Methods and Systems for Efficient Enforcement of Cybersecurity Policies in Network Communications

Abstract

This document discloses methods, apparatus, systems, and computer-readable media for improving packet filtering efficiency by reducing processing time and / or memory usage. The packet filtering appliance uses one of various types of data structures, such as flat hash maps and / or rule trees, to look up packet filtering rules of a cybersecurity policy that should be applied to packets in transit. The packet filtering appliance searches an index data structure for matches of the search object against the matching criteria of the policy rule's threat indicators, in the form of values ​​that the packet filtering appliance extracts from packets in transit. Each of the index data structures maps the rule identifiers (rule IDs) of the policy rules to keys based on (or containing) the matching criteria of those rules.

Description

[Technical Field]

[0001] This disclosure relates to methods and systems for the efficient enforcement of cybersecurity policies in network communications. Cross-reference with related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 547,166, filed on 3 November 2023, and also claims priority to U.S. Provisional Patent Application No. 63 / 468,401, filed on 23 May 2023. Application Nos. 63 / 547,166 and 63 / 468,401 are incorporated herein by reference in their entirety. [Background technology]

[0003] Nothing described in this background art should be treated as an acknowledgment that any system, apparatus, method, or other function described or referred to by such description is prior art, nor should it be treated as an acknowledgment that such description reflects the general knowledge or insights of a person skilled in the art.

[0004] As the information age progresses, the importance of network security is increasing. Network threats / attacks can take various forms, such as malicious requests or data transfers, viruses, malware, and massive amounts of traffic designed to overwhelm resources. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] To counter these threats and attacks, various cyber defense methodologies and systems have been developed and deployed. Some cyber defense systems use packet security gateways to protect networks from threats / attacks. A packet security gateway consists of a network packet filtering device (e.g., an inline non-endpoint device) that mediates data transmission between communication endpoints and has the capability to apply an ordered list of packet filtering rules, which represent and / or may be composed of a network protection policy, to TCP / IP packets in transit, thereby enforcing the policy and protecting the network. If the packet matching criteria for the policy's packet filtering rules originate from cyber threat intelligence (CTI), the packet security gateway may be called a threat intelligence gateway (TIG), and the policy may be called a cybersecurity policy. A TIG may function as a gateway or interface between a network protected or secure by CTI (e.g., a private corporate network connected to the internet) and a similarly protected / unprotected network (e.g., the internet). Therefore, a TIG or multiple TIGs may be located at the boundary between networks or multiple boundaries.

[0006] Packet filtering rules can be derived from a database of threat indicators provided by CTI sources, such as CTI provider organizations. A sequential list of packet filtering rules can represent the cybersecurity policies implemented by TIG to secure / protect the network. Threat indicators are used as packet matching criteria for the rules to identify packets associated with threats. Threat indicators may also be indicators supplied by CTI providers or other CTI sources, including Internet network addresses (IP addresses, IP address ranges, IP subnet addresses expressed in Classless Interdomain Routing (CIDR) notation, L3 A threat indicator may consist of a combination of an IP address, an L4 port, and an L3 protocol type (e.g., a "5-tuple"), a domain name, or a unified resource identifier (which may also be in the form of a URI, e.g., a unified resource locator (URL) and unified resource name (URN)), and may include, for example, the internet network address of a network resource such as a network host computer that may be controlled or manipulated by a threat actor, or a network host computer that may otherwise be associated with malicious activity. Note that, in current disclosures, URIs in CTI are usually URLs (not URNs). Threat indicators may also include identifiers of certificates and associated certificate authorities used to protect certain TCP / IP communications (e.g., X.509 certificates used in the Transport Layer Security (TLS) protocol to protect communications such as sessions mediated by the Hypertext Transfer Protocol (HTTP)). In a cybersecurity context, threat indicators may also consist of geoIP information that associates an IP address with a geographical location, and may also consist of geopolitical information such as country or city.A packet in transit can be said to "match" a "CTI-derived" packet filtering rule, or a CTI-derived rule, if any of the packet field values ​​corresponding to an IP address, 5-tuple, domain name, URI, certificate identifier, or certificate authority match a threat indicator or matching criterion defined by that packet filtering rule. A CTI-derived cybersecurity policy (or CTI-derived policy) consists of a list of CTI-derived packet filtering rules and can be enforced by TIG.

[0007] A CTI provider can associate threat metadata with each threat indicator, which may include, for example, threat / attack type, threat name, threat risk score and / or confidence level, and the attributable threat actor. Furthermore, the CTI provider can provide an information service for searching for threat intelligence reports related to the threat indicator. TIGs and associated applications can use threat metadata and other threat intelligence information to further enhance threat protection and improve the network security posture.

[0008] A network security policy manager can create a CTI-derived network protection / cybersecurity policy as an ordered list of CTI-derived packet filtering rules by receiving threat indicators and associated threat metadata from one or more CTI providers and generating packet filtering rules based on the indicators and metadata. Each packet filtering rule may consist of (a) criteria for matching packet data, such as one or more threat indicators; (b) an action or combination of actions that is applied to the packet if it matches the criteria, including a definitive action of blocking / dropping / denying the packet or allowing / forwarding / passing the packet; and (c) metadata associated with the threat indicator. Generally, a packet filtering device applies a policy to a packet (in transit) by applying matching packet filtering rules to the packet in the order in which the rules are stored in the list. A packet filtering rule may match a packet if the rule's criteria match the value of one or more fields in the packet corresponding to the criteria. Within a policy, each packet filtering rule may be associated with a rule identifier that may be unique relative to other rules in the policy. A rule identifier may be associated with a rank or ranking order that indicates the order in which a set of rules is applied to a packet; for example, a higher-ranked rule may be applied to a packet before a lower-ranked rule is applied to it. In some contexts, a rule identifier may correspond to the rank or position of a rule in an ordered list of rules that make up a policy. For example, the rule at the top of the list may have the highest rank, and the rule at the bottom of the list may have the lowest rank. The ranking order of rules in a list / policy may correspond to the order in which TIG applies matching rules to packets in transit. The matching criteria for each rule consists of one or more pairs of packet field names (e.g., the L3 / Internet layer destination IP address field) and field values.The value of a field(s) can be one or more threat indicators in the form of a network address (e.g., IP address, IP address range, 5-tuple, domain name, URI, etc.) or a certificate or certificate authority identifier. Actions or combinations of actions can consist of combinations of blocking / dropping / denying packets, allowing / forwarding / passing packets, logging packets, capturing packets, redirecting or rerouting packets, or modifying or transforming packets in some way to protect the network. In the context of policies derived from TIG and CTI, rule actions are called packet transformation functions (PTFs) that transform packets to protect the network. Multiple rule actions / PTFs can be combined to further protect the network and can be applied sequentially, concurrently, or both. For example, a rule action / PTF might block a TCP SYN packet that could initiate connection setup with a threat endpoint, but it might also translate the TCP SYN packet into a corresponding TCP RST packet and send it back to the sender of the TCP SYN packet to stop the connection setup process. Another rule action / PTF can log TCP SYN packets and packetize the logs for a network protection application. As another example, a PTF can convert blocked / dropped packets to null. Threat metadata can be used, for example, to inform (human or machine) cyber analysts of the location of relevant cyber threat intelligence reports, to calculate threat risk scores, to select threat analysis techniques to apply to packets and communications associated with those packets, and to select or calculate the action(s) to apply to packets according to the threat risk.

[0009] One or more TIGs can be, for example, inserted inline into one or more Internet access links of a (protected) network. A policy manager associated with one or more TIGs may be configured to (a) receive CTI (e.g., threat indicators and associated metadata) from one or more CTI providers, (b) generate one or more (network protection / cybersecurity) policies consisting of packet filtering rules derived from the threat indicators and metadata, and (c) download one or more policies or forward one or more policies to one or more (subscribed) TIGs and may be associated with the policy manager by policy subscription. TIGs consist of policies and can enforce policies by applying the relevant packet filtering rules to each packet in transit as packets traverse the Internet access link. If a match is found between a rule and a packet in transit, the rule's action is applied to the packet (according to the relevant policy) to protect the network.

[0010] A CTI provider can update the CTI it provides to subscribers by, for example, adding or removing threat indicators that make up an existing CTI feed, providing a new CTI feed, or deleting an existing CTI feed. Therefore, a subscriber policy manager that provides / distributes policies derived from the CTI to the TIG can update the policies with the updated CTI and distribute the updated policies to the TIG. Upon receiving the updated policies, the TIG can process them for efficient enforcement and replace existing enforced policies with the updated policies. The cycle of policy updating, processing, and replacement can be performed regularly, for example, every 15 minutes, depending on the rate at which the CTI provider updates the CTI.

[0011] The effectiveness of network protection using TIG and associated CTI is often a function of the threat indicators and metadata applied by TIG, the scope and quality of associated packet filtering rules, and the performance of TIG. At the time of disclosure, there are hundreds of millions of threat indicators (and associated threat metadata) available from multiple sources of CTI, for example, from hundreds of CTI providers. Threat indicators are in the form of IP addresses, IP address ranges (which can be represented, for example, in CIDR notation), 5-tuples, domain names, URIs, and certificate identifiers, certificate authority identifiers, etc. A large database of such threat indicators can be translated into a similar or corresponding number of packet filtering rules. Because threat indicators observed in the communication traffic of a particular network may not be known before they are observed, all available CTI or associated packet filtering rules can be applied by TIG at any time to ensure effective protection. At the time of disclosure, CTI providers may provide hundreds of millions of threat indicators in total. Therefore, a TIG policy can consist of millions, tens of millions, or even hundreds of millions of rules. Given the growth trend of CTI, the threat indicators available from CTI providers may eventually reach billions, and therefore TIG policies may eventually contain billions of packet filtering rules. Consequently, TIG must be able to apply millions of packet filtering rules to each packet in transit without negatively impacting network performance, for example, without causing latency of more than a few microseconds, and without dropping packets due to, for example, packet buffer overflows in transit (which can occur due to excessive latency). Furthermore, as the number of available CTIs and associated TIG policies increases, the main memory (space) requirements for TIGs for policy enforcement logic may also increase.

[0012] At the time of this disclosure, 100Mb, 1Gb, 10Gb, and / or 40 / 100Gb network links are typically used at TIG insertion points, meaning packet transmission rates can be millions or tens of millions of packets per second. TIG must filter each packet in transit through millions of rules to enforce policies without negatively impacting network performance. Such large policies and high packet rates can lead to insufficient TIG performance and unacceptable performance degradation, even when policies and associated logic are stored in main memory and fast (e.g., sublinear) policy lookup algorithms are used. For example, high latency can lead to packet drops due to packet buffer overflows. Latency during packet filtering should be low, and packets should not be dropped (e.g., due to buffer overflows).

[0013] TIG may be designed, built, and configured to achieve the performance required to apply policies derived from CTI, and the performance of TIG may be measured by the time and (main) memory, or space, required to apply policies to packets in transit, through a combination of (a) a high-performance central processing unit (CPU) and associated computer platform (which may be designed and / or configured for network packet processing) and a software development kit (SDK) designed to leverage the CPU architecture and resources (e.g., multithreading, multicore, high-performance registers and cache) to maximize packet processing performance; (b) a fast / efficient algorithm and associated temporal / spatially efficient data structure for searching rules for matches between the rule indicators / matching criteria and the corresponding field values ​​of the packet currently in transit; and (c) storing the policies (which may include millions of packet filtering rules) and associated policy enforcement logic in fast local memory (e.g., onboard SDRAM, which is often referred to as "main memory") and allowing the CPU(s) to quickly access main memory via a fast, high-bandwidth data bus.

[0014] As of the date of this disclosure, with respect to (a) above, there are several available CPUs / processors, such as the x86 family, ARM family, and PowerPC family, which can be integrated with a computer platform and combined with an SDK such as a Data Plane Development Kit (DPDK) to support the network packet processing requirements of TIG. With respect to (b) above, fast / efficient algorithms and associated data structures for retrieving matching rules in a policy, such as those described in U.S. Patent No. 11,012,417 ("Patent No. 417") (incorporated herein by reference), can support the policy retrieval requirements of TIG. With respect to (c) above, Patent No. 417 describes methods, algorithms, and data structures, such as efficient probabilistic aggregate data structures (e.g., Bloom filters, cuckoo filters, etc.), for reducing the memory requirements of a policy relative to its size, so that (large) CTI-derived policies can be stored in the available main memory of TIG and implemented on network packet traffic while satisfying latency or packet filtering rate requirements.

[0015] However, several current trends in cyber threat intelligence and network protection, such as the continued rapid growth in the volume and diversity of CTI, new and emerging threats, new and emerging threat indicator types (IPv6 metrics, base / registrable domain names, etc.), faster link speeds, and faster network traffic rates, may leave existing methods, algorithms, and data structures insufficient to meet the requirements for implementing modern and future CTI-derived policies regarding network packet communications. For example, overcoming these shortcomings by significantly increasing CPU / processor speed or resources, or by significantly increasing main memory, is often impractical. The negative consequences of these deficiencies may include reduced effectiveness or gaps in network protection against global cyber threats.

[0016] Therefore, there is a need for new technologies (e.g., new, more efficient methods, algorithms, and data structures) to apply comprehensive CTI-derived policies to network communications while meeting performance requirements without significantly increasing TIG's computing resources. [Means for solving the problem]

[0017] This summary is intended to provide a simplified introduction to several concepts as a preliminary step to the "Detailed Description of the Invention (Modes for Carrying Out the Invention)." This summary is not intended to identify any important or essential features.

[0018] Using one or more of the various index data structures disclosed herein, a packet filtering appliance can efficiently retrieve packet filtering rules of a cybersecurity policy for rules that should be applied to packets in transit. The packet filtering appliance can retrieve the index data structures for matches of search objects against the matching criteria of the policy rule's threat indicators in the form of values ​​that the packet filtering appliance extracts from packets in transit. The extracted values ​​and matching criteria consist, for example, of an IP address (or part thereof), a domain name (or part thereof), a URI (or part thereof), and / or other types of values. Each index data structure can map keys based on (or constituting) the matching criteria of a policy rule to the rule identifier (rule ID) of that policy rule.

[0019] The index data structure may consist of a flat hash map that maps keys that make up the hashes of threat indicators from the policy rule matching criteria to the rule IDs of those policy rules. The flat hash map can be efficiently retrieved and / or stored by the packet filtering appliance. The hashes may consist of hashes less than all parts of a value that may be contained in a packet in transit. For example, a flat hash map may map the hash of a threat indicator consisting of a top-level domain (TLD) and one or more subdomains below the TLD from the policy rule matching criteria to the rule ID of that policy rule. However, this threat indicator does not include all subdomains that may be contained in a fully qualified domain name (FQDN) extracted from a packet in transit. As another example, a flat hash map may map the hash of a threat indicator consisting of less than all path segments that may be contained in a URL extracted from a packet in transit to the rule ID of that policy rule from the policy rule matching criteria. When searching a flat hash map associated with such threat indicators, domain names, URLs, or other values ​​extracted from packets in transit may be hashed incrementally, and the flat hash map may be searched for the incrementally hashed portion of the domain name, URL, or other value.

[0020] The index data structure may consist of a flat hash map that maps keys constituting the compression of threat indicators (e.g., lossless compression) from policy rule matching criteria to the rule IDs of those policy rules. Such flat hash maps can also be efficiently retrieved and / or stored by the packet filtering appliance. The compression consists of a compression of less than all portions of a value that may be contained in a packet in transit. For example, a flat hash map may map a compression of a threat indicator from a policy rule matching criteria to the rule ID of a policy rule, which consists of a TLD and one or more subdomains below the TLD, but not necessarily all subdomains that may be contained in an FQDN extracted from a packet in transit. As another example, a flat hash map may map a compression of a threat indicator from a policy rule matching criteria to the rule ID of a policy rule, which consists of less than all path segments that may be contained in a URL extracted from a packet in transit. When searching for such a flat hash map associated with a threat indicator and compression, the domain name, URL, or other value extracted from a packet in transit may be incrementally compressed, and the flat hash map may be searched for the incrementally compressed portion of the domain name, URL, or other value.

[0021] The index data structure may consist of a rule tree that maps bit string representations of threat indicators (such as IP addresses) from the policy rule matching criteria to the rule IDs of those policy rules. The rule tree may be efficiently searched and / or stored by a packet filtering appliance and can be easily updated dynamically at runtime. The rule tree may consist of multiple levels of internal nodes that are searched based on consecutive k-bit chunks of the search object. During the search, at any internal node of the rule tree, multiple rule IDs may be added to a list that makes up the search results based on the match between the k-bit chunk of the search object and the associated value / rule ID associated with the internal node. The search of the rule tree continues after the first match and associated rule ID is found, and other matches and associated rule IDs are found at other internal nodes at deeper levels of the rule tree based on additional parts / chunks of the search object.

[0022] The index data structure search described herein can be combined with membership testing based on set data structure searches. Before searching the index data structure, a search object can be used to search the set data structure to determine whether the search object (or part thereof, or a value based on the search object or part thereof) is a member of all keys represented in any of the threat criteria-rule ID mappings of the relevant index data structure. If the search object (or part thereof, or a value based on the search object or part thereof) is a member of that set, the index data structure search can proceed. If the search object (or part thereof, or a value based on the search object or part thereof) is not a member of that set, the index data structure search is unnecessary and can be skipped to save processing time. Alternatively, the set data structure search and the related index data structure search can be performed simultaneously. If the set data structure search indicates that the search object (or part thereof, or a value based on the search object or part thereof) is not a member of all keys represented in the relevant index data structure, the related index data structure search can be terminated before completion, thereby saving processing resources.

[0023] The index data structure can be selected for each of several types of threat indicators (e.g., a first type of threat indicator consisting of URLs, a second type of threat indicator consisting of domain names, a third type of threat indicator consisting of first type IP addresses, a fourth type of indicator consisting of second type IP addresses, etc.), thereby facilitating more efficient use of processing and memory resources. Multiple instances of a particular type of data structure may be used for a particular type of threat indicator (e.g., an instance for a threat indicator of a particular type that shares a first common characteristic, and an instance for a threat indicator of that particular type that shares a second common characteristic), or for different types of threat indicators (e.g., a first rule tree for IPv4 threat indicators and a second rule tree for IPv6 threat indicators). Alternatively, multiple instances of different types of index data structures may be used for a particular type of threat indicator. A single data structure instance may be used for multiple types of threat indicators (e.g., a single flat hash map used for both domain name and URL threat indicators). The packet filtering appliance can use multiple concurrent processing threads to simultaneously look up multiple index data structures based on different values ​​extracted from packets in transit. Each of these concurrent processing threads can output one or more rule IDs determined based on the index data structure lookup performed in that processing thread (or it can determine that no rule IDs were found by the lookup). The rule IDs determined by each processing thread (if any) are collected, and rules corresponding to some or all of those rule IDs may be applied to packets in transit based on the rank / priority of those rules.

[0024] These and other features will be explained in detail below. [Brief explanation of the drawing]

[0025] Some features are shown in the figures of the attached drawings and by examples in which similar reference numbers refer to similar elements, but are not limited to those examples. [Figure 1] Figure 1 shows an example of a network environment for efficient cybersecurity policy enforcement. [Figure 2A] Figure 2A is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2B] Figure 2B is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2C] Figure 2C is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2D] Figure 2D is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2E] Figure 2E is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2F] Figure 2F is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 2G] Figure 2G is a flowchart illustrating an example of how to retrieve a flat hash map index data structure based on a key derived from a value obtained from a field of a packet in transit. [Figure 3A] Figure 3A shows an example of a rule tree and its associated data structure. [Figure 3B]Figure 3B is a flowchart showing an example of how to insert a rule tree key. [Figure 3C] Figure 3C is a flowchart showing an example of how to search a rule tree like the one in Figure 3A. [Figure 3D] Figure 3D is a flowchart illustrating another example method for searching a rule tree, similar to the rule tree in the example in Figure 3A. [Figure 4] Figure 4 is a block diagram illustrating an example of a method for efficient policy enforcement, comprising selecting and using one or more data structures and associated usage methods described herein. [Figure 5] Figure 5 is a block diagram showing an example of how to select a data structure for a network protection policy. [Figure 6] Figure 6 is a block diagram showing an example of how to input the data structure for a network protection policy. [Figure 7] Figure 7 is a block of images showing an example of policy search and enforcement. [Figure 8] Figure 8 is a flowchart illustrating an example method for calculating the memory requirements for a collection of data structures (e.g., index data structures and set data structures) selected for a network protection policy. [Figure 9] Figure 9 is a block diagram showing an example of how to determine the k-parameter values ​​of a rule tree. [Figure 10] Figure 10 is a block diagram showing an example of a computing device. [Figure 11] Figure 11 is a block diagram showing an example of a packet filtering appliance. [Figure 12] Figure 12 illustrates an example of a network environment in which rule trees and / or flat hash maps can be used as membership test set data structures, for example, in relation to the efficient cyber protection of a corporate network associated with mobile devices. [Figure 13]Figure 13 is a flowchart illustrating an example of a method for configuring mobile devices and associated enterprise network elements. [Figure 14] Figure 14 is a flowchart illustrating an example of a packet filtering method in a corporate network associated with mobile devices. [Figure 15] Figure 15 shows another example of a network environment where a rule tree and / or a flat hash map are used as the membership test set data structure. [Figure 16] Figure 16 is a flowchart illustrating an example of how to configure a remote network, associated tunnel gateways, and associated central network elements. [Figure 17] Figure 17 is a flowchart illustrating an example of a method for packet filtering and efficient backhaul processing for network protection in a remote network. [Figure 18] Figure 18 shows an example of another network environment in which a rule tree and / or a flat hash map can be used as a set membership test data structure. [Figure 19] Figure 19 is a block diagram showing an example of a DNS gatekeeper. [Figure 20] Figure 20 is a flowchart showing an example of how to operate a DNS gatekeeper. [Figure 21] Figure 21 is a flowchart showing another example of how to operate a DNS gatekeeper. [Figure 22] Figure 22 is a flowchart showing an example of how a DNS filter administrator operates. [Figure 23] Figure 23 shows an event sequence for an internetwork with a DNS gatekeeper that can potentially mitigate or prevent DDoS attacks on DNS. [Figure 24] Figure 24 shows an event sequence in an internetwork with a DNS gatekeeper that mitigates or prevents DNS tunneling attacks. [Modes for carrying out the invention]

[0026] The embodiments described herein generally relate to computer hardware and software, as well as network security. In particular, one or more embodiments of this disclosure generally relate to computer hardware and software for efficiently filtering packets in transit based on packet filtering rules derived from cyber threat intelligence (CTI). Disclosed herein are methods, apparatus, systems, and machine-readable media relating to efficient packet filtering for CTI-based applications, including, for example, the enforcement of CTI-derived cybersecurity policies on packets in transit by TIG. The methods, apparatus, systems, and machine-readable media disclosed herein may facilitate improvements in packet filtering efficiency, which may include improvements in processing time and / or memory usage.

[0027] In this detailed description, reference is made to the accompanying drawings, which constitute part of this specification and illustrate various embodiments in which aspects of this disclosure may be carried out. It should be understood that the implementation of this disclosure may take other forms and that structural and functional modifications may be made without departing from the scope of this disclosure. Furthermore, specific uses, protocols, and embodiments in which aspects of this disclosure may be carried out are mentioned. It should be understood that other uses, protocols, and embodiments may be utilized and structural and functional modifications may be made without departing from the scope of this disclosure.

[0028] The various connections between elements are described below. These connections are general in nature and, unless otherwise specified, can be direct or indirect (e.g., connected via one or more networks), wired or wireless, physical or logical (e.g., virtual or software-defined), or any combination.

[0029] Figure 1 shows an example network environment 100 for more efficient cybersecurity policy enforcement, and some or all aspects of this disclosure can be implemented. The network environment 100 consists of multiple private networks 101.1 to 101.n, where "n" can be any integer value. Any one of networks 101.1 to 101.n may be referred to as network 101, and networks 101.1 to 101.n may be collectively referred to as network 101. The same convention is used for other elements with similar numbering in the figure.

[0030] Each of the networks 101 may be an enterprise network that includes, for example, one or more packet filtering devices that enforce one or more cybersecurity policies using one or more methods described herein. In the network environment 100, network 101.1 configures TIG102.1 at the boundary between network 101.1 and network 105, network 101.2 configures TIG102.2 at the boundary between network 101.2 and network 105, and network 101.n configures TIG102.n at the boundary between network 101.n and network 105. Network 105 may be, for example, a public network such as the Internet and / or an unprotected network. TIG102 (which may be configured as a gateway or other interface) may be configured such that all traffic between a protected network 101 and an unprotected network (such as network 105) must pass through TIG102, thereby protecting data communications across the boundary between network 101 and the unprotected network. The network environment 100 may further include one or more CTI providers (CTIPs) 106 and one or more security policy management servers (or services) (SPMSs) 107 connected to network 105. The network environment 100 may also include one or more additional networks 109.1 to 109.n connected to network 105. Although shown for convenience in Figure 1 as directly connected to network 105, the CTIPs 106 and SPMSs may be hosts in other networks connected to network 105. The network environment 100 may consist of other types of computing devices connected to network 105 (or connected as part of a network connected to network 105), such as Domain Name System (DNS) servers / devices, Security Operations Center (SOC) servers / devices, Global Threat Context Servers (GTCS), and / or other devices.

[0031] Network 101 may consist of hosts H (e.g., servers, laptops or desktop computers, and / or other networked computing devices) configured to communicate with hosts within Network 105 or with networks connected to Network 105 via Network 105. Some hosts in Network 105 and / or with networks connected to Network 105 may be malicious. For example, Network 109 may contain a malicious host MH that incites, assists, or otherwise associates one or more types of network threats or attacks. To detect, prevent, stop, and / or otherwise address such network threats / attacks, each TIG 102 can filter data traffic (incoming and outgoing) between hosts H within Network 101 and hosts outside of Network 101. Herein, TIG 102 is used as an example of a packet filtering appliance configured to perform the operations described herein, but other types of packet filtering appliances may also, or alternatively, be configured to perform the operations described herein for TIGs.

[0032] CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP(or CTIP))))))) ) ) ) ) ) ) ) ) ) ) CTIP(or CTIP(or CTIP(or CTIP) ) ) ) ) ) ) ) ) CTIP(orCTIP(or CTIP(or CTIP) ) ) ) ) CTIP(or CTIP(or CTIP) ) ) ) ) CTIP(or CTIP(or CTIP) ) ) ) ) CTIP(or CTIP(or CTIP) ) ) ) ) CTIP(or CTIP(or CTIP) ) ) ) CTIP(or CTIP(or SPMS(plural)107 and / or TIG can also determine an index data structure used to map search keys (e.g., corresponding matching criteria from a rule) to rule identifiers, determine a set data structure (e.g., membership tests) for determining whether a set of rules contains criteria that would match a particular value, populate the index and set data structures, and / or perform other operations.

[0033] Each TIG102 may receive policies / updates, store policies / updates, and / or filter packet traffic based on the packet filtering rules of those policies / updates. Traffic may consist of packets in transit sent from (or from) a host(s) H in network 101 protected by its TIG102 to an external host outside of network 101 (e.g., a host in network 105 or a host in another network connected to network 105). Packet filtering rules may include packet matching criteria having one or more pairs of packet field names and values ​​based on one or more threat indicators. Examples of threat indicators, but not limited to, include IP addresses, IP address ranges, IP subnet addresses expressed in CIDR notation, combinations of L3 IP addresses with L4 ports and L3 protocol types (e.g., "5-tuple"), domain names or parts of domain names, URIs (e.g., URL or URN) or parts of URIs, certificates and / or associated certificate authority identifiers, geoIP information, and / or other types of threat indicators. Packet matching criteria based on threat indicators may consist of the indicator's value (or a portion of the indicator's value) and / or values ​​derived from the indicator's value (or a portion of it). A packet matching criterion in a packet filtering rule means that a packet may match if it contains a value in a field or field indicated by the criterion that matches the criterion's value (directly or after processing such as hashing), or a value that falls within the range indicated by the criterion's value. If a packet matches one or more packet matching criteria of a rule, the TIG102 may perform one or more actions specified by the rule, such as blocking / dropping / denying the packet, allowing / forwarding / passing the packet, logging the packet, capturing the packet, redirecting or rerouting the packet, modifying or transforming the packet in any way to protect the network, generating and / or sending a response to the packet, etc.

[0034] As part of filtering packet traffic, and to determine whether a policy consists of rules with packet matching criteria that may match a packet, TIG102 may perform one or more methods described herein. These methods may include searching a set data structure to determine the membership of values ​​in the relevant index data structure for the policy rule that match values ​​based on the packet's fields. A packet field-based value (packet field-based value or field-based value) may consist of the actual value (or a portion of the actual value) of that field, and / or a value derived from the actual value (or a portion of the actual value) of that field (e.g., a hash value or a compressed data value). The methods performed by TIG102 may also, or alternatively, include searching the relevant index data structure for values ​​that match the field-based value, and, if a matching value is found, determining the rule identifier corresponding to that matching value. As also described herein, searching a set data structure may be omitted, and / or may be performed before or concurrently with searching the index data structure associated with that set data structure.

[0035] TIG102 can filter a large volume of packets based on a large number of policies and rules. However, in reality, only a small fraction of the packets in transit filtered by TIG102 typically match the packet filtering rules of the cybersecurity policies enforced by TIG, which originate from CTI. Therefore, a time- and space-efficient method for determining whether a packet in transit matches (or does not match) any cybersecurity policy rules before searching (or not searching) the security policy rules could eliminate much of the policy lookup and thus reduce the average or expected time to filter packets.

[0036] A set data structure with time- and / or space-efficient set operations for inserting elements into a set, removing elements from a set, and testing elements for membership or presence within a set can support such methods. Each CTI-derived packet filtering rule in a policy may be characterized by one or more threat indicators that constitute the packet matching criteria for the rule. For each CTI-derived rule in a policy, the associated threat indicator (or a value based on a threat indicator) may be inserted as an element into one or more set data structures. When filtering packets in transit, TIG102 may first perform a membership test to determine whether the packet matches any rule in a policy before searching the policy, by testing whether a value based on any of the packet's field values ​​corresponding to a threat indicator value (e.g., IP address, 5-tuple, domain name, URI, certificate identifier, certificate authority, etc.) is a member of or exists within a set data structure(s). If the membership / existence test returns True, the policy lookup for the associated index data structure can proceed; otherwise (i.e., if the membership / existence test returns False), the policy lookup can be skipped, and the TIG102 can immediately send the packet to its destination.

[0037] If membership / existence tests are expected to be on average / significantly faster than lookups through the associated policy index data structure, the expected policy lookup time per packet can be significantly reduced when averaged over a sufficiently large volume of representative packet traffic. Furthermore, the overall packet filtering time can be further reduced by starting and running membership / existence tests concurrently with the policy lookup, or in other ways, instead of waiting for the membership / existence tests to complete before initiating the policy lookup. If the membership test returns True, the policy lookup completes earlier (by a time commensurate with the time the membership test takes to run). If the membership test returns False, the policy lookup may be abandoned or otherwise ignored (and, for example, TIG102 may immediately send the packet toward its destination without waiting for the policy lookup to complete).

[0038] In addition to supporting fast / time-efficient membership / existence determination, the relevant set data structures can also support space efficiency; for example, the memory requirements for storing elements of a set may be relatively small and may increase slowly, for example, non-linearly or by small constant multiples, as elements are added / inserted into the set. Data compression techniques may also be effective in controlling memory requirements, but often need to be balanced with time efficiency. Furthermore, the type of set data structure for membership / existence testing may differ depending on the type of threat indicator. For example, in the case of IPv4 addresses represented in 32 bits (potentially 2^32 IPv4 addresses), a bit array of length / size 2^32 bits (requiring 4 Gbit, or 512 MBytes of memory) might be a good choice of set data structure because it can support very fast element insertion, element deletion, and element membership testing. Each bit in such an array corresponds to one of 2^32 distinct IPv4 addresses and can hold a value (e.g., 1) indicating that the corresponding IPv4 address is part of a set (e.g., a set of IPv4 addresses that are part of a policy rule matching criterion), or a value (e.g., 0) indicating that the corresponding IPv4 address is not part of the set. To insert (or remove) an IPv4 address from the set, set the bit value in the array at the index or position corresponding to the 32-bit representation of that IPv4 address to 1 (or 0). Similarly, an IPv4 address / element membership test reads the bit value in the array at the index or position corresponding to the 32-bit representation of the IPv4 address and returns True if the bit value is 1, and False if it is 0.

[0039] A bit array set data structure for IPv4 addresses may be time- and space-efficient for relevant applications (e.g., enforcement of cybersecurity policies by TIG), but not for other types of threat indicators. For example, an IPv6 address represented in 128 bits would require a corresponding bit array of length / size 2^128 bits, which may be impractical (and therefore space-inefficient). However, a bit array set data structure is still effectively usable for IPv6 addresses. For example, a bit array of length / size 2^32 bits (requiring 4 Gbit, or 512 MBytes of memory) could be allocated, and the bit value of the array at the index or position corresponding to the / 32 prefix bit representation of the IPv6 address (i.e., the most significant 32 bits of the IPv6 address) could be set to 1 in order to insert an IPv6 address / element into the bit array / set. Similarly, a membership test for an IPv6 address / element could read the bit value of the array index or position corresponding to the / 32 prefix bit representation of the IPv6 address, returning True if the bit value is 1 and False if it is 0. However, IPv6 addresses in such bit arrays can lead to performance degradation, such as element membership lookups returning a non-zero false positive rate or the inability to dynamically remove IPv6 addresses from the bit array.

[0040] Note that the above adaptation of element membership tests for IPv6 address prefixes may also be applied to IPv4 addresses to reduce memory requirements. For example, if a 2^32 bit array (requiring 4 Gbit, or 512 MBytes of memory) for membership tests of a full IPv4 address is considered too large, a smaller array can be used instead by storing elements corresponding to the IPv4 address prefix. For example, a bit array of length / size 2^30 bits (which requires 1 Gbit, or 128 MBytes of memory) may be allocated, and the bit value of the array index or position corresponding to the / 30 prefix bit representation of the IPv4 address (i.e., the most significant 30 bits of the IPv4 address) may be set to 1 in order to insert the IPv4 address / element into the bit array / set. Similarly, the IPv4 address / element membership test reads the bit value of the array index or position corresponding to the / 30 prefix bit representation of the IPv4 address, and returns True if the bit value is 1, and False if it is 0. The IP address can be a full IP address or a partial IP address. A full IP address is an IP address represented by 32 bits (IPv4) or 128 bits (IPv6). In CIDR / prefix notation, a full IP address is represented as a / 32 (IPv4) or / 128 (IPv6) address. A partial IP address may consist of an IP address prefix, which is an IP address represented by 32 bits or less (IPv4) or 128 bits or less (IPv6), with missing bits being the least significant bits. An IP address prefix may be denoted as / X, where X is a number 32 or less (IPv4) or 128 or less (IPv6), and "X" refers to the X most significant bits (MSB) of the IP address. An IP address prefix may be associated with a subnet address.

[0041] For other types of threat indicators, such as domain names and URIs, bit array set data structures are similarly inefficient and may not have a direct mapping from the threat indicator to a bit array index. Therefore, other types of set data structures can also be used to perform membership / existence tests. For example, the 417 patent describes methods, algorithms, and data structures that may be used to perform membership / existence tests on domain name and URI threat indicators (and IP address threat indicators, as well as other types of threat indicators), such as efficient probabilistic set data structures (e.g., Bloom filters, cuckoo filters, etc.). In such probabilistic set data structures, time efficiency and spatial efficiency can be functions of the false positive rate (e.g., a false positive occurs if the membership / existence test returns True for an element not included in the set). Generally, increasing the false positive rate increases the time and spatial efficiency of these probabilistic set data structures. However, each false positive may cause unnecessary policy lookups that may not find a match between the packet and the packet filtering rule; therefore, the false positive rate may be chosen so that the time spent performing unnecessary policy lookups does not unnecessarily increase the expected packet filtering time.

[0042] As mentioned above, there is also a need to improve the temporal and spatial efficiency of policy search. For example, it may be impractical to meet some of the requirements for implementing the latest / future CTI-derived policies by significantly increasing CPU / processor speed and resources, and / or significantly increasing main memory. Failure to meet these requirements may reduce the effectiveness of network protection from global cyber threats or create gaps.

[0043] While techniques such as bloom filters and cuckoo filters can significantly improve the temporal and spatial efficiency of policy discovery, highly temporal and spatially efficient packet filtering methods based on probabilistic set data structures may have problems related to the type of threat indicator. For example, probabilistic set data structures may be inefficient or impractical for processing IPv4 and IPv6 threat indicators expressed in CIDR notation, such as subnet addresses. Another example is that probabilistic set data structures may be partially inefficient when storing domain name threat indicators (and URI threat indicators that may constitute domain name threat indicators), for example, because they do not take advantage of the redundancy or similarity that may occur across the entire set of domain name threat indicators. Specifically, this is the case when the information entropy of the set is relatively low. For example, they may not take advantage of the redundancy of valid top-level domains (eTLDs) or commonly used prefix subdomain labels (such as "www", "mail", "remote", "blog", "webmail") across the entire set of domain name threat indicators. As yet another example, CTI for domain names is often provided in the form of “base domain names,” “registrable domain names,” or “eTLD+1” domain names, which effectively have “starred” prefix subdomains that may not be fully qualified domain names (FQDNs). Base / registrable domain names can introduce further inefficiencies when probabilistic set data structures are used in packet filtering techniques. As yet another example, CTI for URLs may often be provided in the form of partial paths, or effectively “starred” paths, i.e., paths where only a subset of the path segments that make up the URL path can constitute the URL. Also, the aforementioned membership / existence testing method, which first checks whether the network address of a packet that may correspond to a threat indicator exists in the policy rules before searching the policy for a matching rule, means that each threat indicator may be stored redundantly, i.e., twice, in main memory.Such inefficiencies may prevent the temporal and / or spatial requirements necessary for policy implementation from being met. For example, there may be insufficient main memory to store policies, policy retrieval times may be too long, or both.

[0044] This specification describes a novel, efficient index data structure and associated algorithm that can address the inefficiencies and gaps in threat indicator processing associated with probabilistic set data structures while meeting the time and space requirements for CTI-derived policy enforcement by TIG. Generally, an index data structure can map keys to values. For example, in the context of searching for and enforcing CTI-derived policies, the key may be a threat indicator (or a value derived from a threat indicator) associated with one or more packet filtering rules in the policy, and the value may be a packet filtering rule identifier. Then, for example, a policy search through the index data structure for a potential threat indicator may output one or more rule identifiers, if the threat indicator is comprised of the packet matching criteria for each rule in which it is identified. If the output is empty (e.g., no rule identifiers), the potential threat indicator is not in the policy.

[0045] These new index data structures for efficient policy lookup include “rule trees” that are based on prefix tree index data structures and adaptive cardinality tree index data structures adapted, extended, parameterized, and / or optimized for using threat indicators, and may include flat hash maps that map keys, which are bit string representations of threat indicators, to rule identifiers, and keys, which are hashed or compressed threat indicators, to rule identifiers.

[0046] One problem with matching threat indicators in packet filtering rules with domain names and URIs contained in packets (in transit) is that (1) domain name threat indicators may be provided in the form of base / registrable domain names or "starred" domain names (e.g., domain names that do not have a clear entry in DNS, such as *.example.com), and may never be observed in the packet; and (2) similarly, URL threat indicators may be provided in "starred" paths, for example, only a portion of the path segment that makes up the complete path may be represented by the URL indicator (e.g., http: / / www.example.com / path-segment-1 / *), which may also never be observed in the packet. However, in both cases, domain names and URLs contained in packets in transit that partially match these starred indicators may be treated as matching the corresponding packet filtering rule. For example, the (fully qualified) domain name "www.example.com" may be observed in a packet in transit and may match a packet filtering rule with the matching criterion "example.com". Similarly, a complete or full URL http: / / www.example.com / path-segment-1 / path-segment-2 / path-segment-3 that may be observed in a packet in transit may match a packet filtering rule with the matching criterion http: / / www.example.com / path-segment-1 / .

[0047] Therefore, for both domain names and URLs, threat indicators of the domain name and URL type can often be considered as prefixes of domain names and URLs that may be contained in packets (in transit). In packet filtering applications such as cybersecurity policy enforcement, a match between domain name and URL threat indicators that constitute the matching criteria of a packet filtering rule and the corresponding values ​​contained in a packet (in transit) can be determined if the matching criteria may be prefixes of values ​​contained in the packet. Similarly, IP threat indicators are often provided in the form of prefixes, such as CIDR notation, and likewise, a match between IP threat indicators that constitute the matching criteria of a packet filtering rule and the corresponding values ​​contained in a packet (in transit) can be determined if the matching criteria may be prefixes of values ​​contained in the packet.

[0048] Another issue with domain name and URI threat indicators is that their native character / text representations (e.g., the representations used in packets) can have higher memory / space requirements for storage in data structures compared to a typical TIG main memory budget for cybersecurity policy enforcement. For example, at the time of this disclosure, a CTI provider may be supplying tens of millions of domain name indicators (e.g., 50 million) and hundreds of millions of URI indicators (e.g., 200 million) in total, and these indicators, in their raw form, could require more than 20 GB of main memory / space, while a typical TIG main memory budget for cybersecurity policy enforcement may be 5-10 GB.

[0049] One or more of these problems can be addressed by using a flat hash map index data structure to look up policy rule IDs. A flat hash map index data structure can map threat indicators (such as domain names and URIs) and / or hashes of parts of threat indicators to rule IDs. Alternatively, a flat hash map index data structure can map threat indicators (e.g., domain names and URIs) and / or compressed parts of threat indicators (e.g., lossless compression) to rule IDs. As described below, leveraging domain name and URI structures and combining them with hashing and / or lossless compression methods can support efficient storage and efficient retrieval of these types of threat indicators. A flat hash map index data structure can significantly reduce main memory requirements within a typical TIG budget while supporting efficient matching between partial domain name and URI indicators in CTI, and between complete domain names and URIs observed in packets in transit. A flat hash map index data structure simultaneously leverages the structures of domain names and URIs and the data compression effects of (lossy) hashing and lossless compression methods.

[0050] In an exemplary flat hash map, each key / indicator is represented, for example, as a 64-bit (8-byte) hash, or fingerprint, and mapped to a hash table of a size corresponding to the number of keys. At the time of disclosure, considering that the length of a domain name in CTI is typically 7–27 octets / byte, up to a maximum of 255 octets / byte, and the length of a URI in CTI is typically 35–175 octets / byte, up to a maximum of 2048 octets / byte, a 64-bit / 8-byte representation, for example, for domain names and URIs in CTI can result in a significant reduction in memory / space requirements. However, it should be noted that the hash collision rate changes inversely with the fingerprint size. Therefore, for example, a 32-bit fingerprint size can reduce memory requirements to about half the memory required for a 64-bit fingerprint, but the collision rate may actually be too high. Approaches for managing / balancing the collision rate and fingerprint size (and thus memory requirements) are described below.

[0051] The exemplary technique used to create a fingerprint of a domain name is sometimes called "incremental label hashing," and similarly, the exemplary technique used to create a fingerprint of a URL is sometimes called "incremental path segment hashing."

[0052] Incremental label hashing can be used to create a fingerprint of a domain name. A domain name can be represented as a series of subdomain labels separated by dots ".", where the rightmost label represents a top-level subdomain (e.g., "com", "net", "edu", "org", etc.) relative to the (implicit) root domain. For example, domain name D i i subdomain labels L i Li-1 ...As a list L1, each label is represented by a dot character ".", and H is a hash function that outputs an X-bit hash (e.g., a 64-bit hash) for an input of any size. Examples of such hash functions include xxh3, murmur2, abseil, and anker1, which are commonly available in the standard library, but are not limited to these. "||" is used as the concatenation operator. Domain name D i The process for creating the fingerprint is defined recursively as follows:

number

number

[0053] Since domain names are processed from the rightmost / topmost subdomain label for each subdomain label, when applying a CTI-derived policy to packet traffic in transit, the fully qualified domain name (FQDN) contained in the packet can be easily and efficiently matched with any CTI subdomain name or base / registrable domain name that may constitute the policy's rules. For example, if the domain name CBA (but not EDCBA, DCBA, BA, or A) is in the CTI, and the rule ID R is configured in the policy... kAssume it is related to a packet filtering rule having. When a policy is created and processed for efficient enforcement, the fingerprint H(CBA) key and the rule ID R k The value can be inserted into the flat hash map index data structure F. Assume that the domain name E.D.C.B.A is included in the packet (during transfer). The TIG policy enforcement logic can efficiently check whether E.D.C.B.A matches one or more packet filtering rules in the policy using the following exemplary policy search algorithm (pseudo code).

Number

[0054] Note that in each of the above steps, the hash function H is calculated at most twice because it processes each subdomain label. Therefore, this algorithm is efficient in both time and memory / space because the time complexity is a linear function of the number of subdomain labels and the space complexity is a constant function of the hash size X. Also note that the corresponding algorithm for inserting the domain name indicator into the flat hash map index data structure has a similar time and space complexity. Also note that a variation of the fingerprint creation process is as follows when the hash for each label is not required.

Number

[0055] Incremental path segment hashing can be used to create a fingerprint of the URL. URLs in CTI are often <scheme> : / / <host> <path> <query>It may take the form of this. Here, <scheme>In many cases, this may be "http" or "https" (as of the time of this disclosure, URLs in CTI are primarily schemed as http or https), <host>This may be a hostname (e.g., domain name Di) or an IP address. <path>P j This consists of j path segments S1S2S3...S j It may also be a sequence of each path segment S i It may also begin with a slash " / " character. <query>This may also be a string consisting of a sequence of pairs of parameter names and their associated values. <query>While often ignored in the context of CTI applications such as cybersecurity policy enforcement, query strings can be processed in a similar manner to labels or path segments, where applicable. For example, <query>This is processed incrementally in units of {parameter name, value} pairs. URL U ij =D i P j of <host> <path>The process of creating a fingerprint for a part is similar to the incremental label hashing process described above for creating a fingerprint for a domain name. ij ) is, for example, H(U ij ) = H(H(D i ) ||H(P j )) or can be calculated as similar, where H(D i ) can be calculated as above, and H(P j ) is defined recursively as follows:

number

number

[0056] For the purpose of generating hashes to map to rule IDs within a flat hash map index data structure, and for the purpose of incrementally hashing search objects to search that flat hash map index data structure, domain names (or URLs in domain name format) <host>The URL path portion can be processed from right to left (for example, from the top-level domain (or TLD) to the lowest-level subdomain). Conversely, the URL path portion can be processed from left to right for the purpose of generating hashes to map to rule IDs in a flat hash map index data structure, and for the purpose of incrementally hashing search objects to look up that flat hash map index data structure. More generally, URL Uij=D i P j In the case of D i =L i ·L i-1 ·L···L3·L2·L1 and P j =S1 / S2 / ··· / S j Therefore, the domain name label L and the path segment S may be processed in this order.

number

number

number

[0057] By using the incremental hashing described above, you can create rules based on domain name CTI (for example, rules where the matching criterion is a domain name) and rules based on URL CTI (for example, rules where the matching criterion is a URL). <host>and <path>A single, flat hash map index data structure can be used for a set of rules (which are sub-rules). This simplifies the search logic (e.g., using the same search logic for different types of threat indicators) and reduces memory usage (e.g., using a single index data structure for rules with different types of matching criteria). URL U ij Fingerprint H(U ij An efficient approach to calculating ) is as follows: (1) Constitute a single instance of a flat hash map to store and process all relevant domain name CTIs (and associated policy rules) and all relevant URL CTIs (and associated policy rules). (2) Hash-based key K c Using the above formula, the domain name D i and URLD i P j Calculate the fingerprint of the domain name D. i =L i .L i-1 ...For L3, L2, and L1, the fingerprint added to a single flat hash map is H(L i ||H(L i-1 ||...||H(L3||H(L2||H(L1)))...)) and URLU ij =D i P j =L i .L i-1 ....L3.L2.L1 / S1 / S2 / ... / S j-1 / S j The fingerprint added to a single flat hash map for H(S) j ||H(S j-1 ...||H(S2||H(S1||H(L i ||H(L i-1 ||...||H(L3||H(L2||H(L1))))...))

[0058] As with the domain name mentioned above, the URL <path>Since each path segment is processed starting from the leftmost path segment, when applying a CTI-derived policy to packet traffic in transit, the complete URL contained in the packet can be easily and efficiently matched with the URL in CTI that has the subpaths that make up the policy rules. For example, the following can be found in a packet in transit: <host>EDCBA and <path>URLs with / S1 / S2 / S3 are the same <host>Has EDCBA, but (partial) <path> / S 1 / This may match a packet filtering rule that uses CTI URLs with S2 as a matching criterion.

[0059] URL <host>Note that the part can also be an IP address (IPv4 or IPv6). In this case, the IP address may be incrementally hashed in a way similar to the incremental label hash of a domain name, but proceeding from left to right. For example, an IPv4 address can be represented in dotted decimal notation, which is a sequence of four integers N, each in the range [0, 255], and each integer is separated by a dot "." character. Thus the URL <host>If the part is an IPv4 address, the IPv4 address is treated similarly to a domain name, with the four integers corresponding to four subdomain labels separated by a dot "." character, and the four integers numbered from left to right (e.g., N1.N2.N3.N4). The hash-based key K above. c In the formula, for example, <host>When incrementally hashing a URL that has an IPv4 address as part of it, i = 4, and L1 to L4 become N1 to N4 respectively. An IPv6 address can be represented by eight groups of four-digit hexadecimal numbers, each group separated by a colon ":". Therefore, the URL <host>If the part is an IPv6 address, the IPv6 address is treated similarly to a domain name, with the eight groups corresponding to eight subdomain labels, separated by colons ":" instead of periods ".", and the eight groups numbered from left to right (e.g., H1:H2:H3:H4:H5:H6:H7:H8). The hash-based key K above. c In the formula, for example, <host>When incrementally hashing a URL that has an IPv6 address as part of it, i=8, and L1 to L8 become H1 to H8 respectively.

[0060] Note that URL formats found in CTIs, such as email addresses, VoIP addresses, and messaging addresses, can also be processed similarly using the techniques described above. Such URLs are: <scheme> : <name> @ <host>This format can be adopted, and here, <scheme>Examples include "mailto", "sip", "sips", "h323", "im", and "xmpp". <name>For example, a typical username is <host>This is the hostname (for example, the domain name D as shown above). i ). A similar approach to the fingerprinting approach described above can be used to handle such URLs.

[0061] When processing policies originating from CTI for efficient enforcement, the domain names and URLs of the packet filtering rules comprised by the policy may be fingerprinted as described above, and the fingerprint (key) may be paired with the associated rule identifier / rule ID (value), and the {fingerprint, rule ID} pair may be inserted into a hash table, which may be a flat hash map index data structure. However, flat hash maps may not natively support efficient dynamic runtime updates of packet filtering rules for policies currently being executed by TIG (see explanation below). If dynamic runtime policy updates are required, an alternative index data structure that natively supports dynamic runtime updates, such as a rule tree (described below), can be used instead. In this case, the {fingerprint, rule ID} pair is inserted into a rule tree instead of a hash table. This index data structure is a rule tree that stores keys, which are hash-based fingerprints in bit string representation, and is sometimes called a fingerprint rule tree. Note that fingerprint rule trees may have different memory requirements than the corresponding flat hash map, and the trade-off is support for dynamic runtime updates.

[0062] In some cases, hash-based fingerprints and associated index data structures can have problems related to hash collisions. For example, as an index data structure contains more fingerprints, hash collisions between a fingerprint H(A) already contained in the index data structure and a fingerprint H(B) being inserted or retrieved (where A and B can be domain names or URLs) become more likely, meaning the hash collision rate can increase. Because hash functions are irreversible and can be irreversible, it can be difficult or impossible to determine whether A is equal to B or not. The hash collision rate can be reduced, for example, as follows: Collisions can be reduced by (1) limiting the number of entries / keys / elements in the associated index data structure, and / or (2) using a perfect (collision-free) hash function, and / or (3) partitioning the domain name and URL threat indicator / element set and inserting the elements within each partition into different instances of the index data structure, and / or (4) increasing the hash size, and / or (5) using multiple different hash functions or multiple different fingerprints for each key, combined with logic to reduce the effective collision rate. However, it should be noted that there are trade-offs to these exemplary methods. For example, (1) and (2) may not be practical depending on the application, (3) may increase time requirements, memory requirements and logical complexity, and (4) and (5) may increase memory requirements. On the other hand, these techniques can be used to reduce the effective collision rate so that true collisions, i.e., H(A)=HB) but A is not equal to B, do not actually occur or occur very rarely, while still meeting time and memory / space requirements.

[0063] In the context of this disclosure, for an X-bit fingerprint, if multiple different hash functions are used to generate the X-bit fingerprint, the effective collision rate may be lower than if a single hash function is used to generate the X-bit fingerprint. This can be caused by several factors, but the main factor is that, for temporal performance reasons, an NCHF may be used to compute the fingerprint instead of a cryptographic hash function (CHF), which is relatively slower than a non-cryptographic hash function (NCHF), and this relative slowness may negatively impact performance in the context of at least some of the applications described herein. If a (strong) CHF is available, an effective way to reliably reduce the collision rate is to increase the hash size (and fingerprint size) while using a single CHF, as described in (4) above. However, compared to a strong CHF, the estimated collision rate of an NCHF is less reliable, dependent on the data being hashed, and can vary. Also, different NCHFs may have different data dependencies and variability. Furthermore, since NCHFs available in the standard library are often optimized for specific hash sizes, an NCHF optimized for calculating a 64-bit output hash may perform poorly (e.g., have a higher collision rate) when generating hashes of a different size. To mitigate the negative impact of these various factors on the collision rate, multiple different NCHFs can be selected to calculate an X-bit fingerprint, each optimized for a specific output size, and the outputs of the NCHFs are concatenated to form a fingerprint with a total output size of X. In this way, when calculating a fingerprint of size X, the effective collision rate of multiple NCHFs can be reduced compared to the collision rate of any one of the NCHFs.

[0064] The following is an example of how the effective collision rate can be reduced by using multiple unencrypted hash functions. Let H1 be the first unencrypted hash function, H2 be a second unencrypted hash function different from H1, and H3 be a third unencrypted hash function different from H1 and H2. URL U ij =D i P j (D i =L i .L i-1 .L.…L3.L2.L1 and P j =S1 / S2 / ... / S j In the case of ), c ​​is a counter with a final value equal to the sum of the hostname label L and path segment S of the URL (i.e., c = i + j), Kc is the c-th key, and K_1 c K_2c is a component of the c-th key generated using H1, K_3 is a component of the c-th key generated using H2, and K_3 c This is the component of the c-th key generated using H3.

number

[0065] The above example of incremental hashing using three non-cryptographic hash functions is an example of how to use multiple hash functions. Using fewer hash functions (e.g., K) is also an option. c =K _2c ||K_1 c You can use ) or more hash functions (e.g., K c =K _ 4 c ||K _ 3 c ||K _ 2 c || K_1 c ) may be used. A non-cryptographic hash function can be used in combination with one or more cryptographic hash functions. Also, a cryptographic hash function can be used (for example, without using a non-cryptographic hash function), although a cryptographic hash function may be undesirable depending on the application. Furthermore, multiple hash functions can be combined to perform incremental hashing in ways other than those described above. Such other combinations may produce different outputs for a given input, but may achieve the same utility and effect (e.g., reduced collision rate) as using the combination of H1, H2, and H3 described above for a large population of threat indicators (e.g., hundreds of millions of unique threat indicators). For example, using the non-cryptographic hash functions H1, H2, and H3 described above, for the URL Uij described above, K c It is generated as follows:

number

[0066] Efficient lossless compression methods can be used in combination with or as an alternative to the hash function-based methods described above. Lossless compression can maintain time and space / memory efficiency while mitigating or eliminating uncertainties such as hash collisions. An efficient lossless compression function C can encode and process domain names and URLs in combination with or as an alternative to the hash function-based methods described above, which is seen as an irreversible compression method. For example, C can be based on Huffman coding, a type of prefix coding, which is a lossless compression method that provides time and space / memory efficiency and avoids the collision problem associated with hash functions. Other lossless compression methods include, for example, arithmetic coding and asymmetric numerical schemes. Note that lossless compression functions are often lossless functions, and this disclosure assumes this unless otherwise specified. These lossless compression methods and associated compression functions are characterized by their compression ratio. The compression ratio for a given compression function can be the expected value obtained by dividing the size of the output of the compression function by the size of the input to the compression function. In other words, the lower the information entropy of the input, the lower the compression ratio (and therefore the better the compression ratio). Information entropy corresponds to the amount of redundant information contained in the input. For example, as of the time of this disclosure, typical public domain names in CTI consist of eTLDs formed from a relatively small set of labels such as "com", "net", and country codes such as "ru" and "cn" (e.g., several thousand elements). Furthermore, as of the time of this disclosure, the "com" label may be found in approximately 35% of eTLDs for domain names in CTI. Such redundancy can be used by lossless compression methods to achieve good compression (i.e., a relatively low compression ratio). For example, coding schemes such as Huffman coding utilize redundancy and low information entropy to achieve better compression ratios (for example, because the data required to represent an information element of size X and N-1 repetitions of that element is often less than the data required to represent N distinct elements of size X).Furthermore, at the time of publication, the Public Suffix List (PSL) contains approximately 10,000 eTLDs. The enumeration of these eTLDs in the PSL, and the associated 1-1 mapping between numbers and eTLDs, are encoded in 14 bits. Therefore, significant (lossless) compression can be achieved by representing domain name eTLDs with 14-bit enumeration. In practice, 16 bits or 2 bytes may be used to accommodate the growth of the PSL (up to 64K eTLDs) and to align with byte-size logic.

[0067] Exemplary techniques used to create efficient compressed representations or compressions of domain names are sometimes called "incremental label compression." Similarly, exemplary techniques used to create efficient compressions of URLs are sometimes called "incremental path segment compression."

[0068] Incremental label compression can be used to create an efficient compressed representation (compression) of a domain name. This technique is similar in approach to incremental label hashing (described above), but uses a (reversible) compression function C instead of a hash function H. As before, a domain name can be represented as a series of subdomain labels separated by dot characters ".", where the rightmost label represents a top-level subdomain (e.g., "com", "net", "edu", "org", etc.) relative to the (implicit) root domain. For example, domain name D i This consists of i subdomain labels L, each separated by a dot "." character. i L i-1 ...represented as a list of L1, with C being a compression function that outputs an efficiently encoded version of the input, and "||" being the concatenation operator. Domain names D with i labels. i Compression of C(D i The process of creating ) can be defined recursively as follows:

Number

Number

[0069] Since the domain name is processed from the rightmost / top-level subdomain label for each subdomain label, during the enforcement of CTI-derived policies on packet traffic in transit, the fully qualified domain name (FQDN) contained in the packet can be easily and efficiently matched with the partial domain name or base / registrable domain name within the CTI that constitutes the policy rules. For example, if the domain name C.B.A (but not the domain names E.D.C.B.A, D.C.B.A, B.A, or A) is in the CTI and is associated with a packet filtering rule having a rule ID R k . When the policy is created and processed for efficient enforcement, the compressed C(CBA) key and the rule ID R k value can be inserted into the flat hash map index data structure F. Suppose the domain name E.D.C.B.A is contained in a packet in transit. The TIG policy enforcement logic can efficiently check whether E.D.C.B.A matches one or more packet filtering rules in the policy using the following exemplary policy search algorithm (pseudo-code).

Number

[0070] Note that in each of the above steps, processing is performed for each subdomain label, so the compression function C is calculated only once for each subdomain label. Therefore, this algorithm is efficient in both time and space / memory, because the time complexity is a linear function of the number of subdomain labels, and the space complexity can be a function of the compression ratio (which can be less than 1). Also note that the corresponding algorithm for inserting domain indices into a flat hash map index data structure has similar time and space complexity.

[0071] Incremental path segment compression can be used to create efficient compression of URLs. This technique is similar in approach to incremental path segment hashing (described above), but uses a (reversible) compression function C instead of a hash function H. URLs are, <scheme> : / / <host> <path> <query>It may take the form of this. Here, <scheme>This is either "http" or "https". <host>This is the hostname (for example, domain name D i =) or an IP address may also be used. <path>P j This consists of j path segments S1S2S3...S j It may also be a sequence of each path segment S i It may also begin with a slash " / " character. <query>This may be a string consisting of a parameter name and its associated value. <query>In the context of CTI applications such as cyber security policy enforcement, which may often be ignored, if applicable, query strings can be processed in a similar manner to labels or path segments. URL U ij =D i P j of <host> <path>The process of creating compression for a portion is similar to the incremental label compression process described above for creating compression for a domain name. Compression C(U ij ) = C(D i ) ||C(P j ), here C(D i ) is calculated as above, C(P j ) is defined recursively as follows:

number

[0072] As with the domain name mentioned above, the URL <path>Since each path segment is processed starting from the leftmost path segment, when applying a CTI-derived policy to packet traffic in transit, the complete URL contained in the packet can be easily and efficiently matched with the URL in CTI that has the subpaths that make up the policy rules. For example, the following can be found in a packet in transit: <host>EDCBA and <path>URLs with / S1 / S2 / S3 are the same <host>Has EDCBA, but (partial) <path> / S 1 / This may match a packet filtering rule that uses CTI URLs with S2 as a matching criterion.

[0073] For the purpose of generating compression to map to rule IDs within a flat hash map index data structure, and for the purpose of incrementally compressing search objects to search that flat hash map index data structure, domain names (or URLs in domain name format) <host>The part of the URL (the 'part') is processed from right to left, and the path part of the URL can be processed from left to right. i =L i .L i-1 .L.…L3.L2.L1, and P j =S1 / S2 / ... / S j The URL is Uij=D i P j The compression can be generated incrementally as follows: Here, "K c " is the key generated by incremental compression, and c is a counter whose final value is equal to the sum of the hostname label L and path segment S of the URL (i.e., i + j).

number

number

[0074] By using the incremental compression described above, rules based on domain name CTI (e.g., rules where the matching criterion is a domain name) and rules based on URL CTI (e.g., rules where the matching criterion is a URL) <host>and <path>A single flat hash map index data structure can be used for a part of a rule. This simplifies the search logic (e.g., using the same search logic for different types of threat indicators) and reduces memory usage (e.g., using a single index data structure for rules with different types of matching criteria). ij Compression C(U ij An efficient approach to calculating ) is as follows: (1) Constitute a single instance of a flat hash map to store and process all relevant domain name CTIs (and associated policy rules) and all relevant URL CTIs (and associated policy rules). (2) The above compression-based key K c Using the formula, Domain name D i and URLD i P j Calculate the compression ratio. For example, domain name D i =L i .L i-1 ...The compression added to a single flat hash map for L3, L2, and L1 is C(L i )||C(L i-1 )||…||C(L3||C(L2||C(L1)))…)) and URLU ij =D i P j =L i .L i■1 ...L3.L2.L1 / S1 / S2 / ... / S j-1 / S j The compression added to a single flat hash map is C(P j )||C(P j-1 )||…||C(P2)||C(P1)||C(L i )||C(L i-1 )||...||C(L3)||C(L2)||C(L1).

[0075] URL <host>Note that the part can also be an IP address (IPv4 or IPv6). In this case, the IP address is incrementally compressed in a way similar to incremental label compression of domain names, but proceeding from left to right. An IPv4 address can be represented in dotted decimal notation, which is a sequence of four integers N, each in the range [0, 255], separated by a dot character ".". Thus the URL <host>If the part is an IPv4 address, the IPv4 address is treated similarly to a domain name, with the four integers corresponding to four subdomain labels separated by a dot "." character, and the four integers numbered from left to right (e.g., N1.N2.N3.N4). The above compression-based key K c In the formula, for example, <host>When compressing a URL that has an IPv4 address as part of it incrementally, i=4, and L1 to L4 become N1 to N4 respectively. An IPv6 address can be represented by eight groups H, each group having a 4-digit hexadecimal number separated by a colon ":". Therefore, the URL <host>If the part is an IPv6 address, the IPv6 address is treated similarly to a domain name, with the eight groups corresponding to eight subdomain labels, separated by a colon ":" instead of a "." character, and the eight groups numbered from left to right (e.g., H1:H2:H3:H4:H5:H6:H7:H8). The above compression-based key K c In the formula, for example, <host>When incrementally compressing a URL that has an IPv6 address as part of it, i=8, and L1 to L8 become H1 to H8 respectively.

[0076] Similarly, when processing policies originating from CTI for efficient enforcement, the domain names and URLs of the packet filtering rules comprised by the policy may be compressed as described above, the compression (key) may be paired with the associated rule identifier / rule ID (value), and the {compression, rule ID} pair may be inserted into a table, which is a flat hash map index data structure. However, flat hash maps may not natively support efficient dynamic runtime updates of packet filtering rules for policies currently being enforced by TIG (see explanation below). In this context, dynamic runtime updates mean updating the matching criteria of a packet filtering rule by inserting or removing threat indicators from the set of metrics that constitute the matching criteria while the associated policy is being applied to packets in transit. If dynamic runtime policy updates are required, an alternative index data structure that natively supports dynamic runtime updates, such as a rule tree, can be used instead. In this case, the {compression, rule ID} pair is inserted into a rule tree instead of a hash table. This index data structure is a rule tree that stores keys compressed in bit string representation and is sometimes called a compressed rule tree. Note that compressed rule trees require more memory than their corresponding flat hash maps, and the trade-off is the support for dynamic runtime updates.

[0077] In some cases, a combination of irreversible hashing and reversible compression methods can be effective. For example, since domain name redundancy is often higher than path redundancy, URLs can be processed using a (reversible) compression function for the hostname (i.e., domain name) portion and an (irregular) hashing function for the path portion.

[0078] Figure 2A is a flowchart illustrating an example of how to retrieve a flat hash map index data structure consisting of {fingerprint, rule ID} index pairs of a policy (or part of a policy) based on a lookup object, which is a value obtained from a field in a packet in transit. For each index pair in the flat hash map index data structure, the rule ID may constitute the identifier of the policy rule, and the fingerprint may constitute a value generated using incremental hashing as described above, based on the policy rule's matching criteria (e.g., threat indicators). There may be multiple index pairs that constitute the same fingerprint (e.g., there may be multiple rules that constitute the same matching criteria). In the example in Figure 2A, a domain name / subdomain name is used as an example of the type of policy rule matching criteria and packet field value on which the method is performed, but the method in Figure 2A may also be performed in relation to other types of matching criteria and packet field values, or alternatively. Figure 2A may be performed, for example, by TIG102. One or more steps of the exemplary method in Figure 2A can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0079] Step 201 may receive a lookup object in the form of a domain name (for example, from a process of parsing packet fields specified by policy rule criteria and extracting data from those fields). Step 202 searches one or more set data structures for membership of the domain name (and / or subdomain of that domain name) received in Step 201 in a set of domain names and subdomain names that are policy rule matching criteria corresponding to index pairs of flat hash map index data structures. Step 202 may include the use of one or more Bloom filters, cuckoo filters, or other types of probabilistic set data structures and related methods described in Patent No. 417. Alternatively, Step 202 may include testing membership using a non-probabilistic set data structure. If in Step 202 TIG102 determines that there is no membership (i.e., the lookup object is not included in the set of lookup objects that match the policy rule matching criteria corresponding to index pairs of flat hash map index data structures), TIG102 may skip the lookup of the flat hash map data structures and apply rules that may be determined using other values ​​from the packet in transit, if any. As will be explained in more detail in relation to Figure 7, TIG102 can simultaneously search for a group of policy rules by searching multiple index data structures based on multiple different values ​​from a packet.

[0080] If TIG102 determines membership in step 202 (for example, if the search object or part of the search object is within the set of matching criteria of the policy rule corresponding to the index pair of the flat hash map index data structure), TIG102 may initialize counter c to 1 in step 203. In step 204, TIG102 determines the c-th label L of the domain name search object. c Based on this, the hash base key K c can be calculated. For example, when c = 1, K c is calculated as H(L1), and when 1 < c ≤ i, K c is calculated as H(L c ||K c-1 ). In step 205, TIG102 searches a flat hash map index data structure (FHM) to find a fingerprint that matches the value of K c calculated in step 204. Each key / fingerprint in the flat hash map index data structure is calculated from a policy rule match criterion (e.g., a threat indicator) corresponding to the rule ID to which the key / fingerprint is mapped within the flat hash map index data structure using incremental hash generation consisting of one or more steps such as step 204. If one or more matching fingerprints are found, in step 206, TIG102 can store the value of the rule ID of each index pair having the matching fingerprint in a result list data structure. As used herein, a "result list" generally refers to a data structure that can store multiple values such as rule IDs, and the result list data structure does not necessarily have to take the form of an actual list.

[0081] Alternatively, or alternatively, step 204 of the method of FIG. 2A includes calculating the key using a different procedure, and step 205 includes searching a flat hash map index data structure consisting of a key / fingerprint calculated using a different procedure. For example, the key calculation in step 204 and the key / fingerprint calculation of the flat hash map index data structure searched in step 205 are hashes based on the hashes of multiple individual labels (e.g., when c = 1, K c = H(L1), when 1 < c ≤ i, K c = H(H(L c )||K c-1 )) may also be based on. As another example, the key calculation in step 204 and the key / fingerprint calculation of the flat hash map index data structure retrieved in step 205 may be based on multiple hash functions (e.g., any of the above procedures using hash functions H1, H2, and H3).

[0082] After step 206, or in step 205, the flat hash map index data structure is calculated in step 204. c After determining that it lacks a key / fingerprint that matches the value, TIG102 calculates K based on the additional portion of the search object value. c The search of the flat hash map index data structure can continue for a key / fingerprint that matches the additional value. For example, consider the domain name "abc.example.com". The first policy rule that can be obtained based on the first CTI showing an association between the domain "example.com" and a cyber threat / attack might indicate that packets with this field value should be allowed, but the presence of such packets should be logged. The second policy rule can be obtained based on the second CTI showing another association between the subdomain "abc.example.com" and a cyber threat / attack, and might indicate that packets with this field value should be blocked. The fingerprint of example.com, i.e., the first rule, will likely be found before the fingerprint of abc.example.com, i.e., the second rule. If the search is stopped after finding the first rule, more critical rules may be missed.

[0083] In step 207, TIG102 can determine whether all relevant parts of the search object have been processed (for example, whether all labels L of a domain / subdomain have been processed). If No (for example, if c is not equal to i), the c counter is incremented by 1 in step 208, and TIG102 can repeat part of the process starting from step 204. If Yes (for example, if c = i), TIG102 adds the stored rule ID(s) (stored in the results list during one or more executions of step 206) to the rule ID(s) that can be determined using any other values ​​from the packet in transit, and the rules(s) corresponding to all determined rule ID(s) can be applied to the packet. This will be further explained in relation to Figure 7.

[0084] Figure 2B is a flowchart illustrating another example method for retrieving a flat hash map index data structure consisting of {fingerprint, rule ID} index pairs of a policy (or part of a policy) based on a lookup object obtained from a field in a packet in transit. One or more steps in the exemplary method of Figure 2B can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. The method of Figure 2B is similar to the method of Figure 2A and includes steps 215 and 219-224, which may be the same as steps 201 and 203-208 of Figure 2A, respectively. Similarly, step 216 of Figure 2B may be the same as step 202 of Figure 2A. However, in the method of Figure 2B, step 216 is performed in a first processing thread A1, and steps 219-224 are performed in a concurrent second processing thread B1. In the method of Figure 2B, upon receiving the lookup object in step 215, thread A1 initiates a membership check, and thread B1 initiates the retrieval of the flat hash map index data structure. If step 216 determines that TIG102 is a member, thread A1 can be terminated in step 217. If step 216 determines that TIG102 is not a member, thread B1 is terminated in step 218, and any rules determined using other values ​​from the packet in transit may be applied.

[0085] The method in Figure 2B may be selected based on the fact that the estimated or expected time to perform step 216 is shorter than the estimated or expected time to complete the lookup of the index data structure. Thread B1 may include step 225, which may be performed after the "No" determination in step 223, and if those time estimates are inaccurate, or if the deviation from the expected time results in the lookup being faster than the membership test, it may cause thread A1 to terminate. In step 225, TIG102 can determine whether thread A1 is still running, and if so, it can terminate the execution of thread A1.

[0086] The membership test may be omitted. For example, if it seems that the membership test will take significantly less time than searching an index data structure, it may be inefficient to allocate memory resources to the set data structure. FIG. 2C is a flowchart showing another example method for searching a flat hash map index data structure consisting of {fingerprint, rule ID} index pairs of a policy (or part of a policy) based on a search object obtained from fields of a packet being transferred. The method of FIG. 2C is similar to the method of FIG. 2A except that step 202 is omitted.

[0087] FIG. 2D is a flowchart showing another example method for searching a flat hash map index data structure consisting of index pairs of a policy (or part of a policy) based on a search object obtained from fields of a packet being transferred. One or more steps of the exemplary method of FIG. 2D can be reordered (e.g., executed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. The method of FIG. 2D may be executed by TIG102 and is similar to the method of FIG. 2B except that incremental label compression is used instead of incremental label hash. In the example method of FIG. 2D, the flat hash map index data structure is composed of a compression (compressed value) calculated as described above using a reversible compression function C instead of a hash-based fingerprint. Steps 235-239 and 242-245 of FIG. 2D are the same as steps 215-219 and 222-225 of FIG. 2B, respectively. Step 240 of FIG. 2D is, in step 240, the same as step 220 of FIG. 2B except that TIG102 can calculate a compression-based key K c based on the c-th label L c of the domain name search object. For example, when c = 1, K c is calculated as C(L1), and when 1 < c ≦ i, K c is C(L c )||C(K c-1 Step 241 in Figure 2D is calculated as follows: In step 241, TIG102 is calculated as K calculated in step 240. c This is similar to step 221 in Figure 2B, except that the flat hash map index data structure can be searched for a key / compression that matches the value. Each key / compression in the flat hash map index data structure is computed using incremental compression consisting of one or more steps, such as step 240, from the matching criteria (e.g., threat indicator) of the policy rule corresponding to the rule ID to which the key / compression is mapped in the flat hash map index data structure.

[0088] In a variation of the method in Figure 2D, step 236 is performed before step 239, and a "Yes" membership determination becomes a prerequisite for performing the remaining method steps, allowing the method to be executed in a single thread, similar to the method in Figure 2A, with steps 237, 238, and 245 omitted. In yet another variation of the method in Figure 2D, the membership test may be omitted (similar to the method in Figure 2C).

[0089] Figure 2E is a flowchart illustrating another example method for retrieving a flat hash map index data structure consisting of index pairs of a policy (or part of a policy) based on a lookup object obtained from a field of a packet in transit. The exemplary method in Figure 2E, which can be performed by TIG102, may consist of an incremental label hash and an incremental path segment hash, as also described above. One or more steps of the exemplary method in Figure 2E can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0090] In step 250, a lookup object can be received (for example, from a process that parses the specified packet fields of a policy rule criterion and extracts data from those fields). The lookup object may consist of a domain name. Alternatively, the lookup object may be: <host>parts and, <path>It can consist of URLs having parts. In particular, the method in Figure 2E can be performed using a single instance of a flat hash map index data structure, which is configured based on threat indicators for rules created based on domain name CTI and also based on threat indicators for rules created based on URL CTI. For example, that single instance of the flat hash index data structure can map keys generated by incrementally hashing domain names (or parts of domain names) from the matching criteria of a first policy rule based on all relevant domain name CTI to the rule IDs of those first policy rules. That single instance of the flat hash index data structure can also map keys generated by incrementally hashing URLs (or parts of URLs) from the matching criteria of a second policy rule based on all relevant URL CTI to the rule IDs of those second policy rules.

[0091] Similar to the methods in Figures 2B and 2D, the method in Figure 2E includes concurrent processing threads A3 and B3. Based on the receipt of the search object in step 250, thread A3 starts a membership test and thread B3 starts searching for a flat hash map data structure. If TIG102 determines membership in step 251 (which may be similar to steps 202, 216, and 236), thread A3 may terminate in step 252. If TIG102 determines no membership in step 251, thread B3 terminates in step 253, and any rules determined using other values ​​from the packet in transit may be applied.

[0092] In step 254, TIG102 can determine the portion of the search object that is a domain name label, the quantity (i) of the label, and the order of the label. Also in step 254, TIG102 can determine whether the search object has a path portion, and if so, the number of segments (s), quantity (j), and order of the segments of that path portion. If the search object does not have a path portion, TIG102 can set the value of j to zero for the purposes of step 259 (described later). Alternatively, in step 254, TIG102 can determine whether the search object is in the form of an IPv4 address or an IPv6 address. <host>It can be configured to determine whether a URL has a part. The search object is an IPv4 address <host>If it is a partial URL, TIG102 may set i=4 in step 254 and determine integers N1, N2, N3, N4 that handle labels L1 to L4 in step 256 (described later). <host>If it is a URL to be segmented, TIG102 may set i = 8 in step 254 and determine the hexadecimal groups H1:H2:H3:H4:H5:H6:H7:H8 that handle labels L1 to L8 in step 256.

[0093] In step 255, TIG102 can initialize the counter c to 1. In step 256, TIG102 can calculate the hash-based key K c based on the label or path segment of the search object. For example, when c = 1, K c is calculated as H(L1). When 1 < c ≤ i, K c is calculated as H(L c ||K c-1 ). If the search object has a path part and i < c ≤ j, K c is calculated as H(S c-i ||K c-1 ). In step 257, TIG102 can search for the key / fingerprint that matches the value of K c calculated in step 256 from the flat hash map index data structure. Each key / fingerprint in the flat hash map index data structure is calculated from the match criteria (e.g., threat indicators) of the policy rule corresponding to the rule ID to which the key / fingerprint in the flat hash map index data structure is mapped using incremental hash generation consisting of one or more steps like step 256. If one or more matching fingerprints are found, in step 258, TIG102 can store the value of the rule ID of each index pair with the matching fingerprint in the result list data structure.

[0094] Alternatively, or instead, step 256 of the method of FIG. 2E includes calculating a key using a different procedure, and step 257 includes searching a flat hash map index data structure consisting of keys / fingerprints calculated using a different procedure. For example, the key calculation in step 256 and the key / fingerprint calculation of the flat hash map index data structure searched in step 257 may be based on a hash based on the hash of a plurality of individual labels (e.g., when c = 1, K c = H(L1), when 1 < c ≤ i, K c = H(H(L c ) || K c-1 )). As another example, the key calculation in step 256 and the key / fingerprint calculation of the flat hash map index data structure searched in step 257 may be based on a plurality of hash functions (e.g., any of the above-described procedures using hash functions H1, H2, and H3).

[0095] After step 258, or after it is determined in step 257 that the flat hash map index data structure is missing a key / fingerprint that matches the value of K c calculated in step 256, TIG102 calculates K c The search of the flat hash map index data structure can continue for a key / fingerprint that matches the additional value. In step 259, TIG102 can determine whether it has processed all labels and path segments (if any) of the search object. If no (e.g., c has not reached the value of i+j), in step 260 the c counter is incremented by 1, and TIG102 can repeat part of the process starting from step 256. If yes (e.g., c = i+j), in step 261 TIG102 determines whether thread A3 is still running, and if so, can terminate the execution of thread A3. After step 261, TIG102 can add the stored rule ID(s) (stored in the result list during one or more executions of step 258) to any other rule ID(s) that may be determined using other values ​​from the packet in transit, and apply the rule(s) corresponding to all determined rule ID(s) to the packet. This will be further explained in relation to Figure 7.

[0096] In a modified version of the method in Figure 2E, step 251 is performed before step 254, and a "Yes" membership determination becomes a prerequisite for the execution of the remaining method steps, allowing the method to be executed in a single thread, similar to the method in Figure 2A, with steps 252, 253, and 261 omitted. In yet another modification of the method in Figure 2E, the membership test may be omitted (similar to the method in Figure 2C).

[0097] Figure 2F is a flowchart illustrating another example method for retrieving a flat hash map index data structure consisting of index pairs of policies (or parts of policies) based on lookup objects obtained from fields of a packet in transit. One or more steps of the exemplary method in Figure 2F can be reordered (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. The method in Figure 2F may also be performed by TIG102 and is similar to the method in Figure 2E, except that incremental label (and segment, in the case of URL lookup objects) compression is used instead of incremental label and segment hash. In the example method in Figure 2E, the flat hash map index data structure consists of compressions (compressed values) calculated as described above using a lossless compression function C, instead of hash-based fingerprints. Similar to the method in Figure 2E, the method in Figure 2F may be performed using a single instance of a flat hash map index data structure configured based on threat indicators of rules created based on domain name CTI and based on threat indicators of rules created based on URL CTI. For example, a single instance of the flat hash index data structure could map keys generated by incrementally compressing domain names (or parts of domain names) from the matching criteria of a first policy rule based on all relevant domain name CTIs to the rule IDs of those first policy rules. A single instance of the flat hash index data structure could also map keys generated by incrementally compressing URLs (or parts of URLs) from the matching criteria of a second policy rule based on all relevant URL CTIs to the rule IDs of those second policy rules.

[0098] Steps 265-270 and 273-276 in Figure 2F are the same as steps 250-255 and 258-261 in Figure 2E, respectively. Step 271 in Figure 2F is the same as step 256 in Figure 2E, except that in step 271, TIG102 compresses the key K based on the label or path segment of the search object. c You can also calculate this. For example, if c=1, K c This is calculated as C(L1). If 1 < c ≤ i, then K c is C(L c ) ||C(K c-1 It is calculated as C(S). If the search object has a path part and i < c ≤ j, then Kc is C(S c-i )|| K c-1 It is calculated as follows. Step 272 in Figure 2D is where, in step 272, TIG102 is calculated as K calculated in step 271. c This is similar to step 257 in Figure 2E, except that the flat hash map index data structure can be searched for a key / compression that matches the value. Each key / compression in the flat hash map index data structure is computed using incremental compression consisting of one or more steps, such as step 271, from the matching criteria (e.g., threat indicator) of the policy rule corresponding to the rule ID to which the key / compression is mapped in the flat hash map index data structure.

[0099] In a modified version of the method in Figure 2F, step 266 is executed before step 269, and a "Yes" membership determination becomes a prerequisite for executing the remaining method steps, allowing the method to be executed in a single thread, similar to the method in Figure 2A, with steps 267, 268, and 276 omitted. In yet another modification of the method in Figure 2F, the membership test may be omitted (similar to the method in Figure 2C).

[0100] Figure 2G is a flowchart illustrating another example method for retrieving a flat hash map index data structure consisting of index pairs of policies (or parts of policies) based on lookup objects obtained from fields of a packet in transit. One or more steps of the exemplary method in Figure 2G can be reordered (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. The method in Figure 2G may also be performed by TIG102 and is similar to the methods in Figures 2E and 2F, except that a combination of incremental hashing and incremental compression is used. In the exemplary method in Figure 2G, the flat hash map index data structure may consist of compression and fingerprinting, based on compression and hashing. Similar to the methods in Figures 2E and 2F, the method in Figure 2G may be performed using a single instance of a flat hash map index data structure configured based on threat indicators of rules created based on domain name CTI and based on threat indicators of rules created based on URL CTI. For example, a single instance of the flat hash index data structure could map keys generated by incrementally compressing domain names (or parts of domain names) from the matching criteria of a first policy rule based on all relevant domain name CTIs to the rule IDs of those first policy rules. A single instance of the flat hash index data structure could also map keys generated by incrementally compressing and hashing URLs (or parts of URLs) from the matching criteria of a second policy rule based on all relevant URL CTIs to the rule IDs of those second policy rules.

[0101] Steps 280 to 285 and 288 to 291 in FIG. 2G are the same as steps 250 to 255 and 258 to 261 in FIG. 2E, respectively. Step 286 in FIG. 2G is the same as step 256 in FIG. 2E, but in step 286, TIG102 may calculate the key K c based on compression or a combination of compression and hash. For example, when c = 1, K c is calculated as C(L1). When 1 < c ≤ i, K c is calculated as C(L c )||C(K c-1 ). If the search object has a path part and i < c ≤ j, K c is calculated as H(S c-i ||K c-1 ). Step 287 in FIG. 2G is similar to step 257 in FIG. 2E, except that in step 287, TIG102 can search the flat hash map index data structure for a key that matches the value of K c calculated in step 286. Each key in the flat hash map index data structure is calculated using incremental compression / hash consisting of one or more steps such as step 286 from the policy rule match criteria (e.g., threat indicator) corresponding to the rule ID to which the key is mapped in the flat hash map index data structure.

[0102] In a variation of the method of FIG. 2G, step 281 is executed before step 284, a "Yes" membership determination is a prerequisite for the execution of the remaining method steps, and steps 282, 283, and 291 are omitted, and the method can be executed in a single thread similar to the method of FIG. 2A. In yet another variation of the method of FIG. 2G, the membership test may be omitted (similar to the method of FIG. 2C).

[0103] The rule tree index data structure disclosed herein is designed to efficiently store keys, which are binary strings representing full or prefix IPv4 and IPv6 addresses, and which can index identifiers of CTI-derived rules that constitute a cybersecurity policy. The rule tree can be viewed as a succession of well-known index data structures.

[0104] These known tri-index data structures include prefix trees (tria), which are N-ary trees designed to efficiently store keys that share a prefix. Links / edges between nodes, i.e., between parent and child nodes, are labeled with a single character that makes up the key. For binary strings, prefix trees can be 2-ary trees where links / edges are labeled with 0 or 1. Efficient adaptations of prefix trees include Patricia tries, radix trees, and adaptive radix trees.

[0105] These known tri-index data structures have M=2 when the key is a binary string. k This includes multi-way tries (M-way tries) or M-ary tries (where k is an integer). The parameter k is called the stride, and the value of k is called the stride length. Each (internal) node of the trie is 2 k It consists of an array with n elements. If the key is a binary string, the array consists of all possible bit sequences of length k. For example, if k=3, the node array would be {000,001,010,011,100,101,110,111}, which requires at least 24 bits to represent. Each node has a maximum of 2 k It can have 2 children (child nodes). k -The depth (number of levels) of an ary try corresponds to L / k, where L is the length of the longest key / bit sequence.

[0106] These known tri-index data structures also include poptrie (H. Asai and Y. Ohara, "Poptrie": A Compressed Trie with Population Count for Fast and Scalable Software IP Routing Table Lookup, ACM SIGCOMM Computer Communication Review, vol.45, no.4, pp.57-70, 2015), which stores keys that are bit strings representing prefixes of arbitrary length and is a multi-directional trie (2) adapted and optimized for looking up keys bit by bit. k PopTry is a multiway / 2-bit IP routing table lookup target application. The input being searched / looked up is a bit string, and a given input can match any key / prefix of any length less than or equal to the length of the input. PopTry is memory-efficient and optimized for high-speed IP routing table lookup target applications. The search / lookup searches for a single longest prefix match in the Internet router's table and stops when the first matching prefix is ​​found. The key is a bit string representing an IPv4 or IPv6 address, and can be a full address (32 bits for IPv4, 128 bits for IPv6) or a prefix address (less than 32 or 128 bits in length, respectively). The key is mapped to a router's Forwarding Information Base (FIB) entry. PopTry improves the time, space / memory complexity for IP routing table lookup target applications in multiple ways. k - Adapt the ary trial.

[0107] For example, PopTry can fix the stride length k to 6. In PopTry's target application, which is IP routing table lookup, this value of k can jointly optimize time and space / memory complexity. For example, 2 k = 64, which corresponds to the register size of a 64-bit CPU architecture (used in modern internet routers). Therefore, many poptree data structures, such as index arrays, are 64 bits in size, or integer multiples or less than 64, so that the CPU can process them efficiently. Furthermore, the data stored in poptrees, i.e., keys (prefixes for IPv4 and IPv6 addresses) and inputs to lookups / searches (i.e., typically complete IPv4 and IPv6 addresses), are often bit sequences that are multiples of 64 bits (i.e., 128-bit IPv6 addresses) or submultiples (i.e., 32-bit IPv4 addresses).

[0108] As another example, 2 k -ary Tri configures 2 internal nodes k A portion of the element's descendant array is pop-tray's 2 k Replace with a descendant array of bit vectors. 2 k The descendant array of a bit vector is 2 k -ary trial corresponding 2 k It may require k times less memory than an array of descendants of elements, and may require less processing logic and better time complexity. k The use of bit vectors is made possible by the aforementioned pop-try requirement, which allows the search / lookup to find a single longest prefix match in the Internet router's table and stop the search / lookup when the first matching prefix is ​​found.

[0109] As an additional example, in poptry, each key is mapped to one leaf node, and each leaf node consists of an index to the corresponding router FIB entry (which may also directly correspond to a RIB entry). This is the corresponding multiway / 2 without poptry adaptation and optimization. k -ary's tridata structure can have many duplicates and redundant leaves. This arises from the characteristics of the target application, where many keys may map to the same router FIB entry. PopTri leverages this redundancy to significantly reduce space and memory requirements. PopTri's internal nodes consist of bit vectors called leafbecks, which, combined with the associated logic, can compress duplicates and redundant leaves, drastically reducing memory requirements.

[0110] As an additional example, PopTry can use a technique called "direct pointing" to significantly improve search time performance at the cost of adding a relatively small amount of memory. Direct pointing takes advantage of the fact that many keys have prefixes longer than the stride length k. PopTry's root has a length of 2 s By adding an array where s is an integer multiple of k, for example s=12, k=6, and using the s most significant bits of the key as a direct index to the array, subsequent searches of the poptry avoid searching the first s / k level of the poptry and instead jump directly to an internal node or (leaf) FIB value in O(1) constant time.

[0111] A rule tree shares the characteristics of PopTry, stores keys that can be bit strings representing prefixes of arbitrary length, and can be optimized for bitwise key retrieval. When enforcing cybersecurity policies composed of CTI-derived rules, it modifies PopTry's data structure to minimize the time, space, and memory complexity of the target application's lookup / retrieval of CTI-derived rules. The target application of a rule tree has different requirements and constraints than the target application of PopTry, and accordingly, the internal data structure and logic of the rule tree differ from those of PopTry. For example, (i) in PopTry, a key is mapped to a single leaf node, whereas in a rule tree, a key may be mapped to multiple leaf nodes; (ii) in a rule tree, a key / prefix has one or more subkeys / subprefixes, each of which is mapped to one or more leaf nodes as in (i) above, whereas in PopTry, a key is atomic, meaning a key / prefix does not contain any subkeys / subprefixes; and (iii) in PopTry, a search / lookup stops when the first leaf node is encountered, whereas in a rule tree, a search / lookup does not stop when the first leaf node is encountered. Therefore, PopTry cannot be used for target applications of rule trees. This is not mere theory; in reality, in a typical collection of approximately 10 million IPv4 addresses in a representative CTI, about 9 million are complete IPv4 addresses ( / 32 IPv4 addresses), and it should be noted that when this IPv4 CTI is stored in a rule tree, more than 99% of complete IPv4 addresses / keys will have two or more leaf nodes. Rule tree adaptation includes, but is not limited to, the following:

[0112] (a) Rule tree internal nodes consist of different internal data structures and logic than pop-trays, and accordingly support requirements (i), (ii), and (iii) above. For example, a pop-tray internal node contains a "vector" data structure which is a 2k-bit array that constitutes a descendant array. The bit value of each index / position in the array indicates whether the corresponding descendant / child node, i.e., the destination, is a leaf node (e.g., bit value "0"), in which case the search stops. Or, if it is another internal node (e.g., bit value "1"), the search continues. While this pop-tray "vector" bit array may be memory efficient, it cannot support the rule tree requirements for rule lookup / search derived from CTI. Regarding rule tree requirement (i), the single bit value "0" indicating that a descendant node is a leaf node cannot indicate whether there are multiple leaf nodes. Regarding rule tree requirements (ii) and (iii), the single bit value "0" indicating that a descendant node is a leaf node cannot indicate whether the key indexing the related leaf node data is a subkey, and therefore whether the search should continue.

[0113] (b) The stride k of the poptry is set to 6 to optimize the target application of the poptry, however the stride k of the rule tree may be parameterized, and the associated stride length value may be adjusted to jointly optimize the time and space / memory complexity in relation to the time and space / memory requirements and the data distribution characteristics of the key. For example:

[0114] (bi) Generally, a lower value of k can increase the depth or number of levels in the rule tree compared to a higher value of k, which can increase the predictive search time for the rule tree.

[0115] (b.ii) For a given set of keys, different values ​​of k may result in different memory / space values ​​required to store the keys in the rule tree.

[0116] (b.iii) As of the time of this disclosure, for the key sets that make up the majority of available IPv4 CTIs (millions of IPv4 addresses), setting k=4 can jointly optimize the complexity in terms of time and space / memory.

[0117] (b.iv) When the key length distribution is highly modal, the time and space complexity can be optimized by a particular value of k. For example, if most or all of the keys are 64-bit and / or 128-bit strings, as occurs in IPv6 CTI and 64-bit or 128-bit fingerprint rule trees, k=6 can optimize the time and space complexity. Also, in the case of such a modal distribution, a pop-try direct pointing strategy can significantly improve the time complexity and associated search performance. Furthermore, the time complexity can be further improved by using direct pointing at one or more non-root levels. In addition, a highly modal key length distribution can reduce the number of internal leaf nodes, resulting in space savings.

[0118] (bv) Optimal performance can be achieved by using different values ​​of k for different subtrees of the rule tree for a given set of keys.

[0119] (c) In a pop-tree where the target application may be a fast IP routing table lookup, leaf nodes can index FIB entries for keys (i.e., IP address prefixes). In a rule tree, on the other hand, when enforcing a cybersecurity policy consisting of CTI-derived rules, the target application may be a lookup / search for CTI-derived rules, and one or more leaf nodes of a key are associated with a set of rule identifiers, each rule identifier being associated with a rule that has matching criteria that make up the key. The rule identifiers, or elements, in the set may be stored in an efficient data structure that can maintain a ranking of the rule identifiers, where the ranking indicates the order in which rules are applied to matching packets (i.e., packets in transit containing values ​​that match the key). For example, elements may be stored in an ordered linked list in rank, with the first element of the list constituting the rule identifier of the highest-ranked rule, and the last element of the list constituting the rule identifier of the lowest-ranked rule. In some cases, for example, if the rules contain a "quick" keyword / directive, only the highest-ranked rule may be applied to matching packets. In such cases, the rule identifier corresponding to the top-level rule can be stored in a fast cache to improve time performance, and / or the rule identifier corresponding to the lower-level rule can be discarded to reduce space / memory usage.

[0120] (d) In PopTry, the lookup / search stops when the first leaf node is encountered, i.e., when the input first matches a key stored in PopTry. For example, in an application that rapidly looks up an IP routing table, the key corresponds to the longest prefix in the routing table that matches the input, which is a 32-bit IPv4 address or a 128-bit IPv6 address. Because the search / lookup stops when the first leaf node is encountered, the complexity of the logic and memory requirements of the internal nodes of PopTry can be greatly reduced. However, in a rule tree, reaching the first leaf node is not the cause of the lookup / search stopping. Instead, the search continues until all prefixes that match the entire input have been found. Consequently, the internal nodes and leaf nodes of a rule tree require more memory and more complex logic and data structures, as described in (4)(c) above.

[0121] Figures 3A-3D, along with the following explanation of these figures, provide examples of flowchart / algorithm and rule tree implementations in the context of cybersecurity policy enforcement applications. The following explanation shows that rule trees may meet the requirements of such applications, while pop-try may not.

[0122] Figure 3A shows an exemplary rule tree 320 and its associated data structures, including an exemplary rule tree internal node 321 and associated (external) data structures 327 (labeled "Search Results List") and 330 (labeled "Internal Data and External Data"). Each internal node of the rule tree processes a k-bit portion (chunk) of a bit string, where k is the stride and also the stride length. In the exemplary internal node 321, the stride length k=2. While k=2 is used here for simplicity in the example, in practice, the stride length k should be chosen to provide the best performance, which may be measured as a combination of time complexity and space / memory complexity in the context of the expected data elements to be stored in the rule tree and the associated application.

[0123] When k=2, the chunk processed by the internal node has six possible values: "0", "1", "00", "01", "10", and "11". Generally, for the reasons explained below, 2 for a k-bit chunk k+1 - There are two possible chunk values. The bit strings processed by the rule tree may be keys (indexes) mapped to values ​​(e.g., rule IDs), or the bit strings may be lookup objects. In an exemplary application, the keys and lookup objects are IPv4 and IPv6 addresses, prefixes or full IP addresses, and the values ​​are rule identifiers / rule IDs related to cybersecurity policies.

[0124] The internal node 321 constitutes the external vector 321b, with length 2 k+1 A bit array of -2, indexed by each possible value of the current chunk being processed by the internal node 321, as indicated by the small numbers on the box in the outer vector 321b. A value of "0" in the bit array at the position / index corresponding to a chunk value may indicate that there is no value to which the corresponding key or subkey is mapped, such as a rule ID. On the other hand, a value of "1" in the bit array at the position / index corresponding to a chunk value may indicate that there is a value to which the corresponding key or subkey is mapped, such as one or more rule IDs. The mapped values, such as rule IDs, may be stored in memory outside the internal node 321 structure or outside of it (for example, a sparse array, as described later for internal and external data 330), and are therefore sometimes called outer-data nodes. In PopTry terminology, a PopTry "leaf node" may somewhat correspond to an external data node in a rule tree, but in PopTry and general tree data structures, a "leaf node" is usually associated with a data value representing the final / terminal internal node of a path through PopTry starting from the PopTry's root node, whereas a rule tree can have one or more data values ​​associated with any node on a path through the rule tree, hence the use of different terminology in rule trees. PopTry does not support associating data values ​​with nodes on a path other than the final / terminal node, while rule trees support associating multiple data values ​​with any node on a path through the rule tree. Therefore, a PopTry search returns at most one data value, while a rule tree search may return multiple values / rule IDs, which can be further stored in the "search result list" data structure 327 described later. For example, an external data node consisting of rule IDs can be efficiently stored in an (outer) data structure such as a well-known sparse array or sparse matrix data structure, illustrated in Figure 3A by the "internal and external data" sparse array data structure 330.

[0125] An external data node associated with the "1" element of the external vector bit array 321b may be referenced to the internal data and external data sparse array 330 via the "external data reference list" data structure 321c, which includes a pointer to the sparse array and an indication of how many external data nodes (e.g., rule IDs) there are associated with the corresponding key or subkey (e.g., key prefix). For example, index position "0" of the external vector is set to 1, indicating that there is one or more external data nodes or rule IDs associated with the corresponding key or subkey. The external data node is referenced by a corresponding "0-Ptr" pointer that points to the location in the internal data and external data sparse array 330 where the Outer-Date node is located. The "0-#RuleIDs" value indicates the number / quantity of external data nodes, i.e., rule IDs, associated with the corresponding key or subkey. If the bit array element of the external vector 321b is set to "0" at index / position "01", for example, the corresponding 01-Ptr value is NULL and the 01-#RuleIDs value is 0.

[0126] For convenience, Figure 3A shows that the internal node 321 also constitutes a sparse-array-head pointer 321d. However, an internal node of a rule tree does not need to contain both a pointer like pointer 321d and a pointer to an external data reference list like external data reference list 321c. If rule IDs are stored in discontinuous memory blocks, multiple pointers like those shown in external data reference list 321c may be used. However, if the rule IDs associated with the internal node reside in a single memory block, and the memory location of a particular rule ID can be determined based on an offset from the memory location indicated by a single pointer, multiple pointers may not be necessary. In fact, an internal node of a rule tree may omit a reference list like reference list 321c, and the information shown in reference list 321c may be inferred and / or calculated from other information of the internal node. For example, if a sparse array is used to store reference list information, a single pointer like the sparse-array-head pointer 321d may refer to the head of the sparse array in memory (e.g., local main memory) and may be included in the internal node of the rule tree. The values ​​and amounts of pointers to external data nodes in the internal and external data sparse arrays 330 are dynamically determined / calculated as needed. For example, an external data pointer value such as 0-Ptr can be efficiently calculated as an offset from the sparse array head pointer 321d by counting the "1" entries in the internal vector 321a bit array and the external vector 321b bit array that precede the "1" value / element in the external vector 321b bit array currently being processed. Note that rule tree search differs from similar pop-tri search in that pop-tri search stops when it encounters a data value / leaf node, for example when the first matching prefix is ​​found, whereas rule tree search does not stop when it encounters a data value, which may be indicated by a "1" element / value in the bit array of the external vector 321b.The presence of a "1" element anywhere in the external vector bit array of an internal node in the rule tree does not indicate whether the search will stop.

[0127] The internal vector 321a has a length of 2 k This is a bit array, indexed by each of the possible k-bit values ​​of the next chunk processed by the descendant internal node. If k=2, the four possible values ​​of the next chunk processed by the descendant internal node are "00", "01", "10", and "11". If the length of the next chunk is less than k, for example if k=2, and the next chunk is "0" or "1" (indicating the end / last bit of the key or search object), the next chunk is mapped to a k-bit index with a trailing zero; for example, if the next chunk is "0", it may be mapped to index "00", and if the next chunk is "1", it may be mapped to index "10". For a given index / next chunk value, if the element / value of the corresponding internal vector's bit array is "1", then there is a descendant internal node to process the next chunk. In Figure 3A, the exemplary internal node shows an internal vector 321a bit array with the positions / indexes of "01" and "10" set to "1", indicating that there are descendant internal nodes 322 and 323 respectively to process the next chunks "01" and "10" and "1". The descendant internal nodes 322 and 323 (and / or other internal nodes of the rule tree 321) may be similar to internal node 321 and may behave in the same way as internal node 321. The positions / indexes of "00" and "11" are set to "0", indicating that there are no descendant internal nodes to process the next chunks "00" and "11". As will be explained in more detail in relation to steps 357 and 358 in Figure 3C, this indicates that the search may stop if the next chunk of the search object is "0", "00", or "11".

[0128] The internal vector 321a and external vector 321b of an internal node constitute an internal data node, which is also stored in the internal and external data sparse array 330. The internal and external data nodes of each internal node can be stored in the same sparse array. Similar to external data, the position of an internal data node in the sparse array can be calculated as an offset from a single pointer that references the beginning of the sparse array, for example, by counting the "1" entries in the internal vector bit array that precede the "1" value in the internal vector bit array currently being processed. In the example in Figure 3A, the internal data nodes and / or external data nodes stored in the internal and external data sparse array 330 are shown as data nodes data-0 to data-n.

[0129] As described above, the search results list 327 may be used to store data values / rule IDs encountered during a search through the rule tree (shown as rule ID 328 in the example in Figure 3A). When the search stops, the search results list 327 is returned to the process that issued the relevant query. If there is a (sum) order associated with the data values / rule IDs, for example, if the rule IDs correspond to non-negative integers 0, 1, 2, 3, ..., the search results list 327 may be maintained in a sorted order. As described elsewhere, the rules associated with the rule IDs in the search results list may be applied to packets in an order corresponding to the order of the rule IDs. For example, the highest-ranking rule applied first is the rule associated with the rule ID corresponding to the smallest non-negative integer in the search results list 327, which may be the rule ID at the top of the search results list 327. By maintaining the search results list 327 in sort order, the application that invoked the search can choose to apply, for example, (1) only the first / highest ranked rule corresponding to the rule ID at the top of the (sorted) search results list 327, (2) the first N rules or the N highest ranked rules of the (sorted) search results list 327, or (3) all the rules of the (sorted) search results list 327 in order.

[0130] Based on the above description of the rule tree 320 and associated data structures shown in Figure 3A, 2 for k-bit chunks k+1 The purpose of having -2 possible chunk values ​​is more easily explained. For at least some types of lookup objects (e.g., IPv4 and IPv6 addresses), there may be one or more rule IDs mapped by a specific combination of values ​​of all bits of the chunk, and one or more different rule IDs mapped by a single bit of the chunk (or, if k>2, a combination of values ​​less than all bits of the chunk). k+1 - Another reason for having two possible chunk values ​​is that the key length may not be evenly divisible by k. For example, with a 7-bit key and k=2, the chunk ultimately processed will be the 1-bit LSB of the 7-bit key.

[0131] Applying the above to k-bit chunks (k>2) is easily understood. Each subpotion of a chunk containing the chunk's MSB(MSB) may have different rule IDs(s) mapped by the bit value combinations of that subpotion. A rule ID(s) mapped by the bit value or bit value combination for the first subpart of a chunk (e.g., "11" for k=4) may differ from a rule ID(s) mapped by the bit value combination for the second subpart of the chunk containing the first subpart (e.g., "110"), and may also differ from a rule ID(s) mapped by the bit value combination for the entire chunk (e.g., "1101").

[0132] Figure 3B is a flowchart illustrating an example of how to insert a rule tree key. An exemplary rule tree may consist of internal nodes and other data structures, as illustrated in Figure 3A. A rule tree instance may be created and initialized with a single root internal node (not shown in Figure 3B) configured with a stride k, for example k=2 as in Figure 3A, where both the internal and external vector bit arrays may be initialized to all zeros, e.g., all "0" elements / bits, and may be stored as internal nodes in the sparse arrays of internal data and external data of the root internal node. The method in Figure 3B can be performed, for example, by TIG102. Other computing devices can also perform the method in Figure 3B (e.g., one or more SPMS107), but for convenience, the method is described in the context of its execution by TIG102. One or more steps of the exemplary method in Figure 3B can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0133] Step 331 can accept a "Key" value as input. The key value can be, for example, a bit string representing an IP address prefix (which can be a complete IP address). The RuleID value is, for example, a RuleID, which is a non-negative integer corresponding to a packet filtering rule (composed by a policy) with a matching criterion consisting of the key value / IP address prefix. The key value (IP address prefix) may also be an index mapped to the RuleID value.

[0134] In step 332, the key bit sequence / IP address prefix is ​​divided into substrings of length k. For example, if the stride k=2, the bit sequence is divided into 2-bit subsequences, i.e., 2-bit chunks. The chunks are processed from the most significant bit to the least significant bit of the associated IP address prefix.

[0135] In step 333, the root internal node, which may be identified as the current internal node, may be accessed to process the first chunk, which may be identified as the current chunk.

[0136] In step 334, TIG102 can determine whether the current chunk, which may be a bit sequence of length k or less, is a chunk consisting of the k-th least significant bit of the last chunk of the key, for example, an IP address prefix (which may be a full IP address, i.e., a / 32 IPv4 address prefix or a / 128 IPv6 prefix).

[0137] If the current chunk is the last chunk, in step 335 TIG102 may set or reset the external vector bit array element at the index / position corresponding to the current / last chunk to "1". For example, referring to Figure 3A, if the last chunk is the bit sequence "10", TIG102 may set the external vector bit array 321b element at the position corresponding to "10" to "1". In step 335, TIG102 may also insert the data value / rule ID into the internal data and external data sparse array 330 data structure associated with the current internal node (conceptually potentially updating the external data reference list of the internal node), return / signal that the insertion was successful, and stop the insertion process.

[0138] If the current chunk is not the last chunk, TIG102 can access the next chunk of the key in step 336 and set or reset the internal vector bit array element at the index / position corresponding to the next chunk to "1". As part of step 336, TIG102 can create and initialize a descendant internal node if a corresponding descendant internal node does not exist.

[0139] In step 337, TIG102 can move to / access the descendant internal node corresponding to the next chunk, identify the next chunk as the current chunk, identify the descendant internal node as the current internal node, and repeat part of the method starting from step 334.

[0140] The method in Figure 3B may correspond to different rule IDs, but can be performed separately in relation to different keys that may correspond to overlapping portions of the search object. For example, the method in Figure 3B may be performed in relation to a first key (e.g., "11010010") that corresponds to the 8 MSB of an IP address and maps to one or more first rule IDs. The method in Figure 3B may be repeated in relation to a second key (e.g., "110100101101") that corresponds to the 12 MSB of an IP address and maps to one or more second rule IDs. When the resulting rule tree is subsequently searched based on the search object "110100101101..." (where "..." is an abbreviation of the IP address for simplicity), for example, as described later in relation to Figure 3C, both one or more first rule IDs and one or more second rule IDs may be added to the search results list.

[0141] Figure 3C is a flowchart illustrating an exemplary method for searching a rule tree, such as exemplary rule tree 320. For an input IP address, the process can search a rule tree that matches all prefixes of the input IP address and return all rule ID values ​​associated with each matching prefix. The method in Figure 3C may be performed, for example, by TIG102. One or more steps of the exemplary method in Figure 3C can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0142] In step 351, the method may accept a search object value as input, which could be, for example, a bit string representing a full IP address, or an IP address that constitutes a packet (in transit) to be filtered against a cybersecurity policy that includes packet filtering rules. The search results list can be initialized as an empty list.

[0143] In step 352, the search object value / bit string / IP address may be divided into substrings of length k. For example, if the stride k=2, the bit string is divided into a 2-bit substring, or 2-bit chunk. The chunks are processed from the most significant bit to the least significant bit of the associated IP address.

[0144] In step 353, the root internal node, which may be identified as the current internal node, may be accessed to process the first chunk, which may be identified as the current chunk.

[0145] In step 354, the external vector bit array of the current internal node can be accessed. For each position / index in the bit array corresponding to the current chunk, if the corresponding element of the bit array is "1", there may be one or more rule ID values ​​associated with the search object value / IP address. For example, referring to internal node 321 in Figure 3A, if the current chunk is bit string "10", the "1" element of the external vector 321b bit array at position / index "10" may indicate that there is one or more associated rule ID values ​​(which may be stored as external nodes in the internal data and external data sparse array 330). Note that in Figure 3A, the first bit of the current chunk is "1", and the "0" element of the external vector bit array at position / index "1" may indicate that there are no associated rule ID values.

[0146] In step 355, all external data or rule ID values related to the current chunk are collected from the internal data and external data sparse array (e.g., internal data and external data sparse array 330), inserted into the search result list (e.g., search result list 327), and can be maintained in sorted order.

[0147] In step 356, access can be made to the internal vector bit array of the current internal node (e.g., internal vector bit array 321a).

[0148] In step 357, for the next chunk of the search object value / bit string / IP address (if any), if the internal vector bit array element at the position / index corresponding to the next chunk is "1", there may be a descendant internal node for the next chunk. If the bit array element of the internal vector at the position / index corresponding to the next chunk is "0", there may be no descendant internal node for the next chunk.

[0149] If there is a descendant internal node for the next chunk, in step 359, TIG102 can migrate / access to the descendant internal node corresponding to the next chunk, identify the next chunk as the current chunk, identify the descendant internal node as the current internal node, and repeat a part of the method starting from step 354.

[0150] If there is no descendant internal node for the next chunk, in step 358, TIG102 can return the search result list and stop searching the rule tree.

[0151] As can be understood from the above, each chunk of the search object corresponds to a combination of bit values that can result in no rule ID, one, or multiple rule IDs being added to the search result list. As an example, the search object is k1k2k3k 4··· k n It consists of the following chunks. Chunk k1 may consist of a combination of bit values ​​in which the corresponding internal vector array elements of the root internal node indicate the corresponding descendant internal node (at the second level of the rule tree), but there are no external vector array elements indicating that there is one or more corresponding rule IDs. Chunk k2 may consist of a combination of bit values ​​in which the corresponding internal vector array elements of the second-level descendant internal node indicate the corresponding descendant internal node (at the third level of the rule tree), but there are one or more external vector array elements indicating that there is one or more corresponding rule IDs. Chunk k3 may consist of a combination of bit values ​​in which the corresponding internal vector array elements of the third-level descendant internal node indicate the corresponding descendant internal node (at the fourth level of the rule tree), but there are no external vector array elements indicating that there is one or more corresponding rule IDs. Chunk k4 may consist of a combination of bit values ​​in which the corresponding internal vector array elements of the fourth-level descendant internal node indicate that there is a corresponding descendant internal node (at the fifth level of the rule tree), but there are one or more external vector array elements indicating that there is one or more corresponding rule IDs. Each of the additional chunks k5 through kn may similarly consist of a set of bit values ​​that may or may not contain an external vector array element indicating the presence of one or more corresponding rule IDs. Furthermore, each of the chunks k5 through kn may consist of a set of bit values ​​that indicates the internal vector does not have a descendant internal node corresponding to the next lower level.

[0152] Figure 3D shows how the method in Figure 3C is performed concurrently with the membership test of the set data structure, similar to the method described in relation to the previous figure. One or more steps of the exemplary method in Figure 3D may be performed by, for example, TIG102, but can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. In the method of step 3D, step 362 (which may be the same as step 216 in Figure 2B) is performed by the first processing thread A6, and steps 365-373 are performed by a concurrently running second processing thread B6. In the method of Figure 3D, when a lookup object is received in step 361, thread A6 starts the membership check and thread B6 starts the rule tree lookup. If TIG102 determines in step 362 that there is membership, thread A6 can terminate in step 363. If TIG102 determines in step 362 that there is no membership, thread B6 terminates in step 364, and any rules determined using other values ​​from the packet in transit may be applied. Steps 365-373 may be the same as steps 351-359 in Figure 3C, but step 372 may further include terminating the execution of thread A6 if necessary. In a variation of the method in Figure 3D, step 362 is executed before step 365, the "Yes" membership determination becomes a prerequisite for executing the remaining method steps, and steps 363 and 364 are omitted, allowing the method to be executed in a single thread similar to the method in Figure 2A. In yet another modification of the method in Figure 3D, the membership test may be omitted (similar to the method in Figure 2C).

[0153] The optimal index data structure for a given threat indicator type can vary depending on requirements and performance. For example, for IPv4 and IPv6 threat indicators, a rule tree with specific parameter values, such as a stride length of 4, might be better than a flat hash map. Further, if both IPv4 and IPv6 threat indicators are being used (e.g., within a policy), separate rule tree instances can be used to improve performance, such as one rule tree instance for mapping IPv4 threat indicators to corresponding rules and another for mapping IPv6 threat indicators to corresponding rules within the policy. However, note that this can be an error as a rule tree might interpret an IPv4 indicator as a subkey of an IPv6 indicator, regardless of factors like performance, and separate rule tree instances for IPv4 and IPv6 might be needed. Also, IPv6 prefix indicators of 32 bits or less can be misinterpreted as subkeys of IPv4 indicators by a rule tree, which is also an error. On the other hand, for threat indicators such as domain names or URIs, a hash map might be more suitable. Further, performance might be improved if a single instance of a hash map is used to map base / registrable domain name indicators, FQDN indicators, and URLs to corresponding rules in the policy.

[0154] In addition to supporting a temporally and spatially efficient policy search, the index data structure can also support other requirements and / or characteristics related to efficient TIG-based cybersecurity policy enforcement for network protection, similar to a set data structure.

[0155] These requirements and / or characteristics can constitute a time-efficient key / threat indicator lookup. For example, an index data structure may be selected (at least partially) such that the estimated time to look up key / threat indicators within the index data structure is shorter than the current average packet transmission rate in transit, so as not to overflow the packet buffer in transit.

[0156] These requirements and / or characteristics may include time-efficient and / or space-efficient key / threat indicator insertion. For example, the index data structure may be selected (at least partially) based on the expected time to insert keys (e.g., threat indicators) and their associated mapped values ​​(e.g., one or more rule identifiers) into the index data structure, and / or the expected additional space / memory required to incorporate the keys and values ​​into the index data structure.

[0157] These requirements and / or characteristics may include the removal of key / threat indicators in a time-efficient and / or space-efficient manner. For example, an index data structure may be selected (at least partially) based on the expected time to remove keys (e.g., threat indicators) and their associated mapped values ​​(e.g., one or more rule identifiers) from the index data structure, and / or the expected space reduction as a result of the removal. Note that in some scenarios, a removal operation may not be necessary. In such cases, note that the relevant index data structure may differ from that in scenarios where a removal operation is necessary.

[0158] These requirements and / or characteristics may consist of a time- and / or space-efficient collection of index data structures for the overall policy. For example, the index data structures may be selected (at least partially) based on the time it takes to populate the index data structures with all key / threat indicators and associated mapped values ​​for the entire policy. Note that in some scenarios, for example, policies enforced by TIG may be updated periodically or cyclically with different information, such as in response to changes in CTI, which may influence the selection of index data structures.

[0159] Requirements and / or characteristics associated with efficient TIG-based cybersecurity policy enforcement to protect networks may consist of dynamic (runtime) insertion and removal of key / threat indicators. For example, in some scenarios, an index data structure may be (at least partially) chosen because it may be required to update the policy lookup data structure during execution, such as when TIG is actively enforcing policies on packets in transit. For example, instead of replacing the currently enforced policy with a new / updated policy, TIG may dynamically update the currently enforced policy during execution by, for example, inserting a new CTI-derived packet filtering rule into the currently enforced policy, in which case the relevant threat indicators and rule IDs must be inserted into the relevant set and index data structure during execution, and / or the currently enforced policy may be updated by removing the CTI-derived packet filtering rule, in which case the relevant threat indicators and rule IDs may be removed from the relevant set and index data structure during execution. For example, a rule tree may be implemented to support dynamic / runtime insertion and removal, and as a result be able to support these scenarios.

[0160] Requirements and / or characteristics related to the efficient implementation of TIG-based cybersecurity policies to protect networks may consist of information security for key / threat indicators. For example, in some scenarios, an index data structure may be chosen (at least partially) because it is necessary to protect key / threat indicators within a set or index data structure so that they are not stored in plaintext or native bit string representations (e.g., 32-bit IPv4 addresses) or using reversible functions, for example, so that key / threat indicators can be concealed from (malicious) observers. As mentioned above, key / threat indicators in flat hash maps and fingerprint rule trees that store fingerprints may be protected because the hash function used to compute the fingerprint may be an irreversible function and therefore may not be substantially reversed to the original representation. Storing rule trees, e.g., IP addresses or reversible compression, may not be protected in this way, and if not, a (malicious) observer could read the key / threat indicators in their original representation. Similarly, in probabilistic set data structures like Bloom filters and cuckoo filters, the use of (irreversible) hash functions may mean that the threat indicators included in the set may not be determined by a (malicious) observer.

[0161] In general, regarding the above requirements and characteristics for set and index data structures, trade-offs may be necessary between the characteristics and requirements of different index and set data structures in relation to TIG performance in policy enforcement. Time requirements are traded off with space requirements; for example, more space / memory is used to reduce policy lookup time. One example mentioned and discussed above is the use of set data structures to perform existence / membership tests before or concurrently with performing policy lookups. Set data structures are not necessary, and space / memory requirements increase, but TIG policy enforcement time performance can be improved.

[0162] As described above, the index data structure for searching CTI-derived policies may consist of one or more packet filtering rules of CTI-derived policies, or it may consist of keys corresponding to associated threat indicators, which are mapped to one or more rule identifiers of one or more rules associated with the key / threat indicator. For example, the domain name threat indicator "www.badactors.com" that constitutes the matching criterion for the 10th rule in an ordered list of rules (e.g., policies) with rule identifier "rule-10" may be represented in the index data structure as the key "www.badactors.com" mapped to the value "rule-10". Then, for example, when TIG102 is filtering packets in transit that contain the domain name "www.badactors.com", TIG102 searches the index data structure for the key / threat indicator "www.badactors.com", and for example, TIG102 tests whether the key / threat indicator exists in the index data structure (which is True). Therefore, in effect, searching for a key / threat indicator in an index data structure is equivalent to testing for the existence / membership of an element / threat indicator in the set of all keys / threat indicators contained in the index data structure. Thus, an index data structure corresponding to a CTI-derived policy can eliminate the need for a set data structure to test for the membership of threat indicators that constitute the matching criteria of the packet filtering rules that make up the CTI-derived policy.

[0163] However, in some cases, using set data structures in addition to index data structures can improve the overall performance of TIG packet filtering. Existence / membership tests of set data structures can be significantly faster (on average) than corresponding key lookups of index data structures. As mentioned above, in reality, only a small fraction of the actual packets in transit that are filtered by TIG102 will match the CTI-derived packet filtering rules of the cybersecurity policy enforced by TIG. Therefore, by performing existence / membership tests of potential threat indicators in set data structures before (or simultaneously with) performing the (slower) key lookups of potential threat indicators in index data structures, many unnecessary key lookups can be avoided (or stopped or abandoned before completing the key lookups). This can improve the overall packet filtering performance of TIG102 by reducing the average / expected time to filter each packet. Therefore, set data structures can be added to main memory until the budget / allocation is met, as long as it does not exceed TIG's main memory budget or allocation for storing all of the policy's index data structures. For example, a 2^32-bit array set data structure to represent IPv4 addresses within a policy supports very fast existence / member testing and requires 512MB (i.e., 2^29 bytes) of memory. If you have more than 512MB of main memory budget remaining, you can add this set data structure to improve overall packet filtering and policy enforcement performance.

[0164] The index data structure and set data structure described above can be used by TIG102 to efficiently enforce CTI-derived policies on packet traffic in transit traversing the boundary between a protected network and an unprotected network (for example, TIG102 may be an interface between a protected network and an unprotected network). Figure 4 is a block diagram illustrating an example of a method for efficient policy enforcement, consisting of selecting and using one or more data structures and associated usages described herein. The method in Figure 4 consists of three steps. Step 411 (Policy Specification) determines an (ordered) list of packet filtering rules and a threat indicator associated with each rule. Step 412 (Data Structure Creation) creates and populates an instance of an index data structure that maps threat indicators (keys) to rule identifiers (values). Step 412 may include creating an instance of a set data structure that can reduce policy lookup latency. In step 413 (Policy Search and Enforcement), one or more TIG102s can process a packet in transit by (a) identifying potential threat indicators in the packet, (b) searching for potential threat indicators in the input data structure created in step 412, and (c) applying arbitrary matching rules to the packet, for example, by blocking, logging, or capturing the packet.

[0165] The policy designation in step 411, which may be performed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices, may take into account and / or consider a large number of threats and potential threats. In a typical scenario, for example, and as of the time of this disclosure, the policy may be derived from millions (e.g., hundreds of millions) of threat indicators that may be delivered by thousands of CTI feeds that may be supplied by tens or hundreds of CTIP106s. Typically, each CTI feed consists of one or more (e.g., many) threat indicators of a single type, the type of which may be, for example, IPv4, IPv6, domain name, or URL. Each CTI feed may be mapped to or associated with a single packet filtering rule that includes, for example, matching criteria which may be threat indicators composed of the CTI feed, a rule decision action which may block / deny / drop matching packets or allow / pass / forward matching packets, other packet filtering actions such as log / flow log, capture / flow capture, redirect, spoof response, a rule policy processing instruction which may be either "quick" (e.g., stop policy search) or "continue" (e.g., continue policy search for other matching rules), and metadata such as CTI provider and CTI feed name, indicator type, associated threat / attack type, confidence level, risk score, etc. Rules consist of and / or may be ordered in an ordered list (policy), primarily ordered by indicator fidelity (e.g., URLs are more fidelity than domain names and more fidelity than IP addresses (IPv4 or IPv6)) and subordered by disposition action (e.g., one of "block" or "allow," with "block" taking precedence over "allow").Therefore, for example, in an ordered list, a "block" URL rule may precede a "allow" URL rule, an "allow" URL rule may precede a "block" domain name rule, a "block" domain name rule may precede an "allow" domain name rule, an "allow" domain name rule may precede a "block" IP address rule, and a "block" IP address rule may precede an "allow" IP address rule. Each rule in an ordered list is associated with a rule identifier that corresponds to the order or rank of the rules in the ordered list, and the relative rank of two different rules in an ordered list is determined by the associated rule identifier (rule ID).

[0166] Step 412 may be executed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices, and instances of data structures are selected according to desired requirements and characteristics (such as those described above), and then in step 411, threat indicators and rule IDs for the specified policy may be populated. For example, the selected index data structure may be (a) one instance of a rule tree with a stride length k=4 for all IPv4 ( / 32) threat indicators, (b) one instance of a rule tree with a stride length k=4 for all other (non- / 32) IPv4 CIDR indicators, and (c) one instance of a flat hash map for domain name and URL indicators. (In this example, there are no IPv6 threat indicators). Alternatively, an instance of a set data structure consisting of a bit array of size 2^32 bits may be selected to reduce the policy lookup time for IPv4 addresses included in the policy. In this example, a set data structure (e.g., a Bloom filter) is not used to reduce the policy lookup time for domain names and URLs that may be included in the policy. As another example, suppose the initial selection of data structures exceeds the memory requirements. Instead of modifying the data structure to reduce memory usage, or in addition to doing so, the policy can be modified to reduce memory usage by repeating step 411, for example, by removing one or more unreliable CTI feeds.

[0167] Figure 5 is a block diagram illustrating an example of a method for selecting a data structure for a network protection policy. The method in Figure 5 may be performed as part of step 412 (Figure 4). For convenience, Figure 5 illustrates an example in which the method is performed by SPMS107. However, the method in Figure 5 may be performed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices. One or more steps of the exemplary method in Figure 5 can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0168] In step 501, SPMS107 receives the policy specified in step 411. In step 502, SPMS107 parses the policy and divides the policy rules into n rule groups G1~G n Rules are grouped based on the type of threat indicator associated with them. For example, rules whose matching criterion is an IPv4 address are assigned to one rule group, rules whose matching criterion is an IPv6 address are assigned to another rule group, rules whose matching criterion is a domain name are assigned to yet another rule group, rules whose matching criterion is a URL are assigned to yet another rule group, and so on. The grouping can be more granular and may consist of multiple groups based on different variations of a particular type of threat indicator. For example, one group may consist of rules whose matching criterion is an IP address associated with a certain geographical region, and another group may consist of rules whose matching criterion is an IP address associated with a different geographical region. Another example is a group of rules whose matching criterion is the host portion of a URL, and another group may consist of rules whose matching criterion is the host portion and path portion of a URL.

[0169] Alternatively, rules may be grouped based on rules that have a matching criterion for threat indicators that share one or more common characteristics. For example, all rules with a matching criterion for the domain name threat indicator where the TLD is ".com", and all rules with a matching criterion for the URL domain name threat indicator where the hostname portion of the URL TLD is also ".com", may be grouped (for example, so that a single flat hash map index data structure can be used for all those rules). Rules may similarly be grouped based on other TLDs (for example, for each of the 10 most popular TLDs), with a separate flat hash map index data structure used for each of those groups. As a further example, one group may consist of rules whose matching criterion is the host portion of an IP address (instead of a domain name), and another group may consist of rules whose matching criterion is the host portion of a domain name. As yet another example, one group may consist of rules whose matching criterion is the complete ( / 32) IPv4 address, and another group may consist of rules whose matching criterion is the partial (non- / 32) IPv4 address (e.g., prefix).

[0170] In step 503, SPMS107 initializes counter m to 1. In step 504, SPMS107 sets rule group G m Select an index data structure. The selectable index data structures can consist of those described herein (e.g., flat hash map, rule tree) and / or other types of data structures. The selection in step 504 is for rule group G m Based on the type of threat indicator(s) associated with the rule, based on the characteristics / features of the selectable type of data structure, and based on any predetermined considerations related to the policy (e.g., prioritizing improved search speed over reduced memory usage, or vice versa, enabling runtime updates, protecting information within the data structure), step 504 can be performed in various ways. For example, rule group G m A default selection is made based on the type of threat indicator associated with that group, and the user may be prompted to accept that default selection or make an alternative selection. As another example, each of the various data structures may be weighted based on one or more policy-related considerations, and the data structure with the highest weight may be selected by default.

[0171] In step 505, SPMS107 can determine whether the index data structure selected in step 504 is a rule tree. If yes, the stride k parameter value is determined in step 506. Step 506 may be configured as shown in Figure 9, which will be described later. If the index data structure selected in step 504 is not a rule tree, step 506 can be omitted. In step 507, SPMS107 determines the rule group G selected in step 504. m For the selected index data structure, it can be determined whether to use the set data structure for the membership test. The decision in step 507 can be made in various ways. For example, if memory resources are likely to be unavailable, the membership test (and any associated set data structure(s) that require memory resources) may be omitted. As another example, a default selection (e.g., "Yes" or "No") may be made for all rule groups, and the user may be prompted to override the default selection. As yet another example, the default selection may be based on the estimated time to perform the lookup of the selected index data structure and the estimated time to perform the membership test of the set data structure, and the default decision in step 507 may be "No" unless the estimated time for the membership test is less than the estimated time to look up the index data structure (or less than a given coefficient). The estimated time is, for example, for rule group G m It can be determined based on the size (for example, based on the number of rules and / or the number of levels in the index data structure). SPMS107 is in step 507, rule group G m If, in relation to the selected index data structure, it is decided to use a set data structure for the membership test, SPMS107 selects a set data structure type in step 508. The selection in step 508 may be based on user input, a default selection, and / or other considerations (e.g., memory requirements). The data structure selected in step 508 may consist of a Bloom filter or other probabilistic data structure. Alternatively, the data structure selected in step 508 may consist of a flat hash map data structure and / or a rule tree data structure modified for use as a membership test / set data structure. Such modifications, and the use of such modified flat hash map data structures or rule tree data structures for membership testing, are described in relation to Figures 18-24. If the decision in step 507 is not to use a set data structure for membership, step 508 may be omitted.

[0172] In step 509, SPMS107 determines whether m=n, and if m is not equal to n, in step 510 increments the m counter by 1. After step 510, SPMS107 may repeat part of the method that began in step 504. If m=n in step 509, the memory requirements for the policy may be estimated based on the selected index data structure and the selected set data structure (if any). Step 511 may consist of the method shown in Figure 8, described later. In step 512, SPMS107 may determine whether the memory requirements estimated in step 511 are acceptable (e.g., whether the requirements are within an acceptable memory budget corresponding to the TIGs that enforce the policy). If no, in step 513 SPMS107 may modify the selection of one or more data structures, modify the grouping of one or more rules (e.g., combine separate groups corresponding to the same threat indicator type into one group), and / or decide not to use set data structures for membership testing in relation to one or more groups. Step 513 may include, for example, outputting a display of groups and their associated threat indicator types, selected index data structures and the memory requirements for each of those selected data structures, set data structures (if selected) and the memory requirements for those set data structures, estimated memory requirements for other types of index data structures or set data structures if used alternatively, and / or estimated search times associated with the selected index data structures and / or alternative index data structures. Step 513 may further include receiving, based on those output displays, one or more selections for alternative groupings, one or more selections for alternative index data structures, and / or one or more decisions not to use membership tests. After step 513, the SPMS 107 may repeat part of the method that began in step 511.If SPMS107 determines in step 512 that the memory requirements estimated in step 511 are acceptable, it can then populate the data structure with data.

[0173] Figure 6 is a block diagram illustrating an example of a method for inputting the data structure of a network protection policy. The method in Figure 6 may be performed as part of step 412 (Figure 4). For convenience, Figure 6 is illustrated based on an example in which the method is performed by SPMS107. However, the method in Figure 6 may be performed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices. One or more steps of the exemplary method in Figure 6 can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0174] In step 601, SPMS107 can initialize counter m to 1. In step 602, SPMS107 can initialize rule group G m For this purpose, it is possible to determine whether the selected index data structure is a flat hash map or a rule tree. If the selected index data structure is a flat hash map, SPMS107 can initialize counter p to 1 in step 603. In step 604, SPMS107 determines rule group G m Rule R p For a threat indicator of the matching criteria, the segment of that threat indicator can be determined. The segment is, for example, the label L of the domain name D, the path segment S of the URL path P, etc. In step 605, SPMS107 can hash and / or compress the segment. Whether a segment is hashed or compressed can be determined, for example, based on whether the index data structure relies on hashing (for example, as in the examples in Figures 2A, 2B, and 2D), data compression (for example, as in the examples in Figures 2C and 2E), or a combination of hashing and data compression (for example, as in the example in Figure 2F). In step 606, SPMS107 applies the hashed and / or compressed segment to rule R p Along with the rule ID, it can be inserted into the flat hash map index data structure. In step 607, SPMS107 is rule group G m Whether there are further rules (for example, whether p=x, where x is rule group G) m The number of rules in rule group G can be determined. m If there are further rules, SPMS107 can increment the p counter by 1 in step 608 and repeat part of the method starting from step 604. Rule group G m If there are no further rules, SPMS107 will use rule group G. m If the use of a set data structure is chosen for membership testing of the index data structure (for example, in the method shown in Figure 5), then the set data structure is entered in step 609. Rule group G m If the use of a set data structure is not chosen for the index data structure, SPMS107 may skip step 609. In step 610, SPMS107 determines whether m=n. If yes, SPMS107 can increment the m counter by 1 in step 611 and repeat part of the process starting from step 602. If no, the input data structure of the policy can be used for policy lookup and enforcement.

[0175] In step 602, when SPMS107 determines that the selected index data structure of rule group G m is a rule tree, SPMS107 can execute steps 632 to 638. Steps 632 to 634 may be the same as or similar to steps 332 to 334 of the method in FIG. 3B, and steps 635 and 636 may be the same as or similar to steps 336 and 337 of the method in FIG. 3B. Step 637 may be the same as or similar to step 335 of FIG. 3B except that step 638 is executed at the end of step 637. In step 638, when it is selected to use the set data structure for the membership test of the rule group G m index data structure, that set data structure may be input. When the use of the set data structure for the rule group G m index data structure is not selected, SPMS107 may skip step 638. In step 639, SPMS107 determines whether m = n. If No, SPMS107 can increment the m counter by 1 in step 611 and repeat a part of the method starting from step 602. If Yes, the input data structure of the policy can be used for policy search and enforcement.

[0176] As part of step 413 (FIG. 4), TIG102 (or each of multiple TIG102s) can load or configure the input data structure (e.g., using the method in FIG. 6) into its main memory in step 412 so that the packet processing and policy enforcement logic of TIG102 can efficiently access them. Then, TIG102 can process the packets in transit by searching those data structures based on the values extracted from the fields of those packets in transit and enforce the policy rules found as a result of that search.

[0177] Figure 7 is a block diagram illustrating an example of a policy lookup and enforcement method using one or more data structures as described herein. One or more steps of the exemplary method in Figure 7 can be rearranged (e.g., executed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. In step 711, TIG102 can receive a packet in transit. In step 712, TIG102 can extract one or more field values ​​from each of several fields of the packet. The extracted field values ​​are then forwarded to the corresponding processing thread among several concurrently running processing threads to be used as lookup objects for the policy (or part of the policy) and for looking up the input index data structure. For example, the field value 700.1 from a packet field of type 1 is forwarded to thread 713.1, the field value 700.2 from a packet field of type 2 is forwarded to thread 713.2, the field value 700.3 from a packet field of type 3 is forwarded to thread 713.3, and so on, with the field value 700.n from a packet field of type n being forwarded to thread 713.n. Each of the threads 713 may constitute one of the index data structure lookup methods described herein (e.g., the method according to Figures 2A-2F, 3C, or 3D), any or all of the threads 713 may constitute a separate processing thread for testing set membership (e.g., the method according to Figures 2B-2F and 3D), any or all of the threads 713 may configure set membership testing in the same processing thread as the index data structure lookup, and / or any or all of the threads 713 may omit set membership testing.

[0178] Alternatively, a single field value from a packet may be forwarded to each of multiple concurrent processing threads (for example, the field value 700.4 may be forwarded to the first processing thread 713.4(1), the concurrent second processing thread 713.4(2), the third concurrent processing thread 713.4(3), and so on). This may occur, for example, if the first processing thread consists of a first index data structure created for a first group of rules sharing a first characteristic, and the concurrently running second processing thread consists of a second index data structure created for a second group of rules sharing a second characteristic, and the field value from the packet may match a key in either the first or second index data structure.

[0179] As an example of a scenario in which packet field values ​​may be forwarded to multiple processing threads, the first index data structure (of the first processing thread) may be a flat hash map index data structure containing keys corresponding to domain names and / or URLs whose top-level domain is ".com", mapped to the rule IDs of the first group of policy rules. The second index data structure (of the second processing thread) may be a flat hash map index data structure consisting of keys corresponding to domain names and / or URLs whose top-level domain is ".net", mapped to the rule IDs of the second group of policy rules. The third index data structure (of the third processing thread) may be a flat hash map index data structure consisting of keys corresponding to domain names and / or URLs whose top-level domain or effective top-level domain is not ".com" or ".net", mapped to the rule IDs of the third group of policy rules. The field values ​​from the packet may be domains or URLs forwarded to the first, second, and third processing threads, respectively, in order to simultaneously look up the first, second, and third index data structures.

[0180] Another example of a scenario where packet field values ​​may be forwarded to multiple processing threads is that the fourth index data structure (of the fourth processing thread) may be a rule tree index data structure that maps keys corresponding to full ( / 32) IPv4 addresses to rule IDs of the fourth group of policy rules. The fifth index data structure (of the fifth processing thread) may be a rule tree index data structure that maps keys corresponding to partial (e.g., non- / 32) IPv4 addresses to rule IDs of the fifth group of policy rules. The field values ​​from the packet may be full IPv4 addresses that are forwarded to the fourth and fifth processing threads, respectively, to simultaneously look up the fourth and fifth index data structures.

[0181] For a given packet, one, some, all, or any of the processing threads 713 may output one or more rule IDs determined based on an index data structure lookup. In step 714, TIG102 may sort the rules corresponding to those rule IDs based on their priority and apply some or all of those rules based on their priority. Applying a rule may involve performing one or more actions specified by the rule (e.g., blocking / dropping / rejecting the packet, allowing / forwarding / passing the packet, logging the packet, capturing the packet (e.g., saving a copy of the packet), redirecting or rerouting the packet, modifying or transforming the packet in any way to protect the network, generating and / or sending a response to the packet, etc.). As shown in connector C41, TIG102 may repeat the method in Figure 102 for the next packet in transit.

[0182] The data structure described above can be used to solve the following problems in the policy enforcement application of this disclosure. For example, when there is a large number of CTIs, such as tens of millions or hundreds of millions of indicators, creating a TIG-enforceable policy from the indicators may take a relatively long time, e.g., several minutes, consisting of inputting the indicators and rule IDs into the data structure, which can be problematic if, for example, the time to input the data structure approaches the policy update cycle time (e.g., every 15 minutes). Furthermore, the size of the input data structure, or the main memory consumed by it, may exceed the main memory budget of TIG102 allocated for policy enforcement. In such cases, one or more corrective actions may be required to reduce the size, e.g., by selecting a more space-efficient data structure, reducing the number of threat indicators, or removing one or more set data structures for testing indicator membership in the policy. Subsequently, the data structure selection / configuration process and / or data structure grouping process may need to be repeated to check whether the memory budget has been exceeded.

[0183] It is useful if the main memory requirements needed for a given set of threat indicators, the policies to be enforced (e.g., an ordered list of CTI-derived packet filtering rules), and the selection of a given data structure can be accurately estimated before inputting the threat indicators and rule IDs into the data structure. For all data structures described herein, an accurate estimate of the memory / space requirements can be efficiently calculated as follows: For a bit array set data structure for IPv4 address indicators, the memory requirement is 2^32 bits (e.g., "4Gb") or 2^29 bytes (e.g., "512MB"). For a bit array set data structure for IPv6 address indicators, the value Z of the most significant bit of the IPv6 address indicator is selected and used to input the bit array; for example, if Z=32, the memory requirement may be 2^Z bits. For probabilistic set data structures, such as Bloom filters and cuckoo filters, well-known formulas for the memory requirements of the set of elements and a given false positive rate can be used, and by treating the threat indicators as elements, an accurate estimate can be calculated using such formulas. For index data structures, there generally exists an experimentally determined constant coefficient X that can be used to calculate an accurate estimate of the memory requirements for storing N indicators of a specific type (e.g., IPv4, IPv6, domain name, URL, etc.) in a particular type of index data structure (e.g., rule tree or flat hash map). That is, for a particular implementation of the index data structure type (e.g., flat hash map), a number N of indicators of a particular type (e.g., URL), and an (experimentally determined) coefficient X for flat hash map and URL, the memory requirement M can be efficiently calculated as M = NX bytes. Thus, for a set of indicators, policies to be enforced, and a choice of data structure, an accurate estimate of the main memory requirements for TIG performing policy enforcement can be easily, quickly, and efficiently calculated before the threat indicators and rule IDs are entered into the data structure.Furthermore, it is possible to immediately determine whether the main memory requirements exceed or fall below the budget. If the main memory requirements exceed the budget, corrective measures can be taken, such as selecting a more memory / space-efficient set of data structures (e.g., deselecting the IPv4 indicator bit array), modifying the policy (e.g., removing one or more low-confidence, high-capacity CTI feeds), or increasing the TIG main memory budget for policy enforcement.

[0184] Figure 8 is a flowchart illustrating an example method for calculating the memory requirements for a collection of data structures (e.g., index data structures and set data structures) selected for a network protection policy. As described above, the example method in Figure 8 may be performed, for example, as part of the method in Figure 5. For convenience, Figure 8 is illustrated based on an example in which the method is performed by an SPMS107. However, the method in Figure 8 may be performed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices. One or more steps of the exemplary method in Figure 8 can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added.

[0185] In step 801, SPMS107 can set counter m to 1. In step 802, SPMS107 can initialize the value of variable Mem_size to 0. In step 803, SPMS107 uses one of the methods described above to set rule group G m IDS_mem is a value representing the estimated amount of memory required for the selected index data structure. m It can be determined. In step 804, SPMS107 uses one of the methods described above to determine rule group G m SDS is a value representing the estimated amount of memory required for the set data structure. _ mem m This can be determined. Rule group G m If a set data structure is not used, SPMS107 will use SDS_mem m You may also set the value to 0. In step 805, SPMS107 sets the value of Mem_size to Mem_size + IDS_mem m +SDS_mem m It can be set to this. In step 806, SPMS determines whether m=n. If no, SPMS can increment m by 1 in step 807 and repeat part of the method starting from step 803. If yes, the value of Mem_size may be returned (for example, as part of the method in Figure 5).

[0186] Figure 9 is a block diagram illustrating an example of a method for determining the stride k parameter value of a rule tree (e.g., the chunk size value of the internal nodes at each level of the rule tree). For convenience, Figure 9 is illustrated based on an example where the method is performed by an SPMS107. However, the method in Figure 9 may be performed by one or more SPMS107s, one or more TIG102s, and / or one or more other computing devices. One or more steps of the exemplary method in Figure 9 can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or other steps can be added. In the examples so far, a rule tree has been used where the k value can be the same at all levels of the rule tree, but this is not required. For example, k may have different values ​​for nodes at different levels of the rule tree, and the internal rule tree nodes may be further modified to include one or more data structures indicating applicable k values.

[0187] In step 901, SPMS107 can receive the rule group (for example, as the output of step 502 of the method in Figure 5). In step 903, SPMS107 can initialize counter s to 1. In step 903, SPMS107 can initialize array K s Initial values ​​for elements k(1) to k(L) may be selected, where L is the number of levels in the rule tree below the root node, k(1) is the k value for the root node, k(2) is the k value for the internal nodes of the second level of the rule tree, and so on, and k(L) is the k value for the lowest level of the rule tree. SPMS107 can select initial k element values ​​based on a given default value, for example (for example, by setting all k element values ​​to 4).

[0188] In step 904, SPMS107 is sequence K s Memory requirement for a rule tree with k values ​​(Mem-req s ) and average search time (Ave-time) s ) can be estimated. In step 905, SPMS107 uses the estimated memory requirements and estimated average search time values ​​determined in step 904 for array K s It can be remembered together with it.

[0189] In step 906, the SPMS can determine whether a stopping condition has been reached. The stopping condition may consist of the completion of steps 904 and 905 for all possible combinations of k values, the completion of all possible combinations of k values ​​between given boundaries (e.g., all possible combinations when the initial k element values ​​change only by ±1, ±2, or ± other constants), the completion of a given number of iterations of steps 904 and 905, and / or other stopping conditions. If the stopping condition has not been reached, the SPMS 107 increments the s counter by 1 in step 907. In step 908, the SPMS 107 checks the array K s array K s-1 The first value is input, and then the values ​​of the k elements are changed according to a predetermined pattern, thereby creating the array K s It is possible to create this. After step 908, SPMS107 can repeat part of the method that began in step 904.

[0190] If SPMS107 determines in step 906 that the stopping condition has been reached, SPMS909 will check the array K for each value (or some values ​​of s) s And, Mem-req s and Ave-time s It can output a table or other display showing the values ​​of . Alternatively, SPMS107 will output the array K that has the minimum estimated memory requirement among all values ​​of s. s and the corresponding Mem-req s and Ave-time s Among the values ​​of and all values ​​of s, the array K that minimizes the estimated average search time. s and the corresponding Mem-req s and Ave-time s The value of can be highlighted (or output limited). In step 910, SPMS107 is array K s It can receive input to select an array K. In step 911, SPMS selects the array K (for example, as part of the method in Figure 5). s It can return the parameter value of k.

[0191] Figure 10 is a block diagram of an exemplary computing device 1000. Any device described herein or illustrated in any of the figures, such as any of the TIG102 or any of the SPMS107, may be partially or completely implemented using one or more computing devices such as computing device 1000. Computing device 1000 may consist of a general-purpose computing device having general-purpose hardware configured to perform one or more (or a part thereof) of the methods described herein using specific software and / or firmware, for example, or it may consist of a purpose-specific computing device having purpose-specific hardware (and / or purpose-specific software and / or purpose-specific firmware) customized for a particular function. The hardware elements of computing device 1000, and / or computing device 1000 itself, may be emulated in a virtual version of computing device 1000. Computing device 1000 may include one or more processors 1001 capable of executing computer-readable instructions of a computer program in order to perform any of the functions or other operations described herein. Instructions, along with other data, can be stored in memory 1002, which may include, for example, memory such as read-only memory (ROM) and / or random access memory (RAM), a hard drive, a magnetic disk or optical disk, a Universal Serial Bus (USB) drive, and / or any other type of computer-readable media. The data can be organized in any desired way, such as being stored in memory 1002 as instructions and organized so that it can be accessed via database software executed by one or more processors 1001.The computing device 1000 may also include a user interface 1004 for interface with one or more input devices 1005, such as a keyboard, mouse, or voice input, and for interface with one or more output devices 1006, such as a display, speaker, or printer. The computing device 1000 may also include a network interface 1003 for interface with one or more external devices, which may be part of a network outside the computing device 1000. Figure 10 shows an exemplary hardware configuration, but one or more elements of the computing device 1000 may be implemented as software or as a combination of hardware and software. The components of the computing device 1000 can be modified, such as by adding, removing, combining, or splitting. Furthermore, the elements shown in Figure 10 may be implemented using basic computing devices and components configured to perform operations as described herein. The processor 1001 and / or memory 1002 may also be implemented via one or more integrated circuits (ICs). The ICs may be, for example, microprocessors that access programming instructions and other data stored in ROM, and / or hardwired to the ICs. For example, an IC may consist of an application-specific integrated circuit (ASIC) having gates and / or other logic dedicated to the computations and other operations described herein. The IC may perform some operations based on the execution of programming instructions read from ROM or RAM, while other operations may be hardwired to gates or other logic.

[0192] Figure 11 is a block diagram showing an exemplary packet filtering appliance 1100 (e.g., TIG102), which may be located at the boundary between a first network (e.g., network 101) and a second network (e.g., network 105). The packet filtering appliance 1100 may consist of one or more processors 1104 (which may be the same as processor(s) 1001), memory 1106 (which may be the same as memory 1002), network interfaces 1108 and / or 1110 (which may be the same as network interface 1003), a packet filter 1112, and a management interface 1114 (which may be performed by processor(s) 1001, user interface 1004, input device 1005, and / or output device 1006). The processor 1104, memory 1106, network interfaces 1108 and / or 1110, packet filter 1112, and / or management interface 1114 may be interconnected via a data bus 1116 (which may consist of arrows interconnecting any of the various elements in Figures 10 and 11). Network interface 1108 can connect the packet filtering appliance 1100 to a first network. Similarly, network interface 1110 can connect the packet filtering appliance 1100 to a second network. Memory 1106 may contain one or more program modules that, when executed by the processor 1104, can configure the packet filtering appliance 1100 to perform one or more of the various operations described herein. Memory 1106 may also be used to store rules, databases, logs, and / or other information used and generated by the packet filtering appliance 1100.

[0193] The packet filtering appliance 1100 may be configured to receive policies from one or more SPMS 107s that include one or more of the index data structures and / or set data structures described herein. For example, the packet filtering appliance 1100 may receive policy 1118 from an SPMS 107 server via the management interface 1114 (e.g., by out-of-band signaling) or via the network interface 1108 (e.g., by in-band signaling). The packet filtering appliance 1100 may include one or more packet filters or packet identifiers for performing packet filtering operations (e.g., for implementing one or more processing threads for index data structure lookup and / or membership testing), and / or logic for implementing one or more packet filters or packet identifiers. For example, the packet filtering appliance 1100 may include a packet filter 1112 (which may be implemented by processor 1104 and / or one or more other processors) that can be configured to examine information related to packets received by the packet filtering appliance 1100 (e.g., from network 101) and, based on the examined information, forward such packets to one or more operators 1120.1 through 1120.n. For example, the packet filter 1112 can examine information related to packets received by the packet filtering appliance 1100 (e.g., packets received from network 101 via network interface 1108) (e.g., using one or more of the methods described herein to look up set and / or index data structures) and, based on the examined information, forward those packets to one or more operators 1120. These operators may, for example, implement dispositions and directives related to packet filtering rules that match the packets.

[0194] The configuration of the packet filter 1112 may be based on the rules included in policy 1118. For example, policy 1118 may include one or more rules specifying that packets containing specified information should be forwarded to one of the operators 1120, packets containing other specified information should be forwarded to another of the operators 1120, packets containing yet another specified information should be forwarded to yet another of the operators 1120, and so on. Each of the operators 1120 may be configured to perform one or more functions on packets received from the packet filter 1112 that correspond to the application of the rules in policy 1118. For example, one or more operators 1120 may be configured to forward packets received from packet filter 1112 to network 105, one or more operators 1120 may be configured to forward packets received from packet filter 1112 to network 105 and log the packets, one or more operators 1120 may be configured to forward packets received from packet filter 1112 to network 105 and capture a copy of the packets, one or more operators 1120 may be configured to block packets received from packet filter 1112 from proceeding to network 105, one or more operators 1120 may be configured to block packets received from packet filter 1112 from proceeding to network 105 and generate and send another packet, and so on.

[0195] As shown by the dashed arrows in Figure 11, the packet filtering appliance 1100 can perform the same operations as described above for packets received from network 105 via interface 1110 as it does for packets received from network 101 via interface 1108. For example, the packet filtering appliance 1100 may receive packets from network 105 via interface 1110, filter those packets in packet filter 1112 based on policy 1118, forward those packets to one or more operators 1120, and forward at least some of those packets from one or more operators 1120 to network 101.

[0196] The packet filtering appliance 1100 can obtain threat context information from various sources (or generate threat context information based on information obtained from various sources), some of which may be local (sources inside the packet filtering appliance 1100) and some may be remote (sources outside the packet filtering appliance 1100). For example, the processor(s) 1104 may include a clock that can be used to maintain the current time and determine the observation time of packets in transit. Referring to other examples of threat context information described above, the appliance location, appliance ID, administrator and associated security policy preferences, network type, active threat type, multi-packet multi-flow threat / attack analysis results, CTI provider and related information, threat indicator type and fidelity, threat indicator age, flow origin, flow direction, flow state, connection state, global threat context, domain name, URI, URL, domain name popularity, domain name registration status, data transfer protocol method, protocol risk, context CTI noise, etc., can each be determined by the processor 1104 for packets in transit and in response to the reception of packets in transit, based on the processing of information stored in memory 1106, based on calculations by the processor 1104, based on information received from network 101 via network interface 1108, based on information received via network interface 1110 via network 105, and / or based on information received via management interface 1114.

[0197] The functions and steps described herein may be embodied in computer-readable data or computer-executable instructions, such as within one or more program modules, which are executed by one or more computers or other devices (e.g., computing device 1000, packet filtering appliance 1100) to perform one or more of the functions described herein. Generally, program modules include routines, programs, objects, components, data structures, etc., which, when executed by one or more processors of a computer or other data processing device, perform specific tasks or implement specific abstract data types. Computer-executable instructions may be stored in computer-readable media such as magnetic disks, optical disks, removable storage media, solid-state memory, random-access memory (RAM), Ready-Only memory (ROM), flash memory (e.g., memory 1001, memory 1106). As is understood, the functions of program modules may be combined or distributed as desired. Furthermore, all or part of the functions may be embodied in firmware or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs). Certain data structures can be used to more effectively implement one or more aspects of this disclosure, and such data structures are intended to be within the scope of computer executable instructions and computer usable data described herein.

[0198] Rule trees and flat hash maps similar to those described herein can also be used as set data structures / membership test data structures. Such rule trees and flat hash maps can be used in combination with (and / or in place of) Bloom filters and / or other set data structures / membership test data structures in, for example, systems, methods, apparatus, and computer executable instructions (stored on computer-readable media), as described, for example, in U.S. Patent Application No. 2024 / 0007437, titled "Cyber ​​Protections of Remote Networks via Selective Policy Enforcement at a Central Network," incorporated herein by reference, and / or U.S. Patent No. 11,902,250, titled "Methods and Systems for Prevention of Attacks Associated with the Domain Name System," incorporated herein by reference. A rule tree can be configured as a set data structure (e.g., for IP address membership testing) by inserting only keys and not mapping those keys to rule IDs or other data values. A flat hash map can be configured as a set data structure (e.g., for domain name and / or URL membership testing) by inserting only keys and not mapping those keys to rule IDs or other data values. In either case, if the data structure is used only for set membership testing, there is no need to store rule IDs (or other data mapped to the keys). Similar to bloom filters, rule trees and / or flat hash maps are suitable data structures for use in set membership testing.This is because rule trees and / or flat hash maps facilitate efficient membership testing and allow for significant compression (and reduced memory requirements) compared to raw data (e.g., threat indicators in their original form). Furthermore, the compressed memory usage of flat hash maps can be easily adjusted by changing the hash size. Rule trees and / or flat hash maps are sometimes used interchangeably with Bloom filters. In some cases, rule trees and / or flat hash maps may be more efficient and / or perform better than Bloom filters. For example, rule trees can store IPv6 indicators more efficiently than Bloom filters.

[0199] Figure 12 shows an example network environment 2100 in which rule trees and / or flat hash maps may be used as set membership test data structures, for example, in relation to efficient cyber protection of mobile devices and associated corporate networks. The elements shown in Figure 12 can also perform the operations described in relation to the elements shown in Figure 1, and the elements shown in Figure 1 can also perform the operations described in relation to Figure 12. Mobile devices MBDV 2101 and MBDV 2102 are personally owned / operated by users belonging to a company that operates the private corporate network ENET 2160. Mobile devices MBDV2101 and MBDV2102 are subscribed to a mobile provider that operates the radio access network RNET 2120 and the mobile core network MCNET 2150. When mobile devices MBDV2101 and MBDV2102 make a mobile phone call, signaling and communication are routed through the mobile core network MCNET 2150, and may also be routed through the mobile core networks of other mobile providers (not shown in Figure 12). Mobile devices and mobile networks from 2G onward support access to the Internet and associated TCP / IP networks. Mobile devices MBDV2101 and MBDV2102 can access commonly addressed Internet servers such as the web server WSVR2151 and the malware server MALSVR2152 via the Internet 2130. Mobile device MBDV2101 can download and install the mobile device security application MBL-CYBER-APP, which implements the packet inspection, packet filtering, and packet tunneling functions described herein. The mobile device security application MBL-CYBER-APP can perform client configuration, tunneling, management and / or use of the TUNNEL-M / T data structure (described below), and other functions as described elsewhere in this disclosure.The mobile device MBDV2102 is not protected by the methods and systems of this disclosure because it has not downloaded the mobile device security application MBL-CYBER-APP.

[0200] Mobile devices and / or remote network tunnel gateways can utilize data structures that represent the rules of a security policy by characterizing threat indicators to determine whether an internet traffic packet matches the rules of the policy. To represent the entire policy, the system can collect some or all of the indicators, such as IP addresses, domain names, URIs, and certificate IDs, characterize each rule in the policy, and insert such indicators into one or more set data structures for membership testing, for example. These one or more set data structures may consist of one or more rule trees and / or one or more flat hash maps. Optionally, one or more set data structures may also include one or more probabilistic data structures (e.g., one or more Bloom filters). The set data structures may be tested to determine whether an element (such as an IP address, domain name, URI, or certificate ID) is a member of the set data structure (e.g., whether the element is a member of the set of indicators used to generate the set data structure). The set data structures may be generated by a policy management server, distributed to each mobile device and / or remote network tunnel gateway, or stored on each mobile device and / or remote network that may be protected by the policy. When a mobile device and / or remote network sends or receives an internet traffic packet, computer logic and / or applications on the mobile device and / or remote network tunnel gateway can extract any elements contained in the packet that may correspond to a threat indicator in a set data structure, such as an IP address, domain name, URI, or certificate ID, and test the set data structure(s) to determine whether such elements are members of the set of threat indicators in the set data structure(s).If a test indicates that any packet element is a member of a set of threat indicators, the packet, or a copy of the packet, may be tunneled to a tunnel gateway (e.g., a central network tunnel gateway) that may be located in the corporate private network (e.g., a central network). Once leaving the tunnel gateway, the packet is sent to a TIG where it can be filtered through policies to determine which packet filtering rules match the packet. Once the policy test by the TIG determines matching rules, actions or PTFs associated with the rules are applied to the packet to protect the network. If the action(s) or PTF(s) cause the packet to be forwarded toward the internet, the packet may pass through the associated security stack.

[0201] A policy creation and distribution management server (e.g., one associated with a policy creation and management system) can receive CTIs from multiple CTIPs and create CTI-derived policies based on the received CTIs. A CTI-derived policy may contain multiple rules. For membership testing (M / T), the management server may create a data structure consisting of one or more rule trees and / or one or more flat hash maps based on a set of elements representing multiple rules. Such a data structure (hereinafter referred to as TUNNEL-M / T) may contain one or more flat hash map data structures containing values ​​(e.g., hashes and / or compressions) based on metrics extracted from the rules in the policy, and / or one or more rule tree data structures containing or based on metrics extracted from the rules in the policy. The system may download the policy to a TIG and send the data structure TUNNEL-M / T to each mobile device associated with the enterprise network and / or each remote network tunnel gateway associated with the corresponding remote network, and other network elements including the TIG. Alternatively, TIG may, after receiving policies originating from CTI from the policy creation and distribution management server, create a data structure TUNNEL-M / T, and then send the data structure TUNNEL-M / T to each mobile device and / or remote network tunnel gateway.

[0202] Mobile devices can send packets to (or receive packets from) internet hosts. Packet values ​​such as IP addresses, domain names, and URIs can be tested for membership in the data structure TUNNEL-M / T.

[0203] TUNNEL-M / T's flat hash map data structures can be used for such membership tests. These flat hash map data structures may consist only of keys, for example, as described herein (e.g., keys based on incremental hashing, incremental compression, or a combination of incremental hashing and compression), and may not contain rule IDs or other data mapped to those keys. These flat hash maps can be searched using a method similar to that described herein, by incrementally hashing (and / or incrementally compressing) a portion of a domain name (or a portion of a URL, or a portion of another value from a packet) and comparing those incrementally hashed (and / or compressed) portions with the keys in the flat hash map. However, instead of returning a rule ID when a match is found with a key, the output will be a determination of whether the domain name (or URL or other packet value) matches the threat criteria of the policy rule. A domain name (or URL or other packet value) matches the policy rule's threat criterion (e.g., membership=TRUE) if the incrementally hashed (and / or compressed) domain name (or URL or other packet value) matches a key in the flat hash map. If the incrementally hashed (and / or compressed) domain name (or URL or other packet value) does not match any key in the flat hash map, the domain name (or URL or other packet value) does not match the policy rule's threat criterion (e.g., membership=FALSE).

[0204] Such membership tests can also use rule tree data structures. These rule tree data structures may consist only of keys (e.g., IP addresses or other parts of values, from or based on values ​​from packets) as described herein, and may lack rule IDs or other data mapped to those keys. Such rule trees can be searched using a method similar to that described herein, by traversing the level of internal nodes using a contiguous portion of IP addresses (or other values ​​from or based on values ​​from packets). However, instead of returning a rule ID when a match is found for one or more keys, the output will be a determination of whether the IP address (or other values ​​from or based on values ​​from packets) matches the threat criteria of the policy rule. An IP address (or any other value from or based on a packet) matches the policy rule's threat criterion (e.g., membership=TRUE) if it traverses the rule tree to an internal node where the possible values ​​of the IP address portion corresponding to that internal node are mapped to an external vector array element that stores "1", but does not match the policy rule's threat criterion (e.g., membership=FALSE) if it traverses the rule tree to an internal node where the possible values ​​of the IP address portion corresponding to that internal node are mapped to an external vector array element that stores "1".

[0205] If the membership test returns "True," the system can tunnel the packet into the corporate network. The packet may be filtered through TIG and the security stack. Each subsequent packet in the communication flow will be tunneled into the corporate network and filtered through TIG and the security stack. If the membership test returns "False," the system can either forward the packet directly to the internet host or process it normally without tunneling it into the corporate network.

[0206] Returning to Figure 12, the private enterprise network ENET 2160 can provide internet access to internal hosts such as PC 2161. The private enterprise network ENET 2160 may be configured so that traffic between these internal hosts and internet hosts must pass through the threat intelligence gateway TIG2170, which enforces policies originating from CTI. Traffic may also pass through the security stack SSTK2175, which may include at least one of the following: a conventional network firewall or other enterprise network security devices such as a web proxy, SSL proxy, IDS, or IPS. The threat intelligence gateway TIG2170 and the security stack SSTK 2175 can be located at or near the internet access point of the private enterprise network ENET 2160. When applying policies to packets, the threat intelligence gateway TIG2170 may generate a log of packets that match the policy rules, which is sent over the internet to the security operations center SOC2140 for attack analysis by cyber analysts using, for example, SIEM applications or packet analyzer applications.

[0207] The enterprise system server ESVR 2162 may be a private web application server hosted by the enterprise for use by enterprise users, including users who own / operate mobile devices MBDV 2101 and MBDV 2102. Mobile devices MBDV 2101 and MBDV 2102 access the enterprise system server ESVR 2162 using enterprise-provided applications that include an HTTPS client for accessing the enterprise system server ESVR 2162 on port 443 (the port for HTTPS). Once the application connects to the enterprise system server ESVR 2162, the web application displays a login form. Users can then enter their enterprise credentials to securely access the web application. Network security administrators have opened port 443 on the network firewall of the security stack SSTK 2175, allowing unsolicited inbound HTTPS connections, such as those originating from applications on mobile devices MBDV 2101 and MBDV 2102, to initiate sessions with the enterprise system server ESVR 2162.

[0208] The host tunnel gateway TGW 2163, connected to the private enterprise network ENET 2160, terminates and centralizes tunnels that may be established with mobile devices, such as the mobile device MBDV 2101 with the mobile device security application MBL-CYBER-APP installed. Similar to the enterprise system server ESVR 2162, the network administrator opens one or more ports on the network firewall of the security stack SSTK 2175 to allow unauthorized inbound tunnel traffic, such as that originating from the mobile device MBDV 2101 and other mobile endpoints with the mobile device security application MBL-CYBER-APP installed, to access TGW 2163. For example, if the tunneling protocol is in IPsec tunnel mode, the administrator opens well-known IPsec ports 500, 50, and 51. TGW2163 can decapsulate and / or decrypt incoming packets at the exit from the tunnel. TGW2163 can forward the decapsulated and / or decrypted packets to the private enterprise network ENET2160. Since these packets have a public internet address in the destination IP address field of the IP header, routers and / or switches on the private enterprise network ENET2160 can forward these packets to the internet access point and therefore to the threat intelligence gateway TIG2170, which applies CTI-derived policies to the packets.

[0209] The security policy creation and management server SPMS2141 can collect CTI from one or more CTIPs, including, for example, CTIP2142 and 2143. The SPMS2141 can also aggregate the CTI, create at least one security policy based on the CTI, and publish the security policy to subscribers, including multiple network devices such as host computers and threat intelligence gateways like the TIG2170. The SPMS2141 can create a data structure TUNNEL-M / T for each security policy and publish the TUNNEL-M / T data structure to each subscriber associated with each security policy, such as the mobile device security application MBL-CYBER-APP hosted by the mobile device MBDV2101. Alternatively, the Threat Intelligence Gateway TIG2170 may create a data structure TUNNEL-M / T for each associated mobile device, such as associated instances of the Mobile Device Security Application MBL-CYBER-APP associated with the Threat Intelligence Gateway TIG2170 for cyber protection, and may expose the current data structure TUNNEL-M / T to associated mobile devices, including each subscribed instance of the Mobile Device Security Application MBL-CYBER-APP.

[0210] Figure 13 is a flowchart illustrating the operational concept of a security policy creation and management server for configuring mobile devices and associated corporate network elements, for the mobile device and associated corporate network protection system depicted in Figure 12.

[0211] In step 2201, the security policy creation and management server SPMS2141 can download CTIs published by threat intelligence providers such as CTIP2142 and CTIP2143. In step 2202, the security policy creation and management server SPMS2141 can aggregate the CTI indicators and process the CTIs to generate security policies. The security policy creation and management server SPMS2141 can create at least one packet filtering rule, each rule containing matching criteria corresponding to a threat indicator and a corresponding action / PTF to be applied to the packet when a match is determined. The corresponding action / PTF may be determined based on several factors, including requirements provided by the operator / administrator of the threat intelligence gateway TIG2170. The security policy creation and management server SPMS2141 can collect each generated rule to create a security policy and distribute the security policy to multiple security devices, including the threat intelligence gateway TIG2170. In step 2203, the security policy creation and management server SPMS2141 can generate a TUNNEL-M / T associated with the security policy by collecting all threat indicators, including IP addresses, domain names, URIs, certificate IDs, etc., that characterize each packet filtering rule in the security policy. The security policy creation and management server SPMS2141 may insert keys that constitute or are based on indicators into one or more rule trees and / or one or more flat hash maps of the TUNNEL-M / T, and may test whether the rules of the security policy match the packet elements being tested.

[0212] In step 2204, the SPMS may expose at least one of the security policy and data structure (TUNNEL-M / T) to multiple network security devices. The security policy creation and management server SPMS2141 can send at least one of the security policy and data structure TUNNEL-M / T to multiple network subscribers, including the threat intelligence gateway TIG2170 and multiple mobile devices associated with the corporate network. The threat intelligence gateway TIG2170 can download the security policy and configure packet filtering logic with the security policy. In step 2205, the mobile device security application MBL-CYBER-APP on mobile device MBDV2101 can download the data structure TUNNEL-M / T and configure tunneling logic using the data structure TUNNEL-M / T. If mobile device MBDV2102 does not have the mobile device security application MBL-CYBER-APP installed, it will not download the data structure TUNNEL-M / T and will not be protected by the security policy. The TUNNEL-M / T policy and data structure may be updated, and updates to the security policy and data structure TUNNEL-M / T may be generated and distributed. Based on such updates, the mobile device security application MBL-CYBER-APP may tunnel packets for filtering that were previously allowed to proceed to their intended destination without being tunneled into the corporate network.

[0213] Figure 14 shows a flowchart of the operational concept for packet filtering on the mobile device and associated enterprise network, relative to the protection system for the mobile device and associated enterprise network depicted in Figure 12. The flowchart / operational concept assumes that all devices and hosts are already configured for operation. In particular, mobile device MBDV 2101 may have already downloaded, installed, and configured the mobile device security application MBL-CYBER-APP and established a tunnel with TGW 2163. Conversely, mobile device MBDV2102 does not have the mobile device security application MBL-CYBER-APP configured.

[0214] In step 2211, a user who owns and / or operates a mobile device MBDV, which is either MBDV2101 or MBDV2102, can check their email via an email application and click an embedded URL that links to a resource on the web server WSVR2151. The mobile device MBDV2101 or 2102 can resolve the hostname of the URL authority, or fully qualified domain name (FQDN), to the IP address of the web server WSVR2151, e.g., 12.34.56.78, by querying DNS (not shown in Figure 14). The mobile device MBDV can initiate a TCP connection with the web server WSVR 2151 on port 80 (HTTP) by creating a TCP SYN packet with destination port 80, encapsulating the TCP packet in an IP packet with the destination IP address field set to 12.34.56.78 and the corresponding MBDV's source IP address set.

[0215] In step 2212, before the packet is forwarded (to the wireless access network RNET2120), the MBDV2101's mobile device security application MBL-CYBER-APP can determine each packet element related to a threat indicator that is a member of the data structure TUNNEL-M / T. For example, the mobile device security application MBL-CYBER-APP extracts an IP address (e.g., 12.34.56.78) from the destination IP address field and tests whether 12.34.56.78 is a member of the data structure TUNNEL-M / T. Based on the determination that there are no packet filtering rules that match the security policy of the threat intelligence gateway TIG2170, the membership test returns FALSE. The mobile device security application MBL-CYBER-APP may determine that there is no threat risk associated with the tested packet data (e.g., the IP address 12.34.56.78 of the web server WSVR 2151). The mobile device security application MBL-CYBER-APP may determine that it does not need to tunnel packets that are being filtered. The mobile device security application MBL-CYBER-APP may not forward the packets to TGW 2163, threat intelligence gateway TIG2170, and / or security stack SSTK 2175.

[0216] In step 2213, based on the determination that there is no need to tunnel the filtered packets, the mobile device MBDV2101 can forward the packets directly to the web server WSVR2151 via the wireless access network RNET2120. Similarly, without performing tunneling decision / decision logic, MBDV2102 can forward packets directly to the web server WSVR2151 via the wireless access network RNET2120. Packet forwarding initiates a TCP handshake, and then establishes a TCP connection on port 80 of 12.34.56.78. The mobile device MBDV can issue an HTTP GET request for the URL. The web server WSVR2151 responds with the requested resource, the web session is terminated, and the TCP connection may be disconnected. During such a communication session, MBDV2101 can test the membership of the data structure TUNNEL-M / T for all inbound and outbound packets, using relevant packet field values ​​such as the IP address field, domain name field, and URI field. When each packet in a communication flow is associated with a secure destination, the mobile device security application MBL-CYBER-APP can determine that there is no need to tunnel the packets to be filtered because the test of data structure TUNNEL-M / T always returns FALSE, and therefore none of the packets constituting the session are tunneled.

[0217] In step 2211, a user who owns and / or operates the mobile device MBDV 2101 may be tricked into reading a spear-phishing email and clicking an embedded URL that links to a resource on the malware server MALSVR 2152, which may contain a web page disguised as a login page for a corporate web application on the corporate system server ESVR 2162. The mobile device MBDV 2101 can resolve the authoritative hostname, or fully qualified domain name (FQDN), of the URL to the IP address of the malware server MALSVR 2152, e.g., 87.65.43.21, by querying DNS (not shown in Figure 14). The mobile device MBDV 2101 can attempt to initiate a TCP connection with the malware server MALSVR 2152 on port 80 (HTTP) by creating a TCP SYN packet with a destination port of 80, and can encapsulate the TCP packet in an IP packet with the destination IP address field set to 87.65.43.21. In step 2212, before the packet is forwarded by device MBDV 2101 to the wireless access network RNET 2120, the mobile device security application MBL-CYBER-APP may extract packet elements (e.g., the IP address 87.65.43.21 from the destination IP address field) and test to determine whether any of the packet elements are members of the data structure TUNNEL-M / T.

[0218] The membership test may return a TRUE value or other indication that a matching packet filtering rule exists in the security policy associated with the data structure TUNNEL-M / T. Based on the TRUE value or other indication that a matching packet filtering rule exists in the security policy associated with the data structure TUNNEL-M / T, the mobile device security application MBL-CYBER-APP can determine that the packet should be filtered by the associated enterprise threat intelligence gateway TIG2170 and that there is some threat risk associated with the packet element (e.g., the IP address 87.65.43.21 of the malware server MALSVR2152). In step 2214, the mobile device MBDV2101 can send the packet to TGW2163 over the network tunnel based on the TRUE value or other indication that a matching packet filtering rule exists in the security policy associated with the data structure TUNNEL-M / T. In step 2215, the packet is received on the corporate network side of the tunnel and forwarded through the private corporate network ENET2160 toward the internet access link, where it is received by the threat intelligence gateway TIG2170 for packet filtering. In step 2216, the threat intelligence gateway TIG2170 applies a security policy to the packet and can determine a rule that matches the packet element (e.g., IP address 87.65.43.21). The network protection action / PTF associated with the determined rule can specify that the action(s) are at least one of block / drop, log, and / or capture.

[0219] Alternatively, before sending packets based on a match determination by testing the data structure TUNNEL-M / T, the system may test a secondary data structure to reduce the amount of network traffic. For example, the system may test a block rule data structure, which may represent each rule of a policy associated with a block action or PTF. Based on a TRUE value or other indication that a matching packet filtering rule exists in the security policy associated with the block rule data structure, the mobile device security application MBL-CYBER-APP may perform a block rule action without transmitting such packets through the tunnel to a packet filtering device on the corporate network in order to prevent packets associated with the block rule from proceeding to their intended destination. The block rule data structure may be a rule tree, a flat hash map, or a Bloom filter, and the block rule data structure may be a smaller data structure than the data structure TUNNEL-M / T associated with all rules of the policy being enforced.

[0220] In step 2217, the threat intelligence gateway TIG2170 can send logs to the Security Operations Center (SOC) 2140 or other network devices. The logs can be analyzed by the Security Operations Center (SOC) 2140, for example, by cyber analysts using a SIEM application. Based on the determination of the threat risk associated with the malware server MALSVR 2152 by the CTIP(s) that provided the CTI for threat indicators (e.g., IP address 87.65.43.21), the system can take corrective action. For example, the threat analysis device can report the incident to the user of the mobile device MBDV 2101 or recommend that the user watch a cybersecurity training video explaining spear-phishing email attacks and how to avoid them. Similarly, the mobile device security application MBL-CYBER-APP can generate a log of threat events and send the logs to the Security Operations Center (SOC) 2140 or other network security applications(s) or devices(s). The mobile device security application MBL-CYBER-APP can selectively generate threat event logs based on the indication that a packet filtering rule exists that matches the security policy associated with the block rule probabilistic data structure.

[0221] Figure 15 shows another example network environment 2100 in which a rule tree and / or flat hash map can be used as a set membership test data structure to implement efficient cyber protection for remote networks and associated central networks that constitute an enterprise network. The elements shown in Figure 15 may also perform the operations described in relation to the elements shown in Figure 1 and / or Figure 12, and the elements shown in Figure 1 and / or Figure 12 may also perform the operations described in relation to Figure 15. In at least some configurations, the remote networks and associated central networks may be located in different geographical locations. Figure 15 depicts an enterprise network consisting of a single remote network RMT-NET2720 and a single associated central network CENT-NET2760, but it should be noted that the embodiments described herein can be easily extended to apply to multiple remote networks and one or more associated central networks. Figure 15 does not limit or restrict the disclosed invention in any way to the case of a single remote network and a single central network.

[0222] One or more remote networks RMT-NET2720 may be associated with a central network CENT-NET2760 operated by the enterprise. Hosts connected to the remote network RMT-NET2720, such as PC2721 and PC2722 (e.g., desktop personal computers, mobile devices (e.g., connected to a local Wi-Fi access network)), can directly access commonly addressed internet servers such as the web server WSVR2751 and the malware server MALSVR2752 via the internet 2730. When accessing the internet 2730 directly, hosts connected to the remote network RMT-NET2720, such as PC2721 and PC2722, may not be protected by the enterprise security stack SSTK2775 and threat intelligence gateway TIG2770 associated with the central network CENT-NET2760, and therefore may not be protected from internet threats (e.g., threats associated with the malware server MALSVR2752). For protection, an enterprise may install an inline remote network tunnel gateway RMT-NET-TGW2725 at or near the network boundary / interface between the remote network RMT-NET2720 and the Internet 2730. The remote network tunnel gateway RMT-NET-TGW2725 can implement the packet inspection, packet testing, and packet tunneling functions described herein. The remote network tunnel gateway RMT-NET-TGW2725 may be configured to tunnel selected packets, manage TUNNEL-M / T data structures, and perform other functions described elsewhere in this disclosure. The remote network tunnel gateway RMT-NET-TGW2725 may be integrated with or otherwise located in the same place as other network boundary devices such as network firewalls and VPN tunnels.

[0223] The central network CENT-NET2760 can provide internet access to internal hosts such as PC2761 (e.g., desktop personal computers, mobile devices (e.g., connecting to a local Wi-Fi access network)). The central network CENT-NET2760 can be configured so that traffic between these internal hosts and threat internet hosts such as the malware server MALSVR2752 must pass through the threat intelligence gateway TIG2770. The threat intelligence gateway TIG2770 may be running policies originating from CTI. Such traffic may also pass through the security stack SSTK2775, which may include traditional network firewalls and at least one other enterprise network security device such as a web proxy, SSL / TLS proxy, IDS, IPS, or packet capture device. The threat intelligence gateway TIG2770 and security stack SSTK2775 can be located at or near the internet access point of the central network CENT-NET2760. When applying policies to packets, the threat intelligence gateway TIG2770 may generate logs of packets that match the policy rules. These logs may be sent over the internet to the security operations center SOC2740 for attack analysis by cyber analysts using SIEM applications or packet analyzer applications, for example.

[0224] The enterprise system server ESVR2762 may be, for example, a private web application server hosted by an enterprise on a central network for use by enterprise users, including enterprise users who own / operate host PCs 2721 and 2722 connected to the remote network RMT-NET2720. Host PCs 2721 and 2722 can access the enterprise system server ESVR2762 using a web browser that includes an HTTPS client for accessing the enterprise system server ESVR2762 on port 443 (the port for HTTPS). Once a host connects to the enterprise system server ESVR2762 (via a web browser), the web application can present a login form. Enterprise users can enter their enterprise credentials to securely access the web application. Network security administrators may open a port (e.g., port 443) on the network firewall of the security stack SSTK2775 to allow unsolicited inbound HTTPS connections, such as those originating from host PCs 2721 and 2722, to initiate a session with the enterprise system server ESVR2762.

[0225] The central network tunnel gateway TGW2763, connected to the central network CENT-NET2760, can terminate and centralize tunnels that may be established with associated remote networks (e.g., RMT-NET2720). The associated remote networks may have corresponding tunnel gateways (e.g., RMT-NET-GTW2725) to terminate tunnels connected to the tunnel gateway TGW2763 (via the Internet 2730). Similar to the enterprise system server ESVR2762, network administrators can open one or more ports on the network firewall of the security stack SSTK2775 to allow unauthorized inbound tunnel traffic (e.g., originating from RMT-NET-GTW2725) to access TGW2763. For example, if the tunneling protocol is in IPsec tunnel mode, the administrator opens well-known IPsec ports 500, 50, and 51. The tunnel gateway TGW2763 can decapsulate and / or decrypt incoming packets as they exit the tunnel. The tunnel gateway TGW2163 can forward decapsulated and / or decrypted packets to the (private enterprise) central network CENT-NET2760. Since these packets have a public internet address in the destination IP address field of their IP header, routers and / or switches on the central network CENT-NET2760 can forward these packets toward an internet access point, and therefore toward the threat intelligence gateway TIG2770, which applies CTI-derived policies to the packets.

[0226] The security policy creation and management server SPMS2741 can collect CTI from one or more CTIPs, including, for example, CTIP2742 and 2743. The SPMS2741 can also create at least one security policy based on the CTI and publish the security policy to subscribers. Subscribers include multiple network devices such as host computers and threat intelligence gateways like TIG2770. The SPMS2741 may create a data structure TUNNEL-M / T (as described above) for each security policy and publish the TUNNEL-M / T data structure to each subscriber associated with each security policy, such as the remote network tunnel gateway RMT-NET-TGW2725. Alternatively, the threat intelligence gateway TIG2770 may create a data structure TUNNEL-M / T for each associated remote network tunnel gateway, such as RMT-NET-TGW2725, that is associated with the threat intelligence gateway TIG2770 for cyber protection, and may expose the current data structure TUNNEL-M / T to each associated remote network tunnel gateway.

[0227] In one configuration, the security policy creation and management server SPMS2741 is located outside the central network CENT-NET2760 and the remote network RMT-NET2720, and can communicate with them via the internet 2730. In another configuration, the security policy creation and management server SPMS2741 is connected to either the central network CENT-NET2760 or the remote network RMT-NET2720.

[0228] One or more elements shown in Figure 15 may be identical, similar, substantially similar, or share one or more hardware, software, and / or firmware components with one or more elements shown in Figure 12. For example, the central network CENT-NET2760, which has the associated threat intelligence gateway TIG2770 and security stack SSTK2775, may be the same, similar, or substantially similar to the enterprise network ENET2160, which has the associated threat intelligence gateway 2170 and security stack SSTK2175 in Figure 12. Thus, the threat intelligence gateway 2170 and security stack 2175 can enforce CTI-derived policies not only for tunnel traffic from mobile devices (e.g., MBDV 2101) but also for tunnel traffic from remote network tunnel gateways (e.g., RMT-NET-TGW 2725). Similarly, the internal host PC 2161, enterprise system server 2162, tunnel gateway 2163, security operations center SOC 2140, security policy creation and management server SPMS 2141, CTI providers CTIP 2142 and 2143, malware server MALSVR 2152, and web server WSVR 2151 described in Figure 12 may be identical, similar, or substantially similar to the internal host PC 2761, enterprise system server 2762, tunnel gateway 2763, security operations center SOC 2740, security policy creation and management server SPMS 2741, CTI providers CTIP 2742 and 2743, malware server MALSVR 2752, and web server WSVR 2751. The mobile device 2101 (and / or mobile device security application MBL-CYBER-APP) may be identical, similar, substantially similar, or share hardware, software, and / or firmware components with the remote network tunnel gateway RMT-NET-TGW 2725.

[0229] Figure 16 is a flowchart illustrating the operational concept of security policy creation and management servers for configuring remote networks, associated tunnel gateways, and associated central network elements in a corporate network protection system consisting of a remote network and a central network, as shown in Figure 15.

[0230] Figure 16 shows one process in which CTI can be collected and used to generate one or more rule tree data structures and / or one or more flat hash map data structures configured for set membership testing. Optionally, CTI can also be used to generate one or more probabilistic data structures (e.g., Bloom filter, cuckoo filter, block Bloom filter, XOR filter, etc.) for set membership testing. The data structures generated for set membership testing can be distributed to a remote device (e.g., RMT-NET-TGW 2725) (e.g., as TUNNEL-M / T data structures). In this way, the remote device (e.g., RMT-NET-TGW 2725) can use the received data structure(s) to test packets and decide whether those packets should be sent to a central device / network for further analysis. For example, if one or more attributes of a packet are present in / represented by the data structure, the packet is forwarded to the central device for further analysis. In contrast, if one or more attributes are not present in the data structure or are not represented by the data structure, the packet is forwarded without further analysis.

[0231] In step 2801, the security policy creation and management server SPMS2741 can receive (e.g., download) CTIs published by threat intelligence providers such as CTIP2742 and CTIP2743. This step may be the same as or similar to step 2201 in Figure 13.

[0232] In step 2802, the security policy creation and management server SPMS2741 can process CTI to aggregate CTI indicators and generate security policies. The security policy creation and management server SPMS2741 can create at least one packet filtering rule, each rule including a matching criterion corresponding to a threat indicator and a corresponding action / PTF to be applied to the packet if a match is determined. The corresponding action / PTF may be determined based on several factors, including requirements provided by the operator / administrator of the threat intelligence gateway TIG2770. The security policy creation and management server SPMS2741 can collect each generated rule to create a security policy and distribute the security policy to multiple security devices, including the threat intelligence gateway TIG2770. This step may be the same as or similar to step 2202 in Figure 13.

[0233] In step 2803, the security policy creation and management server SPMS741 can generate a data structure TUNNEL-M / T related to the security policy by collecting threat indicators (e.g., by collecting one or more of the following: IP addresses, IP address ranges, domain names, URIs, certificate IDs, etc.). This structure includes one or more rule tree data structures configured for set membership testing, and / or one or more flat hash map data structures configured for set membership testing, and optionally one or more probabilistic data structures. In this way, TUNNEL-M / T can characterize each packet filtering rule of the security policy. The security policy creation and management server SPMS2741 can insert indicators (or values ​​based on indicators) into the data structure TUNNEL-M / T, which can then be tested to determine whether the rules in the security policy match the packet element being tested. Thus, TUNNEL-M / T can be used to efficiently test whether a particular packet should be tunneled to the central network (e.g., for further processing such as further rule-based testing). This step is the same as or similar to step 2203 in Figure 13.

[0234] The security policy creation and management server SPMS2741 may expose at least one of the security policy and the data structure TUNNEL-M / T to multiple network security devices. The security policy creation and management server SPMS2741 may send at least one of the security policy and the data structure TUNNEL-M / T to multiple network security devices. Multiple network security devices may include the threat intelligence gateway TIG2770 and / or one or more remote network tunnel gateways (e.g., remote network tunnel gateway RMT-NET-TGW2725). For example, as shown in step 2804, the security policy creation and management server SPMS2741 may send a security policy to the threat intelligence gateway TIG2770 associated with the central network CENT-NET2760. The threat intelligence gateway TIG2770 can download the security policy and configure packet filtering logic with the security policy. This step is the same as or similar to step 2204 in Figure 13.

[0235] For example, as shown in step 2805, the security policy creation and management server SPMS2741 may send the data structure TUNNEL-M / T to the remote network tunnel gateway RMT-NET-TGW2725. The remote network tunnel gateway RMT-NET-TGW2725 may download the data structure TUNNEL-M / T and use the data structure TUNNEL-M / T to configure tunneling logic. Based on the security policy and the data structure TUNNEL-M / T, the remote network tunnel gateway RMT-NET-TGW2725 may or may not tunnel packets to the central network CENT-NET2760 for filtering at the threat intelligence gateway TIG2770 (as further explained, for example, with reference to Figure 17). This step is the same as or similar to step 2205 in Figure 13.

[0236] The security policy and data structure TUNNEL-M / T may also be updated (for example, by the security policy creation and management server SPMS2741), and updates to the security policy and data structure TUNNEL-M / T may be generated and distributed (for example, periodically based on updates to CTI provided by CTIP). Updates to the security policy and data structure TUNNEL-M / T may change the operation of network security devices (e.g., the threat intelligence gateway TIG2770 and / or one or more remote network tunnel gateways). For example, based on such an update, the remote network tunnel gateway RMT-NET-TGW2725 may tunnel packets for filtering to their intended destination without tunneling them to the central network CENT-NET2760, which were previously allowed to proceed (e.g., based on a previous version of the data structure TUNNEL-M / T).

[0237] Additionally and / or alternatively (though not illustrated in Figure 16), after receiving a security policy from the security policy creation and management server SPMS2741, the threat intelligence gateway TIG2770 may generate a data structure TUNNEL-M / T associated with the security policy (including one or more rule tree data structures configured for set membership testing, and / or one or more flat hash map data structures configured for set membership testing, and optionally one or more probabilistic data structures) by collecting all threat indicators, including IP addresses, domain names, URIs, certificate IDs, etc., to characterize each packet filtering rule of the security policy. The threat intelligence gateway TIG2770 may insert indicators (or values ​​based on indicators) into the data structure TUNNEL-M / T and test them to determine whether the rules of the security policy match the packet element being tested. Next, the threat intelligence gateway TIG2770 can send TUNNEL-M / T to the remote network tunnel gateway (e.g., RMT-NET-TGW2725) of the associated remote network (e.g., RMT-NET2725).

[0238] Generally, remote fixed networks (especially remote network tunnel gateways) may not have resource limitations associated with mobile devices (e.g., related to power, memory, processing power, etc.). Thus, a remote network tunnel gateway (e.g., RMT-NET-TGW2725) can perform one or more of the operations described above in relation to the security policy creation and management server SPMS2741 (e.g., generation of the data structure TUNNEL M / T). For example, the security policy creation and management server SPMS2741 can generate security policies based on multiple CTI indicators and distribute the security policies to the remote network tunnel gateway RMT-NET-TGW2725. The remote network tunnel gateway RMT-NET-TGW2725 may generate the data structure TUNNEL-M / T related to the security policies. Furthermore, remote fixed networks and associated remote network tunnel gateways may have looser memory constraints compared to mobile devices, which could lead to increased sizes for, for example, a given rule tree data structure configured for set membership testing, a given flat hash map data structure configured for set membership testing, and / or a given probabilistic data structure (e.g., a given Bloom filter).

[0239] Figure 17 is a flowchart illustrating operational concepts for packet filtering and efficient backhaul for network protection in a remote network, associated tunnel gateway, and associated central network. For a remote network (e.g., the remote network RMT-NET2720 in Figure 15 with multiple connected hosts such as PC2721 and PC2722), a single tunnel can be used to selectively tunnel or backhaul only packet traffic known to be threat-related to the central network CENT-NET2760. Otherwise, packets could bypass the tunnel and instead be forwarded directly to their destination.

[0240] As an introduction, Figure 17 illustrates a process in which a remote device (e.g., RMT-NET-TGW2725) can test packets (e.g., packets related to internet communication) using a received TUNNEL-M / T data structure to determine whether those packets should be sent to a central device / network for further analysis. For example, if one or more attributes of a packet are present in / represented by the data structure, the packet may be forwarded to the central device for further analysis. In contrast, if one or more attributes are not present in / represented by the data structure, the packet may be forwarded (e.g., to a web server) without further analysis. As explained with respect to Figure 16, the TUNNEL-M / T data structure may be generated by the security policy creation and management server SPMS2741.

[0241] The received TUNNEL-M / T data structure may correspond to a security policy consisting of multiple packet filtering rules. Each packet filtering rule in the security policy may consist of one or more packet matching criteria and a corresponding rule action (e.g., a network protection action, or PTF) to be taken for packets that match the packet matching criteria. At least the first packet filtering rule of the security policy may be automatically generated based on a CTI report provided by an independent CTI provider. At least the second packet filtering rule of the security policy may be automatically generated based on a second CTI report provided by a different independent CTI provider.

[0242] Referring to Figure 15, one or more CTIPs (e.g., CTIP2742 and / or CTIP2743) may identify the malware server MALSVR2752 as a threat but not the web server WSVR2751. Therefore, for example, the IP address of the malware server MALSVR2752 (or other threat indicators) can be included in the CTI supplied to the security policy creation and management server SPMS2741. A process similar to that shown and described with reference to Figure 16 may result in (a) the threat intelligence gateway TIG2770 being configured with a security policy that includes packet filtering rules, where the packet filtering rules specify packet matching criteria corresponding to the IP address of the malware server MALSVR2752 (or any other threat indicators), and (b) the remote network tunnel gateway RMT-NET-TGW2725 being configured with the data structure TUNNEL-M / T. The data structure TUNNEL M / T may be generated based on a set of packet matching criteria, which may include the IP address (or other threat indicator) of the malware server MALSVR2752 as an element of that set. The set of packet matching criteria may not include the IP address (or threat indicator) of the web server WSVR2751 as an element of that set.

[0243] In step 2811a, host PC 2721 initiates communication with web server WSVR2751. Communication can be initiated, for example, by host PC 2721 sending a TCP SYN contained in packet P1 with a destination IP address corresponding to web server WSVR2751. A user associated with host PC 2721 can initiate communication with web server WSVR2751, for example, through a web browser or application associated with web server WSVR2751 installed on host PC 2721. This step is the same as or similar to step 2211 in Figure 14.

[0244] For example, a user operating host PC2721 can access resources on web server WSVR2751 using a URL. Host PC2721 can resolve the URL's authority hostname (FQDN) to the IP address of web server WSVR2751 (e.g., 12.34.56.78) by querying DNS (not shown in Figure 15). Host PC2721 may attempt to initiate a TCP connection with web server WSVR2751 on port 80 (HTTP) by creating a TCP SYN packet for destination port 80, encapsulating the TCP SYN packet in an IP packet. The IP packet may contain a destination IP address field (e.g., set to 12.34.56.78) and a source IP address (e.g., of host PC2721).

[0245] In step 2811b, although it does not need to occur at a specific time with respect to step 2811a, host PC 722 initiates communication with the malware server MALSVR2752. Communication can be initiated by host PC 2722 with the malware server MALSVR2752 by sending a TCP SYN contained in packet P2 with a destination IP address corresponding to the malware server MALSVR2752. For example, a malicious application installed on host PC 2722 may secretly initiate communication with the malware server 2752, resulting in the sending of a TCP SYN. ​​In another example, host PC 2722 may initiate communication with the malware server MALSVR2752 based on a user clicking a link in a malicious email (e.g., a spear-phishing email) or other form of electronic communication.

[0246] For example, a user operating host PC2722 might be tricked into reading a spear-phishing email and clicking an embedded URL linking to a resource on malware server MALSVR2752. The resource could include a webpage disguised as a login page for a corporate web application on corporate system server ESVR2762. Host PC2722 can resolve the authoritative hostname or FQDN of the URL to the IP address of malware server MALSVR2752 (e.g., 87.65.43.21) by querying DNS (not shown in Figure 15). Host PC2722 could attempt to initiate a TCP connection with malware server MALSVR2752 on port 80 (HTTP) by creating a TCP SYN packet with destination port 80, encapsulating the TCP packet in an IP packet. The IP packet contains a destination IP address field set to 87.65.43.21.

[0247] In step 2812, the remote network tunnel gateway RMT-NET-TGW2725 receives packets P1 and / or P2. The packets may originate from the remote network RMT-NET2720. In steps 2813 and 2814, the remote network tunnel gateway RMT-NET-TGW2725 can test whether one or more packet elements of the received packets P1 and / or P2 (e.g., packet matching criteria such as an IP address, domain name, URI, certificate ID, or any other packet element) are elements of the data structure TUNNEL M / T (e.g., generated in step 2803 and distributed in step 2805) (e.g., whether they are members of it or represented therein). The data structure TUNNEL M / T may be associated with security policies enforced by the threat intelligence gateway TIG2770. These steps may be the same as or similar to step 2212 in Figure 14.

[0248] In step 2813, the remote network tunnel gateway RMT-NET-TGW2725 may determine the respective packet matching criteria / packet elements for each packet P1 and / or P2. The remote network tunnel gateway RMT-NET-TGW2725 may test the rule tree data structure and / or flat hash map data structure (e.g., data structure TUNNEL-B / F) for the determined packet matching criteria. The data structure TUNNEL-M / T can represent each of the packet filtering rules of the security policy configured on the threat intelligence gateway TIG2770 (e.g., as described with respect to Figure 16). Based on the determination that the packet matching criteria / packet elements for a packet (e.g., packet P1) are not represented in the data structure TUNNEL-M / T, the remote network tunnel gateway RMT-NET-TGW2725 may forward packet P1 to its intended destination (e.g., web server WSVR2751). Based on the determination that the packet matching criteria / packet elements of a packet (e.g., packet P2) are represented in the data structure TUNNEL-M / T, the remote network tunnel gateway RMT-NET-TGW2725 can tunnel packet P2 to the central network CENT-NET2760. The central network CENT-NET2760 can filter packets based on the packet filtering rules of its security policy. For example, the threat intelligence gateway TIG2770 on the central network CENT-NET2760 can filter packets based on the packet filtering rules of its security policy. Further details related to packet tunneling and filtering will be discussed later.

[0249] The remote network tunnel gateway RMT-NET-TGW2725 can extract packet elements from packet P1. For example, the remote network tunnel gateway RMT-NET-TGW2725 can extract the destination IP address, which is the IP address of the web server WSVR2751, from packet P1. The remote network tunnel gateway RMT-NET-TGW2725 can test whether a packet element (e.g., the destination IP address) is an element in the data structure TUNNEL-M / T or a member of the data structure TUNNEL-M / T. For example, if a packet element is neither an element nor a member of the data structure TUNNEL-M / T, the membership test may return FALSE. A membership test returning FALSE means that packet P1 does not fall under any packet filtering rules of the policy enforced by the threat intelligence gateway TIG2770. Therefore, in step 9814, the remote network tunnel gateway RMT-NET-TGW2725 can forward packet P1 to the web server WSVR2751 via the internet 2730 if the membership test returns FALSE.

[0250] For example, the remote network tunnel gateway RMT-NET-TGW2725 can extract an IP address (e.g., 12.34.56.78) from the destination IP address field of packet P1 and test whether 12.34.56.78 is a member of the data structure TUNNEL-B / F. Since the IP address 12.34.56.78 is not a member of the data structure TUNNEL-B / F, the membership test may return FALSE. Based on the membership test returning FALSE, the remote network tunnel gateway RMT-NET-TGW2725 may determine that there is no matching packet filtering rule for packet P1 in the security policy enforced by the threat intelligence gateway TIG2770. The remote network tunnel gateway RMT-NET-TGW2725 determines, for example, that there is no threat risk associated with the tested packet data (e.g., the IP address 12.34.56.78 of web server WSVR2751) based on the membership test returning FALSE. The remote network tunnel gateway RMT-NET-TGW2725 determines, for example, that there is no need to tunnel packet P1 (which is filtered in the central network CENT-NET2760) based on the membership test returning FALSE. As a result, the remote network tunnel gateway RMT-NET-TGW2725 may not forward packet P1 to TGW2763, threat intelligence gateway TIG2770, and / or security stack SSTK2775.

[0251] In step 2815, the remote network tunnel gateway RMT-NET-TGW2725 can extract packet elements from packet P2. For example, the remote network tunnel gateway RMT-NET-TGW2725 can extract the destination IP address, which is the IP address of the malware server MALSVR2752, from packet P2. The remote network tunnel gateway RMT-NET-TGW2725 can test whether the packet element (e.g., the destination IP address) is an element in the data structure TUNNEL-M / T or a member of the data structure TUNNEL-M / T. The membership test may return TRUE if, for example, the packet element is an element or member of the data structure TUNNEL-M / T. The membership test returning TRUE means that packet P2 corresponds to a packet filtering rule of the policy enforced by the threat intelligence gateway TIG2770. Therefore, in step 2816, the remote network tunnel gateway RMT-NET-TGW2725 tunnels packet P2 to tunnel gateway TGW2763, and tunnel gateway TGW2763 terminates at the central network CENT-NET2760.

[0252] For example, the remote network tunnel gateway RMT-NET-TGW2725 can extract an IP address (e.g., 87.65.43.21) from the destination IP address field of packet P2 and test whether 87.65.43.21 is a member of the data structure TUNNEL-M / T. The membership test may return a TRUE value based on the IP address 87.65.43.21 being a member of the data structure TUNNEL-M / T. Based on the membership test returning TRUE, the remote network tunnel gateway RMT-NET-TGW2725 may determine that there is at least one matching packet filtering rule for packet P2 in the security policy implemented by the threat intelligence gateway TIG2770. Based on the membership test returning TRUE, the remote network tunnel gateway RMT-NET-TGW2725 may determine that there is some threat risk associated with the tested packet data (e.g., the IP address 87.65.43.21 of the web server MALSVR2752). The remote network tunnel gateway RMT-NET-TGW2725 determines, for example, based on a membership test that returns TRUE, that packet P2 needs to be tunneled to the central network CENT-NET2760 for further processing (e.g., filtering). As a result, the remote network tunnel gateway RMT-NET-TGW2725 forwards packet P2 to TGW2763, threat intelligence gateway TIG2770, and / or security stack SSTK2775.

[0253] Steps 2813 and 2815 describe the use of the packet's destination IP address to perform a membership test, but any other element (or more elements) of the packet that may correspond to an element inserted into TUNNEL-M / T may be used.

[0254] Additionally and / or alternatively, before sending any packet based on a match determination from testing the data structure TUNNEL-M / T, the remote network tunnel gateway RMT-NET-TGW2725 may test secondary data structures to reduce the amount of network traffic. For example, the remote network tunnel gateway RMT-NET-TGW2725 may test a block rule data structure, which may represent each rule of the security policy associated with a block action or PTF. Based on a TRUE value or other indication that there is a matching packet filtering rule in the security policy associated with the block rule data structure, the remote network tunnel gateway RMT-NET-TGW2725 may perform a block rule action to prevent packets associated with the block rule from proceeding to their intended destination without transmitting such packets through the tunnel to the tunnel gateway TGW2763. The block rule data structure can be a rule tree data structure configured for set membership testing, a flat hash map data structure configured for set membership testing, a Bloom filter, and / or any other probabilistic data structure.

[0255] For example, referring to the above example where it is determined that a packet element of packet P2 is an element within the data structure TUNNEL-M / T or a member of the data structure TUNNEL-M / T, the remote network tunnel gateway RMT-NET-TGW2725 can extract a second packet element of packet P2. For example, the second packet element could be the URI associated with packet P2. The remote network tunnel gateway RMT-NET-TGW2725 may extract the URI of packet P2 and test whether the URI is a member of the block rule data structure BLOCK-M / T. The membership test may return a TRUE value based on the URI being a member of the data structure BLOCK-M / T. Based on the membership test returning TRUE, for example, the remote network tunnel gateway RMT-NET-TGW2725 may determine that, in the security policy enforced by the threat intelligence gateway TIG2770, there is at least one matching packet filtering rule for packet P2 to block the packet. The remote network tunnel gateway RMT-NET-TGW2725 determines, for example, that packet P2 has a high threat risk associated with the tested packet data, based on the membership test returning TRUE. Based on the membership test returning TRUE, the remote network tunnel gateway RMT-NET-TGW2725 can block packet P2 from leaving the remote network RMT-NET2720.

[0256] Performing packet membership tests (e.g., testing whether packet elements associated with a packet are members of the data structure TUNNEL-M / T) in the remote network tunnel gateway RMT-NET-TGW2725 reduces processing requirements on hosts (e.g., PC2721 and PC2722). Since hosts do not need to perform membership tests, resource availability on the hosts is improved. Furthermore, higher resource availability in the remote network tunnel gateway RMT-NET-TGW2725 may enable faster packet processing.

[0257] In step 2817, packet P2 can exit the tunnel and be forwarded through the central network CENT-NET2760 toward the malware server MALSVR2752 (corresponding to the destination IP address of P2). Along the path toward the malware server MALSVR2752, packet P2 may be received by the threat intelligence gateway TIG2770.

[0258] In step 2818, the threat intelligence gateway TIG2770 can apply a security policy to packet P2. Applying a security policy may involve applying a packet filtering rule with matching criteria corresponding to the packet elements (e.g., destination IP address) of packet P2 (e.g., the IP address of MALSVR2752). The processing of the matching rule (e.g., network protection action, PTF, or rule action) is to block the packet, and the directives are to log and capture the packet. Thus, by blocking, logging, and / or capturing packet P2, PC2722 and the corporate network can be protected from malware or malicious actions that may result from connecting to the malware server MALSVR2752.

[0259] In step 2819, the packet P2 log is sent to the Security Operations Center (SOC) 2740, where one or more cyber analysis and threat awareness applications process the log. This processing may lead to corrective actions, such as cleaning PC 2722 for malware, or other corrective and / or protective measures.

[0260] Set membership tests can also be used to prevent attacks related to the Internet's Domain Name System (DNS). DNS requests can be used as an attack vector for certain types of attacks that, if successful, could damage and / or cause loss to network assets. For example, an attack might include a DNS query request message containing a domain name that is not registered in DNS (e.g., registered and / or recorded). Such requests can disrupt the performance of DNS servers, especially in large numbers. Set membership tests can be used to check whether a domain name included in an incoming request is a registered (e.g., registered and / or recorded) domain name. If a domain name included in an incoming request is not a member of a set of registered (e.g., registered and / or recorded) domain names, appropriate action may be taken (e.g., dropping the request to prevent an attack). Flat hash map data structures and rule tree data structures can be used for such set membership tests, as described above.

[0261] Figure 18 shows another example network environment 3100 in which a rule tree and / or flat hash map can be used as a set membership test data structure. The elements shown in Figure 18 may also perform the operations described in relation to the elements shown in Figures 1, 12 and / or 15, and the elements shown in Figures 1, 12 and / or 15 may also perform the operations described in relation to Figure 18. Network environment 3100 may include a DNS attack prevention system that operates to protect the Domain Name System (DNS) of Internet networks, such as the public internet and private networks that may be interconnected by the public internet, from attacks that may use false DNS requests as an attack vector. Network 3110 may be the public internet interconnecting networks 3112, 3114, 3116, 3118 and 3120.

[0262] Network 3112 may be a private enterprise network with multiple hosts. These hosts may include at least one Internet of Things (IoT) device, IOT3140, such as a security camera or appliance, which can be remotely managed and operated over the Internet, and a Personally Identifiable Information (PII) host, PII3141. Host IOT3140 and other IoT devices (not shown) connected to Network 3112 may be infected with malware that generates a DNS attack by issuing numerous spoofed DNS requests with domain names designed to attack specific authoritative name servers (such as host NS3144 connected to Network 3116, described below). Host PII3141 is a representative of a personal computer or server that manages personally identifiable information (PII), sensitive information such as credit card information. Host PII3141 may be infected with malware that collects credit card information, and the infected host may use DNS tunneling to leak the information to a collection server CSVR3148 connected to Network 3170 (described below). Both attacks may issue requests using domain names that are not registered (e.g., registered and / or recorded) in DNS.

[0263] Network 3114 and its associated hosts IOT3142 and PII3143 are configured and can be compromised in the same way as network 3112 and its hosts IOT3140 and PII3141. However, network 3114 may also consist of a DNS gatekeeper, DNS-G / K3150, located at or near the boundary between network 3114 and the internet 3110. Network 3114 can be configured so that DNS-G / K3150 filters all DNS requests that cross the boundary of network 3114 and controls which DNS requests can access DNS.

[0264] Network 3116 may be operated by the DNS infrastructure provider organization DIPO-X. Authoritative name servers NS3144 and NS3145 are the DNS authoritative name servers for the domains example-3144.com and example-3145.com, and DIPO-X may operate them on behalf of the associated registrar organization. Both name servers NS3144 and NS3145 are configured to provide authorized requesters with a zone file containing a list of all registered domain names managed by those servers. Network 3116 may have a DNS gatekeeper DNS-G / K3180 at or near the boundary between Network 3116 and Internet 3110. Network 3116 may be configured so that the DNS gatekeeper DNS-G / K3180 filters all DNS requests crossing the boundary and controls which DNS requests can access the DNS name servers for example-3144.com (authoritative name server NS3144) and example-3145.com (authoritative name server NS3145).

[0265] Network 3118 may be operated by a DNS infrastructure provider organization (e.g., DIPO-Y). Host authoritative name servers NS3146 and NS3147 may be the DNS authoritative name servers for the domains example-3146.com and example-3147.com, and DIPO-Y may operate them on behalf of the associated registrar company. Both name servers NS3146 and NS3147 are configured to provide authorized requesters with a zone file containing a list of all registered domain names managed by those servers. Network 3118 does not need to have a DNS gatekeeper to control DNS requests that can access the DNS name servers for example-3146.com (authoritative name server NS3146) and example-3147.com (authoritative name server NS3147).

[0266] Network 3170 may be operated by at least one malicious actor. Network 3170 may include a data collection server host CSVR3148 that collects data leaked through fake DNS requests, and may include a botnet command control host system CNC3149 that executes a botnet command control process that instructs infected devices to launch DDoS attacks on DNS. The malicious data collection process running on data collection server CSV3148 may receive fake DNS requests on port 53, a well-known port for DNS. The public IP address of data collection server CSVR3148 is 21.43.65.87 in the example below. A network firewall (not shown) associated with network 3170 can be configured to allow inbound packets with destination IP address 21.43.65.87 and destination port 53 to data collection server CSVR3148. Network 3170 may not have a DNS gatekeeper that controls access to DNS.

[0267] The DNS filter manager (DFM) 3160, which functions as a system administration server, can run as a host connected to a private network (not shown) that serves as an interface to the Internet 3110. The DFM 3160 can continuously collect a list of domain names from all DNS authoritative name servers, such as DNS authoritative name servers NS3144-3147, and / or from associated domain list aggregators (not shown in Figure 18), and / or from other sources of registered (e.g., registered and / or recorded) domain names. The DFM 3160 can encode and store all registered (e.g., registered and / or recorded) domain names in at least one data structure DNS-REG. The at least one data structure DNS-REG may consist of one or more flat hash maps configured for set membership testing (e.g., for membership testing in a set of registered (e.g., registered and / or recorded) domain names), and / or one or more rule tree data structures configured for set membership testing. Optionally, the at least one data structure DNS-REG may also include a Bloom filter or other probabilistic set data structure for set membership testing. The DFM3160 may be distributed to DNS gatekeepers that subscribe to the DNS-REG data structure (e.g., DNS-G / K3150 and DNS-G / K3180).

[0268] The DFM3160 may also include the ability to manage information supplied or uploaded by subscribed DNS gatekeepers (e.g., DNS-G / Ks3150 and 3180). This information may include unregistered (e.g., unregistered and / or unrecorded) domain names and associated metadata (e.g., attack type estimates, timestamps, rate estimates, origin hosts, DNS-G / K IDs, etc.) discovered by subscribed DNS gatekeepers DNS-G / Ks3150 and 3180, which may be used as threat intelligence to determine or detect other attacks. This information may also include statistics on DNS requests and responses observed by DNS-G / Ks3150 and 3180, as well as element management information for DNS-G / Ks3150 and 3180. The DFM3160 may also include the ability to provide information collected from DNS-G / Ks3150 and 3180 to subscribed applications (not shown in Figure 18), such as threat intelligence providers, cyber analytics applications, network element administrators, DNS monitoring applications, DNS protection applications, and internet protection applications.

[0269] The DNS-G / K3150 and DNS-G / K3180 shown in Figure 18 (and Figure 19 below) may or may not be all or part of a network device such as TIG. In general, the DNS-G / K150 and DNS-G / K180 can run on any computing device, including but not limited to the computing devices described above or below.

[0270] Figure 19 is a typical system configuration diagram of DNS-G / K3150 and / or DNS-G / K3180 supporting aspects of this disclosure. Components of DNS-G / K3150 may include a processor CPU 3520 that executes the logic for configuring and operating DNS-G / K3150, network interfaces 3512 and 3514 for connecting to networks 3114 and 3110, respectively, a management interface MGMT I / F 3540 that connects to network 3110, a main memory module MAIN-MEMORY 3530 which may contain the data structure DNS-REG 3531, and a memory store MEMORY-STORE 3532 for persistent storage. The components are connected to a bus BUS 3510 used for transferring information between components of DNS-G / K3150. DNS-REG 3531 is provided to DNS-G / K3150 from DFM 3160 via the management interface 3540 and network 3110. The system components of DNS-G / K3150 can be any combination of processes and applications running on the same host (e.g., co-located), processes running on different hosts, processes running on virtual infrastructure such as a hypervisor, or other deployments of components and software. Similarly, the bus BUS3510 can be any combination of an integrated / embedded data bus on a printed circuit board (PCB), such as an L2 / L3 switch network, an L3 routing network, or L2 / L3 network links connecting logical components. The data bus can be any combination of wired, wireless, physical, logical, virtual, or software-defined. Network interfaces 3512 and 3514 can be L3 transparent. Network interfaces 3512 and 3514 may not have IP addresses assigned and may not participate in the L3 routing process. Similarly, network interfaces 3512 and 3514 can be L2 transparent. Network interfaces 3512 and 3514 may not have MAC addresses assigned and may not participate in the ARP process.The management interface MGMT I / F3540 may be assigned an IP address for communication with management devices such as the DFM3160.

[0271] Figure 20 is a flowchart of a typical operational concept for DNS-G / K3150. In step 3601, DNS-G / K3150 selects a data structure (e.g., a flat hash map) DNS-REG3531 provided by DFM3160, downloads DNS-REG3531, and stores it in main memory MAIN-MEMORY3530. DNS-REG3531 can contain keys for all domain names currently registered (e.g., registered and / or recorded) in DNS, which are available to or determined by DFM3160. DFM3160 can provide multiple versions of DNS-REG3531 for use by DNS gatekeepers, which are characterized by their corresponding memory requirements. Versions of DNS-REG3531 may also be characterized by their creation time, and the selection and download of DNS-REG3531 may be part of the DNS-G / K3150 update process. The version of DNS-REG3531 can be selected by DNS-G / K3150 based on the corresponding memory requirements for storing DNS-REG3531. In step 3601, DNS-G / K3150 uploads the information collected in steps 3603, 3607, and 3608 to DFM3160. Uploading may be part of the update process. Uploading may include data such as malicious, fake, or unregistered (e.g., unregistered and / or unrecorded) domain names and associated metadata (e.g., timestamps, origin host identifiers, estimated attack types, rate estimates, etc.) detected by DNS-G / K3150, which may be used, for example, as threat intelligence and in threat intelligence reports. The uploaded data may include statistics on DNS requests and responses observed by DNS-G / K3150, and domain names that DNS-G / K3150 has determined may be registerable in DNS (e.g., registered and / or recorded) but may not be included in the local instance of DNS-REG3531 (e.g., due to synchronization delays). Such domain names are determined by DNS-G / K3150 in step 3612 of Figure 21 (described later).Other information, such as DNS-G / K3150 element management information, can also be uploaded.

[0272] In step 3602, DNS-G / K3150 can receive a packet in transit from network 3114 that is intended to be forwarded to network 3110. DNS-G / K3150 can also process packets sent in the opposite direction, from network 3110 to network 3114, but for illustrative purposes only, only one direction is described. In step 3603, DNS-G / K3150 determines whether the packet contains a DNS query request. If the packet does not contain a DNS query request, in step 3604F, the packet is forwarded to network 3110, and DNS-G / K3150 returns to step 3602 to process the additional packet. If the packet contains a DNS query request, in step 3604T, the domain names that may be contained in the QNAME field are extracted.

[0273] In step 3605, DNS-G / K3150 tests whether the domain name is a member of the set contained in DNS-REG3531. DNS-REG3531 may contain keys for the set of all domain names currently registered (e.g., registered and / or recorded) in DNS, based on the last update according to DFM3160. If the membership test returns TRUE, or if it is determined that the name is registered (e.g., registered and / or recorded) in DNS, in step 3606T, the packet and associated DNS requests and metadata may be recorded for analysis and statistical purposes and stored in MEMORY-STORE3532. Furthermore, DNS-G / K3150 updates the percentage of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests relative to the percentage of all DNS requests, and / or the percentage change in the percentage of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests relative to the percentage of all DNS requests, forwards the packet to network 3110, and DNS-G / K3150 returns to step 3602 to process additional packets.

[0274] In step 3606F, if the membership test returns FALSE, and / or if the name is determined not to be registered in DNS (e.g., not registered and / or not recorded), and the DNS request is determined to be associated with a malicious, false, or unregistered (e.g., not registered and / or not recorded) domain name, the packet may be dropped or blocked. The packet and associated information and metadata (e.g., domain name, time, location, direction, origin, attack type estimate, rate estimate, etc.) may be recorded for statistical purposes, the packet and associated information and metadata may be logged and / or captured, and the packet and associated information and metadata may be stored in MEMORY-STORE3532. This data is sent to or uploaded to DFM3160, as in step 3601 above. The packet and associated information and metadata can be made available for use by cyber analytics applications, threat intelligence applications, and network protection applications such as applications for securing and protecting global DNS and the global internet (not shown in Figure 20). For example, during a high-rate DDoS attack using fake DNS requests as an attack vector, logging and / or capturing packets associated with the attack can consume an unacceptably large amount of computing resources. Therefore, statistical information that requires fewer resources is collected instead. Cyber ​​analytics applications can identify infected hosts that may be the source of the fake DNS requests. The packets and associated information and metadata are provided to a cyber threat intelligence database and then used by applications and systems that protect the network. For example, DNS infrastructure providers and internet service providers can use the data collectively to promote global DNS and global internet protection applications.

[0275] In step 3607, the occurrence rate of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests may be updated to incorporate the occurrence of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests. DNS-G / K3150 can determine or calculate the occurrence rate of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests, or the occurrence rate of all DNS requests. DNS-G / K3150 can determine or calculate the rate of change in the occurrence of spoofed DNS requests, or the rate of change in the occurrence rate of all DNS requests. The rate of DNS requests and / or the rate of change in the rate of DNS requests may be used in step 3608 to determine the type of attack associated with the DNS request(s) of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests, and may be used to determine the possible DNS responses (if any) that may be generated. In step 3608, if DNS-G / K3150 has a DNS query response to generate in response to a bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS query request, it may decide what type of DNS query response to generate and forward the generated DNS response to network 3114. When generating a DNS response, DNS-G / K3150 may spoof a real DNS responder and send a response that spoofs the source of the associated DNS request and an intermediate device that may have observed the DNS request. This spoofed response may appear to the requesting device as a response from an authenticated DNS name server.

[0276] Factors used to determine a DNS response may include the current estimated rate of malicious, false, or unregistered (e.g., unregistered and / or unrecorded) domain name requests versus legitimate DNS requests, the rate of change between malicious, false, or unregistered (e.g., unregistered and / or unrecorded) domain name requests versus legitimate DNS requests, the false positive rate P of a data structure (e.g., DNS-REG1531), the estimated lag if the domain name contains a country code in its suffix, an estimate of the type of attack for which a DNS request for a bad, false, or unregistered (e.g., unregistered and / or unrecorded) domain name is an attack vector, and / or syntactic characteristics of the domain name (e.g., information entropy value, correlation with human language words, label length, alphanumeric characteristics, etc.). For example, based on the current percentage of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests, and / or the rate of change in the percentage of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests, DNS-G / K3150 can determine that a large-scale DDoS attack is occurring. Based on the determination that an attack is occurring, DNS-G / K3150 can decide not to generate DNS responses to send back to the source of the DNS requests. Generating DNS responses (e.g., responses with response code NXDOMAIN) to bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests during such a DDoS attack can exacerbate the attack by consuming processing resources and filling the DNS cache of intermediate DNS proxies with NXDOMAIN responses to bad, spoofed, or unregistered (e.g., unrecorded and / or unrecorded) domain name DNS requests, thus increasing the load on DNS as legitimate DNS requests and associated responses are not stored in the DNS cache.

[0277] In another example, non-DDoS attacks, such as DNS tunneling attacks or leaks, may occur based on the current percentage of bad, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS requests and the rate of change in that percentage. Such attacks should be prevented, but are unlikely to over-consume processing resources or fill up the DNS cache. DNS-G / K3150 may determine that a non-DDoS attack, such as a DNS tunneling attack or leak, is occurring. Based on this determination, DNS-G / K3150 may generate a DNS response with response code NXDOMAIN and send that response to the source of the DNS request. The DNS cache of the intermediate DNS proxy may store the NXDOMAIN response generated by DNS-G / K3150, and subsequent DNS requests for bad, fake, or unregistered (e.g., unregistered and / or unrecorded) domain names (including retries of previous bad, fake, or unregistered (e.g., unregistered and / or unrecorded) domain names) may be handled by the intermediate DNS proxy. Conversely, DNS-G / K3150 may decide not to generate a DNS response and send it back to the source of the DNS request, for example, to avoid revealing that the source has been discovered or otherwise identified as a malicious actor.

[0278] In yet another example, DNS-G / K3150, while operating an application that generates DNS requests, such as a web browser, may determine that a faulty, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS request is the result of human error or a simulated human error generated by malware. Based on such determination, DNS-G / K3150 can generate a DNS response with response code NXDOMAIN. Based on the response code, the application may not time out while waiting for a response to the request, nor will it automatically reissue the same faulty, spoofed, or unregistered (e.g., unregistered and / or unrecorded) domain name DNS request. For example, a web browser generating a faulty DNS request may repeat this retry cycle multiple times if it does not receive a DNS response to the DNS request, temporarily rendering the browser unavailable to a human operator until the browser gives up. Repeatedly issuing the same faulty DNS request can be considered a denial-of-service attack. By generating and sending a DNS response with response code NXDOMAIN, DNS-G / K3150 can prevent the repetition of the same faulty DNS request.

[0279] Figure 21 is a flowchart illustrating the operation of a DNS gatekeeper such as DNS-G / K3150. Steps 3611 through 3614 in Figure 21 are substantially the same as steps 3601 through 3604 in Figure 20, so the explanation of these steps will not be repeated here or in the diagram. The other steps concern the processing of DNS requests that can be forwarded to DNS by DNS-G / K3150 even though the domain name membership test in step 3605 of Figure 21 returned FALSE.

[0280] As mentioned above, some requests concern domain names that do not exist in DNS-REG3531 due to reasons such as country code lags. For example, there may be some gaps in the collection of all country code domains in the domain names registered (e.g., registered and / or recorded) in local DNS-REG3531, and the country code domain CCD-X, registered (e.g., registered and / or recorded) in global DNS, may not be included in the DNS-REG3531 distributed by DFM160 in step 3601 of Figure 20. Similarly, synchronization delays may be a factor, for example, if DNS-G / K3150 determines, based on data received from DFM3160, that the local DNS-REG3531 stored in DNS-G / K3150 has not been updated for a relatively long time. Furthermore, requests related to domain names not registered in DNS-REG3531 may also arise due to the use of unregistered (e.g., unregistered and / or unrecorded) domain names by legitimate information exchange services.

[0281] In step 3605, the CCD-X membership test for DNS-REG3531 returns a FALSE value, and in step 3606F, the associated packet is dropped. However, when a packet is dropped, DNS-G / K3150 does not collect any new information. As an alternative to dropping, DNS-G / K3150 may decide not to drop a DNS request packet and instead forward it to DNS if it determines there is sufficient probability that the domain name is actually registered (e.g., registered and / or recorded) in DNS, or for other possible reasons such as information gathering, or because it may determine that an unregistered (e.g., unregistered and / or unrecorded) domain name is part of a legitimate information exchange service. DNS-G / K3150 may monitor and observe the corresponding DNS response, which may either contain a resolved IP address that indicates the domain name is registered (e.g., registered and / or recorded) in DNS, or an error code such as NXDOMAIN that indicates the domain name is not registered (e.g., unregistered and / or unrecorded) in DNS, or no response indicating a DNS tunneling attack. From this perspective, additional processing may be provided instead of the determination in step 3605. In Figure 21, such determination is performed in steps 3615.1 and 3615.2.

[0282] In step 3615.1, DNS-G / K3150 can test whether the domain name is a member of the set represented by the key contained in DNS-REG3531 and represents all domain names currently registered (e.g., registered and / or recorded) in DNS. If the membership test returns TRUE, so that DNS-G / K3150 determines that the name is registered (e.g., registered and / or recorded) in DNS, in step 3616T, the packet and associated DNS request are recorded for statistical purposes and stored in memory store 3532. DNS-G / K3150 forwards the packet to network 3110, and DNS-G / K3150 returns to step 3612 to process the packet further.

[0283] If the membership test returns FALSE, in which it is determined that the domain name is not registered in DNS-REG3531 (e.g., not registered and / or not recorded), then in step 3615.2, the DNS-G / K3150 can decide whether to forward the packet to DNS. The forwarding decision may be based on the DNS-G / K3150's determination that the FALSE result of the membership test is due to some kind of lag. For example, the DNS-G / K3150 may determine whether the domain name contains a country code suffix. Based on the determination that the domain name contains a country code suffix, the DNS-G / K3150 may conclude that the FALSE result was due to a country code lag (as described above). Alternatively, the DNS-G / K3150 may determine that the local copy of DNS-REG3531 has not been updated by the DFM3160 for a relatively long time, and conclude that the FALSE result is due to a synchronization delay (as described above). Forwarding decisions may also be based on a determination by DNS-G / K3150 that the membership test for the use of an unregistered (e.g., unregistered and / or unrecorded) domain name as part of a legitimate information exchange service is FALSE. For example, DNS-G / K3150 may determine that an unregistered (e.g., unregistered and / or unrecorded) domain name contains a suffix corresponding to an authoritative name server operated by a legitimate information exchange service.

[0284] In step 3615.2, if DNS-G / K3150 decides not to forward the DNS request packet to DNS, steps 3616F, 3617, and 3618 are performed. These steps are substantially the same as steps 3606F, 3607, and 3608 in Figure 20, so a detailed explanation will not be repeated here. If DNS-G / K3150 decides to forward the DNS request packet, in step 3619, DNS-G / K3150 forwards the DNS request packet to DNS via network 3110. If DNS-G / K3150 determines that the domain name of the request is not registered in DNS-REG3531 (e.g., not registered and / or recorded), but the request packet should be forwarded, it can activate logic to monitor communications related to the forwarded request packet and detect the corresponding DNS response. In step 3620, DNS-G / K3150 detects or observes the corresponding DNS response, if any, destined for network 3114.

[0285] In step 3621, DNS-G / K3150 may check whether the DNS response indicates a resolved IP address so that it can be determined that the domain name is not represented in DNS-REG3531 but is registered in DNS (e.g., registered and / or recorded). If DNS-G / K3150 does not observe a DNS response indicating a resolved IP address, DNS-G / K3150 performs steps 3616F, 3617, and 3618 (as described above) and returns to step 3612 to process additional packets. If DNS-G / K3150 observes a DNS response indicating a resolved IP address, in step 3622, DNS-G / K3150 may generate keys based on the domain name and insert those keys into the local DNS-REG3531, or it may record the domain name for informational purposes and upload it to DFM3160 in step 3611. In step 3623, DNS-G / K3150 forwards the DNS response packet to its destination (via network 3114), and DNS-G / K3150 returns to step 3612 to process additional requests.

[0286] In step 3620, a DNS response may not be observed. For example, if the associated DNS request is a vector for a DNS tunneling attack (e.g., data leak),...

Claims

1. A method comprising receiving multiple packets transmitted from one or more hosts in a first network to one or more hosts in a second network using a packet filtering appliance, The packet filtering appliance is located at the boundary between the first network and the second network and stores a flat hash map index data structure associated with multiple rules of cybersecurity policies implemented by the packet filtering appliance. Each of the aforementioned rules includes a matching criterion that indicates one or more threat indicators related to the packet to which the rule applies, The flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of which rule identifiers corresponds to a rule among the plurality of rules, and each of which keys includes a value incrementally generated based on a portion of threat indicators from the matching criteria of the rule corresponding to the rule identifier to which the key is mapped. The process comprises extracting values ​​from fields of the received plurality of packets, and searching the flat hash map index data structure for rule identifiers of rules applicable to the received plurality of packets based on the extracted values, wherein each of the extracted values ​​includes a domain name, and the search is performed for each of the received plurality of packets. For each of the multiple labels of the domain name extracted from the packet, a value is generated for the portion of the extracted value that includes the label, based on the label. Each of the generated values ​​is compared with the key of the flat hash map index data structure, The process includes determining whether one or more of the rules are applicable to the packet based on comparing the generated value with the key of the flat hash map index data structure, The process involves applying one or more rules from the plurality of rules that are determined to be applicable to one or more of the aforementioned packets to one or more of the aforementioned packets, A method that, if applied, includes one or more of the following: blocking or dropping a packet, forwarding a packet, logging a packet, capturing a packet, redirecting or rerouting a packet, modifying or transforming a packet, or generating or sending a response to a packet.

2. For each packet of the aforementioned plurality of packets, the value is generated. To generate a hash of a first label indicating the top-level domain of the domain name extracted from the packet, and The method according to claim 1, comprising: for each of the one or more additional labels that indicate one or more subdomains of a domain name extracted from the packet, generating a hash of the portion of the domain name that includes the additional label and the top-level domain, based on the additional label.

3. For each packet of the aforementioned plurality of packets, the value is generated. To generate a lossless compression of a first label indicating the top-level domain of the domain name extracted from the packet, and The method according to claim 1, comprising: for each of the one or more additional labels indicating one or more subdomains of the domain name extracted from the packet, generating a lossless compression of the portion of the domain name including the additional label and the top-level domain based on the additional label.

4. The value extracted from the first packet among the one or more packets includes a uniform resource locator (URL) which includes a domain name as the host portion and a path portion which has one or more path segments, and The method according to claim 1, wherein generating the value includes, for the first packet, generating a hash based on the path segment for each of the one or more path segments, with respect to the portion of the URL that includes the path segment and the domain name of the URL.

5. The aforementioned flat hash map index data structure is A key containing a value incrementally generated based on a threat indicator that has only a domain name, The method according to claim 1, comprising: a key containing a value incrementally generated based on a threat indicator having a uniform resource locator (URL) having a domain name portion and a path portion.

6. The packet filtering appliance stores a second flat hash map index data structure associated with a second set of rules of the cybersecurity policy. The plurality of keys in the flat hash map index data structure are associated with a threat indicator that includes a domain name having the first characteristic, and The method according to claim 1, wherein the second flat hash map index data structure includes a second plurality of keys mapped to a second plurality of rule identifiers, and includes values ​​incrementally generated based on a portion of a threat indicator from a matching criterion that includes a domain name having a second characteristic different from the first characteristic.

7. The packet filtering appliance stores a second index data structure associated with a second set of rules of the cybersecurity policy, the second index data structure includes a set of internal nodes, and for each of the set of internal nodes, The internal node includes a first bit array, a second bit array, and one or more pointers. The first bit array includes elements mapped to possible values ​​of a k-bit chunk of the search object, and each of the first bit array elements stores a value indicating, for the possible value mapped to the first bit array element, either the presence of a corresponding descendant internal node among the plurality of internal nodes, or the absence of a corresponding descendant internal node. The second bit array includes elements mapped to possible values ​​of the k-bit chunk and possible values ​​of portions of the k-bit chunk, and each of the second bit array elements stores a value indicating, for a possible value mapped to the second bit array element, either the presence of one or more corresponding rule identifiers or the absence of a corresponding rule identifier. The method according to claim 1, wherein the one or more pointers indicate a memory location associated with one or more rule identifiers corresponding to a k-bit chunk.

8. The second index data structure maps a key containing at least a portion of an Internet Protocol (IP) address to a rule identifier of the second set of rules, and the method further Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet, The method according to claim 7, comprising: storing, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes, and a second rule identifier associated with a second internal node that is a descendant node of the first internal node among the plurality of internal nodes, in an order sorted by the priority of the related rules.

9. The second index data structure maps a key containing at least a portion of an Internet Protocol (IP) address to a rule identifier of the second set of rules, and the method further Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet, The method according to claim 7, comprising storing, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes, and a second rule identifier associated with the first internal node, in an order sorted by the priority of the related rules.

10. The aforementioned search is In the first processing thread, by searching the set data structure based on the value extracted from the first packet among the plurality of packets, it is determined whether or not the flat hash map index data structure contains a first key corresponding to the value from the first packet. In a second processing thread that runs concurrently with the first processing thread, a search of the flat hash map index data structure is initiated based on the value extracted from the first packet. The method according to claim 1, further comprising: in the first processing thread, determining that the flat hash map index data structure lacks a key corresponding to a value extracted from the first packet, and then terminating the search of the flat hash map index data structure.

11. A packet filtering appliance comprising one or more processors, wherein when executed by one or more processors, the packet filtering appliance, Receive multiple packets transmitted from one or more hosts in the first network to one or more hosts in the second network. The packet filtering appliance is located at the boundary between the first network and the second network and stores a flat hash map index data structure associated with multiple rules of cybersecurity policies implemented by the packet filtering appliance. Each of the aforementioned rules includes a matching criterion that indicates one or more threat indicators related to the packet to which the rule applies, The flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of which rule identifiers corresponds to a rule among the plurality of rules, and each of which keys includes a value incrementally generated based on a portion of threat indicators from the matching criteria of the rule corresponding to the rule identifier to which the key is mapped. The following steps are performed: extract values ​​from the fields of the received multiple packets, search the flat hash map index data structure for the rule identifier of the rule applicable to the received multiple packets based on the extracted values, and ensure that each of the extracted values ​​includes a domain name, and that the search is performed for each of the received multiple packets. For each of the multiple labels of the domain name extracted from the packet, a value is generated for the portion of the extracted value that includes the label, based on the label. Each of the generated values ​​is compared with the key of the flat hash map index data structure, The process includes determining whether one or more of the rules are applicable to the packet based on comparing the generated value with the key of the flat hash map index data structure, For one or more of the aforementioned multiple packets, apply one or more rules from the aforementioned multiple rules that are determined to be applicable to the aforementioned one or more packets. A packet filtering appliance having memory for storing instructions, configured such that the application includes one or more of the following: blocking or dropping a packet, forwarding a packet, logging a packet, capturing a packet, redirecting or rerouting a packet, modifying or transforming a packet, or generating or sending a response to a packet.

12. For each packet of the aforementioned plurality of packets, the value is generated. To generate a hash of a first label indicating the top-level domain of the domain name extracted from the packet, and The packet filtering appliance according to claim 11, comprising: for each of the one or more additional labels indicating one or more subdomains of a domain name extracted from the packet, generating a hash of the portion of the domain name including the additional label and the top-level domain based on the additional label.

13. For each packet of the aforementioned plurality of packets, the value is generated. To generate a lossless compression of a first label indicating the top-level domain of the domain name extracted from the packet, and The packet filtering appliance according to claim 11, comprising: for each of the one or more additional labels indicating one or more subdomains of a domain name extracted from the packet, generating a lossless compression of the portion of the domain name including the additional label and the top-level domain, based on the additional label.

14. The value extracted from the first packet among the one or more packets includes a uniform resource locator (URL) which includes a domain name as the host portion and a path portion which has one or more path segments, and The packet filtering appliance according to claim 11, wherein generating the value includes, for the first packet, for each of the one or more path segments, generating a hash based on the path segment for the portion of the URL that includes the path segment and the domain name of the URL.

15. The aforementioned flat hash map index data structure is A key containing a value incrementally generated based on a threat indicator that has only a domain name, A packet filtering appliance according to claim 11, comprising: a key containing a value incrementally generated based on a threat indicator having a uniform resource locator (URL) having a domain name portion and a path portion.

16. The packet filtering appliance stores a second flat hash map index data structure associated with a second set of rules of the cybersecurity policy. The plurality of keys in the flat hash map index data structure are associated with a threat indicator that includes a domain name having the first characteristic, and The packet filtering appliance according to claim 11, wherein the second flat hash map index data structure includes a second plurality of keys mapped to a second plurality of rule identifiers, and includes values ​​incrementally generated based on a portion of a threat indicator from a matching criterion that includes a domain name having a second characteristic different from the first characteristic.

17. The packet filtering appliance stores a second index data structure associated with a second set of rules of the cybersecurity policy, the second index data structure includes a set of internal nodes, and for each of the set of internal nodes, The internal node includes a first bit array, a second bit array, and one or more pointers. The first bit array includes elements mapped to possible values ​​of a k-bit chunk of the search object, and each of the first bit array elements stores a value indicating, for the possible value mapped to the first bit array element, either the presence of a corresponding descendant internal node among the plurality of internal nodes, or the absence of a corresponding descendant internal node. The second bit array includes elements mapped to possible values ​​of the k-bit chunk and possible values ​​of portions of the k-bit chunk, and each of the second bit array elements stores a value indicating, for a possible value mapped to the second bit array element, either the presence of one or more corresponding rule identifiers or the absence of a corresponding rule identifier. The packet filtering appliance according to claim 11, wherein the one or more pointers indicate a memory location associated with one or more rule identifiers corresponding to a k-bit chunk.

18. A packet filtering appliance wherein the second index data structure maps keys, which include at least a portion of Internet Protocol (IP) addresses, to rule identifiers of the second plurality of rules, and when the instruction is executed by one or more processors, the packet filtering appliance, Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet. The packet filtering appliance according to claim 17, configured to store, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes, and a second rule identifier associated with a second internal node that is a descendant node of the first internal node among the plurality of internal nodes, in an order sorted by the priority of the related rules.

19. A packet filtering appliance wherein the second index data structure maps keys, which include at least a portion of Internet Protocol (IP) addresses, to rule identifiers of the second plurality of rules, and when the instruction is executed by one or more processors, the packet filtering appliance, Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet. The packet filtering appliance according to claim 17, configured to store, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes and a second rule identifier associated with the first internal node, in an order sorted by the priority of the relevant rules.

20. The aforementioned search is In the first processing thread, by searching the set data structure based on the value extracted from the first packet among the plurality of packets, it is determined whether or not the flat hash map index data structure contains a first key corresponding to the value from the first packet. In a second processing thread that runs concurrently with the first processing thread, a search of the flat hash map index data structure is initiated based on the value extracted from the first packet. The packet filtering appliance according to claim 11, further comprising: in the first processing thread, determining that the flat hash map index data structure lacks a key corresponding to a value extracted from the first packet, and then terminating the search of the flat hash map index data structure.

21. One or more non-temporary computer-readable media, which, when executed by one or more processors of a packet filtering appliance, the packet filtering appliance, Receive multiple packets transmitted from one or more hosts in the first network to one or more hosts in the second network. The packet filtering appliance is located at the boundary between the first network and the second network and stores a flat hash map index data structure associated with multiple rules of cybersecurity policies implemented by the packet filtering appliance. Each of the aforementioned rules includes a matching criterion that indicates one or more threat indicators related to the packet to which the rule applies, The flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of which rule identifiers corresponds to a rule among the plurality of rules, and each of which keys includes a value incrementally generated based on a portion of threat indicators from the matching criteria of the rule corresponding to the rule identifier to which the key is mapped. The following steps are performed: extract values ​​from the fields of the received multiple packets, search the flat hash map index data structure for the rule identifier of the rule applicable to the received multiple packets based on the extracted values, and ensure that each of the extracted values ​​includes a domain name, and that the search is performed for each of the received multiple packets. For each of the multiple labels of the domain name extracted from the packet, a value is generated for the portion of the extracted value that includes the label, based on the label. Each of the generated values ​​is compared with the key of the flat hash map index data structure, The process includes determining whether one or more of the rules are applicable to the packet based on comparing the generated value with the key of the flat hash map index data structure, For one or more of the aforementioned multiple packets, apply one or more rules from the aforementioned multiple rules that are determined to be applicable to the aforementioned one or more packets. One or more non-temporary computer-readable media comprising stored instructions configured such that the application includes one or more of the following: blocking or dropping a packet; forwarding a packet; logging a packet; capturing a packet; redirecting or rerouting a packet; modifying or transforming a packet; or generating or transmitting a response to a packet.

22. For each packet of the aforementioned plurality of packets, the value is generated. To generate a hash of a first label indicating the top-level domain of the domain name extracted from the packet, and One or more non-temporary computer-readable media according to claim 21, comprising: for each of the one or more additional labels indicating one or more subdomains of a domain name extracted from the packet, generating a hash of the portion of the domain name including the additional label and the top-level domain based on the additional label.

23. For each packet of the aforementioned plurality of packets, the value is generated. To generate a lossless compression of a first label indicating the top-level domain of the domain name extracted from the packet, and One or more non-temporary computer-readable media according to claim 21, comprising: for each of one or more additional labels indicating one or more subdomains of a domain name extracted from the packet, generating a lossless compression of the portion of the domain name including the additional label and the top-level domain based on the additional label.

24. The value extracted from the first packet among the one or more packets includes a uniform resource locator (URL) which includes a domain name as the host portion and a path portion which has one or more path segments, and One or more non-temporary computer-readable media according to claim 21, wherein generating the value includes, for each of the one or more path segments of the first packet, generating a hash based on the path segment for the portion of the URL that includes the path segment and the domain name of the URL.

25. The aforementioned flat hash map index data structure is A key containing a value incrementally generated based on a threat indicator that has only a domain name, One or more non-temporary computer-readable media according to claim 21, comprising: a key containing a value incrementally generated based on a threat indicator having a uniform resource locator (URL) having a domain name portion and a path portion;

26. The packet filtering appliance stores a second flat hash map index data structure associated with a second set of rules of the cybersecurity policy. The plurality of keys in the flat hash map index data structure are associated with a threat indicator that includes a domain name having the first characteristic, and One or more non-temporary computer-readable media according to claim 21, wherein the second flat hash map index data structure includes a second plurality of keys mapped to a second plurality of rule identifiers, and includes values ​​incrementally generated based on a portion of a threat indicator from a matching criterion that includes a domain name having a second characteristic different from the first characteristic.

27. The packet filtering appliance stores a second index data structure associated with a second set of rules of the cybersecurity policy, the second index data structure includes a set of internal nodes, and for each of the set of internal nodes, The internal node includes a first bit array, a second bit array, and one or more pointers. The first bit array includes elements mapped to possible values ​​of a k-bit chunk of the search object, and each of the first bit array elements stores a value indicating, for the possible value mapped to the first bit array element, either the presence of a corresponding descendant internal node among the plurality of internal nodes, or the absence of a corresponding descendant internal node. The second bit array includes elements mapped to possible values ​​of the k-bit chunk and possible values ​​of portions of the k-bit chunk, and each of the second bit array elements stores a value indicating, for a possible value mapped to the second bit array element, either the presence of one or more corresponding rule identifiers or the absence of a corresponding rule identifier. The one or more non-temporary computer-readable media according to claim 21, wherein the one or more pointers indicate a memory location associated with one or more rule identifiers corresponding to a k-bit chunk.

28. The second index data structure is one or more non-temporary computer-readable media that maps keys, which include at least a portion of Internet Protocol (IP) addresses, to rule identifiers of the second plurality of rules, and when the instruction is executed by one or more processors, the packet filtering appliance, Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet. One or more non-temporary computer-readable media according to claim 27, configured to store, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes, and a second rule identifier associated with a second internal node that is a descendant node of the first internal node among the plurality of internal nodes, in an order sorted by the priority of the related rules.

29. The second index data structure is one or more non-temporary computer-readable media that maps keys, which include at least a portion of Internet Protocol (IP) addresses, to rule identifiers of the second plurality of rules, and when the instruction is executed by one or more processors, the packet filtering appliance, Based on the IP address extracted from the field of the first packet received by the packet filtering appliance, the second index data structure is searched for the rule identifier of the rule among the second plurality of rules that is applicable to the first packet. One or more non-temporary computer-readable media according to claim 27, configured to store, based on the search, a first rule identifier associated with a first internal node among the plurality of internal nodes and a second rule identifier associated with the first internal node, in an order sorted by the priority of the relevant rules.

30. The aforementioned search is In the first processing thread, by searching the set data structure based on the value extracted from the first packet among the plurality of packets, it is determined whether or not the flat hash map index data structure contains a first key corresponding to the value from the first packet. In a second processing thread that runs concurrently with the first processing thread, a search of the flat hash map index data structure is initiated based on the value extracted from the first packet. One or more non-temporary computer-readable media according to claim 21, further comprising: in the first processing thread, determining that the flat hash map index data structure lacks a key corresponding to a value extracted from the first packet, and terminating the search of the flat hash map index data structure.

Images