Method and system for efficiently and mandatorily executing network security policy in network communication

Abstract

Methods, devices, systems, and machine-readable media are disclosed for improving packet filtering efficiency by reducing processing time and / or reducing memory usage. A packet filtering device may use any of various types of data structures, such as a flat hash map and / or rule tree, to search for network security policy packet filtering rules that should be applied to packets in transmission. The packet filtering device may search for an index data structure to find a match of a search object (in the form of a value extracted by the packet filtering device from a transmitted packet) to a threat indicator matching criterion of the policy rule. Each of the index data structures may map rule identifiers (rule IDs) for policy rules to keys that are based on (or include) matching criteria for the rules.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 547,166, filed November 3, 2023, and also claims priority to U.S. Provisional Patent Application Serial No. 63 / 468,401, filed May 23, 2023. The entire contents of each of Application Serial No. 63 / 547,166 and Application Serial No. 63 / 468,401 are incorporated herein by reference. BACKGROUND

[0002] Any statements as to the background of the technology in this background section are not to be taken as an admission that any system, apparatus, method, or other feature described in or mentioned as prior art is art to the present technology, nor are they to be taken as an admission that any of the descriptions reflect or were made in view of the prior art or common general knowledge in the art.

[0003] As the information age continues to evolve, network security is becoming increasingly important. Network threats / attacks can take many forms (e.g., unauthorized requests or data transfers, viruses, malware, large volumes of traffic intended to occupy resources, and so on).

[0004] To combat such threats and attacks, various network defense methods and systems have been developed and deployed. One network defense system uses a packet security gateway to protect a network from threats / attacks. The packet security gateway can include a network packet filtering appliance (e.g., an in-line, non-endpoint appliance) that mediates data transfer between communicating endpoints, is capable of applying an ordered list of packet filtering rules (which can represent and / or constitute a network protection policy) to TCP / IP packets in transit to enforce the policy, thereby protecting the network. When the packet matching criteria of the policy’s packet filtering rules originate from cyber threat intelligence (CTI), then the packet security gateway can be referred to as a threat intelligence gateway (TIG), and the policy can be referred to as a cyber security policy. The TIG can act as a gateway or interface between a network protected or shielded by the CTI (e.g., a private enterprise network connected to the Internet) and a network that is not similarly shielded / protected (e.g., the Internet). Thus, one or more TIGs can be located at the boundary between networks.

[0005] The packet filtering rules can be sourced from a database of threat indicators provided by a CTI source, such as a CTI provider organization. The ordered list of packet filtering rules can represent a network security policy for TIG-enforced protection of a network security. The threat indicators can be used as packet matching criteria for the rules to identify packets associated with threats. The threat indicators, which can be indicators provided by a CTI provider or other CTI source, can include internet network addresses (which can be IP addresses, IP address ranges, IP subnet addresses expressed in Classless Inter-Domain Routing (CIDR) notation, combinations of L3 IP addresses with L4 ports and L3 protocol types (e.g., "5-tuples")), domain names, Uniform Resource Identifiers (URIs, such as Uniform Resource Locators (URLs) and Uniform Resource Names (URNs), etc.) of networking resources (e.g., network host computers) that can be controlled or operated by threat actors, or that can otherwise be associated with malicious activity. It should be noted that at the current time of the present disclosure, URIs in CTI are typically URLs (and not URNs). The threat indicators can also include identifiers of certificates and associated certificate authorities used for securing certain TCP / IP communications (e.g., X.509 certificates used for securing communications by the Transport Layer Security (TLS) protocol, e.g., HyperText Transfer Protocol (HTTP)-mediated sessions). In the context of network security, the threat indicators can also include geoIP information that associates IP addresses with geographic locations, and can include geopolitical information such as countries and cities. A packet in transit can be said to "match" a "CTI-derived" packet filtering rule or CTI-derived rule when any field value in the packet in transit corresponding to an IP address, 5-tuple, domain name, URI, certificate identifier, certificate authority, etc. matches a threat indicator or matching criterion included by the packet filtering rule. The TIG can enforce a CTI-derived network security policy (or CTI-derived policy), which can include a list of CTI-derived packet filtering rules.

[0006] The CTI provider can associate threat metadata with each threat indicator, which can include, for example, threat / attack type, threat name, threat risk score and / or confidence level, threat actor (attribution), etc. In addition, the CTI provider can provide an information service for looking up threat intelligence reports associated with a threat indicator. The TIG and associated applications can use the threat metadata and other threat intelligence information to further harden defenses against threats, and to further improve the network security posture.

[0007] The network security policy manager can create a CTI-derived network protection / network security policy as an ordered list of CTI-derived packet filtering rules by receiving threat indicators and associated threat metadata from one or more CTI providers and generating packet filtering rules based on these indicators and metadata. Each packet filtering rule can include: (a) criteria for matching packet data, such as one or more threat indicators; (b) an action or combination of actions to apply to a packet upon matching the criteria, including a disposition action of blocking / dropping / rejecting the packet or allowing / forwarding / passing the packet; and (c) metadata associated with the threat indicator(s). Generally, a packet filtering device will apply any matching packet filtering rules to a packet (in transit) in the order that the rules are stored in the list, thereby enforcing the policy on the packet. A packet filtering rule can match a packet if the criteria of the packet filtering rule matches the values in one or more fields of the packet corresponding to the criteria. Each packet filtering rule in a policy can be associated with a rule identifier, which can be unique with respect to other rules in the policy. The rule identifier can be associated with a rank or rank order, which can indicate an order in which a set of rules are to be applied to a packet, e.g., a higher ranked rule can be applied to a packet first and a lower ranked rule can be applied to the packet second. In some contexts, the rule identifier can correspond to a rank or rank order of the rule in an ordered list of rules that make up the policy. For example, a rule at the head or front of the list can have a highest rank and a rule at the end of the list can have a lowest rank. The rank order of rules in a list / policy can correspond to an order in which a TIG can apply matching rules to a packet in transit. The matching criteria of each rule can include one or more pairs consisting of a packet field name (e.g., a L3 / Internet layer destination IP address field) and a field value. The field value(s) can be in the form of one or more threat indicators that are network addresses (e.g., IP addresses, IP address ranges, 5-tuples, domain names, URIs, etc.) or identifiers of certificates or certificate authorities. The action or combination of actions can include some combination of blocking / dropping / rejecting the packet, allowing / forwarding / passing the packet, logging the packet, capturing the packet, redirecting or re-routing the packet, modifying or transforming the packet in some way to protect the network, etc. In the context of TIGs and CTI-derived policies, a rule action can be referred to as a packet transformation function (PTF), which can transform a packet to protect the network. Multiple rule actions / PTFs can be combined to further protect the network and can be applied in sequence or concurrently, or both. For example, a rule action / PTF can block a TCP SYN packet that can be initiating a connection establishment with a threat endpoint, but also transform the TCP SYN packet into a corresponding TCP RST packet that can be sent to the sender of the TCP SYN packet to stop the connection establishment process, e.g., which can stop an attack that can be associated with a threat, thereby protecting an associated network.Another rule action / PTF can create a log of TCP SYN packets, and this log can be packaged and sent to a network protection application. For another example, a PTF can transform blocked / dropped packets to null, which can similarly protect a network. For example, threat metadata can be used to signal a (human or machine) network analyst to a location of an associated network threat intelligence report, compute a threat risk score, select a threat analysis method to apply to a packet and communications associated with the packet, select or compute an action(s) to apply to a packet based on a threat risk, etc.

[0008] For example, one or more TIGs can be inserted into one or more of the Internet access links of a (protected) network. A policy manager associated with the one or more TIGs can be configured to: (a) receive CTI (e.g., threat indicators and associated metadata) from one or more CTI providers; (b) generate one or more (network protection / network security) policies, which are comprised of packet filtering rules derived from the threat indicators and metadata; and (c) download or communicate the one or more policies to one or more (subscribing) TIGs, which can be associated with the policy manager through a policy subscription. The TIGs can be configured with the policies, which can then be enforced by applying the associated packet filtering rules to packets in each transmission as the packets traverse the Internet access links (in either direction). If a match is found between a rule and a packet in a transmission, the action(s) of the rule can be applied to the packet to (according to the associated policy) protect the network.

[0009] CTI providers can update the CTI they provide to their subscribers, e.g., by adding or deleting threat indicators that comprise an existing CTI feed, providing a new CTI feed or deleting an existing CTI feed. Accordingly, a subscribing policy manager that can provide / distribute CTI-derived policies to TIGs can use the updated CTI to update the policies and distribute the updated policies to the TIGs. Upon receiving the updated policies, the TIGs can process the updated policies for efficient enforcement and then replace the existing policies being enforced with the updated policies. The policy update, processing, and replacement cycle can be performed periodically, e.g., every 15 minutes, depending on the rate of updates to the CTI by the CTI providers.

[0010] The effectiveness of using TIGs and associated CTI to protect a network generally depends on the scope and quality of the threat indicators and metadata and the associated packet filtering rules applied by the TIG, as well as the performance of the TIG. At the current time of the present disclosure, a total of several hundred million threat indicators (and associated threat metadata) are available from a number of CTI sources (e.g., hundreds of CTI providers), in forms including IP addresses, IP address ranges (which can be expressed in CIDR notation), 5-tuples, domain names, URIs, etc., as well as certificate identifiers, certificate authority identifiers, etc. These vast databases of threat indicators can translate into a corresponding or similar number of packet filtering rules. Since threat indicators that can be observed in the communication traffic of a given network can be unknown until they are observed, a TIG can apply all available CTI or associated packet filtering rules at any time to ensure effective protection. At the current time of the present disclosure, CTI providers can collectively provide several hundred million threat indicators; thus, a TIG policy can consist of millions, tens of millions, or even hundreds of millions of rules. Given the growth trend of CTI, the number of threat indicators available from CTI providers can soon reach the tens of billions, and thus a TIG policy can soon include tens of billions of packet filtering rules. Accordingly, a TIG must be able to apply millions of packet filtering rules to each packet in transit without adversely affecting network performance (e.g., without introducing more than a few microseconds of latency or dropping packets due to, e.g., packet buffer overflow, which can result from excessive latency). Additionally, as the available CTI and associated TIG policy grows, the TIG's main memory or space requirements for policy enforcement logic can also grow.

[0011] At the current time of the present disclosure, network links of 100 Mb, 1 Gb, 10 Gb, and / or 40 / 100 Gb can typically be used at the TIG insertion point, which means that packet transmission rates can reach millions or tens of millions of packets per second. A TIG must filter each packet in transit through millions of rules without impacting network performance to enforce the policy. For such a large policy and high packet rates, even if the policy and associated logic can be stored in main memory and even if fast (e.g., sub-linear) policy search algorithms are used, the performance of the TIG can be insufficient and can result in unacceptable performance degradation. For example, high latency can result in dropped packets due to packet buffer overflow. Latency incurred during packet filtering should be low, and packets should not be dropped (e.g., due to buffer overflow).

[0012] The TIG can be designed, architected, and configured to achieve the performance required for enforcement of CTI-derived policies through some combination of: (a) high-performance central processing units (CPUs) and associated computer platforms (which can be architected and / or configured for network packet processing) in combination with software development kits (SDKs) designed to leverage the architecture and resources of the CPUs (e.g., multi-threading, multi-core, high-performance registers and caches) to maximize packet processing performance; (b) fast / efficient algorithms and associated time-efficient and space-efficient data structures for searching rules to find matches between the rule’s criteria / matching standards and corresponding field values of a packet in current transmission; and (c) storing policies (which can include millions of packet filtering rules) and associated policy enforcement logic in high-speed local memory (e.g., on-board SDRAM, which is commonly referred to as “main memory”) so that the CPU(s) can quickly access the main memory via high-speed, high-bandwidth data buses.

[0013] At the time of this disclosure, with respect to (a) above, there are multiple available CPUs / processors (e.g., x86 family, ARM family, and PowerPC family) that can be integrated with computer platforms and combined with SDKs like the Data Plane Development Kit (DPDK) to support the network packet processing requirements of the TIG. With respect to (b) above, the fast / efficient algorithms and associated data structures for searching for matching rules in policies as described in U.S. Patent No. 11,012,417 (“the ‘417 patent”) (incorporated by reference herein) can support the policy searching requirements of the TIG. With respect to (c) above, the ‘417 patent describes methods, algorithms, and data structures, such as efficient probabilistic set data structures (e.g., Bloom filters, cuckoo filters, etc.), for reducing the memory requirements of a policy relative to the size of the policy so that (large) CTI-derived policies can be stored in the available main memory of the TIG and enforced against network packet traffic while meeting latency or packet filtering rate requirements.

[0014] However, some current trends in cyber threat intelligence and cyber protection, including the continuing rapid growth in the number and diversity of CTIs, new / emerging threats, new / emerging types of threat indicators (e.g., IPv6 indicators, basic / registerable domain names), faster link speeds, higher network traffic rates, etc., can cause existing methods, algorithms, and data structures to be inadequate to meet the requirements for enforcing modern / future CTI-derived policies on network packet communications. Overcoming these inadequacies is often impractical, e.g., by substantially increasing CPU / processor speed and resources and / or by substantially increasing main memory. The adverse effects of these inadequacies can include reduced effectiveness or vulnerabilities in cyber protection against global cyber threats.

[0015] Accordingly, there is a need for new techniques (e.g., new, more efficient methods, algorithms, and data structures) to enforce comprehensive CTI-derived policies on network communications while meeting performance requirements without requiring substantial increases in computer resources of TIGs. SUMMARY

[0016] This Summary is provided to introduce some selected concepts in a simplified form, as a prelude to the detailed description. This Summary is not intended to be limiting in scope.

[0017] Using one or more of the various index data structures disclosed herein, a packet filtering device can efficiently search packet filtering rules of a network security policy for rules that should be applied to a packet in transit. The packet filtering device can search the index data structures for a match of a search object (in the form of values extracted by the packet filtering device from the packet in transit) to matching criteria of threat indicators of the policy rules. The extracted values and matching criteria can include, for example, IP addresses (or portions thereof), domain names (or portions thereof), URIs (or portions thereof), and / or other types of values. Each of the index data structures can map keys based on (or including) the policy rule matching criteria to rule identifiers (rule IDs) of those policy rules.

[0018] The index data structure can include flat hash maps that map keys that include hashes of threat indicators that originate from policy rule matching criteria to rule IDs of those policy rules. The flat hash maps can be efficiently searched and / or stored by the packet filtering device. The hashes can include hashes of less than all portions of values that can be found in packets in transit. For example, a flat hash map can map a hash of a threat indicator (that originates from a policy rule matching criterion and that includes a top-level domain (TLD) and one or more subdomains below the TLD) to a rule ID of that policy rule. However, the threat indicator can not include all subdomains that can be found in a fully qualified domain name (FQDN) extracted from a packet in transit. As another example, a flat hash map can map a hash of a threat indicator (that originates from a policy rule matching criterion and that includes less than all path segments that can be found in a URL extracted from a packet in transit) to a rule ID of that policy rule. In searching the flat hash map associated with such a threat indicator, a domain name, URL, or other value extracted from a packet in transit can be incrementally hashed, and the incrementally hashed portions of the domain name, URL, or other value can be searched in the flat hash map.

[0019] The index data structure can include flat hash maps that map keys that include hashes of threat indicators that originate from policy rule matching criteria to rule IDs of those policy rules. The flat hash maps can be efficiently searched and / or stored by the packet filtering device. The hashes can include hashes of less than all portions of values that can be found in packets in transit. For example, a flat hash map can map a hash of a threat indicator (that originates from a policy rule matching criterion and that includes a top-level domain (TLD) and one or more subdomains below the TLD) to a rule ID of that policy rule. However, the threat indicator can not include all subdomains that can be found in a fully qualified domain name (FQDN) extracted from a packet in transit. As another example, a flat hash map can map a hash of a threat indicator (that originates from a policy rule matching criterion and that includes less than all path segments that can be found in a URL extracted from a packet in transit) to a rule ID of that policy rule. In searching the flat hash map associated with such a threat indicator, a domain name, URL, or other value extracted from a packet in transit can be incrementally hashed, and the incrementally hashed portions of the domain name, URL, or other value can be searched in the flat hash map.

[0020] The index data structure can include rule trees that map bit string representations of threat indicators (e.g., IP addresses) that originate from policy rule matching criteria to rule IDs of those policy rules. The rule trees can be efficiently searched and / or stored by the packet filtering device and can facilitate dynamic runtime updates. The rule trees can include multiple levels of internal nodes that can be searched based on successive k-bit blocks of the search object. During a search, at any internal node of the rule tree, multiple rule IDs can be added to a list that includes search results based on a match of a k-bit block of the search object to an associated value / rule ID associated with that internal node. The search of the rule tree can continue after the first match and associated rule ID is found and can find other matches and associated rule IDs at other internal nodes of the rule tree at deeper levels based on more parts / blocks of the search object.

[0021] The searches of the index data structures described herein can be combined with membership tests based on searches of the set data structures. Prior to searching the index data structure, the search object can be used to search the set data structure to determine whether the search object (or a portion thereof, or a value based on the search object or a portion thereof) is a member of a set of all keys represented in the associated index data structure. If the search object (or a portion thereof, or a value based on the search object or a portion thereof) is a member of the set, the search of the index data structure can continue. If the search object (or a portion thereof, or a value based on the search object or a portion thereof) is not a member of the set, the search of the index data structure can be unnecessary and can be omitted, saving processing time. Alternatively, the searches of the set data structure and the associated index data structure can be performed concurrently. If the search of the set data structure indicates that the search object (or a portion thereof, or a value based on the search object or a portion thereof) is not a member of the set of all keys represented in the associated index data structure, the search of the associated index data structure can be terminated prior to completion, saving processing resources.

[0022] An index data structure can be selected for each of multiple types of threat indicators (e.g., a first type of threat indicator includes URLs, a second type of threat indicator includes domain names, a third type of threat indicator includes first-class IP addresses, a fourth type of indicator includes second-class IP addresses, etc.), facilitating more efficient utilization of processing and memory resources. Multiple instances of a particular type of data structure can be used for a particular type of threat indicator (e.g., one instance for threat indicators of that particular type that share a first common characteristic, another instance for threat indicators of that particular type that share a second common characteristic), or for different types of threat indicators (e.g., a first rule tree for IPv4 threat indicators, and a second rule tree for IPv6 threat indicators). Alternatively, multiple instances of different types of index data structure can be used for a particular type of threat indicator. A single data structure instance can be used for multiple types of threat indicators (e.g., a single flat hash map can be used for domain name threat indicators and URL threat indicators). A packet filtering device can use multiple concurrent processing threads to simultaneously search multiple index data structures based on different values extracted from a packet in transit. Each of these concurrent processing threads can output one or more rule IDs determined based on the index data structure search performed in that processing thread (or can determine that the search found no rule IDs). The rule IDs determined in each processing thread (if any) can be collected, and the rules corresponding to some or all of these rule IDs can be applied to the packet in transit based on the ranking / priority of the rules.

[0023] These and other features are described in more detail below. BRIEF DESCRIPTION OF DRAWINGS

[0024] Some features are shown in the drawings in example rather than limiting detail, and similar elements are referred to by similar reference numbers throughout the several views.

[0025] FIG. 1 An example network environment for efficient network security policy enforcement is shown.

[0026] FIG. 2A , FIG. 2B , FIG. 2C , FIG. 2D , FIG. 2E , FIG. 2F and FIG. 2G is a flowchart showing an example method of searching a flat hash map index data structure based on a key derived from values obtained from fields of a packet in transit.

[0027] FIG. 3A An example rule tree and related data structures are shown.

[0028] FIG. 3Bis a flowchart showing an example method for rule tree key insertion.

[0029] FIG. 3C is a flowchart showing an example method for searching an example rule tree, such as the rule tree in FIG. 3A .

[0030] FIG. 3D is a flowchart showing another example method for searching an example rule tree, such as the rule tree in FIG. 3A .

[0031] FIG. 4 is a block diagram showing an example method for efficient policy enforcement, which can include selecting and using one or more of the data structures and associated methods of use described herein.

[0032] FIG. 5 is a block diagram showing an example method for selecting data structures for a network protection policy.

[0033] FIG. 6 is a block diagram showing an example method for populating data structures for a network protection policy.

[0034] FIG. 7 is a block diagram showing an example of policy searching and enforcement.

[0035] FIG. 8 is a flowchart showing an example method for calculating memory requirements for a set of data structures selected for a network protection policy (e.g., index data structures and set data structures).

[0036] FIG. 9 is a block diagram showing an example method for determining a k parameter value for a rule tree.

[0037] FIG. 10 is a block diagram showing an example computing device.

[0038] FIG. 11 is a block diagram showing an example packet filtering device.

[0039] FIG. 12 shows an example network environment, for example, related to efficient network protection for mobile devices and associated enterprise networks, where rule trees and / or flat hash maps can be used as membership test set data structures.

[0040] FIG. 13 is a flowchart showing an example method for configuring mobile device and associated enterprise network elements.

[0041] FIG. 14is a flow diagram illustrating an example method for packet filtering in a mobile device and associated enterprise network.

[0042] FIG. 15 Another example network environment is illustrated in which a rule tree and / or flat hash map can be used as a set membership test data structure.

[0043] FIG. 16 is a flow diagram illustrating an example method for configuring a remote network, associated tunnel gateway, and associated central network element.

[0044] FIG. 17 is a flow diagram illustrating an example method for packet filtering and efficient backhauling for network protection in a remote network.

[0045] FIG. 18 Another network environment example is illustrated in which a rule tree and / or flat hash map can be used as a set membership test data structure.

[0046] FIG. 19 is a block diagram illustrating an example DNS gatekeeper.

[0047] FIG. 20 is a flow diagram illustrating an example method for operating a DNS gatekeeper.

[0048] FIG. 21 is a flow diagram illustrating another example method for operating a DNS gatekeeper.

[0049] FIG. 22 is a flow diagram illustrating an example method for operating a DNS filter manager.

[0050] FIG. 23 An event sequence is illustrated for an interconnected network having a DNS gatekeeper that can mitigate or prevent DDoS attacks on DNS.

[0051] FIG. 24 An event sequence is illustrated for an interconnected network having a DNS gatekeeper that can mitigate or prevent DNS tunneling attacks. DETAILED DESCRIPTION

[0052] Aspects described herein generally relate to computer hardware and software and cybersecurity. In particular, one or more aspects of the present disclosure generally relate to computer hardware and software for efficiently filtering packets in transit based on packet filtering rules derived from cyber threat intelligence (CTI). Disclosed herein are, for example, methods, devices, systems, and machine-readable media that can be relevant to efficient packet filtering for CTI-based applications, including enforcement of CTI-derived cybersecurity policies on packets in transit by a TIG. The methods, devices, systems, and machine-readable media disclosed herein can help improve packet filtering efficiency and can include improvements related to processing time and / or memory usage.

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various examples of how aspects of the disclosure can be put into practice. It should be understood that the

[0054] Various connections between elements are discussed in the following description. These connections are general and, unless otherwise specified, can be direct or indirect (e.g., via one or more networks), wired or wireless, physical or logical (e.g., virtual or software-defined), any combination of the above, and so forth.

[0055] FIG. 1 An example network environment 100 is shown for more efficient cybersecurity policy enforcement, and some or all aspects of the present disclosure can be implemented in this environment. The network environment 100 includes a plurality of private networks 101.1 - 101.n, where “n” can be any integer value. Any of the networks 101.1 - 101.n can be referred to as network 101, and the networks 101.1 - 101.n can be collectively referred to as networks 101. Similar conventions will be used for other groups of similarly numbered elements in the figures.

[0056] Each of the networks 101 can be, for example, an enterprise network that includes one or more packet filtering devices that enforce one or more network security policies using one or more of the methods described herein. In the network environment 100, network 101.1 includes a TIG 102.1 located at a boundary between network 101.1 and network 105, network 101.2 includes a TIG 102.2 located at a boundary between network 101.2 and network 105, and network 101.n includes a TIG 102.n located at a boundary between network 101.n and network 105. Network 105 can be, for example, a public and / or unsecured network such as the Internet. The TIGs 102, which can also or alternatively be configured as gateways or other interfaces, can be configured such that all traffic between the secured networks 101 and unsecured networks such as network 105 must pass through the TIGs 102, thereby securing data communications across the boundary between the networks 101 and unsecured networks. The network environment 100 can further include one or more CTI providers (CTIPs) 106 and one or more security policy management servers (or services) (SPMSs) 107 connected to network 105. The network environment 100 can also include one or more additional networks 109.1 - 109.n connected to network 105. Although shown as being directly connected to network 105 in FIG. 1 for convenience, the CTIP(s) 106 and SPMS(s) can be hosts within other networks connected to network 105. The network environment 100 can include other types of computing devices connected to network 105 (or as part of a network connected to network 105), for example, domain name system (DNS) servers / devices, security operations center (SOC) servers / devices, global threat context servers (GTCSs), and / or other devices. FIG. 1

[0057] ​Network 101 can include hosts H (e.g., servers, laptop or desktop computers, and / or other networked computing devices) that are configured to communicate with hosts in network 105 or networks connected to network 105 via network 105. Some of the hosts in network 105 and / or in networks connected to network 105 can be malicious. For example, network 109 can include malicious hosts MH that can instigate, support, or otherwise be associated with one or more types of network threats or attacks. To detect, prevent, stop, and / or otherwise take action in response to such network threats / attacks, each TIG 102 can filter data traffic (incoming and outgoing) between hosts H in network 101 and hosts outside of network 101. Although TIGs 102 are used herein as an example of a packet filtering device that is configured to perform the operations described herein, other types of packet filtering devices can also or alternatively be configured to perform the operations described herein for TIGs.

[0058] CTIP(s) 106 can continuously identify associated threats or potential threats in network 105 (or networks accessible via network 105), create threat intelligence reports regarding the threats / potential threats, determine indicators associated with the threats / potential threats, and publish (e.g., stream) a list of threat indicators or push. SPMS(s) 107 can receive the push from CTIP(s) 106, can create rules based on the threat indicators from the push and combine the rules into policies, can output the policies to TIGs 102, can generate (e.g., based on additional pushes from CTIP(s) 106) policy and / or rule updates and output to TIGs 102, and / or can otherwise manage or control policy-related functions of TIGs 102. As described further herein, SPMS(s) 107 and / or TIGs can also determine index data structures to be used to map search keys (e.g., corresponding matching criteria from rules) and rule identifiers, determine set (e.g., membership test) data structures to be used to determine whether a set of rules includes criteria that will match a particular value, populate the index data structures and set data structures, and / or perform other operations.

[0059] Each TIG 102 can receive policies / updates, can store policies / updates, and / or can filter packet traffic based on packet filtering rules for those policies / updates. Traffic can include packets in transit from host(s) H of network 101 protected by the TIG 102 to external hosts outside the network 101 (e.g., hosts in network 105 or hosts in another network connected to network 105) (or from external hosts to host(s) H). Packet filtering rules can include packet matching criteria having one or more pairs including a packet field name and a value based on one or more threat indicators. Examples of threat indicators can include, but are not limited to, IP address, IP address range, IP subnet address expressed in CIDR notation, a combination of L3 IP address and L4 port and L3 protocol type (e.g., a "5-tuple"), domain name or portions of a domain name, URI (e.g., URL or URN) or portions of a URI, certificate and / or an identifier of an associated certificate authority, geoIP information, and / or other types of threat indicators. A packet matching criteria value based on a threat indicator can include a value of the indicator (or a portion of the indicator value) and / or a value derived from the indicator value (or a portion thereof). A packet matching criteria can match a packet if the packet includes a value in one or more fields indicated by the packet matching criteria of a packet filtering rule that matches (either directly matches or matches after hashing or other processing) the value of the criteria, or includes a value within a range indicated by the value of the criteria. If a packet matches one or more packet matching criteria of a rule, the TIG 102 can perform one or more actions specified by the rule (e.g., block / drop / reject the packet, allow / forward / pass the packet, log the packet, capture the packet, redirect or re-route the packet, modify or transform the packet in some way to protect the network, generate and / or send a response to the packet, etc.).

[0060] As part of filtering packet traffic, and to determine whether a policy includes a rule with packet matching criteria that can match a packet, the TIG 102 can perform one or more of the methods described herein. These methods can include searching a set data structure to determine membership of a value matching a value based on a packet field in an associated index data structure of a policy rule. The value based on a packet field (value based on packet field or field-based value) can include the actual value (or a portion of the actual value) in the field and / or can include a value (e.g., a hashed or compressed data value) that has been derived from the actual value (or from a portion of the actual value) in the field. The methods performed by the TIG 102 can also or alternatively include searching the associated index data structure for a value matching the field-based value and, if a matching value is found, determining a rule identifier corresponding to the matching value. As also described herein, the search of the set data structure can be omitted and / or can occur prior to or concurrently with the search of the index data structure associated with the set data structure.

[0061] The TIG 102 can filter a large number of packets based on a large number of policies and rules. In practice, however, only a small fraction of the packets in a transmission that are filtered by the TIG 102 will match any CTI-derived packet filtering rules in the network security policies enforced by the TIG. Thus, a time- and space-efficient method for determining, prior to traversing the rules of a security policy for searching (or not searching) for a matching rule, whether a packet in a transmission will match (or not match) any rules in a network security policy can eliminate a large number of policy searches, which can reduce the average or expected time to filter a packet.

[0062] A set data structure that employs set operations that are time and / or space efficient for inserting elements into a set, deleting elements from a set, and testing membership or existence of elements in a set can support such an approach. Each CTI-derived packet filtering rule of a policy can be characterized by one or more threat indicators that make up the packet matching criteria of the rule. For each CTI-derived rule in a policy, the associated threat indicators (or values based on the threat indicators) can be inserted as elements into one or more set data structures. When filtering packets in transit, the TIG 102 can first perform a membership test by testing whether any of the values in the packet field values corresponding to the threat indicator values (e.g., IP address, 5-tuple, domain name, URI, certificate identifier, certificate authority, etc.) are members of or exist in the set data structure(s) before searching the policy to determine whether the packet matches any rule(s) in the policy. If the membership / existence test returns true, then the policy search can proceed on the associated index data structure; otherwise (i.e., the membership / existence test returns false), the policy search can be omitted and the TIG 102 can immediately send the packet to its destination.

[0063] If the membership / existence test is significantly faster on average / expected than a search that traverses the associated policy index data structure, then the expected policy search time per packet can be significantly reduced when averaged over a sufficiently large representative volume of packet traffic. Moreover, by initiating and performing the membership / existence test(s) concurrently with or otherwise in parallel with the policy search, rather than waiting for the membership / existence test to complete before initiating the policy search, the overall packet filtering time can be further reduced. If the membership test returns true, then the policy search will complete early (by some amount of time equivalent to the time to perform the membership test). If the membership test returns false, then the policy search can be abandoned or otherwise ignored (and, for example, the TIG 102 can immediately send the packet to its destination without waiting for the policy search to complete).

[0064] In addition to supporting fast / time-efficient membership / existence tests, the associated set data structure can also support spatial efficiency, e.g., the memory requirements for storing the elements of the set can be relatively small and can grow slowly, e.g., sub-linearly or by a small constant, when elements are added / inserted into the set. Data compression techniques can also be used to control the memory requirements, but often must be balanced against time efficiency. Moreover, the type of set data structure used for membership / existence tests can differ depending on the type of threat indicator. For example, for IPv4 addresses, which can be represented as 32 bits (and which allow for 2^32 possible IPv4 addresses), a bit array of length / size 2^32 bits (which requires 4 Gb or 512 MB of memory) can be a good choice for the set data structure, as it can support very fast element insertion, element deletion, and element membership testing. Each bit of such an array would correspond to a different one of the 2^32 possible IPv4 addresses, and can hold a value (e.g., 1) indicating that the corresponding IPv4 address is part of the set (e.g., a set of IPv4 addresses that are part of the matching criteria of a policy rule) or a value (e.g., 0) indicating that the corresponding IPv4 address is not part of the set. To insert an IPv4 address into the set (correspondingly, to delete an IPv4 address from the set), the bit value at the index or location of the array corresponding to the 32-bit representation of the IPv4 address can be set to 1 (correspondingly, 0). Similarly, membership testing of an IPv4 address / element can be reading the bit value at the index or location of the array corresponding to the 32-bit representation of the IPv4 address, and returning true if the bit value is 1 and false if the bit value is 0.

[0065] While the bit array set data structure for IPv4 addresses can be time and space efficient for related applications (e.g., TIG’s network security policy enforcement), it can not be for other types of threat indicators. For example, an IPv6 address with a 128-bit representation would correspondingly use a bit array of length / size 2^128 bits, which can be impractically large (and thus not space efficient). However, the bit array set data structure can still be effectively used for IPv6 addresses. For example, a bit array of length / size 2^32 bits (which requires 4 Gb or 512 MB of memory) can be allocated, and to insert an IPv6 address / element into the bit array / set, the bit value at the index or location of the array corresponding to the / 32 prefix bit representation of the IPv6 address (i.e., the most significant 32 bits of the IPv6 address) can be set to 1. Similarly, membership testing of an IPv6 address / element can be reading the bit value at the index or location of the array corresponding to the / 32 prefix representation of the IPv6 address, and returning true if the bit value is 1, and false if the bit value is 0. However, IPv6 addresses in such bit arrays can result in performance degradation, such as false positive rates of element membership searches returning non-zero, and inability to dynamically remove IPv6 addresses from the bit array.

[0066] It should be noted that the above adaptation of the element membership test for IPv6 address prefixes can be applied to IPv4 addresses to reduce memory requirements. For example, for IPv4 addresses, if the 2^32 bit array for the membership test of a full IPv4 address, which requires 4 Gb or 512 MB of memory, is deemed too large, a smaller array can instead be used by storing elements corresponding to IPv4 address prefixes. For example, a bit array of length / size 2^30 bits, which requires 1 Gb or 128 MB of memory, can be allocated, and to insert an IPv4 address / element into the bit array / set, the bit value at the index or location of the array corresponding to the / 30 prefix bit representation of the IPv4 address (i.e., the most significant 30 bits of the IPv4 address) can be set to 1. Similarly, the membership test of an IPv4 address / element can be to read the bit value at the index or location of the array corresponding to the / 30 prefix bit representation of the IPv4 address, and return true if the bit value is 1, and false if the bit value is 0. The IP address can be a full IP address or a partial IP address. A full IP address can be an IP address represented by all 32 bits (IPv4) or 128 bits (IPv6). In CIDR / prefix notation, a full IP address can be represented as a / 32 (IPv4) or / 128 (IPv6) address. A partial IP address can include an IP address prefix, which is an IP address represented by 32 bits or fewer (IPv4) or 128 bits or fewer (IPv6), where the missing bits are the least significant bits. An IP address prefix can be represented as / X, where X can be a number that is 32 or fewer (IPv4) or 128 or fewer (IPv6), and “X” refers to the X most significant bits (MSB) of the IP address. The IP address prefix can be associated with a subnet address.

[0067] For other types of threat indicators, such as domain names and URIs, bit array set data structures can also be inefficient and can not have a direct mapping from a threat indicator to a bit array index. Thus, other types of set data structures can be used to perform membership / existence tests. For example, the '417 patent describes methods, algorithms, and data structures that can be used to perform membership / existence tests on domain name and URI threat indicators (and IP address threat indicators and other types of threat indicators), such as efficient probabilistic set data structures (e.g., Bloom filters, cuckoo filters, etc.). For these probabilistic set data structures, time efficiency and space efficiency can be a function of the false positive rate (e.g., a false positive occurs when a membership / existence test returns true for an element that is not in the set). Generally, increasing the false positive rate increases the time efficiency and space efficiency of these probabilistic set data structures. However, each false positive can result in an unnecessary policy search that can not find a match between a packet and a packet filtering rule; thus, the false positive rate can be selected so that the time spent performing unnecessary policy searches does not increase the expected packet filtering time beyond what is required.

[0068] As noted above, there is also a need to improve the time efficiency and space efficiency of policy searches. For example, it can be impractical to meet some of the requirements for enforcing modern / future CTI-derived policies by significantly increasing CPU / processor speed and resources and / or by significantly increasing main memory. Failure to meet these requirements can result in a decrease in the effectiveness of network protection against global network threats or vulnerabilities.

[0069] While bloom filters, cuckoo filters, and the like can be used to significantly improve the time and space efficiency of policy searches, time and space efficient packet filtering methods based on probabilistic set data structures can be problematic in some cases, which can be related to the type of threat indicators. For example, probabilistic set data structures can be inefficient or impractical for handling IPv4 and IPv6 threat indicators represented in CIDR notation (e.g., subnet addresses). For another example, probabilistic set data structures can be partially inefficient in storing domain name threat indicators (and URI threat indicators, which can include domain name threat indicators), as they do not exploit redundancies or similarities that can occur over a set of domain name threat indicators, e.g., when the set has a relatively low information entropy; for example, they can exploit neither redundancies in effective top-level domains (eTLDs) across a set of domain name threat indicators, nor redundancies in commonly used prefix subdomain labels (e.g., “www,” “mail,” “remote,” “blog,” “webmail,” etc.). For yet another example, CTIs for domains can often be provided in the form of “base domain” or “registrable domain” or “eTLD+1” domain names, e.g., domain names that actually have a “starred” prefix subdomain that can not be a fully qualified domain name (FQDN). Base / registrable domain names can cause further inefficiencies when probabilistic set data structures are used in packet filtering methods. For yet another example, CTIs for URLs can often be set with partial paths or actually “starred” paths, i.e., a URL can have only a subset of path segments that make up the URL’s path. Moreover, the aforementioned membership / existence testing methods (wherein a packet’s network address(es) that can correspond to threat indicator(s) are first checked against the rules of a policy before a policy is searched for a matching rule) imply that each threat indicator can be stored redundantly in main memory, i.e., twice. These inefficiencies can result in time and / or space requirements for policy enforcement that cannot be met. For example, there can not be enough main memory to store a policy, or policy search times can be too long, or both.

[0070] Described herein are new, efficient index data structures and associated algorithms that can address the threat indicator processing inefficiencies and gaps associated with probabilistic set data structures, while meeting the time and space demands of CTI-derived policy enforcement strategies of TIGs. Generally, an index data structure can map keys to values. For example, in the relevant context of CTI-derived policy search and enforcement, the keys can be threat indicators (or values derived from threat indicators) associated with one or more packet filtering rules in a policy, and the values can be packet filtering rule identifiers. Then, for example, if a threat indicator is included in the packet matching criteria of each identified rule, a policy search for the potential threat indicator that traverses the index data structure can output one or more rule identifiers. If the output is empty (e.g., no rule identifiers), the potential threat indicator is not in the policy.

[0071] These new index data structures for efficient policy search can include: a “rule tree” that can be based on an adapted, extended, parameterized, and / or optimized prefix tree index data structure and adaptive radix tree index data structure for threat indicators, and that maps keys that are bit string representations of threat indicators to rule identifiers; and a flat hash map that maps keys that can be hashed or compressed threat indicators to rule identifiers.

[0072] One problem with matching threat indicators of packet filtering rules to domain names and URIs contained in packets (in a transmission) is that: (1) domain name threat indicators can be provided in the form of base / registrable domain names or "starred" domain names (e.g., domain names for which there are no explicit entries in DNS, such as *.example.com), which can never be observed in packets, and (2) similarly, URL threat indicators can be provided in the form of "starred" paths, e.g., only a portion of the path segments that make up a complete path can be represented in a URL indicator (e.g., http: / / www.example.com / path-segment-1 / *), which can also never be observed in packets. However, in both cases, domain names and URLs contained in packets in a transmission that match these starred indicator portions can be considered to match the corresponding packet filtering rules. For example, a (fully qualified) domain name "www.example.com" that can be observed in a packet in a transmission can match a packet filtering rule with a matching criterion of "example.com". Similarly, a complete or full URL http: / / www.example.com / path-segment-1 / path-segment-2 / path-segment-3 that can be observed in a packet in a transmission can match a packet filtering rule with a matching criterion of http: / / www.example.com / path-segment-1 / .

[0073] Thus, for both domain names and URLs, domain name and URL type threat indicators can generally be considered to be prefixes of domain names and URLs that can be contained in packets (in a transmission). In packet filtering applications such as network security policy enforcement, a match between a domain name and URL threat indicator that makes up a matching criterion of a packet filtering rule and a corresponding value contained in a packet (in a transmission) can be determined if the matching criterion can be a prefix of the value contained in the packet. Similarly, IP threat indicators are generally provided in the form of prefixes, e.g., in CIDR notation, and similarly a match between an IP threat indicator that makes up a matching criterion of a packet filtering rule and a corresponding value contained in a packet (in a transmission) can be determined if the matching criterion can be a prefix of the value contained in the packet.

[0074] Another issue with domain name and URI threat indicators is that their memory / space requirements, when stored in their native character / text representation (e.g., as used in the packets) in the data structure, can be high compared to the typical TIG main memory budget for cyber-security policy enforcement. For example, at the time of the present disclosure, a CTI provider can be providing a total of tens of millions of domain name indicators (e.g., 50 million) and hundreds of millions of URI indicators (e.g., 200 million), which in their unprocessed form can require more than 20 GB of main memory / space; whereas the typical TIG main memory budget for cyber-security policy enforcement can be 5-10 GB.

[0075] One or more of these issues can be addressed by using a flat hash map index data structure to search for policy rule IDs. The flat hash map index data structure can map hashes of threat indicators (e.g., domain names and URIs) and / or portions of threat indicators to rule IDs. Additionally or alternatively, the flat hash map index data structure can map compressions (e.g., lossless compressions) of threat indicators (e.g., domain names and URIs) and / or portions of threat indicators to rule IDs. As described below, the structure of domain names and URIs can be exploited in conjunction with hash and / or lossless compression methods to support efficient storage and efficient searching of these types of threat indicators. The flat hash map index data structure can reduce main memory requirements substantially within the typical TIG budget, while supporting efficient matching between partial domain name indicators and partial URI indicators in the CTI and full domain names and URIs observed in the packets in transit. The flat hash map index data structure simultaneously exploits the structure of domain names and URIs and the data compression effects of (lossy) hash and / or lossless compression methods.

[0076] In an example flat hash map, each key / indicator can be represented as, for example, a 64-bit (8-byte) hash or fingerprint, and mapped into a hash table of size corresponding to the number of keys. Given that at the time of the present disclosure, domain names in the CTI are typically 7-27 octets / bytes long, up to 255 octets / bytes, and URIs in the CTI are typically 35-175 octets / bytes long, up to 2048 octets / bytes, for example, a 64-bit / 8-byte representation can result in a significant reduction in memory / space requirements for domain names and URIs in the CTI. Note, however, that the collision rate of a hash is inversely proportional to the size of the fingerprint. Thus, for example, while a 32-bit fingerprint size can reduce memory requirements to about half of what is required for a 64-bit fingerprint, in practice the collision rate can be too high. Methods for managing / balancing the collision rate and the fingerprint size (and thus the memory requirements) are described below.

[0077] An exemplary technique for creating a domain name fingerprint may be referred to as "incremental tag hashing"; similarly, an exemplary technique for creating a URL fingerprint may be referred to as "incremental path segment hashing".

[0078] Incremental label hashing can be used to create fingerprints of domain names. A domain name can be represented as a sequence of subdomain labels separated by the dot "." character, where the rightmost label represents the top-level subdomain relative to the (implicit) root domain (e.g., "com", "net", "edu", "org", etc.). For example, let domain name D... i Represented as i subdomain labels L i L i-1 ...a list of L1, where each label is separated by a dot "." character, and set... H This is a hash function that outputs an X-bit hash (e.g., a 64-bit hash) from an input of arbitrary size. Examples of such hash functions commonly available in the standard library include, but are not limited to, xxh3, murmur2, abseil, and anker1. Let "||" be the concatenation operator. It is used to concatenate the domain name D. i The process of creating a fingerprint can be recursively defined as follows: H (D1) = H (L1), and H (D i ) = H ( H (L i ) || H (D i-1 )), This generates domain name D. i X-position fingerprint H (D i When creating a fingerprint, wildcards in the threat indicators (e.g., "*" in "*.example.com") (if present) can be omitted. It should be noted that domain name D... i It starts from the rightmost or top-level subdomain tag and processes each subdomain tag one by one. For example, the domain name CBA can be processed as... H (CBA) = H ( H (C) || H ( H (B) || H (A))).

[0079] Because domain names are processed from rightmost / top-level subdomain label to left, one subdomain label at a time, fully qualified domain names (FQDNs) contained in packets during enforcement of CTI-derived policies on packet traffic can be easily and efficiently matched against portions or base / registerable domain names that can form policy rules in CTI. For example, assume that the domain name C.B.A (but not the domain names E.D.C.B.A, D.C.B.A, B.A, or A) is in CTI and is associated with a packet filter rule that includes the rule ID R k in the policy. When the policy is created and processed for efficient enforcement, the fingerprint H (CBA) key and rule ID R k value can be inserted into the flat hash map index data structure F. Assume that the packet (in transit) contains the domain name E.D.C.B.A. The TIG policy enforcement logic can efficiently check whether E.D.C.B.A matches one or more packet filter rules in the policy using the following example policy search algorithm (in pseudo code): Step 1: Compute H (A); Search(F, H (A)) = Null; Step 2: Compute H (B); Compute H ( H (B) || H (A)) = H (BA); Search(F, H (BA)) = Null; Step 3: Compute H (C); Compute H ( H (C) || H (BA)) = H (CBA); Search(F, H (CBA)) = {R k}; Step 4: Compute H (D); Compute H ( H (D) || H (CBA)) = H (DCBA); Search(F, H (DCBA) = Null; Step 5: Compute H (E); Compute H ( H (E) || H (DCBA)) = H (EDCBA); Search(F,H (EDCBA) = Null; It should be noted that in each of the above steps, since the hash function processes each subdomain label individually, H At most two calculations are performed. Therefore, the algorithm is efficient in terms of both time and memory / space, as the time complexity is a linear function of the number of subdomain labels and the space complexity is a constant function of the hash size X. It should also be noted that the corresponding algorithm for inserting domain name indicators into a flat hash map index data structure has similar time and space complexity. Furthermore, it should be noted that variations of the fingerprint creation process can be used if the hash of each individual label is not required. H (D1) = H (L1), and H (D i ) = H (L i || H (D i-1 )), This results in the hash function being calculated only once for each label in the corresponding strategy search algorithm. H .

[0080] Incremental path segment hashes can be used to create fingerprints of URLs. URLs in CTIs can typically have the form <protocol>: / / <host>><path>><query>, where <protocol> can typically be "http" or "https" (at the time of this disclosure, URLs in CTIs primarily use the http or https protocol), and <host> can be a hostname (e.g., domain name D). i ) or IP address, <path>P j It can be j The sequence of path segments S1S2S3…S j Each path segment S i The query string can begin with a forward slash " / " character and can be a string consisting of a sequence of pairs of parameter names and associated values. In the context of CTI applications (such as network security policy enforcement), the query string can often be omitted; however, if applicable, the query string can be processed in a manner similar to tags or path segments. For example, the query string can be incrementally processed in pairs of {parameter name, value}. (For URL U) ij = D i P j The process of creating a fingerprint for the <host> and <path> sections is similar to the incremental tag hashing process described above for creating fingerprints for domain names. H (U ij ) can be calculated as, for example, as H (Uij ) = H ( H (D i ) || H (P j )) or similar form, in which, H (D i ) can be calculated as described above, and where, H (P j Recursively defined as H (P1) = H (S1), and H (P j ) = H ( H (S j ) || H (P j-1 )), This generates URL U ij X-position fingerprint H (U ij It should be noted that path P j It starts from the leftmost path segment and processes each path segment one by one. Furthermore, it's similar to the fingerprinting process for domain names. H (P j Variations of the calculation of ) can be H (P1) = H (S1), and H (P j ) = H (S j || H (P j-1 )), This results in the hash function being calculated only once for each path segment in the corresponding strategy search algorithm. H .

[0081] For the purpose of generating hashes for rule IDs mapped into the flat hash map index data structure, and for the purpose of incrementally hashing search objects to search the flat hash map index data structure, domain names (or the <host> portion of a domain name-formatted URL) can be processed from right to left (e.g., from the highest-level domain (or TLD) to the lowest-level subdomain). Conversely, for the purpose of generating hashes for rule IDs mapped into the flat hash map index data structure, and for the purpose of incrementally hashing search objects to search the flat hash map index data structure, the path portion of the URL can be processed from left to right. More generally, and for URL Uij = D i Pj , where D i = L i .L i-1 .L... L3.L2.L1 and P j = S1 / S2 / ... / S j , the domain name label L and the path segment S can be processed in the following order: L1.L2.L3... L i-1 .L i / S1 / S2 / ... / S j The hash value of the above URL can be incrementally generated as follows, where "K c " is the key generated by incremental hashing, and c is a counter whose final value is equal to the total number of the host name label L and the path segment S of the URL (i.e., i + j): If c = 1, then K c = H (L1), If 1 < c ≤ i, then K c = H (L c || K c-1 ), or If i < c ≤ j, then K c = H (S c-i || K c-1 ) For example, for the URL "http: / / www.example.com / pathseg-1 / pathseg-2 / pathseg-3", i = 3 and j = 3, and the hash-based keys K1 to K6 can be incrementally generated as follows: K1 = H (com) K2 = H (example || K1) = H (example || H ​​​​​​​​​​​​​​​​​​​​(example || H (com)))) K5= H (pathseg-2 || K4) = H (pathseg-2 || H (pathseg-1 || H (www || H (example || H (com))))) K6= H (pathseg-3 || K5) = H (pathseg-3 || H (pathseg-2 || H (pathseg-1 || H (www || H (example || H (com)))))) Using the incremental hashing described above allows a single flat hash map index data structure to be used for both domain-based CTI rules (e.g., rules where the matching criterion is the domain name) and URL-based CTI rules (e.g., rules where the matching criterion is the <host> and <path> parts of the URL). This allows for simplified search logic (e.g., using the same search logic for different types of threat indicators) and reduced memory usage (e.g., by using a single index data structure for rules with different types of matching criteria). Calculate URL U ij fingerprints H (U ij An efficient method could be: (1) Configure a single instance of a flat hash map to store and process all relevant domain name CTIs (and associated policy rules) and all relevant URL CTIs (and associated policy rules), and (2) Using the hash-based key K mentioned above c The formula for calculating domain name D i and URL D i P j The fingerprint. For example, added to domain D. i = L i .L i-1 The fingerprint of a single flat hash map of L3, L2, and L1 will be... H (L i || H (L i-1 || … || H (L3|| H(L2 H (L1)))…)) and added to the URL U ij = D i P j = L i .L i-1 . …L3.L2.L1 / S1 / S2 / … / S j-1 / S j The fingerprint of the single flat hash map of L3.L2.L1 / S1 / S2 / … / S H (S j || H (S j-1 … || H (S2|| H (S1|| H (L i || H (L i-1 || … || H (L3|| H (L2|| H (L1)))…))))…))).

[0082] Similar to the case described above for domain names, because the <path> part of the URL is processed from left-most path segment to right-most path segment, the full URL contained in a packet during enforcement of CTI-derived policies on packet traffic in transit can be easily and efficiently matched against URLs in the CTI that have partial paths that constitute the policy rules. For example, a URL contained in a packet in transit with <host> E.D.C.B.A and <path> / S1 / S2 / S3 can match a packet filtering rule with matching criteria of a CTI URL with the same <host> E.D.C.B.A but with a (partial) <path> / S1 / S2.

[0083] It should be noted that the <host> part of the URL can also be an IP address (IPv4 or IPv6). In this case, the IP address can be incrementally hashed in a similar fashion to the incremental label hashing of domain names, but from left to right from the IP address. For example, an IPv4 address can be expressed in dotted decimal notation, which can be a sequence of four (4) integers N, each in the range of [0, 255] and separated by a dot “.” character. Thus, when the <host> part of the URL is an IPv4 address, the IPv4 address can be processed similarly to a domain name, where the four (4) integers correspond to four subdomain labels, separated by a dot “.” character, and the four integers are numbered from left to right (e.g., N1.N2.N3.N4). In the example above, the hash-based key K cin the formula above, and when incrementally hashing a URL having an IPv6 address as the <host> part, i = 8, and L1 through L8 will become H1 through H8, respectively. The formula above can be modified to accommodate IPv6 addresses as follows: c i

[0084] It should be noted that URLs in the form that can exist in CTI and be used for, e.g., email addresses, VoIP addresses, message addresses, etc., can be similarly handled using the techniques described above. Such URLs can have the form of <protocol>:<name>@<host>, where <protocol> can be, e.g., “mailto”, “sip”, “sips”, “h323”, “im”, “xmpp”, etc., <name> can be, e.g., a typical username, and <host> can be a host name (e.g., a domain name D i

[0085] When handling CTI-derived policies for efficient enforcement, the domain names and URLs of the packet filtering rules included by the policies can be fingerprinted as described above, the fingerprints (keys) can be paired with their associated rule identifiers / rule IDs (values), and the {fingerprint, rule ID} pairs can be inserted into a hash table, which can be a flat hash map index data structure. However, flat hash maps can not inherently support efficient dynamic runtime updates to the packet filtering rules of the policies currently enforced by the TIG (see description below). If dynamic runtime policy updates are required, an alternative index data structure that inherently supports dynamic runtime updates, such as a rule tree (described below), can be used instead. In this case, the {fingerprint, rule ID} pairs are inserted into a rule tree instead of a hash table. Such an index data structure (storing a rule tree as a hash-based fingerprinted key in its bit string representation) can be referred to as a fingerprinted rule tree. It should be noted that a fingerprinted rule tree can have different memory requirements than a corresponding flat hash map, with the tradeoff being support for dynamic runtime updates.

[0086] ​In some cases, hash-based fingerprints and associated index data structures can have issues associated with hash collisions. For example, as more and more fingerprints are populated in the index data structure, the likelihood of a hash collision between a fingerprint that has already been included in the index data structure H (A) and the fingerprint that is being inserted or searched for H (B) (where A and B can be domain names or URLs) increases, i.e., the hash collision rate can increase. Because hash functions can be lossy and can be non-invertible, it can be difficult or impossible to determine whether A is equal to B or whether A is not equal to B. The hash collision rate can be reduced, for example, by: (1) limiting the number of entries / keys / elements in the associated index data structure; and / or (2) using a perfect (zero collision) hash function; and / or (3) partitioning the set of domain name and URL threat indicators / elements and inserting the elements in each partition into different instances of the index data structure; and / or (4) increasing the hash size; and / or (5) using multiple different hash functions or multiple different fingerprints for each key, combined with logic that reduces the effective collision rate. However, it should be noted that these exemplary approaches can have tradeoffs. For example, (1) and (2) can not be practical for some applications, (3) can increase the time requirements, memory requirements, and logic complexity, and (4) and (5) can increase the memory requirements. On the other hand, these techniques can be used to reduce the effective collision rate so that real collisions (i.e., when H (A) = H (B) but A is not equal to B) can never occur or can rarely occur in practice, but at the same time still meet the time requirements and memory / space requirements.

[0087] In the context of this disclosure, for X-bit fingerprints, if multiple different hash functions are used to generate X-bit fingerprints, the effective collision rate can be lower than if a single hash function is used to generate X-bit fingerprints. This can be due to a variety of factors, but a primary factor is that for time performance reasons, a non-cryptographic hash function (NCHF) can be used to compute the fingerprints, rather than a cryptographic hash function (CHF), as CHFs are relatively slower compared to NCHFs, and this relative slowness can have an adverse impact on performance in the context of at least some applications described herein. If a (strong) CHF is available, an effective way to reliably reduce the collision rate is to use a single CHF but increase the hash size (and the size of the fingerprints), as described in (4) above. However, the estimated collision rate for NCHFs can be less reliable compared to strong CHFs, and can depend on the data being hashed and vary with it. Furthermore, different NCHFs can have different data dependencies and variances. In addition, NCHFs available in standard libraries are typically optimized for a particular hash size, and thus, for example, an NCHF optimized for computing 64-bit output hashes can have reduced performance (e.g., higher collision rate) when producing hashes of a size different from 64 bits. To reduce the adverse impact of these various factors on the collision rate, multiple different NCHFs can be selected to compute X-bit fingerprints, where each different NCHF can be optimized for a particular size of output, where the outputs of the NCHFs can be concatenated to form the fingerprint, and where the sum of the output sizes totals X. In this way, the effective collision rate of the multiple NCHFs can be reduced compared to the collision rate of any single NCHF when computing fingerprints of size X.

[0088] The following is an example of how the effective collision rate can be reduced by using multiple non-cryptographic hash functions. Let H 1 be a first non-cryptographic hash function, let H 2 be a second non-cryptographic hash function different from H 1, and let H 3 be a third non-cryptographic hash function different from H 1 and H 2. For a URL Uij = D i P j = S1 / S2 / … / S i ), where D i = L i-1 .L j .L. … L3.L2.L1 and P j = S1 / S2 / … / S c , where c is a counter whose final value is equal to the total number of hostname labels L and path segments S of the URL (i.e., c = i + j), where K c is the cth key, where K_1 H1 the cth component of the generated key, where K_2 c is generated using H 2 the cth component of the generated key, and where K_3 c is generated using H 3 the cth component of the generated key, K c can be generated as follows: K c = K_3 c || K_2 c || K_1 c where, if c = 1, then K_3 c = H 3(L1), K_2 c = H 2(L1), and K_1 c = H 1(L1), if 1 < c ≤ i, then K_3 c = H 3(L c || K_3 c-1 ), K_2 c = H 2(L c || K_2 c-1 ), and K_1 c = H 1(L c || K_1 c-1 ), or if i < c ≤ j, then K_3 c = H 3(S c-i || K_3 c-1 ), K_2 c = H 2(S c-i || K_2 c-1 ), and K_1 c = H 1(S c-i || K_1 c-1 ) if H 1 is a hash function that generates an 8-byte (64-bit) hash, if H 2 is a hash function that generates a 3-byte (24-bit) hash, and if H3 is a hash function that generates a 3-byte (24-bit) hash, then each key K generated using the above procedure will be 14 bytes (112 bits).

[0089] The above example of incremental hashing using three non-cryptographic hash functions is just one example of how multiple hash functions can be used. Fewer hash functions can be used (e.g., K c = K_2 c || K_1 c ) or more hash functions can be used (e.g., K c = K_4 c || K_3 c || K_2 c || K_1 c ). Non-cryptographic hash functions can be used with one or more cryptographic hash functions. And while cryptographic hash functions can not be preferred in some applications, cryptographic hash functions can also be used (e.g., without non-cryptographic hash functions). Moreover, multiple hash functions can be combined for incremental hashing in ways other than as described above. Such other combinations can produce different outputs for a given input, but for a large number of threat indicators (e.g., hundreds of millions of unique threat indicators), the same utility and effect (e.g., reduced collision rates) as the above combinations of H 1, H 2, and H 3 can be achieved. For example, using the above non-cryptographic hash functions H 1, H 2, and H 3, and for a URL Uij as described above, K c can be generated as follows: K c = K_3 c || K_2 c || K_1 c where, if c = 1, then K_3 c = H 3(L1), K_2 c = H 2(L1), and K_1 c = H 1(L1), if 1 < c ≤ i, then K_3 c = H 3(L c || K c-1 ), K_2 c = H2(L c || K c-1 ), and K_1 c = H 1(L c || K c-1 ), or K_3 c = H 3(S c-i || K c-1 ), K_2 c = H 2(S c-i || K c-1 ), and K_1 c = H 1(S c-i || K c-1 ) Efficient lossless compression methods can be used in combination with or as an alternative to the above-described hash function-based methods. Lossless compression can mitigate or eliminate uncertainties such as hash collisions, while maintaining time and space / memory efficiency. Efficient lossless compression functions C can encode and process domain names and URLs in combination with or as an alternative to the above-described hash function-based methods, which can be considered lossy compression methods. For example, CHuffman encoding (a type of prefix encoding method), which is a lossless compression method that provides temporal efficiency and spatial / memory efficiency and can avoid collision issues associated with hash functions. Other lossless compression methods can be, for example, arithmetic encoding and asymmetric numeral systems. It should be noted that lossless compression functions are typically invertible functions, and this can be assumed in the present disclosure unless otherwise noted. These lossless compression methods and associated compression functions can be characterized by their compression ratio. The compression ratio for a given compression function can be the expected value of the compression function output size divided by the compression function input size. The compression ratio can vary with the information entropy of the input, i.e., the lower the information entropy of the input, the lower (and thus better) the compression ratio. Information entropy can correspond to the amount of redundant information in the input. For example, at the present time of the present disclosure, typical public domain names in the CTI are composed of eTLDs, which are formed from a relatively small set of labels (e.g., a few thousand elements) (such as “com,” “net”) and country codes (such as “ru” and “cn”). Furthermore, at the time of the present disclosure, it can be that approximately 35% of eTLDs in domain names in the CTI contain the “com” label. Lossless compression methods can exploit this redundancy to achieve good compression (i.e., a relatively low compression ratio). For example, Huffman encoding and other encoding methods exploit redundancy and low information entropy to achieve better compression factors (e.g., because it is typically less data to represent an information element of size X and N-1 repetitions of that element than it can take to represent N different elements of size X). Additionally, at the present time of the present disclosure, there are approximately 10,000 eTLDs in the public suffix list (PSL). The enumeration of these eTLDs in the PSL and the associated 1-1 mapping between numbers and eTLDs can be encoded in 14 bits. Thus, significant (lossless) compression can be achieved by representing the eTLD of any domain name with its 14-bit enumeration. In practice, 16 bits or 2 bytes can be used to account for future growth of the PSL (up to 64K eTLDs) and to align with byte-sized logic.

[0090] Exemplary techniques for creating an efficient compressed representation or compression of domain names can be referred to as “incremental label compression”; similarly, exemplary techniques for creating an efficient compression of URLs can be referred to as “incremental path segment compression.”

[0091] Incremental label compression can be used to create an efficient compressed representation or compression of domain names. This technique is similar in approach to incremental label hashing (as described above), but uses a (lossless) compression function C instead of a hash function HAs mentioned above, a domain name can be represented as a sequence of subdomain tags separated by the dot "." character, where the rightmost tag represents a top-level subdomain relative to the (implicit) root domain (e.g., "com", "net", "edu", "org", etc.). For example, let the domain name D... i Represented as i subdomain labels L i L i-1 ...a list of L1, where each label is separated by a dot "." character, let... C This is a compression function with an efficient encoded version of the input and output, and let "||" be the concatenation operator. It is used to encode inputs and outputs... i Domain D of the tag i Create a "compression" C (D i The process can be recursively defined as follows: C (D1) = C (L1), and C (D i ) = C (L i ) || C (D i-1 ), This generates domain name D. i compression C (D i It should be noted that the domain name D i It starts from the rightmost or top-level subdomain tag and processes each subdomain tag one by one. For example, the domain name CBA can be processed as... C (CBA) = C (C) || C (B) || C (A)

[0092] Because domain names are processed tag by tag, starting from the rightmost / top-level subdomain, during the enforcement of CTI derivation policies on in-transit packet traffic, the fully qualified domain name (FQDN) contained in the packet can be easily and efficiently matched with a portion or base / registerable domain name constituting the policy rule in the CTI. For example, suppose the domain name CBA (but excluding domain names EDCBA, DCBA, BA, or A) is in the CTI, and the rule constituting the policy is ID R k This is associated with packet filtering rules. Compression is performed when policies are created and processed for efficient enforcement. C (CBA) Key and Rule ID R kValues ​​can be inserted into a flat hash map index data structure F. Assume a packet in transit contains the domain name EDCBA. The TIG policy enforcement logic can efficiently check whether EDCBA matches one or more packet filtering rules in the policy using the following exemplary policy search algorithm (represented in pseudocode): Step 1: Calculation C (A); Search (F, C (A)) = Null; Step 2: Calculation C (BA); Search (F, C (BA)) = Null; Step 3: Calculation C (CBA); Search (F, C (CBA)) = {R k}; Step 4: Calculation C (DCBA); Search (F, C (DCBA) = Nul; Step 5: Calculation C (EDBCA); Search (F, C (EDCBA) = Null.

[0093] It should be noted that in each of the above steps, since the subdomain tags are processed one by one, the compression function... C Each subdomain label is computed only once. Therefore, the algorithm is efficient in both time and space / memory, as the time complexity is a linear function of the number of subdomain labels, and the space complexity is a function of the compression ratio (which can be less than 1). It should also be noted that the corresponding algorithm for inserting domain indices into a flat hash map index data structure can have similar time and space complexities.

[0094] Incremental path segment compression can be used to create efficient URL compression. This technique is methodologically similar to incremental path segment hashing (as described above), but uses a (lossless) compression function. C Replaced hash function H A URL can have the following form: <protocol>: / / <host>><path>><query>, where <protocol> can be "http" or "https", and <host> can be the hostname (e.g., domain name D). i = or IP address, <path>P j It can be j Path segments S1S2S3…S j The sequence, where each path segment S iA query can begin with a forward slash " / " character, and the <query> can be a string that includes parameter names and associated values. In the context of a CTI application such as a network security policy enforcement, the <query> can typically be ignored; however, if applicable, the query string can be processed in a similar manner to a label or path segment. For the URL U ij = D i P j The process of creating a compression for the <host><path> portion of URL U C (U ij ) = C (D i ) || C (P j ), where C (D i ) can be computed as above, and where C (P j ) is recursively defined as C (P1) = C (S1), and C (P j ) = C (S j ) || C (P j-1 ), This yields a compression C (U ij ) for URL U ij . It should be noted that the path P j is processed from leftmost path segment to rightmost path segment.

[0095] Similar to the case described above for domain names, because the <path> portion of a URL is processed from leftmost path segment to rightmost path segment, a complete URL contained in a packet during enforcement of CTI-derived policies on packet traffic in transit can be easily and efficiently matched against CTI URLs having partial paths that constitute policy rules. For example, a URL contained in a packet in transit having a <host> of E.D.C.B.A and a <path> of / S1 / S2 / S3 can match a packet filtering rule having a match criterion of a CTI URL having the same <host> of E.D.C.B.A but having a (partial) <path> of / S1 / S2.

[0096] For the purpose of generating a compression for a rule ID that maps into a flat hash map index data structure, and for the purpose of incrementally compressing a search object to search the flat hash map index data structure, a domain name (or the <host> part of a domain name formatted URL) can be processed from right to left, and the path part of a URL can be processed from left to right. The URL Uij = D i P j (where D i = L i .L i-1 .L. … L3.L2.L1 and P j = S1 / S2 / … / S j ) can be incrementally generated as follows, where "K c " is the key generated by incremental compression, and c is a counter whose final value is equal to the total number of host name labels L and path segments S of the URL (i.e., c = i + j): If c = 1, then K c = C (L1), If 1 < c ≤ i, then K c = C (L c ) || K c-1 , or If i < c ≤ j, then K c = C (S c-i ) || K c-1 For example, for the URL "http: / / www.example.com / pathseg-1 / pathseg-2 / pathseg-3", the compression based keys K1 through K6 can be incrementally generated as follows: K1= C (com) K2= C (example) || K1= C (example) || C (com) K3= C (www) || K2= C (www) || C (example) || C (com) K4= C (pathseg-1) || K3= C (pathseg-1) || C (www) ||C (example) || C (com) K5= C (pathseg-2) || K4= C (pathseg-2) || C (pathseg-1) || C (www) || C (example) || C (com) K6= C (pathseg-3) || K5= C (pathseg-3) || C (pathseg-2) || C (pathseg-1) || C (www) || C (example) || C (com) Using the incremental compression described above allows a single flat hash map index data structure to be used for both domain-based CTI rules (e.g., rules where the matching criterion is the domain name) and URL-based CTI rules (e.g., rules where the matching criterion is the <host> and <path> parts of the URL). This allows for simplified search logic (e.g., using the same search logic for different types of threat indicators) and reduced memory usage (e.g., by using a single index data structure for rules with different types of matching criteria). Calculate URL U ij compression C (U ij An efficient method could be: (1) Configure a single instance of a flat hash map to store and process all relevant domain name CTIs (and associated policy rules) and all relevant URL CTIs (and associated policy rules), and (2) Using the above compression-based key K c The formula for calculating domain name D i and URL D i P j Compression. For example, adding to domain D. i = L i .L i-1 …The compression of a single flat hash map of L3, L2, and L1 will be… C (L i ) || C (L i-1 )|| … || C (L3||C (L2 C ij = D i P j = L i .L i-1 . …L3.L2.L1 / S1 / S2 / … / S j-1 / S j The compression of the single flat hash map of C (P j ) || C (P j-1 ) || … || C (P2) || C (P1) || C (L i )|| C (L i-1 )|| … || C (L3) || C (L2) || C (L1).

[0097] It should be noted that the <host> part of the URL can also be an IP address (IPv4 or IPv6). In this case, the IP address can be incrementally compressed in a similar way to the incremental label compression of domain names, but from left to right from the IP address. An IPv4 address can be expressed in dotted decimal notation, which can be a sequence of four (4) integers N, each of which is in the range [0, 255], and each of which is separated by a dot “.” character. Thus, when the <host> part of the URL is an IPv4 address, the IPv4 address can be treated similarly to a domain name, where the four (4) integers correspond to four subdomain labels, separated by dot “.” characters, and the four integers are numbered from left to right (e.g., N1.N2.N3.N4). In the formula of the compression-based key K c above, and when incrementally compressing a URL with an IPv4 address as the <host> part, i = 4, and L1 to L4 will become N1 to N4, respectively. An IPv6 address can be expressed as eight (8) groups H, where each group has four (4) hexadecimal digits, and where each group is separated by a colon “:” character. Thus, when the <host> part of the URL is an IPv6 address, the IPv6 address can be treated similarly to a domain name, where the eight (8) groups correspond to eight subdomain labels, separated by colon “:” characters instead of “.” characters, and the eight groups are numbered from left to right (e.g., H1:H2:H3:H4:H5:H6:H7:H8). In the formula of the compression-based key K c ​in the formula above, and when incrementally compressing URLs having IPv6 addresses as the <host> part, i = 8, and L1 through L8 will become H1 through H8, respectively.

[0098] Similar to the above, when processing CTI-derived policies for efficient enforcement, the domain names and URLs of the packet filter rules included in the policies can be compressed as described above, the compression (key) can be paired with the rule identifier / rule ID (value) to which it relates, and the {compression, rule ID} pair can be inserted into a table, which is a flat hash map index data structure. However, flat hash maps can not inherently support efficient dynamic runtime updates to the packet filter rules of the policies currently enforced by the TIG (see description below). In this context, dynamic runtime updates mean updating the match criteria of a packet filter rule by inserting or deleting a threat indicator from the set of indicators that make up the match criteria while enforcing the associated policy on packets in transit. If dynamic runtime policy updates are required, an alternative index data structure that inherently supports dynamic runtime updates, such as a rule tree, can be used instead. In this case, the {compression, rule ID} pair is inserted into a rule tree instead of a hash table. Such an index data structure (stored as a rule tree as a compressed key in its bit string representation) can be referred to as a compressed rule tree. It should be noted that compressed rule trees can require more memory than the corresponding flat hash map, with the tradeoff being support for dynamic runtime updates.

[0099] In some cases, a combination of lossy hashing and lossless compression methods can be effective. For example, because the redundancy measure of domain names can generally be greater than the redundancy measure of paths, a URL can be processed by using a (lossless) compression function on the hostname (i.e., domain name) part of the URL and a (lossy) hashing function on the path part of the URL.

[0100] FIG. 2A is a flowchart illustrating an example method for searching a flat hash map index data structure based on a search object (i.e., a value obtained from a field of a packet in transit), the flat hash map index data structure including {fingerprint, rule ID} index pairs of a policy (or a portion of a policy). For each of the index pairs of the flat hash map index data structure, the rule ID can include an identifier of a policy rule, and the fingerprint can include a value that has been generated based on the match criteria (e.g., threat indicators) of the policy rule using an incremental hash as described above. There can be multiple index pairs that include the same fingerprint (e.g., there can be multiple rules that include the same match criteria). While FIG. 2A The example uses domain names / subdomain names as an example of the type of policy rule match criteria and packet field values for which the method is performed, but FIG. 2AThe method of can also or alternatively be performed in relation to other types of matching criteria and packet field values. FIG. 2A The method of can be performed, for example, by the TIG 102. FIG. 2A One or more steps of the example method of can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps can be added.

[0101] In step 201, a search object in the form of a domain name can be received (e.g., from a process that resolves packet fields specified by the policy rule criteria and extracts data from those fields). In step 202, one or more set data structures are searched to determine membership of the domain name (and / or subdomain names of that domain name) received in step 201 in a set of domain names and subdomain names that match the criteria of a policy rule corresponding to the index pair of the flat hash map index data structure. Step 202 can include using one or more Bloom filters, cuckoo filters, or other types of probabilistic set data structures and associated methods described in the '417 patent. In addition or alternatively, step 202 can include using non-probabilistic set data structures to test for membership. If the TIG 102 determines non-membership in step 202 (i.e., the search object is not in the set of search objects that would match the matching criteria of a policy rule corresponding to the index pair of the flat hash map index data structure), the TIG 102 can skip searching the flat hash map data structure and can apply the rule(s) (if any) that can be determined using other values from the packet in transit. As described in more detail in connection with the '417 patent, the TIG 102 can search multiple sets of policy rules concurrently by searching multiple index data structures based on multiple different values from the packet. FIG. 7 As described in more detail, the TIG 102 can search multiple sets of policy rules concurrently by searching multiple index data structures based on multiple different values from the packet.

[0102] If the TIG 102 determines membership in step 202 (e.g., the search object or a portion of the search object is in the set of matching criteria of a policy rule corresponding to the index pair of the flat hash map index data structure), the TIG 102 can initialize a counter c to 1 in step 203. In step 204, the TIG 102 can compute a hash-based key K c based on the cthlabel L c of the domain name search object. For example, if c = 1, K c may be computed as H (L1), and if 1 < c < i, K c may be computed as H (L c || K c-1 ). In step 205, the TIG 102 can search the flat hash map index data structure (FHM) for a value corresponding to Kc Each of the keys / fingerprints in the flat hash map index data structure has been computed using an incremental hash generation method including one or more steps (e.g., step 204) from matching criteria (e.g., threat indicators) of policy rules corresponding to the rule IDs to which the keys / fingerprints are mapped in the flat hash map index data structure. If one or more matching fingerprints are found, then in step 206, the TIG 102 can store the value of the rule ID of each index pair having a matching fingerprint in a result list data structure. As used herein, a "result list" broadly refers to a data structure that can store multiple values (e.g., rule IDs); a result list data structure does not necessarily take the form of an actual list.

[0103] Additionally or alternatively, FIG. 2A Step 204 of the method of FIG. 1 can include using a different procedure to compute the keys, and step 205 can include searching the flat hash map index data structure including keys / fingerprints that are also computed using the different procedure. For example, the key computation in step 204 and the key / fingerprint computation of the flat hash map index data structure searched in step 205 can be based on a hash that is based on a hash of a plurality of individual tags (e.g., if c = 1, then K c H (L1), if if 1 < c < i, then K c H H (L c ) || K c-1 ). As another example, the key computation in step 204 and the key / fingerprint computation of the flat hash map index data structure searched in step 205 can be based on a plurality of hash functions (e.g., any of the procedures using hash functions H 1, H 2, and H 3, as described above).

[0104] After step 206, or after determining in step 205 that the flat hash map index data structure lacks a key / fingerprint matching the value of K c computed in step 204, the TIG 102 can continue searching the flat hash map index data structure for a key / fingerprint matching K c ​​​The additional value matches the key / fingerprint. For example, consider the example domain "abc.example.com". A first policy rule (which may be based on a first CTI indicating the association between the domain "example.com" and a network threat / attack) could instruct that packets with that field value should be allowed, but their presence should be logged. A second policy rule (which may be based on a second CTI indicating another association between the subdomain "abc.example.com" and a network threat / attack) could instruct that packets with that field value should be blocked. The fingerprint of example.com (and therefore the first rule) will be found before the fingerprint of abc.example.com, and thus before the second rule. If the search stops after finding the first rule, more serious rules may be missed.

[0105] In step 207, TIG 102 can determine whether it has processed all relevant parts of the search object (e.g., whether all tags L of the domain / subdomain have been processed). If no (e.g., if c is not equal to i), the c counter is incremented by 1 in step 208, and TIG 102 can repeat the portion of the method that began in step 204. If yes (e.g., if c = i), TIG 102 adds the stored rule(s) IDs (stored in the results list during one or more executions in step 206) to the rule(s) IDs (if any) that can be determined using other values ​​from packets in transit, and can apply the rule(s) corresponding to all determined rule(s) IDs to the packet. This will combine FIG. 7 Further description.

[0106] FIG. 2B This is a flowchart illustrating another example method for searching a flat hash map index data structure based on a search object obtained from a field in a packet in transit. The flat hash map index data structure includes {fingerprint, rule ID} index pairs of a policy (or part of a policy). FIG. 2B One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. FIG. 2B The method is similar to FIG. 2A The method includes steps 215 and 219-224, which can be respectively connected with... FIG. 2A Steps 201 and 203-208 are the same. FIG. 2B Step 216 can be similar to FIG. 2A Step 202 is the same. However, in FIG. 2B In the method, step 216 is executed in the first processing thread A1, and steps 219-224 are executed concurrently in the second processing thread B1.FIG. 2B In this method, based on the search object received in step 215, a membership test is initiated in thread A1, and a search for the flat hash map index data structure is initiated in thread B1. If TIG 102 determines membership in step 216, thread A1 can be terminated in step 217. If TIG 102 determines no membership in step 216, thread B1 can be terminated in step 218, and rules determined using other values ​​from packets in transit (if any) can be applied.

[0107] The selection can be based on the fact that the estimated or expected time for performing step 216 is less than the estimated or expected time for completing the search of the index data structure. FIG. 2B The method. Thread B1 may include step 225, which may be executed after a "No" determination in step 223, and may cause thread A1 to terminate if those time estimates are incorrect or deviations from expected times cause the search to be faster than the membership test. In step 225, TIG 102 may determine whether thread A1 is still executing, and if so, terminate the execution of thread A1.

[0108] Membership tests can be omitted. For example, if the time required for membership tests is unlikely to be much less than that required for searching the indexed data structure, then dedicating memory resources to the collection data structure may be inefficient. FIG. 2C This is a flowchart illustrating another example method for searching a flat hash map index data structure based on a search object obtained from a field in a packet in transit. The flat hash map index data structure includes {fingerprint, rule ID} index pairs of a policy (or part of a policy). FIG. 2C The method is similar to FIG. 2A The method differs in that step 202 is omitted.

[0109] FIG. 2D This is a flowchart illustrating another example method for searching a flat hash map index data structure based on a search object obtained from a field in a packet in transit, the flat hash map index data structure including index pairs of a strategy (or part of a strategy). FIG. 2D One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. FIG. 2D The method can be executed by TIG102, and is similar to... FIG. 2B The difference lies in using incremental label compression instead of incremental label hashing. FIG. 2D In the example method, the flat hash map index data structure includes lossless compression functions as described above. Ca computed compression (compression value) rather than a hash-based fingerprint. FIG. 2D Steps 235-239 and 242-245 of FIG. 2B Method 200 are analogous to steps 215-219 and 222-225 of FIG. 2D Method 200, respectively. FIG. 2B Step 240 of Method 200 is analogous to step 220 of Method 200, with the difference being that in step 240, TIG 102 can compute a compression-based key K c based on the cth label L c of the search object. c For example, if c = 1, then K c can be computed as C (L1), and if 1 < c < i, then K c can be computed as C (L c-1 ) || C (K c ). FIG. 2D Step 241 of Method 200 is analogous to step 221 of Method 200, with the difference being that in step 241, TIG 102 can search the flat hash map index data structure for a key / compression that matches the value of K c computed in step 240. Each of the keys / compressions in the flat hash map index data structure has been computed using incremental compression including one or more steps (such as step 240), from matching criteria (e.g., threat indicators) according to policy rules corresponding to the rule ID to which the key / compression maps in the flat hash map index data structure.

[0110] In a variant of FIG. 2D Method, the method can be executed in a single thread, similar to the method of FIG. 2A Method, where step 236 is executed before step 239, the membership determination of “yes” is a prerequisite for executing the remaining method steps, and steps 237, 238, and 245 are omitted. In yet another variant of FIG. 2D Method, the membership test can be omitted (similar to the method of FIG. 2C Method).

[0111] FIG. 2E is a flowchart illustrating another example method for searching a flat hash map index data structure for an index pair of a policy (or a portion of a policy) based on a search object obtained from a field of a packet in transit. FIG. 2E The example method of FIG. 2EOne or more steps of the example methods can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps can be added.

[0112] In step 250, a search object can be received (e.g., from a process that parses the specified packet fields for policy rule criteria and extracts data from those fields). The search object can include a domain name. Additionally or alternatively, the search object can include a URL having a <host> portion and a <path> portion. Specifically, FIG. 2E The method of may use a single instance of a flat hash map index data structure that has been configured based on threat indicators in rules created with domain name CTI as a basis and also has been configured based on threat indicators in rules created with URL CTI as a basis. For example, the single instance of the flat hash index data structure can map keys generated by incrementally hashing domain names (or portions of domain names) in matching criteria of first policy rules based on all relevant domain name CTI to rule IDs of those first policy rules. The single instance of the flat hash index data structure can also map keys generated by incrementally hashing URLs (or portions of URLs) in matching criteria of second policy rules based on all relevant URL CTI to rule IDs of those second policy rules.

[0113] Similar to FIG. 2B and FIG. 2D The method of FIG. 2E The method of includes concurrent processing of threads A3 and B3. Based on receiving a search object in step 250, a membership test can be initiated in thread A3, and a search of the flat hash map data structure can be initiated in thread B3. If the TIG 102 determines membership in step 251 (which can be similar to steps 202, 216, and 236), thread A3 can be terminated in step 252. If the TIG 102 determines no membership in step 251, thread B3 can be terminated in step 253, and rules determined using other values from the packet in the transmission (if any) can be applied.

[0114] In step 254, the TIG 102 may determine the part of the search object that is a label of the domain name, the number (i) of labels, and the order of the labels. The TIG 102 may also determine in step 254 whether there is a path part of the search object, and if there is a path part, determine the segment(s) S of the path part, the number (j) of segments, and the order of the segments. If there is no path part of the search object, the TIG 102 may set the value of j to zero for the purposes of step 259 (as described below). Additionally or alternatively, the TIG 102 may be configured in step 254 to determine whether the search object is a URL having a <host> part in the form of an IPv4 address or and an IPv6 address. If the search object is a URL having an IPv4 address as the <host> part, the TIG 102 may set i = 4 in step 254 and determine the integers N1.N2.N3.N4 that will be processed as labels L1 to L4 in step 256 (as described below). If the search object is a URL having an IPv6 address as the <host> part, the TIG 102 may set i = 8 in step 254 and determine the hexadecimal digit groups H1:H2:H3:H4:H5:H6:H7:H8 that will be processed as labels L1 to L8 in step 256.

[0115] In step 255, the TIG 102 may initialize the counter c to 1. In step 256, the TIG 102 may calculate a hash-based key K based on the labels or path segments of the search object c . For example, if c = 1, then K c may be calculated as H (L1). If 1 < c ≤ i, then K c may be calculated as H (L c || K c-1 ). If there is a path part of the search object, and if i < c ≤ j, then K c may be calculated as H (S c-i || K c-1 ). In step 257, the TIG 102 may search the flat hash map index data structure to find the entry associated with the K calculated in step 256 cany of the keys / fingerprints in the flat hash map index data structure have been computed using an incremental hash generation method including one or more steps (e.g., step 256) from matching criteria (e.g., threat indicators) of policy rules corresponding to the rule IDs to which the keys / fingerprints are mapped in the flat hash map index data structure. If one or more matching fingerprints are found, then in step 258, the TIG 102 can store the value of the rule ID of each index pair having a matching fingerprint in a result list data structure.

[0116] Additionally or alternatively, FIG. 2E Step 256 of the method of FIG. 2 can include using a different procedure to compute the keys, and step 257 can include searching the flat hash map index data structure including keys / fingerprints that are also computed using the different procedure. For example, the key computation in step 256 and the key / fingerprint computation of the flat hash map index data structure searched in step 257 can be based on a hash that is based on a hash of a plurality of individual tags (e.g., if c = 1, then K c = H (L1), if if 1 < c < i, then K c = H ( H (L c ) || K c-1 ) as another example, the key computation in step 256 and the key / fingerprint computation of the flat hash map index data structure searched in step 257 can be based on a plurality of hash functions (e.g., any of the procedures using hash functions H 1, H 2, and H 3 as described above).

[0117] After step 258, or after determining in step 257 that the flat hash map index data structure lacks a key / fingerprint matching the value of K c computed in step 256, the TIG 102 can continue searching the flat hash map index data structure for a key / fingerprint matching K cThe additional value matches the key / fingerprint. In step 259, TIG 102 can determine whether it has processed all tags and path segments of the search object (if any). If not (e.g., if c has not yet reached the value i+j), the c counter is incremented by 1 in step 260, and TIG 102 can repeat the portion of the method starting from step 256. If yes (e.g., if c = i+j), TIG 102 can determine in step 261 whether thread A3 is still executing, and if so, terminate the execution of thread A3. After step 261, TIG 102 can add the stored rule(s) IDs (stored in the result list during one or more executions in step 258) to the rule(s) IDs (if any) that can be determined using other values ​​from the packet in transit, and can apply the rule(s) corresponding to all determined rule(s) IDs to the packet. This will combine FIG. 7 Further description.

[0118] exist FIG. 2E In a variant of the method, the method can be executed in a single thread, similar to... FIG. 2A The method, in which step 251 is performed before step 254, where the "yes" membership determination is a prerequisite for performing the remaining method steps, and steps 252, 253, and 261 are omitted. FIG. 2E In another variation of the method, the membership test can be omitted (similar to...). FIG. 2C (Method).

[0119] FIG. 2F This is a flowchart illustrating another example method for searching a flat hash map index data structure based on a search object obtained from a field in a packet in transit, the flat hash map index data structure including index pairs of a strategy (or part of a strategy). FIG. 2F One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. FIG. 2F The method can be executed by TIG102, and is similar to... FIG. 2E The method differs in that it uses incremental tags (and segments for URL search objects) for compression instead of incremental tags and segment hashes. FIG. 2E In the example method, the flat hash map index data structure includes lossless compression functions as described above. C The computation is compressed (compressed value), rather than a hash-based fingerprint. Similar to... FIG. 2E The method FIG. 2FThe method can be performed using a single instance of a flat hash map index data structure that has been configured based on threat indicators in rules created based on domain name CTI and has also been configured based on threat indicators in rules created based on URL CTI. For example, the single instance of the flat hash index data structure can map keys generated by incrementally compressing domain names (or portions of domain names) in the matching criteria of a first set of policy rules based on all relevant domain name CTI to the rule IDs of those first set of policy rules. The single instance of the flat hash index data structure can also map keys generated by incrementally compressing URLs (or portions of URLs) in the matching criteria of a second set of policy rules based on all relevant URL CTI to the rule IDs of those second set of policy rules.

[0120] FIG. 2F Steps 265 - 270 and 273 - 276 of FIG. 2E are respectively similar to FIG. 2F Steps 250 - 255 and 258 - 261 of FIG. 2E Step 271 of c is similar to c Step 256 of C except that in step 271, TIG 102 can calculate a compression-based key K based on the label or path segment of the search object c For example, if c = 1, then K C can be calculated as c (L1). If 1 < c ≤ i, then K C can be calculated as c-1 (L c ) || C (K c-i ). If there is a path portion of the search object and if i < c ≤ j, then K c-1 can be calculated as FIG. 2D Step 272 of FIG. 2E is similar to Step 257 of

[0121] except that in step 272, TIG 102 can search the flat hash map index data structure to find a key / compression that matches the value of K c calculated in step 271. Each key / compression in the flat hash map index data structure has been calculated using incremental compression including one or more steps such as step 271, according to the matching criteria (e.g., threat indicators) of the policy rule corresponding to the rule ID to which the key / compression is mapped in the flat hash map index data structure. In a variant of the FIG. 2F method, the method can be performed in a single thread, similar to FIG. 2AThe method, in which step 266 is performed before step 269, where the "yes" membership determination is a prerequisite for performing the remaining method steps, and steps 267, 268, and 276 are omitted. FIG. 2F In another variation of the method, the membership test can be omitted (similar to...). FIG. 2C (Method).

[0122] FIG. 2G This is a flowchart illustrating another example method for searching a flat hash map index data structure based on a search object obtained from a field in a packet in transit, the flat hash map index data structure including index pairs of a strategy (or part of a strategy). FIG. 2G One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. FIG. 2G The method can be executed by TIG102, and is similar to... FIG. 2E and FIG. 2F The method differs in that it uses a combination of incremental hashing and incremental compression. FIG. 2G In the example methods, the flat hash map index data structure can include compression and fingerprinting based on compression and hashing. Similar to... FIG. 2E and FIG. 2F The method FIG. 2G This method can be implemented using a single instance of a flat hash map index data structure, configured based on threat indicators in rules created using domain name CTIs and also based on threat indicators in rules created using URL CTIs. For example, this single instance of the flat hash index data structure can map keys generated by incrementally compressing domain names (or portions of domain names) in the matching criteria of first policy rules based on all relevant domain name CTIs to the rule IDs of those first policy rules. Similarly, this single instance of the flat hash index data structure can map keys generated by incrementally compressing and hashing URLs (or portions of URLs) in the matching criteria of second policy rules based on all relevant URL CTIs to the rule IDs of those second policy rules.

[0123] FIG. 2G Steps 280-285 and 288-291 are respectively similar to FIG. 2E Steps 250-255 and 258-261. FIG. 2G Step 286 is similar to FIG. 2E Step 256 differs from step 286 in that TIG 102 can calculate the key K based on compression or a combination of compression and hashing. c For example, if c = 1, then K c It can be calculated as C(L1). If 1 < c ≤ i, then K c can be calculated as C (L c ) || C (K c-1 ). If there is a path part of the search object and if i < c ≤ j, then K c can be calculated as H (S c-i || K c-1 ). FIG. 2G Step 287 of FIG. 2E is similar to step 257 of c , except that in step 287, TIG 102 can search a flat hash map index data structure to find keys that match the value of K c calculated in step 286. Each of the keys in the flat hash map index data structure has been calculated using incremental compression / hashing including one or more steps such as step 286, according to a matching criterion (e.g., threat indicators) of a policy rule corresponding to the rule ID to which the key maps in the flat hash map index data structure.

[0124] In FIG. 2G a variant of the method, the method can be executed in a single thread, similar to FIG. 2A [[ID=三十一]]the method of FIG. 2A , where step 281 is executed before step 284, determination of membership of "yes" is a prerequisite for executing the remaining method steps, and steps 282, 283, and 291 are omitted. In FIG. 2G yet another variant of the method, the membership test can be omitted (similar to FIG. 2C the method of [[ID=3%]] FIG. 2C ). [[ID=三十七]]

[0125] The rule tree index data structure disclosed herein is designed to efficiently store keys, which are binary strings that can represent full IPv4 and IPv6 addresses or their prefixes, where these keys can be indexed to identifiers of CTI-derived rules that make up a network security policy. The rule tree can be regarded as an adapted version of a well-known index data structure after a series of adaptations.

[0126] These known trie index data structures include prefix trees or tries, i.e., n-ary trees designed to efficiently store keys with shared prefixes. The links / edges between nodes (i.e., between parent nodes and child nodes) are labeled by a single character that makes up the (multiple) keys. For binary strings, the prefix tree can be a binary tree with links / edges labeled by 0 or 1. Efficient adaptations of the prefix tree include Patricia trie, radix tree, and adaptive radix tree.

[0127] Known trie index data structures also include multi-way tries, or M-way tries, or M-ary tries, where M=2. k When the key is a binary string, k is an integer. The parameter k is called the stride, and the value of k is called the stride length. Each (internal) node in the trie includes nodes with a stride length of 2. k An array of k elements. When the key is a binary string, the array includes all possible bit strings of length k. For example, if k=3, the array of nodes could be {000,001,010,011,100,101,110,111}, which requires at least 24 bits to represent. Each node can have at most 2 k 2 child nodes or child progeny. k The depth or number of levels of a trie can correspond to L / k, where L is the length of the longest key / bit string.

[0128] Those known trie indexed data structures further include poptrie (in H. Asai and Y. Ohara, “Poptrie: A Compressed Trie with Population Count for Fast and Scalable Software IP Routing Table Lookup”, ACM SIGCOMM Computer Communications Review, Vol. 45, No. 4, pp. 57-70, 2015), which is a multi-way trie or 2-way trie adapted and optimized for storing keys (which are bit strings representing prefixes of any length) and searching these keys bit by bit. k A trie. The input to be searched / looked up is a bit string, and given input, it can match keys / prefixes of any length, which can be less than or equal to the input length. Poptrie is optimized for memory-efficient, fast IP routing table lookups, where the search / lookup is to search for a single longest prefix match in the Internet router table, and where the search / lookup stops when the first matching prefix is ​​found. These keys are bit strings representing IPv4 and IPv6 addresses, which can be full addresses (32-bit strings for IPv4 and 128-bit strings for IPv6) and prefix addresses (less than 32 or less than 128 bits in length, respectively). These keys are mapped to entries in the router's Forwarding Information Base (FIB). Poptrie is adapted for multi-way / 2 in several ways. k Triple trees are used to improve the time and space / memory complexity of IP routing table lookups for target applications.

[0129] For example, a poptrie can have a fixed stride length k = 6. For poptrie applications that require IP routing table lookups, this value of k can collectively optimize time and space / memory complexity. For example, 2 k = 64, which corresponds to the register size of a 64-bit CPU architecture (used in modern Internet routers). Therefore, many poptrie data structures (such as index arrays) are 64 bits in size or multiples or sub-multiples of 64 bits, and thus can be processed efficiently by the CPU. Furthermore, the data stored in the poptrie (i.e., the keys (IPv4 and IPv6 address prefixes)) and the lookup / search input (i.e., typically the complete IPv4 and IPv6 addresses) are usually bit strings that are multiples or sub-multiples of 64 bits (i.e., 128-bit IPv6 addresses) or 32-bit IPv4 addresses.

[0130] As another example, constituting 2 k 2 of the internal nodes in the trie k The descendant array of elements is partially replaced with 2 in the poptrie. k Bit vector descendant array. (and 2) k Corresponding 2 in the trie k Compared to the descendant array of elements, 2 k The descendant array of a bit vector requires k times less memory and may require less processing logic, with better time complexity. Using the 2^k descendant array in a poptrie... k Bit vectors are possible because the poptrie described above requires that the search / lookup can search for a single longest prefix match in the Internet router table, and that the search / lookup stops when the first matching prefix is ​​found.

[0131] As an additional example, in a poptrie, each key maps to a single leaf node, where each leaf node includes an index pointing to the corresponding router FIB entry (which can directly correspond to a RIB entry). This contrasts with the corresponding multi-way / 2 configuration without poptrie adaptation and optimization. k In a trie data structure, there may be many duplicate and redundant leaves, which arises due to the characteristics of the target application (i.e., many keys can map to the same router FIB entry). Poptrie utilizes this redundancy to significantly reduce space / memory requirements. Poptrie internal nodes can include bit vectors called leafvecs, which, when combined with associated logical elements, can compress duplicate and redundant leaves, further reducing memory requirements.

[0132] As an additional example, poptrie uses a technique called "direct pointers," which can significantly improve search time performance at the cost of relatively little additional memory. Direct pointers take advantage of the fact that the prefix length of many keys is greater than the stride length k. By adding a length of 2 at the root of the poptrie... s An array of s, where s is an integer multiple of k, for example s=12 and k=6, and using the s most significant bits of the key as direct indices of the array, then subsequent searches of the poptrie will avoid searching the first s / k layers of the poptrie, and instead jump directly to the internal node or (leaf) FIB value in O(1) constant time.

[0133] The rule tree (which can share some features of the poptrie and is likely optimized for storing keys (which could be bit strings representing prefixes of any length) and searching for these keys bit by bit) modifies the poptrie data structure to minimize the time and space / memory complexity of the target application when enforcing network security policies composed of CTI-derived rules. The target application of the rule tree has different requirements and constraints than the target application of the poptrie, and therefore the internal data structure and logic of the rule tree differ from those of the poptrie. For example: (i) in a poptrie, a key maps to a single leaf node, while in a rule tree, a key can map to multiple leaf nodes; (ii) in a rule tree, a key / prefix can include one or more subkeys / subprefixes, where each subkey / subprefix can map to one or more leaf nodes as described in (i) above, while in a poptrie, the key is atomic, i.e., the key / prefix does not include any subkeys / subprefixes; and (iii) in a poptrie, the search / find stops when the first leaf node is encountered, while in a rule tree, the search / find does not stop when the first leaf node is encountered. Therefore, a poptrie cannot be used for the target application of a rule tree. It should be noted that this is not only theoretical—in practice, it should be noted that in a representative set of approximately 10 million IPv4 addresses in a representative CTI, approximately 9 million are full IPv4 addresses or / 32 IPv4 addresses, and when that IPv4 CTI is stored in a rule tree, >99% of the full IPv4 addresses / keys have two or more leaf nodes. Rule tree adaptations may include, but are not limited to, the following: (a) The internal nodes of the rule tree include internal data structures and logic that differ from the internal data structures and logic of the poptrie and accordingly support the requirements (i), (ii), and (iii) above. For example, the poptrie internal nodes include a “vector” data structure, which is a 2k-bit array that constitutes the descendant array. The bit value at each index / position in the array indicates whether the corresponding descendant / child node or jump destination is a leaf node (e.g., bit value “0”, in which case the search stops) or another internal node (e.g., bit value “1”, in which case the search continues). Although this poptrie “vector” bit array may be memory efficient, it does not support the rule tree requirements for rule lookup / search derived from CTI. Regarding rule tree requirement (i), a single bit value “0” indicating that a descendant node is a leaf node cannot indicate whether there is more than one leaf node. Regarding rule tree requirements (ii) and (iii), a single bit value “0” indicating that a descendant node is a leaf node cannot indicate whether the key of the index associated with the leaf node data is a child key, and therefore cannot indicate whether the search should continue.

[0134] (b) Although the poptrie's stride k is set to 6 to optimize the target application of the poptrie, the stride k of the rule tree can be parameterized, and the associated stride length value can be adjusted according to time and space / memory requirements and the data distribution characteristics of the keys to jointly optimize time and space / memory complexity. For example: (bi) Generally, a lower k value may increase the expected search time of the rule tree when compared to a higher k value, because the depth or number of layers of the rule tree may increase.

[0135] (b.ii) For a given set of keys, different values ​​of k may result in different values ​​of memory / space required to store the keys in the rule tree.

[0136] (b.iii) At the time of this disclosure, for a set of keys that includes most of the available IPv4 CTIs (which may be millions of IPv4 addresses), k=4 can collectively optimize time and space / memory complexity.

[0137] (b.iv) If the key length distribution of the key set is highly modal, a specific value of k can lead to jointly optimized time and space complexity. For example, if the keys are mostly or entirely 64-bit and / or 128-bit strings (as is possible, for example, in IPv6 CTI, or as is possible, in a 64-bit or 128-bit fingerprint rule tree), then k=6 can optimize time and space complexity. Furthermore, for such a modal distribution, the direct-pointing strategy of the poptrie can significantly improve time complexity and associated search performance. Additionally, direct-pointing can be used at one or more non-root levels to further improve time complexity. Moreover, a highly modal key length distribution can reduce the number of internal leaf nodes, thus saving more space; (bv) For a given set of keys, optimal performance can be achieved by using different k values ​​for different subtrees of the rule tree; (c) In the poptrie (where the target application could be a fast IP routing table lookup), leaf nodes can index to the FIB entry (i.e., the IP address prefix) of the key. In the rule tree (where the target application could be a CTI-derived rule lookup / search when enforcing a network security policy comprised of CTI-derived rules), one or more leaf nodes of the key can be associated with a set of rule identifiers, where each rule identifier can be associated with a rule having a matching criterion including the key. The rule identifiers or elements in this set can be stored in an efficient data structure that preserves the rank order of the rule identifiers, indicating the order in which rules will be applied to matching packets (i.e., packets in transit containing values ​​matching the key). For example, elements can be stored in a rank-ordered linked list, where the head of the list includes the rule identifier of the highest-ranking rule, and the last element of the list includes the rule identifier of the lowest-ranking rule. In some cases, such as when a rule includes the keyword / instruction "fast," only the highest-ranking rule can be applied to the matching packet. In this case, the rule identifier corresponding to the highest-ranked rule can be stored in the cache to improve time performance, and / or the rule identifier corresponding to the lower-ranked rule can be discarded to reduce space / memory usage; (d) In a poptrie, the lookup / search stops when a leaf node is first encountered, i.e., when the input first matches a key stored in the poptrie. For example, for a target application with fast IP routing table lookup, the key corresponds to the longest prefix in the routing table that matches the input, which can be a 32-bit IPv4 address or a 128-bit IPv6 address. Since the lookup / search stops when a leaf node is first encountered, the logical complexity and memory requirements of the poptrie's internal nodes can be significantly reduced. However, for a rule tree, reaching the first leaf node does not cause the lookup / search to stop. Instead, the search continues until the entire input has been searched to find all matching prefixes. Therefore, the internal nodes and leaf nodes of a rule tree require more memory and more complex logic and data structures, as described in (4)(c) above.

[0138] FIG. 3A through FIG. 3D Together with the following descriptions of these diagrams, flowcharts / algorithms and descriptions of example implementations of rule trees in the context of network security policy enforcement applications are provided. The following description shows that rule trees can meet the requirements of such applications, while poptrie cannot.

[0139] FIG. 3A An example rule tree 320 and related data structures are shown, including an example rule tree internal node 321 and associated (external) data structures 327 (labeled "Search Results List") and 330 (labeled "Internal Data and External Data"). Each internal node in the rule tree processes a k-bit portion or block of a bit string, where k is the stride, which is also the stride length. In the exemplary internal node 321, the stride length k = 2. It should be noted that k = 2 is used here to simplify the example, but in practice, the stride length k should be chosen to provide optimal performance, which can be measured by the combination of time complexity and space / memory complexity in the context of the expected data elements stored in the rule tree and the associated application.

[0140] For k=2, there are six (6) possible values ​​for the internal node processing block: “0”, “1”, “00”, “01”, “10”, “11”. Typically, and for reasons explained below, a k-bit block may have 2... k+1 -2 possible block values. The bit string processed by the rule tree can be a key or index mapped to a value (e.g., a rule ID), or the bit string can be a search object. For an exemplary application, the key and search object can be IPv4 and IPv6 addresses, whether prefixed or full IP addresses, and the value can be a rule identifier / rule ID associated with a network security policy.

[0141] Internal node 321 may include an outer vector 321b, which may be of length 2. k+1A bit array of -2, indexed by each possible value of the current block being processed by the internal node 321, as shown by the smaller number above the box in the outer vector 321b. A "0" value at the position / index corresponding to the block value in the bit array can indicate that there is no value mapped to the corresponding key or subkey, e.g., no rule ID; while a "1" value at the position / index corresponding to the block value in the bit array can indicate that there is a value mapped to the corresponding key or subkey, e.g., one or more rule IDs. The mapped value (e.g., rule ID) can be referred to as an "outer data" node because the associated data can be stored in memory outside or outside the structure of the internal node 321 (e.g., in a sparse array, as described below for internal and outer data 330). In poptrie terminology, a poptrie "leaf node" may correspond to an outer data node in a regular tree to some extent; however, regular trees use different terminology because, for poptrie and tree data structures in general, a "leaf node" is typically associated with a data value representing the final / terminal inner node of a path through the poptrie originating from the poptrie root node; while a regular tree can have one or more data values ​​associated with any node on a path through the regular tree. Poptrie does not support associating data values ​​with any node on a path other than the final / terminal node, while regular trees support associating multiple data values ​​with any node on a path through the regular tree. Therefore, a poptrie search will return at most one data value, while a regular tree search can return multiple values / rule IDs, which can be stored in a "search results list" data structure 327, as described further below. Outer data nodes (e.g., including rule IDs) can be efficiently stored in (outer / external) data structures, such as well-known sparse array or sparse matrix data structures, like... FIG. 3A The sparse array data structure 330, which consists of "internal data and external data", is used as an example.

[0142] Multiple external data nodes associated with an element "1" in the external vector bit array 321b can be referenced in the internal and external data sparse arrays 330 via an "external data reference list" data structure 321c. This data structure includes pointers to the sparse arrays and an indication of how many external data nodes (e.g., rule IDs) are associated with a corresponding key or subkey (e.g., a prefix of a key). For example, setting an external vector index position "0" to 1 indicates the presence of one or more external data nodes or rule IDs associated with the corresponding key or subkey. Multiple external data nodes can be referenced by corresponding "0-Ptr" pointers, which point to the location of the multiple external date nodes in the internal and external data sparse arrays 330. The value of "0-#RuleIDs" indicates the number of external data nodes or rule IDs associated with the corresponding key or subkey. When an element in the external vector 321b bit array is set to "0," such as at index / position "01," the corresponding 01-Ptr value is empty, and the 01-#RuleIDs value is zero.

[0143] For convenience, FIG. 3AIt is also shown that internal node 321 includes a sparse array head pointer 321d. However, internal nodes of the rule tree do not need to simultaneously contain pointers such as pointer 321d and pointers in the external data reference list such as external data reference list 321c. If the rule IDs are stored in non-contiguous blocks of memory, multiple pointers, such as those shown in external data reference list 321c, can be used. However, if the rule IDs associated with an internal node are in a single block of memory, making it possible to determine the memory location of a particular rule ID based on an offset relative to the memory location indicated by a single pointer, multiple pointers may not be necessary. In fact, internal nodes of the rule tree can omit reference lists such as reference list 321c, and the information shown in reference list 321c can be inferred and / or computed based on other information in the internal nodes. For example, if a sparse array is used to store reference list information, a single pointer (such as sparse array head pointer 321d) can reference the beginning or head of the sparse array in memory (e.g., local main memory) and can be included in the internal node of the rule tree. The values ​​of pointers to the inner and outer data sparse arrays 330, as well as the number of outer data nodes in these sparse arrays, can be dynamically determined / calculated as needed. For example, the outer data pointer value, such as 0-Ptr, can be efficiently calculated as an offset relative to the sparse array head pointer 321d by counting the entries of "1" values / elements preceding the "1" values ​​in the currently processed outer vector 321b bit array, for example. It should be noted that the rule tree search differs from a similar poptrie search in that the poptrie search stops if it encounters a data value / leaf node (e.g., when the first matching prefix is ​​found); however, the rule tree search may not stop if it encounters a data value that can be indicated by a "1" element / value in the outer vector 321b bit array. The presence of a "1" element anywhere in the outer vector bit array of an inner node of the rule tree does not in any way indicate whether the search has stopped.

[0144] The inner vector 321a is of length 2 kThe bit array can be indexed by each possible k-bit value of the next block to be processed by the descendant internal node. For k=2, the four (4) possible values ​​of the next block to be processed by the descendant internal node can be "00", "01", "10", "11". If the length of the next block is less than k, for example, for k=2, if the next block is "0" or "1" (which may indicate the end / last bit of the key or search object), the next block can be mapped to a k-bit index with trailing zeros, for example, if the next block is "0", it can be mapped to index "00", and if the next block is "1", it can then be mapped to index "10". For a given index / next block value, if the corresponding internal vector bit array element / value is "1", then there exists a descendant internal node to process the next block. FIG. 3A In the example, an internal node is shown as an internal vector 321a bit array, where the positions / indices of "01" and "10" are set to "1", indicating the existence of corresponding descendant internal nodes 322 and 323 for processing the following blocks "01", "10", and "1". Descendant internal nodes 322 and 323 (and / or other internal nodes of the rule tree 321) can be similar to internal node 321 and operate in the same manner as that internal node. The positions / indices of "00" and "11" are set to "0", indicating that there are no descendant internal nodes for processing the following blocks "00" and "11". (See also...) FIG. 3C Steps 357 and 358 are explained in more detail, indicating that the search can stop if the next block of the search object is “0”, “00”, or “11”.

[0145] The inner vectors 321a and 321b of the internal nodes include inner data nodes, which can also be stored in the sparse arrays 330 of inner and outer data. The inner and outer data nodes of each internal node can be stored in the same sparse array. Similar to the outer data, the position of the inner data node in the sparse array can be calculated as an offset relative to a single pointer referencing the head of the sparse array, for example, by counting the entries in the inner vector bit array preceding the "1" value in the currently processed inner vector bit array. FIG. 3A In the example, the inner data nodes and / or outer data nodes stored in the sparse array 330 of inner and outer data are shown as data nodes Data-0 to Data-n.

[0146] As mentioned above, search result list 327 can be used to store data values / rule IDs encountered during the search process while traversing the rule tree (in... FIG. 3AThe example is shown as RuleID 328. When the search stops, the search results list 327 can be returned to the process that issued the associated query. If there is a (total) sorting associated with the data values / rule IDs, for example, rule IDs can correspond to non-negative integers 0, 1, 2, 3, ..., the search results list 327 can be maintained in sorted order. As described elsewhere, rules associated with rule IDs in the search results list can be applied to the package in the order corresponding to the sorting of the rule IDs. For example, the highest-ranking rule to be applied first could be the rule associated with the rule ID corresponding to the lowest non-negative integer value in the search results list 327, which could be the rule ID located at the head of the search results list 327. By maintaining the search results list 327 in sorted order, the application that invokes the search can choose to apply the search results sequentially, for example: (1) only the first rule / highest ranked rule corresponding to the rule ID located at the head of the (sorted) search results list 327; (2) the first N rules or the N highest ranked rules in the (sorted) search results list 327; or (3) all rules in the (sorted) search results list 327.

[0147] Based on the above FIG. 3A The description of rule tree 320 and related data structures shown makes it easier to explain the 2 k-bit block. k+1 -2 possible block values. For at least some types of search objects (e.g., IPv4 or IPv6 addresses), there can exist one or more rule IDs mapped to a specific combination of the values ​​of all bits in a block, and one or more distinct rule IDs mapped to a single bit of the block (or, if k > 2, a combination of the values ​​of all bits in the block). A k-bit block has 2 k +1 Another reason for having -2 possible block values ​​is that there might be cases where the key length is not divisible by k. For example, for a 7-bit key with k = 2, the last block processed would be a single bit of the 7-bit key's LSB.

[0148] The application of the above to k-bit blocks with k>2 is self-evident. For each sub-part of a block comprising multiple MSBs, there can be multiple different rule IDs mapped to by combinations of bit values ​​of that sub-part. The rule ID mapped to by the bit values ​​or combinations of bit values ​​of the first sub-part of the block (e.g., "11" for k=4) may be different from the rule ID mapped to by the combination of bit values ​​of the second sub-part of the block comprising the first sub-part (e.g., "110"), and may also be different from the rule ID mapped to by the combination of bit values ​​of the entire block (e.g., "1101").

[0149] FIG. 3BThis is a flowchart illustrating an example method for key insertion in a rule tree. An exemplary rule tree can consist of internal nodes and other data structures, such as... FIG. 3A As illustrated, a rule tree instance can be created and initialized with a single root internal node ( FIG. 3B (not shown in the image), this node is configured with a stride k, for example, k=2, as... FIG. 3B As shown, both the inner vector bit array and the outer vector bit array can be initialized to all zeros, for example, all "0" elements / bits, and can be stored as inner nodes in the sparse array of inner and outer data of the root inner node. FIG. 3B The method can be performed, for example, by TIG 102. Although other computing devices (e.g., one or more SPMS 107) can also or alternatively perform the same operation. FIG. 3A The method is described in the context of being executed by TIG 102, but for convenience, it is described in the context of being executed by TIG 102. FIG. 3B One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added.

[0150] In step 331, a "key" value can be accepted as input. The key value can be, for example, a bit string representing an IP address prefix (which can be a complete IP address). The "RuleID" value, such as a rule ID, can be, for example, a non-negative integer corresponding to a packet filtering rule (included by the policy) with matching criteria including the key value / IP address prefix. The key value (IP address prefix) can be an index mapped to the RuleID value.

[0151] In step 332, the key string / IP address prefix can be partitioned into substrings of length k. For example, when the stride k=2, the key string is partitioned into 2-bit substrings or 2-bit blocks. The blocks can be processed in order from the most significant bit to the least significant bit of the associated IP address prefix.

[0152] In step 333, the root internal node (which can be identified as the current internal node) can be accessed to process the first block (which can be identified as the current block).

[0153] In step 334, TIG 102 can determine whether the current block (which can be a bit string of length k or shorter) is the last block of the key, for example, a block that includes the k least significant bits of the IP address prefix (which can be a complete IP address, i.e., a / 32 IPv4 address prefix or a / 128 IPv6 prefix).

[0154] If the current block is the last block, then in step 335, TIG 102 can set or reset the outer vector bit array element at the index / position corresponding to the current / last block to "1", for example, refer toFIG. 3B If the last block is the bit string "10", then the element of the outer vector bit array 321b at the position corresponding to "10" can be set to "1". In step 335, TIG 102 can also insert the data value / rule ID into the sparse array 330 data structure of inner and outer data associated with the current inner node (conceptually, this may update the outer data reference list of the inner node), and can return / signal a successful insertion and stop the insertion process.

[0155] If the current block is not the last block, TIG 102 can access the next block of the key in step 336 and set or reset the element of the inner vector bit array to "1" at the index / position corresponding to the next block. As part of step 336, if the corresponding descendant inner node does not exist, TIG 102 can create and initialize the descendant inner node.

[0156] In step 337, TIG 102 can jump to / access the descendant internal node corresponding to the next block, identify the next block as the current block, identify the descendant internal node as the current internal node, and repeat the part of the method starting from step 334.

[0157] FIG. 3B This method can be executed separately for different keys that can be mapped to different rule IDs but may correspond to overlapping parts of the search object. For example, FIG. 3C The method can be executed for a first key (e.g., "11010010") that corresponds to 8 MSBs of an IP address and maps to one or more first rule IDs. FIG. 3C The method can be repeated for a second key (e.g., "110100101101") that corresponds to 12 MSBs of an IP address and maps to one or more second rule IDs. This is repeated when the resulting rule tree is later searched based on the search object "110100101101…" (where "…" indicates that the IP address portion is omitted for simplicity) (e.g., as combined below). FIG. 3C (As described), one or more first rule IDs and one or more second rule IDs can be added to the search results list.

[0158] FIG. 3C This is a flowchart illustrating an example method for searching a rule tree (such as example rule tree 320). For an input IP address, the process can search the rule tree to find all matching prefixes of the input IP address and can return all rule ID values ​​associated with each matching prefix. FIG. 3A The method can be performed, for example, by TIG 102. FIG. 3AOne or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added.

[0159] In step 351, the method may accept a search object value as input, which may be, for example, a bit string representing a complete IP address, which may include the IP address of a (transmitting) packet filtered according to a network security policy including packet filtering rules. The search results list may be initialized as an empty list.

[0160] In step 352, the search target value / bit string / IP address can be partitioned into substrings of length k. For example, when the stride k=2, the bit string is partitioned into 2-bit substrings or 2-bit blocks. The blocks can be processed in order from the most significant bit to the least significant bit of the associated IP address.

[0161] In step 353, the root internal node (which can be identified as the current internal node) can be accessed to process the first block (which can be identified as the current block).

[0162] In step 354, the outer vector bit array of the current internal node can be accessed. For each position / index in the bit array corresponding to the current block, if the corresponding element in the bit array is "1", then one or more rule ID values ​​associated with the search object value / IP address can exist. For example, refer to... FIG. 3D For internal node 321, if the current block is the bit string "10", then the "1" element at position / index "10" in the bit array of outer vector 321b can indicate the existence of one or more associated rule ID values ​​(which can be stored as outer nodes in the sparse arrays of internal and external data 330). It should be noted that in FIG. 3C In the current block, the first bit is "1", and the "0" element at position / index "1" in the outer vector bit array can indicate that there is no associated rule ID value.

[0163] In step 355, all external data or rule ID values ​​associated with the current block can be collected from the sparse array of internal and external data (e.g., sparse array of internal and external data 330) and inserted into the search results list (e.g., search results list 327), which can be maintained in a sorted order.

[0164] In step 356, the internal vector bit array of the current internal node can be accessed (e.g., internal vector bit array 321a).

[0165] In step 357, for the next block (if any) of the search object value / bit string / IP address, if the inner vector bit array element at the position / index corresponding to the next block is "1", then a descendant inner node for the next block may exist. If the inner vector bit array element at the position / index corresponding to the next block is "0", then a descendant inner node for the next block may not exist.

[0166] If a descendant internal node exists for the next block, then in step 359, TIG 102 can jump to / access the descendant internal node corresponding to the next block, identify the next block as the current block, identify the descendant internal node as the current internal node, and repeat the part of the method starting from step 354.

[0167] If no descendant internal node exists for the next block, then in step 358, TIG 102 can return the list of search results and stop searching the rule tree.

[0168] As can be understood from the above, each block of the search object can correspond to a combination of bit values ​​that result in zero, one, or more rule IDs being added to the search results list. As just one possible example, the search object could include blocks k1k2k3k4…k n Block k1 may include corresponding internal vector array elements of the root internal nodes indicating the corresponding descendant internal nodes (at the second level of the rule tree), but no external vector array elements indicating the existence of one or more combinations of bit values ​​corresponding to rule IDs. Block k2 may include corresponding internal vector array elements of the second-level descendant internal nodes indicating the corresponding descendant internal nodes (at the third level of the rule tree) and one or more external vector array elements indicating the existence of one or more combinations of bit values ​​corresponding to rule IDs. Block k3 may include corresponding internal vector array elements of the third-level descendant internal nodes indicating the corresponding descendant internal nodes (at the fourth level of the rule tree), but no external vector array elements indicating the existence of one or more combinations of bit values ​​corresponding to rule IDs. Block k4 may include corresponding internal vector array elements of the fourth-level descendant internal nodes indicating the existence of corresponding descendant internal nodes (at the fifth level of the rule tree) and one or more external vector array elements indicating the existence of one or more combinations of bit values ​​corresponding to rule IDs. Additional blocks k5 to k n Any of these can similarly include combinations of bit values ​​that may or may not have external vector array elements indicating the presence of one or more corresponding rule IDs. Furthermore, blocks k5 to k... n Any of these can include a combination of internal vectors indicating bit values ​​of internal nodes that do not have a corresponding descendant at the next lower level.

[0169] FIG. 3D It shows FIG. 2BHow can the method (similar to the method described in the previous diagram) be executed concurrently with membership tests of collection data structures? FIG. 3D One or more steps of the example method (which may be performed, for example, by TIG 102) may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. In the method of step 3D, step 362 (which may be combined with...) FIG. 3C (The same as step 216) is executed in the first processing thread A6, and steps 365-373 are executed concurrently in the second processing thread B6. FIG. 3D In the method, based on the search object received in step 361, a membership test is initiated in thread A6, and a search of the rule tree is initiated in thread B6. If TIG 102 determines membership in step 362, thread A6 can be terminated in step 363. If TIG 102 determines no membership in step 362, thread B6 can be terminated in step 364, and rules determined using other values ​​from packets in transit (if any) can be applied. Steps 365-373 can be combined with... FIG. 2A Steps 351-359 are the same, the difference being that step 372 may further include terminating the execution of thread A6 (if necessary). FIG. 3D In a variant of the method, the method can be executed in a single thread, similar to... FIG. 2C The method, in which step 362 is performed before step 365, the "yes" membership determination is a prerequisite for performing the remaining method steps, and steps 363 and 364 are omitted. FIG. 4 In another variation of the method, the membership test can be omitted (similar to...). FIG. 4 (Method).

[0170] The optimal indexing data structure for a given threat indicator type may vary depending on requirements and performance. For example, for both IPv4 and IPv6 threat indicators, a rule tree with certain parameter values ​​(e.g., a stride length of 4) might be a better choice than a flat hash map. Furthermore, when both IPv4 and IPv6 threat indicators are in use (e.g., within a policy), separate rule tree instances can be used to improve performance; for example, one rule tree instance for mapping IPv4 threat indicators to corresponding rules, and another separate rule tree instance for mapping IPv6 threat indicators to corresponding rules in the policy. However, it should be noted that regardless of factors such as performance, separate rule tree instances may be necessary for IPv4 and IPv6, as the rule tree might incorrectly interpret IPv4 indicators as subkeys of IPv6 indicators. Additionally, the rule tree might incorrectly interpret 32-bit or shorter IPv6 prefix indicators as subkeys of IPv4 indicators. On the other hand, for example, for domain name threat indicators and URI threat indicators, a hash map might be a better choice. Furthermore, performance can be improved by using a single instance of hash mapping to map the basic / registerable domain name metrics, FQDN metrics, and URLs to the corresponding rules in the policy.

[0171] In addition to supporting time- and space-efficient policy searches, the index data structure and set data structure can also support other requirements and / or properties associated with efficient TIG-based network security policy enforcement to protect the network.

[0172] These requirements and / or properties can include time-efficient key / threat indicator searches. For example, the index data structure can be (at least in part) chosen such that the expected time to search for key / threat indicators in the index data structure is less than the average transmission rate of packets in the current transmission, so that the packet buffer in the transmission does not overflow.

[0173] These requirements and / or properties may also include, or alternatively, time- and / or space-efficient key / threat indicator insertion. For example, an index data structure may be selected (at least in part) based on the expected time required to insert keys (e.g., threat indicators) and their associated mapping values ​​(e.g., one or more rule identifiers) into the index data structure, and / or the expected additional space / memory required to incorporate keys and values ​​into the index data structure.

[0174] These requirements and / or properties may also include, or alternatively, time- and / or space-efficient key / threat indicator deletion. For example, the index data structure may be selected (at least in part) based on the expected time required to delete a key (e.g., a threat indicator) and its associated mapping value (e.g., one or more rule identifiers) from the index data structure, and / or the expected space reduction resulting from the deletion. It should be noted that in some scenarios, deletion operations may not be necessary, in which case the associated index data structure may differ from that required in the scenario.

[0175] These requirements and / or characteristics may also include, or alternatively, time- and / or space-efficient index data structure population for the entire strategy. For example, the index data structure may be selected (at least in part) based on the time required to populate it with all key / threat indicators and associated mapping values ​​for the entire strategy. It should be noted that in some scenarios, such as when a strategy enforced by TIG is periodically or periodically updated with different information exchanges, replacements, or other means in response to changes in CTI, this may influence the selection of the index data structure.

[0176] Requirements and / or characteristics associated with efficient TIG-based network security policy enforcement to protect networks may include dynamic (runtime) key / threat indicator insertion and deletion. For example, in some scenarios, an index data structure may be chosen (at least partially) because updating the policy search data structure may be necessary during runtime, such as when the TIG proactively enforces policies on packets in transit. For instance, instead of replacing the currently enforced policy with a new / updated policy, the TIG may dynamically update the currently enforced policy during runtime / enforcement by, for example, inserting new CTI-derived packet filtering rules into the currently enforced policy. This requires inserting associated threat indicators and rule IDs into associated set data structures and index data structures during runtime / enforcement; and / or the currently enforced policy may be updated by deleting CTI-derived packet filtering rules, and associated threat indicators and rule IDs may be deleted from associated set data structures and index data structures during runtime / enforcement. For example, rule trees may be implemented to support dynamic / runtime insertion and deletion, and thus can support these scenarios.

[0177] Requirements and / or characteristics associated with enforcing efficient TIG-based network security strategies to protect networks may include the information security of key / threat indicators. For example, in some scenarios, index data structures may be chosen (at least partially) because it may be necessary to protect key / threat indicators in set or index data structures such that they are not stored in their plaintext or raw bit string representation (e.g., a 32-bit IPv4 address) or are not stored using reversible functions, so that, for example, the key / threat indicators can be kept confidential from (malicious) observers. As mentioned above, key / threat indicators storing a flat hash map of fingerprints and key / threat indicators storing a fingerprint rule tree can be protected because the hash function used to compute the fingerprint can be an irreversible function and therefore may not actually be reversed to its original representation. Storing, for example, IP addresses or reversibly compressed rule trees may not be protected in this way, and if not protected, a (malicious) observer could read the original representation of the key / threat indicators. Similarly, for probabilistic set data structures that can use (irreversible) hash functions (such as Bloom filters and Cuckoo filters), the threat indicators contained in the set may not be determinable by a (malicious) observer.

[0178] Generally, regarding the requirements and properties of set and indexed data structures mentioned above, trade-offs may need to be made between the properties and requirements of different indexed and set data structures in terms of TIG performance during policy enforcement. For example, time requirements may be traded against space requirements; for instance, more space / memory can be used to reduce policy search time. An example described above and below is using a set data structure to perform existence / membership tests before or concurrently with policy search. The set data structure is not required and will increase space / memory requirements, but it can improve the time performance of TIG policy enforcement.

[0179] As described above, the index data structure used to search CTI-derived policies can consist of keys that correspond to threat indicators included or otherwise associated with one or more packet filtering rules of the CTI-derived policy. These keys map to one or more rule identifiers of one or more rules associated with the key / threat indicator. For example, the domain threat indicator "www.badactors.com" matching the 10th rule with the rule identifier "rule-10" in an ordered list of rules (e.g., a policy) can be represented in the index data structure as the key "www.badactors.com" mapping to the value "rule-10". Then, for example, when TIG 102 filters packets in transit containing the domain name "www.badactors.com", TIG 102 searches the index data structure to look for the key / threat indicator "www.badactors.com", for example, TIG 102 tests whether the key / threat indicator exists in the index data structure (if the result is true). Therefore, in practice, searching for a key / threat indicator in the index data structure is equivalent to testing the existence / membership of the element / threat indicator within the set of all keys / threat indicators contained in the index data structure. And thus, the index data structure corresponding to the CTI derivation strategy can be used without employing any set data structure to test the membership of threat indicators that constitute the matching criteria of the packet filtering rules (which constitute the CTI derivation strategy).

[0180] However, in some cases, using set data structures in addition to index data structures can improve overall TIG packet filtering performance. Existence / membership tests on set data structures can be (on average) significantly faster than corresponding key searches on index data structures. As mentioned above, in practice, typically only a small fraction of packets in real-world transmissions filtered by TIG 102 will match any CTI-derived packet filtering rules of the cybersecurity policies enforced by TIG. Therefore, performing existence / membership tests for potential threat indicators in set data structures before (or concurrently) performing (slower) key searches for potential threat indicators in index data structures can eliminate many unnecessary key searches (or stop or abandon concurrent key searches before completion). This can improve the overall packet filtering performance of TIG 102 by reducing the average / expected time to filter each packet. Therefore, if the TIG main memory budget or allocation for all index data structures used for the storage policy is not exceeded, set data structures can be added to main memory until the budget / allocation is met. For example, a 2^32-bit array set data structure used to indicate IPv4 addresses in a policy supports very fast existence / membership testing and requires 512MB (i.e., 2^29 bytes) of memory. If more than 512MB remains in the main memory budget, this set data structure can be added, which can improve overall packet filtering and policy enforcement performance.

[0181] The aforementioned index data structure and set data structure can be used, for example, by TIG 102 to efficiently enforce CTI derivation policies on packet traffic in transmissions crossing the boundary between protected and unprotected networks (e.g., TIG102 can be an interface between protected and unprotected networks). FIG. 5 This is a block diagram illustrating an example method for efficient policy enforcement, which may include selecting and using one or more of the data structures and associated usage methods described herein. FIG. 5 The method may include three steps. In step 411 (policy specification), an (ordered) list of packet filtering rules and a threat indicator associated with each rule may be determined. In step 412 (data structure creation), an instance of an indexed data structure that maps threat indicators (keys) to rule identifiers (values) may be created and populated. Step 412 may include creating an instance of a set of data structures that can reduce policy search latency. In step 413 (policy search and enforcement), one or more TIG 102s may process packets in transit by: (a) identifying potential threat indicators in the packets, (b) searching for potential threat indicators in the data structures created and populated in step 412, and (c) applying any matching rules to the packets, such as blocking, logging, and capturing packets.

[0182] The policy specification in step 411 (which may be executed by one or more SPMS 107, one or more TIG 102, and / or one or more other computing devices) may consider and / or incorporate a large number of threats and potential threats. For example, in a typical scenario, and at the time of this disclosure, the policy may be derived from millions (e.g., hundreds of millions) of threat indicators, which may be delivered by thousands of CTI pushes, which may be provided by dozens or hundreds of CTIP 106. Typically, each CTI push may include one or more (e.g., many) threat indicators of a single type, where the type is, for example, IPv4, IPv6, domain name, or URL. Each CTI push can be mapped to or associated with a single packet filtering rule, which includes, for example: matching criteria, which may be threat metrics included by the CTI push; the rule's dispositional action, which may be blocking / denying / dropping the matching packet or allowing / passing / forwarding the matching packet; other packet filtering actions, such as logging / stream logging, capture / stream capture, redirection, spoofing response, etc.; policy processing instructions for the rule, which may be "quick" (e.g., stop policy search) or "continue" (e.g., continue policy search for other matching rules); and metadata, such as the CTI provider and CTI push name, metric type, associated threat / attack type, confidence level, risk score, etc. Rules may include and / or be arranged in an ordered list (policy), wherein rules may be primarily ordered by metric fidelity (where, for example, URLs have higher fidelity than domain names, and domain names have higher fidelity than IP addresses (IPv4 or IPv6), and secondarily ordered by dispositional action (e.g., one of "block" or "allow," where "block" takes precedence over "allow"). Therefore, for example, in an ordered list, a "block" URL rule can take precedence over an "allow" URL rule, an "allow" URL rule can take precedence over a "block" domain rule, a "block" domain rule can take precedence over an "allow" domain rule, an "allow" domain rule can take precedence over an "block" IP address rule, and a "block" IP address rule can take precedence over an "allow" IP address rule. Each rule in the ordered list can be associated with a rule identifier, which can correspond to the rule's order or rank in the ordered list, such that the relative rank of two different rules in the ordered list can be determined by their associated rule identifier (rule ID).

[0183] In step 412, which can be performed by one or more SPMS 107s, one or more TIG 102s, and / or one or more other computing devices, instances of data structures can be selected based on desired requirements and properties (as described above), and these instances are then populated with threat indicators and rule IDs of the policy specified in step 411. For example, the selected index data structure could be (a) an instance of a rule tree with a stride length k = 4 for full IPv4 ( / 32) threat indicators, (b) an instance of a rule tree with a stride length k = 4 for all other (non- / 32) IPv4 CIDR indicators, and (c) an instance of a flat hash map for domain name and URL indicators. (In this example, there are no IPv6 threat indicators). Furthermore, instances of set data structures including bit groups of size 2^32 bits can be selected to reduce policy search time for IPv4 addresses that may be present in the policy. In this example, set data structures (e.g., Bloom filters) are not used to reduce policy search time for domain names and URLs that may be present in the policy. As another example, assume that the data structure selected initially exceeds memory requirements. As an alternative or supplement to modifying the data structure to reduce memory usage, step 411 can be repeated, and the strategy can be modified to reduce memory usage, for example by removing one or more low-confidence CTI pushes.

[0184] FIG. 4 This is a block diagram illustrating an example method for selecting data structures for network protection strategies. FIG. 5 The method can be used as step 412 ( FIG. 5 It is executed as part of ) . For convenience, FIG. 5 This description is based on an example of how the method is executed using SPMS 107. However, FIG. 9 The method can be performed by one or more SPMS 107, one or more TIG 102 and / or one or more other computing devices. FIG. 18 through FIG. 24 One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added.

[0185] In step 501, SPMS 107 receives the policy specified in step 411. In step 502, SPMS 107 can parse the policy and group the policy rules into n rule groups G1-G2. nRules can be grouped based on the type of threat indicators associated with them. For example, a rule whose matching criteria include IPv4 addresses can be assigned to one rule group, a rule whose matching criteria include IPv6 addresses can be assigned to another rule group, a rule whose matching criteria include domain names can be assigned to yet another rule group, a rule whose matching criteria include URLs can be assigned to yet another rule group, and so on. Grouping can be more granular and can include multiple groups based on different variations of a specific type of threat indicator. For example, one group can include rules whose matching criteria include IP addresses associated with a geographic region, and another group can include rules whose matching criteria include IP addresses associated with another geographic region. As another example, one group can include rules whose matching criteria include the host portion of a URL, and another group can include rules whose matching criteria include both the host portion and the path portion of a URL.

[0186] Alternatively, rules can be grouped based on matching criteria that allow them to share one or more common characteristics in their threat indicators. For example, all rules with a domain threat indicator matching criterion where the TLD is ".com" and all rules with a URL domain threat indicator matching criterion where the hostname portion of the URL TLD is also ".com" can be grouped together (e.g., so that a single flat hash map index data structure can be used for all of these rules). Similarly, rules can be grouped based on other TLDs (e.g., for each of the 10 most popular TLDs) and a separate flat hash map index data structure can be used for each group. As a further example, one group can include rules whose matching criteria include the host portion of the URL having an IP address (not a domain name), and another group can include rules whose matching criteria include the host portion of the URL having a domain name. As yet another example, one group can include rules whose matching criteria include the full ( / 32) IPv4 address, and another group can include rules whose matching criteria include the partial (non- / 32) IPv4 address (e.g., a prefix).

[0187] In step 503, SPMS 107 initializes counter m to 1. In step 504, SPMS 107 initializes rule group G. m Choose an index data structure. Optional index data structures may include any of those described herein (e.g., flat hash maps, rule trees) and / or other types of data structures. The selection in step 504 may be based on the rule group G. mThe type of threat indicator(s) associated with the rule, the nature / characteristics of the data structure based on the optional type, and any predetermined considerations associated with the strategy (e.g., prioritizing improved search speed over reduced memory usage or vice versa, allowing runtime updates, protecting information in the data structure). Step 504 can be performed in various ways. For example, it can be based on the rule group G m The type of associated threat indicator sets a default selection for the rule group, and the user can be prompted to accept the default selection or make an alternative. As another example, each of the various data structures can be weighted based on one or more considerations associated with the policy, and the data structure with the highest weight can be selected by default.

[0188] In step 505, SPMS 107 can determine whether the index data structure selected in step 504 is a rule tree. If so, the parameter value of the stride k can be determined in step 506. Step 506 may include... FIG. 8 The method is as follows. If the index data structure selected in step 504 is not a rule tree, step 506 can be omitted. In step 507, SPMS 107 can determine whether to use a set data structure for the rule group G selected in step 504. m The selected index data structure undergoes a membership test. The determination in step 507 can also be performed in various ways. For example, if memory resources may be unavailable, the membership test (and the associated collection data structures(s) that may require memory resources) can be omitted. As another example, a default selection (e.g., yes or no) can be made for all rule groups, and the user can be prompted to override the default selection. As a further example, the default selection can be based on an estimate of the time required to perform a search on the selected index data structure and an estimate of the time required to perform a membership test on the collection data structure, and the default determination in step 507 can be "no" unless the estimated time for the membership test is less than (or a smaller predetermined factor) the estimated time for searching the index data structure. This can be based, for example, on rule group G. m The size (e.g., based on the number of rules and / or the number of levels in the index data structure) determines the estimation time. If SPMS 107 determines in step 507 to use a set data structure with rule group G... mIf the membership test is related to the selected index data structure, SPMS 107 selects the type of set data structure in step 508. The selection in step 508 can be based on user input, a default selection, and / or other considerations (e.g., memory requirements). The data structure selected in step 508 may include a Bloom filter or other probabilistic data structures. Additionally or alternatively, the data structure selected in step 508 may include a flat hash map data structure and / or a rule tree data structure, which have been modified for use as the membership test / set data structure. This modification, and the use of such a modified flat hash map data structure or rule tree data structure for the membership test, will combine... FIG. 6 The description is as follows. If the determination in step 507 does not use a set data structure for membership, step 508 can be omitted.

[0189] In step 509, SPMS 107 can determine whether m = n, and if m is not equal to n, the m counter can be incremented by 1 in step 510. After step 510, SPMS 107 can repeat the portion of the method that begins at step 504. If m = n in step 509, the memory requirements of the strategy can be estimated based on the selected index data structure and the selected set data structure (if included). Step 511 may include... FIG. 6The method will be described below. In step 512, SPMS 107 may determine whether the memory requirements estimated in step 511 are acceptable (e.g., whether the requirements are within the acceptable memory budget corresponding to the TIG that will enforce the policy). If not, SPMS 107 may revise one or more data structure selections, revise one or more rule groupings (e.g., grouping individuals corresponding to the same threat indicator type into one group) and / or determine not to use a set data structure for membership testing associated with one or more groups in step 513. Step 513 may include, for example, an indication of outputting the following: the group and its associated threat indicator type, the selected index data structures and the memory requirements of each of these selected data structures, the set data structures (if selected) and the memory requirements of these set data structures, the estimated memory requirements of other types of index data structures or set data structures (if used alternatively), and / or the estimated search time associated with the selected index data structures and / or alternative index data structures. Step 513 may further include receiving one or more selections for alternative groups, one or more selections for alternative index data structures, and / or one or more determinations not to use membership testing based on these output indications. After step 513, SPMS 107 may repeat the portion of the method that begins with step 511. If SPMS 107 determines in step 512 that the memory requirements estimated in step 511 are acceptable, the data structure can be populated.

[0190] FIG. 4 This is a block diagram illustrating an example method for populating data structures for network protection policies. FIG. 6 The method can be used as step 412 ( FIG. 6 It is executed as part of ) . For convenience, FIG. 6 This description is based on an example of how the method is executed using SPMS 107. However, FIG. 2A The method can be performed by one or more SPMS 107, one or more TIG 102 and / or one or more other computing devices. FIG. 2B One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added.

[0191] In step 601, SPMS 107 can initialize counter m to 1. In step 602, SPMS 107 can determine rule group G. m The selected index data structure is either a flat hash map or a rule tree. If the selected index data structure is a flat hash map, SPMS 107 can initialize counter p to 1 in step 603. In step 604, SPMS 107 can target rule group G.m Rule R p The threat index matching the criteria determines the segment of the threat index. The segment can be, for example, tag L of domain D, path segment S of URL path P, etc. In step 605, SPMS 107 can hash and / or compress these segments. Whether these segments are hashed or compressed can be based, for example, on whether the index data structure depends on hashing (e.g., as...). FIG. 2D , FIG. 2C and FIG. 2E Examples), data compression (e.g., such as...) FIG. 2F and FIG. 5 (example), or a combination of hashing and data compression (e.g., as shown in the example), FIG. 3B (Example) to determine. In step 606, SPMS 107 can compare the hashed and / or compressed segments with rule R. p The rule IDs are inserted together into the flat hash map index data structure. In step 607, SPMS 107 can determine rule group G. m Are there more rules in the set (e.g., is p = x, where x is a rule group G)? m The number of rules in the rule group G. m If there are more rules, SPMS 107 can increment the p counter by 1 in step 608, and can repeat the part of the method starting from step 604. If rule group G m If there are no more rules in the set, then if a set data structure is chosen for rule group G... m Membership testing is performed on the indexed data structure (e.g., in...). FIG. 3B In the method), SPMS 107 can populate the set data structure in step 609. If there is no rule group G... m If the index data structure is chosen to use a set data structure, then SPMS 107 can skip step 609. In step 610, SPMS 107 can determine whether m = n. If yes, then SPMS 107 can increment the m counter by one in step 611 and repeat the portion of the method starting from step 602. If no, then policy search and enforcement can be performed using the data structure already populated in the policy.

[0192] If SPMS 107 determines rule group G in step 602 m If the selected index data structure is a rule tree, then SPMS107 can execute steps 632-638. Steps 632-634 can be combined with... FIG. 3B Steps 332-334 of the method are the same as or similar to those of the method, while steps 635 and 636 can be the same as those of the method. FIG. 4 Steps 336 and 337 of the method are the same or similar. Step 637 can be the same as...FIG. 6 Step 335 is the same as or similar to step 635, except that step 638 is executed at the end of step 637. In step 638, and if a set data structure is chosen for rule group G... m If the membership test of the index data structure is performed, then the set data structure can be populated. If there is no rule group G... m If the index data structure is chosen to use a set data structure, then SPMS 107 can skip step 638. In step 639, SPMS 107 can determine whether m = n. If not, then SPMS 107 can increment the m counter by one in step 611 and repeat the portion of the method starting from step 602. If yes, then policy search and enforcement can be performed using the data structure already populated in the policy.

[0193] As in step 413 ( FIG. 7 As part of ), TIG 102 (or each of the multiple TIG 102s) can populate the data structure in step 412 (e.g., using...) FIG. 7 The method loads or configures these data structures into its main memory, allowing TIG 102's packet processing and policy enforcement logic to access them efficiently. TIG 102 can then process these packets in transit by searching these data structures based on values ​​extracted from fields of the packets in transit, and enforce policy rules discovered as search results.

[0194] FIG. 2A through FIG. 2F This is a block diagram illustrating an example method for policy search and enforcement using one or more data structures, such as those described in this article. FIG. 3C One or more steps of the example method may be rearranged (e.g., executed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added. In step 711, TIG 102 may receive packets in transit. In step 712, TIG 102 may extract one or more field values ​​from each of the multiple fields of the packet. The extracted field values ​​may then be forwarded to the corresponding processing thread among multiple concurrently executing processing threads to be used as search objects to search for index data structures generated and populated for a strategy (or part of a strategy). For example, field value 700.1 from a packet field of a first type may be forwarded to thread 713.1, field value 700.2 from a packet field of a second type may be forwarded to thread 713.2, field value 700.3 from a packet field of a third type may be forwarded to thread 713.3, and so on, with field value 700.n from a packet field of type n being forwarded to thread 713.n. Each of threads 713 may include any of the index data structure search methods described herein (e.g., according to...). FIG. 3D ,FIG. 2B through FIG. 2F or FIG. 3D Any one of the methods in ), any one or all of threads in thread 713 may include a separate processing thread for test set membership (e.g., such as FIG. 8 and FIG. 8 (In the method), any one or all threads in thread 713 may include a set membership test in the same processing thread as the index data structure search, and / or any one or all threads in thread 713 may omit the set membership test.

[0195] In addition, or alternatively, a single field value from a packet may be forwarded to each of multiple concurrent processing threads (e.g., field value 700.4 may be forwarded to a first processing thread 713.4(1), a concurrent second processing thread 713.4(2), a third concurrent processing thread 713.4(3), etc.). This may occur, for example, if the first processing thread includes a first index data structure created for a first set of rules sharing a first characteristic, if the concurrent second processing thread includes a second index data structure created for a second set of rules sharing a second characteristic, and if the field value from the packet may potentially match a key in either the first or second index data structure.

[0196] As an example of a scenario where packet field values ​​can be forwarded to multiple processing threads, the first index data structure (of the first processing thread) can be a flat hash-mapped index data structure including keys that map to rule IDs of the first set of policy rules, corresponding to domain names and / or URLs where the top-level domain is ".com". The second index data structure (of the second processing thread) can be a flat hash-mapped index data structure including keys that map to rule IDs of the second set of policy rules, corresponding to domain names and / or URLs where the top-level domain is ".net". The third index data structure (of the third processing thread) can be a flat hash-mapped index data structure including keys that map to rule IDs of the third set of policy rules, corresponding to a top-level domain or a domain name and / or URL where the effective top-level domain is anything other than ".com" or ".net". Field values ​​from packets can be domains or URLs that are forwarded to each of the first, second, and third processing threads for concurrent searching of the first, second, and third index data structures.

[0197] As another example of a scenario where packet field values ​​can be forwarded to multiple processing threads, the fourth index data structure (of the fourth processing thread) could be a rule tree index data structure that maps keys corresponding to full ( / 32) IPv4 addresses to rule IDs of fourth group policy rules. The fifth index data structure (of the fifth processing thread) could be a rule tree index data structure that maps keys corresponding to partial (e.g., non- / 32) IPv4 addresses to rule IDs of fifth group policy rules. Field values ​​from packets could be the full IPv4 address forwarded to each of the fourth and fifth processing threads for concurrent searching of the fourth and fifth index data structures.

[0198] For a given packet, processing threads 713 may output one, some, or all of one or more rule IDs determined by searching the index data structure, or none of them. In step 714, TIG 102 may sort these rules based on the priority of the rules corresponding to these rule IDs, and may apply some or all of these rules based on their priority. Applying rules may include performing one or more actions specified by the rules (e.g., blocking / dropping / rejecting packets, allowing / forwarding / passing packets, logging packets, capturing packets (e.g., storing a copy of the packets), redirecting or rerouting packets, modifying or transforming packets in some way to protect the network, generating and / or sending responses to packets, etc.). As indicated by connector C41, TIG 102 may repeat the method of Figure 102 for packets in the next transmission.

[0199] The aforementioned data structure can be used to address the following issues in the policy enforcement application of this disclosure: Given a large number of CTIs, such as tens or hundreds of millions of metrics, creating a TIG-enforceable policy from these metrics (which includes populating a data structure with these metrics and rule IDs) can take a relatively long time, such as several minutes. This can be problematic, for example, when the data structure population time is close to the policy update cycle (e.g., every 15 minutes). Furthermore, it is possible that the size of the populated data structure, or the main memory consumed by it, may exceed the main memory budget allocated to policy enforcement by TIG 102. When this occurs, one or more remedial actions may be required to reduce the size, such as: selecting a more space-efficient data structure; reducing the number of threat metrics; eliminating one or more aggregate data structures used to test the membership of metrics in the policy; and so on. Subsequently, the data structure selection / configuration process and / or the data structure population process may need to be repeated to check that the memory budget has not been exceeded.

[0200] It would be useful to accurately estimate the main memory requirements for a given set of threat indicators, the policy to be enforced (e.g., an ordered list of packet filtering rules derived from CTI), and a given data structure selection before populating the data structure with threat indicators and rule IDs. For all the data structures described in this paper, accurate estimates of memory / space requirements can be efficiently computed as follows: For a set of bit array data structures for IPv4 address indicators, the memory requirement could be 2^32 bits (e.g., "4Gb") or 2^29 bytes (e.g., "512MB"). For a set of bit array data structures for IPv6 address indicators, the value Z of the Z most significant bits of the IPv6 address indicator can be chosen and used to populate the bit array, e.g., Z=32, and the memory requirement could then be 2^Z bits. For probabilistic set data structures, such as Bloom filters, Cuckoo filters, etc., well-known formulas for memory requirements for a set of elements and for a given false positive rate can be used to compute accurate estimates by treating threat indicators as elements when using these formulas. For index data structures, there is typically a constant factor X (which can be determined experimentally) that can be used to accurately estimate the memory requirements for storing N metrics of a specific type (e.g., IPv4, IPv6, domain names, URLs, etc.) in a particular type of index data structure (e.g., rule trees or flat hash maps). That is, for a specific implementation of the index data structure type (e.g., flat hash map), the number N of metrics of a specific type (e.g., URLs), and the factor X for flat hash maps and URLs (determined experimentally), the memory requirement M can be efficiently calculated as M = NX bytes. Therefore, for the choice of metric set, the policy to be enforced, and the data structure, an accurate estimate of the main memory requirements for enforcing a policy can be easily and quickly / efficiently calculated before the data structure is populated with threat metrics and rule IDs. It can then be immediately determined whether the main memory requirements exceed or fall below budget. If main memory requirements exceed the budget, remedial actions can be taken, such as choosing a more memory / space efficient set of data structures (e.g., deselecting bit arrays for IPv4 metrics), modifying policies (e.g., eliminating one or more low-confidence, high-capacity CTI pushes), increasing the main memory budget for TIG for policy enforcement, and so on.

[0201] FIG. 5 This is a flowchart illustrating an example method for calculating the memory requirements of a set of data structures (e.g., indexed data structures and set data structures) selected for a network protection strategy. As described above, FIG. 8 Example methods can be, for example, as FIG. 8 It is executed as part of the method. For convenience, FIG. 8 This description is based on an example of how the method is executed using SPMS 107. However,FIG. 5 The method can be performed by one or more SPMS 107, one or more TIG 102 and / or one or more other computing devices. FIG. 9 One or more steps of the example method may be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps may be added.

[0202] In step 801, SPMS 107 can set the counter m to 1. In step 802, SPMS 107 can initialize the value of the variable Mem_size to 0. In step 803, SPMS 107 can determine the value IDS_mem using one of the methods described above. m This value represents rule group G. m The estimated memory required for the selected index data structure. In step 804, SPMS 107 can determine the value SDS_mem using one of the methods described above. m This value represents rule group G. m The estimated memory required for the set data structure. If the set data structure is not used for the rule group G. m Then SPMS 107 can store SDS_mem m The value is set to 0. In step 805, SPMS 107 can set the value of Mem_size to Mem_size + IDS_mem. m + SDS_mem m In step 806, SPMS can determine whether m = n. If not, SPMS can increment m by 1 in step 807 and repeat the portion of the method that began in step 803. If yes, the value of Mem_size can be returned (e.g., as...). FIG. 9 (Part of the method).

[0203] FIG. 9 This is a block diagram illustrating an example method for determining the stride k parameter value of a rule tree (e.g., the block size value of internal nodes at each level of the rule tree). For convenience, FIG. 9 This description is based on an example of how the method is executed using SPMS 107. However, FIG. 5 The method can be performed by one or more SPMS 107, one or more TIG 102 and / or one or more other computing devices. FIG. 5One or more steps in the example method can be rearranged (e.g., performed in a different order), combined, omitted, and / or otherwise modified, and / or additional steps can be added. While the previous example involved a rule tree where the value of k might be the same for all levels of the rule tree, this is not necessarily the case. For example, k can have different values ​​for nodes at different levels of the rule tree, and internal rule tree nodes can be further modified to include one or more data structures to indicate the applicable value of k.

[0204] In step 901, SPMS 107 may receive (e.g., as...) FIG. 10 (Output of method step 502) Rule group. In step 903, SPMS 107 can initialize counter s to 1. In step 903, SPMS 107 can select array K. s The initial values ​​of elements k(1) to k(L) are given, where L is the number of levels below the root node of the rule tree, k(1) is the k value of the root node, k(2) is the k value of the internal node at the second level of the rule tree, and so on, where k(L) is the k value of the lowest level of the rule tree. SPMS 107 may select the initial k element values, for example, based on a predetermined default value (e.g., by setting all k element values ​​to 4).

[0205] In step 904, SPMS 107 can estimate the array K. s Memory requirements of a rule tree with k values ​​(Mem-req) s ) and average search time (Ave-time) s In step 905, SPMS 107 can compare the estimated memory requirements and estimated average search time determined in step 904 with the array K. s Store them together.

[0206] In step 906, SPMS can determine whether a stopping condition has been met. The stopping condition may include completing steps 904 and 905 for all possible combinations of k values, completing all possible combinations of k values ​​within a predetermined limit (e.g., completing all possible combinations if the initial k element values ​​only change by ±1, ±2, or ± some other constant), completing a predetermined number of iterations of steps 904 and 905, and / or some other stopping condition. If the stopping condition has not been met, SPMS 107 can increment the s counter by 1 in step 907. In step 908, SPMS 107 can initially use array K... s-1 Fill array K with the values s Then, array K is created by changing the values ​​of k elements according to a predetermined pattern. s After step 908, SPMS 107 may repeat the portion of the method that began in step 904.

[0207] If SPMS 107 determines in step 906 that the stopping condition has been met, then SPMS 909 can output array K for each value of s (or for some values ​​of s). s and Mem-req s and Ave-time s The values ​​of s can be displayed in a table or other way. Additionally or alternatively, SPMS 107 can highlight (or restrict the output to) the array K with the lowest estimated memory requirements among all values ​​of s. s and Mem-req s and Ave-time s The corresponding value of , and the array K that estimates the lowest average search time among all values ​​of s. s and Mem-req s and Ave-time s The corresponding value. In step 910, SPMS 107 can receive the selection array K. s The input. In step 911, SPMS can return (e.g., as input). FIG. 10 (Part of the method) Selected array K s The value of the k parameter.

[0208] FIG. 10This is a block diagram of an example computing device 1000. Any device described herein or shown in any of the accompanying drawings (such as any of TIG 102 or SPMS 107) may be implemented, in part or in whole, using one or more computing devices (such as computing device 1000). Computing device 1000 may, for example, include a general-purpose computing device having general-purpose hardware configured to perform one or more methods (or parts thereof) described herein using specific software and / or firmware; or it may, for example, include a dedicated computing device having dedicated hardware (and / or dedicated software and / or dedicated firmware) tailored for specific functions. Any hardware element of computing device 1000 and / or computing device 1000 itself may be emulated in a virtual version of computing device 1000. Computing device 1000 may include one or more processors 1001 capable of executing computer-readable instructions of a computer program to perform any of the functions or other operations described herein. Instructions and other data may be stored in memory 1002, which may include, for example, memory such as read-only memory (ROM) and / or random access memory (RAM), hard disk, disk or optical disk, universal serial bus (USB) drive and / or any other type of computer-readable medium. Data may be organized in any desired manner, such as being organized to be accessible via database software stored as instructions in memory 1002 and executed by one or more processors 1001. The computing device 1000 may also include a user interface 1004 for interfacing with one or more input devices 1005 (e.g., keyboard, mouse, voice input, etc.) and one or more output devices 1006 (e.g., monitor, speaker, printer, etc.). The computing device 1000 may also include a network interface 1003 for interfacing with one or more external devices, which may be part of an external network of the computing device 1000. Although FIG. 11 An example hardware configuration is shown, but one or more elements of the computing device 1000 can be implemented using software or a combination of hardware and software. Components of the computing device 1000 can be modified to add, delete, combine, divide, etc. Furthermore, FIG. 10The elements shown can be implemented using basic computing devices and components configured to perform the operations described herein. Processor 1001 and / or memory 1002 can also, or alternatively, be implemented using one or more integrated circuits (ICs). For example, an IC can be a microprocessor that accesses programming instructions or other data stored in ROM and / or hardwired into the IC. For example, an IC can include an application-specific integrated circuit (ASIC) having gates and / or other logic dedicated to the computations and other operations described herein. The IC can perform some operations based on executing programming instructions read from ROM or RAM, while other operations are hardwired into gates or other logic.

[0209] FIG. 11 This is a block diagram illustrating an example packet filtering device 1100 (e.g., TIG 102), which may be located at the boundary between a first network (e.g., network 101) and a second network (e.g., network 105). The packet filtering device 1100 may include one or more processors 1104 (which may be the same as processor 1001), memory 1106 (which may be the same as memory 1002), network interfaces 1108 and / or 1110 (which may be the same as network interface 1003), packet filter 1112, and management interface 1114 (which may be executed by processor(s) 1001, user interface 1004, input device 1005, and / or output device 1006). Processor 1104, memory 1106, network interfaces 1108 and / or 1110, packet filter 1112, and / or management interface 1114 may be interconnected via a data bus 1116 (which may include interconnects). FIG. 11 and FIG. 12 (Arrows for the various elements). Network interface 1108 can connect packet filtering device 1100 to a first network. Similarly, network interface 1110 can connect packet filtering device 1100 to a second network. Memory 1106 may include one or more program modules that, when executed by processor(s)1104, can configure packet filtering device 1100 to perform one or more of the various operations described herein. Memory 1106 may also be used to store rules, databases, logs, and / or any other information used and generated by packet filtering device 1100.

[0210] Packet filtering device 1100 can be configured to receive policies from one or more SPMS 107s, which include one or more index data structures and / or set data structures as described herein. For example, packet filtering device 1100 can receive policies 1118 from the SPMS 107 server via management interface 1114 (e.g., via out-of-band signaling) or network interface 1108 (e.g., via in-band signaling). Packet filtering device 1100 may include one or more packet filters or packet discriminators for performing packet filtering operations (e.g., implementing one or more processing threads for index data structure searches and / or membership tests), and / or logic for implementing one or more packet filters or packet discriminators. For example, packet filtering device 1100 may include packet filter 1112 (which may be implemented by processor 1104 and / or one or more other processors) configured to examine information associated with packets received by packet filtering device 1100 (e.g., from network 101) and forward such packets to one or more operators 1120.1 to 1120.n (which may be implemented as hardware and / or software executed by processor(s) 1104 and / or one or more other processors) based on the examined information. For example, packet filter 1112 may (e.g., using one or more of the methods described herein for searching set data structures and / or indexing data structures) examine information associated with packets received by packet filtering device 1100 (e.g., packets received from network 101 via network interface 1108) and forward these packets to one or more operators 1120 based on the examined information. For example, these operators may implement actions and instructions associated with packet filtering rules that match packets.

[0211] The configuration of packet filter 1112 can be based on the rules included in policy 1118. For example, policy 1118 may include one or more rules specifying that packets with specified information should be forwarded to one of operators 1120, packets with other specified information should be forwarded to another of operators 1120, packets with even more specified information should be forwarded to yet another of operators 1120, and so on. Each operator 1120 can be configured to perform one or more functions corresponding to the application of the rules of policy 1118 on packets received from packet filter 1112. For example, one or more operators 1120 can be configured to forward packets received from packet filter 1112 to network 105, one or more operators 1120 can be configured to forward packets received from packet filter 1112 to network 105 and record the packets, one or more operators 1120 can be configured to forward packets received from packet filter 1112 to network 105 and capture a copy of the packets, one or more operators 1120 can be configured to block packets received from packet filter 1112 from reaching network 105, one or more operators 1120 can be configured to block packets received from packet filter 1112 from reaching network 105, and generate and send different packets, etc.

[0212] like FIG. 12 As indicated by the dashed arrows, packet filtering device 1100 can also perform actions on packets received from network 105 via interface 1110 (similar to the actions described above for packets received from network 101 via interface 1108). For example, packet filtering device 1100 can receive packets from network 105 via interface 1110, filter these packets in packet filter 1112 based on policy 1118, forward these packets to one or more operators 1120, and forward at least a portion of these packets from one or more operators 1120 to network 101.

[0213] Packet filtering device 1100 can obtain threat context information from various sources (or generate threat context information based on information retrieved from various sources), some of which may be local (sources within packet filtering device 1100) and others may be remote (sources outside packet filtering device 100). For example, processor (multiple) 1104 may include a clock that holds the current time and can be used to determine the observation time of packets in transit. Other examples of the threat context information mentioned above include: device location, device ID, administrator and associated security policy preferences, network type, active threat type, multi-packet multi-flow threat / attack analysis results, (multiple) CTI providers and associated information, threat indicator type and fidelity, threat indicator duration, flow origin, flow direction, flow status, connection status, global threat context, domain name, URI, URL, domain popularity, domain registration status, data transmission protocol method, protocol risk, contextual CTI noise, etc., each of which can be determined by (multiple) processors 1104 for and in response to receiving a packet in transit, based on: processing of information stored in memory 1106; calculations performed by (multiple) processors 1104; information received from network 101 via network interface 1108; information received from network 105 via network interface 1110; and / or information received via management interface 1114.

[0214] The functions and steps described herein may be embodied in computer-usable data or computer-executable instructions, such as in one or more program modules, and executed by one or more computers or other devices (e.g., computing device 1000, packet filtering device 1100) to perform one or more functions described herein. Typically, program modules include routines, programs, objects, components, data structures, etc., which, when executed by one or more processors in a computer or other data processing device, perform a specific task or implement a specific abstract data type. Computer-executable instructions may be stored on computer-readable media (e.g., memory 1001, memory 1106), such as disks, optical disks, removable storage media, solid-state storage, random access memory (RAM), read-only memory (ROM), flash memory, etc. It should be understood that the functions of program modules can be combined or distributed as needed. Furthermore, the functions may be wholly or partially embodied in firmware or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Specific data structures may be used to more efficiently implement one or more aspects of this disclosure, and such data structures are contemplated within the scope of the computer-executable instructions and computer-usable data described herein.

[0215] Similar rule trees and flat hash maps described herein can also be used as set data structures / membership test data structures. For example, such rule trees and flat hash maps can be combined with (and / or replace) Bloom filters and / or other set data structures / membership test data structures in the systems, methods, apparatus, and computer-executable instructions (stored on a computer-readable medium) described in U.S. Patent Application Publication No. 2024 / 0007437 (titled “Cyber ​​Protections of Remote Networks via Selective Policy Enforcement at Central Network,” incorporated herein by reference) and / or U.S. Patent No. 11,902,250 (titled “Methods and Systems for Prevention of Attacks Associated with the Domain Name System,” incorporated herein by reference). A rule tree can be configured as a set data structure (e.g., for membership testing of IP addresses) by simply inserting keys without mapping these keys to rule IDs or other data values. Flat hash maps can be configured as set data structures (e.g., for domain name and / or URL membership testing) by simply inserting keys without mapping them to rule IDs or other data values. In both cases, if the data structure is only used for set membership testing, there is no need to store rule IDs (or other data mapped to keys). Similar to Bloom filters, rule trees and / or flat hash maps are good data structures for set membership testing because they facilitate efficient membership testing and allow for significant compression (and reduced memory requirements) compared to raw data (e.g., threat metrics in their raw form). Furthermore, the compressed memory usage of flat hash maps can be directly tuned by changing the hash size. Rule trees and / or flat hash maps can be used interchangeably with Bloom filters. In some cases, rule trees and / or flat hash maps may be more efficient and / or perform better than Bloom filters. For example, a rule tree may be more efficient than a Bloom filter for storing IPv6 metrics.

[0216] FIG. 1 An example network environment 2100 is shown, for example, related to efficient network protection of mobile devices and associated enterprise networks, where rule trees and / or flat hash maps can be used as set membership test data structures. FIG. 1 The elements shown can also be combined. FIG. 12The elements shown describe the operations, and FIG. 12 The elements shown can also be combined. FIG. 12 The operation described. Mobile devices MBDV 2101 and MBDV 2102 are owned / operated by individual users who may be associated with a company operating the private enterprise network ENET2160. Both mobile devices MBDV 2101 and MBDV 2102 are subscribed to by mobile providers operating the radio access network RNET 2120 and the mobile core network MCNET 2150. When mobile devices MBDV 2101 and MBDV 2102 make cellular phone calls, signaling and communications are routed through the mobile core network MCNET 2150 and may be routed through the mobile core networks of other mobile providers (…). FIG. 13 (Not shown in the image). 2G and next-generation mobile devices and mobile networks support access to the Internet and associated TCP / IP networks. Mobile devices MBDV 2101 and MBDV 2102 can access publicly addressed Internet servers, such as web server WSVR 2151 and malware server MALSVR 2152, via Internet 2130. Mobile device MBDV 2101 can download and install the mobile device security application MBL-CYBER-APP, which implements the packet testing, packet filtering, and packet tunneling described herein. The mobile device security application MBL-CYBER-APP can configure, tunnel (multiple) clients, manage and / or use the data structure TUNNEL-M / T (described below), and perform other functions described in other parts of this disclosure. Mobile device MBDV 2102 has not downloaded the mobile device security application MBL-CYBER-APP and is therefore not protected by the methods and systems of this disclosure.

[0217] Mobile device and / or remote network tunneling gateways can utilize data structures (representing rules in a security policy by characterizing threat metrics) to determine whether internet traffic packets match rules in the policy. To represent the entire policy, the system can collect some or all metrics (IP address, domain name, URI, certificate ID, etc.) characterizing each rule in the policy and insert each such metric into, for example, one or more set data structures for membership testing. These one or more set data structures may include one or more rule trees and / or one or more flat hash maps. Optionally, the one or more set data structures may also include one or more probabilistic data structures (e.g., one or more Bloom filters). The set data structure can be tested to determine whether an element (e.g., IP address, domain name, URI, certificate ID, etc.) is a member of that set data structure (e.g., whether the element is a member of the set of metrics used to generate the set data structure). The set data structure can be generated by a policy management server, distributed to each mobile device and / or remote network tunneling gateway, and stored on each mobile device and / or remote network that can be protected by the policy. When a mobile device and / or remote network initiates or receives an Internet traffic packet, the computer logic and / or application on the mobile device and / or remote network tunnel gateway can extract any element contained in the packet that can correspond to threat indicators in a set data structure, such as IP address, domain name, URI, certificate ID, etc., and can test (multiple) set data structures to determine if any such element is a member of the threat indicator set in (multiple) set data structures. If the test indicates that any packet element is a member of the threat indicator set, the packet or a copy of the packet can be tunneled to a tunnel gateway (e.g., a central network tunnel gateway), which may be located in a corporate private network (e.g., a central network). After leaving the tunnel gateway, the packet may be sent to a TIG, which can filter the packet through policies to determine which packet filtering rules match the packet. When the TIG determines (multiple) matching rules through policy testing, (multiple) actions or (multiple) PTFs associated with those rules can be applied to the packet to protect the network. If (multiple) actions or (multiple) PTFs cause the packet to be forwarded to the Internet, the packet can be passed through the associated security stack.

[0218] A policy creation and distribution management server (e.g., a server associated with a policy creation and management system) can receive CTIs from multiple CTIPs and can create CTI-derived policies based on the received CTIs. CTI-derived policies can include multiple rules. The management server can create a data structure for membership testing (M / T) based on a set of elements representing multiple rules, which includes one or more rule trees and / or one or more flat hash maps. This data structure (hereinafter referred to as TUNNEL-M / T) may contain one or more flat hash map data structures (containing values ​​of metrics extracted from rules in the policy, such as hashes and / or compressions), and / or may contain one or more rule tree data structures (containing values ​​of metrics extracted from rules in the policy or based on those values). The system can download policies to a TIG and can transmit the TUNNEL-M / T data structure to each mobile device associated with the enterprise network and / or each remote network tunnel gateway associated with a corresponding remote network, as well as other network elements, including the TIG. Alternatively, TIG can create a data structure TUNNEL-M / T after receiving the CTI-derived policy from the policy creation and distribution management server, and can then transmit the data structure TUNNEL-M / T to each mobile device and / or remote network tunnel gateway.

[0219] Mobile devices can send packets to (or receive packets from) Internet hosts. The membership of packet values ​​(such as IP addresses, domain names, and URIs) in the TUNNEL-M / T data structure can be tested.

[0220] The flat hash map data structure TUNNEL-M / T can be used for this membership test. For example, these flat hash map data structures may only include keys, such as those described herein (e.g., keys based on incremental hashing, incremental compression, or a combination of incremental hashing and compression), and may lack rule IDs or other data mapped to these keys. These flat hash maps can be searched using methods similar to those described herein, by incrementally hashing (and / or compressing) the domain name (or a portion of the URL, or a portion of other values ​​from the packet) and comparing these incrementally hashed (and / or compressed) portions with the keys in the flat hash map. However, if a match is found with the key, the output is not the rule ID, but a determination of whether the domain name (or URL or other packet value) matches the threat criteria of the policy rule. If the incrementally hashed (and / or compressed) domain name (or URL or other packet value) matches a key in the flat hash map, then the domain name (or URL or other packet value) matches the threat criteria of the policy rule (e.g., membership = true). If the domain name (or URL or other packet value) after incremental hashing (and / or compression) does not match any key in the flat hash map, then the domain name (or URL or other packet value) will not match the policy rule threat criteria (e.g., membership = false).

[0221] Such membership testing can also be performed using rule tree data structures, or alternatively. For example, these rule tree data structures may only include keys, such as those described herein (e.g., a portion of an IP address or other values ​​from or based on packets), and may lack rule IDs or other data mapped to these keys. Such a rule tree can be searched using methods similar to those described herein, by traversing internal nodes level by level, using consecutive portions of the IP address (or other values ​​from or based on packets). However, if a match is found with one or more keys, the output is not the rule ID, but rather a determination of whether the IP address (or other values ​​from or based on packets) matches the threat criteria of the policy rule. If you traverse the rule tree and reach an internal node where the possible values ​​of the IP address portion corresponding to that internal node are mapped to an external vector array element storing "1", then the IP address (or another value from the packet or another value based on the value from the packet) will match the policy rule threat criterion (e.g., membership = true); however, if you traverse the rule tree but do not reach an internal node where the possible values ​​of the IP address portion corresponding to that internal node are mapped to an external vector array element storing "1", then the IP address will not match the policy rule threat criterion (e.g., membership = false).

[0222] If the membership test returns true, the system can tunnel the packet to the corporate network. The packet can be filtered through TIG and security stack. Each subsequent packet in the communication flow can be tunneled to the corporate network and can also be filtered through TIG and security stack. If the membership test returns false, the system can forward the packet directly to an Internet host or otherwise process the packet normally without tunneling it to the corporate network.

[0223] return FIG. 12 The private enterprise network ENET 2160 can provide internet access to internal hosts (such as PC 2161). The ENET 2160 can be configured to require all traffic between these internal hosts and internet hosts to pass through the Threat Intelligence Gateway TIG 2170, which enforces CTI-derived policies. Traffic can also pass through the Security Stack SSTK 2175, which may include at least one of traditional network firewalls and other enterprise network security devices such as web proxies, SSL proxies, IDS, IPS, etc. The Threat Intelligence Gateway TIG 2170 and the Security Stack SSTK 2175 can be located at or near the internet access point of the ENET 2160. When policies are applied to packets, the Threat Intelligence Gateway TIG 2170 can generate packet logs that match the rules in the policies. These logs can be sent via the internet to the Security Operations Center (SOC) 2140 for network analysts to use for attack analysis, such as with SIEM applications and packet analyzer applications.

[0224] The Enterprise System Server ESVR 2162 can be a privately hosted web application server for enterprise users, including those who own / operate the mobile devices MBDV 2101 and MBDV 2102. The mobile devices MBDV 2101 and MBDV 2102 access the Enterprise System Server ESVR 2162 using an enterprise-provided application. This application includes an HTTPS client for accessing the Enterprise System Server ESVR 2162 via port 443 (HTTPS port). When the application connects to the Enterprise System Server ESVR 2162, the web application presents a login form. The user can then enter their enterprise credentials to securely access the web application. The network security administrator has opened port 443 of the network firewall in the security stack SSTK 2175 so that connections such as unsolicited inbound HTTPS connections originating from applications on the mobile devices MBDV 2101 and MBDV 2102 can initiate sessions with the Enterprise System Server ESVR 2162.

[0225] The host tunneling gateway TGW 2163, attached to the private enterprise network ENET 2160, terminates and centralizes tunnels that can be established with mobile devices such as the MBDV 2101, which has the mobile device security application MBL-CYBER-APP installed. Similar to the enterprise system server ESVR 2162, the network administrator has opened one or more ports on the network firewall in the security stack SSTK 2175 so that traffic, such as unsolicited inbound tunnel traffic originating from the MBDV 2101 and other mobile endpoints with the MBL-CYBER-APP installed, can enter the TGW 2163. For example, if the tunneling protocol is IPsec tunneling mode, the administrator may have opened well-known IPsec ports 500, 50, and 51. The TGW 2163 can decapsulate and / or decrypt received packets at the tunnel exit. The TGW 2163 can then forward the decapsulated and / or decrypted packets to the private enterprise network ENET 2160. Because these packets have a public Internet address in the destination IP address field of their IP header, routers and / or switches in the private enterprise network ENET 2160 can forward these packets to the Internet access point, and thus to the threat intelligence gateway TIG 2170, which applies a CTI derivation policy to the packets.

[0226] The Security Policy Creation and Management Server SPMS 2141 can collect CTIs from one or more CTIPs, including, for example, CTIPs 2142 and 2143. SPMS 2141 can also aggregate CTIs, create at least one security policy based on the CTIs, and publish the security policies to subscribers, which may include multiple network devices such as host computers and threat intelligence gateways TIG 2170. SPMS 2141 can create a data structure TUNNEL-M / T for each security policy and can publish the data structure TUNNEL-M / T (such as the mobile device security application MBL-CYBER-APP hosted by mobile device MBDV 2101) to each subscriber associated with each security policy. Alternatively, the Threat Intelligence Gateway TIG 2170 can create a data structure TUNNEL-M / T for each associated mobile device (such as an associated instance of the mobile device security application MBL-CYBER-APP associated with the Threat Intelligence Gateway TIG 2170) for network protection, and can publish the current data structure TUNNEL-M / T (including each subscription instance of the mobile device security application MBL-CYBER-APP) to the associated mobile devices.

[0227] FIG. 14 againstFIG. 12 The protection system for mobile devices and associated enterprise networks depicted in the text describes an operational concept flowchart for creating and managing servers to configure security policies for mobile devices and associated enterprise network elements.

[0228] In step 2201, the Security Policy Creation and Management Server SPMS 2141 can download CTIs published by threat intelligence providers, such as CTIP 2142 and CTIP 2143. In step 2202, the Security Policy Creation and Management Server SPMS 2141 can aggregate CTI metrics and process the CTIs to generate a security policy. The Security Policy Creation and Management Server SPMS 2141 can create at least one packet filtering rule, where each rule includes a matching criterion corresponding to a threat metric and a corresponding action / PTF applied to the packet when a match is determined. The corresponding action / PTF can be determined based on multiple factors, including requirements provided by the operator / administrator of the Threat Intelligence Gateway TIG 2170. The Security Policy Creation and Management Server SPMS 2141 can collect each of the generated rules to generate a security policy and can distribute the security policy to multiple security devices, including the Threat Intelligence Gateway TIG 2170. In step 2203, the Security Policy Creation and Management Server SPMS 2141 can generate a TUNNEL-M / T associated with the security policy by collecting all threat metrics (including IP address, domain name, URI, certificate ID, etc.) characterizing each packet filtering rule in the security policy. The Security Policy Creation and Management Server SPMS 2141 can insert keys, including or based on metrics, into one or more rule trees and / or one or more flat hash maps of the TUNNEL-M / T, and can test the rule trees and / or flat hash maps to determine if any rule in the security policy matches the tested packet element.

[0229] In step 2204, SPMS can publish at least one of the security policy and the TUNNEL-M / T data structure to multiple network security devices. The security policy creation and management server SPMS 2141 can transmit at least one of the security policy and the TUNNEL-M / T data structure to multiple network subscribers, which may include the threat intelligence gateway TIG 2170 and multiple mobile devices associated with the enterprise network. The threat intelligence gateway TIG 2170 can download the security policy and configure packet filtering logic using the security policy. In step 2205, the mobile device security application MBL-CYBER-APP on mobile device MBDV 2101 can download the TUNNEL-M / T data structure and configure tunneling logic using the TUNNEL-M / T data structure. If mobile device MBDV 2102 has not yet installed the mobile device security application MBL-CYBER-APP, it cannot download the TUNNEL-M / T data structure and is not protected by the security policy. It can also update policies and the TUNNEL-M / T data structure, and can generate and distribute updates to security policies and the TUNNEL-M / T data structure. Based on such updates, the mobile device security application MBL-CYBER-APP can filter packet tunneling transmissions that previously did not require tunneling through the corporate network to reach their intended destination.

[0230] FIG. 14 against FIG. 14 The protection system for mobile devices and associated enterprise networks described herein depicts an operational concept flowchart for packet filtering within these networks. The flowchart / operational concept assumes that all devices and hosts are configured and operational. Specifically, mobile device MBDV 2101 has downloaded, installed, and configured the mobile device security application MBL-CYBER-APP, and may have already established a tunnel with TGW 2163. Conversely, mobile device MBDV 2102 has not yet configured the mobile device security application MBL-CYBER-APP.

[0231] In step 2211, a user who owns and / or operates a mobile device MBDV (which may be MBDV 2101 or MBDV 2102) can check emails via an email application and can click on an embedded URL linking to a resource on the web server WSVR 2151. The mobile device MBDV 2101 or 2102 can query DNS (… FIG. 15(Not shown in the image) Resolve the hostname or fully qualified domain name (FQDN) of the URL issuing authority to the IP address of the web server WSVR 2151, for example, 12.34.56.78. The mobile device MBDV can attempt to initiate a TCP connection with the web server WSVR 2151 on port 80 (HTTP) by creating a TCP SYN packet with a destination port 80. This TCP packet can be encapsulated in an IP packet, which can be configured to have a destination IP address field set to 12.34.56.78 and a source IP address corresponding to the MBDV.

[0232] In step 2212, before the packet can be forwarded (to the radio access network RNET 2120), the mobile device security application MBL-CYBER-APP of MBDV 2101 can determine each packet element associated with a threat indicator that is a member of the data structure TUNNEL-M / T. For example, the mobile device security application MBL-CYBER-APP can extract an IP address (e.g., 12.34.56.78) from the destination IP address field and can test whether 12.34.56.78 is a member of the data structure TUNNEL-M / T. The membership test returns false based on the determination that there is no matching packet filtering rule in the security policy of the threat intelligence gateway TIG 2170. The mobile device security application MBL-CYBER-APP can determine that there is no threat risk associated with the tested packet data (e.g., the IP address 12.34.56.78 of web server WSVR 2151). The mobile device security application MBL-CYBER-APP can determine that there is no need to tunnel the packet to be filtered. The mobile device security application MBL-CYBER-APP may not forward packets to TGW 2163, threat intelligence gateway TIG 2170, and / or security stack SSTK2175.

[0233] In step 2213, based on the determination that no tunneling is required for the packets to be filtered, mobile device MBDV 2101 can directly forward packets to web server WSVR 2151 via radio access network RNET 2120. Similarly, but without performing any tunneling determination / decision logic, MBDV 2102 can directly forward packets to web server WSVR 2151 via radio access network RNET 2120. Packet forwarding may initiate a TCP handshake and subsequently establish a TCP connection on port 80 of 12.34.56.78. Mobile device MBDV can issue an HTTP GET request to the URL. Web server WSVR 2151 can respond with the requested resource, the web session can be terminated, and the TCP connection can be closed. During this communication session, for MBDV 2101, the membership of relevant packet field values ​​(such as IP address field, domain name field, URI field, etc.) of all inbound and outbound packets in the data structure TUNNEL-M / T can be tested. When each packet in the communication stream is associated with a secure destination, the mobile device security application MBL-CYBER-APP can determine that there is no need to tunnel the packets to be filtered because the test of the data structure TUNNEL-M / T will always return false. Therefore, no packets constituting the session will be tunneled.

[0234] In step 2211, a user who owns and / or operates the mobile device MBDV 2101 may read a spear-phishing email and may be tricked into clicking an embedded URL that links to a resource on the malware server MALSVR2152. This resource may contain a webpage that forges a login page for an enterprise web application on the enterprise systems server ESVR 2162. The mobile device MBDV 2101 can then query DNS (… FIG. 15(Not shown in the image) The hostname or fully qualified domain name (FQDN) of the URL issuing authority is resolved to the IP address of the malware server MALSVR 2152, for example, 87.65.43.21. The mobile device MBDV 2101 may attempt to initiate a TCP connection with the malware server MALSVR 2152 on port 80 (HTTP) by creating a TCP SYN packet with a destination port 80, and may encapsulate the TCP packet in an IP packet with a destination IP address field set to 87.65.43.21. In step 2212, before the device MBDV 2101 forwards the packet to the radio access network RNET 2120, the mobile device security application MBL-CYBER-APP may extract packet elements (e.g., extract the IP address 87.65.43.21 from the destination IP address field) and may test to determine if any packet elements are members of the data structure TUNNEL-M / T.

[0235] Membership testing can return a truth value or other indication that a matching packet filtering rule exists in the security policy associated with the TUNNEL-M / T data structure. The mobile device security application MBL-CYBER-APP can determine, based on the truth value or other indication that a matching packet filtering rule exists in the security policy associated with the TUNNEL-M / T data structure, that the packet needs to be filtered by the associated enterprise threat intelligence gateway TIG 2170 and that there are some threat risks associated with the packet element (e.g., the IP address 87.65.43.21 of the malware server MALSVR 2152). In step 2214, the mobile device MBDV 2101 can transmit the packet through the network tunnel to TGW 2163 based on the truth value or other indication that a matching packet filtering rule exists in the security policy associated with the TUNNEL-M / T data structure. In step 2215, the packet can be received on the enterprise network side of the tunnel, forwarded to the Internet access link via the private enterprise network ENET 2160, and received by the threat intelligence gateway TIG 2170 for packet filtering. In step 2216, the Threat Intelligence Gateway (TIG) 2170 can apply security policies to the packet and determine rules that match packet elements (such as IP address 87.65.43.21). The network protection action / PTF associated with the determined rule can specify that the action(s) can be at least one of blocking / dropping, logging, and / or capturing.

[0236] Alternatively, before transmitting any packet based on a match determined from the TUNNEL-M / T data structure, the system can test auxiliary data structures to reduce the amount of network traffic. For example, the system can test a blocking rule data structure that represents each rule of the policy associated with a blocking action or PTF. Based on truth values ​​or other indications showing the existence of matching packet filtering rules associated with the blocking rule data structure in the security policy, the mobile device security application MBL-CYBER-APP can execute blocking rule actions to prevent packets associated with the blocking rule from reaching their intended destination without tunneling such packets to packet filtering devices in the enterprise network. The blocking rule data structure can be a rule tree, a flat hash map, or a Bloom filter, and it can be a smaller data structure than the TUNNEL-M / T data structure associated with all rules of the policy to be enforced.

[0237] In step 2217, the Threat Intelligence Gateway TIG 2170 can send the logs to the Security Operations Center (SOC) 2140 or other network devices. The SOC 2140 (e.g., a network analyst) can analyze the logs using a SIEM application. Based on the determination of the threat risk associated with the malware server MALSVR 2152 by the CTI(s) provided by multiple CTIPs targeting threat indicators (e.g., IP address 87.65.43.21), the system can take corrective action. For example, the threat analysis device can report the incident to the user of the mobile device MBDV 2101, or recommend that the user watch a cybersecurity training video discussing spear-phishing email attacks and how to avoid them. Similarly, the mobile security application MBL-CYBER-APP can generate logs of threat events and can send the logs to the SOC 2140 or other cybersecurity applications or devices. The mobile device security application MBL-CYBER-APP can selectively generate threat event logs based on indications that there are matching packet filtering rules in the security policy that are associated with the probability data structure of blocking rules.

[0238] FIG. 1 Another example network environment 2100 is shown, in which rule trees and / or flat hash maps can be used as set membership test data structures to implement efficient network protection for (multiple) remote networks and (multiple) associated central networks that may constitute an enterprise network. FIG. 12 The elements shown can also be combined. FIG. 1 and / or FIG. 12 The elements shown describe the operations, and FIG. 15 and / or FIG. 15The elements shown can also be combined. FIG. 15 The described operation. In at least some arrangements, (multiple) remote networks and (multiple) associated central networks can be located in geographically different locations. Note that, although FIG. 15 The document describes an enterprise network consisting of a single remote network RMT-NET 2720 and a single associated central network CENT-NET 2760, but the aspects described herein can be easily extended to multiple remote networks and one or more associated central networks. FIG. 12 The invention is not limited or restricted in any way to the case of a single remote network and a single central network.

[0239] One or more remote networks RMT-NET 2720 may be associated with a central network CENT-NET 2760 operated by the enterprise. Hosts connected to the remote network RMT-NET 2720, such as PCs 2721 and 2722 (e.g., desktop PCs, mobile devices (e.g., connected to a local Wi-Fi access network), etc.), can directly access publicly addressed Internet servers, such as web server WSVR 2751 and malware server MALSVR 2752, via Internet 2730. When directly accessing Internet 2730, hosts connected to the remote network RMT-NET 2720 (e.g., PCs 2721 and 2722) may not be protected by the enterprise security stack SSTK 2775 and threat intelligence gateway TIG2770 associated with the central network CENT-NET 2760, and therefore may not be protected from Internet threats, such as those associated with malware server MALSVR 2752. For protection, enterprises can install an online remote network tunneling gateway RMT-NET-TGW 2725 at or near the network boundary / interface between the remote network RMT-NET 2720 and the Internet 2730. The remote network tunneling gateway RMT-NET-TGW 2725 can implement the packet inspection, packet testing, and packet tunneling functions described herein. The remote network tunneling gateway RMT-NET-TGW 2725 can be configured to tunnel selected packets, manage TUNNEL-M / T data structures, and perform other functions described in other parts of this disclosure. The remote network tunneling gateway RMT-NET-TGW 2725 can be integrated with or otherwise coexist with other network boundary devices, such as network firewalls, VPN tunnels, etc.

[0240] The central network CENT-NET 2760 can provide internet access for internal hosts such as PC 2761 (e.g., desktop PCs, mobile devices connected to a local Wi-Fi access network, etc.). The central network CENT-NET 2760 can be configured such that traffic between these internal hosts and threat internet hosts (e.g., malware server MALSVR 2752) must be routed through the threat intelligence gateway TIG 2770. The threat intelligence gateway TIG 2770 can enforce CTI derivation policies. This traffic can also be routed through the security stack SSTK 2775, which can include at least one of traditional network firewalls and other enterprise network security devices (e.g., web proxies, SSL / TLS proxies, IDS, IPS, packet capture, etc.). The threat intelligence gateway TIG 2770 and the security stack SSTK 2775 can be located at or near the internet access point of the central network CENT-NET 2760. When policies are applied to packets, the Threat Intelligence Gateway TIG 2770 can generate packet logs that match the rules in the policy. These logs can be sent via the Internet to the Security Operations Center (SOC) 2740 for network analysts to use for attack analysis, such as with SIEM applications and packet analyzer applications.

[0241] For example, the Enterprise System Server ESVR 2762 could be a private web application server hosted on a central network for use by enterprise users, including those who own / operate host PCs 2721 and 2722 connected to the remote network RMT-NET 2720. Host PCs 2721 and 2722 can access the Enterprise System Server ESVR 2762 using, for example, a web browser that includes an HTTPS client for accessing the Enterprise System Server ESVR 2762 via port 443 (HTTPS port). When a host connects to the Enterprise System Server ESVR 2762 (via a web browser), the web application can present a login form. The enterprise user can then enter their enterprise credentials to securely access the web application. A network security administrator can open a port (e.g., port 443) in the security stack SSTK 2775 on the network firewall so that connections such as unsolicited inbound HTTPS connections originating from host PCs 2721 and 2722 can initiate sessions with the Enterprise System Server ESVR 2762.

[0242] The central network tunneling gateway TGW 2763, attached to the central network CENT-NET 2760, can terminate and centralize tunnels that can be established with associated remote networks (e.g., RMT-NET 2720). The associated remote networks may have a corresponding tunneling gateway installed (e.g., RMT-NET-GTW 2725) to terminate tunnels connected to tunneling gateway TGW 2763 (via Internet 2730). Similar to the enterprise system server ESVR 2762, network administrators can open one or more ports on the network firewall in the security stack SSTK 2775 to allow unsolicited inbound tunnel traffic (e.g., originating from RMT-NET-GTW 2725) to enter TGW 2763. For example, if the tunneling protocol is IPsec tunneling mode, the administrator may have already opened well-known IPsec ports 500, 50, and 51. Tunneling gateway TGW 2763 can decapsulate and / or decrypt received packets at the tunnel exit. The tunnel gateway TGW 2163 can forward decapsulated and / or decrypted packets to the (private enterprise) central network CENT-NET 2760. Because these packets have a public Internet address in the destination IP address field of their IP headers, routers and / or switches in the private central network CENT-NET 2760 can forward these packets to the Internet access point, and thus to the threat intelligence gateway TIG 2770, which applies a CTI derivation policy to the packets.

[0243] The Security Policy Creation and Management Server SPMS 2741 can collect CTIs from one or more CTIPs, including, for example, CTIPs 2742 and 2743. SPMS 2741 can also create at least one security policy based on the CTI and can publish the security policy to subscribers. Subscribers can include multiple network devices, such as host computers and threat intelligence gateways TIG 2770. SPMS 2741 can create a data structure TUNNEL-M / T for each security policy (as described above) and can publish the data structure TUNNEL-M / T (such as the Remote Network Tunneling Gateway RMT-NET-TGW 2725) to each subscriber associated with each security policy. Alternatively, the Threat Intelligence Gateway TIG 2770 can create a data structure TUNNEL-M / T associated with the Threat Intelligence Gateway TIG 2770 for network protection for each associated remote network tunnel gateway (such as RMT-NET-TGW 2725), and can publish the current data structure TUNNEL-M / T to each associated remote network tunnel gateway.

[0244] In one arrangement, the security policy creation and management server SPMS 2741 can be located outside the central network CENT-NET 2760 and the remote network RMT-NET 2720, and can communicate with the central network CENT-NET 2760 and the remote network RMT-NET 2720 via the Internet 2730. In another arrangement, the security policy creation and management server SPMS 2741 can be attached to either the central network CENT-NET 2760 or the remote network RMT-NET 2720.

[0245] FIG. 12 One or more elements described in the text can be related to... FIG. 12 One or more elements described herein are identical, similar, substantially similar, or may share one or more hardware, software, and / or firmware components. For example, the central network CENT-NET 2760 (and its associated threat intelligence gateway TIG 2770 and security stack SSTK 2775) may be similar to... FIG. 16 The enterprise network ENET 2160 (and its associated threat intelligence gateway 2170 and security stack SSTK 2175) is the same, similar, or substantially similar. Therefore, the threat intelligence gateway 2170 and security stack 2175 can enforce CTI derivation policies not only on tunnel traffic from mobile devices (e.g., MBDV 2101) but also on tunnel traffic from remote network tunnel gateways (e.g., RMT-NET-TGW2725). Similarly, as per [the relevant information]... FIG. 15 The internal host PC 2161, enterprise system server 2162, tunnel gateway 2163, security operations center SOC 2140, security policy creation and management server SPMS 2141, CTI provider CTIP 2142 and 2143, malware server MALSVR 2152, and web server WSVR 2151 may be the same as, similar to, or substantially similar to the internal host PC 2761, enterprise system server 2762, tunnel gateway 2763, security operations center SOC 2740, security policy creation and management server SPMS 2741, CTI provider CTIP 2742 and 2743, malware server MALSVR 2752, and web server WSVR 2751. The mobile device 2101 (and / or the mobile device security application MBL-CYBER-APP) may be the same as, similar to, or substantially similar to the remote network tunnel gateway RMT-NET-TGW 2725, or may share one or more hardware, software, and / or firmware components with it.

[0246] FIG. 16 For use by FIG. 13The protection system for an enterprise network, consisting of a remote network and a central network, is described in the diagram. It includes an operational concept flowchart for configuring security policies for the remote network, associated tunnel gateways, and associated central network elements, as well as for the server's operation concept.

[0247] FIG. 13 A process is described that can be used to collect CTI and use the CTI to generate one or more rule tree data structures and / or one or more flat hash map data structures configured for set membership testing. Optionally, the CTI can also be used to generate one or more probabilistic data structures (e.g., Bloom filters, Cuckoo filters, blocking Bloom filters, XOR filters, etc.) for set membership testing. The data structures(s) generated for set membership testing can be distributed (e.g., as TUNNEL-M / T data structures) to remote devices (e.g., RMT-NET-TGW 2725). In this way, the remote device (e.g., RMT-NET-TGW 2725) can use the received data structures(s) to test packets and determine whether these packets should be transmitted to a central device / network for further analysis. For example, if one or more attributes of a packet exist in / are represented by the data structures(s), the packet can be forwarded to the central device for further analysis. Conversely, if one or more attributes do not exist in or are not represented by the data structure(s), the packet can be forwarded without further analysis.

[0248] In step 2801, the security policy creation and management server SPMS 2741 may receive (e.g., download) CTIs published by threat intelligence providers, such as CTIP 2742 and CTIP 2743. This step can be combined with... FIG. 13 The steps in step 2201 are the same or similar.

[0249] In step 2802, the Security Policy Creation and Management Server SPMS 2741 can aggregate CTI metrics and process the CTIs to generate a security policy. The SPMS 2741 can create at least one packet filtering rule, where each rule includes a matching criterion corresponding to a threat metric and a corresponding action / PTF applied to the packet when a match is determined. The corresponding action / PTF can be determined based on multiple factors, including requirements provided by the operator / administrator of the Threat Intelligence Gateway TIG 2770. The SPMS 2741 can collect each of the generated rules to generate a security policy and can distribute the security policy to multiple security devices, including the Threat Intelligence Gateway TIG 2770. This step can be combined with... FIG. 13The steps in step 2202 are the same or similar.

[0250] In step 2803, the security policy creation and management server SPMS 741 can generate a data structure TUNNEL-M / T associated with the security policy by collecting threat indicators (e.g., by collecting one or more IP addresses, IP address ranges, domain names, URIs, certificate IDs, etc.). This TUNNEL-M / T includes one or more rule tree data structures configured for aggregate membership testing and / or one or more flat hash map data structures configured for aggregate membership testing, as well as optional one or more probabilistic data structures. In this way, TUNNEL-M / T can characterize each packet filtering rule in the security policy. The security policy creation and management server SPMS 2741 can insert indicators (or indicator-based values) into the TUNNEL-M / T data structure, which can then be tested to determine if any rules in the security policy match the tested packet element. In this way, TUNNEL-M / T can be used to efficiently test whether a specific packet should be tunneled to the central network (e.g., for further processing, such as rule-based further testing). This step can be combined with... FIG. 17 The steps in step 2203 are the same or similar.

[0251] The Security Policy Creation and Management Server SPMS 2741 can publish at least one of the security policies and the TUNNEL-M / T data structure to multiple network security devices. The Security Policy Creation and Management Server SPMS 2741 can also transmit at least one of the security policies and the TUNNEL-M / T data structure to multiple network security devices. These multiple network security devices may include a Threat Intelligence Gateway TIG 2770 and / or one or more remote network tunnel gateways (e.g., a remote network tunnel gateway RMT-NET-TGW 2725). For example, as shown in step 2804, the Security Policy Creation and Management Server SPMS 2741 can transmit the security policy to the Threat Intelligence Gateway TIG 2770 associated with the central network CENT-NET 2760. The Threat Intelligence Gateway TIG 2770 can download the security policy and configure packet filtering logic using the security policy. This step can be combined with... FIG. 13 The steps in step 2204 are the same or similar.

[0252] For example, as shown in step 2805, the security policy creation and management server SPMS 2741 can transmit the data structure TUNNEL-M / T to the remote network tunneling gateway RMT-NET-TGW 2725. The remote network tunneling gateway RMT-NET-TGW 2725 can download the data structure TUNNEL-M / T and configure tunnel logic using the data structure TUNNEL-M / T. Based on the security policy and the data structure TUNNEL-M / T, the remote network tunneling gateway RMT-NET-TGW 2725 may or may not transmit the packet tunnel to the central network CENT-NET 2760 for filtering by the threat intelligence gateway TIG 2770 (e.g., as referenced). FIG. 16 (As further described). This step can be combined with... FIG. 17 The steps in step 2205 are the same or similar.

[0253] The security policy and data structure TUNNEL-M / T can also be updated (e.g., by the security policy creation and management server SPMS 2741), and updates to the security policy and data structure TUNNEL-M / T can be generated and distributed (e.g., periodically, based on updates to CTI provided by CTIP). Updates to the security policy and data structure TUNNEL-M / T can alter the operation of network security devices (e.g., the threat intelligence gateway TIG 2770 and / or one or more remote network tunneling gateways). For example, based on such an update, the remote network tunneling gateway RMT-NET-TGW 2725 can filter packet tunneling transmissions that previously did not require tunneling to the central network CENT-NET 2760 to reach their intended destination (e.g., based on a previous version of the data structure TUNNEL-M / T).

[0254] In addition and / or alternatively (but) FIG. 15(Not shown in the image), after receiving a security policy from the security policy creation and management server SPMS2741, the threat intelligence gateway TIG 2770 can generate a data structure TUNNEL-M / T associated with the security policy by collecting all threat metrics (including IP address, domain name, URI, certificate ID, etc.) characterizing each packet filtering rule in the security policy. This data structure includes one or more rule tree data structures configured for aggregate membership testing and / or one or more flat hash map data structures configured for aggregate membership testing, as well as one or more optional probabilistic data structures. The threat intelligence gateway TIG 2770 can insert metrics (or metric-based values) into the TUNNEL-M / T data structure and test it to determine if any rules in the security policy match the tested packet element. The threat intelligence gateway TIG 2770 can then transmit the TUNNEL-M / T to a remote network tunnel gateway (e.g., RMT-NET-TGW 2725) of the associated remote network (e.g., RMT-NET 2725).

[0255] Typically, remote fixed networks (especially remote network tunneling gateways) may not have resource limitations that can be associated with mobile devices (e.g., resource limitations associated with power supply, memory, processing power, etc.). Therefore, a remote network tunneling gateway (e.g., RMT-NET-TGW 2725) can perform one or more of the operations described above related to the security policy creation and management server SPMS 2741 (e.g., generating the data structure TUNNEL M / T). For example, the security policy creation and management server SPMS 2741 can generate a security policy based on multiple CTI metrics and distribute that security policy to the remote network tunneling gateway RMT-NET-TGW 2725. The remote network tunneling gateway RMT-NET-TGW 2725 can then generate the data structure TUNNEL-M / T associated with the security policy. Furthermore, since remote fixed networks and associated remote network tunnel gateways may have more relaxed memory constraints compared to mobile devices, the size of, for example, a given rule tree data structure configured for set membership testing, a given flat hash map data structure configured for set membership testing, and / or a given probability data structure (e.g., a given Bloom filter) can be increased.

[0256] FIG. 17 A flowchart illustrating the operational concepts of packet filtering and efficient backhaul for network protection in a remote network, associated tunnel gateways, and an associated central network is provided. For the case of a remote network (e.g., FIG. 16The remote network RMT-NET 2720 (with multiple connected hosts, such as PC 2721 and PC 2722) can selectively tunnel only packet traffic known to be associated with threats to the central network CENT-NET 2760 using a single tunnel. Otherwise, packets may bypass the tunnel and could instead be forwarded directly to their destination.

[0257] As an introduction, FIG. 15 This describes how a remote device (e.g., RMT-NET-TGW 2725) can use a received TUNNEL-M / T data structure to test packets (e.g., packets associated with Internet communication) and determine whether these packets should be transmitted to a central device / network for further analysis. For example, if one or more attributes of the packet exist in / are represented by the data structure, the packet can be forwarded to the central device for further analysis. Conversely, if one or more attributes do not exist in / are not represented by the data structure, the packet can be forwarded without further analysis (e.g., forwarded to a web server). (See also: [link to relevant documentation]). FIG. 16 The TUNNEL-M / T data structure can be generated by the security policy creation and management server SPMS 2741.

[0258] The received TUNNEL-M / T data structure can correspond to a security policy that includes multiple packet filtering rules. Each packet filtering rule in the security policy can include one or more packet matching criteria and a corresponding rule action (e.g., a network protection action or PTF) to process packets that match the packet matching criteria. At least the first packet filtering rule of the security policy can be automatically generated based on a CTI report provided by an independent CTI provider. At least the second packet filtering rule of the security policy can be automatically generated based on a second CTI report provided by a different independent CTI provider.

[0259] Combination FIG. 14 One or more CTIPs (e.g., CTIP 2742 and / or CTIP 2743) may have identified the malware server MALSVR 2752 as a threat, but may not have identified the web server WSVR 2751 as a threat. Therefore, for example, the IP address of the malware server MALSVR 2752 (or any other threat indicator) could be included in the CTI provided to the security policy creation and management server SPMS 2741. (This is in conjunction with...) FIG. 15A process similar to the one described may result in: (a) a threat intelligence gateway TIG 2770 being configured with a security policy that includes packet filtering rules, wherein the packet filtering rules may specify packet matching criteria corresponding to the IP address (or any other threat indicator) of the malware server MALSVR 2752; and (b) a remote network tunneling gateway RMT-NET-TGW 2725 being configured with a data structure TUNNEL-M / T. The data structure TUNNEL M / T may be generated based on a set of packet matching criteria, which may include the IP address (or any other threat indicator) of the malware server MALSVR 2752 as an element in that set. The set of packet matching criteria may exclude the IP address (or threat indicator) of the web server WSVR 2751 as an element in that set.

[0260] In step 2811a, host PC 2721 can initiate communication with web server WSVR 2751. Host PC 2721 can initiate communication with web server WSVR 2751, for example, by sending a TCPSYN packet P1 containing a destination IP address corresponding to web server WSVR 2751. A user associated with host PC 2721 can initiate communication with web server WSVR 2751, for example, via a web browser or application associated with web server WSVR 2751 installed on host PC 2721. This step can be combined with... FIG. 15 The steps in step 2211 are the same or similar.

[0261] For example, a user operating host PC 2721 can access resources on web server WSVR 2751 using a URL. Host PC 2721 can then query DNS (… FIG. 14 (Not shown in the image) The URL issuing authority's hostname or FQDN is resolved to the IP address of web server WSVR 2751 (e.g., 12.34.56.78). Host PC 2721 can attempt to initiate a TCP connection with web server WSVR 2751 on port 80 (HTTP) by creating a TCP SYN packet with a destination port 80, and the TCP SYN packet can be encapsulated in an IP packet. The IP packet may include a destination IP address field (e.g., set to 12.34.56.78) and a source IP address (e.g., the source IP address of host PC 2721).

[0262] In step 2811b (which does not need to occur at any specific time relative to step 2811a), host PC 722 may initiate communication with malware server MALSVR 2752. Host PC 2722 may initiate communication with malware server MALSVR 2752, for example, by sending a TCP SYN packet contained in packet P2 whose destination IP address corresponds to malware server MALSVR 2752. For example, a malicious application installed on host PC 2722 may initiate clandestine communication with malware server 2752 by sending a TCP SYN. ​​In another example, host PC 2722 may initiate communication with malware server MALSVR 2752 based on a user clicking a link in a malicious email (e.g., a spear-phishing email) or any other form of electronic communication.

[0263] For example, a user operating host PC 2722 might read a spear-phishing email and be tricked into clicking an embedded URL that links to a resource on malware server MALSVR 2752. This resource might contain a webpage that forges a login page for an enterprise web application on enterprise systems server ESVR 2762. Host PC 2722 can then query DNS (… FIG. 16 (Not shown in the image) The URL issuing authority's hostname or FQDN is resolved to the IP address of the malware server MALSVR 2752 (e.g., 87.65.43.21). Host PC 2722 can attempt to initiate a TCP connection with the malware server MALSVR 2752 on port 80 (HTTP) by creating a TCP SYN packet with a destination port 80, and can encapsulate the TCP packet within an IP packet. The IP packet may include a destination IP address field set to 87.65.43.21.

[0264] In step 2812, the remote network tunneling gateway RMT-NET-TGW 2725 may receive packets P1 and / or P2. These packets may originate from the remote network RMT-NET 2720. In steps 2813 and 2814, the remote network tunneling gateway RMT-NET-TGW 2725 may test whether one or more packet elements in the received packets P1 and / or P2 (e.g., packet matching criteria, such as IP address, domain name, URI, certificate ID, or any other packet element) are (e.g., generated in step 2803 and distributed in step 2805) elements in the data structure TUNNEL M / T (e.g., represented therein as a member). The data structure TUNNEL M / T may be associated with a security policy enforced by the threat intelligence gateway TIG 2770. These steps may be related to...FIG. 18 The steps in step 2212 are the same or similar.

[0265] In step 2813, the remote network tunneling gateway RMT-NET-TGW 2725 can determine the corresponding packet matching criteria / packet element for each packet P1 and / or P2. The remote network tunneling gateway RMT-NET-TGW 2725 can test the rule tree data structure and / or flat hash map data structure (e.g., data structure TUNNEL-B / F) against the determined packet matching criteria. The data structure TUNNEL-M / T can represent each packet filtering rule of the security policy configured on the threat intelligence gateway TIG 2770 (e.g., regarding...). FIG. 18 (As described above). Based on packet matching criteria / packet elements not represented in the data structure TUNNEL-M / T, the remote network tunneling gateway RMT-NET-TGW 2725 can forward packet P1 to its intended destination (e.g., web server WSVR 2751). Based on packet matching criteria / packet elements represented in the data structure TUNNEL-M / T, the remote network tunneling gateway RMT-NET-TGW 2725 can tunnel packet P2 to the central network CENT-NET 2760. The central network CENT-NET 2760 can filter packets based on packet filtering rules of a security policy. For example, the threat intelligence gateway TIG 2770 on the central network CENT-NET 2760 can filter packets based on packet filtering rules of a security policy. Further details related to packet tunneling and filtering are described below.

[0266] The remote network tunneling gateway RMT-NET-TGW 2725 can extract packet elements from packet P1. For example, the RMT-NET-TGW 2725 can extract the destination IP address from packet P1, which could be the IP address of web server WSVR 2751. The RMT-NET-TGW 2725 can test whether the packet elements (e.g., the destination IP address) are elements or members of the data structure TUNNEL-M / T. For example, if the packet elements are not elements or members of the data structure TUNNEL-M / T, the membership test can return false. A false membership test could mean that packet P1 does not correspond to any packet filtering rules in the policy enforced by the threat intelligence gateway TIG 2770. Therefore, in step 9814, if the membership test returns false, the RMT-NET-TGW 2725 can forward packet P1 to web server WSVR 2751 via the Internet 2730.

[0267] For example, the remote network tunneling gateway RMT-NET-TGW 2725 can extract the IP address (e.g., 12.34.56.78) from the destination IP address field of packet P1 and can test whether 12.34.56.78 is a member of the data structure TUNNEL-B / F. If the IP address 12.34.56.78 is not a member of the data structure TUNNEL-B / F, the membership test can return false. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can return false based on the membership test to determine that there is no packet filtering rule matching packet P1 in the security policy enforced by the threat intelligence gateway TIG 2770. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can return false based on the membership test to determine that the tested packet data (e.g., the IP address 12.34.56.78 of web server WSVR 2751) does not pose a threat risk. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine that tunneled packet P1 is unnecessary to filter (on the central network CENT-NET 2760) based on a false return from a membership test. Therefore, the remote network tunneling gateway RMT-NET-TGW 2725 may not forward packet P1 to TGW 2763, threat intelligence gateway TIG 2770, and / or security stack SSTK 2775.

[0268] In step 2815, the remote network tunneling gateway RMT-NET-TGW 2725 can extract packet elements from packet P2. For example, the RMT-NET-TGW 2725 can extract the destination IP address from packet P2, which is the IP address of the malware server MALSVR 2752. The RMT-NET-TGW 2725 can test whether the packet elements (e.g., the destination IP address) are elements or members of the data structure TUNNEL-M / T. For example, if the packet elements are elements or members of the data structure TUNNEL-M / T, the membership test can return true. A true membership test can mean that packet P2 corresponds to packet filtering rules in the policy enforced by the threat intelligence gateway TIG 2770. Therefore, in step 2816, the RMT-NET-TGW 2725 can tunnel packet P2 to tunneling gateway TGW 2763, which can terminate at the central network CENT-NET 2760.

[0269] For example, the remote network tunneling gateway RMT-NET-TGW 2725 can extract the IP address (e.g., 87.65.43.21) from the destination IP address field of packet P2 and can test whether 87.65.43.21 is a member of the data structure TUNNEL-M / T. Based on the IP address 87.65.43.21 being a member of the data structure TUNNEL-M / T, the membership test can return a true value. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine, based on the membership test returning true, that at least one packet filtering rule matching packet P2 may exist in the security policy enforced by the threat intelligence gateway TIG 2770. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine, based on the membership test returning true, that the tested packet data (e.g., the IP address 87.65.43.21 of the web server MALSVR 2752) poses a certain threat risk. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine that packet P2 needs to be tunneled for further processing (e.g., filtering) at the central network CENT-NET 2760 based on a true return from a membership test. Therefore, the remote network tunneling gateway RMT-NET-TGW 2725 can forward packet P2 to TGW 2763, threat intelligence gateway TIG 2770, and / or security stack SSTK2775.

[0270] Although steps 2813 and 2815 describe performing a membership test using the destination IP address of the packet, any other element (or multiple elements) in the packet that can correspond to the element inserted into TUNNEL-M / T may also be used.

[0271] Additionally and / or alternatively, before transmitting any packet based on a match determined from the TUNNEL-M / T data structure, the remote network tunneling gateway RMT-NET-TGW 2725 can test auxiliary data structures to reduce the amount of network traffic. For example, the RMT-NET-TGW 2725 can test a blocking rule data structure that represents each rule of the security policy associated with a blocking action or PTF. Based on a truth value or other indication indicating the presence of a matching packet filtering rule associated with the blocking rule data structure in the security policy, the RMT-NET-TGW 2725 can execute a blocking rule action to prevent packets associated with the blocking rule from reaching their intended destination without tunneling such packets to the tunneling gateway TGW 2763. The blocking rule data structure can be a rule tree data structure configured for set membership testing, a flat hash map data structure configured for set membership testing, a Bloom filter, and / or any other probabilistic data structure.

[0272] For example, referring to the example above where it is determined that the packet element of packet P2 is an element or a member of the data structure TUNNEL-M / T, the remote network tunneling gateway RMT-NET-TGW 2725 can extract a second packet element of packet P2. For example, the second packet element could be a URI associated with packet P2. The remote network tunneling gateway RMT-NET-TGW 2725 can extract the URI of packet P2 and can test whether that URI is a member of the blocking rule data structure BLOCK-M / T. Based on the URI being a member of the data structure BLOCK-M / T, the membership test can return a true value. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine, based on the membership test returning true, that at least one packet filtering rule matching packet P2 exists in the security policy enforced by the threat intelligence gateway TIG 2770 for blocking packets. For example, the remote network tunneling gateway RMT-NET-TGW 2725 can determine, based on the membership test returning true, that the tested packet data of packet P2 poses a high threat risk. The remote network tunnel gateway RMT-NET-TGW 2725 can prevent packet P2 from leaving the remote network RMT-NET 2720 based on a true return from a membership test.

[0273] Performing membership tests on packets on the RMT-NET-TGW 2725 remote network tunneling gateway (e.g., testing whether the packet element associated with the packet is a member of the TUNNEL-M / T data structure) reduces the processing requirements of hosts (e.g., PC 2721 and PC 2722). Hosts do not need to perform membership tests, thus improving host resource availability. Furthermore, the higher available resources on the RMT-NET-TGW 2725 remote network tunneling gateway enable faster packet processing.

[0274] In step 2817, packet P2 can leave / be sent from the tunnel and be forwarded via the central network CENT-NET2760 to the malware server MALSVR 2752 (corresponding to the destination IP address of P2). On the path to the malware server MALSVR 2752, packet P2 can be received by the threat intelligence gateway TIG 2770.

[0275] In step 2818, the threat intelligence gateway TIG 2770 can apply a security policy to packet P2. The applied security policy may include packet filtering rules with matching criteria corresponding to packet elements (e.g., destination IP address) of packet P2 (e.g., the IP address of MALSVR 2752). The processing of the matching rules (e.g., network protection actions, PTF, or rule actions) may involve blocking the packet, and the instruction may be to log and capture the packet. Therefore, packet P2 may be blocked, logged, and / or captured, thereby protecting PC 2722 and the enterprise network from any malware or malicious actions that may result from connecting to the malware server MALSVR 2752.

[0276] In step 2819, the log of packet P2 can be sent to the Security Operations Center (SOC) 2740, where one or more network analysis and threat awareness applications can process the log. For example, this processing may lead to remedial actions, such as a malware scan of PC 2722, or any other remedial and / or protective actions.

[0277] Set membership testing can also be used to prevent attacks associated with the Internet Domain Name System (DNS). DNS requests can be used as a medium for certain types of attacks, which, if successful, can cause damage and / or loss to network assets. For example, an attack could include DNS query request messages containing domain names that are not registered in the DNS (e.g., unregistered and / or unrecorded). Such requests, especially in large numbers, can disrupt the performance of DNS servers. Set membership testing can be used to check whether the domain names contained in an incoming request are registered (e.g., registered and / or recorded). If the domain names contained in the incoming request are not members of a set of registered (e.g., registered and / or recorded) domain names, appropriate actions can be taken (e.g., dropping the request to prevent the attack). As mentioned above, such set membership testing can be performed using flat hash map data structures and rule tree data structures.

[0278] FIG. 1 Another example of a network environment 3100 is shown, in which a rule tree and / or a flat hash map can be used as a data structure for set membership testing. FIG. 12 The elements shown can also be combined. FIG. 15 , FIG. 1 and / or FIG. 12 The elements shown describe the operations, and FIG. 15 , FIG. 18 and / or FIG. 18 The elements shown can also be combined. FIG. 18 The described operation. Network environment 3100 may include a DNS attack prevention system for protecting the Domain Name System (DNS) of interconnected networks (such as the public Internet and private networks that can be interconnected via the public Internet) from attacks that may use spoofed DNS requests as attack vectors. Network 3110 may be the public Internet, which may interconnect networks 3112, 3114, 3116, 3118, and 3120.

[0279] Network 3112 may be a private enterprise network with multiple hosts. These hosts may include at least one "Internet of Things" (IoT) device, IOT 3140 (such as a security camera or device that can be remotely managed and operated via the Internet), and a Personally Identifiable Information (PII) host, PII 3141. Host IOT 3140 and other IoT devices (not shown) attached to network 3112 may be infected with malware that could generate DNS attacks by issuing numerous fake DNS requests with domain names designed to attack specific authoritative name servers (such as host NS 3144 attached to network 3116, as described below). Host PII 3141 represents a personal computer or server that manages sensitive information such as Personally Identifiable Information (PII), credit card information, etc. Host PII 3141 may be infected with malware that collects credit card information, and the infected host may use DNS tunneling methods to leak information to the collection server CSVR3148 (described below) attached to network 3170. Both of these attacks could send requests with domain names that may not be registered in the DNS (e.g., unregistered and / or undocumented).

[0280] The configuration of network 3114 and its attached hosts IOT 3142 and PII 3143 may be similar to that of network 3112 and hosts IOT 3140 and PII 3141, and may be subject to similar harm. However, network 3114 may be configured with a DNS gatekeeper, DNS-G / K 3150, located at or near the boundary between network 3114 and the Internet 3110. Network 3114 may be configured such that DNS-G / K 3150 filters all DNS requests crossing the boundary of network 3114 and can control which DNS requests can enter the DNS.

[0281] Network 3116 may be operated by the DNS infrastructure provider organization DIPO-X. Authoritative name servers NS 3144 and NS 3145 may be the authoritative DNS name servers for the domains example-3144.com and example-3145.com, and DIPO-X may operate these servers on behalf of the associated registrar organization. Name servers NS 3144 and NS 3145 can both be configured to provide their zone files to authorized requesters, which may include a list of all registered domain names managed by these servers. Network 3116 may have a DNS gatekeeper, DNS-G / K 3180, located at or near the boundary between network 3116 and Internet 3110. Network 3116 can be configured so that the DNS gatekeeper DNS-G / K 3180 filters all DNS requests crossing the boundary and can control which DNS requests can reach the DNS name servers for example-3144.com (authoritative name server NS 3144) and example-3145.com (authoritative name server NS 3145).

[0282] Network 3118 can be operated by a DNS infrastructure provider organization (e.g., DIPO-Y). Authoritative name servers NS 3146 and NS 3147 can be the authoritative name servers for the domains example-3146.com and example-3147.com, and DIPO-Y can operate these servers on behalf of an associated registrar company. Both name servers NS 3146 and NS 3147 can be configured to provide their zone files to authorized requesters, which may include a list of all registered domains managed by these servers. Network 3118 does not require a DNS gatekeeper to control which DNS requests can be directed to the DNS name servers for example-3146.com (authoritative name server NS 3146) and example-3147.com (authoritative name server NS 3147).

[0283] Network 3170 may be controlled by at least one malicious actor. Network 3170 may include a data collection server host CSVR 3148, which can collect data leaked through spoofed DNS requests, and may include a botnet command and control host system CNC 3149, which can execute botnet commands and control processes instructing infected devices to launch DDoS attacks against DNS. Malicious data collection processes executing on the data collection server CSVR 3148 can receive spoofed DNS requests through port 53 (a well-known DNS port). For example, in the following example, the public IP address of the data collection server CSVR 3148 could be 21.43.65.87. A network firewall (not shown) associated with network 3170 can be configured to allow inbound packets destined for the data collection server CSVR 3148 at port 53. Network 3170 may not have a DNS gatekeeper to control access to DNS.

[0284] The DNS Filter Manager (DFM) 3160 can function as a system management server or as a host attached to a private network (not shown) connected to the Internet 3110 interface. The DFM 3160 can continuously obtain data from all authoritative DNS name servers (e.g., authoritative DNS name servers NS 3144-3147) and / or from associated domain name list aggregators (…). FIG. 18 (Not shown in the diagram) and a list of registered (e.g., registered and / or recorded) domain names collected from other registered (e.g., registered and / or recorded) domain name sources. DFM 3160 can encode all registered (e.g., registered and / or recorded) domain names and can store all of these domain names in at least one data structure, DNS-REG. At least one data structure, DNS-REG, can include one or more flat hash maps configured for set membership testing (e.g., for testing membership in a set of registered (e.g., registered and / or recorded) domain names) and / or one or more rule tree data structures configured for set membership testing. Optionally, at least one data structure, DNS-REG, can also include a Bloom filter or other probabilistic set data structures for set membership testing. DFM 3160 can distribute the DNS-REG data structures to subscribing DNS gatekeepers (e.g., DNS-G / K 3150 and DNS-G / K 3180).

[0285] DFM 3160 may also include functionality for managing information provided or uploaded by subscription DNS gatekeepers (e.g., DNS-G / K 3150 and 3180). This information may include unregistered (e.g., unregistered and / or undocumented) domain names detected by the subscription DNS gatekeepers DNS-G / K 3150 and 3180, along with associated metadata (e.g., attack type estimates, timestamps, rate estimates, source host, DNS-G / K ID, etc.), which can be used as threat intelligence to identify or detect other attacks. This information may also include statistics on DNS requests and responses observed by DNS-G / K 3150 and 3180, as well as element management information for DNS-G / K 3150 and 3180. DFM 3160 may also include functionality for managing information provided to subscription applications ( FIG. 19 (Not shown in the text) such as threat intelligence providers, network analysis applications, network element managers, DNS monitoring applications, DNS protection applications, Internet protection applications, etc., provide the ability to provide information collected from DNS-G / K 3150 and 3180.

[0286] FIG. 19 (and discussed below) FIG. 20 The DNS-G / K 3150 and DNS-G / K 3180 shown in the diagram can be all or part of a network device (such as TIG), but are not required to be. Typically, the DNS-G / K 150 and DNS-G / K 180 can be executed on any computing device (including, but not limited to, those computing devices discussed above or below).

[0287] FIG. 21A representative system component diagram of the DNS-G / K 3150 and / or DNS-G / K 3180 supporting various aspects of this disclosure is shown. The DNS-G / K 3150 components may include a processor CPU 3520 capable of executing logic for configuring and operating the DNS-G / K 3150; network interfaces 3512 and 3514 for respectively interfaced to networks 3114 and 3110; a management interface MGMT I / F 3540 connected to network 3110; a main memory module MAIN-MEMORY 3530, which may contain a data structure DNS-REG 3531; and a memory storage device MEMORY-STORE 3532 for persistent storage. These components may be connected to a bus BUS 3510, which can be used to transfer information between the components of the DNS-G / K 3150. DNS-REG 3531 can be provided to DNS-G / K 3150 via DFM 3160 through management interface 3540 and network 3110. System components of DNS-G / K 3150 can be any combination of the following: processes or applications executing on the same host (e.g., coexisting), processes executing on different hosts, processes executing on virtual infrastructure (e.g., hypervisors), or other arrangements of components and software. Similarly, bus 3510 can be not only an integrated / embedded data bus on a printed circuit board (PCB), but also any combination of, for example, L2 / L3 switching networks, L3 routing networks, L2 / L3 network links connecting logical components, etc. The data bus can be any combination of wireless, physical, logical, virtual, software-defined, etc. Network interfaces 3512 and 3514 can be L3 transparent. Network interfaces 3512 and 3514 may not be assigned IP addresses and may not participate in the L3 routing process. Similarly, network interfaces 3512 and 3514 can be L2 transparent. Network interfaces 3512 and 3514 may not be assigned MAC addresses and therefore may not participate in the ARP process. The management interface MGMT I / F 3540 may be assigned an IP address for communication with management devices (such as DFM 3160).

[0288] FIG. 20This is a flowchart illustrating a representative operational concept of DNS-G / K 3150. In step 3601, DNS-G / K 3150 can select a data structure (e.g., a flat hash map) DNS-REG 3531 provided by DFM 3160, and can download DNS-REG 3531 and store it in its main memory MAIN-MEMORY 3530. DNS-REG 3531 may contain keys for all domain names currently registered in the DNS (e.g., registered and / or documented), which are available to or determined by DFM 3160. DFM 3160 can provide multiple versions of DNS-REG 3531 for use by the DNS gatekeeper, which can be characterized by corresponding memory requirements. The version of DNS-REG 3531 can also be characterized by its creation time, and the selection and download of DNS-REG 3531 can be part of the DNS-G / K 3150 update process. The DNS-G / K 3150 can select a version of DNS-REG 3531 based on the corresponding memory requirements for storing DNS-REG 3531. In step 3601, the DNS-G / K 3150 can also upload the information collected in steps 3603, 3607, and 3608 to the DFM 3160. This upload can be part of an update process. The upload may include malicious, fraudulent, or unregistered (e.g., unregistered and / or undocumented) domain names detected by the DNS-G / K 3150, along with associated metadata (e.g., timestamps, source host identifiers, estimated attack types, rate estimates, etc.), which can be used, for example, for threat intelligence and threat intelligence reporting. Uploaded data may include statistics on DNS requests and responses observed by DNS-G / K 3150, as well as domain names that DNS-G / K 3150 determines may be registered in the DNS (e.g., registered and / or documented) but may not be included in the local instance DNS-REG 3531 (e.g., due to synchronization lag). FIG. 21 In step 3612, DNS-G / K 3150 can determine such a domain name (as described below). Other information can also be uploaded, such as DNS-G / K 3150 element management information.

[0289] In step 3602, DNS-G / K 3150 can receive packets in transit from network 3114, which are expected to be forwarded to network 3110. DNS-G / K 3150 can also process packets transmitted in the reverse direction from network 3110 to network 3114, but for ease of illustration, only one direction is described. In step 3603, DNS-G / K 3150 can determine whether the packet contains a DNS query request. If the packet does not contain a DNS query request, then in step 3604F, the packet can be forwarded to network 3110, and DNS-G / K 3150 will loop back to step 3602 to process additional packets. If the packet does contain a DNS query request, then in step 3604T, the domain name that may be contained in the QNAME field can be extracted.

[0290] In step 3605, DNS-G / K 3150 may test whether a domain name is a member of the set included in DNS-REG 3531. DNS-REG 3531 may include a key to the set of all domain names currently registered (e.g., registered and / or recorded) in the DNS, based on the latest update of DFM 3160. When the membership test returns true, or it is determined that the name is registered in the DNS (e.g., registered and / or recorded), in step 3606T, the packet and associated DNS requests and metadata may be logged for analytical and statistical purposes and stored in MEMORY-STORE 3532. In addition, DNS-G / K 3150 can update the ratio of DNS request rates for bad, fake, or unregistered (e.g., unregistered and / or undocumented) domains to all DNS request rates, and / or the rate of change of DNS request rates for bad, fake, or unregistered (e.g., unregistered and / or undocumented) domains to all DNS request rates. The packet can be forwarded to network 3110, and DNS-G / K 3150 can loop back to step 3602 to process the additional packet.

[0291] When a membership test returns false, and / or it is determined that the name is not registered in the DNS (e.g., unregistered and / or undocumented), and it is determined that the DNS request is associated with a bad, fake, or unregistered (e.g., unregistered and / or undocumented) domain name, the packet may be dropped or blocked in step 3606F. The packet and its associated information and metadata (e.g., domain name, time, location, direction, source, attack type estimate, rate estimate, etc.) may be logged for statistical purposes. The packet and its associated information and metadata may be recorded and / or captured, and the packet and its associated information and metadata may be stored in MEMORY-STORE 3532. Subsequently, this data may be transferred to or uploaded to DFM 3160 according to step 3601 above. The packet and its associated information and metadata may be made available for network analysis applications, threat intelligence applications, and network protection applications, such as applications for securing and protecting global DNS and the global Internet. FIG. 21 (Not shown in the image). For example, during a high-rate DDoS attack using fake DNS requests as the attack medium, logging and / or capturing packets associated with the attack can consume a very large amount of computing resources. Therefore, it is possible to instead collect statistics that require fewer resources. Network analysis applications can identify infected hosts that may be initiators of fake DNS requests. Packets and their associated information and metadata can be provided to a network threat intelligence database, and subsequently, the packets and their associated information and metadata can be used by applications and systems that protect the network. For example, DNS infrastructure providers and Internet service providers can jointly use this data to drive global DNS and global Internet protection applications.

[0292] In step 3607, the DNS request rate for malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain names can be updated to incorporate the occurrence of DNS requests for such domain names. DNS-G / K 3150 can determine or calculate the occurrence rate of DNS requests for malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain names, or the occurrence rate of all DNS requests. DNS-G / K 3150 can determine or calculate the rate of change of fake DNS request occurrences, or the rate of change of all DNS request occurrences. In step 3608, the DNS request rate and / or the rate of change of DNS request rates can be used to determine the type of attack associated with DNS requests for malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain names, and to determine the possible DNS responses (if any). In step 3608, DNS-G / K 3150 can determine what type of DNS query response (if any) to generate in response to a malicious, fraudulent, or unregistered (e.g., unregistered and / or undocumented) domain name DNS query request, and can forward any generated DNS response to network 3114. When generating a DNS response, DNS-G / K 3150 may forge the actual DNS response procedure and may send the forged response to the initiator of the associated DNS request and any intermediate device that may observe the DNS request. To the requesting device, the forged response may appear to be a response from an authorized name server in the DNS.

[0293] Factors used to determine DNS responses may include: the current estimated rate of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain name requests and legitimate DNS requests; the rate of change of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain name requests and legitimate DNS requests; the false positive rate P of the data structure (e.g., DNS-REG 1531); the estimated hysteresis (in cases where the domain name includes a country code in its suffix); an estimate of the attack type of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain name DNS requests as an attack vector; and / or the syntactic properties of the domain name (e.g., information entropy value, relevance to human language vocabulary, tag length, alphanumeric characteristics, etc.). For example, based on the current value of the rate of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain name DNS requests and / or the rate of change of the rate of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domain name DNS requests, DNS-G / K 3150 may determine that a high-volume DDoS attack is occurring. Based on the determination that an attack is occurring, the DNS-G / K 3150 can determine not to generate any DNS response to send back to the source of the DNS request. During a DDoS attack, generating DNS responses (e.g., responses with the NXDOMAIN response code) for DNS requests of malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domains can exacerbate the attack by consuming processing resources or filling the DNS cache in intermediate DNS proxies with NXDOMAIN responses to DNS requests for malicious, fake, or unregistered (e.g., unregistered and / or undocumented) domains. This results in legitimate DNS requests and associated responses not being stored in the DNS cache, thus placing a greater load on the DNS.

[0294] In another example, non-DDoS attacks, such as DNS tunneling attacks or leaks, may occur based on the current value of the DNS request rate for bad, fake, or unregistered (e.g., unregistered and / or undocumented) domain names and the rate of change of such DNS request rates. Such attacks should be prevented, although the likelihood of them excessively consuming processing resources or filling DNS caches is low. The DNS-G / K 3150 can determine that a non-DDoS attack, such as a DNS tunneling attack or leak, may be occurring. Based on this determination, the DNS-G / K 3150 can generate a DNS response with the response code NXDOMAIN and can transmit this response to the source of the DNS request. The DNS cache in any intermediate DNS proxy can store the NXDOMAIN response generated by the DNS-G / K 3150, and subsequent DNS requests for bad, fake, or unregistered (e.g., unregistered and / or undocumented) domain names, including retries of previous DNS requests for bad, fake, or unregistered (e.g., unregistered and / or undocumented) domain names, can be handled by the intermediate DNS proxy. Conversely, DNS-G / K 3150 can decide not to generate a DNS response and instead transmit it back to the source of the DNS request, for example, to prevent the source from realizing that it may have been discovered or otherwise identified as a malicious actor.

[0295] In another example, DNS-G / K 3150 can determine that a bad, spurious, or unregistered (e.g., unregistered and / or undocumented) DNS request for a domain name may be caused by human error while running an application (e.g., a web browser) that generates the DNS request, or by simulated human error generated by malware. Based on this determination, DNS-G / K 3150 can generate a DNS response with the response code NXDOMAIN. Based on the response code, the application may not time out while waiting for a response and may not automatically reissue the same bad, spurious, or unregistered (e.g., unregistered and / or undocumented) DNS request for the domain name. For example, a web browser that generates an erroneous DNS request may repeat this retry loop multiple times if it does not receive a DNS response to the DNS request, which could cause the browser to become temporarily unusable by a human operator until the browser gives up. Repeatedly issuing the same bad DNS request can be considered a denial-of-service attack. DNS-G / K 3150 generates and sends a DNS response with the response code NXDOMAIN to prevent the repeated issuance of the same bad DNS request.

[0296] FIG. 20 This is a flowchart of the operation of a DNS gatekeeper (e.g., DNS-G / K 3150). FIG. 21 Steps 3611 to 3614 in the middle FIG. 20Steps 3601 to 3604 are essentially the same; therefore, these steps will not be repeated here or in the accompanying drawings. Other steps relate to processing DNS requests, even... FIG. 21 Even if the domain membership test in step 3605 returns false, DNS-G / K 3150 can still forward these requests to the DNS.

[0297] As mentioned above, due to lag (such as country code lag), some requests may involve domain names not included in DNS-REG 3531. For example, there may be gaps in the set of all country code domains among domain names registered (e.g., registered and / or recorded) in the local DNS-REG 3531, and the country code domain CCD-X registered (e.g., registered and / or recorded) in the global DNS may not be included. FIG. 22 In step 3601, the DNS-REG 3531 distributed by DFM 160 is used. Similarly, for example, if DNS-G / K 3150 determines that the local DNS-REG 3531 stored in DNS-G / K 3150 has not yet been updated based on data that can be received from DFM 3160, and the time has been relatively long, then synchronization lag may be a factor. Furthermore, because the legitimate information exchange service uses unregistered (e.g., unregistered and / or undocumented) domain names, some requests may relate to domain names not in DNS-REG 3531.

[0298] In step 3605, the membership test of CCD-X in DNS-REG 3531 may return a false value, and in step 3606F, the associated packet may be dropped. However, when the packet is dropped, DNS-G / K 3150 does not gather any new information. As an alternative to dropping, if it is determined that there is sufficient probability that the domain name is actually registered in the DNS (e.g., registered and / or recorded), or for other possible reasons, such as intelligence gathering, or because it can be determined that an unregistered (e.g., unregistered and / or unrecorded) domain name is part of a legitimate information exchange service, DNS-G / K 3150 may decide not to drop the DNS request packet, but instead forward the DNS request packet to the DNS. The DNS-G / K 3150 can monitor and observe corresponding DNS responses, which may contain resolved IP addresses, indicating that the domain name is registered in the DNS (e.g., registered and / or recorded), or contain error codes such as NXDOMAIN, indicating that the domain name is not registered in the DNS (e.g., not registered and / or not recorded); or there may be no response, which could indicate a DNS tunneling attack. G...

Claims

1. A method comprising: receiving, by a packet filtering device, a plurality of packets transmitted from one or more hosts in a first network to one or more hosts in a second network, wherein: the packet filtering device is located at a boundary between the first network and the second network and stores a flat hash map index data structure associated with a plurality of rules of a network security policy enforced by the packet filtering device, each rule of the plurality of rules includes a match criterion that indicates one or more threat indicators associated with packets to which the rule applies, and the flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of the rule identifiers corresponding to a rule of the plurality of rules, and each of the keys includes a value that is incrementally generated based on a portion of a threat indicator in a match criterion of the rule corresponding to the rule identifier to which the key is mapped; extracting values from fields of the received plurality of packets, and searching the flat hash map index data structure based on the extracted values to find rule identifiers of rules of the plurality of rules that apply to the received plurality of packets, wherein each of the extracted values includes a domain name, and wherein, for each packet of the received plurality of packets, the searching includes: for each label of a plurality of labels of the domain name extracted from the packet, generating, based on the label, a value for a portion of the extracted value that includes the label, comparing each of the generated values to a key of the flat hash map index data structure, and based on the comparison of the generated values to the keys of the flat hash map index data structure, determining whether one or more rules of the plurality of rules apply to the packet; and for one or more packets of the plurality of packets, applying one or more rules of the plurality of rules that are determined to apply to the one or more packets, wherein the applying includes one or more of: blocking or dropping a packet, forwarding a packet, logging a packet, capturing a packet, redirecting or re-routing a packet, modifying or transforming a packet, or generating or sending a response to the packet.

2. The method of claim 1, wherein, for each packet of the plurality of packets, the generating includes: generating a hash of a first label that indicates a top-level domain of a domain name extracted from the packet, and for each additional label of one or more additional labels that indicate one or more subdomains of the domain name extracted from the packet, generating, based on the additional label, a hash of a portion of the domain name that includes the additional label and the top-level domain.

3. The method of claim 1, wherein, for each packet of the plurality of packets, the generating includes: generating a lossless compression of a first label that indicates a top-level domain of a domain name extracted from the packet, and for each additional label of one or more additional labels that indicate one or more subdomains of the domain name extracted from the packet, generating, based on the additional label, a lossless compression of a portion of the domain name that includes the additional label and the top-level domain.

4. The method of claim 1, wherein, The value extracted from a first packet of the one or more packets includes a uniform resource locator (URL) that includes a domain name as a host portion and includes a path portion having one or more path segments, and wherein, for the first packet, the generated value includes: for each path segment of the one or more path segments, a hash of a portion of the URL that includes the path segment and the domain name of the URL based on the path segment.

5. The method of claim 1, wherein, The flat hash map index data structure includes: keys that include values that are incrementally generated based on threat indicators having only domain names, and keys that include values that are incrementally generated based on threat indicators having uniform resource locators (URLs) that include domain name portions and path portions.

6. The method of claim 1, wherein, The packet filtering device stores a second flat hash map index data structure associated with a second plurality of rules of the network security policy, and wherein: the plurality of keys of the flat hash map index data structure are associated with threat indicators that include domain names having a first characteristic, and the second flat hash map index data structure includes a second plurality of keys that map to a second plurality of rule identifiers, the second plurality of keys including values that are incrementally generated based on portions of threat indicators in matching criteria that include domain names having a second characteristic that is different from the first characteristic.

7. The method of claim 1, wherein, The packet filtering device stores a second index data structure associated with a second plurality of rules of the network security policy, wherein the second index data structure includes a plurality of internal nodes, and wherein, for each internal node of the plurality of internal nodes: the internal node includes a first bit array, a second bit array, and one or more pointers, the first bit array includes elements that map to possible values of a k-bit block of search objects, each of the first bit array elements stores a value that indicates, for the possible value mapped to the first bit array element, one of: a presence of a corresponding descendant internal node of the plurality of internal nodes, or an absence of a corresponding descendant internal node, the second bit array includes elements that map to possible values of the k-bit block and possible values of a portion of the k-bit block, each of the second bit array elements stores a value that indicates, for the possible value mapped to the second bit array element, one of: a presence of one or more corresponding rule identifiers, or an absence of a corresponding rule identifier, and the one or more pointers indicate memory locations associated with one or more rule identifiers corresponding to the k-bit block.

8. The method of claim 7, wherein, The second index data structure maps keys that include at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, the method further comprising: searching the second index data structure to find a rule identifier of a rule of the second plurality of rules that is applicable to a first packet received by the packet filtering device based on an IP address extracted from a field of the first packet; and based on the search and in an order prioritized by associated rule precedence, storing a first rule identifier associated with a first of the plurality of internal nodes and a second rule identifier associated with a second of the plurality of internal nodes that is a descendant node of the first internal node.

9. The method of claim 7, wherein, The second index data structure is to map keys comprising at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, the method further comprising: based on an IP address extracted from a field of a first packet received by the packet filtering device, searching the second index data structure to find a rule identifier of a rule of the second plurality of rules that is applicable to the first packet; and based on the search and in an order prioritized by associated rule precedence, storing a first rule identifier associated with a first of the plurality of internal nodes and a second rule identifier associated with the first internal node.

10. The method of claim 1, wherein, The search comprises: in a first processing thread and by searching the set data structure based on a value extracted from a first packet of the plurality of packets, determining whether the flat hash map index data structure includes a first key corresponding to the value from the first packet; in a second processing thread executed concurrently with the first processing thread, initiating a search of the flat hash map index data structure based on the value extracted from the first packet; and based on a determination in the first processing thread that the flat hash map index data structure lacks a key corresponding to the value extracted from the first packet, terminating the search of the flat hash map index data structure.

11. A packet filtering device comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, configure the packet filtering device to: receive a plurality of packets transmitted from one or more hosts in a first network to one or more hosts in a second network, wherein: the packet filtering device is located at a boundary between the first network and the second network and stores a flat hash map index data structure associated with a plurality of rules of a network security policy enforced by the packet filtering device, each rule of the plurality of rules includes a match criterion that indicates one or more threat indicators associated with a packet to which the rule applies, and the flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of the rule identifiers corresponding to a rule of the plurality of rules, and each of the keys including a value incrementally generated based on a threat indicator in a match criterion of a rule corresponding to the rule identifier to which the key is mapped; extract values from fields of the received plurality of packets and search the flat hash map index data structure based on the extracted values to find rule identifiers of rules applicable to the received plurality of packets, wherein each of the extracted values comprises a domain name, and wherein, for each packet of the received plurality of packets, the search comprises: for each of a plurality of labels of the domain name extracted from the packet, based on the label, generating a value for a portion of the extracted value that includes the label, comparing each of the generated values to keys of the flat hash map index data structure, and based on the comparison of the generated values to the keys of the flat hash map index data structure, determining whether one or more of the plurality of rules apply to the packet; and applying, for one or more of the plurality of packets, one or more of the plurality of rules determined to apply to the one or more packets, wherein the applying includes one or more of: blocking or dropping a packet, forwarding a packet, logging a packet, capturing a packet, redirecting or re-routing a packet, modifying or transforming a packet, or generating or sending a response to the packet.

12. The packet filtering device of claim 11, wherein, for each of the plurality of packets, the generated value includes: generating a hash of a first label that indicates a top-level domain of a domain name extracted from the packet, and for each of one or more additional labels that indicate one or more subdomains of the domain name extracted from the packet, based on the additional label, generating a hash of a portion of the domain name that includes the additional label and the top-level domain.

13. The packet filtering device of claim 11, wherein, for each of the plurality of packets, the generated value includes: generating a lossless compression of a first label that indicates a top-level domain of a domain name extracted from the packet, and for each of one or more additional labels that indicate one or more subdomains of the domain name extracted from the packet, based on the additional label, generating a lossless compression of a portion of the domain name that includes the additional label and the top-level domain.

14. The packet filtering device of claim 11, wherein, the value extracted from a first packet of the one or more packets includes a uniform resource locator (URL) that includes a domain name as a host portion and includes a path portion having one or more path segments, and wherein, for the first packet, the generated value includes: for each of the one or more path segments, based on the path segment, generating a hash of a portion of the URL that includes the path segment and the domain name of the URL.

15. The packet filtering device of claim 11, wherein, the flat hash map index data structure includes: keys that include values that are incrementally generated based on threat indicators that have only domain names, and keys that include values that are incrementally generated based on threat indicators that have uniform resource locators (URLs) that contain domain name portions and path portions.

16. The packet filtering device of claim 11, wherein, the packet filtering device stores a second flat hash map index data structure that is associated with a second plurality of rules of the network security policy, and wherein: the plurality of keys of the flat hash map index data structure are associated with threat indicators that include domain names having a first characteristic, and the second flat hash map index data structure includes a second plurality of keys that map to a second plurality of rule identifiers, the second plurality of keys including values that are incrementally generated based on portions of threat indicators in matching criteria that include domain names having a second characteristic that is different from the first characteristic.

17. The packet filtering device of claim 11, wherein, The packet filtering device stores a second index data structure associated with a second plurality of rules of the network security policy, wherein the second index data structure includes a plurality of internal nodes, and wherein for each internal node of the plurality of internal nodes: The internal node includes a first bit array, a second bit array, and one or more pointers, The first bit array includes elements that map to possible values of a k-bit block of a search object, each of the first bit array elements stores a value that indicates, for the possible value mapped to the first bit array element, one of: a presence of a corresponding descendant internal node of the plurality of internal nodes, or an absence of a corresponding descendant internal node, The second bit array includes elements that map to possible values of the k-bit block and possible values of a portion of the k-bit block, each of the second bit array elements stores a value that indicates, for the possible value mapped to the second bit array element, one of: a presence of one or more corresponding rule identifiers, or an absence of a corresponding rule identifier, and The one or more pointers indicate memory locations associated with one or more rule identifiers corresponding to the k-bit block.

18. The packet filtering device of claim 17, wherein, The second index data structure maps keys that include at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, and wherein the instructions, when executed by the one or more processors, configure the packet filtering device to: search the second index data structure to find rule identifiers of the second plurality of rules that are applicable to a first packet received by the packet filtering device based on an IP address extracted from a field of the first packet; and store, based on the search and in an order ordered by associated rule priority, a first rule identifier associated with a first internal node of the plurality of internal nodes and a second rule identifier associated with a second internal node of the plurality of internal nodes that is a descendant node of the first internal node.

19. The packet filtering device of claim 17, wherein, The second index data structure maps keys that include at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, and wherein the instructions, when executed by the one or more processors, configure the packet filtering device to: search the second index data structure to find rule identifiers of the second plurality of rules that are applicable to a first packet received by the packet filtering device based on an IP address extracted from a field of the first packet; and store, based on the search and in an order ordered by associated rule priority, a first rule identifier associated with a first internal node of the plurality of internal nodes and a second rule identifier associated with the first internal node.

20. The packet filtering device of claim 11, wherein, The search includes: determining, in a first processing thread and by searching the set data structure based on a value extracted from a first packet of the plurality of packets, whether the flat hash map index data structure includes a first key that corresponds to the value from the first packet; in a second processing thread that executes concurrently with the first processing thread, initiating a search of the flat hash map index data structure based on the value extracted from the first packet; and terminating the search of the flat hash map index data structure based on a determination in the first processing thread that the flat hash map index data structure lacks a key corresponding to the value extracted from the first packet.

21. One or more non-transitory computer-readable media comprising stored instructions that, when executed by one or more processors of a packet filtering device, configure the packet filtering device to: receive a plurality of packets transmitted from one or more hosts in a first network to one or more hosts in a second network, wherein: the packet filtering device is located at a boundary between the first network and the second network and stores a flat hash map index data structure associated with a plurality of rules of a network security policy enforced by the packet filtering device, each rule of the plurality of rules includes a match criterion that indicates one or more threat indicators associated with packets to which the rule applies, and the flat hash map index data structure includes a plurality of keys mapped to a plurality of rule identifiers, each of the rule identifiers corresponding to a rule of the plurality of rules, and each of the keys including a value that is incrementally generated based on a portion of a threat indicator in a match criterion of the rule corresponding to the rule identifier to which the key is mapped; extract values from fields of the received plurality of packets and search the flat hash map index data structure based on the extracted values to find rule identifiers of rules of the plurality of rules that apply to the received plurality of packets, wherein each of the extracted values includes a domain name, and wherein, for each packet of the received plurality of packets, the search includes: for each label of a plurality of labels of the domain name extracted from the packet, generating, based on the label, a value for a portion of the extracted value that includes the label, compare each of the generated values to a key of the flat hash map index data structure, and determine, based on the comparison of the generated values to the keys of the flat hash map index data structure, whether one or more rules of the plurality of rules apply to the packet; and apply, for one or more packets of the plurality of packets, one or more rules of the plurality of rules that are determined to apply to the one or more packets, wherein the applying includes one or more of: blocking or dropping packets, forwarding packets, logging packets, capturing packets, redirecting or re-routing packets, modifying or transforming packets, or generating or sending responses to the packets.

22. The one or more non-transitory computer-readable media of claim 21, wherein, for each packet of the plurality of packets, the generating values includes: generating a hash of a first label that indicates a top-level domain of a domain name extracted from the packet, and for each additional label of one or more additional labels that indicate one or more subdomains of the domain name extracted from the packet, generating, based on the additional label, a hash of a portion of the domain name that includes the additional label and the top-level domain.

23. The one or more non-transitory computer-readable media of claim 21, wherein, for each packet of the plurality of packets, the generating values includes: generating, for each of one or more additional labels indicative of one or more subdomains of the domain name extracted from the packet, a lossless compression of the domain name based on the additional label that includes a portion of the additional label and the top-level domain. The value extracted from a first packet of the one or more packets includes a uniform resource locator (URL) that includes a domain name as a host portion and includes a path portion having one or more path segments, and wherein, for the first packet, the generated value includes:

24. The one or more non-transitory computer-readable media of claim 21, wherein, generating, for each of the one or more path segments, a hash of the URL based on the path segment that includes a portion of the path segment and the domain name of the URL. The flat hash map index data structure includes:

25. The one or more non-transitory computer-readable media of claim 21, wherein, keys that include values incrementally generated based on threat indicators having only domain names, and keys that include values incrementally generated based on threat indicators having uniform resource locators (URLs) that include a domain name portion and a path portion. The packet filtering device stores a second flat hash map index data structure associated with a second plurality of rules of the network security policy, and wherein:

26. The one or more non-transitory computer-readable media of claim 21, wherein, the plurality of keys of the flat hash map index data structure are associated with threat indicators that include domain names having a first characteristic, and the second flat hash map index data structure includes a second plurality of keys that map to a second plurality of rule identifiers, the second plurality of keys including values incrementally generated based on a portion of threat indicators in a matching criterion that includes domain names having a second characteristic different from the first characteristic. The packet filtering device stores a second index data structure associated with a second plurality of rules of the network security policy, wherein the second index data structure includes a plurality of internal nodes, and wherein, for each of the plurality of internal nodes:

27. The one or more non-transitory computer-readable media of claim 21, wherein, the internal node includes a first bit array, a second bit array, and one or more pointers, the first bit array includes elements that map to possible values of a k-bit block of search objects, each of the first bit array elements stores a value that indicates, for the possible value that maps to the first bit array element, one of: a presence of a corresponding descendant internal node of the plurality of internal nodes, or an absence of a corresponding descendant internal node, the second bit array includes elements that map to possible values of the k-bit block and possible values of a portion of the k-bit block, each of the second bit array elements stores a value that indicates, for the possible value that maps to the second bit array element, one of: a presence of one or more corresponding rule identifiers, or an absence of a corresponding rule identifier, and the one or more pointers indicate memory locations associated with one or more rule identifiers corresponding to the k-bit block. ​ 28. The one or more non-transitory computer-readable media of claim 27, wherein, The second index data structure maps keys comprising at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, and wherein the instructions, when executed by the one or more processors, configure the packet filtering device to: search the second index data structure based on an IP address extracted from a field of a first packet received by the packet filtering device to find a rule identifier of a rule of the second plurality of rules that applies to the first packet; and store, based on the search and in an order ordered by associated rule priority, a first rule identifier associated with a first internal node of the plurality of internal nodes and a second rule identifier associated with a second internal node of the plurality of internal nodes that is a descendant node of the first internal node.

29. The one or more non-transitory computer-readable media of claim 27, wherein, The second index data structure maps keys comprising at least a portion of an Internet Protocol (IP) address to rule identifiers of the second plurality of rules, and wherein the instructions, when executed by the one or more processors, configure the packet filtering device to: search the second index data structure based on an IP address extracted from a field of a first packet received by the packet filtering device to find a rule identifier of a rule of the second plurality of rules that applies to the first packet; and store, based on the search and in an order ordered by associated rule priority, a first rule identifier associated with a first internal node of the plurality of internal nodes and a second rule identifier associated with a second internal node of the plurality of internal nodes that is a descendant node of the first internal node.

30. The one or more non-transitory computer-readable media of claim 21, wherein The search includes: determining, in a first processing thread and by searching the set data structure based on a value extracted from a first packet of the plurality of packets, whether the flat hash map index data structure includes a first key corresponding to the value from the first packet; initiating, in a second processing thread that executes concurrently with the first processing thread, a search of the flat hash map index data structure based on the value extracted from the first packet; and terminating the search of the flat hash map index data structure based on determining, in the first processing thread, that the flat hash map index data structure lacks the key corresponding to the value extracted from the first packet.

Images