Rapid identification of violations and attack execution in network traffic patterns

Abstract

A method, apparatus, and computer system for identifying threats on TCP / IP-based networks. The technique utilizes a set of reference patterns (or "network spectra") associated with one or more predefined indicators of compromise (IoCs). At least one reference pattern is time-bounded, payload-neutral, and profiles network traffic patterns using a set of session data (e.g., volume, direction, traffic metadata) that may be derived in part by time-series compression of at least one non-variable coded interval. Network traffic data associated with the traffic pattern under test is received and coded to generate a test spectrum. A stream-based real-time comparison is performed to determine whether the test spectrum matches any of the reference spectra. In response to identifying a match, a given remediation or mitigation action is then taken. The reference spectrum may represent a bidirectional or multidirectional flow, and a multidirectional flow may involve multiple entities.

Description

[Technical Field]

[0001] The present disclosure relates generally to network security, and more particularly to tools and techniques that provide characterization of various network threats and attacks occurring on data networks (e.g., TCP / IP networks). [Background technology]

[0002] With regard to instrumentation for network cybersecurity positioning or other applications such as traffic engineering, networks are often in a position to be monitored by promiscuous network taps or by receiving network summary information exported from multiport switches or other network forwarding devices that provide flow data with varying degrees of redundancy or simplicity. When the application of this network traffic data is the subject of cybersecurity analysis, there are multiple ways the data can be used. The first use case is forensic collection, where raw packet data is collected for sorting, indexing, and analysis within a time frame following an attack incident. Another use case is real-time attack detection, which involves detecting a combination of network flow metadata such as source address information, application data signatures embedded in network traffic (e.g., malware), and other patterns known (via threat intelligence sources) to be indicative of some type of known attack. While a rich product space has grown around these two paths, both approaches have limitations.

[0003] With regard to forensic collection in particular, this technique is by definition post-incident and therefore only useful when cyber threat experts (e.g., security operations centers (SOCs)) have the skills and time to identify historical actual traffic for post-mortem analysis or evidence gathering. Real-time attack detection can incur significant processing overhead, and a fully utilized data center network backbone can easily overwhelm packet capture and / or flow analysis. Indeed, even if a device can be dedicated to this function, the CPU load for flow analysis often impairs scalability. Furthermore, when such processing occurs on the traffic flow (e.g., a switch), the job of performing the function may be assigned lower priority than the device's primary job (zero-loss packet forwarding), thereby impacting performance. Summary of the Invention

[0004] A further obstacle to application traffic analysis is the fact that encrypted traffic is impervious to this type of analysis without significant operational investment and policies that would be considered highly intrusive outside of highly secure sites. With the overall Internet traffic mix currently reflecting a significant HTTPS traffic balance, this becomes a necessary obstacle to overcome for any realistic volume or effectiveness of traffic analysis.

[0005] Known mechanisms to address these flaws include the use of network web proxies and HTTPS key escrow. Both approaches impose significant operational overhead. For example, proxying traffic is a costly operation, primarily in terms of storage and transmission processing cycles, because each transmission must be re-encrypted at the transfer point. Key escrow systems, which offer additional security protection, can typically only be deployed in mutually respecting secure enterprise network sites; otherwise, providers may be held liable for compromising the integrity of secure user traffic.

[0006] There remains a need in the art to provide new techniques for detecting malware and other network threats, preferably in near real-time and regardless of whether the payload is in the clear or encrypted.

[0007] A method, apparatus, and computer system for identifying threats on TCP / IP-based networks. The technique utilizes a set of reference "network spectrum" patterns associated with one or more defined indicators of compromise. A network spectrum is an encoding of a packet capture. Typically, a network spectrum includes a set of interval-boundary traffic rate measurements for each pair of addressable network interfaces in the captured traffic, data identifying the duration of the measurement interval, directionality data, source and destination peers, IP bearer and application protocol, and optionally other detectable metadata such as summary flow metadata about measurement entropy. Thus, a network spectrum typically profiles two-party or multi-party network traffic patterns using a set of session data that is payload-neutral. If traffic is idle (unchanging) during one or more intervals encoded in the network spectrum, time-series compression may be selectively applied to reduce the amount of encoded data, thereby resulting in a more compact representation. Preferably, a library of such network spectrums is generated and made available for evaluation.

[0008] To that end, network traffic data associated with a traffic pattern under test is received and encoded into a test network spectrum. A real-time comparison is performed to determine whether the test network spectrum matches any of the reference spectrums. Preferably, this comparison is performed in a continuous streaming manner, with each of the set of reference network spectrums initially identified as a potential match. When the confidence in a given match between the test network spectrum and a given reference network spectrum falls below a configurable threshold, the reference network spectrum is removed from the set. The process then continues until at least one potential match remains. In response to identifying at least one potential match, the system then takes a given remediation or mitigation action. In one example use case, the reference network spectrum indicates a set of traffic patterns that should be flagged, and when a potential match is found, the system provides an indication to a SIEM or other network security device or system that a match has been found.

[0009] The above has outlined some of the more pertinent features of the subject matter. These features should be construed as merely exemplary. Many other beneficial results can be attained by applying the disclosed subject matter in a different manner or by modifying the subject matter as described.

[0010] For a more complete understanding of the subject matter and its advantages, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an exemplary block diagram of a distributed data processing environment in which exemplary aspects of the illustrative embodiments may be implemented. [Figure 2]1 is an exemplary block diagram of a data processing system in which exemplary aspects of the illustrative embodiments may be implemented; [Figure 3] FIG. 1 illustrates a security intelligence platform in which the techniques of the present disclosure may be implemented. [Figure 4] 1 illustrates a network flow data collection system in which the techniques of the present disclosure may be implemented. [Figure 5] Figure 1 shows a representative network spectrum showing HTTPS-based traffic flows. [Figure 6] Figure 1 shows another representative network spectrum showing malware bots. [Figure 7] FIG. 1 illustrates the network spectrum analysis technique of the present disclosure. [Figure 8] A representative pseudocode listing for a spectral pattern matching routine. [Figure 9] Figure 7 shows a more detailed example of the network spectrum analysis technique. [Figure 10] An example network spectrum is processed using soft-axis time-series compression to account for variability in zero-volume (non-entropy) coding intervals that can affect spectral accuracy. [Figure 11] FIG. 1 illustrates a representative multi-party operating scenario in which the techniques of this disclosure may also be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0012] Referring now to the drawings, and in particular to Figures 1-2, diagrams of example data processing environments are provided in which exemplary embodiments of the present disclosure may be implemented. Figures 1-2 are merely exemplary and are not intended to assert or imply any limitation with respect to the environments in which aspects or embodiments of the disclosed subject matter may be implemented. Many modifications to the depicted environments may be made without departing from the spirit and scope of the present invention.

[0013] Referring now to the drawings, Figure 1 depicts a visual representation of an exemplary distributed data processing system in which aspects of the illustrative embodiments may be implemented. Distributed data processing system 100 may include a network of computers in which aspects of the illustrative embodiments may be implemented. Distributed data processing system 100 includes at least one network 102, which is the medium used to provide communications links between the various devices and computers connected together within distributed data processing system 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0014] In the illustrated embodiment, server 104 and server 106 are connected to network 102 along with storage device 108. In addition, clients 110, 112, and 114 are also connected to network 102. Clients 110, 112, and 114 may be, for example, personal computers, network computers, etc. In the illustrated embodiment, server 104 provides data such as boot files, operating system images, and applications to clients 110, 112, and 114. Clients 110, 112, and 114 are clients to server 104 in the illustrated embodiment. Distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0015] In the depicted example, distributed data processing system 100 is the Internet with network 102 representing a worldwide collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, comprised of thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, distributed data processing system 100 may also be implemented to include several different types of networks, such as, for example, an intranet, a local area network (LAN), a wide area network (WAN), etc. As noted above, FIG. 1 is intended as an example, and not as an architectural limitation for different embodiments of the disclosed subject matter, and thus the specific elements illustrated in FIG. 1 should not be considered limitations on the environments in which exemplary embodiments of the invention may be implemented.

[0016] 2, a block diagram of an example data processing system 200 is shown in which aspects of the exemplary embodiments may be implemented. Data processing system 200 is an example of a computer, such as client 110 of FIG. 1, on which computer-usable code or instructions for implementing processes for the exemplary embodiments of the present disclosure may be located.

[0017] 2, a block diagram of a data processing system is shown in which an illustrative embodiment may be implemented. Data processing system 200 is an example of a computer, such as server 104 or client 110 in FIG. 1, where computer-usable program code or instructions implementing processes may be located for the illustrative embodiment. In this illustrative example, data processing system 200 includes a communications fabric 202 that provides communications between a processor unit 204, a memory 206, persistent storage 208, a communications unit 210, an input / output (I / O) unit 212, and a display 214.

[0018] Processor unit 204 serves to execute instructions for software that may be loaded into memory 206. Processor unit 204 may be a set of one or more processors or may be a multi-processor core, depending on the particular implementation. Additionally, processor unit 204 may be implemented using one or more heterogeneous processor systems, in which a main processor resides on a single chip with secondary processors. As another illustrative example, processor unit 204 may be a symmetric multi-processor (SMP) system that includes multiple processors of the same type.

[0019] Memory 206 and persistent storage 208 are examples of storage devices. A storage device is any piece of hardware capable of storing information, either temporarily or permanently, or both. Memory 206 in these examples may be, for example, random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 208 may take various forms depending on the particular implementation. For example, persistent storage 208 may include one or more components or devices. For example, persistent storage 208 may be a hard drive, flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The media used by persistent storage 208 may also be removable. For example, a removable hard drive may be used for persistent storage 208.

[0020] In these examples, communications unit 210 provides for communications with other data processing systems or devices. In these examples, communications unit 210 is a network interface card. Communications unit 210 may provide communications through the use of either or both physical and wireless communications links.

[0021] Input / output unit 212 allows for the input and output of data with other devices that may be connected to data processing system 200. For example, input / output unit 212 may provide a connection for user input through a keyboard and mouse. Additionally, input / output unit 212 may send output to a printer. Display 214 provides a mechanism for displaying information to a user.

[0022] Instructions for the operating system and applications or programs are located on persistent storage 208. These instructions may be loaded into memory 206 for execution by processor unit 204. The processes of the different embodiments may be performed by processor unit 204 using computer-implemented instructions, which may be located in a memory, such as memory 206. These instructions are referred to as program code, computer-usable program code, or computer-readable program code, which may be read and executed by a processor in processor unit 204. The program code in the different embodiments may be embodied on different physical or tangible computer-readable media, such as memory 206 or persistent storage 208.

[0023] Program code 216 is located in a functional form on computer-readable medium 218, which is selectively removable, and may be loaded onto or transferred to data processing system 200 for execution by processor unit 204. Program code 216 and computer-readable medium 218 form computer program product 220 in these examples. In one example, computer-readable medium 218 may be in a tangible form, such as an optical or magnetic disk, that is inserted into or placed into a drive or other device that is part of persistent storage 208 for transfer onto a storage device, such as a hard drive that is part of persistent storage 208. In a tangible form, computer-readable medium 218 may also take the form of a persistent storage device, such as a hard drive, thumb drive, or flash memory, that is connected to data processing system 200. The tangible form of computer-readable medium 218 is also referred to as a computer-recordable storage medium. In some cases, computer-recordable medium 218 may not be removable.

[0024] Alternatively, program code 216 may be transferred to data processing system 200 from computer-readable medium 218 through a communications link to communications unit 210 and / or a connection to input / output unit 212. The communications link and / or connection may be physical or wireless in an exemplary embodiment. The computer-readable medium may also take the form of non-tangible media, such as a communications link or wireless transmission containing program code. The different components illustrated for data processing system 200 are not meant to imply architectural limitations to the manner in which different embodiments may be implemented. Different exemplary embodiments may be implemented in a data processing system including components in addition to or instead of those illustrated for data processing system 200. Other components shown in FIG. 2 may vary from the illustrated exemplary embodiment. As one example, storage devices in data processing system 200 are any hardware devices capable of storing data. Memory 206, persistent storage 208, and computer-readable medium 218 are examples of storage devices in tangible form.

[0025] In another example, a bus system may be used to implement communications fabric 202 and may be comprised of one or more buses, such as a system bus or an input / output bus. Of course, the bus system may be implemented using any suitable type of architecture that provides for a transfer of data between different components or devices attached to the bus system. Additionally, a communications unit may include one or more devices used to transmit and receive data, such as a modem or a network adapter. Further, memory may be, for example, memory 206 or a cache such as found in the interface and memory controller hub that may be present in communications fabric 202.

[0026] Computer program code implementing operations of aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider).

[0027] Those skilled in the art will appreciate that the hardware in Figures 1-2 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, may be used in addition to or in place of the hardware depicted in Figures 1-2. The processes of the illustrative embodiments may also be applied to multiprocessor data processing systems other than the SMP systems described above without departing from the spirit and scope of the disclosed subject matter.

[0028] As can be seen, the techniques described herein may operate within a standard client-server paradigm, as shown in FIG. 1, in which a client machine communicates with an Internet-accessible, web-based portal running on a set of one or more machines. End users operate Internet-enabled devices (e.g., desktop computers, notebook computers, Internet-enabled mobile devices, etc.) that can access and interact with the portal. Typically, each client or server machine is a data processing system, as shown in FIG. 2, including hardware and software, and these entities communicate with each other over a network, such as the Internet, an intranet, an extranet, a private network, or any other communications medium or link. A data processing system typically includes one or more processors, an operating system, one or more applications, and one or more utilities. Applications on the data processing system provide native support for Web services, including, but not limited to, support for HTTP, SOAP, XML, WSDL, UDDI, and WSFL, among others. Information about SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards, and further information about HTTP and XML is available from the Internet Engineering Task Force (IETF). Familiarity with these standards is assumed.

[0029] Security Intelligence Platform Today's networks are larger and more complex than ever before, and securing them against malicious activity is a never-ending task. Organizations looking to safeguard intellectual property, protect customer identities, avoid business disruptions, and more need to do more than simply monitor logs and network flow data. In reality, many organizations generate millions or even billions of events per day, and narrowing that data down to a short list of priority violations can be extremely difficult.

[0030] Known security products include security incident and event management (SIEM) solutions, which are built on a rules-based mechanism for evaluating observed security events. SIEM systems and methods collect, normalize, and correlate available network data. One such security intelligence product of this type is IBM® QRadar® SIEM, which provides a set of platform technologies that examine network flow data to discover and classify active hosts and servers (assets) on the network and track the applications, protocols, services, and ports they use. The product collects, stores, and analyzes this data, which performs real-time event correlation for use in threat detection and compliance reporting and auditing. Using this platform, billions of events and flows can then be reduced to and prioritized into a handful of actionable violations according to their business impact.

[0031] Network transactions typically consist of a series of multiple packets between a server and a client. With the widespread use of security monitoring and SIEM, many entities are choosing to collect this flow information to detect cyber threats. With the ever-increasing volume of traffic on networks, the need for mechanisms that allow for fast correlation of disparate records is becoming increasingly important. One such view, useful in security operations, is the ability to see what happened in a particular flow session. However, current mechanisms for defining and querying flows in a given flow session are typically tightly coupled to knowledge of which flow collectors saw the traffic, which protocols were used, and what flow fields were present. For example, correlation may be needed between two different network flow collectors looking at different halves of the same session (i.e., asymmetric routing and recombination), or correlation may be needed between a network flow collector and an event log collector.

[0032] A known type of security intelligence platform is illustrated in FIG. 3. Generally, the platform provides search-driven data exploration, session reconstruction, and forensic intelligence to assist in security incident investigations. In relevant part, the platform 300 includes a set of packet capture devices 302, an incident forensics module device 304, a distributed database 306, and a security intelligence console 308. The packet capture and module devices may be configured as network devices, or they may be configured as virtual devices. The packet capture devices 302 are operable to capture packets from the network (using a known packet capture (pcap) application programming interface (API) or other known techniques) and provide such data (e.g., real-time log events and network flows) to the distributed database 306, where the data is stored and available for analysis by the forensics module 304 and the security intelligence console 308. Packet capture devices operate in a session-oriented manner, capturing all packets in a flow and indexing the metadata and payload to enable fast, search-driven data exploration. Database 306 provides a forensic repository, its distributed and heterogeneous data set containing information collected by packet capture devices. Console 308 provides a web-accessible or cloud-accessible user interface (UI) that exposes a "Forensics" dashboard tab to facilitate the investigator's incident investigation workflow. Using the dashboard, the investigator selects a security incident. Incident forensics module 304 retrieves all packets (including metadata, payload, etc.) for the selected security incident and reconstructs the session for analysis.

[0033] A typical commercial product that implements this type of incident investigation workflow is IBM® Security QRadar® Incident Forensics. Using this platform, investigators search across distributed and heterogeneous data sets stored in a database and receive a consolidated list of search results. Search results may be merged in a grid, and they may be visualized in a "digital impression" tool, allowing users to explore relationships between identities.

[0034] Specifically, a typical incident forensic investigation is described here for extracting relevant data from network traffic and documents in a forensic repository. According to this approach, the platform allows for both simple and advanced methods of first searching and bookmarking many records, then allows the investigator to focus on the bookmarked records to identify a final set of records. In a typical workflow, the investigator determines which material is relevant. The investigator then uses that material to prove a hypothesis or "case" to develop new leads that can be pursued by using other methods in existing cases. Typically, the investigator first focuses the investigation through coarse-grained actions and then proceeds to refine those findings into a relevant final result set. The bottom of Figure 3 illustrates this basic workflow. Visualization and analysis tools within the platform can then be used to manually and automatically assess results for relevance. Relevant records can be printed, exported, or submitted for processing.

[0035] As described above, the platform console provides a user interface to facilitate this workflow. Thus, for example, the platform provides a search results page as a default page on the interface display tab. An investigator uses the search results to search for and access documents. The investigator may use other tools to conduct further investigation. Known entities or people found in network traffic and documents are automatically tagged. The forensic incident module 304 is operable to correlate interacting tagged identifiers. Typically, a collection relationship represents a sequentially collected electronic presence associated with an attacker or network-related entity.

[0036] Typically, appliances used in the above-described systems are implemented as network-connected, non-display devices. For example, appliances purpose-built to perform traditional middleware service-oriented architecture (SOA) functions are widespread in computing environments. SOA middleware appliances can simplify, secure, and accelerate XML and Web services deployment while extending existing SOA infrastructure across the enterprise. The use of hardware and lightweight middleware stacks for middleware can address the performance burden experienced by traditional software solutions. Additionally, appliance form factors provide a secure, consumable package for implementing middleware SOA functions. One particular advantage these types of devices offer is offloading processing from back-end systems. This type of network appliance is typically rack-mounted. The devices include physical security, allowing the appliance to act as a secure vault for sensitive information. Typically, the appliances are manufactured, pre-loaded with software, and then deployed within or in conjunction with an enterprise or other network operating environment. Alternatively, the box may be deployed locally and then provisioned with a standard or customized middleware virtual image that can be securely deployed and managed, for example, within a private or on-premise cloud computing environment. The equipment may include hardware and firmware encryption support, possibly for encrypting data on the hard disk. No users, including administrative users, can access any data on the physical disk. In particular, the operating system (e.g., Linux) preferably locks down the root account, does not provide a command shell, and users do not have file system access. Typically, the equipment does not include a display device, a CD or other optical drive, or any USB, Firewire, or other ports to allow devices to be connected.It is designed to be a closed, secure environment with limited accessibility, and only to authenticated and authorized individuals.

[0037] This type of device can facilitate security information event management (SIEM). For example, IBM® Security QRadar® SIEM is an enterprise solution that includes a packet data capture device that can be configured as this type of device. Such a device is operable to capture real-time Layer 4 network flow data, from which Layer 7 application payloads can then be analyzed, for example, using deep packet inspection and other techniques. It provides situational awareness and compliance support using a combination of flow-based network knowledge, security event correlation, and asset-based vulnerability assessment. In a basic QRadar® SIEM installation, a system such as that shown in Figure 3 is configured to collect event and flow data and generate reports. As described above, users (e.g., SOC analysts) can investigate violations and determine the root cause of network issues.

[0038] Generally speaking, security information and event management (SIEM) tools offer a range of services for analyzing, managing, monitoring, and reporting on IT security events and vulnerabilities. Such services typically involve the collection of monitored access and unexpected occurrence events across data networks and analyzing them in a correlation context to determine their contribution to profiled higher-level security events. They may also include support for analyzing firewall configurations, network topology and connectivity visualization tools to view current and potential network traffic patterns, correlation of asset vulnerabilities with network configurations and traffic to identify active attack paths and high-risk assets, and policy compliance monitoring of network traffic, topology, and vulnerability exposure. Some SIEM tools have the ability to build topologies of managed network devices, such as routers, firewalls, and switches, based on translational analysis of device configurations processed through a common network information model. The result is a locational organization that can be used for security threat simulation, firewall filter behavior analysis, and other applications. However, the primary device criteria are generally network- and network configuration-based. While there are several ways to invoke discovery capabilities for managed assets / systems, and containment in the user interface can be managed semi-automatically (i.e., methods through the user interface that allow semi-automated human input-based deployment with topology, and its display and formatting, are data-driven based on discovery of both initial configuration and changes / deletions in the underlying network), nothing is provided for fully automated deployment analysis and deployment analysis that generates recommendations.

[0039] Therefore, in this approach, violation details are extracted from a SIEM system such as QRadar®. The details typically include the violation type, rule, category, source and destination IP addresses, and user names. For example, the violation may be a malware category violation, indicating that malicious software was detected on a machine. Accordingly, activity on the machines surrounding the violation needs to be examined to determine infection vectors and potential data leaks. Naturally, the nature of the activity that needs to be investigated depends on the nature of the violation.

[0040] Many different types of network flow collectors exist, and collectors are often specific to a particular network flow protocol. One example is IPFIX, and another is NetFlow. The latter is a router-based feature that provides the ability to collect Internet Protocol (IP) network traffic as it enters or exits a router interface. By analyzing the data collected from such tools, network administrators can identify traffic sources and destinations, classes of service, and causes of congestion. As shown in Figure 4, a typical flow monitoring setup includes a flow exporter 400 that aggregates packets in flows and exports flow records to one or more collectors; a flow collector 402 that receives, stores, and preprocesses the flow data received from the flow exporter; and an analysis application 404 that analyzes the received flow data, for example, for intrusion detection or traffic profiling. Typically, a flow 406 is defined as a unidirectional sequence of packets that all share the following values: ingress interface, source IP address, destination IP address, IP protocol, source port for UDP / TCP (0 for other protocols), destination port for UDP / TCP (or type and code for ICMP or 0 for other protocols), and type of service.As shown in FIG. 4, data record 408 may include a variety of information about the traffic in a given flow, such as the input interface index used by SNMP, the output interface index (or zero if the packet was discarded), timestamps for the flow start and end times (typically milliseconds since last boot), the number of bytes and packets observed in the flow, Layer 3 headers (source and destination IP addresses, ICMP type and code, IP protocol, Type of Service (ToS) value), source and destination port numbers for TCP, UDP, and SCTP for TCP flows, the union of all TCP flags observed over the lifetime of the flow, Layer 3 routing information (e.g., the IP address of the immediate next hop along the route to the destination), and source and destination IP masks (prefix lengths in CIDR notation).

[0041] Many other types of network flow export protocols and associated collector mechanisms exist. Another example is sFlow (short for "sampled flow"), which is an industry standard for packet export at OSI Layer 2. It provides a means to export aborted packets along with interface counters for network monitoring purposes. Another example includes IPFIX, as mentioned above. Each flow export protocol produces a flow record. As described herein, the techniques of this disclosure work with any network flow export protocol, regardless of the nature and semantics of the flow record that is generated.

[0042] The packet and network flow capture techniques described above represent various operating environments in which the techniques of this disclosure may be implemented. As described, the fast identification and detection techniques herein may be implemented using different data sources (real-time records of network activity), including, but not limited to, actual packet capture with microsecond (if not nanosecond) accuracy for recording packet arrival times, and exported network flows where packet counts, packet sizes, and other data (e.g., interframe gaps, which are periods of silence between when a source sends a data packet over the network and when a destination responds) are collected as part of the exported flow data.

[0043] For network data to be propagated to a process performing any type of analysis, there must be a fixed period during which the network data is collected and packaged (e.g., ubiquitously in PCAP format for packet captures, and as IPFIX messages sent from the exporting process or device for network flows). In the context of the remainder of this disclosure, this fixed period is referred to as the "collection interval." More generally, in network traffic analysis, the collection interval represents the overall collection and reporting unit of interest, e.g., how a flow exporter assembles its overall data. As can be seen, the existence and adjustment of this interval can be a balancing factor between throughput and real-time effectiveness of the analysis described herein.

[0044] Rapid identification of violations and attack execution in network traffic patterns With the above as background, the techniques of the present disclosure will now be described. According to one aspect, traffic flows of interest, including both the traffic under test and a set of candidate traffic flows to which the system attempts to match the test traffic, are encoded into a representation format. As noted above, this format is referred to herein as a "network spectrum." The motivation for this approach is the recognition that network traffic itself bears a conceptual analogy to the colors of the visible light spectrum and its spectral representations. Thus, just as the visible spectrum conveys such characteristics or properties as magnitude, hue, and coloration, in a similar manner, the techniques herein encode a set of characteristics or properties of network traffic. The nature and type of these characteristics / properties may vary, but may typically include quantity, protocol identification, directionality, and unencrypted traffic metadata (e.g., TCP protocol and port equivalence). For example, assuming a baseline pattern over a reasonably fine-grained collection interval, a network spectrum for a given flow can be easily derived based on quantized IP traffic volume (or lack thereof in inter-frame transmissions), multi-party direction, and other detectable metadata (i.e., IP bearer protocols such as TCP, UDP, ICMP, and GGP), regardless of the actual traffic payload and content. The pattern (network spectrum) may be multidimensional or simply replicated within the number of nodes identified in any aggregated flow set. Furthermore, as will be explained, this technique can be used for simple bilateral cases (e.g., flows across two endpoints) but can also be used when multiple parties are involved in the traffic flow.

[0045] More formally, a network spectrum is an encoding of a packet capture. Typically, a network spectrum includes a set of segment-boundary traffic rate measurements for each pair of addressable network interfaces in the captured traffic, data identifying the duration of the measurement segment, directionality data, source and destination peers, IP bearer and application protocol, and optionally other discoverable metadata such as summary flow metadata about measurement entropy. Given this encoding, a network spectrum typically profiles bilateral or multilateral network traffic patterns using a set of session data that is payload-neutral. If traffic is idle (unchanging) while one or more segments are encoded into the network spectrum, and as described below, time series compression may be selectively applied to reduce the amount of encoded data, thereby resulting in a more compact representation.

[0046] 5 illustrates an example network spectrum 500 for a simple two-party traffic flow between a source 502 and a destination 504. These entities may be computing systems such as those shown in FIG. 2 operating in a client-server relationship such as that shown in FIG. 1.

[0047] As seen in Figure 5, this visual representation shows the traffic flow over the lifetime of a typical HTTP session with the "Connection: keep-alive" header present in the original HTTP request. While no two sessions of this type are exactly identical, an expected pattern exists here. In particular, the amount of request-centric traffic (above the line) is significantly outweighed by the response traffic (below). It is also clear that the initial response (on the left) forms the overwhelming majority of the traffic, along with subsequent smaller requests (e.g., paralleling the retrieval of a multimedia content page followed by the retrieval of a banner ad embed, a one-pixel advertiser tracking mechanism, etc.). There are two distinct protocols identified in this diagram: TCP control traffic and TCP session traffic (thick line). As can be seen, the network spectrum for this flow conveys a significant amount of information about the entire session, without any need for content (payload) analysis (e.g., by a security analyst). Thus, if this example flow were an HTTPS protocol session, it would be analyzed using the network spectrum techniques of this disclosure while avoiding intrusive management techniques such as key escrow or the use of specific decryption devices at the time of traffic monitoring.

[0048] Figure 6 shows another example network spectrum 600, in this case a pattern characteristic of a potential security incident. Again, this is a bilateral flow between a source 602 and a destination 604. In this example, the bold lines represent certificate exchange and transport control traffic. Data exfiltration is indicated by reference numeral 606, and command and control traffic is indicated at 608. This network spectrum is therefore a graphical pattern representative of a malware bot. As in the previous example, even though the traffic is anonymized and there is no content transparency, the network spectrum still provides meaningful information. Instead, relevant information is obtained from available data—i.e., data volume and direction, TCP packet header information, alternating protocol exchanges (e.g., X.509), etc.—and this information is identified as closely characteristic of malware-mediated exchanges. Continuing with the visual spectrum analogy, in this example, these available data elements resemble various colors or color characteristics that together form a visual representation.

[0049] Representations such as those shown in Figures 5 and 6 are examples of reference network spectra. Representations such as these may be grouped together, for example, into families of spectra associated with more generalized indicators of compromise (IoCs), in which case the spectra may also have associated "confidence" values ​​(or other prioritization) that reflect a judgment of their relative likelihood to be indicative of the IoC of interest.

[0050] Generally speaking, the various data elements that comprise a traffic flow of interest, particularly the traffic segment being analyzed, may be referred to herein as dimensions. While the dimensions that comprise a network spectrum can vary depending on the nature and type of flow as well as the number of flow participants, they are typically payload-neutral and defined at the level of session traffic. Session boundaries are defined by a flow's five-tuple: source IP address, destination IP address, source port, source-destination, and protocol. These are relevant dimensions that may be included in the encoding. Others include, but are not limited to, byte counts (amounts), packet counts, sequences of byte counts, observations paired with packet counts, direction, and unencrypted traffic metadata (e.g., TCP protocol and port equivalence). As noted above, a given network spectrum may be based on bidirectional network traffic patterns, multidirectional network traffic patterns, or a combination thereof, with the patterns generally based on connection establishment and teardown signatures for the connections (typically TCP transport connections) that transmit those patterns across the underlying IP network fabric. As can be seen, using network spectrum techniques, traffic segments of interest may be analyzed against a set of reference patterns that are encoded (typically in an offline process) to represent the occurrence of network threats and / or other high-value events in the network that indicate potential security breaches or attacks. This analysis may occur in real time or substantially real time, thereby enabling "high-speed identification" of security or other threats associated with the traffic segment. In one example (and non-limiting) embodiment, the network spectrum comparison set qualifies very high-speed execution of network events (HiVEs) (High Value Events) in the customer premises network.

[0051] FIG. 7 illustrates a conceptual diagram of the basic network spectrum capture and evaluation technique of the present disclosure. As shown, in this example embodiment, functionality is implemented in a network spectrum analyzer 700, which may be packaged as a standalone application, a library layer, or other processing capabilities (including, but not limited to, a native PCAP application such as Wireshark). The analyzer 700 includes or has the capability to access a set (library) of reference network spectrums 702. Traffic under test (e.g., a PCAP stream) is captured and encoded into a network spectrum by a spectrum converter component 704. The traffic under test is sometimes referred to herein as traffic under analysis or test traffic. The encoding used to generate the reference network spectrum may be different from the encoding used to encode the traffic under test, or the encoding techniques may be the same. The output of the converter component 704 is a network spectrum 706, which is then compared to reference network spectra in a comparison component 708, which provides a spectrum pattern match. Preferably, the comparison component 708 operates in a continuous streaming manner, with each reference network spectrum (of some given number of such reference spectra) being considered a match. As the matching process progresses, the comparison component 708 generates an output 710 indicating the degree of overlap between the captured traffic and one or more reference network spectra. When confidence in a given match between the test network spectrum and a given reference network spectrum decreases such that the reference network spectrum is no longer considered a (potential) match, that reference network spectrum is removed from further consideration. The matching process then continues until at least one potential match remains.Based on the pattern matching, the system then characterizes the test traffic, and if the flow (represented by the matched reference network spectrum) is suspicious, it takes appropriate remediation or mitigation actions, possibly as needed. While one or more reference network spectrums are typically found to match the test traffic, there may be situations where no match remains from the pattern matching. In such cases, the system returns an indication that the test traffic is either indeterminate or unrecognized.

[0052] In general terms, network spectrum assessment (encoding and pattern matching of test traffic) is accomplished by a system (typically a software system) that is fed either captured packet traffic (e.g., PCAP files) or data derived from live-captured traffic (e.g., IPFIX messages).

[0053] The following sections provide additional details regarding implementation of the techniques described above.

[0054] Traffic, Pattern, and Flow Monitoring As used herein and as shown in FIG. 7, one or more reference spectra (s ref ) is the traffic segment (s eval) are compared. In a typical use case, a network spectrum analyzer considers information encoded in the network spectrum, e.g., packet counts, byte counts, and network traffic metadata (e.g., primary IP bearer protocol, source / destination ports) across one or more traffic flow dimensions, e.g., the traffic patterns observed as samples over a given collection interval. For any given time-bounded interval (e.g., every 5 seconds), the formulated comparative analysis is shown by the formula:

number

[0055] This formula represents the formula m for matching all candidate reference network spectra, and is run over the matching interval i. Formally, a matching interval typically represents some particular subdivision within (and before the end of) the collection interval. A matching interval is typically an integer coding interval. A coding interval is the smallest period within the collection interval for qualifying and inferring the presence and match of IoCs, as shown in the reference network spectrum, to the same matching interval for the analyzed traffic. A coding interval is the atomic duration unit of comparison. Preferably, the formula shown above is run iteratively over all matching intervals in the matching space (in the PCAP or across the current boundaries of the captured traffic), with each iteration yielding a set of matched and partially matched spectra M', where M'∋(s x ,o sx) is the combination of identifying a spectrum (partially matched) and encoding an offset to that spectrum for which a match has been identified by the end of the matching interval. In stream-based pattern matching, at the beginning of any match in matching interval 0, preferably all possible reference spectra are viable match candidates (i.e., none are removed). When traffic is evaluated in the matching interval, the system takes into account current match candidates from their offset 0.

[0056] Confidence interval weighting As described above, during stream-based pattern matching, the encoding of the traffic under test is compared to the encoding of each reference network spectrum that is a potential match to the test traffic. As pattern matching progresses, the packet analyzer's confidence in the match typically varies, typically with respect to some predefined confidence interval. More formally, the confidence interval is the number of consecutive positive comparisons over the matching interval that serves to qualify or confirm the threshold likelihood of the comparison to the IoC as indicated in the reference network spectrum collection interval. Typically, this number is a property of the reference network spectrum, perhaps based in part on the confidence in the accuracy of the IoC indicated by the reference network spectrum. In alternative embodiments, the number is adjusted through system or user preferences based on external factors or other factors (e.g., time of day, source or destination IP addresses, etc.).

[0057] As described above, in one embodiment, a match according to a coding implementation typically includes a sequence of observed <byte count, packet count> pairs across a transition of the port, byte count, packet count, and match interval. However, any of several factors can cause a baseline spectrum match to deviate across the match point, thereby potentially impacting the analysis. These factors include, for example, port variations (especially for any port >1024) for a pattern that could potentially match the spectrum in all other respects, variations in packet payload volume due to variability in arguments within the range, etc. As an example, a phishing indicator of compromise (IoC) match may include a baseline HTTP redirect with a 78-byte destination URL, but the observed traffic has a 112-byte match (perhaps due to dynamic changes in hosting for the command and control (C&C) center directing the phishing attack). To address these types of errors, the network spectrum analyzer preferably derives and maintains additional confidence interval weighting, as described herein.

[0058] In particular, at any point when the analyzer considers a match in the reference spectrum, the analyzer preferably records the following state information (which constitutes some term of M or M' described above): (i) the identity of the network spectrum under consideration, (ii) the mapping of endpoint identities shown in the network spectrum to endpoint identities observed in the traffic, (iii) the offset to the successful match for that point, and (iv) the balance of the remaining confidence matching interval. The second item (ii) holds: the spectrum can relate to any two IP addresses, not just those observed in the reference network spectrum. The third item (iii) indicates that each match starts with a "reserve" of confidence that can be reduced over a given match. As an example, assume an arbitrary 100-point confidence. A mismatch between >1024 ports may be a small decrease from that balance, e.g., 1 point. On the other hand, a mismatch between <1024 ports may be a more significant deviation from the reference, e.g., 10 points. The change in size between two expected packets in a matching sequence may be variably weighted against a defined deviation range, e.g., [+(s ref -s eval ) / s ref ] points decrease. As another example, exhaustion of the criteria match space (e.g., transport connection) that leaves a significant amount of the evaluation match space transport connection can be weighted to very quickly deplete confidence points for the match. These are merely representative examples.

[0059] By applying confidence interval weighting as described above, the spectrum comparison component of the analyzer may quickly identify mismatches between the spectrum for the traffic segment being evaluated and a set of reference network spectra. In particular, the lack of viability for a given reference network spectrum may be quickly determined, leaving all but those reference network spectrum candidates that have some possibility of match.

[0060] Although not required, it may be desirable to constrain the match space under consideration, for example, by network-level protocol boundaries.

[0061] Network Spectrum Formation The formation and encoding of the network spectrum preferably results in a distinct interval value, referred to as the encoding interval, as described above. As explained, the encoding interval constitutes the smallest point for a given spectrum at which traffic volume can be identified and assessed for a given actor in the underlying network activity that the spectrum models and represents. Preferably, during encoding of the network spectrum (e.g., by the converter component 704 of FIG. 7), the system declares this value to ensure that confidence intervals in any matching process for any given comparison traffic measurement are appropriately adjusted. As described below, the encoding interval also allows the system to adjust traffic measurements to match the traffic measurements in the network spectrum. Preferably, a network spectrum with the smallest encoding interval (and therefore encoded with the most likely sample values) results in the most entropy network traffic pattern based on a particular network spectrum measurement.

[0062] Spectral Pattern Matching As noted above, the encoding (of both the test traffic and each reference) facilitates the comparison of observed network traffic to the reference network spectrum. As noted above in FIG. 7, this comparison is performed by comparison component 708. In practice, this comparison typically involves matching and adjusting interval measurements for each encoding to match those in the traffic pattern being compared. This is because, for efficient pattern matching, it is assumed that such dissimilarity in interval boundary divisions is the rule, not the exception. Therefore, to facilitate efficient pattern matching, several additional operations are preferably performed, including initial smoothing and interpolation. Smoothing is performed to spread closely related traffic spikes. This operation processes the data, for example, using Fourier analysis, to convert a series of spikes into a more continuous waveform. The degree of smoothing (or smoothing tolerance) may be varied and is preferably a system-configurable or user-configurable parameter. Interpolation refers to the process of providing statistically linear and discrete inputs to the smoothing process. Interpolation and smoothing are preferably performed so that the interval section of interest is first matched across the reference spectrum and the packet flow / capture to be compared. The matching itself then becomes a conventional search, albeit using matched cells for the interval boundaries of the interpolated encodings. Figure 8 shows a representative pseudocode listing for a stream-based pattern matching algorithm using this approach.

[0063] Spectral Report As described above and shown in FIG. 7, as stream-based matching proceeds, the spectrum analyzer provides an output representing the match of the traffic under test to a given reference network spectrum. A simple match allows for a simple (match(spectrum, binary)) result (e.g., output 710 in FIG. 7). As described above, preferably, pattern matching is performed continuously in near real time on the traffic under test, so that the encoding of the traffic itself is dynamic (as more data is received). Thus, as the traffic under test is compared to the reference network spectrum, the degree of match typically changes continuously (as more data is received), causing various reference spectra to be dropped (as mismatches). At any point in time, when pattern matching returns a response, typically at least one match, this match then represents the system's best characterization of the traffic under test. This final result can then be used for other purposes depending on the implementation. As one example, the comparison output can be used to trigger more advanced threat reporting processes, such as for SIEM, IPS processes, etc. Additionally, the spectral pattern matching algorithm (Figure 8) can be extended to more than match / mismatch results, thus enabling more versatile or fine-grained IoC analysis.

[0064] For example, there are some points at which the entropy terms of network sampling may introduce perturbations that can be tolerated by a cumulative tolerance confidence interval. More formally, a network spectrum may have an associated measured entropy. The entropy calculation is typically measured over the collection interval present in the reference network spectrum. A highly dissimilar entropy value provides the system with an immediate indication of a negative comparison, thereby allowing the system to reject the reference network spectrum as a match without expending processing cycles on the more computationally intensive process of comparing over the collection interval and potentially eliminating the calculation of the encoding interval.

[0065] The entropy calculation may proceed as follows: An additional match_interval function (in the comparison component 708) takes a given confidence interval and adjusts it during the matching process. To this end, the "compare spectral against next measure" code line (FIG. 8) measures that cumulative interval against a configured (or per-spectral) high level. This operation then approximates a "confidence entropy" value as follows: H(Y∨X)=Σ(x∈X)Σ(y∈Y)p(x,y)log(p(y∨x)

[0066] In other words, partial matches produced by spectral comparisons may have their own matching metrics to communicate. These metrics may also be particularly targeted for pattern spaces with high interpolated and native entropy. These entropy measures may also be provided as input to a spectrum analyzer to allow the analyzer to calculate the "maximum" entropy deviation that is acceptable in determining a match. One application of this type of spectral reporting includes, but is not limited to, the detection of anomalies within the comparison space of network spectral coding interval boundaries themselves.

[0067] The following provides additional details regarding the implementation of a network spectrum analyzer according to an example embodiment. As noted above, referring now to FIG. 9, the network spectrum analyzer takes as one input a reference network spectrum 900 and the output 902 of a PCAP spectrum transformer component and performs a comparison. As shown, the reference spectrum component preferably performs all of the above-described coding interval matching, smoothing, interpolation (outlier reduction), and entropy deviation (e.g., perfect match = 0, perfect dissimilarity = 1) calculations. Smoothing and interpolation are optional. The final result 904 optionally provides the degree of overlap between the captured traffic and the reference network spectrum, along with a notation and duration of any precise overlap identified by the comparison.

[0068] The smoothing, interpolation, and entropy calculation (which occurs during matching) is preferably per coding interval.

[0069] Construction of a reference spectral library Assuming a persistent network spectrum format (e.g., from a pcap-converted spectrum), the following describes a utility that can be used to generate a library of reference spectra 702, as shown in FIG. 7. As described therein, the analyzer may store the library natively or may access the library from an external repository or any other data source. The library may be updated periodically upon receiving new reference network spectra. In one implementation (e.g., with a pcap data stream), the utility takes a pcap file name and an output network spectrum file name. The utility then performs several functions. In particular, it evaluates the inter-frame spacing in the data stream to estimate the optimal usable coding interval. The utility then encodes the spectrum interval and frame level based on the coding interval, adjusting the interval based on any user overrides (although unattended operation, e.g., for automated test applications, is also supported), including any necessary smoothing, interpolation, and other functions to enable interval-bounded traffic level reporting. During this process, the utility may also provide other spectral metadata, such as entropy reporting. According to a further aspect of the present disclosure, the utility preferably also provides an additional coding option for the spectrum, referred to herein as soft axis.

[0070] Preferably, the system has an available library of network spectrum that is built based on measurements of judged or simulated attack traffic. Network spectrum may be provided by a trusted third-party source, for example, via a security community distribution model.

[0071] Alternative encoding techniques for generating the reference spectrum may be based on recorded packet-by-packet flow data, for example, in IPFIX and OASIS STIX. In the case of IPFIX, interval-by-interval exchanges may be encoded using the Basic List Template structure (defined in RFC6313). The basic interval-by-interval transmission (where interval is defined by the IPFIX Exporter Reporting Interval) reports according to the overall metadata template, and then publishes n list elements, where n = the number of sampling intervals within the IPFIX Reporting Interval. STIX utilizes a mechanism similar to that defined by the multi-element list structure in Volume 5 of the 2.0 STIX standard.

[0072] Soft-Axis Time Series Compression As mentioned above, the reference network spectrum is preferably encoded in a very compact manner (to facilitate pattern matching) by processing the encoding to ignore or omit intervals where traffic is idle (unchanging). When traffic is idle (unchanging) during one or more intervals encoded in the network spectrum, time series compression is preferably utilized to reduce the amount of encoded data, thereby resulting in an even more compact representation. This process is sometimes referred to herein as using soft-axis encoding. This encoding ensures that the analyzed traffic pattern cannot avoid detection as a result of ambient influences (on the flow) resulting from idle traffic conditions, i.e., idle activity that does not represent meaningful traffic. The nature and causes of idle activity can vary, but typically include application protocol activity such as inter-frame intervals and keep-alives across intervals. These real-world conditions can adversely affect the presence across intervals of measurements useful for profiling IoCs represented by the network spectrum, and the soft-axis approach removes this activity from the encoding.

[0073] For a typical IoC, a representative network spectrum includes intervals representing several non-zero measurements of traffic levels going from one side to the other, as well as zero traffic levels in either direction. These zero traffic levels may represent processing and microservice startup times on the spectrum participants' side, or, specifically, transmission latencies across long transmission fabrics where the sampling intervals are fine-grained. The soft-axis comparison and encoding described herein reduces false negatives that can arise from these non-zero intra-measurement intervals. Referring now to FIG. 10, in this simplified network spectrum 1000 illustration, there are multiple zero measurement intervals 1002. In this two-way, bidirectional traffic pattern, the open triangle 1004 represents traffic from A to B, and the solid triangle represents traffic from B to A. While it is possible to effectively match the spectrum pattern of actual traffic measurement levels at either endpoint of the flow, these values ​​typically remain relatively constant for both the reference spectrum and the measured, captured traffic (indicating the IoC associated with that spectrum). The reliability of the measurement (if not the overall ability to match the spectrum) may, however, be adversely affected by the presence of some defined sequence (e.g., greater than two) of zero measurement intervals, which causes the captured traffic to be inaccurate, for example, due to transient conditions in system processing or network latency. To address this, soft-axis coding provides time-series compression along non-changing (idle traffic) coding intervals. While this compression may be used whenever and if the spectrum contains a given sequence of zero measurement intervals, the preferred approach is to configure the compression more selectively depending on the nature of the traffic flow and conditions that may impair the accuracy of the representation.For example, compression of zero measurement intervals may be controlled by a matching element of a regular expression that tests for the presence of one or more conditions, such as transport vagaries (e.g., keep alive), interframe grap, etc., that may impair measurement accuracy. To provide a concrete example, a regular expression may enforce one or more types of matches of the matching element of the regular expression. Upon match, time series compression is performed on the sequence of zero measurement intervals. This coding is beneficial when considering the variability of zero-volume (non-entropy) coding intervals.

[0074] Generalized, soft-axis (time series) coding provides a mechanism for filtering (from the coding) measurements that do not represent meaningful traffic data useful for comparison purposes.

[0075] Multi-party, multi-directional network traffic patterns The techniques described above allow for the formulation and use of bidirectional network patterns between entities. Figures 5 and 6 represent typical bilateral flows. According to further aspects, the techniques herein are also readily applied to multilateral traffic patterns, including those involving multiple parties. This is desirable because not all IoCs conform to a bilateral (e.g., client-server) paradigm. This is illustrated as an example in Figure 11, which depicts a multilateral scenario showing traffic patterns and byte pairs corresponding to a primary client sender and multiple peers exchanging data using The Onion Router (TOR) across its peer routing fabric. This use case is not intended to be limiting. The final destination, once identified and anonymized, is not visible. In this scenario, there are multiple actors in the exchange. Comparing the two plots, it can be seen that the very presence of this peering set within a short distance provides some indication of a match to the reference spectrum. However, the identities of those peers must be extracted and matched to a given evaluation candidate for matching purposes. For example, a given set of peering actors a...e is mapped to identity IP addresses that appear for a given spectrum. If the spectrum to be matched is generated, for example, from evaluation candidates located on public IP address space versus behind NAT firewalls, the match (in this case, b...e → a single IP) is more robust, leaving multiple matching options open without a significant decrease in the confidence matching interval.

[0076] Generalizing, the techniques herein may be generalized to facilitate multi-party network spectrum comparisons. Preferably, the comparisons are performed using the functionality described above and, if necessary, evaluating traffic flows from each pair of entities (e.g., [a,b]...[b,c]...[a,c]) that contain the flow in question. In this way, the techniques herein enable pattern detection for forward proxying, man-in-the-middle translation, and other peer-related forwarding or data serving protocols (e.g., BitTorrent).

[0077] Mitigation and Remediation As described, the technology herein enables fast and efficient characterization of data network threats and attacks. As described, a basic use case attempts to match network conditions (through the network spectrum) against a set of characterized and recognizable threats. Based on such matching, the system outputs information useful for initiating or controlling some mitigation or remediation of the threat. The nature of the mitigation or remediation may vary depending on the severity of the threat, the degree of match, some external factors (e.g., time of day, target of attack, etc.), or some other consideration that may be defined in a security policy. Typical mitigation or remediation actions may be updating a firewall block, suspending the connection, sandboxing the connection, issuing an alert, logging, or providing a control signal or other indication to another security device, application, or system.

[0078] The techniques herein offer significant advantages. As described, the techniques herein enable fast and efficient characterization of data network threats and attacks. First, the techniques provide for the generation and use of a network spectrum in which bidirectional and multidirectional network traffic patterns can be analyzed and characterized. This approach enables a network spectrum analyzer to provide real-time or near-real-time traffic pattern analysis based on data encoded in a reference spectrum, such as data containing volume, direction, and unencrypted traffic metadata (e.g., TCP protocol and port equivalence) and employing unencrypted patterns based on connection establishment and teardown signatures for connections transmitting those patterns across an underlying IP fabric. The techniques herein enable real-time comparison of those traffic patterns against reference patterns that represent the occurrence of network threats or other high-value network events for security purposes. The techniques herein (particularly the "soft axis" concept) ensure that the reference spectrum comparison is robust and accurate when the traffic under test can be adversely affected by ambient network conditions such as interframe intervals, application protocol keep-alive activity over time, etc. The approach is also easily scalable to model multi-party (p>2), thereby enabling the system to detect complex patterns of forward proxies, man-in-the-middle translation entities, and other peer-related forwarding or data serving scenarios.

[0079] Without intending to be limiting, the techniques herein may be used in conjunction with other network flow monitoring and packet analysis techniques. The network spectrum analyzer of the present disclosure may be implemented in a variety of ways, including on-premise, network-based, or cloud-based.

[0080] This network spectrum analyzer subject matter may be implemented as a service. The subject matter may also be implemented in or in connection with a data center that provides data storage or related services related to cloud-based computing, other network security products and services. The spectrum generator utility (used to construct a reference spectrum) and spectrum analyzer functionality (used to perform real-time high-speed identification and analysis) may each be provided as standalone functionality, or each may leverage functionality from other products and services, including, without limitation, any other security monitoring and analysis systems, products, devices, programs, or processes.

[0081] In a typical use case, a SIEM or other security system is used to visualize data flow information, and is associated with an interface that may search and retrieve relevant information from alerts or other databases, and perform other known input and output functions thereon.

[0082] As noted above, the techniques herein are designed to be implemented in an automated manner within or in conjunction with security systems such as SIEMs, cybersecurity analytics platforms, and the like.

[0083] The functionality described in this disclosure may be implemented in whole or in part in a stand-alone manner, e.g., as software-based functions executed by a hardware processor, or may be available as a managed service (including as a web service via a SOAP / XML interface). Details of specific hardware and software implementations described herein are for illustrative purposes only and are not meant to limit the scope of the described subject matter.

[0084] More generally, a computing device within the context of the disclosed subject matter is a data processing system (such as that shown in Figure 2) including hardware and software, where these entities communicate with each other over a network, such as the Internet, an intranet, an extranet, a private network, or any other communications medium or link. Applications on the data processing system provide native support for web services and other known services and protocols, including, but not limited to, support for HTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, and WSFL, among others. Information about SOAP, WSDL, UDDL, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards. Further information about HTTP, FTP, SMTP, and XML is available from the Internet Engineering Task Force (IETF). Familiarity with these known standards and protocols is assumed.

[0085] The methods described herein may be implemented in or in conjunction with a variety of server-side architectures, including simple n-tier architectures, web portals, federated systems, etc. The techniques herein may be implemented in loosely coupled server (including "cloud" based) environments.

[0086] More generally, the subject matter described herein may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In a preferred embodiment, the functionality is implemented in software, including but not limited to firmware, resident software, microcode, etc. Furthermore, as noted above, the identity-context-based access control functionality may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any apparatus that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W) and DVD. A computer-readable medium is a tangible item.

[0087] A computer program product may be a product having program instructions (or program code) for performing one or more of the described functions. Those instructions or code may be stored in a computer-readable storage medium within a data processing system after being downloaded over a network from a remote data processing system. Alternatively, those instructions or code may be stored in a computer-readable storage medium within a server data processing system and adapted to be downloaded over a network to a remote data processing system for use in a computer-readable storage medium within the remote system.

[0088] In an exemplary embodiment, the threat characterization and modeling techniques are implemented in software executed by a dedicated computer, preferably one or more processors. The software is maintained in one or more data stores or memories associated with the one or more processors, and the software may be implemented as one or more computer programs. Collectively, this dedicated hardware and software include the functionality described above.

[0089] While the above describes a particular order of operations performed by certain embodiments of the present invention, it should be understood that such orders are exemplary, as alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc. Reference in the specification to a given embodiment indicates that the described embodiment may include a particular feature, structure, or characteristic, but that not all embodiments necessarily include that particular feature, structure, or characteristic.

[0090] Finally, although certain components of the system are described separately, those skilled in the art will understand that some of the functionality may be combined or shared in a given set of instructions, program sequences, code portions, etc.

[0091] The technology herein provides an improvement to another technology or technology field, for example, an improvement to security incident and event management (SIEM) systems, other security systems, and automation-based cybersecurity analytics.

[0092] The concepts of "real-time" or "quasi-real-time" as used herein do not imply any absolute time relative to a given clock value; rather, "real-time" may be relative. In the context of spectrum pattern matching as described, real-time refers to the matching process occurring as the traffic under test is received and encoded. As described, as encoding is in progress, the traffic is being matched with a reference spectrum. This pattern matching of the stream of data (traffic under test) occurs in real time or quasi-real time.

Claims

1. 1. A method for identifying threats on a TCP / IP based network, comprising: obtaining a set of reference patterns associated with one or more predefined Indicators of Compromise (IoCs), the reference patterns being encodings of a packet capture and including a set of interval-bound traffic measurements for each pair of one or more addressable network interfaces in the packet capture along with any other detectable session data, the any other detectable session data including at least one of traffic volume, directionality data, and unencrypted traffic metadata associated with a transport connection; receiving network traffic data associated with a traffic pattern as a data stream; encoding the received network traffic data to generate a network spectrum, the encoding also including at least one of traffic volume, directionality data, and unencrypted traffic metadata associated with the transport connection; real-time pattern matching the generated network spectrum against a set of reference patterns including the reference pattern as the data stream is received; taking a given remediation or mitigation action in response to identifying a match between the generated network spectrum and at least one reference pattern in the set of reference patterns; A method comprising:

2. Real-time pattern matching identifying each of the set of reference patterns as a match to the network spectrum; removing the given reference pattern from further consideration when confidence in a match between the network spectrum and the given reference pattern falls below a configurable threshold; The method of claim 1 , comprising:

3. 2. The method of claim 1, wherein the at least one reference pattern is generated by encoding the packet capture and filtering one or more time-bounded intervals indicative of idle traffic activity.

4. The method of claim 1 , wherein the one or more addressable network interfaces are associated with at least two separate computing entities.

5. The method of claim 4 , wherein the at least one reference pattern is a multi-directional traffic pattern involving more than two separate computing entities.

6. The method of claim 1 , wherein the received network traffic data is one of captured packet traffic and data derived from live captured traffic.

7. 1. A computer program in a computer readable medium in a data processing system for identifying threats on a TCP / IP based network, comprising: The processor A computer program for causing a computer to carry out the method according to any one of claims 1 to 6.

8. 1. An apparatus comprising: a processor; a computer memory holding a computer program as claimed in claim 7, executed by said processor to identify threats on a TCP / IP-based network; Equipped with Device.

9. 1. A network threat detection system, comprising: a first software-based utility configured to execute on hardware and to obtain a packet capture and generate a first encoding, the first encoding including a set of interval boundary traffic measurements for each pair of one or more addressable network interfaces in the packet capture along with any other detectable session data, the any other detectable session data including at least one of traffic volume, directionality data, and unencrypted traffic metadata associated with a transport connection; a second software-based utility configured to execute on the hardware and receive a data stream and, upon receipt of the data stream, perform real-time pattern matching of an encoding of the received data stream against a set of network spectrum codings including the first network spectrum coding, the encoding also including at least one of traffic volume, directional data, and unencrypted traffic metadata associated with the transport connection; and a third software-based utility configured to take a given remediation or mitigation action in response to a pattern match between a network spectrum coding of the received data stream and at least one network spectrum coding in a set of network spectrum codings including the first network spectrum coding; and A network threat detection system comprising:

Images