Automated detection of observables and automatic processing of alerts in endpoint detection and response (EDR) systems using machine learning

Abstract

A technique for threat response associated with an endpoint detection and response (EDR) system. The system uses a combination of automatic observable detection, threat intelligence augmentation, graph analysis, and supervised machine learning to machine-predict analyst behavior in classifying EDR alerts (as "true" or "false" positives) and supports either (i) automatic suppression of alerts that the system classifies with sufficient confidence as either true or false, or (ii) providing recommendations to analysts for alerts that cannot be so classified to expedite their actions. The automatic detection of observables for graph-based feature detection, coupled with automatic processing of alerts when possible, significantly reduces the overall analyst workload for the EDR system. Furthermore, and even when the machine-based predictions are not sufficiently reliable to allow bypassing the analyst, the system provides additional context and augmentation to the analyst, facilitating faster (or at least more efficient) alert handling.

Description

[Technical Field]

[0001] The present disclosure relates generally to alert handling in endpoint detection and response (EDR) systems. [Background technology]

[0002] Endpoint detection and response (EDR) systems are rule-based systems that perform well at correlating endpoint activity on host systems and generating alerts about suspicious or malicious behavior. However, one of the major drawbacks of these EDR systems is that they can generate a large number of detections, ranging from many duplicates to false positives, which can lead to analyst fatigue. Approximately 60–70% of the alerts generated by these systems are suspicious behavioral activity triggered by system administrators, custom applications or scripts, and reputation-based detections from next-generation antivirus (NGAV) sensors, which can result in a large number of false positives until tuned. Analysts spend approximately 10–15 minutes evaluating and investigating these “false positive” alerts, increasing the time required to triage and handle true positive detections. The resulting analyst fatigue and delayed investigation of more relevant alerts can lead to increased customer losses in the event of an actual cyberattack on a customer’s host system. Another drawback of EDR systems is that while they correlate suspicious events and malicious behavior, they do not do a good job of explaining anomalies, for example, through contextualization and data enrichment. They rarely list suspicious observables, for example, in command lines that have a history of previously exhibited suspicious behavior. The task of enriching these suspicious observables with the correct threat intelligence and asset context is left to the analyst conducting the investigation, further delaying response actions. Summary of the Invention

[0003] The present disclosure provides methods, apparatus, and computer program products for reducing alert volume and alert processing time in association with an endpoint detection and response (EDR) system. The techniques herein implement a machine-based predictive service (e.g., a microservice) that operates in association with or as part of an EDR system. In response to receiving an alert from the EDR system, the alert body is analyzed to automatically detect one or more “observables” within the alert that indicate potentially suspicious activity. Exemplary observables may include, for example, high-risk hashes, encoded commands, the number of command scripts, the presence of an IP address in the command line, the presence of a shell script, and the like. The set of one or more observables identified in the alert body (e.g., the value of the hash, the specific IP address identified in the command line, etc.) is then evaluated against a threat intelligence service, which returns a set of observables (or at least a portion thereof) augmented with additional information (collected from the threat intelligence service) that are useful for assessing the degree of risk presented by each such observable. The set of observables and their threat intelligence service-provided enrichment comprises a pattern, and an additional level of enrichment is then performed on one or more (or all) of the enriched observables in the pattern using graph-based analysis.

[0004] To that end, the enriched observables (as patterns) are then provided to a graph database, on which graph analysis is then performed to identify a set of graph features representing the complete set of enriched characteristics for the EDR alert. Representative graph features typically include alert-related data (e.g., number of alerts, number of alerts over a period of time, number of devices associated with the alerts, percentage of alerts escalated, and the like) and one or more graph-related metrics, such as page rank, community, centrality, proximity, and degree, for each node in the pattern. The output of this processing pipeline (including threat intelligence and graph analysis enrichment) is an EDR normalized alert, which is then applied to a machine learning model pre-trained on previous EDR alerts collected and processed in a similar manner. As a result, a prediction is generated for the EDR alert. Preferably, the prediction classifies the alert as either a "true" positive or a "false" positive, along with a confidence level representing the degree of confidence in the machine-generated prediction.

[0005] Depending on the machine confidence, the system handles the alert without providing it to an analyst. This is automatic processing. Thus, if the machine confidence level is high enough, e.g., at or above a configurable threshold such as 90%, the system can take automatic action. For example, for a "true" positive, the system automatically handles the alert, e.g., by escalating the alert, providing command or control to tooling to initiate automatic remediation, or the like; conversely, for a "false" positive, if the machine confidence is high enough, the system automatically closes the alert. In either case, the alert does not need to be directly handled by an analyst, thereby reducing alert volume (alert suppression). However, if the machine confidence for the alert is not high enough, and instead of automatic processing, the system returns an alert prediction (a "true" or "false" positive prediction and the accompanying machine confidence in that prediction) to the analyst dashboard for the EDR alert. To facilitate handling of alerts where machine confidence is not high enough to trigger automated processing, the alert prediction preferably includes various types of additional information to assist the analyst in quickly handling the alert. To this end, the additional information may include, for example, observables detected in the alert body, historical analysis explaining the machine decision, and identification of historically related actions or events.

[0006] The automatic detection of observables for graph-based feature detection, coupled with automatic processing of alerts where possible (based on machine confidence in the predictions), significantly reduces the overall analyst workload for the EDR system. Additionally, and even when machine-based predictions do not have sufficient confidence to allow the analyst to be bypassed, the system provides additional context and reinforcement to the analyst, facilitating faster (or at least more efficient) alert handling.

[0007] The foregoing has outlined some of the more pertinent features of the disclosed subject matter. These features should be construed as merely exemplary. Many other beneficial results can be attained by applying the disclosed subject matter in a different manner or by modifying the subject matter, as will be explained below. [Brief explanation of the drawings]

[0008] For a more complete understanding of the subject matter herein and its advantages, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] 1 depicts an exemplary block diagram of a data processing system in which exemplary aspects of the illustrative embodiments may be implemented; [Figure 2] 1 illustrates a representative EDR system in which the techniques of this disclosure may be implemented. [Figure 3] 1 illustrates a high-level representation of the disclosed architecture for automated threat handling for endpoints and network assets. [Figure 4] 1 shows a detailed representation of a preferred processing workflow. [Figure 5] We present a representative implementation of the system as a containerized cloud-native application that is portable and can be freely scaled and deployed across multiple clusters and regions, e.g., as a cloud-based solution. [Figure 6] We present a solution implemented as a Service Oriented Architecture (SOA). [Figure 7] A typical EDR-generated alert is shown below. [Figure 8] It shows the set of observables associated with the alert and enriched with threat intelligence data. [Figure 9] 1 shows a generalized high-level graph data model of entities (nodes) and relationships for customer alerts. [Figure 10]Shows a visualization of the customer alerts data model using the Neo4J browser. DETAILED DESCRIPTION OF THE INVENTION

[0010] Various aspects of the present disclosure are described through text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, depending again on the technology involved, two operations shown in successive flowchart blocks may be performed in the reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner.

[0011] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media"), collectively contained in one or more storage devices, that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as pits / lands formed on a major surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage in the form of a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data is typically moved at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device temporary because the data is not temporary while it is stored.

[0012] Computing environment 100 comprises an example of an environment for the execution of at least a portion of computer code involved in performing the methods of the present invention, such as the disclosed automated threat handler code 200, which uses a combination of automated observable detection, threat intelligence augmentation, graph analysis, and supervised machine learning to machine-predict analyst behavior in classifying EDR alerts (as “true” or “false” positives) and supports (i) the automatic suppression of alerts that the system classifies as either true or false with sufficient confidence, or (ii) providing recommendations to the analyst to facilitate analyst action for alerts that cannot be so classified, all of which are described in detail below. In addition to block 200, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), a communications fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 200, as identified above), a set of peripheral devices 114 (including a user interface (UI), a set of devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and a network module 115. Remote server 104 includes a remote database 130. Public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0013] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or multiple locations. However, in this presentation of computing environment 100, to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in FIG. 1 within a cloud, it may be located within a cloud. However, computer 101 is not required to reside within a cloud except to any extent that may be expressly indicated.

[0014] Processor set 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple tailored integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all caches for a processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0015] Computer-readable program instructions are typically loaded onto computer 101 and cause processor set 110 of computer 101 to execute a series of operational steps, thereby enabling a computer-implemented method, such that the instructions so executed instantiate the methods specified in the computer-implemented method flowcharts and / or descriptions contained herein (collectively referred to as the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by processor set 110 to control and direct the execution of the methods of the present invention. In computing environment 100, at least some of the instructions for executing the methods of the present invention may be stored in block 200 within persistent storage 113.

[0016] Communications fabric 111 is the signal-conducting pathway that allows various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as switches and conductive pathways that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication pathways may be used, such as fiber optic communication pathways and / or wireless communication pathways.

[0017] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0018] Persistent storage 113 is any form of non-volatile storage for a computer, now known or later developed. The non-volatility of this storage means that stored data is maintained regardless of whether power is supplied to computer 101 and / or to persistent storage 113 directly. Persistent storage 113 can be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 can take several forms, such as Linux, various known proprietary operating systems, or open-source Portable Operating System Interface-type operating systems employing a kernel. The code contained in block 200 typically includes at least some of the computer code involved in performing the method of the present invention.

[0019] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cable (such as a Universal Serial Bus (USB)-type cable), pluggable connections (e.g., a Secure Digital (SD) card), connections made through a local area communication network, and even connections made through a wide area network such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. The storage 124 may be external storage, such as an external hard drive, or insertable storage, such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (e.g., where computer 101 stores and manages large databases locally), this storage may be provided by a peripheral storage device designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple, geographically distributed computers. IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0020] Network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers over WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control function manages multiple different network hardware devices. Computer-readable program instructions for implementing the methods of the present invention may be downloaded to computer 101 from an external computer or external storage device, typically through a network adapter card or network interface included in network module 115.

[0021] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances using any technology for communicating computer data now known or later developed. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include copper transmission cables, optical fiber transmissions, wireless transmissions, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0022] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from computer 101's network module 115 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device, such as a thin client, a heavy client, a mainframe computer, a desktop computer, and the like.

[0023] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on past data, then this past data may be provided to computer 101 from remote database 130 of remote server 104.

[0024] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer functionality, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of public cloud 105 computing resources is performed by computer hardware and / or software in cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on various computers comprising host physical machine set 142, which is the universe of physical computers within and / or available in public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It is understood that these VCEs can be stored as images and can be transferred among and between various physical machine hosts, either as images or after instantiation of the VCEs. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that enables public cloud 105 to communicate over WAN 102.

[0025] Some further discussion of virtualized computing environments (VCEs) is now provided. A VCE can be stored as an "image." A new, active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances, called containers. These isolated user space instances typically behave as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and of the devices assigned to the container; this feature is known as containerization.

[0026] Private cloud 106 is similar to public cloud 105, except that its computing resources are available only for use by a single enterprise. While private cloud 106 is shown in communication with WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often each implemented by a different vendor. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0027] Endpoint Detection and Response (EDR) Systems FIG. 2 illustrates a typical known endpoint detection and response (EDR) system 200. EDR is software designed to automatically protect an organization's end users, endpoint devices, and IT assets against cyber threats that evade antivirus software and other traditional endpoint security tools. It is typically deployed by a service provider as a managed service on behalf of its enterprise customers. A typical service of this type is IBM® Managed Security Services (MSS). EDR continuously collects data from all endpoints on the customer network, including desktop and laptop computers, servers, mobile devices, IoT (Internet of Things) devices, and the like. It analyzes this data in real time for evidence of known or suspected cyber threats, and it can automatically respond to prevent or minimize damage from the threats it identifies. EDR threat detection analysis and automated response capabilities can identify and contain potential threats that penetrate the network perimeter before they can cause serious damage, often without the need for human intervention. EDR also provides tools that security teams can use to independently discover, investigate, and prevent suspected and emerging threats.

[0028] Specifically, EDR systems typically provide continuous endpoint data collection, real-time analysis and threat detection, automated threat response, threat isolation and remediation, and support for threat hunting. To this end, and as shown, EDR system 200 typically includes a set of components: system event aggregator 202, system correlator 204, rule library 206, alert generator 208, data analysis 210, and management console 212. System aggregator 202 receives events from connected endpoints. Endpoints (not shown) are networked devices that run system management code (software) that enable the EDR system to manage and monitor them. Endpoints are typically data processing systems. The system event aggregator continuously collects data from all endpoint devices on the network, such as data about processes, performance, configuration changes, network connections, file and data downloads or transfers, and end-user or device behavior. The data is stored in a central database or data lake, typically hosted in the cloud. Most EDR security solutions collect this data by installing lightweight data collection tools or agents on every endpoint device; some EDR security solutions may instead rely on features within the endpoint operating system. EDR uses advanced analytics and machine learning algorithms to identify patterns suggestive of known threats or suspicious activity in real time as they unfold. Broadly speaking, EDR looks for two types of indicators: indicators of compromise (IOCs), which are actions or events consistent with a potential attack or breach; and indicators of attack (IOAs), which are actions or events associated with known cyberthreats or cybercriminals.To identify these indicators, and using the system correlator 204, the EDR correlates its own endpoint data in real time with data from a threat intelligence service (not shown), a service that delivers continuously updated information on new and recent cyber threats—the tactics they use, the endpoint or IT infrastructure vulnerabilities they exploit, and the like. Threat intelligence services can be proprietary (operated by the EDR provider), third-party, or community-based. In addition, many EDR solutions also map data to Mitre ATT&CK, a U.S. government-sponsored global knowledge base on hacker cyber threat tactics and techniques. Data analytics 210 includes devices and algorithms that conduct their own investigations, comparing real-time data with historical data and established baselines to identify suspicious activity, anomalous end-user activity, and anything that may indicate a cybersecurity incident or threat. They can also separate the “signal,” i.e., legitimate threats, from the “noise” of false positives, allowing security analysts to focus on critical incidents. EDR summarizes key data and analysis results in a central management console 212, which also serves as the solution's user interface (UI). From there, security team members gain a complete view of all endpoints and endpoint security issues across the enterprise and initiate investigations, threat response, and remediation involving any endpoint.

[0029] Automation provides EDR with rapid response capabilities. Specifically, and based on predefined rules in rule library 206, either set by the security team or learned over time by machine learning algorithms, the EDR solution through alert generator 208 can automatically alert security analysts to specific threats or suspicious activity. For example, alert generator 208 can automatically generate "trackback" reports that trace all incident or threat outages on the network back to their root cause, disconnect endpoint devices or log end users off the network, suspend system or endpoint processes, prevent endpoints from executing (exploding) malicious or suspicious files or email attachments, trigger antivirus or antimalware software to scan other endpoints on the network for the same threat, and the like. Once a threat is isolated, EDR provides functionality that security analysts can use to further investigate the threat. For example, forensic analysis can help security analysts determine the root cause of a threat, identify the various files it affected, and identify the vulnerability or vulnerabilities that the attacker exploited to enter and move within the network, access authentication credentials, or perform other malicious activity. Armed with this information, analysts can use remediation tools to eliminate the threat. Remediation may involve destroying malicious files and erasing them from the endpoint; restoring damaged configurations, registry settings, data, and application files; applying updates or patches to eliminate vulnerabilities; and updating detection rules to prevent recurrence.

[0030] As mentioned, EDR systems can also include support for threat hunting (also known as cyber threat hunting). Threat hunting is a proactive security exercise in which security analysts search a network for unknown threats or known threats that have not yet been detected or remediated by an organization's automated cybersecurity tools. Threat hunters use a variety of tactics and techniques, most of which rely on the same data sources, analysis, and automation capabilities that EDR uses for threat detection, response, and remediation. To support threat hunting, EDRs make these capabilities available to security analysts through UI-driven or programmatic means, allowing them to perform ad-hoc exploratory data queries, correlation with threat intelligence, and other investigations. EDR tools specifically targeted for threat hunting include everything from simple scripting languages ​​(to automate common tasks) to natural language query tools.

[0031] As noted above, while EDR solutions offer significant advantages, they suffer from certain drawbacks, primarily due to high alert volume, the presence of many false positives, and a lack of alert reinforcement or contextualization. These challenges lead to analyst burnout and delayed investigation of more relevant alerts. The techniques of the present disclosure address these and other drawbacks in known EDR solutions.

[0032] Automatic detection of observables and automatic alert processing using machine learning With the above background, the techniques of the present disclosure will now be described. As referenced, the techniques herein are typically implemented in association with an EDR system (such as that shown in FIG. 2) to provide an automated threat handler for endpoints and network assets. The solution can be an adjunct to the EDR system, it can be integrated into the EDR system, or it can be provided as a standalone system or function, for example, as a microservice. As described, the system uses a combination of automated observable detection, threat intelligence augmentation, graph analysis, and supervised machine learning to machine-predict analyst actions in classifying alerts generated by the EDR system (as “true” or “false” positives) and supports either (i) automatic alert suppression for alerts that can be classified with sufficient confidence, or (ii) for alerts that cannot be classified with sufficient confidence, providing recommendations to the analyst to facilitate the analyst's response activities.

[0033] In particular, the techniques herein provide methods, apparatus, and computer program products for reducing alert volume and alert processing time in association with an endpoint detection and response (EDR) system. This benefit is realized using a machine-based predictive service (e.g., a microservice) operating in association with or as part of the EDR system. In response to receiving an alert from the EDR system, the alert body is analyzed to automatically detect one or more “observables” within the alert that indicate potentially suspicious activity. Exemplary observables may include, for example, high-risk hashes, encoded commands, the number of command scripts, the presence of an IP address in the command line, the presence of a shell script, and the like. The set of one or more observables identified in the alert body (e.g., hash values, specific IP addresses identified in the command line, etc.) is then evaluated against a threat intelligence service, which returns a set of observables (or at least a portion thereof) augmented with additional information (collected from the threat intelligence service) that are useful for assessing the degree of risk presented by the observables. The set of observables and their threat intelligence service-provided enrichment comprises a pattern. Herein, the observables may sometimes be referred to as "features." Additional levels of enrichment are then performed on one or more (or all) of the enriched observables in the pattern using graph-based analysis.

[0034] To that end, the enriched observables (as patterns) are then provided to a graph database, on which graph analysis is then performed to identify a set of graph features that represent the complete set of enriched characteristics for the EDR alert. Representative graph features typically include alert-related data (e.g., number of alerts, number of alerts over a period of time, number of devices associated with the alerts, percentage of alerts that are escalated, and the like) as well as one or more graph-related metrics, such as page rank, community, centrality, proximity, and degree, for each node in the pattern. Some or all of the information gleaned from the graph-related enrichment is then included in the alert. The output of this alert processing pipeline is an EDR normalized alert, which is then applied to a machine learning model pre-trained on previous EDR alerts collected and processed in a similar manner. As a result, a prediction is generated for the particular alert.

[0035] Preferably, the prediction classifies the alert as either a "true" positive or a "false" positive, with a confidence level representing the degree of confidence in the machine-generated prediction. Depending on the machine confidence, the system then handles the alert without providing it to an analyst. As described above, this is automatic processing. Thus, if the machine confidence level is high enough, e.g., at or above a configurable threshold such as 90%, the system can take automatic action. For example, for a "true" positive, the system automatically handles the alert, e.g., by escalating the alert, providing command or control to tooling to initiate automatic remediation, or the like; conversely, for a "false" positive, if the machine confidence is high enough, the system automatically closes the alert. In either case, alert volume is reduced because the alert does not need to be handled directly by an analyst. This is referred to above as alert suppression. However, if the machine confidence is not high enough, and instead of automatic processing, the system returns the alert prediction to an analyst dashboard, e.g., for EDR alerts. To facilitate handling of alerts where machine confidence is not high enough to trigger automated processing, the alert prediction preferably includes various types of information to assist analysts in quickly handling the alert, including, for example, observables detected in the alert body, historical analysis explaining the machine decision, and identification of historically related behaviors (e.g., rare events).

[0036] The automatic detection of observables for graph-based feature detection, coupled with automatic processing of alerts where possible (based on machine confidence in the predictions), significantly reduces the overall analyst workload for the EDR system. Additionally, and even when machine-based predictions do not have sufficient confidence to allow the analyst to be bypassed, the system provides additional context and reinforcement to the analyst, facilitating faster (or at least more efficient) alert handling.

[0037] FIG. 3 is a high-level depiction of a threat handling solution 300 of the present disclosure. As described, in one embodiment, the solution is implemented as microservices associated with an EDR system. Microservices are an application architecture in which an application is composed of many separate, network-connected components called microservices. This type of architecture allows large, monolithic applications to be broken down into smaller services. A single, network-accessible service is the smallest deployable unit for a microservices application. Each service runs in its own process, e.g., a container or other lightweight deployment mechanism.

[0038] As shown, the solution is coupled downstream from an EDR system alert generator (such as generator 208 in FIG. 2 ) to receive streaming Managed Detection and Response (MDR) alerts 302. In particular, alerts obtained from a source EDR system are received and routed from the alert generator to the automated threat handler microservice of the present disclosure. In this exemplary embodiment, the alerts are subject to feature engineering, represented as generated by reference numeral 304. As described in more detail below, feature engineering 304 involves identifying / detecting observables within the alert and associating contextual information and enrichment with the alert based on those detected observables. Typically, observables are hidden within the alert body, but they can also be present directly within the alert body. Observables include, but are not limited to, suspicious IP addresses, high-risk hash values ​​such as MD5 and SHA256, encoded embedded commands, download commands, and suspicious shell (e.g., PowerShell) scripts. Key features used for data enrichment and feature engineering include, but are not intended to be limiting. To that end, the first part of feature engineering 304 involves automatic detection of observables, for example, in the command line of input data identified in the alert. Next, as shown in FIG. 3, and after the observables have been identified, the observables (or their values) are run through a threat intelligence service 306, such as IBM X-Force Exchange (XFE), to determine whether any known threat intelligence exists regarding the observables. Supported by human- and machine-generated intelligence, IBM X-Force Exchange is a cloud-based threat intelligence platform that enables users to consume, share, and act on threat intelligence.Other threat intelligence services (e.g., AlienVault, now AT&T® Cybersecurity) can be used for this purpose.

[0039] The threat intelligence service 306 is typically accessed in an automated manner, for example, via an application programming interface (API). The threat intelligence service 306 parses and normalizes the alert data received in the alert and returns base data (one or more observables) supplemented with threat intelligence. The observables, now enriched with threat intelligence data, are then used as further inputs to the next stage of feature engineering, which involves graph analysis. In this aspect, the observables are provided to a pipeline 308, which cleans and formats the observables into patterns (alert graphs) and loads the data into a graph database (DB) 310 for graph analysis. This part of the feature engineering involves computing graph features that identify correlations between and within the nodes of the pattern graph and the importance of each node in the graph. The set of graph features results from the graph analysis feature engineering. Graph features include, but are not limited to, alert-related data (e.g., number of alerts, number of alerts over a period of time, number of devices associated with the alerts, percentage of alerts escalated, and the like), as well as one or more graph-related metrics such as page rank, community, centrality, proximity, and degree for each node in the pattern. As described above, some or all of the information gathered from graph-related enrichment is included in the alert. The output of this alert processing pipeline 308 (including threat intelligence and graph analytics enrichment applied to detected observables) is sometimes referred to herein as an EDR normalized alert.

[0040] As shown, the EDR normalized alert is then provided as a request to a prediction microservice 312. The prediction microservice 312 has an associated model 314 built by supervised machine learning 316 using training data collected from previous analyst actions and stored in an AI database 318. The previous analyst actions are a dataset from previous alerts previously referenced by the system over a period of time (e.g., measured over days, weeks, months, or a similar period), along with their associated “labels” and associated confidence level information. In other words, supervised machine learning is performed on the dataset using the enriched features as predictors and previous analyst actions as labels. The result of the machine learning is a precomputed scoring model that predicts the outcome of the alert as a “true” or “false” positive. As will be described, the system also provides machine confidence and, in certain circumstances, additional information such as the key observables referenced, an easy-to-read explanation of the recommendation, and a statistical analysis of similar previously referenced actions.

[0041] Referring again to FIG. 3 , in response to receiving a request, prediction microservice 312 applies the EDR normalized alert to machine learning model 314 to generate a prediction for the particular alert (“alert prediction”). Each alert prediction includes an indication of whether the alert is considered a “true” or “false” positive by the system, as well as the degree of confidence the machine has in the prediction. Depending on the machine’s confidence in the prediction, one of several actions is taken. In particular, if the machine’s confidence in the prediction is below a configurable confidence level threshold, the alert prediction is further augmented and forwarded by pipeline 308 to an analyst dashboard in integration console 320. Integration console 320 is typically a dashboard shared by a set of analysts responsible for handling EDR alerts. As shown, and in this scenario, the alert prediction typically includes additional information (e.g., observables, machine inferences, historical or statistical information about similar alerts or events, and the like) that can be used to assist the analyst in handling the alert.

[0042] However, if the machine's confidence in the prediction is high enough, there is no reason to forward the alert further for handling by an analyst. In such cases, and in accordance with the present disclosure, pipeline 308 is further configured to perform automated processing of the alert. In this manner, alerts are suppressed, thereby reducing the overall volume of alerts exposed to a set of analysts via a dashboard or otherwise.

[0043] When deployed in an EDR environment (or more generally in conjunction with managed security services), the techniques described above reduce the overall workload of analysts by limiting alert volume. For alerts not subject to automated processing, the system also reduces alert processing time by providing threat intelligence-enhanced data along with automatically detected observables and machine-generated recommendations.

[0044] FIG. 4 shows a more detailed representation of the automated process described above. As mentioned above, the system receives alerts from the EDR alert controller 400. The alerts are received by the prediction microservice 402, which routes the alerts for data cleaning and transformation 404 as needed to facilitate further processing. After data cleaning and transformation, feature engineering 406 is initiated to detect observables within the alert. At 408, the alerts are enriched by providing the identified observables to a threat intelligence service 409. The resulting threat intelligence-enhanced observables include patterns (of observables) for the alert, and at 410, this dataset is then loaded into a graph database 411 for graph feature extraction. Graph analysis 412 is used to generate graph features. Encoding and normalization 414 are then performed on the generated graph features to facilitate subsequent machine learning predictions. To that end, the system can utilize one or more models 415 pre-trained on historical data for prediction, as described above. For example, as the system generates predictions, the model is updated periodically or continuously. At 416, a prediction operation is performed on the current dataset using the model 415 to generate a prediction for the current alert being processed. As noted, the prediction (whether the alert is a "true" or "false" positive) also includes a machine confidence in the prediction. At 418, the system also generates additional information such as an explanation, statistical analysis, similar actions previously seen, and the like. At 420, the system generates a recommendation for handling, which is returned to the prediction microservice 402. As noted above, whether a particular alert is suppressed (handled automatically) or returned for analyst handling typically depends on the machine confidence level associated with the alert prediction. If the alert is not automatically handled, it is provided to the analyst console 422 for display, which is then picked up for handling by an analyst 424.

[0045] In one non-limiting embodiment, the solution described above can be implemented as a set of containerized, cloud-native applications that are portable and can be freely scaled and deployed across multiple clusters and regions. FIG. 5 shows a representative architecture. In this embodiment, there are two separate modules that can be deployed: a Python-based microservice 502 and a Neo4J graph database 504. Preferably, the deployment is automated, flexible, and on-demand infrastructure-oriented. For example, and without limitation, the deployment can be managed by a container platform 500 such as Red Hat® OpenShift®, a continuous delivery infrastructure 506 such as Weaveworks Flux, and GitHub Pull Requests, an approach to Kubernetes cluster management and application delivery called GitOps 505. This deployment approach gives companies greater control and ownership over the release process, enabling true continuous delivery with low risk, highly detailed audit trails, and increased operational velocity. To this end, a YAML (human-readable data serialization language) file on the managed service Github repository for each deployable module enforces system conformance to ensure the OpenShift cluster configuration matches the configuration in the repository, and it automates deployment by triggering deployment within OpenShift using an operator 508 in the cluster. The operator manages all relevant image repositories 510, detects new images, triggers deployment, and updates the desired running configuration based on this file. During deployment, a timed scheduler activity is also configured on the cluster, preferably to run periodically (e.g., daily) to provide model training. The model training is then pushed and stored on an OpenShift cluster persistent volume (PV). The PV is accessed by a Python microservice that fetches the most accurate version of the model.

[0046] Figure 6 shows the solution implemented as a Service Oriented Architecture (SOA). In particular, the processing pipeline is separated into different modules to provide loose coupling. Each module has a set of methods (services) exposed as method APIs. The modules communicate with each other through these APIs. A typical architecture is as follows:

[0047] The resource director module 600 is where training and prediction related resources reside, including cryptographic and SQL queries. Resources such as models and dictionaries are generated and then used during prediction. They reside in the resource directory.

[0048] The mdr_aedpr module 602 is the main module where the main pipeline for training and prediction resides. It has two sub-modules, mdr_aedrp.train and mdr_aedpr.run. The mdr_aedrp.train sub-module 604 supports the training pipeline. From this training pipeline, other modules needed to generate the training model are called in turn as needed. This pipeline also generates models that are saved in the resource directory. The mdr_aedpr.run sub-module 606 is used to design the prediction pipeline. From here, all other modules related to prediction are called in turn as needed. This sub-module also generates correspondence sets for some of the predictions.

[0049] The module mdr_aedpr_ingest608 is used to help generate / prepare the training data by pulling it from a database.

[0050] The module mdrbase610 contains utility scripts required by various modules in the architecture.

[0051] The module mdr_spikedetection 612 is used to remove spikes detected in the training data and return corresponding spike-free data, thereby removing any bias in the training data.

[0052] The module mdr_aedpr_db614 contains all the database (SQL / Neo4j etc.) connectivity APIs along with the scripts needed to create a database (DB) client.

[0053] The module mdr_aedpr_graph_analytics616 contains APIs for graph analytics, including data ingestion in neo4J, running graph analytics on the ingested data, and returning machine learning functions for training (e.g., PageRank, community detection, etc.).

[0054] The module mdr_aedpr_explainmodel 618 contains the API for model explanation. It provides an API for model explanation. In particular, it provides the explanation behind the model's alert prediction. The explanation typically includes the specific observations made during the prediction, along with a historical analysis of similar alerts observed across the training data or per customer.

[0055] Module web620 exposes the system as a Python microservice. This module receives alert requests from the alert controller, which then sends alerts to the prediction pipeline and compiles an API response with the results received from the prediction pipeline. The final response is then sent back to either the controller or the display.

[0056] The implementations shown in Figures 5 and 6 are not intended to be limiting.

[0057] Managed services (or the automated threat handler itself) typically implement a comprehensive auditing process to ensure that accurate, up-to-date feedback is provided to the machine learning of the present disclosure. In particular, and as analysts make processing decisions based on information provided by the system, they can "like" or "dislike" the machine recommendations and supporting explanations, thereby providing input for future learning. In addition, periodic (e.g., daily) audits are preferably performed, for example, on high-confidence alerts, to ensure that the solution is not erroneously automating processing. As described, audit feedback is preferably incorporated into the learning process.

[0058] Alert processing example To illustrate the processing described above, FIG. 7 shows an example of an EDR alert 700. As shown, the alert includes a set of attributes on the left and their values ​​on the right. Some of the values ​​may be masked, as shown. Once the alert data is received into the system, it is automatically enriched, supplemented with threat intelligence, and contextualized to detect specific attributes, i.e., observables, as described above. FIG. 8 shows observables 800, indications or counts 802 for each observable, and threat intelligence 804. In this example, the MD5_Risk observable is "High" because 21 instances of the MD5 signature are found in the threat intelligence data. The data also indicates that no PowerShell or shell scripts or download or CURL scripts are found in the command line (cmdline). One command script was identified. As described above, the next level of enrichment and feature engineering comes from graph analysis. In one embodiment, the system uses Neo4J as a backend graph database, although this is not intended to be limiting as the solution can be implemented using any graph database. A high-level graph data model 900 of entities (nodes) and relationships for a typical customer alert is shown in Figure 9. As shown, the node Cmd refers to the command line, and observables (referenced shell / PS scripts, referenced downloads, referenced encoded commands, referenced IP addresses, number of scripts, IP reputation, etc.) are listed as properties of the Cmd node. Figure 10 shows the customer alert data model 1000 visualized using the Neo4J browser.

[0059] The techniques herein have significant advantages. They provide a unique application of machine learning and graph data science to reduce alert volume by automatically processing alerts for EDR systems. The techniques employ graph data science and analytics to identify community behaviors and relationships and use the identified features to improve the predictive accuracy of machine learning. The system automatically detects objects of interest (observables) in EDR alerts, aiding faster decision-making by both machines and analysts. Rare event detection and automatically generated explanations are also provided to support machine recommendations. The techniques herein conveniently leverage supervised machine learning of analyst actions to learn threat severity. More generally, the techniques provide alert detection and processing based on supervised machine learning techniques and graph analysis. Supervised machine learning and graph analysis are used to establish relationships between nodes in a graph and use those features to guide analyst actions. As described, the techniques enable supervised machine learning and graph analysis for attack / alert processing.

[0060] In general, the method according to the present disclosure may be implemented in a stand-alone manner, e.g., as a software-based function executed by a processor, or it may be available as a managed service (including as a web service via a SOAP / XML interface). Details of specific hardware and software implementations described herein are for illustrative purposes only and are not intended to limit the scope of the described subject matter.

[0061] More generally, computing devices within the context of the disclosed subject matter are data processing systems (such as those shown in FIG. 1 ) each comprising hardware and software, and these entities communicate with each other over a network, such as the Internet, an intranet, an extranet, a private network, or any other communications medium or link. Applications on the data processing systems provide native support for Web and other known services and protocols, including, but not limited to, support for HTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, and WSFL, among others. Information about SOAP, WSDL, UDDI, and WSFL is available from the World Wide Web Consortium (W3C), which is responsible for developing and maintaining these standards; more information about HTTP, FTP, SMTP, and XML is available from the Internet Engineering Task Force (IETF). Familiarity with these known standards and protocols is assumed.

[0062] 1, the schemes described herein may be implemented in or in conjunction with a variety of server-side architectures, including simple n-tier architectures, web portals, federated systems, and the like. The techniques herein may also be practiced in whole or in part in loosely coupled server environments (including "cloud" based).

[0063] Further, more generally, the subject matter described herein may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In a preferred embodiment, functions are implemented in software, including but not limited to firmware, resident software, microcode, and the like. Furthermore, as noted above, the functions described above may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium may be any apparatus that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device). Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and DVD. A computer-readable medium is a tangible item.

[0064] In an exemplary embodiment, the code of the present disclosure (combination of automated observable detection, threat intelligence augmentation, graph analysis, supervised machine learning, and alert handling) is implemented in a special-purpose computer, preferably in software executed by one or more processors. The software is maintained in one or more data stores or memories associated with the one or more processors, and the software may be implemented as one or more computer programs. Collectively, this special-purpose hardware and software comprise the system described above.

[0065] While the above describes a particular order of operations to be performed by certain embodiments of the disclosed subject matter, it should be understood that such order is exemplary, as alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, or do the like. Reference to a given embodiment herein indicates that the described embodiment may include a particular feature, structure, or characteristic, but all embodiments may not necessarily include that particular feature, structure, or characteristic.

[0066] Finally, although certain components of the system have been described individually, those skilled in the art will understand that some of the functionality may be combined or shared among the given instructions, program sequences, code portions, and the like.

[0067] The techniques described herein are particularly suited to, but not limited to, EDR alert handling, as certain alerts may be generated by any other security service, system, device, or solution.

[0068] The techniques herein provide improvements to another technology or field, namely EDR systems, and improved operational capabilities of such systems when used in the manner described.

[0069] Having described the subject matter, the claims are as follows:

Claims

1. 1. A method of threat response associated with an endpoint detection and response (EDR) system, wherein a threat is identified in an alert and comprises: responsive to receiving an alert, automatically detecting one or more observable items within the alert, wherein the observable items are information potentially indicative of suspicious activity; augmenting the one or more observables detected in the alert with threat intelligence to generate a pattern; applying the pattern through graph analysis to identify a set of one or more graph features associated with the pattern; applying the set of one or more graph features through a machine learning model trained to classify prior analyst actions to machine-generate a prediction, the machine-generated prediction having an associated confidence level; and taking one of the following actions based at least in part on the confidence level: automatically suppressing the alert; and returning the prediction with additional contextual information to facilitate alert handling.

1. A method of threat response associated with an endpoint detection and response (EDR) system, comprising:

2. 2. The method of claim 1, wherein the observable item is detected in input data for the alert and is one of an IP address, a hash value, an embedded command, a download command, a command script, the presence of a shell script, and a command script detected in input data for the alert.

3. 2. The method of claim 1 , wherein the one or more graph features are one of page rank, community, centrality, closeness, and degree for one or more nodes in a graph associated with the pattern.

4. The method of claim 1 , wherein the machine-generated prediction is that the alert is one of a true positive and a false positive.

5. 10. The method of claim 1, further comprising training the machine learning model in a supervised manner using previous analyst actions as labels and the set of graph features as predictors.

6. 10. The method of claim 1, wherein the additional contextual information includes one or more of the observable items detected in the alert, historical analysis explaining the machine-generated prediction, and identification of historically related actions or events.

7. The method of claim 1 , wherein the method is implemented as a microservice associated with the EDR system.

8. processor; A computer memory holding computer program instructions that are executed by the processor for threat response associated with an endpoint detection and response (EDR) system. wherein a threat is identified in an alert, and wherein the computer program instructions include: In response to receiving an alert, automatically detecting one or more observable items within the alert, where an observable item is information potentially indicative of suspicious activity; augmenting the one or more observables detected in the alert with threat intelligence to generate a pattern; applying the pattern through graph analysis to identify a set of one or more graph features associated with the pattern; applying the set of one or more graph features through a machine learning model trained to classify prior analyst actions to machine-generate a prediction, the machine-generated prediction having an associated confidence level; and taking one of the following actions based at least in part on the confidence level: automatically suppressing the alert; and returning the prediction with additional contextual information to facilitate alert handling. and a program code configured to Device.

9. 9. The apparatus of claim 8, wherein the observable item is detected in input data related to the alert and is one of an IP address, a hash value, an embedded command, a download command, a command script, the presence of a shell script, and a command script detected in input data related to the alert.

10. 9. The apparatus of claim 8, wherein the one or more graph features are one of page rank, community, centrality, closeness, and degree for one or more nodes in a graph associated with the pattern.

11. The apparatus of claim 8 , wherein the machine-generated prediction is that the alert is one of a true positive and a false positive.

12. 10. The apparatus of claim 8, wherein the program code is further configured to train the machine learning model in a supervised manner using previous analyst actions as labels and the set of graph features as predictors.

13. 9. The apparatus of claim 8, wherein the additional contextual information includes one or more of the observable items detected in the alert, historical analysis explaining the machine-generated prediction, and identification of historically related actions or events.

14. The apparatus of claim 8 , wherein the method is implemented as a microservice associated with the EDR system.

15. 1. A computer program product on a computer-readable medium carrying computer program instructions that are executed by a processor for threat response associated with an endpoint detection and response (EDR) system, wherein a threat is identified in an alert, and the computer program instructions include: In response to receiving an alert, automatically detecting one or more observable items within the alert, where an observable item is information potentially indicative of suspicious activity; augmenting the one or more observables detected in the alert with threat intelligence to generate a pattern; applying the pattern through graph analysis to identify a set of one or more graph features associated with the pattern; applying the set of one or more graph features through a machine learning model trained to classify prior analyst actions to machine-generate a prediction, the machine-generated prediction having an associated confidence level; and taking one of the following actions based at least in part on the confidence level: automatically suppressing the alert; and returning the prediction with additional contextual information to facilitate alert handling. and a program code configured to Computer program products.

16. 16. The computer program product of claim 15, wherein the observable item is detected in input data for the alert and is one of an IP address, a hash value, an embedded command, a download command, a command script, the presence of a shell script, and a command script detected in input data for the alert.

17. 16. The computer program product of claim 15, wherein the one or more graph features are one of page rank, community, centrality, closeness, and degree for one or more nodes in a graph associated with the pattern.

18. 16. The computer program product of claim 15, wherein the machine-generated prediction is that the alert is one of a true positive and a false positive.

19. 16. The computer program product of claim 15, wherein the program code is further configured to train the machine learning model in a supervised manner using previous analyst actions as labels and the set of graph features as predictors.

20. 16. The computer program product of claim 15, wherein the additional contextual information comprises one or more of the observable items detected in the alert, a historical analysis explaining the machine-generated prediction, and an identification of historically related actions or events.

Images