Enhance network security and operational monitoring with alert confidence assignments
By supervising machine learning technology and aggregating vectorized event data, the alarm confidence score is calculated, which solves the problem of alarm priority sorting and false alarm processing in the computing system, and improves security monitoring efficiency and system security.
Patent Information
- Application Number
- CN201980054971.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-20
- Filing Date
- 2019-06-28
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-06-28
AI Technical Summary
The prior art is difficult to effectively distinguish and prioritize substantial threat alerts and false alarms in computing systems, resulting in security analysts being overwhelmed by a large number of potential threats and being unable to effectively allocate attention and resources.
Supervised machine learning technology is adopted to aggregate and vectorize event data, generate eigenvectors, and calculate the confidence scores of alerts using machine learning-driven classifiers to prioritize and filter alerts.
It improves the efficiency and effectiveness of the safety monitoring of the computing system, ensures that high confidence alerts receive rapid attention, reduces false alarm noise, promptly detects and responds to safety accidents, and limits their adverse effects.
Smart Images

Figure CN112602081B_ABST
Abstract
Description
Background Art
[0001] Cybersecurity attempts to reduce or prevent attacks that undermine the desired qualities of data and computing resources, such as confidentiality, availability, integrity, and privacy. Sometimes, cyberattacks focus on specific qualities, for example, distributed denial of service attacks typically focus on eroding or eliminating the availability of data and transaction services. At other times, attacks target multiple qualities, for example, a man-in-the-middle attack reduces confidentiality and privacy, but it may also introduce false or modified data, thereby attacking data integrity. Cyberattacks take many forms, including: social engineering efforts such as phishing, computationally intensive attacks such as brute force attacks on passwords, open attacks such as adware and ransomware, covert attacks such as rootkits and data leaks, attacks that focus on specific resources such as computing power (creating an army of zombie robots) or storage (hijacking web server storage to hold illegal materials), attacks against specific kinds of data (e.g., medical history, credit card data), and many other forms of attacks.
[0002] Some of these tools or techniques automatically monitor events in computing technology systems either when they occur or by examining event logs at a later time. Some tools or techniques simply log or report events, while others may sound an alert in order to direct attention to specific events that are considered suspicious or otherwise deemed more important to ensuring the proper operation of the computing technology. The monitored events may be security-related events, or they may be related to operational health, or both. Many different kinds of events may be monitored, including, for example, device operation result codes, firmware-generated errors, failed or successful attempts to log in, attempts to open or create or modify or delete files, attempts to change directories, attempts to change access privileges, actions such as scanning ports, rebooting, cloning, sending or receiving network communications that match specified patterns, and the like. Summary of the invention
[0003] Some of the teachings herein are motivated by the initial technical challenge of quickly distinguishing security alerts indicating substantial threats from alerts that do not. This can be viewed as a challenge to quickly, efficiently, and effectively clear false positive alerts. The next level of challenge is how to process the raw event data of the monitored computing system into a form suitable for useful classification by machine learning mechanisms. Other technical challenges solved by the innovations taught herein will also be apparent to those skilled in the art from the discussion provided below.
[0004] Some alarm confidence scoring embodiments include: a processor and a memory in operable communication with the processor, and other computing components, such as an aggregator, a vectorizer, and a classifier. The aggregator receives insight instances. Each insight instance of interest has an insight value and an insight type, and is associated with an alert identifier that identifies the alert. The alert is generated by monitoring the monitored system. The aggregator aggregates the insight values of received instances with the same insight type. The vectorizer creates a feature vector containing feature values, where the feature values correspond to the insight type. The feature vector contains feature values, which are aggregated insight values generated by the aggregator from multiple insight values. The feature vector is also associated with the alert. The classifier accepts the feature vector and assigns a confidence score to the alert based at least in part on the feature vector. The confidence score indicates the degree of confidence that the alert represents a threat to one or more operations of the monitored system. The output device can be configured to report the confidence score.
[0005] Some confidence scoring embodiments provide or use specific actions. For example, an embodiment may obtain digital electronic event data, the digital electronic event data including or otherwise identifying at least one of the following: one or more event structures generated by monitoring; an alarm structure generated by monitoring; or an alarm identifier identifying an alarm structure generated by monitoring. An embodiment extracts multiple insight instances from the event data, each insight instance having an insight value and an insight type, and automatically aggregates the insight values of at least two insight instances having the same insight type, thereby generating an aggregated insight value. An embodiment also automatically vectorizes the insight data by creating a feature vector containing feature values. The feature value corresponds to the insight type, and the feature vector contains the aggregated insight value as one of the feature values. An embodiment uses a machine learning driven classifier to classify the feature vector, and assigns at least one confidence score to the event data based on the result of the classification. Each confidence score indicates: the degree of confidence that the event data represents a threat to one or more operations of the monitored system. An embodiment may also configure an output device with at least one assigned confidence score, or it may cause a control mechanism to take action to implement a security control or an operational control.
[0006] Other technical activities related to the teachings of this article will also become apparent to those skilled in the art. The examples given are illustrative only. This summary is neither intended to identify the key features or essential features of the claimed subject matter nor to limit the scope of the claimed subject matter. On the contrary, this summary is provided to introduce some technical concepts in a simplified form, which will be further described in the following specific embodiments. The present invention is defined by the claims, and in the event that this summary conflicts with the claims, the claims shall prevail. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] A more detailed description will be given with reference to the accompanying drawings, which illustrate only selected aspects and are therefore not intended to be exhaustive in terms of coverage or scope.
[0008] Figure 1 is a block diagram illustrating a computer system and also illustrating a configured storage medium;
[0009] Figure 2 is a block diagram illustrating aspects of a computing technology environment including monitored systems, monitoring systems, event data generated by monitoring, and other items;
[0010] Figure 3 is a block diagram illustrating aspects of computing system event data, including events and alarms;
[0011] Figure 4 is a block diagram illustrating aspects of an insight instance;
[0012] Figure 5 is a block diagram illustrating an insight ranker and a ranked insight list;
[0013] Figure 6 is a block diagram illustrating various aspects of a feature vector;
[0014] Figure 7 is a block diagram illustrating aspects of a classifier;
[0015] Figure 8 is a block diagram illustrating some aggregate functions;
[0016] Fig. 9 is a block diagram illustrating some functionality that is avoided in some embodiments;
[0017] Fig.10 is a block diagram illustrating the functionality of some classifiers;
[0018] Fig.11 is a block diagram illustrating some additional operating features of some systems;
[0019] Fig.12 is a block diagram illustrating some examples of insight types;
[0020] Fig.13 is a block diagram illustrating some examples of sources of computing system event data;
[0021] Fig.14 is a flow chart illustrating steps in some alert confidence scoring methods;
[0022] Fig.15 is a data flow diagram illustrating usage of an alert confidence scoring method; and
[0023] Fig.16 is a flow chart further illustrating the steps in some alert confidence scoring methods. DETAILED DESCRIPTION
[0024] Overview
[0025] Innovation can transcend its origins, but understanding the origins of an innovation can help people understand the innovation more fully. In the present case, system monitoring innovations have emerged in the context of inventors seeking ways to improve the security of systems. Computing systems are vulnerable to different types of attacks. In order to improve the security of systems, organizations often track ongoing events in order to catch potential threats. Because false positives are common, security analysts sometimes further investigate suspicious events. These analysts examine different characteristics of a given event and, based on these characteristics, determine whether the event is malicious. The analyst's decision is used to trigger an alert to system customers.
[0026] To track ongoing events, intrusion detection systems and other security tools monitor events in individual machines, computer networks, and other computing systems. These tools can generate alerts when attention from security administrators or more complex security tools may be important. Similarly, resource management tools monitor the use of storage devices, CPUs, bandwidth, process descriptors, file descriptors, ports, virtual machines, server nodes, and other resources, and generate alerts when attention from operations administrators or more complex operations tools may be important.
[0027] By monitoring even relatively small computing systems, the number of security alerts or operational alerts generated per hour can also be hundreds or higher. Depending on security settings, alert thresholds, the number and focus of monitoring agents, monitoring granularity, the number of active users, the number of monitored machines and networks, available event log storage space, and other factors, the number of events recorded or otherwise noted within a given time period (e.g., one hour) may be in the thousands or even higher. In large organizations, security analysts may be overwhelmed by many different security events or many events on many different resources. As a result, security analysts are generally unable to analyze a large number of potential security or resource threats for their organizations. Therefore, tools and techniques that effectively allocate attention (e.g., the attention of a human administrator or the processing cycle of an automated intrusion prevention tool) and computing resources to specific alerts can improve the efficiency and effectiveness of the health status or security monitoring functionality in a computing system, and can directly or indirectly improve the health status and security of the monitored system itself.
[0028] Some embodiments of the teachings herein use or provide a virtual analyst that assigns confidence scores to security alerts. The virtual analyst does not assign confidence scores to alerts in the same manner as human experts assign such scores, but nonetheless produces results consistent with scores derived from analysis by human security and operations experts.
[0029] Some embodiments described herein use supervised machine learning to compute a confidence score for an alert that captures the confidence that the alert is indeed malicious or otherwise worthy of closer attention. This confidence score is highly useful to customers because it enables alerts to be prioritized for investigation. The confidence score can also enable filtering out alerts with scores below a threshold parameter. Ranking a list of alerts can be critical to effectively utilize the limited time of a human analyst.
[0030] Given an event, some embodiments execute a process that follows the logic of a security expert. The process generates a collection of insights that characterize security aspects of the event. Each insight is associated with a type and a value. For example, an insight may count the number of high severity alerts that were generated on a machine 24 hours prior to the event. The number of insight types is represented by N and is fixed. Alerts may generate multiple instances of a particular insight type with potentially different values, and the total number of insights (instances) may vary between events.
[0031] To handle the variable number of insight instances generated per event, some embodiments perform the following operations: (a) for each insight type, aggregate the values of all instances (of a given event) using an aggregation function, (b) represent events by N-dimensional feature vectors over insight types, and (c) apply supervised machine learning techniques to compute confidence scores for alerts. The confidence scores can then be used to focus limited attention and computing resources where their application will most improve the security or operation of the monitored system.
[0032] Some of the embodiments described herein may be viewed by some in a broader context. For example, concepts such as aggregation, classification, confidence, security, and operational status may be considered relevant to specific embodiments. However, from the availability of a broad context, it does not follow that exclusivity is sought for abstract concepts in this article; they are not. Rather, the present disclosure focuses on providing appropriate specific embodiments whose technical effects fully or partially solve specific technical problems, such as how to prioritize alarms. Other configured storage media, systems, and methods involving aggregation, classification, confidence, security, or operational status are outside the scope of this article. Therefore, with a proper understanding of the present disclosure, ambiguity, mere abstraction, lack of technical features, and the accompanying proof problems are also avoided.
[0033] Technical features
[0034] The technical features of the embodiments described herein will be obvious to those of ordinary skill in the art, and will also be obvious to a wide range of dedicated readers in a variety of ways. The embodiments address technical activities rooted in computing technology, which may include various events in monitored systems. Some alarm scoring embodiments improve the functionality of computing systems by helping to prioritize network security control actions or computing resource management actions or both, so that those systems can operate more securely, efficiently and effectively. When alarms are not scored, time and resources may be wasted due to screening alarms that are more informative than urgent in an ad hoc manner, or due to solving alarms that are not as impactful as unresolved alarms. Through the operation of the alarm confidence scoring mechanisms and techniques taught herein, monitored systems will be improved because high-scoring security alarms and operational alarms will be given prompt attention, without being lost in the noise of false positives, or not only taking action hours or days after an incident or threshold crossing rather than within minutes. The earlier a security incident is detected, classified as urgent and measures are taken, the more its adverse effects can be limited. With rapid alert scoring, for example, an intrusion can be limited to a single machine instead of an entire subnet, or a data breach can be terminated after one file is compromised instead of revealing a thousand.
[0035] Some of the technical effects provided by some embodiments include: (a) dividing alarms into alarms with high confidence scores and alarms without high confidence scores; (b) alarm feature vectors, which are based on extensive underlying event data by using instance value aggregation; and (c) continuously improved (refined) machine learning models. Corresponding technical improvements may include: faster and stricter suppression of security intrusions or data leakage, more advanced warnings for changes in resource capacity usage, and other improvements that will be obvious to those skilled in the art.
[0036] Other aspects and advantages of the technical features of the present teachings will also be apparent to those skilled in the art from the description provided.
[0037] Acronyms, Abbreviations and Names
[0038] Some acronyms, abbreviations and names are defined below. Others are defined elsewhere herein or do not require definition here to be understood by those skilled in the art.
[0039] ALU: Arithmetic and Logic Unit
[0040] API: Application Programming Interface
[0041] BIOS: Basic Input / Output System
[0042] CD: Compact Disc
[0043] CPU: Central Processing Unit
[0044] DVD: Digital Versatile Disk or Digital Video Disc
[0045] FPGA: Field Programmable Gate Array
[0046] FPU: Floating Point Processing Unit
[0047] GPU: Graphics Processing Unit
[0048] GUI: Graphical User Interface
[0049] IDS: Intrusion Detection System, typically this could be, for example, a HIDS (Host-based IDS) or a NIDS (Network-based IDS)
[0050] IP: Internet Protocol
[0051] LAN: Local Area Network
[0052] OS: Operating System
[0053] RAM: Random Access Memory
[0054] ROM: Read Only Memory
[0055] UEFI: Unified Extensible Firmware Interface
[0056] WAN: Wide Area Network
[0057] Additional terms
[0058] Reference is made herein to exemplary embodiments such as those shown in the accompanying drawings, and specific language is used herein to describe them. However, changes and further modifications to the features shown herein, as well as additional technical applications of the abstract principles illustrated by the specific embodiments herein, which would occur to a person skilled in the relevant art and having the present disclosure, should be considered within the scope of the claims.
[0059] The meaning of the terms is clarified in this disclosure, so these clarifications should be carefully noted when reading the claims. Specific examples are given, but those skilled in the relevant art will understand that other examples may also fall within the meaning of the terms used and fall within the scope of one or more claims. The terms here do not necessarily have the same meaning as they are in general use (especially in non-technical use), in the use of a specific industry, in a specific dictionary or dictionary collection. Figure numerals can be used with various wordings to help illustrate the breadth of the terms. Omitting figure numerals from a given text fragment does not necessarily mean that the text does not discuss the content of the figure. The inventor claims and exercises the right to compile specific and selected dictionaries. Quoted terms are explicitly defined, but it is also possible to define terms implicitly without using quotation marks. Terms can be explicitly or implicitly defined in a specific embodiment and / or in other locations in the application document.
[0060] As used herein, a "computer system" may include, for example, one or more servers, motherboards, processing nodes, laptops, tablets, personal computers (portable or non-portable), personal digital assistants, smart phones, smart watches, smart bracelets, cellular or mobile phones, other mobile devices having at least a processor and memory, video game systems, augmented reality systems, holographic projection systems, televisions, wearable computing systems, and / or other devices that provide one or more processors controlled at least in part by instructions. The instructions may take the form of firmware or other software in memory and / or dedicated circuit systems.
[0061] A "multithreaded" computer system is one that supports multiple threads of execution. The term "thread" should be understood to include any code that can be scheduled or subject to scheduling (and possibly synchronization), and may also be referred to by another name such as "task", "process" or "coroutine", for example. Threads may run in parallel, sequentially, or in a combination of parallel execution (e.g., multiprocessing) and sequential execution (e.g., time slicing).
[0062] A "processor" is a thread processing unit, such as a core in a simultaneous multithreading implementation. A processor comprises hardware. A given chip may house one or more processors. Processors may be general purpose, or may be customized for specific uses, such as vector processing, graphics processing, signal processing, floating point arithmetic processing, encryption, I / O processing, machine learning, etc.
[0063] The "kernel" includes the operating system, hypervisor, virtual machine, BIOS or UEFI code, and similar hardware interface software.
[0064] "Code" means processor instructions, data (including constants, variables, and data structures), or both. "Code" and "software" are used interchangeably herein. Executable code, interpreted code, and firmware are some examples of code. Code that must be interpreted or compiled in order to execute is called "source code."
[0065] "Program" is used broadly herein to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries, and other code written by programmers (also called developers) and / or automatically generated.
[0066] “Service” means a consumable program provided in a cloud computing environment or other network or computing system environment.
[0067] "Cloud" means pooled resources for computing, storage, and networking that are elastically available for measured, on-demand services. Clouds can be private, public, community, or hybrid, and cloud services can be provided as infrastructure as a service, platform as a service, software as a service, or another service. Unless otherwise specified, any discussion of reading from or writing to files includes reading / writing local files or reading / writing over a network (which may be a cloud network or other network), or both (local and network read / write).
[0068] As used herein, "comprising" allows for additional elements (ie, comprising means including) unless stated otherwise.
[0069] "Optimization" means improvement, not necessarily perfection. For example, further improvements can be made in a program or algorithm that has already been optimized.
[0070] For example, "process" is sometimes used herein as a term in the field of computer science and in a technical sense covers resource users, i.e., coroutines, threads, tasks, interrupt handlers, application procedures, kernel procedures, procedures, and object methods. "Process" is also used herein as a term in patent law, for example, when describing a process claim as opposed to a system claim or an article (configured storage medium) claim. Similarly, "method" is sometimes used herein as a technical term in the field of computer science (a "routine") and as a term in the field of patent law (a "process"). Those skilled in the art will understand the meaning intended to be used in a particular case, and will also understand that a given claimed process or method (in the patent law sense) can sometimes be implemented using one or more processes or methods (in the computer science sense).
[0071] "Automatically" means, as opposed to without automation, utilizing automation (e.g., general-purpose computing hardware configured by software for the specific operations and technical effects discussed herein). In particular, the steps performed "automatically" are not performed manually on paper or in a person's mind, although they may be initiated or interactively directed by a person. The automated steps are performed by a machine to obtain one or more technical effects that could not be achieved without utilizing the technical interaction so provided.
[0072] It is understood by those skilled in the art that the technical effect is the presumed purpose of the technical embodiment. For example, in one embodiment, calculations are involved, and some calculations can also be performed without a technical component (e.g., by paper and pencil, or even as a mental step), and this fact does not remove the existence of the technical effect or change the specific and technical nature of the embodiment. Operations such as extracting insight instances from event data, aggregating insight values, vectorizing insight data, classifying feature vectors, and executing machine learning models should be understood as inherently digital in this article. The human mind cannot be directly connected to a CPU or other processor or to a RAM or other digital storage device to read and write the necessary data to perform the alarm confidence scoring steps taught herein. In view of this disclosure, those skilled in the art will understand this well, but sometimes it may be necessary to inform or remind others of the fact. Unless otherwise stated, it is assumed that the embodiment can also be operated on a large scale in a production environment or in a test laboratory in a production environment (i.e., operating on event data from one hundred or more monitored devices), rather than a simple thought experiment.
[0073] "Computationally" also means that a computing device (at least a processor plus memory) is being used, and excludes results obtained by human thought alone or by human action alone. For example, as understood herein, arithmetic performed with paper and pencil is not computationally performing arithmetic. The computational results are faster, more extensive, deeper, more accurate, more consistent, more comprehensive, and / or otherwise provide a technical effect beyond the scope of human performance alone. A "computational step" is a step that is computationally performed. Neither "automatically" nor "computationally" necessarily means "immediately." "Computationally" and "automatically" are used interchangeably herein.
[0074] "Actively" means without direct request from the user. In fact, the user may not even be aware that active steps of the embodiments are possible until the results of the step have been presented to the user. Unless otherwise stated, any computational and / or automatic steps described herein may also be performed actively.
[0075] Throughout this document, the use of the optional plural "(s)" means that there are one or more of the indicated features. For example, "(s) processors" means "one or more processors" or equivalently "at least one processor".
[0076] For purposes of U.S. law and practice, use of the word "step" herein, in the claims or elsewhere, is not intended to invoke means-plus-function, step-plus-function, or 35 U.S.C. 112(f) requirement interpretation. Any presumption to such effect is hereby expressly rebutted.
[0077] For purposes of U.S. law and practice, unless the claims use the phrase “means for,” they are not intended to invoke a parts-plus-function interpretation. Claim language that is intended to be interpreted as parts-plus-function language, if any, will expressly recite that intent through the use of the phrase “means for.” When a parts-plus-function interpretation is employed, whether through the use of “means for” and / or through legal interpretation of the claim language by a court, the parts listed in the specification for a given noun or a given verb should be understood to be linked together with the claim language and are linked together here in any of the following ways: appearing in the same box in a drawing block diagram, being referred to by the same or similar name, being referred to by the same figure number, describing a functional relationship in any drawing, and indicating a functional relationship in the text of the disclosure. For example, if a claim limitation recites a "zac widget" and that claim limitation is subject to a parts-plus-function interpretation, then all structures identified at least anywhere that "zac widget" is mentioned in any figure, paragraph, or example in the specification or tied together by any figure markings assigned to the zac widget, or disclosed as having a functional relationship to the structure or operation of the zac widget, will be considered part of the structure identified in applications of the zac widget and will help define an equivalent set of zac widget structures.
[0078] Throughout this document, unless otherwise explicitly stated, any reference to a step in a process assumes that the step may be performed directly by an interested party, and / or performed indirectly by a party through intervening mechanisms and / or intervening entities, and still fall within the scope of the step. That is, unless direct performance is an explicitly stated requirement, the step need not be performed directly by the interested party. For example, steps involving an action on a destination or other subject matter by an interested party (such as accessing, aggregating, allocating, avoiding, classifying, commanding, comparing, calculating, connecting, communicating, configuring, creating, determining, displaying, executing, extracting, identifying, indicating, listing, mapping, satisfying, notifying, obtaining, providing, proposing, sorting, receiving, running, sending, specifying, acquiring, using, utilizing, vectorizing (and accessing, being accessed, aggregated, being aggregated, etc.)) may involve an intervening action by another party (such as forwarding, copying, uploading, downloading, encoding, decoding, compressing, decompressing, encrypting, decrypting, authenticating, invoking, etc.), but should still be understood to be performed directly by the interested party.
[0079] Whenever reference is made to data or instructions, it should be understood that, for example, these items configure computer-readable memory and / or computer-readable storage media, thereby converting it into a specific article, as opposed to simply existing on paper, in a person's mind, or as a signal propagating on a wire. For purposes of U.S. patent protection, memory or other computer-readable storage media is not a propagating signal, carrier wave, or energy alone, which is outside the scope of patentable subject matter, as interpreted by the U.S. Patent and Trademark Office (USPTO) in In re Nuijten. In the United States, no claim covers signal per se or energy alone, and any claim interpretation asserting the contrary is unreasonable in light of this disclosure. Unless otherwise expressly stated in a claim granted outside the United States, a claim does not cover signal per se or energy alone.
[0080] Furthermore, despite apparent provisions to the contrary elsewhere herein, the distinct distinction between (a) computer-readable storage media and computer-readable memory and (b) transmission media (also referred to as signal media) should be understood. Transmission media are propagating signal or carrier wave computer-readable media. In contrast, computer-readable storage media and computer-readable memory are not propagating signal or carrier wave computer-readable media. Unless otherwise expressly stated in the claims, "computer-readable medium" means computer-readable storage media, not propagating signals per se, nor energy alone.
[0081] The "embodiments" herein are examples. The term "embodiments" is not interchangeable with "the present invention." The embodiments may freely share or borrow aspects to create other embodiments (as long as the result is operable), even if the resulting combination of aspects is not explicitly described herein itself. It is not necessary for a person skilled in the art to require each permitted combination to be explicitly and individually described, which is contrary to the policy of recognizing that patent specifications are written for those skilled in the art. Formal combinatorial calculations and informal intuition about the number of possible combinations resulting from even a small number of combinable features will show that there are a large number of aspect combinations of the aspects described herein. Therefore, requiring an explicit description of each and every combination would run counter to the policy of requiring concise patent specifications and making the reader knowledgeable about the relevant technical field.
[0082] List of reference numerals
[0083] The following list is provided for the convenience and support of the drawings and as part of the text of the specification, which describes the innovation by reference to multiple items. Nevertheless, items not listed here may still be part of a given embodiment. In order to make the text clearer and easier to read, given reference numerals are described near some (but not all) referenced items in the text. The same reference numerals may be used to refer to different examples or different instances of a given item. The list of reference numerals is:
[0084] 100: Operating environment, also known as computing environment
[0085] 102: Computer system, also called computing system or computing system
[0086] 104: User
[0087] 106: Peripheral devices
[0088] 108: Networks, typically including LANs, WANs, software-defined networks, and other wired or wireless networks
[0089] 110: Processor
[0090] 112: Computer-readable storage media, such as RAM, hard disk
[0091] 114: Removable configuration computer readable storage medium
[0092] 116: Instructions, executable by a processor; may be on a removable storage medium, or in other memory (volatile or non-volatile or both)
[0093] 118: Data
[0094] 120: (multiple) kernels, e.g. (multiple) operating systems, BIOS, UEFI, device drivers
[0095] 122: Tools, for example, antivirus software, firewalls, packet sniffer software, intrusion detection systems (IDS), intrusion prevention systems (IPS), software development tools and tool kits, hardware development tools and tool kits
[0096] 124: Applications, e.g., word processors, web browsers, spreadsheets, games, email tools
[0097] 126: Display screen, also known as "monitor"
[0098] 128: computing hardware, otherwise not associated with reference numerals 106, 108, 110, 112, 114
[0099] 202: A monitored computing system, i.e., a system or device comprising a processor and a memory and being monitored; also referred to as a "monitored system"
[0100] 204: monitor, e.g., hardware, software, or both, that receives a signal and monitors it, or otherwise monitors at least one condition or state, to identify one or more predefined events, and then generates event data in response to the detection of the predefined events; although it is theoretically possible to distinguish between monitors and detectors on some basis, such as portable / fixed or active / passive, as used herein, "detector" is an example of a "monitor"
[0101] 206: Monitoring system, i.e., a system that uses one or more monitors to monitor a monitored system; although they are Figure 2 102 is shown as a separate box, but in some cases, the monitoring system and the monitored system can be the same computing system 102.
[0102] 208: Hardware of the monitored system, which may include computing hardware 106, 108, 110, 112, 114, 126, 128 or a portion thereof, and may also include non-computing hardware that neither supports digital operations nor operates in a digital manner
[0103] 210: software of the monitored system, for example, it may include software 116, 118, 120, 122, 124
[0104] 212: Data repositories, such as logs, databases, or audit trails
[0105] 214: event detector, e.g., an intrusion detector, a detection portion of an intrusion prevention system, a detection portion of an anti-malware tool, a motion detector, a temperature sensor, a humidity sensor, a light detector, a microphone, a switch, and other devices or mechanisms that detect an event and may responsively send a signal indicating the detection of the event; detector 214 is an example of monitor 204
[0106] 216: Controls, e.g., security controls, resource management controls; security controls can include physical controls and logical controls; logical controls are sometimes referred to as "logical access controls" or "technical controls"; controls can reside in a monitor or be functionally closely linked to a monitor
[0107] 218: Administrator; can be a human or a computing system
[0108] 220: Event data
[0109] 222: Command
[0110] 224: Insights Extractor
[0111] 226: Vectorizer
[0112] 228: Eigenvector
[0113] 230: Aggregator
[0114] 232: Aggregate function; may also be called "aggregate function"; accepts N inputs and returns one or more values as output, N>1
[0115] 234: Aggregate Insights Value
[0116] 236: A machine learning based classifier; may also be informally referred to as a "model", it being understood that the model is accompanied by and accessed through input and output interfaces
[0117] 238: Confidence score
[0118] 302: Event
[0119] 304: Event structure, i.e., a data structure containing data about an event
[0120] 306: Event Type
[0121] 308: Event ID, also known as "Event Identifier"
[0122] 310: Generally, event characteristics
[0123] 312: Location of the event
[0124] 314: User ID associated with the event
[0125] 316: Time, such as a point in time or a time interval associated with an event
[0126] 318: Placing event data in context
[0127] 320: Alarm
[0128] 322: Alert structure, i.e., a data structure containing data about an alert
[0129] 324: Alert type
[0130] 326: Alert ID, also known as "Alert Identifier"
[0131] 402: Insights Example
[0132] 404: Insight value
[0133] 406: Insight Type
[0134] 502: Insights Ranker, also known as the Insights Instance Ranker
[0135] 504: List of ranked insight instances
[0136] 602: Eigenvalue; also known as "eigenvector value"
[0137] 702: A trained model, e.g., one that uses machine learning to classify inputs
[0138] 704: A classifier interface, such as an API or other interface, that receives a feature vector to classify the feature vector and feeds the feature vector to a trained model for classification, then receives the classification (confidence score) from the trained model and provides it to the monitoring system software for display to the user or further use to prioritize alarms
[0139] 802: A sum aggregate function, i.e., a function that takes two or more numbers as input and provides their sum as output
[0140] 804: Product aggregate function, i.e., a function that takes two or more numbers as input and provides their product as output
[0141] 806: Minimum aggregate function, i.e., a function that takes two or more numbers as input and provides their minimum value as output
[0142] 808: Maximum aggregation function, that is, a function that takes two or more numbers as input and provides their maximum value as output
[0143] 810: Count aggregation function, i.e., a function that takes N inputs and provides N as outputs, where N is an integer greater than 1
[0144] 812: Distinct count aggregate function, i.e., a function that takes N inputs and provides M as outputs, where M and N are integers, N>1, 1<=M<=N, and M is the number of distinct input values, for example, distinct count (1, 3, 7, 19) = 4, distinct count (1, 3, -7, 19, 3) = 4, distinct count (1, 3, 1, 0.2, 3) = 3, and distinct count (-3, -3, -3, -3, -3, -3, -3) = 1.
[0145] 814: Median aggregate function, i.e., a function that takes two or more numbers as input and provides their median as output; to find the median of a set of N numbers, put the N numbers in order (ascending or descending, either), and the median is the middle number (if N is odd) or the average of the two middle numbers (if N is even)
[0146] 816: Average aggregate function, i.e., a function that takes two or more numbers as input and provides their average (arithmetic mean) as output; the average of N numbers is their sum divided by N
[0147] 818: Mode aggregation function, i.e., a function that takes two or more numbers as input and provides their mode as output; the mode of a set of numbers is the most frequently occurring number or numbers in the set, for example, mode(1,2,3,3,4,5)=3, mode(-1,2,-3,4,2,-1)=-1,2.
[0148] 820: Expression combining aggregate functions, that is, a combination of aggregate functions that takes two or more numbers as input to the combination and produces the result of the combination as output; functions can be combined using any computable operator and can include functions that are not themselves aggregate functions
[0149] 822: In Figure 8 Any aggregate function not specifically called out in , for example, x percent (the number of values below a specified percentage x of all values of interest)
[0150] 902: Functionality that may be specifically excluded from a given embodiment
[0151] 904: vector dimension, that is, the number of eigenvalues in the eigenvector
[0152] 906: Dimensionality reduction code, which, when executed, reduces the number of random variables being considered in the context of machine learning
[0153] 908: Filler code that, when executed, inserts default values to replace undefined values when vectorizing in a machine learning context
[0154] 910: Default values, i.e., values in a feature vector that do not depend on any specific underlying event data
[0155] 912: Undefined value, for example, there is no space or time slot in the feature vector that corresponds to event data
[0156] 1002: Classifier functionality
[0157] 1004: Neural Network Functionality
[0158] 1006: Deep Learning Network Functionality
[0159] 1008: Convolutional Network Functionality
[0160] 1010: Decision Tree Functionality
[0161] 1012: Probabilistic Classifier Functionality
[0162] 1014: Naive Bayes Classifier Model Functionality
[0163] 1016: In Fig.10 Other machine learning classification techniques not specifically mentioned in
[0164] 1100: Fig.11 Any operating characteristics or structure or other functionality of the system not specifically called out in the diagrams
[0165] 1102: confidence score range, e.g. [0.0..1.0]; values outside this range will be normalized to fit within the range, or discarded or considered invalid, depending on the implementation
[0166] 1104: confidence score alarm threshold; in implementations where its hierarchy includes: unmonitored occurrence, monitored event, alarm based on monitored event, alarm based on high confidence alarm, can be used to automatically determine whether to upgrade an alarm to an alarm
[0167] 1106: Duration from obtaining the extracted insight to assigning a confidence score to the alert or event associated with the extracted insight; duration may be measured in system time, real-world clock time, or other units, but unless otherwise specified, real-world clock time is assumed
[0168] 1108: repetition rate, i.e., the rate at which a specified sequence of actions is repeated
[0169] 1202: Qualifiers, which help define the insight type
[0170] 1204: Number of processes created; this is a sample insight type
[0171] 1206: The number of occurrences of a specific kind of event; this is an example insight type and is assumed to state a specific kind of event
[0172] 1208: Insight type qualifiers, which help define insight types by limiting them to specific machines
[0173] 1210: Insight type qualifiers, which help define insight types by limiting them to specific time intervals
[0174] 1212: Measurements of a specific physical item, for example, data from a motion detector, or data from a switch in a burglar alarm system
[0175] 1300: The source of the monitored event data
[0176] 1302: host-based network intrusion detection system, host-based network intrusion prevention system, network-based network intrusion detection system, or network-based network intrusion prevention system; these are examples of tool 122 and are also examples of detector 214
[0177] 1304: Cloud management plane or network management plane, for example, management plane of software defined network
[0178] 1306: Antivirus tool; this is an example of tool 122 and also an example of detector 214
[0179] 1308: Motion detector; this is an example of detector 214
[0180] 1310: Physical security mechanisms not otherwise shown in the attached figure, e.g., man-made traps, perimeter sensors, etc.
[0181] 1312: Firewall; this is an example of tool 122 and also an example of detector 214
[0182] 1314: Storage Controller
[0183] 1316: Cluster controller
[0184] 1318: Logical security mechanisms not otherwise shown in the attached figure, such as keystores, file integrity checkers, etc.
[0185] 1400: Alert confidence scoring method; confidence scores can also be assigned to sets of event data that do not contain alerts
[0186] 1402: Get event data
[0187] 1404: Extracting Insights Example
[0188] 1406: Aggregate insight values
[0189] 1408: Vectorizing Insights Data
[0190] 1410: Classify feature vectors
[0191] 1412: Assigning confidence scores to feature vectors based on classification 1410
[0192] 1414: Compare confidence score to threshold
[0193] 1416: Displaying, speaking, printing, or otherwise configuring a screen or other output device, such as for reporting or visualization enhancement or other purposes
[0194] 1500: Usage Example
[0195] 1502: Virtual advisor, such as an alert confidence scoring service, module, or device
[0196] 1504: Consumer of recommendations, i.e., an entity that consumes the output from the virtual recommender, such as an administrator 218 or an intrusion prevention system
[0197] 1506: Control action, e.g., command 222 to tool 122, or notification or escalation of consumer to more authoritative advice
[0198] 1508: Definition of insight type; for example, you could use something like Fig.12 The examples and qualifiers shown in or otherwise taught herein are implemented
[0199] 1510: Human Security Expert
[0200] 1600: Flowchart
[0201] 1602: Notify Administrator 218
[0202] 1604: Take control action
[0203] 1606: Commands a security tool to implement a security control; this is an example of taking a control action
[0204] 1608: Commands computing resource management tools to implement operational controls; this is an example of taking control actions
[0205] 1610: Provide visibility to users into the alert scoring process or mechanism, such as by otherwise displaying the internal actions taken, the internal values used, or both
[0206] 1612: Visibility into the alert scoring process or mechanism
[0207] 1614: Sending data, commands or other signals over the network
[0208] 1616: Receiving data, commands or other signals through the network
[0209] 1618: Avoid using dimensionality reduction functionality
[0210] 1622: Avoid filling functionality with default values
[0211] 1628: An alarm is raised because the confidence score reaches or exceeds a threshold; whether the threshold is reached or exceeded is the alarm condition.
[0212] 1630: Avoided alerting because the confidence score did not meet or exceed the threshold
[0213] 1634: Satisfy one or more performance criteria, for example, regarding the repetition rate or duration of performing a specified operation
[0214] 1636: Performance criteria, e.g. regarding duration 1106 or repetition rate 1108
[0215] 1638: Access to logs or other repositories of event data
[0216] 1640: Any other steps not specifically called out in the figure but taught herein 1642: Ranking insight instances
[0217] 1644: Configure an output device with a list of ranked insight instances; this is an example of configuration 1416
[0218] 1646: Normalized Insight Value
[0219] 1648: Mapping between alerts and insight instances
[0220] 1650: Prioritize alarms for management attention or control action
[0221] Operating Environment
[0222] refer to Figure 1, an operating environment 100 for an embodiment includes at least one computer system 102. Computer system 102 may or may not be a multiprocessor computer system. The operating environment may include one or more machines in a given computer system, which may be clustered in a cloud, client-server networked, and / or peer-to-peer networked. An individual machine is a computer system, and a group of cooperating machines is also a computer system. A given computer system 102 may be configured for, for example, an end user with an application, for an administrator, as a server, as a distributed processing node, and / or in other ways.
[0223] Human users 104 can interact with computer system 102 via typed text, touch, voice, movement, computer vision, gestures, and / or other forms of I / O using displays, keyboards, and other peripherals 106. Screen 126 can be a removable peripheral 106, or can be an integral part of system 102. A user interface can support interaction between an embodiment and one or more human users. The user interface can include a command line interface, a graphical user interface (GUI), a natural user interface (NUI), a voice command interface, and / or other user interface (UI) presentations, which can be presented as different options or can be integrated.
[0224] System administrators, network administrators, cloud administrators, security personnel, operations personnel, developers, engineers, auditors, and end users are all specific types of users 104, although it is expected that most users will likely play the role of security personnel or operations personnel. Automated agents, scripts, playback software, etc. that represent one or more individuals may also be users 104, for example, to facilitate testing of system 102 or as part of an alarm assessment or escalation business or technical process. In some embodiments, storage devices and / or network devices may be considered peripheral equipment, and in other embodiments, may be considered part of system 102, depending on their separability from processor 110. For example, Figure 1 Other computer systems not shown may interact with computer system 102 or with another system embodiment in a technical manner using one or more connections to network 108 via a network interface device.
[0225] Each computer system 102 includes at least one processor 110. As with other suitable systems, the computer system 102 also includes one or more computer-readable storage media 112. The storage media 112 can be of different physical types. The storage media 112 can be volatile memory, non-volatile memory, local fixed media, removable media, magnetic media, optical media, solid-state media, and / or other types of physical persistent storage media (as opposed to only propagating media signals or only energy). In particular, when inserted or otherwise installed, a configured storage medium 114 (such as a portable (i.e., external) hard drive, CD, DVD, memory stick, or other removable non-volatile storage medium) can functionally become a technical part of the computer system, making its contents accessible for interaction with and use by the processor 110. Removable configured storage media 114 is an example of a computer-readable storage medium 112. Some other examples of computer-readable storage media 112 include built-in RAM, ROM, hard disk, and other memory storage devices that are not readily removable by the user 104. For purposes of compliance with current US patent requirements, under any claim pending or issued in the US, neither the computer-readable medium, computer-readable storage medium, nor the computer-readable memory is a signal per se, or merely energy.
[0226] For example, the storage medium 114 is configured with binary instructions 116 executable by the processor 110; "executable" is used in a broad sense herein to include, for example, machine code, interpretable code, byte code, and / or code running on a virtual machine. The storage medium 114 is also configured with data 118, which is created, modified, referenced, and / or otherwise used for technical effect by the execution of the instructions 116. The instructions 116 and data 118 configure the memory or other storage medium 114 in which they reside; when the memory or other computer-readable storage medium is a functional part of a given computer system, the instructions 116 and data 118 also configure the computer system. In some embodiments, a portion of the data 118 represents real-world items, such as product characteristics, inventory, physical measurements, settings, images, readings, targets, volumes, and the like. Such data is also converted through backup, restore, submission, abort, reformat, and / or other technical operations.
[0227] Although the embodiments may be described as being implemented by software instructions executed by one or more processors in a computing device (e.g., a general-purpose computer, a server, or a cluster), this description is not meant to exhaust all possible embodiments. It will be appreciated by those skilled in the art that the same or similar functionality may also be generally implemented in whole or in part directly in hardware logic to provide the same or similar technical effects. Alternatively, or in addition to software implementation, the technical functions described herein may be at least partially performed by one or more hardware logic components. For example, and without excluding other implementations, the embodiments may include hardware logic components 110, 128, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chip components (SOCs), complex programmable logic devices (CPLDs), and similar components. For example, they may be grouped into interactive functional modules based on the input, output, and / or their technical effects of the components of the embodiments.
[0228] In addition to the processor 110 (e.g., CPU, ALU, FPU and / or GPU), memory / storage medium 112, and screen 126, the operating environment may also include other hardware 128, such as, for example, batteries, buses, power supplies, wired and wireless network interface cards. The terms "screen" and "display" are used interchangeably herein. Screen 126 may include one or more touch screens, a screen that responds to input from a pen or tablet, or a screen that is used only for output. In some embodiments, peripheral devices 106 such as human user I / O devices (screen, keyboard, mouse, tablet, microphone, speaker, motion sensor, etc.) will be in operable communication with one or more processors 110 and memory.
[0229] In some embodiments, the system includes multiple computers connected via a network 108. The network interface device 128 may provide access to the network 108 using, for example, network components that may be present in a given computer system (e.g., a packet-switched network interface card, a wireless transceiver, or a telephone network). Virtualization of networking interface equipment and other network components such as switches or routers or firewalls may also exist in, for example, a software-defined network. A given embodiment may also transmit technical data and / or technical instructions via direct memory access, removable non-volatile storage media, or other information storage retrieval and / or transmission methods.
[0230] Those skilled in the art will appreciate that the foregoing aspects and other aspects presented herein in the context of an "operating environment" may form part of a given embodiment. The headings of this document are not intended to strictly categorize features into embodiment and non-embodiment feature sets.
[0231] One or more items are shown in outline form in the figures, or listed in brackets, to emphasize that they are not necessarily part of the operating environment shown or part of all embodiments, but may interoperate with items in the operating environment or some embodiments discussed herein. In any figure or any embodiment, items that do not follow the non-outline or bracket form are required. In particular, Figure 1 is provided for convenience; Figure 1 The inclusion of an item does not imply that the item or the described use of the item was known prior to the present invention.
[0232] More information about the system
[0233] Examples are provided herein to help illustrate various aspects of the technology, but the examples given in this document do not describe all possible embodiments. The embodiments are not limited to the specific implementations, arrangements, displays, features, methods, or scenarios provided herein. A given embodiment may include, for example, additional or different technical features, mechanisms, sequences, data structures, or functionality, and may otherwise depart from the examples provided herein.
[0234] Figure 2 Aspects of some scoring architectures applicable to embodiments taught herein are illustrated. A monitored system 202 and a monitoring system 206 communicate cyclically over one or more networks 108, wherein event data 220 from the monitored system is evaluated in the monitoring system, and the monitoring system may send a monitored system command 222 in response to the evaluation, resulting in additional event data, which is then evaluated, and so on. For example, the monitoring system's evaluation may identify a security incident, and the resulting command 222 may be part of an incident response that contains, eliminates, or recovers from the security incident. Alternatively, the monitoring system evaluation may identify an operational vulnerability, and the command 222 may be part of an active or passive response that allocates additional storage, swaps drives, reroutes network traffic, reallocates network bandwidth, adds additional computing resources, shifts workloads, or takes other computing resource management actions.
[0235] The monitored system 202 shown includes hardware 208 and may include software 210. A log or other data repository 212 may be maintained external to the monitored system or within the monitored system 202.
[0236] Likewise Figure 2As shown in , monitoring can be performed using one or more monitors 204, which can be external or internal (or both) to the monitored system. Monitor 204 can be implemented in software running on general-purpose hardware, or can be implemented in software and dedicated hardware, or can be implemented entirely in hardware. The monitor can be or can include event detector 214. The monitor can include or be associated with control 216, which takes action in response to command 222. There can also be an administrator 218, for example, to receive escalated alarms from the monitoring system and act on them.
[0237] The monitoring system 206 may include an insight extractor 224 that extracts insight values from the event data 220 to support an assessment of the state of the monitored system and to score the severity of threats to its security or operational capabilities or prospects. The illustrated monitoring system uses components also discussed elsewhere herein to perform the assessment, including: a vectorizer 226 that produces a feature vector 228; an aggregator 230 that uses an aggregation function 232 to produce an aggregate insight value 234 from the extracted insight instances to include in the feature vector 228; and a machine learning classifier 236 that assigns a confidence score 238 to the feature vector. The confidence score indicates the severity, credibility, or urgency of the event data alert associated with the scored feature vector.
[0238] Systems 202 and 206 and monitor 204 may each include, be composed of, or depend on one or more machines. A given machine is an example of system 102, and each machine may be a physical machine or a virtual machine or a container, which runs on underlying physical machine hardware, for example, through a hypervisor or management plane. The hypervisor may be a "bare metal" or Type I hypervisor that runs directly on the hardware, or may be a Type II hypervisor that runs on a host operating system or another kernel, which in turn runs directly on the hardware. The hardware in a given machine includes at least one processor 110 having one or more cores, and RAM or other working memory 112 in operable communication with the processor(s). The physical machine(s) or virtual machine(s) or container(s) may all be located in a public cloud, a private cloud, a hybrid cloud, or a community cloud, or may not be in a cloud.
[0239] Figure 2 The other figures herein provide examples, rather than a complete list of suitable scoring architectures. Those skilled in the art will recognize that the teachings herein about obtaining event data and using machine learning to score alerts and other event data can also be applied with other scoring architectures.
[0240] Figure 3Event data 220 is illustrated. Examples of event data include events 302, which may be represented by an event data structure 304, and alerts 320, which may be represented by an alert structure 322. The event structure 304 may include an event type 306, an event ID 308, and event characteristics 310, such as the location 312 of the event, the ID 314 of the user associated with the event, the time point or time interval 316 at which the event occurred, and other data 318 that puts the event in context. The alert structure 322 may include an alert type 324, an alert ID 326, and underlying event structure 304 data. For example, the IDs 308, 326 may be GUIDs (globally unique identifiers), indexes in a master log, or other identifiers.
[0241] Figure 4 Illustrated are insight instances 402 that may be extracted from event data 220. A given insight instance 402 includes an insight value 404 and an insight type 406 that adds meaning to the value. Fig.12 An example of insight type 406 and qualifier 1202 is shown, which in some cases may be used to define insight type 406 . Fig.12 The insight types called out in 1204 include the number of processes created 1204 and the number of times an event occurred 1206. Each of these insight types can be qualified, for example, as to which machine 1208 or which time interval 1210 the insight type represents. Measurements 1212 of specific physical items (e.g., motion sensors, burglar alarm switches, or other physical devices) can also be used as insight types. Thus, as a very specific example, "front door opened between 10pm and 7am" is an insight type, "processes created on server 0113 in the past hour" is an insight type, and "failed login attempts made on any machine with user ID sysguru in the past five minutes" is an insight type.
[0242] like Figure 5As shown in , insight ranker 502 may rank insight instances 402, for example, by placing insight instances 402 in sorted list 504. Insight instances may be ranked according to any one or more event characteristics 310, and in particular, may be ranked according to their respective insight value contributions to aggregate insight value 234. For example, if three insight instances, represented here as inst-0001, inst-0003, and inst-0347, have respective values of 2, 6, and 4 for some specific insight type T, and an aggregate function sum() aggregates these values together to form 2+6+4=12 as the aggregate insight value, then the ranked list from largest contribution to smallest contribution is: inst-0003, inst-0347, inst-0001. The aggregate function summation may be referred to as "summation," "sum()," or "summation 802," and other aggregate functions may be similarly represented using various terms.
[0243] Figure 6 A feature vector 228 is illustrated, which may be created by a vectorizer based on insight values 404 extracted from event data 220. A given feature vector 228 includes multiple insight values 404, which are referred to as "feature values" in the feature vector. One or more of the feature values 602 are aggregate insight values 234, 404. Although in Figure 6 6, but each feature value 602 has an associated insight type 406 that adds meaning to the value. The type of a feature value may also be referred to as a "feature type" or "feature value type."
[0244] Although the aggregate function sum() is used in several examples herein, other aggregate functions 232 may be used instead of sum() or in combination with sum(). Figure 8 Examples of aggregate functions 232 are shown, including sum 802, product 804, minimum 806 (minimum), maximum 808 (maximum), count 810, distinct count 812, median 814, mean 816, mode 818, expression 820 (including one or more aggregate functions 232 with other evaluable code), and other routines 822 that are not called out (e.g., x percentile).
[0245] Figure 7 Various aspects of a machine learning classifier 236 are illustrated. The illustrated classifier includes a trained model 702 and an interface 704 for feeding a vector 228 to the trained model and receiving a corresponding confidence score 238 from the model 702. The interface may include an API, particularly when the classifier 236 is implemented as a cloud service. Fig.10The functionality of the classifier 236 is further illustrated in FIG. 1002. As indicated, the classifier 236 may be implemented using one or more of a neural network 1004, a deep learning network 1006, a convolutional network 1008, one or more decision trees 1010, a probabilistic classifier 1012, or a naive Bayes classifier model 1014. The classifier 236 may also employ Fig.10 One or more other machine learning techniques 1016 not called out in the example, such as support vector machines, k-nearest neighbor classification techniques, ensemble methods, or linear classification techniques. Fig. 9 Functionality 902 is shown that some embodiments avoid, including dimension reduction code 906 for reducing the dimension 904 to use a smaller vector, and padding code 908 for replacing undefined values 912 in the vector with some default value 910 (such as 0 or 1).
[0246] Fig.11 Some additional characteristics of operating some systems are shown 1100. The monitoring system can impose validity ranges 1102 on the confidence scores, or compare the confidence scores to boost thresholds 1104, or both. Performance can be measured using insight-score duration 1106, repetition rate 1108, or both or other measures.
[0247] Fig.13 Some examples of sources 1300 of event data 220 are shown. The data sources shown include intrusion detection or prevention systems 1302, management planes 1304, antivirus tools 1306, and firewalls 1312 and other logical security mechanisms 1318, motion detectors 1308 and other physical security mechanisms 1310, and computing resource controllers, such as storage controllers 1314 or cluster controllers 1316. Those skilled in the art will recognize that these examples are not necessarily comprehensive for any particular system.
[0248] Some embodiments use or provide a monitoring computing system configured to perform alert confidence scoring. The monitoring system 206 includes one or more processors 110, a memory 112 in operable communication with at least one processor, and a scoring-specific component. In this example, one scoring-specific component is an aggregator 230 that receives insight instances 402 when executed with at least one processor. Each insight instance 402 has an insight value 404 and an insight type 406, and each insight instance is associated with an alert identifier 326 that identifies an alert previously generated from the monitor 204 through monitoring of the monitored system 202. The aggregator 230 is configured to aggregate insight values of received instances having the same insight type.
[0249] In this example, another scoring-specific component is a vectorizer 226 that, when executed with at least one processor, creates a feature vector 228 that includes feature values 602. The feature values correspond to corresponding insight types 406. The feature vector includes at least one feature value 602 that is an aggregated insight value 234 generated by an aggregator 230 from a plurality of insight values 404. The feature vector is also associated with the identified alert 320.
[0250] In this example, another scoring-specific component is a classifier 236 that, when executed with at least one processor, accepts the feature vector 228. The classifier is configured to assign a confidence score 238 to the alert 320 based at least in part on the feature vector. The confidence score indicates a degree of confidence that the alert represents a threat to one or more operations of the monitored system 202. In this example, an output device such as the screen 126 is configured with the confidence score, for example, by displaying the score 238.
[0251] In some embodiments, monitoring computing system 206 includes, is in communication with, or is otherwise combined with insight extractor 224. Insight extractor 224 is configured to receive alert identifier 326 and extract insight instances 402 from event data associated with the alert so vectorizer 226 and aggregator 230 can use at least some of the extracted insight instances described herein.
[0252] In some embodiments, the monitoring computing system 206 is configured for enterprise production deployment in the sense that the total elapsed clock time 1106 from the start point when the insight instance is received by the aggregator until and including the end point when the output device is configured with a confidence score is less than 1 second. In some embodiments, the insight-score duration 1106 is less than 500 milliseconds, and in some embodiments less than 100 milliseconds. Achieving an acceptable insight-score duration 1106 (particularly in a production environment) may also be defined by those skilled in the art, industry standards, industry best practices, or service level agreements. Acceptable execution durations may also be defined using different starting and end points.
[0253] In some embodiments, the aggregator 230 is configured to aggregate the insight values by applying at least one of the following aggregation functions 232: sum 802, minimum 806, maximum 808, count 810, distinct count 812, mode 818, average 816, and in some embodiments, the aggregator 230 applies exactly one aggregation function 232. In some embodiments, the aggregator 230 applies one or more aggregation functions 232 but only generates an aggregated insight value 234 of a single type 406 for inclusion in the feature vector 228, while in other embodiments, some or all of the feature vectors 228 include two or more aggregated insight values 234 of different corresponding types 406.
[0254] In some embodiments, the aggregator 230 is configured to aggregate a given set of multiple insight values at least twice by applying at least two aggregation functions 232 to the multiple insight values, the at least two aggregation functions 232 being represented here as function A and function B. For example, three instance values 1, 3, 5 may be summed (function A being sum 802) to produce an aggregate insight value of 1+3+5=9, and the same three instance values 1, 3, 5 may also be averaged (function B being average 816) to produce an aggregate insight value of (1+3+5) / 3=3. In these example embodiments, the vectorizer 226 is configured to create a feature vector represented here as vector A, vector A containing feature values represented here as value A, which is an aggregate insight value resulting from the aggregator's application of function A to the multiple insight values, and the vectorizer 226 is further configured to create a feature vector represented here as vector B, vector B containing feature values represented here as value B, which is an aggregate insight value resulting from the aggregator's application of function B to the multiple insight values, value B being different from value A. In the particular example, the sum 9 will be placed in feature vector A and the average 3 will be placed in feature vector B. In an alternative approach (also derived from the data of the particular example), the vectorizer will create a single feature vector that contains the aggregate insight value 9 (the sum) and also contains the aggregate insight value 3 (the average). The corresponding insight type will include qualifiers 1202 such as "sum of" and "average of".
[0255] In these example embodiments having multiple aggregated insight values, each calculated from the same underlying insight value and placed in a corresponding feature vector, the classifier 236 is configured to assign a confidence score, represented here as score A, to an alert based at least in part on vector A, and to assign a confidence score, represented here as score B, to an alert based at least in part on vector B. The output device is then configured with at least one of the following: score A, score B, or a combined confidence score based at least on score A and score B, such as a minimum (score A, score B) or a maximum (score A, score B) or an average (score A, score B). In one variation, the classifier inputs the concatenation of vector A and vector B into a third vector C as input. The classifier then operates on a single vector C, producing a single score C.
[0256] In some embodiments, the monitoring computing system 206 does not have dimensionality reduction functionality (e.g., dimensionality reduction code 906). In some embodiments, the monitoring system 206 does not have fill functionality (e.g., fill code 908) to fill undefined values in the feature vector with default values. Some embodiments do not have both dimensionality reduction functionality (i.e., dimensionality reduction code 906) and fill functionality (i.e., fill code 908). As used herein, "no" means that the code in question either exists but is disabled, or does not exist and is not called out (it may be away from the monitoring system). In either case, when the monitoring system 206 does not rely on the functionality to process the event data 220 to assign the confidence score 238, the monitoring system 206 does not have specific functionality. The absence of specific functionality in a given embodiment helps to highlight the difference between the technology used in the embodiment and different technologies that may also involve feature vectors.
[0257] In some embodiments, the classifier 236 includes and uses at least one of the following: a naive Bayes classifier model 1014, a neural network 1004, a deep learning network 1006, a convolutional network 1008, a decision tree 1010, or a probabilistic classifier 1012. Some classifiers use two or more of these functionalities in parallel and then compare or combine their corresponding resulting scores 238 to arrive at a final confidence score 238, which can then be displayed to an administrator or used to drive control of the monitored system.
[0258] Some embodiments of the monitoring computing system 206 include an insight ranker 502 that, when executed with at least one processor, ranks the plurality of insight instances 402 according to their respective contributions to the confidence score 238. An output device is configured with a ranked list 504 of the ranked insight instances. The list 504 can help provide visibility and understanding of how the displayed confidence score 238 was calculated.
[0259] Visibility 1610 may also be provided by displaying event data 220 at one or more stages of processing (e.g., raw data from a monitor, extracted insight instances, aggregated insight values, or feature vectors). Visibility 1610 may also be provided by displaying other internal mechanisms (such as insight type definitions 1508) or by displaying what kind of classifier functionality 1002 is being employed (e.g., text phrases such as "neural network" or "naive Bayes").
[0260] Other system embodiments are also described herein, either directly or as system versions of the described methods or configured media, which are informed by the expanded discussion of computing hardware herein.
[0261] method
[0262] Fig.14 An example alarm confidence scoring method 1400 is illustrated. An entity performing the method obtains 1402 event data 220, for example, by receiving the event data over a network or by a direct electronic connection with a detector 214. The event data recipient entity extracts 1404 insight instances 402 from the event data, aggregates 1406 at least some of the extracted insight values 404, and vectorizes the insight data 404, 304, 322 to create a feature vector 228. The vector is then classified 1410 and assigned 1412 a confidence score 238. The score output from the classifier can be scaled to a desired range and granularity for display, such as [0...1] in increments of 0.1 or [0...100] in increments of 1. The final confidence score 238 can be reported 1416 via an output device, possibly after it is compared 1414 to a threshold 1104 (before or after scaling) to determine that it is large enough to report 1416. Steps may be repeated to process additional event data.
[0263] Fig.15The usage 1500 of an alert confidence scoring method, such as the method 1400, is illustrated. A monitored system 202 experiences events 302, some of which are detected by a detector 214. The detector generates an alert structure 322, such as a raw event (i.e., unscored) alert structure 322, and other electronic representations of the events 304. In this example, the alert has an alert identifier 326. The event data travels to an insight extractor 224, which from one perspective can be viewed as a "black box" that extracts insights 402 from the event data using insight type definitions 1508 previously provided by a human security expert 1510. The human expert may define N insight types based on their domain knowledge. The human expert may have decided, for example, that tracking both failed login attempts and successful login attempts helps improve network security, and therefore may have defined one instance type 406 that represents a failed login attempt and another instance type 406 that represents a successful login attempt. Insight extractor 224 accordingly extracts instance values 404 that match these defined types when the extractor encounters those values 404 in the event data. The extracted instances 402 proceed to aggregator 230 and vectorizer 226, which together produce feature vectors 228 that correspond to alert identifiers 326 and have at least one aggregated insight value 234. Feature vectors 228 are fed to virtual advisor 1502, which applies classifier model 702 to produce confidence scores for the vectors and, therefore, for the associated event data including the identified alert. Consumers of recommendations 1504 can use the confidence scores as triggers or guidance to apply control actions 1506 operating on the monitored system. The monitored system may experience additional events as a result of the control actions, and the monitoring and control loop shown will continue.
[0264] Unless otherwise indicated, the technical methods shown in the figures or otherwise disclosed are automatically performed, for example, by the virtual advisor 1502 or the monitoring system 206. To the extent that actions by a human administrator or other human are involved, the methods may also be performed partially automatically and partially manually, for example, entering a command to display details of an escalated alarm. None of the methods contemplated as innovative herein are entirely manual. In a given embodiment, zero or more steps of a method may be repeated, perhaps with different parameters or data to perform the operation. The steps in the embodiments may also be performed in a manner similar to that of the embodiment of the present invention. Fig.14 and Fig.16 The steps may be performed sequentially, in a partially overlapping manner, or completely in parallel. In particular, traversing flowchart 1400 or flowchart 1600 or Fig.15The event data monitors and controls the loop items to indicate the order of steps executed during the method, which may vary from one execution of the method to another execution of the method. The traversal order may also vary from one method embodiment to another method embodiment. Steps may also be omitted, combined, renamed, recombined, executed on one or more machines, or otherwise depart from the flow shown, so long as the method performed is operable and complies with at least one claim.
[0265] Some embodiments use or provide a method for assigning confidence scores to data, the data being generated by monitoring a monitored system. The method includes obtaining 1402 digital electronic event data, the digital electronic event data including or otherwise identifying at least one of: providing one or more event structures generated by monitoring, an alert structure generated by monitoring, or an alert identifier identifying an alert structure generated by monitoring. The method also includes extracting 1404 a plurality of insight instances from the event data, each insight instance having an insight value and an insight type, and automatically aggregating 1406 the insight values of at least two insight instances having the same insight type, thereby generating an aggregated insight value. The method also includes automatically vectorizing 1408 the insight data by creating a feature vector containing feature values, the feature values corresponding to the insight types, and the feature vector containing the aggregated insight value as one of the feature values. The method also includes classifying 1410 the feature vector using a machine learning driven classifier, and assigning 1412 at least one confidence score to the event data based on the results of the classification, wherein each confidence score indicates a degree of confidence that the event data represents a threat to one or more operations of the monitored system. The method also includes configuring 1416 an output device with the at least one assigned confidence score.
[0266] In some embodiments, the confidence score exceeds 1414 a predetermined threshold 1104, and the method accordingly further includes at least one of the following: notifying 1602 a human administrator of at least a portion of the event data; displaying a list of alarms ranked according to corresponding confidence scores; automatically taking 1604 control actions to achieve control of the monitored system; automatically commanding 1606 security tools to take security control actions to achieve security control of the monitored system; or automatically commanding 1608 resource management tools to take operational control actions to achieve operational control of the monitored system.
[0267] In some embodiments, the method includes providing 1610 visibility 1612 into the confidence score by displaying 1416 on an output device in a human-readable format at least one of: at least a portion of the obtained event data 220; at least one insight type definition 1508 utilized in extracting multiple insight instances; at least two of the extracted insight instances 402; the name of at least one aggregation function 232 utilized in aggregating insight values; or at least one feature value of the feature values 602.
[0268] In some embodiments, the method includes sending 1614 the assigned confidence score to an output device via a network connection. In some embodiments, other data such as visibility data 1612, list 504, raw data 220, or a combination thereof is also sent 1614 over one or more network connections.
[0269] In some embodiments, the method repeats multiple times at least the following steps: extracting 1404, aggregating 1406, vectorizing 1408, classifying 1410, and assigning 1412. In some of these embodiments, the method further comprises: comparing 1414 the resulting assigned confidence scores with predetermined thresholds 1104, issuing 1628 an alarm when one of the assigned confidence scores exceeds the predetermined threshold, and refraining 1630 from issuing an alarm when one of the assigned confidence scores does not exceed the predetermined threshold. Issuing an alarm based on a high confidence score for the alarm is an example of an upgraded alarm.
[0270] In some embodiments, the method performs repetition in the monitoring system by automatically repeating at least the following multiple times: extraction 1404, aggregation 1406, vectorization 1408, classification 1410 and allocation 1412. Other steps can also be used to define repetition, and therefore define a repetition rate. In some of these embodiments, the method performs the repetition at an average rate, and the average rate is at least one repetition per minute over a duration of at least one hundred hours. In some embodiments, the repetition rate is at least one repetition per minute over a duration of at least ten hours. In some embodiments, the repetition rate is at least one repetition per minute over a duration of at least two hours. In some embodiments, the repetition rate is at least ten repetitions per minute over a duration of at least ten hours. In some embodiments, the repetition rate is at least ten repetitions per minute over a duration of at least two hours. An acceptable repetition rate for implementation (particularly in a production environment) can also be defined by those skilled in the art, industry standards, industry best practices, or service level agreements.
[0271] In some embodiments, vectorization 1408 creates a feature vector that includes one or more feature values corresponding to one or more insight types 406, where an insight type represents at least one of the following: the number of processes created on machines in the monitored system; the number of processes created on machines in the monitored system within a specified time interval; the number of times a particular type of event occurs on one or more machines in the monitored system; or the number of times a particular type of event occurs on one or more machines in the monitored system within a specified time interval.
[0272] In some embodiments, the method obtains 1402 digital electronic event data 220, the digital electronic event data 220 including or identifying at least one of the following: digital electronic event data generated by a host-based intrusion detection system 1302; digital electronic event data generated by a network-based intrusion detection system 1302; digital electronic event data generated by an antivirus tool 1306; digital electronic event data generated by a firewall 1312; digital electronic event data generated by a motion detector 1308; digital electronic event data generated by a storage controller 1314; digital electronic event data generated by a cluster controller 1316; digital electronic event data generated by a management plane 1304; digital electronic event data generated by a mechanism 1310 configured to enhance physical security; or digital electronic event data generated by a mechanism 1318 configured to enhance logical security.
[0273] In some embodiments, extracting 1404 a plurality of insight instances from the event data includes accessing 1638 a log or other repository and mapping 1648 alerts to at least some of the insight instances.
[0274] Configured storage media
[0275] Some embodiments include a configured computer-readable storage medium 112. The storage medium 112 may include a disk (magnetic, optical, or otherwise), RAM, EEPROM or other ROM, and / or other configurable memory, and in particular includes a computer-readable storage medium (not just a propagating signal). In particular, the configured storage medium may be a removable storage medium 114, such as a CD, DVD, or flash memory. General purpose memory, which may be removable or non-removable and may be volatile or non-volatile, may be configured into an embodiment using items such as an aggregator 230, confidence score 238, and virtual advisor 1502 in the form of data 118 and instructions 116 (which are read from the removable storage medium 114 and / or another source (such as a network connection)) to form a configured storage medium. The configured storage medium 112 enables a computer system to perform the technical process steps for confidence scoring an alert as disclosed herein. Thus, the accompanying drawings help illustrate configured storage medium embodiments and process embodiments, as well as system and process embodiments. In particular, Fig.14 , Fig.15 or Fig.16 Any of the process steps shown in or otherwise taught herein may be used to help configure a storage medium to form a configured storage medium embodiment.
[0276] Some embodiments use or provide a storage medium 112, 114 configured with code that, when executed by one or more processors, performs a confidence scoring method, the method comprising: obtaining 1402 digital electronic event data that includes or otherwise identifies an alert previously generated from a monitor through monitoring of a monitored system; extracting 1404 a plurality of insight instances from the event data, each insight instance having an insight value and an insight type; aggregating 1406 the insight values of at least two insight instances having the same insight type to produce an aggregated insight value; and vectorizing 1408 the insight data by creating a feature vector comprising feature values, wherein the feature values correspond to the insight types, and the feature vector comprises the aggregated insight value as one of the feature values. The confidence scoring method also comprises: classifying 1410 the feature vector using a machine learning driven classifier, and assigning 1412 at least one confidence score to the alert based on the result of the classification. The confidence score may indicate a degree of confidence that the alert represents a threat to one or more operations of the monitored system. The confidence scoring method also includes configuring 1416 an output device using the at least one assigned confidence score so that functionality of the monitored system can be improved based on high confidence scores and the underlying event data.
[0277] In some embodiments, the method is further characterized by at least three of the improvements listed below: (a) aggregating 1406 includes applying at least one of the following aggregation functions: sum, minimum, maximum, count, distinct count, x-percentile, mode, mean, or median; (b) the method includes: applying at least two aggregation functions to a particular set of insight instances, thereby producing at least two different aggregated insight values 234, and creating and classifying 1410 at least two corresponding feature vectors, which produces at least two corresponding confidence score assignments for the particular set of insight instances; (c) the method includes: obtaining, via a network connection, from a cloud service sending 1614 at least the assigned confidence score to an output device; (d) the method includes: automatically commanding 1606 a tool to take a control action to mitigate a security risk of the monitored system; (e) the method includes: automatically commanding 1608 a tool to take a control action to change the accessibility of a computing resource of the monitored system; (f) the method includes: ranking 1642 a plurality of insight instances according to their respective contributions to the confidence score; and configuring 1644 an output device using the list of ranked insight instances; or (g) extracting 1404 a plurality of insight instances from event data includes extracting 1638 event data from log 212. In some embodiments, the method is characterized by at least four of the listed improvements. In some embodiments, the method is characterized by at least five of the listed improvements. More generally, in a given case, the method may be characterized by from zero to all of the listed improvements.
[0278] Events and Alerts
[0279] Often, in network security, a distinction is sometimes made between events and alerts, treating them as two different but related things. An event is something that occurs and is noticed by some monitoring or logging agent, for example, a failed login attempt, a successful login, an application launched, a packet detected on a particular port, and so on. The number of events noticed in a system can be large. Most events are routine and do not indicate any problems. In contrast, an alert (in this view) is a response to some event or group of events, and indicates that there may be a problem. For example, an alert may be generated by: several failed logins to an administrator account, followed by a successful login to that account; or by the launching of a Metasploit application; or by the detection of traffic on a port commonly used for online gaming.
[0280] This distinction between events and alarms also applies in the context of the present description, with some refinements. For example, the virtual advisor 1502 can be viewed as another layer on top of the detection algorithm in the sense that the virtual advisor takes as input events that the underlying detection algorithm considers to be alarms. The virtual advisor gives these alarms a confidence score to indicate whether they are indeed alarms or just normal events. For example, the virtual advisor can elevate an alarm to an alarm state, or take a control action based on the alarm and its relatively high confidence score.
[0281] drill
[0282] As Fig.15 A further description of the data flow in one embodiment consistent with the usage shown, a drill is provided below. Those skilled in the art will recognize that many other drills and devices with this particular alert scoring architecture or with other alert scoring architectures may include one or more insight types, insight values, alert formats, or other details that differ from this particular drill, even though the device discussed embodies the teachings presented herein.
[0283] These data values are simulated values, but realistically, they help illustrate how specific data values will be processed and generated in specific embodiments. They are not actual real-time production data.
[0284] 1. The detector receives the event: Failed login attempt on device PatsTablet at 23:55:01.
[0285] 2. The detector receives the event: Failed login attempt on device PatsTablet at 23:55:16.
[0286] 3. The detector receives the event: Failed login attempt on device PatsTablet at 23:55:31.
[0287] 4. Detector receives event: Successful login on device PatsTablet at 23:55:46.
[0288] 5. Detector receives event: At 23:56:47, process rmt-devl was started on device PatsTablet.
[0289] 6. Detector receives event: At 23:57:02, process bitminer00 started on device PatsTablet.
[0290] 7. Detector receives event: At 23:57:22, process bitminer01 started on device PatsTablet.
[0291] 8. Detector receives event: At 23:57:42, process bitminer02 started on device PatsTablet.
[0292] 9. The detector sends alert #123456 and the underlying eight events to the insights extractor. This alert and its underlying events can also be logged and can also be sent to other software, such as SIEM.
[0293] 10. The insights extractor sends the following insights instances to the aggregator and vectorizer: {Login failed, 1, 123456}, {Login failed, 1, 123456}, {Login failed, 1, 123456}, {Unknown process, 1, 123456}, {Unknown process, 1, 123456}, {Unknown process, 1, 123456}, {Unknown process, 1, 123456}, {Unknown process, 1, 123456}.
[0294] 11. The aggregator aggregates the login failure instances into {login failure, 3, 123456}.
[0295] 12. The aggregator aggregates the unknown process instances into {unknown process, 4, 123456}.
[0296] 13. The vectorizer produces the following feature vector: [Login failures: 3, unknown processes: 4, alert id: 123456].
[0297] 14. The feature vector is fed to the virtual suggester which uses the model to produce the following results:
[0298] [Login failure: 3, unknown process: 4, alert id: 123456, is it an incident: 0.78], which indicates that the probability that the alert represents a security incident is 0.78. Alternatively, the output may be formatted as [alert id: 123456, is it an incident: 0.78].
[0299] More generally, those skilled in the art will recognize that not every part of the present disclosure or any specific detail therein is required to satisfy legal standards such as enabling, written description, or best mode. Moreover, the embodiments are not limited to the specific programming languages, tool contexts, identifiers, fields, class definitions, or other implementation choices described herein. Any apparent conflict with any other patent disclosure, even from the owner of the present invention, does not interpret the effect of the claims set forth in this patent disclosure.
[0300] Model Training
[0301] During training, the trained model gets alerts and their labels. One constructs an alert feature vector and submits it to the model for learning. During classification, one again constructs a feature vector for the alert and calculates its score. To train the virtual advisor model 702, the model is fed with training data in the form of a feature vector with associated labels. The label indicates whether the feature vector indicates (or does not) a safety incident (for safety training) or an operational incident (for resource management or other operational training). The training data is prepared manually, for example, by a human safety expert or a human operations manager. Note that at this stage, the training algorithm can use machine learning methods to perform feature selection in order to improve the quality of the model. Feature selection can choose to use a subset of the N features.
[0302] Additional Examples and Observations
[0303] and Fig.15 Consistent with aspects of other teachings herein, in some implementations of the alarm confidence scoring technique, some kind of detector 214 receives events, and in response to some of those events, the detector provides an alarm 320. This may be referred to as a "raw alarm" to indicate that it has not yet been processed by the virtual advisor 1502. For example, the detector may be a host-based intrusion detection system, a network-based intrusion detection system, antivirus software, a firewall, a motion detector, or some other physical or technical (i.e., software) security mechanism. In some implementations, the alarm includes a description of one or more events, and the alarm also includes an alarm number so that it can be tracked when managing the alarm. In other implementations, a description of the event is not directly communicated; all the information received by the insight extractor is the alarm number (e.g., a unique ID that is not necessarily assigned in sequence), and some description of the characteristics of the alarm. The description of the alarm may be domain-specific or implementation-specific.
[0304] More generally, the original alert includes a description of the event or alert characteristics, or a reference to such information. The alert 320 itself may be implemented simply as an alert number, when the alert number serves as a pointer, index, or key to locate some description of the circumstances that caused the alert or a description of the alert characteristics that suggest or summarize those circumstances. In addition to indicating the nature of the event 302 that caused the alert, the alert characteristics may also include, for example, (a) a machine ID, subnet address, process ID, or other location information, (b) the user ID(s) involved, (c) the time(s) of occurrence, (d) the security credentials that exist, and (e) other relevant information.
[0305] In some devices, an alert enters module 224, which can be considered a "black box", but is referred to here as an "insight extractor". The insight extractor 224 uses insight type definitions 1508 provided by a human security expert to find insight instances in the event associated with the original alert. Each insight instance includes an insight type and an insight value. The insight extractor generates one or more insight instances from a given alert. In some devices, the insight extractor uses the insight type definitions provided by the human security expert together with logic (hardware and software operating together) based on security expert knowledge to extract 1404 insight instances that characterize the alert. To do this, the insight extractor uses a repository 212 (such as a log of the monitored system) to associate insight instances with the alert. For example, assuming an alert occurred on machine M (M would be part of the alert description), the insight instance could state the number of processes created on machine M within a certain time interval. This information is extracted from the system log.
[0306] In some devices, insight instances and their associated alert numbers enter a module called an "aggregator and vectorizer." The aggregator portion of the module aggregates the corresponding values of multiple instances of the same insight type, for example, by summing them. Other aggregation functions may be used instead of or in addition to summing. The vectorizer portion of the module then uses the aggregated instance values to create a feature vector. Each entry in the feature vector corresponds to an insight type. In some cases, human security experts select N insight types to include in the feature vector. The vector is N-dimensional, and each entry corresponds to an insight type. The selection of the N insight types is done earlier, that is, they are "predefined," as opposed to selecting N insight types in response to an alert. When an instance for a given alert does not match a predefined insight type, an exception may be raised, the undefined values may be filled with default values, or the processing of the insight instance may be terminated without generating a feature vector, depending on the specific implementation.
[0307] As processing proceeds, the feature vector (still tracked with the alert ID) is fed into a virtual advisor 1502, which has a trained machine learning model 702. Based on the model, the virtual advisor assigns a confidence score to the feature vector, which indicates the degree of confidence that the feature vector indicates (or does not indicate) a security event. Therefore, the hierarchy used in this example is event->alert->incident, with events being the lowest severity and incidents being the highest severity. In this example, the feature vector, tracking number (i.e., alert ID), and confidence score 238 are given to a consumer of recommendations 1504. In other versions, the feature vector is not returned to the consumer of recommendations; only the confidence score of the alert is returned from the advisor 1502. The consumer of recommendations 1504 can be a human administrator 104 or software such as security information and event management (SIEM) software. The consumer of recommendations can then take one or more control actions 1506 in response to the recommendations, such as further investigation, escalation of the alert, attempt to contain the incident, failover to redundant resources, and the like.
[0308] Some embodiments include a system 206 having: an input interface configured to receive insight instances 402, each of which includes a type and a value; an aggregator 230 code that aggregates the values of received instances of the same type; a vectorizer 226 code that creates a feature vector 228 containing feature values corresponding to the insight type; an output interface configured to provide the feature vector to a trained model 702, and the trained model is configured to assign a confidence score 238 to the feature vector, the confidence score 238 indicating: the corresponding feature vector represents the confidence of the security incident.
[0309] In some contexts, machine learning algorithms operate on a collection of "examples" from some real-world domain. In some cases of particular interest herein, the examples are alerts 320. To help avoid confusion, examples in the sense of "examples fed to a machine learning algorithm" are denoted herein as "instances" and are shown as insight instances 402.
[0310] Machine learning algorithms transform instances into mathematical N-dimensional vectors of fixed size N. Each vector entry contains a value, such as a number. To perform the transformation, such as through "feature engineering", N "features" are defined based on the problem domain. So if the instance represents a patient, the features could be temperature, blood type, etc. Each vector entry represents a feature. For a given instance, the vector entry holds the value of the feature for that instance.
[0311] As another example, assume that an embodiment identifies two predefined insight types 406, namely, type A, which represents the number of incidents that occurred on a machine, and type B, which represents the number of processes spawned by a given process. Assume that an alert occurs where, for insight type A, there are three insight instances with values: 2, 2, 4, and for insight type B, there are four values: 1, 12, 2, 4. Prior to receiving the alert, the embodiment does not know the number of insight instances it will process. In this example, assume that the feature vector for the event has two entries (i.e., two time slots, two values 602), and the aggregation function used is sum 802. Then, after aggregation and vectorization, the first entry of the vector holds the sum of the values of insight A (2+2+4), and the second entry holds the sum of the values of insight B (1+12+2+4).
[0312] Although the number of features can be fixed (in some arrangements, all feature vectors are the same structure - they are N-dimensional), the number of insight instances underlying a given feature vector can vary. In some cases, an alert is associated with an infinite number of insight instances. These are not structured as multiple N-dimensional vectors. Approaches that reduce vector dimensionality or pad the vector to place a value in each of the N slots, regardless of the underlying event data, do not apply to the challenge of meaningfully processing an arbitrary number of insight instances as feature vectors with a fixed number of values 602. However, in hindsight, it can be observed that each instance insight is associated with an insight type, and the number of insight types is bounded. This observation can help one understand the teachings described herein for constraining insights to match finite vectors.
[0313] In other words, in some embodiments, the number of insight types is fixed to N and is predefined by human security experts. Given an event, the black box program generates multiple insight instances for the event. Each insight instance has a value and it belongs to certain (single) instance types. For each insight type, an embodiment aggregates the values of all its instances into a single value. Each insight type serves as a feature and its aggregate value serves as the value of the feature. These are used by the classifier algorithm. Therefore, there are ultimately N insight types, which correspond to N features, and each feature has a value. In view of this, events can be represented by corresponding N-dimensional feature vectors, and familiar machine learning classifier techniques are utilized. The number of features is bounded and is N. For a given alert, the number of values for each insight is unbounded, so an embodiment aggregates them into a single value.
[0314] In some embodiments, the insight extractor executes a program that follows the logic of a security expert. The program generates a set of insights that characterize the security aspects of an event. The program takes an event as input and generates a set of insights, each with multiple values. This can be achieved, for example, by using a set of rules that output a security insight for each event. Since the event rate is typically higher in a cloud setting, the program is automated. The output of the program can be used as features for the supervised learning part, which is again done by a computer.
[0315] This article gives several examples of aggregation functions that can be used, including sum, minimum, maximum, count, distinct count, average (also known as mean), median, x percentile, etc. In some cases, more than one aggregation function may be useful. For example, you can choose to use minimum, maximum, and average. In this case, for each insight type, an embodiment can have three features (rather than one feature), namely, features that are the minimum, maximum, and average of the values based on the insight instance, respectively. Therefore, an embodiment using three aggregation functions can have a total of 3*N features. In order to determine which aggregation function is useful or ineffective, a machine learning procedure for feature selection can be performed.
[0316] One implementation uses a sum function for aggregation 1406 and accumulates the contributions of all instances of the corresponding feature type. However, other aggregation functions may be used, such as maximum, minimum, etc. The implementation also normalizes the values of each insight type 1646 before aggregating using z-score normalization.
[0317] When an event is represented by an N-dimensional feature vector on an insight type, the implementation sets the value of each insight type feature by its corresponding aggregate value. In the case of a fixed-size feature vector 228, the implementation applies supervised machine learning techniques to calculate a confidence score 238 for the alert. To this end, a classifier is trained on a collection of events, such as process creation events, that have been previously seen in the monitored system. For each event, the implementation constructs its feature vector, for example, as previously described. In addition, a security analyst labels each event as malicious or non-malicious. At runtime, given an unseen event, the trained classifier will run and output its confidence score as to whether the event is malicious. In addition or alternatively, an operations analyst can label each event with a label for non-security monitoring, for example, close to full capacity or not close to full capacity, performing well or performing poorly, thrashing or not thrashing, reliable or unreliable, and so on. Then, at runtime, given an unseen event, the trained classifier will run and output a confidence score as to whether the event indicates an operational problem.
[0318] Some embodiments use logistic regression to implement classification 1410. The classifier can be evaluated on a test set of labeled events using metrics such as average mean precision (MAP) and accuracy. In an internal test on the data set, the MAP was approximately 0.92 and the accuracy was approximately 0.85.
[0319] Some embodiments provide confidence levels for alerts classified by security experts. They do this by combining human expert analysis of training data with machine learning algorithms.
[0320] Some embodiments support or enable the use of supervised machine learning on a variable and dynamic number of insights. The number of insights varies from event to event and is not known a priori, but is instead generated at runtime for a given event based on an expert's logic.
[0321] Some additional combinations and variations
[0322] Any of these combinations of code, data structures, logic, components, communications, and / or their functional equivalents may also be combined with any of the above systems and variations thereof. The process may include any of the steps described herein in any subset or combination or sequence that is operable. Each variation may occur alone or in combination with any one or more other variations. Each variation may occur with any process, and each process may be combined with any one or more other processes. Each process or combination of processes (including variations) may be combined with any of the above configured storage media combinations and variations.
[0323] in conclusion
[0324] In short, benefiting from the teachings provided herein, embodiments can be used to automatically triage 1650 security and operational alerts. Insight instances 402 extracted 1404 from raw event data 220 associated with alerts 320 are aggregated 1406, vectorized 1408, and assigned 1412 confidence scores 238 through machine learning-based classification 1410. Confidence scores enable heavy-duty administrators 218 and controls 216 to focus their attention and computing resources on where they are most likely to protect or improve the functionality of the monitored system 202. Even if the number of instance values is not known before receiving the event data, the feature vector 228 receives a broad basis in the underlying instance values 404 through aggregation 1406. Visibility 1612 to the confidence scoring process can be provided to allow adjustments to the embodiment (e.g., by varying the aggregation function 232 used or limiting the number of values 404 for each aggregated insight value 234). Visibility can also inform further training of the classifier model 702. Performance metrics 1106, 1108, 1636 were defined, and production-level performance 1634 was achieved.
[0325] Although specific embodiments are explicitly shown and described herein as processes, configured storage media, or systems, it should be understood that a discussion of one type of embodiment also generally extends to other types of embodiments. Fig.11 and Fig.13 The description of the processes performed also helps describe the configured storage medium, and helps describe the technical effects and operations of systems and manufactures similar to those discussed in conjunction with the other figures. It is not intended that limitations from one embodiment must be read into another embodiment. In particular, the processes are not necessarily limited to the data structures and arrangements presented when discussing systems or manufactures such as configured memory.
[0326] Those skilled in the art will appreciate that implementation details may be relevant to specific code, such as specific APIs, specific fields, and specific sample programs, and therefore need not appear in each embodiment. Those skilled in the art will also appreciate that program identifiers and some other terms used when discussing details are implementation-specific, and therefore need not be relevant to each embodiment. Nevertheless, although they are not necessarily required to exist here, such details may help some readers by providing context, and / or may illustrate some of the many possible implementations of the technology discussed herein.
[0327] References herein to embodiments having some feature X and references elsewhere herein to embodiments having some feature Y do not exclude embodiments having both feature X and feature Y, unless such exclusion is expressly stated herein. All possible negative claim limitations are within the scope of the present disclosure, since any feature specified as part of an embodiment may be expressly deleted from other embodiments even if no specific exclusion is given in any example herein. The term "embodiment" is used herein merely as a more convenient form of "process, system, article, configured computer-readable storage medium, and / or other example of the teachings herein applied in a manner consistent with applicable law." Thus, a given "embodiment" may include any combination of the features disclosed herein, as long as the embodiment is consistent with at least one claim.
[0328] In each embodiment, not every item shown in the figure needs to exist. On the contrary, the embodiment may include (multiple) items that are not explicitly shown in the figure. Although some possibilities are shown in the text and the drawings by specific examples here, the embodiment can be separated from these examples. For example, the specific technical effects or technical features of the examples can be omitted, renamed, grouped differently, repeated, instantiated differently in hardware and / or software, or a mixture of effects or features that appear in two or more examples. In some embodiments, the functionality shown at one location can also be provided at different locations; for example, those skilled in the art recognize that functional modules can be defined in various ways in a given implementation without ignoring the required technical effects from a collection of interactive modules that are considered as a whole.
[0329] Reference is made to the drawings throughout the text by reference numerals. Any apparent inconsistency in the wording associated with a given reference numeral in the drawings or text should be understood to simply expand the scope to which the numeral refers. Even if the same reference numeral is used, different instances of a given reference numeral may refer to different embodiments. Similarly, a given reference numeral may refer to a different embodiment.
[0330] can be used to refer to verbs, nouns, and / or corresponding instances of each, e.g., processor 110
[0331] The instructions may be processed 110 by executing the instructions.
[0332] As used herein, terms such as "a", "an" and "the" include one or more of the indicated items or steps. In particular, in the claims, reference to an item generally means that there is at least one such item, and reference to a step indicates at least one instance of performing the step.
[0333] Headings are for convenience only; information on a given topic may be found outside the section whose heading indicates that topic.
[0334] All claims and the abstract as filed are part of the specification.
[0335] Although exemplary embodiments have been shown in the drawings and described above, it is clear to those skilled in the art that various modifications may be made without departing from the principles and concepts set forth in the claims, and such modifications do not need to encompass the entire abstract concept. Although the subject matter is described in a language specific to structural features and / or program actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific technical features or actions described in the claims. Each component or aspect or technical effect identified in a given definition or example does not necessarily exist or is utilized in every embodiment. Instead, the specific features, actions, and effects described are disclosed as examples to be considered when implementing the claims.
[0336] To the maximum extent permitted by law, all changes that come within the general abstract concept and within the meaning and range of equivalency of the claims are intended to be embraced within their scope.
Claims
1. A monitoring computing system configured to perform alarm confidence scoring, the monitoring computing system comprising: one or more processors; a memory in operable communication with at least one processor; an aggregator, which, when executed with at least one processor, receives insight instances, each insight instance having an insight value and an insight type, each insight instance being associated with an alert identifier, the alert identifier identifying an alert previously generated from a monitor through monitoring of events of a monitored system, the aggregator being configured to aggregate insight values of received instances having the same insight type; a vectorizer that, when executed with at least one processor, creates a feature vector comprising feature values, the feature values corresponding to insight types, the feature vector comprising at least one feature value, the at least one feature value being an aggregated insight value generated by the aggregator from a plurality of insight values, the feature vector being associated with the alert; a classifier that, when executed with at least one processor, receives the feature vector, the classifier being configured to assign a confidence score to the alarm based at least in part on the feature vector, the confidence score indicating a degree of confidence that the alarm represents a threat to one or more operations of the monitored system; as well as An output device is configured with a list of alarms ranked relative to each other based on respective confidence scores assigned by the classifier.
2. The monitoring computing system of claim 1, in combination with an insight extractor configured to receive the alert identifier and extract the insight instance from event data associated with the alert.
3. A monitoring computing system according to claim 1, wherein the monitoring computing system is configured for enterprise production deployment, wherein the total elapsed clock time from a starting point when the aggregator has received the insight instance until and including an ending point when the output device is configured with the confidence score is less than one second.
4. The monitoring computing system of claim 1 , wherein the aggregator is configured to aggregate insight values by applying at least one of the following aggregation functions: sum, minimum, maximum, count, distinct count, x-percentile, mode, mean, or median.
5. The monitoring computing system according to claim 1, wherein: The aggregator is configured to aggregate the plurality of insight values at least twice by applying at least two aggregation functions to the plurality of insight values, the at least two aggregation functions being denoted herein as Function A and Function B; The vectorizer is configured to create a feature vector, represented herein as vector A, containing feature values, represented herein as value A, which is an aggregated insight value resulting from application of function A to the plurality of insight values by the aggregator; The vectorizer is configured to create a feature vector, represented herein as vector B, containing feature values, represented herein as value B, the value B being an aggregated insight value resulting from application of function B by the aggregator to the plurality of insight values, the value B being different from value A; The classifier is configured to assign a confidence score, denoted herein as score A, to the alert based at least in part on the vector A; The classifier is configured to assign a confidence score, denoted herein as score B, to the alert based at least in part on vector B; as well as The output device is configured by at least one of: score A, score B, or a combined confidence score based at least on score A and score B.
6. The monitoring computing system of claim 1, wherein the monitoring computing system is free of each of the following: Dimensionality reduction functionality; and Fill functionality, which fills undefined values in a feature vector with default values.
7. The monitoring computing system according to claim 1, wherein the classifier comprises at least one of the following: Naive Bayes classifier model; Neural networks; Deep learning networks; Convolutional networks; Decision trees; or Probabilistic classifier.
8. The monitoring computing system according to claim 1 further includes an insight ranker, which, when executed by at least one processor, ranks the multiple insight instances according to their respective contributions to the confidence score, and wherein the output device is also configured with a ranked list of the ranked insight instances.
9. A method for assigning a confidence score to data generated by monitoring events of a monitored system, the method comprising: obtaining digital electronic event data, the digital electronic event data including or otherwise identifying at least one of: one or more event structures generated by the monitoring, an alert structure generated by the monitoring, or an alert identifier identifying an alert structure generated by the monitoring; extracting a plurality of insight instances from the event data, each insight instance having an insight value and an insight type; automatically aggregating insight values of at least two insight instances having the same insight type to produce an aggregated insight value; automatically vectorizing the insight data by creating a feature vector comprising feature values, the feature value corresponding to an insight type, the feature vector comprising the aggregated insight value as one of the feature values; classifying the feature vector using a machine learning driven classifier; assigning at least one confidence score to the event data based on a result of the classification, each confidence score indicating a degree of confidence that the event data represents a threat to one or more operations of the monitored system; as well as configuring an output device with a list of alarms, the list of alarms being ranked according to respective confidence scores; Wherein the method is performed using monitored system data, the monitored system data including monitored events and also including alarms based on the monitored events, and wherein alarms are distinguished from monitored events in that some monitored events are not associated with any alarms.
10. The method according to claim 9, wherein the method further comprises at least one of the following: notifying a human administrator of at least a portion of the event data; Automatically taking control actions to achieve control of the monitored system; Automatically instruct the security tool to take security control actions to achieve security control of the monitored system; or The resource management tool is automatically commanded to take operational control actions to implement operational control of the monitored system.
11. The method of claim 9, further comprising providing visibility into the confidence score by displaying on the output device in a human-readable format at least one of: at least a portion of the event data obtained; at least one insight type definition to be utilized when extracting the plurality of insight instances; at least two of the extracted insight instances; the name of at least one aggregation function to be utilized when aggregating insight values; or At least one of the eigenvalues.
12. The method of claim 9, wherein the method further comprises sending at least the assigned confidence score to the output device via a network connection.
13. The method according to claim 9, wherein the method repeats at least the following multiple times: extracting, aggregating, vectorizing, classifying and assigning, and wherein the method further comprises: comparing the resulting confidence score of the assignment to a predetermined threshold; issuing an alarm when one of the assigned confidence scores exceeds the predetermined threshold; as well as When one of the assigned confidence scores does not exceed the predetermined threshold, refraining from issuing an alarm.
14. The method of claim 9, wherein the method performs repetition in a monitoring system by automatically repeating at least a plurality of the following: extraction, aggregation, vectorization, classification, and assignment, and wherein the method performs the repetition at an average rate of at least one repetition per minute over a duration of at least one hundred hours.
15. The method of claim 9, wherein vectorization creates a feature vector, the feature vector comprising one or more feature values corresponding to one or more insight types, the one or more insight types representing at least one of the following: the number of processes created on the machine of the monitored system; The number of processes created on the machine of the monitored system during a specified time interval; the number of times a predetermined type of event occurs on one or more machines of the monitored system; or The number of times events of a predetermined type occur on one or more machines in the monitored system within a specified time interval.
16. The method of claim 9, wherein the method comprises obtaining digital electronic event data, the digital electronic event data comprising or otherwise identifying at least one of: Digital electronic event data generated by host-based intrusion detection systems; Digital electronic event data generated by network-based intrusion detection systems; Digital electronic event data generated by anti-virus tools; Digital electronic event data generated by the firewall; digital electronic event data generated by a motion detector; digital electronic event data generated by a storage controller; digital electronic event data generated by the cluster controller; Digital electronic event data generated by the management plane; digital electronic event data generated by mechanisms configured to enhance physical security; or Digital electronic event data generated by a mechanism configured to enhance logical security.
17. The method of claim 9, wherein extracting a plurality of insight instances from the event data comprises: A log or other repository is accessed and alerts are mapped to at least some of the insight instances.
18. A storage medium configured with codes, wherein when the codes are executed by one or more processors, a confidence scoring method is performed, the method comprising: obtaining digital electronic event data including or otherwise identifying an alarm previously generated from a monitor through monitoring of a monitored system; extracting a plurality of insight instances from the event data, each insight instance having an insight value and an insight type; aggregating insight values of at least two insight instances having the same insight type to produce an aggregated insight value; vectorizing the insight data by creating a feature vector comprising feature values, the feature value corresponding to an insight type, the feature vector comprising the aggregated insight value as one of the feature values; classifying the feature vector using a machine learning driven classifier; Based on a result of the classification, assigning at least one confidence score to the alarm, each confidence score indicating: a degree of confidence that the alarm represents a threat to one or more operations of the monitored system; as well as configuring an output device with a list of alarms, the list of alarms being ranked according to respective confidence scores; Wherein the method is performed using monitored system data, the monitored system data including monitored events and also including alarms based on the monitored events, and wherein alarms are distinguished from monitored events in that some monitored events are not associated with any alarms.
19. The storage medium according to claim 18, wherein the method is further characterized by at least three of the following improvements: Aggregation includes applying at least one of the following aggregation functions: sum, minimum, maximum, count, distinct count, mode, mean, or median; The method includes applying at least two aggregation functions to a predetermined set of insight instances, thereby producing at least two different aggregated insight values, and creating and classifying at least two corresponding feature vectors, which produce at least two corresponding confidence score assignments for the predetermined set of insight instances; The method comprises: transmitting the assigned confidence score from the cloud service to the output device via a network connection; The method includes: automatically commanding a tool to take control actions to mitigate security risks to the monitored system; The method includes: automatically commanding a tool to take a control action to change the accessibility of a computing resource of the monitored system; The method comprises ranking the plurality of insight instances according to their respective contributions to the confidence score and configuring the output device with a ranked list of the insight instances; or Extracting a plurality of insights from the event data instance includes extracting event data from a log.
20. The storage medium of claim 19, wherein the method is characterized by at least four of the listed improvements.
