Program Execution Anomaly Detection for Cybersecurity

Abstract

The program is executed in a first operating mode in a controlled environment according to normal operation without malicious behavior. An acceptable behavior model is generated based on a sequence of events that occur during normal operation of the program. The acceptable behavior model represents the normal behavior of the program that occurs during normal operation. The program is then executed in a second operating mode in the operating environment. The operational sequence of events (determined during the second operating mode) is compared to the acceptable behavior model. When there is a match between the operational sequence of events and the acceptable behavior model, execution in the second operating mode continues. When there is no match between the operational sequence of events and the acceptable behavior model, execution in the second operating mode is aborted.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 415,852, filed October 13, 2022, and entitled "Program Execution Anomaly Detection System for Cyber ​​Security," which is incorporated herein by reference in its entirety for all purposes. [Background technology]

[0002] Software programs are designed to perform specific expected functions and behave in expected ways. This expected execution and behavior can be described as “normal behavior.” However, in relatively complex systems, errors, bugs, weaknesses, and / or vulnerabilities are inevitable, potentially rendering the program or the computer system on which it runs vulnerable to data leaks, malware, and other security breaches or malicious attacks. Traditional approaches to this problem may detect security breaches after they occur by monitoring “big picture” anomalies, such as anomalous network traffic, CPU usage, memory access, file usage, and / or data usage. Many of these approaches typically involve post-mortem analysis of log files, which may be too late to prevent a security breach or other erroneous or unexpected behavior from occurring. Other traditional approaches may provide tools that can be used during software development to analyze code and identify weaknesses that could potentially be exploited by malicious actors. For example, development tools may scan code for vulnerabilities to known types of attacks. Still other conventional approaches may be able to detect certain known attacks, such as SQL injection, but are unable to detect other behavior that is unexpected, unintended, or potentially a security breach. Summary of the Invention [Means for solving the problem]

[0003] In some aspects, the technology described herein relates to a method (or a computerized system that performs the method) that includes: executing, by a computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events including a sequence of multiple events that occur during the normal operation of the program; generating, by the computer system using the record of events, an acceptable behavioral model that indicates normal behavior of the program that occurs during normal operation without malicious behavior; executing, by the computer system, the program in a second operating mode in the operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second operating mode, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavioral model; and, if the comparing step results in a match between the operational sequence of events and the acceptable behavioral model, performing the current action in the second operating mode; and, if the comparing step does not result in a match between the operational sequence of events and the acceptable behavioral model, not taking the current action and stopping execution of the program or generating an alert at the current action that flags the current action in the second operating mode.

[0004] In some aspects, the techniques described herein relate to a method (or a computerized system that performs a method) that includes executing, by a computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events including a sequence of a plurality of events that occur during the normal operation of the program; and generating, by the computer system using the record of events, an allowable behavior model that indicates the normal behavior of the program that occurs during normal operation without malicious behavior, wherein the allowable behavior model is used to prevent execution of a current action of the program in a second operating mode in the operating environment if the current action of the program is determined to be part of an operational sequence of events that is inconsistent with the allowable behavior model.

[0005] In some aspects, the technology described herein relates to a method (or a computerized system that performs the method) that includes: receiving, by a computer system, an acceptable behavioral model that indicates normal behavior of a program without malicious behavior, as determined by executing the program in a first operating mode in a controlled environment according to the normal behavior without malicious behavior; executing, by the computer system, the program in a second operating mode in the operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second operating mode, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavioral model; and, when the comparing step results in a match between the operational sequence of events and the acceptable behavioral model, performing the current action in the second operating mode; and, when the comparing step results in no match between the operational sequence of events and the acceptable behavioral model, not performing the current action and stopping execution of the program or generating an alert at the current action that flags the current action in the second operating mode. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a simplified schematic block diagram of an improved program execution anomaly detection system, according to some embodiments. [Figure 2] FIG. 2 is a simplified block diagram of functional modules of a detection system, according to some embodiments. [Figure 3] 2 is a simplified flowchart of an exemplary abstraction process for generating an acceptable behavior model for use in the detection system shown in FIG. 1 , according to some embodiments. [Figure 4] 2 is a simplified flowchart of an exemplary summarization process for monitoring a supervised program for anomalous behavior by the detection system shown in FIG. 1 , according to some embodiments. [Figure 5] 2 is a simplified user input / control interface for a user to input control parameters of the detection system shown in FIG. 1, according to some embodiments. [Figure 6] 2 is a simplified flowchart of an exemplary abstraction process for instrumenting and monitoring a supervised program including interpreted language code by the detection system shown in FIG. 1 , according to some embodiments. [Figure 7] 2 is a simplified flowchart of an exemplary abstraction process for instrumenting and monitoring a supervised program using a modified compiler by the detection system shown in FIG. 1 , according to some embodiments. [Figure 8] 2 is a simplified block diagram of a computing system of an exemplary hardware implementation of the detection system shown in FIG. 1, according to some embodiments. [Figure 9] 2 is a simplified flowchart of an exemplary abstraction process for instrumenting and monitoring a supervised program using a hardware implementation of the detection system shown in FIG. 1 , according to some embodiments. [Figure 10]2 is a simplified schematic diagram illustrating an exemplary computer system of the detection system shown in FIG. 1, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] The present invention enables improved systems and / or methods to detect when a supervised program is about to engage in unexpected, anomalous, or malicious behavior, i.e., software anomalies. The improved systems and / or methods can then halt execution of the supervised program before the unexpected, anomalous, or malicious behavior can occur. Alternatively, the improved systems and / or methods may flag the behavior or current function, action, or event. Thus, the improved systems and / or methods provide "program behavior enforcement." In some embodiments, the present invention detects deviations from normal behavior rather than explicitly malicious actions. In other words, the present invention determines what the supervised program should do only under normal circumstances, rather than determining what the supervised program could do under all circumstances.

[0008] 1 is a simplified schematic block diagram of an improved program execution anomaly detection system (detection system) 100, according to some embodiments. The detection system 100 generally includes an instrumentation module 101 and a supervisor (i.e., watcher) 102 for detecting execution anomalies in a supervised program 103. The two main components (instrumentation module 101 and supervisor 102) function together to detect impending unexpected or anomalous behavior in the supervised program 103 and to halt, pause, and / or flag its execution and / or execute one or more exception routines (e.g., specified or requested by the supervised program 103) before the unexpected or anomalous behavior occurs.

[0009] In some embodiments, the instrumentation module 101 generates measurements 105 of the supervised program 103, which in turn communicate with the supervisor 102 to provide the supervisor 102 (i.e., the pattern detection unit and the security system) with information about the behavior of the supervised program 103 (i.e., reporting events or actions of the supervised program 103) during both a model building mode (e.g., a training mode or learning mode) and an operational mode (or real-world execution mode or monitoring mode), as described below. Techniques for generating measurements 105 by the instrumentation module 101 are described below.

[0010] In general, the instrumentation module 101 generates instrumentation of the supervised program 103, which in turn generates and can send to the supervisor 102 information about what the supervised program 103 is doing, i.e., events or actions being performed or about to be performed by the supervised program 103 while in either the model building mode (i.e., a first mode of operation in a controlled environment) or the operational mode (i.e., a second mode of operation in an operational environment). Each recorded event or action is provided in chronological order. Some examples of such actions or events may include, but are not limited to, the following: The supervised program 103 calls a specific function with specific parameters. The supervised program 103 returns from a particular function with a particular result; or The supervised program 103 is throwing a specific type of exception.

[0011] During model construction mode, the supervisor 102 receives, among other things, these types of events from the measurement 105 and constructs (e.g., using artificial intelligence (AI), machine learning (ML), statistical analysis, and / or a heuristic compiler) a "normal behavior" or "acceptable behavior" model 104 based on the behavioral patterns or sequences of events that occur during normal operation of the supervised program 103. Thus, the acceptable behavior model 104 represents the normal behavior of the supervised program 103 that occurs during normal operation in the absence of any attacks, malicious behaviors / actions, or anomalous events, or in the absence of external hostile influences. In other words, the acceptable behavior model 104 is a construction diagram of the types and ranges of flow control and other operating parameters that the supervised program 103 uses during operational mode.

[0012] During operational mode, supervisor 102 receives these events from measurement 105 and compares the pattern or sequence of events (i.e., operational events and operational sequences) to acceptable behavior models 104, e.g., as input to AI models, statistical analysis, and / or other suitable pattern detection techniques, or a combination thereof. This comparison enables supervisor 102 to determine whether a current or immediate action or event, or sequence of events, matches a pattern or sequence of events known or expected to occur during normal operation or behavior of supervised program 103, or known or expected to represent normal behavior. Thus, supervisor 102 performs this comparison to determine when there is a match and when there is not. In some embodiments, the match need not be exact, but must be close enough to be accepted as a match, as may be empirically determined during model building mode. Thus, with instrumentation 105, execution of the instrumented supervised program 103 generates a series of signals or messages indicative of flow control or flow status within the executing instrumented supervised program 103, and these signals are monitored by the supervisor 102 and compared to criteria in the acceptable behavior model 104 to ascertain whether the instrumented supervised program 103 is performing within or outside of expected limits.

[0013] If the comparison results in a match between the operation sequence of events and the allowable behavior model 104, this indicates that no anomaly has occurred or been detected, and the supervisor 102 typically takes no action in response (or instructs the instrumentation 105 or the supervised program 103 to continue operating), thereby allowing the supervised program 103 to perform the current action or event in the normal operation of the supervised program 103. On the other hand, if the comparison results in no match between the operation sequence of events and the allowable behavior model 104 (i.e., the operation sequence of events deviates from the allowable behavior model 104), this indicates that an anomaly (which would occur if the supervised program 103 continued execution) has been detected. In this case, the supervisor 102 responds with an action that prevents the supervised program 103 from performing the current action or event. In some embodiments, the supervisor 102 generates an alert that stops or pauses program execution. Thus, the supervisor 102 may send information back to the supervised program 103 or the instrumentation 105, for example (but not limited to), that the supervised program 103 needs to be terminated or paused during an operational mode, that a current action / event needs to be flagged, and / or that one or more predetermined exception routines need to be executed (e.g., by the supervised program 103).

[0014] Traditional uses of instrumented executable code generally involve searching for known types of vulnerable or unsafe functions / states or malicious actions that could be performed by the code, as can be determined, for example, from a control flow graph and / or data flow model. A control flow graph or data flow model may be generated, for example, to attempt to determine all possible actions or events a program could create or the states the program would enter under all circumstances. This enables analysis of potential vulnerabilities or attack vectors that a malicious actor could exploit. A program developer can then debug the program by modifying the code to ensure that the vulnerability does not appear in the final control flow graph, thereby eliminating or remediating the vulnerability. Such traditional techniques require knowledge of the characteristics of potential vulnerabilities or attack vectors, the structure of the program, and the complete functionality of the program so that potential vulnerabilities or attack vectors can be detected through analysis of the control flow graph or data flow model. On the other hand, because the present invention does not explicitly search for vulnerabilities, attack vectors, or potentially malicious actions within the code of the supervised program 103, the present disclosure advantageously does not require such knowledge. Instead, the present invention advantageously looks for normal behavior of the supervised program 103, regardless of any potential vulnerabilities in the code, and is therefore able to detect deviations from such normal behavior. In model-building mode, the detection system 100 is not involved in the full functionality of the supervised program 103 and may not encounter every possible action, event, sequence, or state that the supervised program 103 is potentially capable of. However, because actions, events, sequences, or states encountered during model-building mode will necessarily be part of normal operation, the allowable behavior model 104 generated therefrom can be used to compare against operational events and sequences during operational mode. In other words, the present invention detects deviations from normal behavior, rather than explicitly malicious actions.In other words, the present invention does not determine what the supervised program 103 will be able to do in all circumstances, but rather determines what the supervised program 103 should do only in normal circumstances.

[0015] 2 is a simplified block diagram of functional modules of the detection system 100, according to some embodiments. During model building and operational modes, functions, actions, or events performed by the supervised program 103 are provided to an instrumentation collection module 201 of the instrumentation 105. The instrumentation collection module 201 collects this information to generate instrumentation data 202. The instrumentation data 202 is provided by the instrumentation 105 to the supervisor 102.

[0016] The supervisor control panel 203 provides a "train" or "build model" signal 204 to the supervisor 102 so that the supervisor 102 operates in a model building mode to generate or train an acceptable behavior model 104. The supervisor control panel 203 also provides a "run" signal 205 to the supervisor 102 before the supervisor 102 operates in an operation mode to detect anomalies in actions or events of the supervised program 103 based on the acceptable behavior model 104. The supervisor control panel 203 also provides a "stop" signal 206 to the supervisor 102 so that the supervisor 102 terminates or pauses operation by the supervised program 103, the instrumentation module 101, and / or the supervisor 102 in either mode.

[0017] The supervisor 102 includes a thread timer 207 (e.g., a "watchdog" timer) that is set for each thread or process being tracked to ensure there is an expected response within an expected amount of time. This is used to ensure that the supervised program 103 is executing code as expected. The time durations for some or all of the timers 207 may be set by determining the length of time that the observed or monitored function, action, or event would normally take during model building mode; thus, if one of the timers 207 exceeds a previously determined duration during operational mode, then this may indicate abnormal behavior or operation or an interference with normal operation. In some cases, a response is expected from the measurement 105 within a period monitored by the timer 207; if no response is received by the supervisor 102 within the period, then an abort / termination / flag alert / message is generated, as described below. The supervisor 102 also includes a "stop" command 208 that it sends to instrumentation 105, the supervised program 103, the code interpreter, or the operating system to stop or pause the supervised program 103 (or its threads or processes) in which the supervisor 102 detects an anomaly. The supervisor 102 also includes a "logging" function 209 that it uses to provide a signal or record that an abnormal or unexpected behavior has occurred. The supervisor 102 also includes an "API" function 210 that it uses to allow external programs (e.g., via an application programming interface (API)) to perform functions with the supervised program 103 upon being notified by the signal, thereby extending the usefulness of the supervisor 102. One example of such an external program might be a debugger application that can help a programmer understand why an anomaly occurred.

[0018] In some embodiments, communication between the instrumentation module 101 and the supervisor 102 preferably uses high bandwidth with low latency communication so as not to significantly slow down the execution of the supervised program 103. Suitable protocols for providing communication may include, but are not limited to: Shared memory where the supervisor 102 and supervised program 103 are running on the same virtual machine (VM) and computing node Inter-VM shared memory where the supervisor 102 and the supervised program 103 are running on different VMs but both running on the same computing node, or A high-performance computing (HPC) interconnect (e.g., Remote Direct Memory Access (RDMA), InfiniBand (IB), RDMA over Converged Ethernet (RoCE), etc.), in which the supervisor 102 and the supervised program 103 may run in different VMs and on different computing nodes, as long as there is an HPC interconnect between them.

[0019] In some embodiments, the supervisor 102 can run together with the instrumentation module 101, the supervised program 103, and the instrumentation 105 in the same process space, but this is not ideal. Such an approach may expose the supervisor 102 to unrestricted access by malicious code that attacks the supervised program 103. In such a situation, it would likely be much easier for unacceptable actions by the supervised program 103 or the malicious code to evade the supervisor 102, thereby preventing detection beyond execution. Therefore, a preferred approach provides isolation for the supervisor 102 and the instrumentation module 101 for security reasons, i.e., between different process spaces.

[0020] Measurement Module The implementation of the instrumentation module 101 generally depends on the technology, e.g., interpreted or compiled language, used for the supervised program 103. For example, an instrumentation module 101 for a supervised program 103 written in the Java programming language will be different from an instrumentation module 101 for a program written in the JavaScript programming language, which will be different from an instrumentation module 101 for a program written in the Python programming language, which will be different from an instrumentation module 101 for a program written in one of the C or C++ programming languages, etc.

[0021] In some embodiments, for supervised programs 103 written in an interpreted or bytecode language, a modified code interpreter (including instrumentation module 101) generates instrumentation 105 while interpreting the code program, such that instrumentation 105 comprises instrumented portions of executable code generated by the modified code interpreter using instrumentation module 101. Alternatively, a code interpreter separate from instrumentation module 101 can interpret code that has already been instrumented with instrumentation 105 by instrumentation module 101, such that instrumentation module 101 includes the capability to instrument interpreted language code, and instrumentation 105 comprises instrumented portions of executable code generated by the code interpreter. In some embodiments, for supervised programs 103 written in a compiled language, previously compiled code is recompiled (or the code is first compiled) by a modified compiler (including instrumentation module 101) to automatically add instrumentation 105 to the generated executable code. In some embodiments, the instrumentation module 101 includes a hardware implementation (optionally in combination with instrumentation code in the supervised program 103) that provides instrumentation 105 to collect instrumentation data from the executing code of the supervising program 103 and provide it to the supervisor 102. These embodiments are described in further detail below.

[0022] Additionally, in some embodiments, a virtual machine (VM) or code interpreter provides instrumentation functionality to the instrumentation module 101 (e.g., an instrumentation API for Java) that could be used to generate instrumentation 105. Instrumentation for these technologies is generally well understood in the art.

[0023] In some embodiments, the instrumentation module 101 identifies functions, events, or actions and inserts additional code (i.e., instrumentation code included as instrumentation 105) at entry, exit, and exception points of the supervised program 103 to monitor execution at these points. For example, this may be done upon compiling or interpreting the code of the supervised program 103 for compiled or interpreted languages, respectively. Alternatively, a hardware implementation detects or identifies instrumentation points or locations that generate instrumentation data. In some embodiments, the instrumentation module 101 may identify all functions, or a selected subset of functions, for insertion of additional code. In some embodiments, program flow control is instrumented in the supervised program 103 along with parameters related to program control.

[0024] In some embodiments, when execution of the supervised program 103 (during either model building mode or operational mode) encounters the additional code (either through instrumentation of compiled code or instrumentation by a code interpreter), the additional code is executed. In doing so, the additional code (i.e., instrumentation 105) generates and sends a message to the supervisor 102 that execution of the supervised program 103 has reached a given or predetermined point, action, or event. Alternatively, a hardware implementation generates and sends the message. Thus, execution of the supervised program 103 is observed by instrumentation 105. Generally, the same types of observations are performed and messages are sent by instrumentation 105 during both model building mode and operational mode.

[0025] In model construction mode, in some embodiments, the supervisor 102 typically does not need to send messages back to the instrumentation 105 or the instrumentation module 101, so that the instrumentation 105 generally continues executing the supervised program 103 uninterrupted. However, in other embodiments, the instrumentation 105 receives an acknowledgment for every request or message returned from the supervisor 102. In operational mode, in some embodiments, after each message is sent, depending on the configuration of the instrumentation 105, the instrumentation 105 may wait for an acknowledgment from the supervisor 102 before continuing execution of the supervised program 103's code to the next step, function, action, or event, so that the supervisor 102 can decide whether to immediately abort execution of the supervised program 103 before being able to perform an immediate action or event. In other embodiments, the instrumentation 105 may instead wait under some circumstances, such as after a predetermined number of messages, or only for certain functions, or may not wait for an acknowledgment. In some embodiments, if the instrumentation 105 is configured not to wait for approval, for example for performance reasons, then if the supervisor 102 later determines that the supervised program 103 should not be allowed to continue running, the instrumentation 105 may receive a signal to terminate / stop / kill / pause the program.

[0026] It is anticipated that in some embodiments, for performance reasons, some types of functionality that may appear in the code may not require instrumentation, such as functionality that is purely computational and does not interact with the environment beyond the supervised program 103 itself, while functionality that interacts with user input or controls flow is preferably instrumented. However, it is contemplated that some embodiments may include instrumenting the entire code of the supervised program 103.

[0027] During model building mode, the instrumentation module 101 executes the supervised program 103 using test cases, which are typically designed by a programmer, developer, or user to test the supervised program 103 using, for example, expected normal user interactions, API interactions, file interactions, etc. with the supervised program 103. Additionally, the instrumentation module 101 executes the supervised program 103 (one or more times) as it operates normally in a closed, isolated, or controlled network environment (closed normal operation). In some embodiments, closed normal operation is necessary because a programmer, developer, or user may not anticipate all possible normal operation scenarios in their use case design, i.e., the test cases may not be "complete."

[0028] In the closed normal operation of the supervised program 103 used in the model building mode, the supervised program is tested, but the system is isolated from the external environment, including, for example, the public Internet or other programs running on the same computer as the supervised program 103. In some embodiments, the model building mode is performed in isolation from external influences in an isolated closed network environment to avoid hostile, malicious, or anomalous behavior being mistakenly recognized as acceptable or normal behavior, which is then incorporated into the acceptable behavior model 104 as acceptable or normal behavior. By monitoring in an isolated closed network environment, the model building mode ensures that only valid behavior can occur and that hostile actions or influences are avoided, or training is kept “clean.” Furthermore, during closed normal operation, training can flag behavior that did not appear during test case execution, thus providing information that can be used to generate more test cases targeting this behavior. Using both test cases and closed normal operation of the supervised program 103 generally improves the supervisor 102's ability to monitor program flow and detect anomalies. Furthermore, after the detection system 100 generates an acceptable behavior model 104 associated with a particular software package, set of packages, and / or system configuration, the acceptable behavior model 104 can be propagated to other suitable or similar detection systems 100 located elsewhere without having to move each such device, machine, or system to a closed, isolated, or controlled network environment for training.

[0029] Supervisor In some embodiments, the supervisor 102 can run in either a model construction mode or an operational mode. In the model construction mode, the supervisor 102 is trained or taught to interpret and learn from sequence of events data received from the instrumentation 105. In some embodiments, in this mode, the supervisor 102 approves all requests or messages from the instrumentation 105 and collects the event / sequence data provided by the instrumentation 105 to create an acceptable behavior model 104. The supervisor 102 typically builds an initial version of the acceptable behavior model 104 or enhances an existing version of the acceptable behavior model 104 for the supervised program 103.

[0030] In some embodiments, generating the allowable behavior model 104 generally involves running representative test cases or examples of normal operation of the supervised program 103 via measurement 105, so that the supervisor 102 can learn or establish the limits of its normal or acceptable behavior, i.e., the range of normal or acceptable behavior. (In some embodiments, the allowable behavior model 104 is generated or trained using AI techniques, such as with statistical data, to determine whether a current or immediate instruction or sequence of events is part of normal, acceptable, or expected behavior.) In other words, the supervisor 102 uses the allowable behavior model 104 to determine whether the current or immediate instruction should be allowed to continue executing or whether the supervised program 103 should be terminated. In one example, to make this determination, the allowable behavior model 104 takes into account information such as the sequence of events (i.e., a window of events that includes the current or immediate event and a sufficient number of preceding events), the function being called, and the input data for this function, among other possible parameters. Thus, in model construction mode, instrumentation 105 sends messages to supervisor 102 about successful network package sends / receives, successful file reads / writes, successful database accesses, successful function calls, and other appropriate successful actions or events that can be performed by supervised program 103. Supervisor 102 then incorporates the data from these messages (as the sequence of events at each instrumentation point or location in the execution of the code) into constructing an acceptable behavior model 104.

[0031] During model construction mode, the supervisor 102 receives function, action, or event messages and constructs an acceptable behavior model 104 based on behavioral patterns or event sequences (i.e., construction mode events and construction mode sequences) for control and data flows that occur during normal operation of the supervised program 103, i.e., without any attacks, malicious behavior / actions, or anomalous events. These patterns or sequences thus become “known,” “reference,” “normal,” or “correct” patterns or sequences that are known to occur during normal operation or behavior of the supervised program 103 or are known to be representative of normal behavior. Thus, the acceptable behavior model 104 represents the normal behavior of the supervised program 103 that occurs during normal operation without any attacks, malicious behavior / actions, or anomalous events. In some embodiments, the supervisor 102 uses AI to train or generate the acceptable behavior model 104 (i.e., an ML model or an AI model) based on the model construction mode events and model construction mode sequences. Thus, the model building mode involves inputting multiple patterns or sequences of events into the AI ​​system to train or generate an acceptable behavior model 104. In some embodiments, the acceptable behavior model 104 involves analyzing data developed or generated by the supervisor 102 for statistical analysis of events or sequences of events that the supervised program 103 may perform during normal operation. In some embodiments, the AI ​​may include statistical analysis, artificial intelligence, or a combination thereof.

[0032] After the acceptable behavior model 104 is developed, verified, and tested in the model-building mode (which may be an iterative process and may use techniques known in the art), the detection system 100 is deployed to enforce program behavior in operational mode within any number of computerized systems, whenever necessary. Thus, many copies of the supervisor 102, each using an acceptable behavior model 104, may operate in operational mode to determine whether the operation of the supervised program 103 is behaving normally or abnormally. During operational mode, the supervisor 102 receives action or event messages from the instrumentation 105 and sends back approval to continue operation and / or an alert (e.g., an out-of-bounds alert) to terminate, stop, or pause execution of the supervised program 103. In some embodiments, termination occurs before the immediate or current event or function (indicated in the most recent message) is executed, immediately after the immediate or current event or function is executed (i.e., before the immediately following event or action), or relatively soon after the immediate or current action or function is executed. Additionally, in some embodiments, in the event of a termination signal, supervisor 102 also preferably performs operating system-level program termination, for example, to ensure that potentially compromised supervised programs are not allowed to continue executing.

[0033] In some embodiments, the supervisor 102 determines whether the supervised program 103 should continue to operate or execute by considering the current message from the instrumentation 105 (i.e., the current execution point of the supervised program 103) and the execution history or context (i.e., the sequence of events the supervised program 103 executed prior to the current execution point). In this way, the supervisor 102 does not analyze or observe each event in isolation from other events, but rather in the context among other events and other data. In other words, the supervisor 102 analyzes the current event and the sequence of events to which it belongs (i.e., the number of consecutive events or “window”), either by AI models or statistical analysis. Thus, the analysis by the supervisor 102 determines one or more events that occur in a given situation or state of the supervised program 103 or that occur following a particular event or history, context, or sequence of events.

[0034] The number of events that the analysis uses to form each window or sequence of events (including the current or immediate event and preceding events) may be configurable and may depend on the type of current or immediate event, the needs of the acceptable behavior model 104, the performance capabilities of the supervisor 102, and any other appropriate considerations. Furthermore, for a relatively small number or short sequences of windows or events (e.g., an empirically determined number), the sequences of events may generally be used as is to generate the acceptable behavior model 104. On the other hand, for a relatively large number or long sequences of windows or events (e.g., an empirically determined number), the sequences of events may be compressed using a compression algorithm to generate the acceptable behavior model 104, so that comparison of operational events or sequences with the acceptable behavior model 104 during operational mode can be performed more quickly or efficiently. Different embodiments may or may not use such compression. Furthermore, if a sequence of events is found to be repeated by the supervised program 103, this sequence of events may be stored only once for use in generating the allowable behavior model 104, thereby reducing the size of the allowable behavior model 104.

[0035] In some embodiments that use machine learning (ML) or artificial intelligence (AI) to determine whether the supervisor 102 should continue or terminate the operation of the supervised program 103, the AI ​​allows the supervisor 102 to model complex behavior (e.g., the expected or normal behavior of the supervised program 103) to recognize patterns in sequences of events. The AI ​​model thus allows the supervisor 102 to recognize the difference between expected or normal behavior (learned during model-building mode) and unexpected or abnormal behavior (encountered during operational mode). On the other hand, in some embodiments that use statistical analysis, statistical analysis is used instead of or as an adjunct to AI implementations to model expected or normal behavior. However, for relatively complex software, AI implementations are typically a more practical approach.

[0036] In operational mode, for each message received from instrumentation 105, supervisor 102 uses the impending behavior (i.e., the instruction or event about to be executed by supervised program 103) and the history, input data, and / or other parameters described above to determine whether the impending behavior falls within a behavior or range of behaviors encapsulated by allowable behavior model 104. Supervisor 102 does this by comparing the sequence of events (including the current or immediate event and a sufficient number of preceding events), input data, etc., with the sequence of normal events in allowable behavior model 104. The supervisor then sends either an approval message or signal for operation to continue with the immediate event, or an alert message or signal to terminate, stop, or pause execution of supervised program 103 and / or execute one or more program-specified or program-requested exception routines. In this manner, in some embodiments, supervisor 102 in operational mode monitors and approves received messages or a subset of messages. In some embodiments, supervisor 102 monitors information in received messages without sending an acknowledgment (i.e., instrumentation 105 assumes acknowledgment), but sends a termination signal / alert if it detects that the immediate event may be anomalous or unexpected behavior. As an example, supervisor 102 generates and sends a termination signal / alert when it detects that an unexpected flow control operation has been invoked, such as may be the case for malicious or rogue code (e.g., the result of malicious intent) or an unexpected error in application code (a programming error that would enable a security breach). In this way, detection system 100 can prevent anomalous or unexpected behavior or prevent execution of a program's current action, thereby providing for execution of the program's behavior.

[0037] Alternatively, in some embodiments, supervisor 102 may represent a learning supervisor and a supervising supervisor that are physically and / or logically separate from one another, where the learning supervisor runs in a model building mode and the supervising supervisor runs in an operational mode, instead of supervisor 102 being able to run in both modes.

[0038] Model Building Mode Overview The operation of the instrumentation module 101 and the supervisor 102 during model construction mode has been described above. FIG. 3 is a simplified flowchart of an exemplary abstraction process 300 for generating an acceptable behavior model 104 in model construction mode, according to some embodiments. The specific steps, combinations of steps, and order of steps in this process are provided for illustrative purposes only. Other processes involving different steps, combinations of steps, or order of steps may also be used to achieve the same or similar results. Features or functionality described with respect to one of the steps performed by one of the components may be enabled in a different step or component in some embodiments. Furthermore, some steps may be performed before, after, or overlap with other steps, despite the illustrated order of steps. Additionally, some of the functions of the process (or alternatives to these functions) are described elsewhere herein.

[0039] At 301, the instrumentation module 101 generates instrumentation 105 for the supervised program 103. For example, in an interpreted language embodiment, the instrumentation module 101 may include a modified code interpreter that generates executable code with instrumented additional code that may interrupt normal interpretation of the supervised program 103 at appropriate taps or measurement points or locations detected by the modified code interpreter, or the instrumentation model 101 generates instrumented code for the supervised program 103 where a traditional code interpreter would generate such executable code. In an example compiled language embodiment, the instrumentation module 101 compiles or recompiles the supervised program 103 and inserts the instrumented additional code into the existing code of the supervised program 103. In an example hardware implementation, the instrumentation module 101 includes an auxiliary processor subsystem that monitors a main processor executing the supervised program 103, which may have instrumented code.

[0040] At 302, the supervised program 103 executes normally as the program executes, but with the augmentation of instrumentation 105, which generates and communicates instrumentation data to the supervisor 102. During such execution, the instrumentation 105 observes or monitors the execution of the supervised program 103 in a closed, isolated, or controlled network environment (according to normal operation without malicious behavior or in the absence of external hostile influence) and generates instrumentation data (e.g., data indicative of events). The instrumentation 105 then sends instrumentation messages along with the instrumentation data to the supervisor 102. In this manner, the instrumentation 105 generates a record of events containing the sequence of events occurring during the normal operation of the supervised program 103. For example, in an exemplary interpreted language embodiment, additional code in the executable code generated by a modified or traditional code interpreter generates instrumentation data during normal execution of the supervised program 103 and sends messages of actions or events by the supervised program 103 to the supervisor 102. In an example compiled language embodiment, execution of the compiled code of the supervised program 103 causes the execution of instrumented code, which causes messages of actions or events by the supervised program 103 to be sent to the supervisor 102. In an example hardware implementation, the auxiliary processor subsystem detects appropriate instrumentation points during execution of the supervised program 103 by the main processor, or responds to execution of instrumented code during execution of the supervised program 103 by the main processor, collects instrumentation data for the actions or events by executing the code, and provides the instrumentation data in messages to the supervisor 102.

[0041] At 303, the supervisor 102 receives and collects information / data from the messages in the order of occurrence of a sequence of actions or events, thereby forming such a sequence. The supervisor 102 may also compress the data, consolidate repeated sequences, and / or otherwise prepare the data for input for generation of an acceptable behavior model 104.

[0042] At 304, the supervisor 102 generates an acceptable behavior model 104 based on the collected information of sequences of actions or events (i.e., records of events). This may be, for example, for training an AI model, generating a statistical analysis model, or another suitable pattern detection technique. At this point, the acceptable behavior model 104 has been created and is ready for use in operational mode. Thus, the supervisor 102 then uses the acceptable behavior model 104 to monitor functions, actions, events, or sequences of events received in the instrumentation data to determine whether to allow the supervised program 103 to continue operating, to pause, terminate, or stop execution of the supervised program 103, and / or to execute one or more program-specified or program-requested exception routines.

[0043] Operational Mode Overview The operation of the instrumentation module 101 and the supervisor 102 during operational mode has been described above. FIG. 4 is a simplified flowchart of an exemplary summary process 400 for monitoring a supervised program 103 for anomalous behavior against an acceptable behavior model 104 in operational mode, according to some embodiments. The specific steps, combinations of steps, and order of steps in this process are provided for illustrative purposes only. Other processes involving different steps, combinations of steps, or order of steps may also be used to achieve the same or similar results. Features or functionality described with respect to one of the steps performed by one of the components may be enabled in a different step or component in some embodiments. Furthermore, some steps may execute before, after, or overlap with other steps, despite the illustrated order of the steps. Additionally, some of the functions of the process (or alternatives to these functions) are described elsewhere herein.

[0044] At 401, the supervised program 103 executes in the normal manner in which a program is executed. During such execution, instrumentation 105 observes or monitors the supervised program 103 in a non-isolated, real-world operational network environment (i.e., with or without unknown malicious behavior) to generate instrumentation data (e.g., data indicative of events). The instrumentation 105 then transmits instrumentation messages along with the instrumentation data to the supervisor 102. In other words, the instrumentation 105 detects or determines one or more actions or events (part of an operational sequence of events) of the supervised program 103 during execution of the supervised program 103 in the operational mode. In this manner, the instrumentation 105 generates event records for sequences of events occurring during the real-world operation of the supervised program 103, and the instrumentation 105 transmits the event records in action or event messages by the supervised program 103 to the supervisor 102 during the operational mode in the same or similar manner as performed at 302 during the model building mode.

[0045] At 402, the supervisor 102 receives and collects information / data from the messages in the chronological order of a sequence of actions or events, thereby forming such a sequence. The supervisor 102 may also compress the data, consolidate repeated sequences, and / or otherwise prepare the data, as needed, for input into the allowable behavior model 104.

[0046] At 403, the supervisor 102 compares the pattern or sequence of events (i.e., operational event or operational sequence of events) to reference patterns or sequences of events in the acceptable behavior model 104. This may be done, for example, as input to an AI model, statistical analysis, or other suitable pattern detection technique.

[0047] At 404, based on the comparison at 403, the supervisor 102 determines whether there is a match between the current or immediate action / event or sequence of events (i.e., operational event or operational sequence of events) and a reference pattern or sequence of events (i.e., in the acceptable behavior model 104) that is expected or known to occur during normal operation or behavior of the supervised program 103. This may be done, for example, as output from an AI model, statistical analysis, or other suitable pattern detection technique.

[0048] At 405, when the determination at 404 is affirmative / yes (i.e., there is a match), the supervisor 102 either does nothing or sends an acknowledgement signal / message (shown in dashed lines as being optional) to the instrumentation 105 or the supervised program 103 to continue operation. Thus, the operation or execution of the supervised program 103 continues (at 406), including performing any current or immediate actions / events.

[0049] At 407, when the determination at 404 is negative / no (i.e., there is no match), the supervisor 102 sends an abort / terminate / flag alert / message to the instrumentation 105, the supervised program 103, or the operating system to pause, terminate, or stop the execution of the supervised program 103 or its current thread, flag the current action, and / or execute an exception routine specified or requested by one or more programs. Thus, the operation or execution of the supervised program 103 is terminated, aborted, or paused, or the current action is flagged (at 408), and the supervised program 103 does not execute a current or immediate action / event in response to the abort / terminate / flag alert / message. In some embodiments, the supervised program 103 executes an exception routine specified or requested by one or more programs. For example, to terminate the supervised program 103 or its thread, the supervisor 102 terminates the supervised program 103 or thread directly, executes an exit procedure, or invokes an exit handler.

[0050] Thus, in some embodiments, in an operational mode, the instrumentation 105 continuously sends messages containing events to the supervisor 102, and the supervisor 102 continuously evaluates or analyzes the event or sequence of events and the watchdog thread timer 207 to determine whether to allow the supervised program 103 to continue executing normally or to terminate the supervised program 103.

[0051] Sample Algorithm The information or data in the message sent by instrumentation 105 to supervisor 102 precisely identifies the location of the current or immediate event, function, or line of code being executed or about to be executed by supervised program 103. Additionally, in some embodiments, the message includes context information (e.g., thread ID, end user ID, etc.) that allows supervisor 102 to distinguish between different logical execution flows. Such distinctions can be important because modern systems are typically not sequential, i.e., they process multiple tasks or threads simultaneously.

[0052] Within a given context (i.e., within a given thread of execution), the sequence of messages from instrumentation 105 describes the execution of supervised program 103, which is used during model construction mode to generate an acceptable behavior model 104. The algorithm for generating an acceptable behavior model 104 from the sequence of messages during model construction mode generally relies on an algorithm for determining whether any sequence of events matches an acceptable behavior model 104 during operational mode.

[0053] The algorithm options generally include at least two types of algorithms: ML / AI-based algorithms similar to the class of pattern recognition approaches, including large-scale language models; and · Statistical analysis algorithms using prefix lookup structures. Both types of algorithms have different structures and characteristics, but both are used to make the same decision: whether the current prefix belongs to an acceptable behavior model 104. In this context, "prefix" means the sequence of events that includes the current event, as well as the limited sequence of events that immediately precedes the current event.

[0054] The ML / AI-based approach results in smaller allowable behavior models 104 and faster monitoring actions by the supervisor 102. However, these benefits may come at the cost of more complex training and / or a higher number of false negatives (i.e., not detecting an anomaly when one occurs) and / or false positives (i.e., detecting an anomaly when one does not occur).

[0055] Lookup structures generated from statistical analysis algorithms may be easier to build and may produce fewer false positives and false negatives, although this may come at the cost of larger models and a greater impact on performance during monitoring during operational mode.

[0056] The above-described operation of detection system 100 differs from conventional approaches that monitor for unusual "big picture" anomalies, such as anomalous network traffic, CPU usage, memory access, file usage, data usage, etc. Such conventional approaches necessarily detect security breaches after they have already occurred, rather than providing a means to halt execution before a breach occurs. Furthermore, whereas such conventional approaches typically assume knowledge of potential attack vectors, malware functions, or vulnerabilities, detection system 100 has the advantage of not needing to assume such knowledge; detection system 100 generates knowledge of normal and acceptable behavior without considering potential attack vectors, malware functions, or vulnerabilities in the code.

[0057] Supervisor Control FIG. 5 illustrates a simplified user input / control interface (UI) 500 for a user to input control parameters to the supervisor 102 for operating in the model building mode and operational mode, according to some embodiments. The specific control parameters, combinations of control parameters, and order of control parameters in this data structure are provided for illustrative purposes only. Other embodiments may use different control parameters, combinations of control parameters, or orders of control parameters to achieve the same or similar results. The exemplary UI 500 generally illustrates an example of a possible implementation of an API or user interface for enabling a client, programmer, developer, or user (with appropriate training) to execute the detection system 100 and provide appropriate code signatures for assurance purposes. Other examples may include other types of controls or control parameters.

[0058] The Company ID parameter 501 and User ID parameter 502 identify the developer (e.g., person or entity) logging into and using the detection system 100. Each registered developer creates a master company ID for the Company ID parameter 501. This is an internal item only and does not appear in the application signature. Additionally, the User ID parameter 502 is unique to each registered developer.

[0059] The supervisor 102 (i.e., the "program anomaly detector") comprises a version parameter 503 and a program name parameter 504. The version parameter 503 identifies the version of the supervised program 103 as set by the developer. The program name parameter 504 identifies the name of the supervised program 103, also set by the developer, and is unique to each developer.

[0060] The policy name parameter 505 (currently "Untitled") is a name (internal to the developer) that allows the developer to identify or remember a particular set of rules or parameters specified in the user input / control interface 500. Thus, the developer can have different sets of rules, each identified by a different policy name parameter 505, with which to experiment with different parameters for the model building and operating modes of the supervised program 103.

[0061] A "No Domain Validation" checkbox 506 is provided for registered developers to select whether the program is to be validated and signed. If checkbox 506 is selected, then the supervised program 103 is validated and signed using a domain that is internal to the detection system 100 and controlled by the operator / owner of the detection system 100 to protect domain identity.

[0062] If checkbox 506 is not selected, the supervised program 103 is verified and signed in this manner. Instead, the supervised program 103 is verified with DNS signature verification 507 using a DNS record and a special server. The special server is accessible via a URL formed with version parameter 503 (automatically filled in as provided above), program name parameter 504 (automatically filled in as provided above), and verified base domain 508, which can only be set by the owner or administrator of the base domain (typically the registered developer). In this way, ownership or provision of the supervised program 103 is proven.

[0063] The training parameters section 509 is provided for setting parameters used during model building mode to generate an acceptable behavior model 104. For example, a first resolution parameter 510 sets the amount of data that the instrumentation module 101 creates for instrumentation 105 to generate and transmit, such as system calls, library calls, procedure calls, and / or flow control, among other types of data. In some embodiments, this is set by a first slide bar 511 (shown as set to library calls).

[0064] For example, if system call data is selected (e.g., when the first slide bar 511 is set over “System Calls”), only system call data is generated and sent by the instrumentation 105 and used by the supervisor 102 in model building mode. If library call data is selected (e.g., when the first slide bar 511 is set over “Library Calls” as shown), system call and library call data is generated and sent by the instrumentation 105 and used by the supervisor 102 in model building mode. If procedure call data is selected (e.g., when the first slide bar 511 is set over “Procedure Calls” as shown), system call, library call, and procedure call data is generated and sent by the instrumentation 105 and used by the supervisor 102 in model building mode. If flow control data is selected (e.g., when the first slide bar 511 is set over "Flow Control" as shown), system call, library call, procedure call, and flow control data are generated and transmitted by the instrumentation 105 and used by the supervisor 102 in model building mode. Alternatively, in some embodiments, check boxes can be used (instead of the first slide bar 511) to individually select each of the first resolution parameters 510. On the other hand, in some embodiments, the supervisor 102 can be instructed (e.g., with another check box) that a test case is being used, in which case the supervisor 102 uses all data that can be generated.

[0065] Additionally, if the supervised program 103 spawns threads or additional processes, as is common in multi-threaded applications, the first spawn ancestor depth parameter 512 sets the number to monitor or the depth to monitor so that the instrumentation module 101 can create appropriate instrumentation 105 for those threads or processes. In some embodiments, this parameter is set to monitor "all" spawned threads or processes (as shown), for example, by default, or at any other desired or appropriate depth. Additionally, the first timing variation tolerance parameter 513 may be used to set the resolution for recording timing.

[0066] The deployment parameters section 514 is provided for setting parameters used when the detection system 100 is deployed during operational mode, such as when the instrumentation 105 generates operational events and sequences and / or when the supervisor 102 uses the acceptable behavior model 104 to determine whether any sequence of events matches the acceptable behavior model 104 and whether to allow the supervised program 103 to continue executing normally or to terminate the supervised program 103. For example, the second resolution parameter 515 sets the amount of data generated and collected by the instrumentation 105 and used by the supervisor 102, such as system calls, library calls, procedure calls, and / or flow control, among other types of data. In some embodiments, this is set by a second slide bar 516 (shown as set to library calls). Alternatively, in some embodiments, check boxes can be used (instead of the slide bar 516) to individually select each of the resolution parameters 515.

[0067] In some embodiments, the second resolution parameter 515 is set by the second slide bar 516 in a manner similar to the first resolution parameter 510 and first slide bar 511 in the training parameters section 509. Thus, if the second resolution parameter 515 selected for the operational mode is the same as the first resolution parameter 510 selected for the model building mode (i.e., the second slide bar 516 is set to the same resolution parameter as the first slide bar 511), then the measurement 105 generates and transmits, and the supervisor 102 collects and uses, the same types of data in the operational mode that were generated, transmitted, collected, and used in the model building mode. However, if the second resolution parameter 515 selected for the operational mode is less than the first resolution parameter 510 selected for the model building mode (i.e., if the second slider bar 516 is set to a resolution parameter that is less than the resolution parameter of the first slider bar 511), then in the operational mode, the measurement 105 generates and transmits, and the supervisor 102 collects and uses, only the selected type of data (i.e., indicated by the second slider bar 516), rather than any additional type of data generated, transmitted, collected, and used in the model building mode. In other words, in this case, the operational mode uses a subset of the data that was used in the model building mode. On the other hand, if the second resolution parameter 515 selected for the operational mode is more than the first resolution parameter 510 selected for the model building mode (i.e., if the second slider bar 516 is set to a resolution parameter that is more than the resolution parameter of the first slider bar 511), then this appears to indicate that additional data that was not used in the model building mode will be used in the operational mode. However, in this case, measurement 105 generates and transmits, and supervisor 102 collects and uses, in operational mode only the same types of data that were generated, transmitted, collected, and used in model building mode. In other words, an acceptable behavioral model 104 will not function with any additional data that was not used to generate it.Therefore, regardless of the selection of the second resolution parameter 515 (i.e., the setting of the second slider bar 516), the detection system 100 generates, transmits, collects, and uses the selected type of data in the operational mode only if it was generated, transmitted, collected, and used in the model building mode.

[0068] Additionally, if the supervised program 103 spawns threads or additional processes, a second spawn ancestor depth parameter 517 sets the number to monitor or the depth to monitor. In some embodiments, this parameter is set to monitor “all” (as shown) spawned threads or processes, for example, by default, or any other desired or appropriate depth. Additionally, a second timing variation tolerance parameter 518 may be used to set the resolution for recording timing, which indicates how much slack is allowed in timing during runtime before the thread timer 207 indicates that an expected response did not occur within the expected amount of time.

[0069] Modified code interpreter For supervised programs 103 written in interpreted or bytecode languages ​​(e.g., but not limited to, PHP (a recursive acronym for PHP: Hypertext Prepresenter), Ruby, Python, and JavaScript), compiling the code is not necessary. Instead, in some embodiments, a modified code interpreter (including instrumentation module 101) generates instrumentation 105 while it interprets the code to generate instrumented executable code for execution. In other embodiments, instrumentation module 101 instruments interpreted language code, and the code interpreter generates instrumented executable code for execution. In comparison, traditional code interpreters can be used to execute, emulate, or debug interpreted code. However, traditional debugging typically requires prior knowledge of the type of vulnerability or potential attack. The present invention, on the other hand, has the advantage of not requiring such knowledge.

[0070] FIG. 6 is a simplified flowchart of an exemplary abstraction process 600 for instrumenting and monitoring a supervised program 103 containing interpreted language code, according to some embodiments. The specific steps, combinations of steps, and order of steps in this process are provided for illustrative purposes only. Other processes involving different steps, combinations of steps, or order of steps may also be used to achieve the same or similar results. Features or functionality described with respect to one of the steps performed by one of the components may be enabled in a different step or component in some embodiments. Furthermore, some steps may be performed before, after, or overlap with other steps, despite the illustrated order of steps. Additionally, some of the functions of the process (or alternatives to these functions) are described elsewhere herein.

[0071] The process 600 generally begins (at 601) with the human-readable source code of the supervised program 103, which is the traditional way of writing instructions for program execution. At 602, the instrumentation module 101 inserts instrumentation 105 before or during interpretation so that the instrumentation 105 can monitor flow and state information. Thus, in some embodiments, the modified code interpreter inserts instrumentation 105 during the interpretation of lines of code in the source code. In this way, the modified code interpreter also monitors flow and state data. The modified code interpreter is thus a separate program that not only interprets the code written for the supervised program 103 (as typically done by traditional code interpreters), but also monitors the code's instructions, actions, or events for appropriate instrumentation points. The modified code interpreter thus installs necessary tap points on the fly during interpretation and execution to generate the necessary instrumentation data to observe the execution of the supervised program 103. Thus, for each line of code, the modified code interpreter interprets the code and determines whether instrumentation data needs to be generated (and sent to the supervisor 102) for the current action or event before executing it. Alternatively, at 602, the instrumentation module 101 instruments the interpreted language code, resulting in the code interpreter generating executable code that includes portions of executable instrumented code (i.e., instrumentation 105) and portions of executable normal code of the supervised program 103. Additionally, the duration of the watchdog thread timer 207 may also be determined at this time. The supervisor 102 then detects (at 603) normal and abnormal events.

[0072] Fixed compiler In some embodiments, previously compiled code of supervised program 103 in a compiled language is recompiled (or the code is compiled for the first time) with a modified compiler that incorporates instrumentation module 101. The modified compiler uses instrumentation module 101 to automatically add instrumentation instructions to the source code before producing the compiled executable code. (Furthermore, in some embodiments of some compiled languages, operating system facilities may be used by instrumentation 105 (e.g., ptrace available in the Linux operating system), or calls to shared libraries may be intercepted by instrumentation 105.)

[0073] FIG. 7 is a simplified flowchart of an exemplary abstract process 700 for instrumenting and monitoring a supervised program 103 using a modified compiler, according to some embodiments. The specific steps, combinations of steps, and order of steps in this process are provided for illustrative purposes only. Other processes involving different steps, combinations of steps, or order of steps may also be used to achieve the same or similar results. Features or functionality described with respect to one of the steps performed by one of the components may be enabled in different steps or components in some embodiments. Furthermore, some steps may be performed before, after, or overlap with other steps, despite the illustrated order of steps. Additionally, some of the functions of the process (or alternatives to these functions) are described elsewhere herein.

[0074] The process 700 generally begins (at 701) with the human-readable source code of the supervised program 103. At 702, a modified compiler inserts instrumentation recording functions using the instrumentation module 101 and then compiles the source code to generate executable code that includes instrumentation 105. Thus, in addition to creating a machine code interpretation of the human-readable source code, the modified compiler adds flow and state recording functions to generate instrumented executable code 703. The instrumented executable code 703 is executed by a hardware processor as a developer or programmer would expect, sending state and flow information to the supervisor 102. At 704, the instrumentation 105 in the instrumented executable code 703 monitors the flow and state of the execution of the instrumented executable code 703. At 705, the supervisor 102 detects normal and abnormal events and flags flow and / or state anomalies.

[0075] Hardware Implementation Hardware implementations generally have the potential to be faster and more efficient than software-only implementations. In some hardware implementation embodiments, the instrumentation module 101 instruments the supervised program 103 similar to, but less extensively than, a modified code interpreter or a modified compiler embodiment, but the operation of the instrumentation 105 and supervisor 102 is accelerated by a separate auxiliary processor subsystem linked to the main processor system executing the supervised program 103. In some hardware implementation embodiments, the auxiliary processor subsystem receives signals on all or some of the main processor's I / O pins and uses these signals to determine main processor actions as the main processor executes the supervised program 103, and then determines the instrumentation data to be generated based on these actions. In model construction mode and operational mode, the auxiliary processor subsystem collects instrumentation data from the (optionally instrumented) executable code of the supervised program 103 and provides the instrumentation data to the supervisor 102, which may be implemented in software.

[0076] In some embodiments, the auxiliary processor subsystem is linked as closely as possible to the main processor system. For example, the auxiliary processor subsystem may be incorporated into a custom integrated circuit (IC) chip (e.g., via a design portability scheme such as HDL), a separate computing device, or a plug-in card with appropriate bus access. Furthermore, in some embodiments, the auxiliary processor subsystem uses RAM / ROM and / or EEPROM or the like that are separate from that of the main processor system.

[0077] In some embodiments, at a high level, an exemplary hardware implementation may include an auxiliary processor subsystem that is either co-located on the same IC chip or the same printed circuit board (PCB) as the main processor system, or located off-board in the form of a daughterboard attached (directly or indirectly) to the main processor system's PCB or to a computing device separate from that containing the main processor system. Figure 8 is a simplified block diagram of an exemplary hardware implementation computing system 800 having a main processor system 801 and an auxiliary processor subsystem 802, according to some embodiments.

[0078] Main processor system 801 generally includes, among other suitable components not shown for simplicity, a main processor (CPU) 803, a main memory 804, a communication bus 805, and an input / output (I / O) subsystem 806. Co-processor subsystem 802 generally includes, among other suitable components not shown for simplicity, a co-processor 807, dedicated memory 808 (e.g., volatile and / or non-volatile), a dedicated communication bus 809, a first dedicated I / O bus 810 for communication with I / O subsystem 806, a second dedicated I / O bus 811 for direct communication with main processor 803, and an AI engine 812.

[0079] In some embodiments, the auxiliary processor subsystem 802 executes the instrumentation module 101 and the supervisor 102. Thus, the hardware implementation of the detection system 100 does not rely on the main processor system 801 for operational instructions, except for optional instrumentation of the supervised program 103. In some hardware embodiments involving instrumentation of the supervised program 103, the auxiliary processor subsystem 802 monitors events flagged by instrumentation 105. (Thus, the optional instrumentation in the supervised program 103, combined with the associated operation of the auxiliary processor 807, form the instrumentation 105 of FIG. 1 , part of an embodiment of the hardware implementation for generating instrumentation data for use by the supervisor 102.) The optional instrumentation serves to reduce the load on the auxiliary processor 807 and accelerate the speed at which decisions can be made in operational modes. Thus, the auxiliary processor 807 detects changes in program flow control indicated by instrumentation running on the main processor system 801 (i.e., instrumentation data), and the supervisor 102 uses those inputs to build an acceptable behavior model 104 in model building mode and monitor the supervised program 103 in operational mode. In some embodiments without added optional instrumentation, the auxiliary processor subsystem 802 may monitor every instruction executed by the main processor system 801, for example, by snooping signals on its I / O pins. Furthermore, in some embodiments, the second dedicated I / O bus 811 is a dedicated I / O channel that allows the supervisor 102 to access the internal state, pipeline, and internal registers of the main processor 803 to monitor control signals and observe some actions of the supervised program 103, for example, to detect when a violation is occurring and determine whether it is normal or abnormal. Furthermore, the AI ​​engine 812 may be a hardware component (e.g., a co-processor) separate from the auxiliary processor 807 or the software modules executing on the auxiliary processor 807.The AI ​​engine is used by the supervisor 102 to build the admissible behavior model 104 in a model building mode and to monitor the supervised program 103 in an operational mode. Additionally, a watchdog thread timer 207 is implemented within the auxiliary processor 807.

[0080] Both the main processor system 801 and the auxiliary processor subsystem 802 are connected to a main memory 804 via a communications bus 805. The main memory 804 contains the supervised program 103 and provides the code via the communications bus 805 to the main processor 803 for execution of the code in the normal manner of a conventional computing system. The main memory 804 also provides the code of the supervised program 103 via the communications bus 805 to the auxiliary processor 807 for monitoring of the code.

[0081] In the model building mode and the operational mode, the coprocessor subsystem 802 monitors the instrumented execution of the supervised program 103 by the main processor system 801. Thus, the monitoring data received by the coprocessor 807 includes instructions of the compiled, instrumented code of the supervised program 103 (via communication bus 805), copies of memory accesses caused by the execution of the supervised program 103 (via communication bus 805), copies of I / O accesses from the I / O server system 806 caused by the execution of the supervised program 103 (via dedicated I / O bus 810), and control signals or instructions received over the second dedicated I / O bus 811. Using this data, the instrumentation 105 (running under the operation of the coprocessor 807) generates and transmits the measurement data to the supervisor 102 (also running on the coprocessor 807), which stores the measurement data in dedicated memory 808 (via dedicated communication bus 809).

[0082] In some embodiments, the auxiliary processor subsystem 802 enters a model construction mode in response to a command received via a dedicated I / O port. In the model construction mode, the auxiliary processor subsystem 802 monitors the instrumented execution of the supervised program 103 by the main processor system 801, which is preferably configured in an isolated, closed environment (in accordance with the operation of the instrumentation 105) to ensure that only valid behavior occurs. Then, in some embodiments, the auxiliary processor subsystem 802 (in accordance with the operation of the supervisor 102) uses the instrumentation data stored in the dedicated memory 808 in the AI ​​engine 812 to generate an acceptable behavior model 104. The acceptable behavior model 104 is stored by the supervisor 102 in the dedicated memory 808 rather than in the main memory 804.

[0083] Because the subsystem AI engine is used to process the measurement data generated by the measurement 105 and collected by the supervisor 102, this data can be processed faster and with greater confidence in lower false alarm rates (i.e., the false positives mentioned above) for a given range of test cases than would be possible with alternative training approaches. This process produces an acceptable behavior model 104 that can be refined and improved as more information is gathered during operational mode. In some embodiments, the auxiliary processor subsystem 802 is programmed to enable the uploading, via a secure interface, of operational metrics for the acceptable behavior model 104 derived from a physically separate, closed, and isolated testbed, thereby enabling rapid deployment of learned metrics to several such systems.

[0084] In operational mode, the auxiliary processor subsystem 802 similarly monitors the instrumented behavior of the main processor system 801 provided by the instrumentation 105. Thus, in operational mode, the auxiliary processor subsystem 802 (executing the supervisor 102) takes each received input of instrumentation data, particularly but not limited to flow-changing operations, actions, or events, and at each step, considers whether such behavior matches "normal behavior" using the acceptable behavior model 104, as described above. If the behavior does not match normal or expected behavior or does not match the normal or expected range of behavior per the acceptable behavior model 104, the auxiliary processor subsystem 802 uses a non-maskable interrupt (NMI) and / or its I / O connection with the main processor system 801 (e.g., via 806, 810, and / or 811) to abort the operation, flag the operation as outside the expected range, and / or execute one or more program-specified or program-requested exception modules. The non-maskable interrupt ensures that the abort cannot be overridden. Additionally, the second dedicated I / O bus 811 provides interrupt instructions to the main processor 803, allowing the supervisor 102 to take appropriate action upon receiving an NMI and aborting the supervised program 103 or its threads, so that potentially malicious code cannot control the operation of the main processor 803.

[0085] Thus, the hardware implementation is similar to the approaches described above with respect to the modified code interpreter implementation and the modified compiler implementation, with the exception that a) any load on the main processor system 801 is minimized, b) it can be expected to speed up the model building mode, and c) the detection system 100 operates with a higher degree of integrity because it does not depend on the underlying operating system or application code of the main processor system 801. Furthermore, as in all embodiments, the generation of the acceptable behavioral model 104 is performed without knowledge (or disclosure) of the code used by the supervised program 103, except for some embodiments that include specific instrumentation features to take advantage of the functionality of the auxiliary processor subsystem 802.

[0086] FIG. 9 is a simplified flowchart of an exemplary abstract process 900 for instrumenting and monitoring a supervised program 103 using a hardware implementation, according to some embodiments. The specific steps, combinations of steps, and order of steps in this process are provided for illustrative purposes only. Other processes involving different steps, combinations of steps, or order of steps may also be used to achieve the same or similar results. Features or functionality described with respect to one of the steps performed by one of the components may be enabled in different steps or components in some embodiments. Furthermore, some steps may be performed before, after, or overlap with other steps, despite the illustrated order of steps. Additionally, some of the functions of the process (or alternatives to these functions) are described elsewhere herein.

[0087] Process 900 generally begins (at 901) with the human-readable source code of the supervised program 103. In some embodiments, at 902, instrumentation recording functionality is inserted into the source code, similar to step 702 for the modified compiler implementation described above. Thus, flow and state recording functionality is added to the compiled machine code interpretation of the human-readable source code to generate (instrumented) executable code 903. On the other hand, in other embodiments, the executable code 903 does not include instrumentation, because the auxiliary processor subsystem 802 is configured to handle the functionality that generates instrumentation data. The executable code 903 (whether instrumented or not) is executed by the main processor system 802 in the manner that a developer or programmer would expect. At 904, the auxiliary processor subsystem 802 monitors the flow and state of execution of the executable code 903 (by performing instrumentation 105). At 905, the auxiliary processor subsystem 802 (by executing the supervisor 102) detects normal and abnormal events, flags flow and / or state anomalies, halts execution of the executable code 903, and / or executes one or more program-specified or program-requested exception routines.

[0088] FIG. 10 is a simplified schematic diagram illustrating an exemplary computer system 1000 (representing any combination of one or more of the computer systems) for use in detection system 100 to execute any of the software programs described herein, according to some embodiments. Other embodiments may use other components and combinations of components. For example, computer system 1000 may represent one or more physical computer devices or servers, such as a web server, a rack-mounted computer, a network storage device, a desktop computer, a laptop / notebook computer, etc., having one or more processors or one or more processor cores, depending on the complexity of detection system 100. In some embodiments implemented at least in part in a cloud network with synchronized data potentially across multiple geographic locations, computer system 1000 may be referred to as one or more cloud servers. In some embodiments, the functionality of computer system 1000 is enabled on a single computer device. In more complex implementations, some of the functionality of the computing system is distributed across multiple computer devices, whether within a single server farm facility or across multiple physical locations, and whether implemented on computer systems with the same or dissimilar architectures or types of processors. Thus, when a claim recites a "computer system" or a "computerized system," it should be understood that this may refer to any of these embodiments.

[0089] In some embodiments in which computer system 1000 represents multiple computing devices or systems, some of the functionality of computer system 1000 is implemented in some of the computing devices, while other functionality is implemented in other computing devices. For example, various portions of detection system 100 can be implemented in the same computing device or separate computing devices, regardless of whether the computing devices have the same or different architectures or types of processors.

[0090] In the illustrated embodiment, computer system 1000 generally includes at least one processor 1002, at least one main electronic memory 1004, at least one data storage device 1006, at least one user I / O 1009, and at least one network I / O 1010 connected or coupled together by a data communications subsystem 1012, among other components not shown for simplicity.

[0091] Processor 1002 represents one or more central processing units, computer processors, coprocessors, processor cores, or clusters of processors within one or more IC chips, on one or more PCBs (printed circuit boards), and / or within one or more housings or enclosures. In some embodiments, processor 1002 represents multiple microprocessor units within multiple computing devices in multiple physical locations interconnected by one or more data channels. Thus, when the claims recite a "processor," it should be understood that this may refer to any of these embodiments. Furthermore, when processor 1002 executes computer-executable instructions to perform the above-described functions of computer system 1000 (i.e., detection system 100) in cooperation with main electronic memory 1004, it becomes a dedicated computer for performing the functions of the instructions.

[0092] Main electronic memory 1004 represents one or more RAM modules on one or more PCBs in one or more housings or enclosures. In some embodiments, main electronic memory 1004 represents multiple memory module units in multiple computer devices in multiple physical locations. In operation with processor 1002, main electronic memory 1004 stores computer-executable instructions executed by processor 1002, and data processed or generated by processor 1002, to perform the above-described functions of computer system 1000 (i.e., detection system 100).

[0093] Data storage 1006 represents or includes any suitable number or combination of internal or external physical mass storage devices, such as hard drives, optical drives, network-attached storage (NAS) devices, flash drives, etc. In some embodiments, data storage 1006 represents multiple mass storage devices in multiple computer devices in multiple physical locations. Data storage 1006 generally provides persistent storage (e.g., in non-transitory computer-readable or machine-readable medium 1008) for programs (e.g., computer-executable instructions) and data used in the operation of processor 1002 and main electronic memory 1004. Non-transitory computer-readable medium 1008 includes instructions (e.g., programs and data 1020-1048) that, when executed by processor 1002, cause processor 1002 to perform operations, including the above-described functionality, of computer system 1000 (i.e., detection system 100).

[0094] In some embodiments, main electronic memory 1004 and data storage 1006 contain all or part of the programs and data (e.g., represented by 1020-1048) required by processor 1002 to perform the methods, processes, and functions disclosed herein (e.g., FIGS. 1-9). Under the control of these programs and using this data, processor 1002, in cooperation with main electronic memory 1004, performs the functions of computer system 1000 (i.e., detection system 100).

[0095] User I / O 1009 represents one or more suitable user interface devices, such as a keyboard, pointing device, display, etc. In some embodiments, User I / O 1009 represents multiple user interface devices for multiple computer devices in multiple physical locations. For example, a system administrator may use these devices to access, set up, and control computer system 1000.

[0096] Network I / O 1010 represents any suitable network device, such as a network adapter, for communicating throughout detection system 100. In some embodiments, network I / O 1010 represents multiple such networking devices for multiple computing devices in multiple physical locations to communicate over multiple data channels.

[0097] Data communications subsystem 1012 represents any suitable communications hardware for connecting other components in a single unit or in a distributed manner, on one or more PCBs, within one or more housings or enclosures, within one or more rack assemblies, within one or more geographic locations, etc.

[0098] Reference has been made in detail to embodiments of the disclosed invention, one or more examples of which are illustrated in the accompanying drawings. Each example is provided for the purpose of explaining the technology, not limiting it. Indeed, while the specification has described in detail certain embodiments of the invention, it will be understood that those skilled in the art, upon gaining an understanding of the foregoing, will readily conceive modifications, variations, and equivalents to these embodiments. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield yet a still further embodiment. Accordingly, the present subject matter is intended to encompass all such modifications and variations within the scope of the appended claims and their equivalents. These and other modifications and variations to the present invention can be practiced by those skilled in the art without departing from the scope of the invention, which is more particularly set forth in the appended claims. Moreover, those skilled in the art will understand that the foregoing description is illustrative only and is not intended to limit the invention.

Claims

1. 1. A method comprising: executing, by the computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events comprising a sequence of events occurring during the normal operation of the program; generating, by the computer system using the record of events, an acceptable behavior model that represents normal behavior of the program that occurs during normal operation without the malicious behavior; executing, by the computer system, the program in a second operating mode in an operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second mode of operation, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavior model; executing the current action in the second operating mode when the comparing step results in a match between the operational sequence of events and the allowable behavior model; When the comparing step results in no match between the operational sequence of events and the acceptable behavioral model, not taking the current action and stopping the execution of the program or generating an alert in the current action that flags the current action in the second operating mode; The method comprising:

2. each sequence of events in the plurality of sequences of events includes a plurality of events that occur during the normal operation of the program; The method of claim 1.

3. said program is written in an interpreted language, The method comprises: instrumenting code of the program and including the instrumented code; interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; The method of claim 1 further comprising:

4. said executing said program in said first operating mode and said second operating mode is performed after interpreting said program by said code interpreter; said generating a record of said event is performed by executing a second portion of said executable code. The method of claim 3.

5. said instrumenting the code of said program to include said instrumented code is performed before said program is provided to said code interpreter; The method of claim 3.

6. said instrumenting the code of the program to include the instrumented code is performed by the code interpreter while interpreting the code of the program; The method of claim 3.

7. instrumenting the code of the program by the computer system using a compiler, the program including instrumented executable code; further comprising said executing said program in said first operating mode results in said generating said records of events according to instrumentation of said instrumented executable code of said program; The method of claim 1.

8. said executing said program in said first mode of operation is performed by a first processor of said computer system; the generating of the record of the event is performed by a second processor separate from the first processor. The method of claim 1.

9. said executing said program in said second mode of operation is performed by a first processor of said computer system; the determining of the operational sequence of the events is performed by a second processor of the computer system separate from the first processor; The method of claim 1.

10. the acceptable behavior model is an artificial intelligence model that is trained on the sequences of events known to occur during the normal operation of the program without the malicious behavior. The method of claim 1.

11. 1. A method comprising: executing, by the computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events comprising a sequence of events occurring during the normal operation of the program; generating, by the computer system using the record of events, an acceptable behavior model that represents normal behavior of the program that occurs during normal operation without the malicious behavior; Including, the allowable behavior model is configured to be used to prevent execution of the current action of the program in the second operating mode in the operating environment when the current action of the program is determined to be part of an operational sequence of events that is inconsistent with the allowable behavior model. The method.

12. each sequence of events in the plurality of sequences of events includes a plurality of events that occur during the normal operation of the program; The method of claim 11.

13. said program is written in an interpreted language, the method further comprising instrumenting code of the program and including the instrumented code; the method further comprising interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; said executing said program in said first operating mode and said second operating mode is performed after interpreting said program by said code interpreter; said generating a record of said event is performed by executing a second portion of said executable code. The method of claim 11.

14. instrumenting the code of the program by the computer system using a compiler, the program including instrumented executable code; further comprising said executing said program in said first operating mode results in said generating said records of events according to instrumentation of said instrumented executable code of said program; The method of claim 11.

15. said executing said program in said first mode of operation is performed by a first processor of said computer system; the generating of the record of the event is performed by a second processor of the computer system separate from the first processor; The method of claim 11.

16. the acceptable behavior model is an artificial intelligence model that is trained on the sequences of events known to occur during the normal operation of the program without the malicious behavior. The method of claim 11.

17. 1. A method comprising: receiving, by the computer system, an acceptable behavior model indicative of normal behavior of the program free of malicious behavior as determined by executing the program in a first mode of operation in a controlled environment according to normal behavior free of malicious behavior; executing, by the computer system, the program in a second operating mode in an operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second mode of operation, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavior model; executing the current action in the second operating mode when the comparing step results in a match between the operational sequence of events and the allowable behavior model; When the comparing step results in no match between the operational sequence of events and the acceptable behavioral model, not taking the current action and stopping the execution of the program or generating an alert in the current action that flags the current action in the second operating mode; The method comprising:

18. said program is written in an interpreted language, the method further comprising instrumenting code of the program and including the instrumented code; the method further comprising interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; said executing said program in said second mode of operation is performed after interpreting said program by said code interpreter; said determining an operational sequence of said events is performed by executing a second portion of said executable code.

18. The method of claim 17.

19. the program includes instrumented executable code; said executing said program in said second operating mode results in said generating said operational sequence of events in accordance with instrumentation of said instrumented executable code of said program; 18. The method of claim 17.

20. said executing said program in said second mode of operation is performed by a first processor of said computer system; the determining of the operational sequence of the events is performed by a second processor of the computer system separate from the first processor; 18. The method of claim 17.

21. 1. A computerized system comprising: a memory for storing executable instructions; a processor, coupled to the memory, for performing a method by executing the instructions stored in the memory; wherein the method comprises: executing, by the computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events comprising a sequence of events occurring during the normal operation of the program; generating, by the computer system using the record of events, an acceptable behavior model that represents normal behavior of the program that occurs during normal operation without the malicious behavior; executing, by the computer system, the program in a second operating mode in an operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second mode of operation, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavior model; executing the current action in the second operating mode when the comparing step results in a match between the operational sequence of events and the allowable behavior model; When the comparing step results in no match between the operational sequence of events and the acceptable behavioral model, not taking the current action and stopping the execution of the program or generating an alert in the current action that flags the current action in the second operating mode; The computerized system comprising:

22. each sequence of events in the plurality of sequences of events includes a plurality of events that occur during the normal operation of the program; 22. The computerized system of claim 21.

23. said program is written in an interpreted language, The method comprises: instrumenting code of the program and including the instrumented code; interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; 22. The computerized system of claim 21, further comprising:

24. said executing said program in said first operating mode and said second operating mode is performed after interpreting said program by said code interpreter; said generating a record of said event is performed by executing a second portion of said executable code.

24. The computerized system of claim 23.

25. said instrumenting the code of said program to include said instrumented code is performed before said program is provided to said code interpreter; 24. The computerized system of claim 23.

26. said instrumenting the code of the program to include the instrumented code is performed by the code interpreter while interpreting the code of the program; 24. The computerized system of claim 23.

27. instrumenting the code of the program by the computer system using a compiler, the program including instrumented executable code; further comprising said executing said program in said first operating mode results in said generating said records of events according to instrumentation of said instrumented executable code of said program; 22. The computerized system of claim 21.

28. said executing said program in said first mode of operation is performed by a first processor of said computer system; the generating of the record of the event is performed by a second processor separate from the first processor.

22. The computerized system of claim 21.

29. said executing said program in said second mode of operation is performed by a first processor of said computer system; said determining an operational sequence of events is performed by a second processor of said computer system separate from said first processor; 22. The computerized system of claim 21.

30. the acceptable behavior model is an artificial intelligence model that is trained on the sequences of events known to occur during the normal operation of the program without the malicious behavior.

22. The computerized system of claim 21.

31. 1. A computerized system comprising: a memory for storing executable instructions; a processor, coupled to the memory, for performing a method by executing the instructions stored in the memory; wherein the method comprises: executing, by the computer system, a program in a first operating mode in a controlled environment according to normal operation without malicious behavior; generating, by the computer system, a record of events comprising a sequence of events occurring during the normal operation of the program; generating, by the computer system using the record of events, an acceptable behavior model that represents normal behavior of the program that occurs during normal operation without the malicious behavior; Including, the allowable behavior model is configured to be used to prevent execution of the current action of the program in the second operating mode in the operating environment when the current action of the program is determined to be part of an operational sequence of events that is inconsistent with the allowable behavior model. The computerized system.

32. each sequence of events in the plurality of sequences of events includes a plurality of events that occur during the normal operation of the program; 32. The computerized system of claim 31.

33. said program is written in an interpreted language, the method further comprising instrumenting code of the program and including the instrumented code; the method further comprising interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; said executing said program in said first operating mode and said second operating mode is performed after interpreting said program by said code interpreter; said generating a record of said event is performed by executing a second portion of said executable code.

32. The computerized system of claim 31.

34. instrumenting the code of the program by the computer system using a compiler, the program including instrumented executable code; further comprising said executing said program in said first operating mode results in said generating said records of events according to instrumentation of said instrumented executable code of said program; 32. The computerized system of claim 31.

35. said executing said program in said first mode of operation is performed by a first processor of said computer system; the generating of the record of the event is performed by a second processor of the computer system separate from the first processor; 32. The computerized system of claim 31.

36. the acceptable behavior model is an artificial intelligence model that is trained on the sequences of events known to occur during the normal operation of the program without the malicious behavior.

32. The computerized system of claim 31.

37. 1. A computerized system comprising: a memory for storing executable instructions; a processor, coupled to the memory, for performing a method by executing the instructions stored in the memory; wherein the method comprises: receiving, by the computer system, an acceptable behavior model indicative of normal behavior of the program free of malicious behavior as determined by executing the program in a first mode of operation in a controlled environment according to normal behavior free of malicious behavior; executing, by the computer system, the program in a second operating mode in an operating environment; determining, by the computer system, an operational sequence of events of the program during execution of the program in the second mode of operation, the operational sequence of events including a current action; comparing, by the computer system, the operational sequence of events with the acceptable behavior model; executing the current action in the second operating mode when the comparing step results in a match between the operational sequence of events and the allowable behavior model; When the comparing step results in no match between the operational sequence of events and the acceptable behavioral model, not taking the current action and stopping the execution of the program or generating an alert in the current action that flags the current action in the second operating mode; The computerized system comprising:

38. said program is written in an interpreted language, the method further comprising instrumenting code of the program and including the instrumented code; the method further comprising interpreting the program with a code interpreter to generate a first portion of executable code based on the code of the program and a second portion of executable code based on the instrumented code; said executing said program in said second mode of operation is performed after interpreting said program by said code interpreter; said determining an operational sequence of said events is performed by executing a second portion of said executable code.

38. The computerized system of claim 37.

39. the program includes instrumented executable code; said executing said program in said second operating mode results in said generating said operational sequence of events in accordance with instrumentation of said instrumented executable code of said program; 38. The computerized system of claim 37.

40. said executing said program in said second mode of operation is performed by a first processor of said computer system; the determining of the operational sequence of the events is performed by a second processor of the computer system separate from the first processor; 38. The computerized system of claim 37.

Images