Techniques for auto-remediating security issues with artificial intelligence

Abstract

To provide methods and memories for auto-remediating security issues with artificial intelligence.SOLUTION: A method includes obtaining a problem detected within a signal from an emitter associated with a user, inferring a first response using a global model having a global set of model parameters learned globally from mappings between problems and responses with respect to preferences of all users using a security architecture, inferring a second response using a local model having a local set of model parameters learned locally from mappings between problems and responses with respect to preferences of the user; evaluating the first response and the second response using criteria, determining a final response for the problem based on the evaluation of the first response and the second response, and selecting a responder from a set of responders based on the final response. The responder is adapted to take one or more actions to respond to the problem.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Patent Application No. 17 / 130,867, filed December 22, 2020, and entitled "TECHNIQUES FOR AUTO-REMEDIATING SECURITY ISSUES WITH ARTIFICIAL INTELLIGENCE," the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE INVENTION The present disclosure relates generally to cybersecurity, and more particularly to techniques for automatically remediating security issues with artificial intelligence. [Background technology]

[0003] background Security information and event management (SIEM) is a subcategory of cybersecurity that combines security information management (SIM) and security event management (SEM) through software solutions and tools. These software solutions and tools perform real-time analysis of security notifications generated by applications and network hardware. SIEM software works by collecting log and event data generated by host systems, security devices, and security applications across an organization's infrastructure and organizing the log and event data on a centralized platform. SIEM software solutions and tools use classification rules to identify various activities within this data (e.g., antivirus events and firewall logs) and sort these activities into categories such as malware activity, failed and successful logins, and other potentially malicious activity. If the software identifies activity that may indicate a threat to the organization, a notification is generated to indicate a potential security issue. These notifications can be assigned various priorities using a set of predefined rules. For example, if a user account has multiple failed login attempts over a period of time, it may be flagged as suspicious activity but assigned the lowest priority because the user has most likely forgotten their login details. However, if there is a large number of failed login attempts to an account over a short period of time, it is more likely that a brute force attack is underway and should be flagged as a very serious incident. Summary of the Invention [Problem to be solved by the invention]

[0004] SIEM software solutions and tools aggregate relevant data from multiple sources, and while this data collection is meaningful, these processes within the SIEM tend to generate more notifications than security teams can be expected to respond to while remaining effective. To assist security teams with these tasks, a collection of software solutions and tools known as SOAR (Security Orchestration, Automation, and Response) enables security teams to manage threats and vulnerabilities and respond to incidents and security events. The orchestration component is , orchestrating various software solutions and tools to seamlessly integrate and communicate with each other to create repeatable, actionable, scalable, and effective incident response processes and workflows. The automation component utilizes a rules- and policy-based system to semi-automatically handle linear-based tasks and steps, reducing or eliminating the mundane operations that must be performed. The response component addresses and manages security incidents once notifications are confirmed, including triage, containment, remediation, formalized workflows, reporting, and collaboration. Thus, leveraging the SIEM's capabilities of ingesting large amounts of data and generating notifications, a SOAR solution can be used to augment the SIEM solution to better manage the incident response process for each notification, automating and orchestrating mundane and repetitive tasks that would otherwise take security teams hours to complete.

[0005] overview Techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for automatically remediating security issues using artificial intelligence. [Means for solving the problem]

[0006] In various embodiments, a method is provided. The method includes a response system of a security architecture obtaining an issue detected in a signal from an emitter associated with a user, and inferring a first response using a global model implemented as part of the response system that takes the issue as input, the global model including a global model parameter set learned globally from a mapping of issues and responses for all user preferences using the security architecture; the method further includes inferring a second response using a local model implemented as part of the response system that takes the issue as input, the local model including a local model parameter set learned locally from a mapping of issues and responses for user preferences; the method further includes the response system evaluating the first and second responses using criteria, the criteria including (1) a confidence score associated with each of the first and second responses and (2) a weight associated with each of the global and local models; the method further includes the response system determining a final response to the issue based on the evaluation of the first and second responses; and the response system selecting a responder from among a set of responders based on the final response, the responder being adapted to take one or more actions to respond to the issue.

[0007] In some embodiments, the method further includes, prior to selecting the correspondence, evaluating a accuracy rate of the final correspondence using the correspondence system, the accuracy rate being evaluated based on a comparison of the final correspondence to a ground truth correspondence that a user would prefer for the problem, and determining the accuracy rate of the final correspondence as acceptable if the final correspondence matches the ground truth correspondence based on the comparison, and determining the accuracy rate as unacceptable if the final correspondence does not match the ground truth correspondence based on the comparison, the method further includes generating a label for the problem in response to the accuracy rate being determined to be unacceptable, the label including the ground truth correspondence, and the method further includes and, in response to determining that the accuracy rate is unacceptable, storing a label including the ground truth and the problem in a local data store, and selecting a correspondence from the set of correspondences based on the ground truth rather than the final correspondence; the method further includes, in response to determining that the accuracy rate is acceptable, generating a label for the problem, the label including the final correspondence; and the method further includes, in response to determining that the accuracy rate is acceptable, storing a label including the final correspondence and the problem in a local data store and a global data store, and selecting a correspondence from the set of correspondences based on the final correspondence.

[0008] In some embodiments, the method further includes, in response to determining that the accuracy rate is unacceptable, storing the label including the ground truth and the problem in a general data store or data queue for evaluation by an administrator; receiving from the administrator either a response of taking no action with respect to the label including the ground truth and the problem or a response of training a global model using the label including the ground truth and the problem; in response to the response of taking no action, removing the label including the ground truth and the problem from the data store or data queue; and in response to the response of training a global model, storing the label including the ground truth and the problem in a global repository.

[0009] In some embodiments, the method further includes training a global model with global training data from a global repository, the global training data including labels that include the final response and the problem, and training a local model with local training data from a local repository, the local training data including labels that include the final response and the problem.

[0010] In some embodiments, the method further includes training the local model with local training data from a local repository, the local training data including labels including ground truth correspondences and problems.

[0011] In some embodiments, the method further includes training the global model with global training data from a global repository, the global training data including labels with ground truth correspondences and problems, and training the local model with local training data from a local repository, the local training data including labels with ground truth correspondences and problems.

[0012] In some embodiments, the method further includes the response system performing one or more actions to respond to the problem. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 shows a block diagram illustrating a computing system for generating notifications of potential security issues and automatically remediating security issues with artificial intelligence, according to various embodiments. [Figure 2] FIG. 1 shows a block diagram illustrating a response system with active learning, according to various embodiments. [Figure 3] 1 shows a flowchart illustrating a process for automatically remediating security issues with artificial intelligence, according to various embodiments. [Figure 4] 1 shows a flowchart illustrating steps for automatically remediating security issues with two or more predictive models implemented with active learning, according to various embodiments. [Figure 5] FIG. 1 depicts a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to various embodiments. [Figure 6] FIG. 1 shows a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to various embodiments. [Figure 7] FIG. 1 shows a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to various embodiments. [Figure 8] FIG. 1 shows a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to various embodiments. [Figure 9] FIG. 1 shows a block diagram illustrating an exemplary computer system, in accordance with various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description In the following description, various embodiments are described. For convenience of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that these embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0015] Introduction Various embodiments provide a security architecture consisting of a SIEM solution layered on top of a SOAR solution as a standalone system to help customers maintain a strong security posture within distributed computing environments, such as within cloud computing. Customers use the security architecture to monitor their tenancy and determine whether resources in their distributed computing environment are vulnerable to security threats or are under attack. Upon detection, the security architecture takes corrective action. In many ways, SOAR solutions are a natural complement to SIEM solutions. SIEMs are effective at aggregating security data from across the network, but typically lack the ability to provide context for all of that data. As a result, SIEM users still must spend significant time manually triaging and investigating tasks, which is becoming increasingly difficult in the face of countless alerts and overwhelming data. Orchestration and automation are needed to reduce the manual effort required, and SOAR solutions can automate repeatable incident response tasks after collecting threat data, thereby removing the burden from employees. However, to be effective, SOAR solutions require rules and policies—repeatable, automated security workflows designed to describe threats and how to address them. The problem is, these rules and policies are only as effective as the data used to build them, and humans still need to direct the overall threat remediation and how to address it. Additionally, like SIEM, SOAR can suffer from data overload, a lack of context from internal systems, and a limited view of external threats.

[0016] To overcome these and other challenges, various embodiments are directed to a security architecture with a SOAR solution that can be used with artificial intelligence to automatically remediate security incidents. A security incident (referred to herein simply as a "problem") is a possible security event that a security architecture is adapted to respond to and remediate. A problem is derived from a collection of signals. The same signal repeated over time may indicate a problem, and multiple signals may indicate the same problem. "Alice's user credentials have been compromised" is an example of a problem. In various embodiments, a technique implemented by a security architecture for automatically remediating a security issue includes: a response system of the security architecture obtaining an issue detected in a signal from an emitter associated with a user; and inferring a first response using a global model implemented as part of the response system that takes the issue as input, the global model including a global model parameter set learned globally from a mapping of issues to responses for all user preferences using the security architecture; the technique further includes inferring a second response using a local model implemented as part of the response system that takes the issue as input, the local model including a local model parameter set learned locally from a mapping of issues to responses for user preferences; The technique further includes the response system evaluating the first response and the second response using criteria, the criteria including (1) a confidence score associated with each of the first response and the second response and (2) a weight associated with each of the global model and the local model, the technique further includes the response system determining a final response to the problem based on the evaluation of the first response and the second response, and the response system selecting a response unit from the set of response units based on the final response, the response unit being adapted to take one or more actions to respond to the problem.

[0017] In some cases, the techniques enabled by the security architecture for automatically remediating security issues further comprise active learning, which includes using the correspondence system to evaluate the accuracy of the final correspondence before selecting a correspondence, the accuracy being evaluated based on a comparison of the final correspondence with a ground truth correspondence that a user would prefer for the problem, and if the final correspondence matches the ground truth correspondence based on the comparison, the accuracy of the final correspondence is determined to be acceptable, and if the final correspondence does not match the ground truth correspondence based on the comparison, the accuracy being determined to be unacceptable.

[0018] In response to determining that the accuracy rate is unacceptable, generate a label for the problem, the label including the ground truth correspondence; in response to determining that the accuracy rate is unacceptable, store the label including the ground truth and the problem in a local data store; and select a correspondence from the set of correspondences based on the ground truth rather than the final correspondence.

[0019] In response to determining that the accuracy rate is unacceptable, the label including the ground truth and the problem is stored in a general data store or put into a data queue for evaluation by an administrator, and either a response of taking no action on the label including the ground truth and the problem or a response of training a global model using the label including the ground truth and the problem is received from the administrator, and in response to taking no action, the label including the ground truth and the problem is deleted from the data store or data queue, and in response to training a global model, the label including the ground truth and the problem is stored in a global repository, and then the global model is trained with global training data from the global repository, where the global training data includes the label including the ground truth and the problem.

[0020] In response to determining that the accuracy rate is acceptable, generate a label for the problem, the label including the final correspondence; further, in response to determining that the accuracy rate is acceptable, store the label including the final correspondence and the problem in a local data store and a global data store; select a correspondence from the set of correspondences based on the final correspondence; train a global model with global training data from the global repository including the label including the final correspondence and the problem; and train a local model with local training data from the local repository including the label including the final correspondence and the problem.

[0021] Active learning security architecture FIG. 1 is a block diagram illustrating a computing system 100 for generating notifications of potential security issues and automatically remediating security issues with artificial intelligence, according to various embodiments. As shown in FIG. 1, computing system 100 includes one or more emitters 105 and a security architecture 110 consisting of a SIEM solution and a SOAR solution layered thereon. Computing system 100 can be computerized such that each of the illustrated components is configured to communicate with the other components. If the components reside on the same computing device, communication can occur via internal communication systems, such as various types of buses. If the components reside on different computing devices, such as different servers, In this case, communication may occur via network 115. Network 115 may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of commercially available protocols, including, but not limited to, TCP / IP, SNA, IPX, AppleTalk, etc. By way of example only, network 115 may be a LAN (Local Area Network), such as an Ethernet network, a token ring network, a WAN (Wide Area Network), a virtual network, including, but not limited to, a VPN (Virtual Private Network), the Internet, an intranet, an extranet, a PSTN (Public Switched Telephone Network), an infrared network, a wireless network (e.g., a network operating under any of the IEEE 802.1X suite of protocols, the Bluetooth protocol, and / or other wireless protocols known in the art), and / or any combination of these and / or other networks. Any other combination of networks, including secure and non-secure network communications, is contemplated for use in the systems described herein. Although an exemplary computing system 100 is shown having three emitters 105 and one security architecture 110, in other embodiments, any number of emitters 105 and / or security architectures 110 may be supported (e.g., a separate security architecture may exist on each distributed computing environment, such as a cloud computing environment).

[0022] Emitter 105 is a generator (source) of raw data. Examples include the Operating System Management Service, Oracle Cloud Infrastructure (OCI) Audit Service, agents running on virtual machines (VMs), raw Simple Network Management Protocol (SNMP), and / or OCI Flow Logs. Security architecture 110 uses adapter 120 (data converter) to consume the raw data generated by emitter 105 and convert it into signals. These signals are transmitted to signal processor 125 via internal application programming interface (API) signal calls. Signals are raw data points of security architecture 110. Signals may or may not be classified. Classified signals are raw data points from well-known sources and have a known schema. Security architecture 110 may convert classified signals into strongly typed internal JSON objects that capture the "who, what, when, and where" information. Examples include configuration change signals (e.g., an Object Storage bucket was exposed), major activity signals (e.g., "user Alice logged in from Thailand"), network signals, and Common Vulnerabilities and Exposures (CVE) signals (CVEs are types of vulnerabilities that are commonly known and classified). An unclassified signal is a raw data point or set of raw data points from an unknown source. This source may be one that Cloud Guard has never seen before. For example, a customer's application logs may be the source of an unclassified signal. Security architecture 110 may attempt to infer "who, what, when, where?" and the relationship between normal and abnormal signal patterns. Security architecture 110 may attempt to learn what an unclassified signal is or solicit customer input to aid in the learning process.In some cases, the adapter 120 obtains the raw data through an external API by polling the emitter 105. In other cases, the emitter 105 pushes the raw data to the adapter 120. In some cases, the adapter 120 receives raw data from external APIs provided by companies such as McAfee, Qualys, etc. Used to integrate with third-party vendors such as Rapid7 to obtain raw data from them.

[0023] The signal processor 125 consumes the signal data and signal hint information from the adapter to determine whether the received signal is a classified signal 130 or an unclassified signal 135. The determination of whether the signal is a classified signal 130 or an unclassified signal 135 can be made based on the signal hint. As used herein, when an action is "based" on something, this means that the action is based at least in part on at least a portion of something. A signal hint is a suggestion about the type of signal being considered and is typically provided by the adapter 120. Signal hints are similar to SQL Query hints or HTML Meta tags such as "keywords" and "description." The signal hint tells the signal processor 125 what type of signal the adapter 120 believes it is processing. The signal processor 125 is under no obligation to use the hint. Based on the determined signal type, the signal processor 125 sends the signal to the strongly typed internal API 140 or to the unclassified signal processing stream 145.

[0024] Classified signals 130 are mapped to corresponding previously learned or programmed signal types (or detector topics), such as netflow, configuration, and activity, using a strongly typed internal API 140. Netflow signals are IP (Internet Protocol) information that describes connections between devices, such as ports, protocols, destination IP addresses, and source IP addresses. Configuration signals are configuration parameters used to set up various components on a computing system using the control plane (e.g., configuring the visibility flag for an object bucket designated as public or private). Activity signals are audit logs of activity, such as API calls, occurring on the control plane and data plane. Unclassified signals 135 are input to a signal processing stream 145 and run through a separate learning process. First, the learning process is adapted to determine basic facts about the unclassified signals 135, including input format, record delimiters, field types, and semantics. Dedicated unclassified signal detectors and topics are then built based on the fundamentals, and a learning process, such as unsupervised machine learning, is used to determine the relationship between normal and anomalous behavior within the unclassified signal 135. Anomalous behavior is activity that deviates from learned or well-understood patterns. In some cases, the learning process for the unclassified signal 135 may also require customer input in the form of labels to aid and provide supervised learning. The learning process also attempts to learn about unclassified signal targets. Targets identify infrastructure compartments or resources that are the subject of the security architecture 110 configuration. The security architecture 110 configuration captures information about which compartments within the infrastructure tenancy the security architecture 110 monitors and which detectors to apply to those compartments. Targets may also be specific SaaS instances, such as instances of Human Capital Management Cloud.The target must have an identifier, such as an OCID (Oracle Cloud ID). The learning process allows the action recommender to select the correct action for the detected problem.

[0025] Once the classified signal 130 and / or unclassified signal 135 have been typed or the detector topic determined, the signal is forwarded to a stream processor 150 for real-time analysis. The detector 155 subscribes to the various signal types and detector topics sent to the stream processor 150. Subscribing essentially creates a mapping between the detector 155 and the various signal types and detector topics. The mapping may be a multiple-to-multiple mapping, where specific detectors are interested in specific data or specific combinations of data, or a multiple-to-multiple mapping, where multiple detectors consume the same data or the same combinations of data. The detector 155 is adapted to derive the problem 160 from the signal. For example, the security architecture 110 may support a network behavior anomaly detector adapted to subscribe to NetFlow signals and detect anomalous behavior in the NetFlow signals that indicates a problem, and a configuration change detector adapted to subscribe to configuration signals and detect anomalous behavior in the configuration signals that indicates a problem. The detector 155 detects the anomalous behavior and derives the problem 160 from the signal using heuristics, machine learning, or a combination of heuristics and machine learning techniques. The detector 155 may be stateless or stateful.

[0026] Traditionally, a problem 160 triggers a notification sent to a member(s) of a security team, who then acknowledges the problem and orchestrates its remediation. However, this traditional process flow is time-consuming and resource-intensive (e.g., not having enough member(s) available to address all problems at a given time), resulting in delayed response or remediation of the problem and being decoupled from the source of the problem. To overcome these and other challenges, various embodiments are directed to a response system 165 that can be used to automatically remediate the problem 160 using artificial intelligence. The problem 160 can be addressed in multiple ways, and a set of responders 170 is provided that are adapted to take specific actions 175 (e.g., mitigating actions, corrective actions, or preventative actions) for the problem 160. For example, if one of the detectors 155 detects a problem with a compute instance, one of the responders 170 may take action 175 to quarantine the instance. A response recommendation is implemented as part of the security architecture 110 to determine which responder to select (i.e., which action to take) for a given problem. The responders 170 are selected by the response recommender 180 based on the type of problem, the target of the problem, the severity of the problem, and other attributes. In some cases, the user can configure the response recommender 180 with guidelines to limit what the response recommender 180 can recommend for a given problem 160. For example, a user might want to indicate: "Don't suspend users unless they have failed multi-factor authentication N times and are from a previously seen IP address." Alternatively, for less severe alerts, the user might opt ​​for a simple notification.

[0027] It should be appreciated that it is important for the response recommender 180 to be able to learn from its mistakes. To that end, the response system 165 provides a feedback loop, also known as active learning. The feedback loop allows the user to correct the actions 175 taken by the response system 165 and learns from the corrections. Active learning is implemented using artificial intelligence. Specifically, the response recommender 180 utilizes two or more predictive models to infer the best response to a given problem. A first model (referred to herein as a local model) learns the responder preferences for a specific user tenancy (e.g., specific to a given entity or organization). Generally, the local model is a powerful model, and the local model will be preferred if it exhibits high confidence. A second model (referred to herein as a global model) is an infrastructure-wide model that learns what all users generally prefer (e.g., across all entities or organizations) and is used early on when the local model is trained first. In some cases, the local model replaces the global model as additional corresponding recommendation part labels are collected through active learning. Alternatively, the local model is used in combination with the global model as additional corresponding recommendation part labels are collected through active learning. An optional third model (referred to herein as intermediate-level model(s)) may be used. ) learns counterpart preferences for a subset of users (e.g., similar entities or organizations, such as healthcare entities or financial institutions) and is used in combination with the local and / or global models.

[0028] 2 is a block diagram illustrating a response system with active learning 200 according to various embodiments. As described in FIG. 1, in this example, the active learning response performed by response system 200 includes several stages: problem acquisition stage 205, model training stage 210, response recommendation stage with active learning 215, and responder stage 220.

[0029] The problem acquisition stage 205 includes one or more detectors (e.g., detector 155 described in FIG. 1 ) for deriving problems 225 from signals of data coming from emitters, and one or more repositories or data stores for storing the problems. The problems 225 raised by the detectors may be correlated and grouped to reduce duplication and improve relevance. For example, if one actor performs two activities from a suspicious IP, the response system 200 may group these activities into one problem. Multiple problems may be correlated with each other by infrastructure, emitter, resource type, tags, etc.

[0030] If new models 230a-230n ("n" represents a natural number) (sometimes referred to herein individually as new models 230 or collectively as models 230) are trained before deployment, problems 225 are obtained from the problem acquisition stage 205 by the model training stage 210 and used to train the new models 230. The model training stage 210 may train one or more new models 230 for use in other stages at runtime. For example, the model training stage may train local models before deployment using artificially created problems 225 or past problems 225 that similar local models encountered at runtime. If new models 230a-230n are not trained before deployment, problems 225 are obtained from the problem acquisition stage 205 by the response recommendation stage 215 and used at runtime to select active learning and responses for the global model 235, local models 240a-240n, and / or intermediate models 245a-245n. The various models (models 230; 235; 240; 245) may be any machine learning ("ML") model, such as gradient boosting, random forest, SVM (support vector machine), convolutional neural network ("CNN"), e.g., inception neural network, residual neural network ("Resnet"), U-Net, V-Net, Single Shot Multibox Detector ("SSD") network, or recurrent neural network ("RNN"), or any combination thereof. Response system 200 may employ the same or different types of models to predict and recommend responses.

[0031] In this example, to train the new model 230, problems 225 are obtained and classified into a subset of problems 225a (e.g., 90%) for training and a subset of images 225b (e.g., 10%) for validation / testing. The subset of problems 225a may be obtained from the detector. In some cases, the subset of problems 225a is obtained from a data storage structure, such as a database, SIEM, or SOAR solution, associated with one or more detectors. In some cases, the subset of problems 225a is preprocessed and / or augmented to prepare problems for training the new model 230. For example, the data in the problem may be normalized to change values ​​in the dataset to a common scale without distorting differences in value ranges, and / or the data in the problem may be artificially augmented to allow for use in training the model without actually collecting new data. This may increase the diversity of the available data. In some cases, a subset of problems 225a is annotated with labels 250. The annotation may be done manually by one or more humans (annotators, such as members of a security team) identifying the action to be taken in response to each problem in the subset of problems 225a and giving the problem a label 250. The primary purpose of labeling and training new models 230 is to improve inference of responses to problems before deploying the model in the response recommendation stage 215.

[0032] The training process involves selecting hyperparameters for the new model 230 and performing an iterative process in which problems from a subset of the problem 225a are input to the new model 230 to find a set of model parameters (e.g., weights and / or biases) that minimize the objective function of the new model 230. Hyperparameters are settings that can be adjusted or optimized to control the behavior of the new model 230. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters include the number of hidden units of the model, the learning rate of the model, weighting between loss terms, the width of the convolution kernel, the number of kernels of the model, the learning rate, the batch size, and the batch composition.

[0033] Each training iteration may include finding a model parameter set for a new model 230 (composed of a defined set of hyperparameters) such that the value of the objective function using the model parameter set is smaller than the value of the objective function using a different model parameter set in the previous iteration. The objective function may be configured to measure the difference between the output inferred using the new model 230 and the ground truth annotated to the problem using labels 250. Once the model parameter set is identified, the new model 230 may be trained and validated using a subset of images 225b (a test or validation dataset). The validation process involves iterative operations of inputting problems from a subset of the problem 225b to the new model 230, adjusting the hyperparameters, and ultimately finding the optimal hyperparameter set using validation techniques such as k-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, or nested cross-validation. Once the optimal hyperparameter set is obtained, the reserved test image set from a subset of problem 225b is input to the new model 230 to obtain an output (in this example, a selection of the inferred correspondence or correspondence), which is evaluated against the ground truth correspondence using correlation techniques such as Bland-Altman and Spearman's rank correlation coefficient, and performance metrics such as error, accuracy, precision, recall, and ROC (Receiver Operating Characteristic Curve) are calculated.

[0034] It should be understood that other training / validation mechanisms are contemplated and may be implemented within the response system 200. For example, a new model 230 may be trained and hyperparameters adjusted for problems from a subset of problems 225a, while problems from a subset of problems 225b may be used solely to test and evaluate the model's performance. Also, while the training mechanisms described for the model 230 focus on training new models, these training mechanisms may be utilized to fine-tune existing models 235; 240; 245 trained from other datasets, or models that are only partially trained or not trained at all prior to deployment. For example, in some cases, the model 235; 240; 245 may be pre-trained using problems detected from other entities or response systems, or may be continuously trained using problems detected in real time during deployment. In such cases, the model 230; 235; 240; 245 may be continuously retrained (and optionally retested / validated) as part of a feedback loop or active learning process (described in more detail later herein).

[0035] The model training stage 210 outputs a trained new model 230, an untrained model 235; 240; 245, or a retrained model 235; 240; 245, which is used by the response recommendation stage 215. The response recommendation stage 215 includes a response recommender 255 adapted to generate an inference or prediction of a response for a given problem 225. The response recommender 255 includes a global model 235, one or more local models 240a-240n, one or more intermediate models 245a-245n (optional), a discriminator 260, and a monitor / analyzer 265. The response recommender 255 uses two or more predictive models (i.e., two or more models selected from the global model 235, the local model 240, and the intermediate model 245) to infer the best response to take for the given problem 225. The local model 240 has learned or is adapted to learn the counterpart preferences for a specific user tenancy (e.g., specific to a given entity or organization). Generally, the local model 240 is a powerful model, and if the local model 240 exhibits a higher confidence score, the local model 240 should be preferred. The global model 235 is an infrastructure-wide model that has learned or is adapted to learn what all users generally like (e.g., across all entities or organizations) and is used early on when the local model 240 is trained first. Once additional counterpart recommender labels are collected through active learning and used to train the local model 240 (by the model training phase 210), the local model 240 can replace the global model 235. Alternatively, once additional counterpart recommender labels are collected through active learning and used to train the local model 240 (by the model training phase 210), the local model 240 may be used in combination with the global model 235.Any intermediate-level model 245 may have learned or be adapted to learn counterpart preferences for a subset of users (e.g., similar entities or organizations, such as healthcare entities or financial institutions) and may be used in combination with the local model 240 and / or the global model 235.

[0036] As described herein, a problem 225 is input to two or more models selected from two or more predictive models (i.e., a global model 235, a local model 240, and an intermediate model 245), each of which generates an inferred response based on the problem's features (e.g., a discovered pattern of features), and a final response (e.g., a best response) for the given problem 225 is determined by a discriminator 260 from the multiple inferred responses. The final response may be determined using one or more criteria. In some cases, the one or more criteria include weighting the two or more predictive models. The weighting of the two or more predictive models may be dynamically changed as the state of the models changes (e.g., as a local model is trained and validated, a model's weight may be increased). For example, when a new user first comes online, the global model 235 may be assigned a higher weight than the local model 240 because the user's local model 240 may have little or no training (optionally, the intermediate model 245 may be assigned a middle or highest weight). As a result, the inferred correspondences of the global model 235 and / or the intermediate models 245 may become the final correspondences based on the weighted ranking of the correspondences from each model.

[0037] In some cases, the one or more criteria include a confidence score of the inferred correspondence from each of the two or more predictive models. For example, the global model 235 may provide an inferred correspondence with a confidence score of 80%, and the local model 240 may provide an inferred correspondence with a confidence score of 85%. As a result, the inferred correspondence of the local model 240 may be the final correspondence based on a ranking of the confidence scores of the inferred correspondence from each model. In some cases, the one or more criteria include a weighting of the two or more predictive models and a confidence score of the inferred correspondence from each of the two or more predictive models. For example, when the local model 240 begins training and its model parameters are learned, the weighting applied to the local model 240 may be increased to equal or exceed the weighting of the global model 235. Furthermore, the global model 235 may provide an inferred correspondence with a confidence score of 87%, and the local model 240 may provide an inferred correspondence with a confidence score of 87%. As a result, the inferred correspondence of the local model 240 may become the final correspondence based on a ranking of the confidence scores that is factored to take into account the weights of the individual models.

[0038] The correspondence recommender 255 is adapted to operate in two modes: (1) a normal mode in which the correspondence recommender 255 infers a correspondence for a given problem 225 and selects the correspondence based on the inferred correspondence; and (2) a test mode in which the correspondence recommender 255 infers a correspondence for a given problem 225 within an active learning environment. In the active learning environment, before a correspondence is selected, the correspondence recommender 255 queries a user (e.g., a member of a security team) and / or the monitor / analyzer 265 to accept or reject the inferred correspondence (optionally providing a correct or ground truth correspondence if the inferred correspondence is rejected). The acceptance or rejection of the inferred correspondence (optionally providing a correct or ground truth correspondence) is used to generate and provide labels 250 for retraining or continuously training the model 235; 240; 245. A user of the correspondence system 200 may turn the two modes on or off (e.g., turn the test mode on or off). In some cases, if the testing / validation of a given model is successful (e.g., if the model achieves an inference accuracy rate above a predetermined threshold), the enabling system 200 may prompt the user to turn off the test mode (or alternatively turn on the normal mode) to stop or temporarily halt the training of the model(s). In some cases, if the enabling system 200 detects model drift for a given model (e.g., if the model's performance has drifted beyond an acceptable standard), the enabling system 200 may prompt the user to turn on the test mode (or alternatively turn off the normal mode) to initiate retraining of the model(s). In some cases, the enabling system 200 may remain in the test mode all the time to enable continuous active learning and training of the model(s).

[0039] The models 235; 240; 245 are trained and actively learn in three contexts: global, local, and regional. (1) Global, which is how the global model 235 globally learns the problem-to-response mapping for all organization or entity preferences using the security architecture. (2) Local, which is how the local model 240 learns the problem-to-response mapping for each organization or entity preference using the security architecture. (3) Regional, which is how the intermediate model 245 learns the problem-to-response mapping for subgroups of organizations or entities using the security architecture. Within each of these contexts during active learning, which obtains correct or ground truth correspondences from a user or monitor / analyzer 265, labels 250 are generated and provided for retraining or continuously training the models 235; 240; 245 using the correct or ground truth correspondences. The labels 250 generated from active learning can be stored in a global repository 270, a local repository 275, and an intermediate repository 280, respectively. Thus, active learning produces two sets of data: (1) a response to a problem (either generated by the model or corrected by the user) that can be used to select a response, and (2) labeled data for retraining or continuously training the model 235; 240; 245. You get two things.

[0040] For example, when a new user comes online, the user may start with a response recommender 255 running a global model 235 that is continuously trained on problems and responses across all entities or organizations, and a local model 240. The local model 240 may either (1) be pre-trained on training data that is not necessarily specific to the new user (e.g., an artificially generated training data set 225a) to learn an initial set of model parameters, or (2) be untrained with pre-selected model parameters ready to learn. Initially, as problems begin to be detected by the security architecture, the new user's response recommender 255 may be configured in test mode. Each problem is input to the global model 235 and the local model 240, and both models 235 and 240 generate inferred responses based on the features associated with the problem using the model parameters of each model. A discriminator 260 evaluates the inferred responses and provides an inferred response determined to be the final inferred response (in this example, the inferred response from the global model 235 is the most likely). The monitor / analyzer 265 monitors the output of the discriminator 260 and, in a test mode, provides the final inferred correspondence to the user to evaluate its accuracy, or automatically evaluates the accuracy of the final inferred correspondence. If the final inferred correspondence is determined to be accurate for the new user's preferences by the new user and / or the monitor / analyzer 265, the correspondence recommender 255 begins generating problem labels 250 that include the final inferred correspondence. The problems and associated labels 250 are stored in both the global repository 270 and the local repository 275 for positive reinforcement training for both the models 235 and 240.

[0041] Alternatively, if the final inferred correspondence for the new user's preference is determined to be inaccurate by the new user and / or the monitor / analyzer 265, the new user and / or the monitor / analyzer 265 provides a correct or ground truth correspondence, and the correspondence recommender 255 starts generating labels 250 including the correct or ground truth correspondence for the problem. For corrective training of the local model 240, the problems and associated labels 250 are stored in the local repository 275. In addition, the problems and associated labels 250 are stored in a general repository and / or queued for evaluation by a security architecture administrator. The administrator evaluates the problems and associated labels 250 to determine whether they should be included in the global repository 270 for corrective training of the global model 235. For example, if a question contains a feature set that the global model 235 has never seen before, then, at first glance, the inference correspondence is likely to be inaccurate, and the curator may determine that the question and associated labels 250 should be included in the global repository 270 to train the global model 240. On the other hand, if the question contains a feature set that the global model 235 has seen before, and the inference correspondence is accurate, at least for the global community of entities or organizations, then the inference correspondence is likely to be inaccurate due to new user preferences (e.g., new users have correspondences that are unique to their situation), and the curator may determine that the question and associated labels 250 should not be included in the global repository 270 to train the global model 235. Thus, global models are highly curated by human curators to avoid introducing noise into their training and use.

[0042] In some cases, the new user and / or monitor / analyzer 265's assessment of the accuracy of the final response includes comparing the final response to a ground truth response that the user would prefer for the problem. If the final correspondence matches the ground truth correspondence, the accuracy rate of the final correspondence is determined to be acceptable. If the final correspondence does not match the ground truth correspondence based on the comparison, the accuracy rate of the final correspondence is determined to be unacceptable. The ground truth may be provided manually by a user and / or automatically by the monitor / analyzer 265 (e.g., retrieved from a repository). A match can be determined in several ways, such as an exact match, a substantially exact match, or a partial match. In some cases, the match is an exact match. For example, the final correspondence "increase multi-factor authentication" is determined to match the ground truth "increase multi-factor authentication." In further or alternative examples, a match is a substantially exact match or at least a partial match. As used herein, the terms "substantially," "approximately," and "about" are defined as nearly as specified, but not necessarily exactly as specified, as understood by one of ordinary skill in the art (including exactly as specified). In the disclosed embodiments, the terms "substantially," "approximately," or "about" may be replaced with "within a [percentage] of," as specified, where percent includes 0.1, 1, 5, and 10 percent. For example, a final response of "increase authentication level" is determined to match a ground truth of "increase multi-factor authentication." Alternatively, a final response of "increase authentication level" is determined to match a ground truth of "increase multi-factor authentication" and / or "suspend user account."

[0043] In some cases, the new user and / or the monitor / analyzer 265's evaluation of the accuracy of the final response further includes analyzing an explanation for the final response. In some cases, each of the models 235; 340; 245 is adapted to explain its inference in simple terms. For example, "This user has generated N MFA problems, and in most of these types of cases, you have so far expressed pausing as the correct response (label)," so I recommend pausing this user." While some ML models are black boxes whose decisions are unexplainable or barely explainable, other algorithms are, in fact, explainable. For example, a gradient boosting machine may be constructed as a set of shallow decision trees. By reducing the number of trees included in the model and traversing the trees to create explanations, an estimator can be created to provide explanations. Other models, such as support vector machines (SVMs), also support explanations in the form of feature importance.

[0044] The response recommender 255 is further adapted to select a response unit 285 from the response unit stage 220 based on the output of the discriminator 260 (i.e., the final inferred response) or the output of the user and / or monitor / analyzer 265 (i.e., the correct response or ground truth response). The response unit stage 220 includes a plurality of response units 285 selected by the response recommender 255. Each response unit is associated with one or more actions 290a-290n to be automatically taken by the security architecture to respond to or remediate a given issue. The response unit 285 includes active response units, including a quarantine response unit, a shutdown and snapshot response unit, a bucket invalidation response unit, an MFA (multi-factor authentication) increase response unit, a user account suspension response unit, a configuration reversion response unit, an add to blacklist response unit, and a public IP invalidation response unit. The response unit 285 may include a plurality of passive response units, including a notification response unit, an event response unit, and a log response unit.

[0045] The isolation response unit takes action to isolate specific infrastructure resources, such as compute resources, by disabling network access to those resources. The shutdown and snapshot response unit takes action to shut down the compute resources and immediately take a snapshot of the instance. This response unit provides the user with a snapshot of the state at the time the problem was detected. The Threat Intelligence Response component stops threats related to compute resources while allowing users to capture them. The Bucket Disable component disables (but does not delete) misconfigured object store buckets. For example, buckets with public access can be disabled through this component. The MFA component forces end users to authenticate with multi-factor authentication. The User Account Suspension component suspends OCI user accounts. The Configuration Reversion component rolls back configuration changes made by users. The Blacklist Add component adds source IP addresses to user CP / DP blacklists. The Public IP Disable component removes public IP addresses from infrastructure resources, such as compute resources. The Notification component configures infrastructure notification services to report issues via available notification methods. The Event component emits cloud events that enable standard event outputs, such as notifications, streams, and functions. The log handling unit takes measures to report the problem as a log using an infrastructure logging service.

[0046] Using artificial intelligence to automatically remediate security issues 3 and 4 illustrate steps and operations for automatically remediating security issues with artificial intelligence. Individual embodiments may be described as steps, which may be depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While a flowchart may depict operations as sequential steps, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may terminate when the operation is completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, or the like. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0047] The processes and / or operations illustrated in FIGS. 3 and 4 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processor cores), hardware, or a combination thereof. The software may be stored in memory (e.g., on a memory element, on a non-transitory computer-readable storage medium). The particular sequence of operations illustrated in FIGS. 3 and 4 is not intended to be limiting. Other sequences of steps may be performed according to alternative embodiments. For example, in alternative embodiments, the steps outlined above may be performed in a different order. Individual steps illustrated in FIGS. 3 and 4 may also include multiple sub-steps that may be performed in various orders depending on the individual step. Further, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0048] FIG. 3 shows a flowchart 300 illustrating steps for automatically remediating security issues with two or more predictive models. In some embodiments, the steps shown in flowchart 300 may be implemented by the architectures, systems, and techniques shown in FIGS. 1 and 2. For example, these steps may be implemented in a response system with active learning to automatically select responses as responses to issues detected within a security architecture. The response system implements at least two models that actively learn in at least two contexts: a global context and a local context. (1) Global, which is how the global model learns the problem-to-response mapping for all user preferences globally using the security architecture; and (2) Local, which is how the local model learns the problem-to-response mapping for each user preference using the security architecture.

[0049] In step 305, a problem is acquired by a response system of the security architecture. The problem can be detected by a detector in a signal from an emitter associated with the user. The problem is a possible security event that the security architecture is adapted to respond to and remediate.

[0050] In step 310, a first response is inferred using a global model implemented as part of a response system that takes the problem as input. The global model includes a global model parameter set learned globally from a mapping of problems and responses for the preferences of all users (e.g., all of an entity or organization) using the security architecture. That is, the global model parameter set is learned from training data including problems and labels with ground truth correspondences that replicate the preferences of all users using the security architecture (i.e., supervised learning). Preferences are responses that users expect to be taken to the problem. An example of a given problem is "Alice's user credentials have been compromised," and examples of responses preferred by the global community of users are "suspend the user account" and / or "shut down and take a snapshot."

[0051] In step 315, a second response is inferred using a local model implemented as part of a response system that takes the problem as input. The local model includes a local model parameter set that is locally learned from a mapping of user (e.g., entity or organization) preferences between problems and responses. That is, the local model parameter set is learned from training data that includes problems and labels that include ground truth correspondences that replicate the preferences of users utilizing the security architecture (i.e., supervised learning). A preference is a response that a user expects to be taken to a problem. An example of a given problem is "Alice's user credentials have been compromised," and an example of a user's preferred response is "increase multi-factor authentication."

[0052] In optional step 320, a third response is inferred using an intermediate model implemented as part of a response system that takes the problem as input. The third response can be inferred instead of the first response or in addition to the first and second responses. The intermediate model includes a regional model parameter set learned regionally from a mapping of problems and responses for the preferences of a subgroup of users (e.g., entities or organizations within a given industry) using the security architecture. That is, the regional model parameter set is learned from training data including problems and labels with ground truth correspondences that replicate the preferences of the subgroup of users using the security architecture (i.e., supervised learning). The preferences are responses that the subgroup of users expect to take to the problem. An example of a given problem is "Alice's user credentials have been compromised," and an example of a response preferred by the local community of users is "suspend the user account" and / or "increase multi-factor authentication."

[0053] In step 325, a criterion is used to evaluate the first and second responses (and optionally, the third response). In some cases, the criterion includes weighting the global model and the local models (and optionally, the intermediate models). Model weights may be dynamically changed as the state of the models changes (e.g., as local models are trained and validated, model weights may be increased). For example, when a new user first comes online, the user's local models may have little or no training, so the global model may be assigned a higher weight than the local models (optionally, the intermediate models may be assigned a weight with a middle or highest value). This results in a ranking of the weighted responses from each model, and thus a ranking of the global model. The inferred correspondence of the model and / or intermediate model can become the final correspondence.

[0054] In some cases, the criteria include a confidence score of the inferred correspondence from each of the global model and the local model (optionally, an intermediate model). The confidence score may be computed according to any known process (e.g., softmax, a prediction function / operation, a Bayesian network, etc. are all known processes that can be implemented to estimate the confidence of a model's prediction). For example, the global model may provide an inferred correspondence with a confidence score of 80%, and the local model may provide an inferred correspondence with a confidence score of 85%. As a result, the inferred correspondence of the local model may be the final correspondence based on a ranking of the confidence scores of the inferred correspondence from each model. In some cases, the criteria include (1) a confidence score associated with each of the first and second correspondences (optionally, a third response) and (2) a weight associated with each of the global model and the local model (optionally, an intermediate model).

[0055] In step 330, the response system determines a final response to the problem based on the evaluation of the first and second responses (and optionally, the third response). For example, if the ranking of models by weight indicates that the global model response is the best response (e.g., the highest ranked model based on the weights), the first response is determined to be the final response. Alternatively, if the ranking of models by confidence score indicates that the local model response is the best response (e.g., the highest ranked response based on the confidence score), the second response is determined to be the final response. Alternatively, if the ranking of models by confidence score factored by weight indicates that the local model response is the best response (e.g., the highest ranked response / model based on the confidence score factored by the weights), the second response is determined to be the final response.

[0056] In step 335, a response part is selected from the set of response parts by the response system based on the final response. The response part is adapted to take one or more actions to respond to the problem. For example, if the final response is "suspend user account," the response system selects the suspend user account response part to take the action of suspending the user account, thereby resolving the problem "Alice's user credentials have been compromised."

[0057] In step 340, the response system takes one or more actions to address the problem based on the selected response unit. For example, the user account suspension response unit takes the action of suspending the user account, thereby resolving the problem "Alice's user credentials have been compromised."

[0058] FIG. 4 shows a flowchart 400 illustrating steps for automatically remediating security issues with two or more predictive models implemented with active learning. In some embodiments, the steps shown in flowchart 400 may be implemented by the architectures, systems, and techniques shown in FIGS. 1 and 2. For example, these steps may be implemented in a response system with active learning to automatically select an action as a response to an issue detected within a security architecture. The response system implements at least two models that actively learn in at least two contexts: a global context and a local context. (1) Global, which is how the global model learns the problem-to-action mapping for all user preferences globally using the security architecture; and (2) Local, which is how the local model learns the problem-to-action mapping for each user preference using the security architecture.

[0059] In step 405, the problem is acquired by the corresponding system of the security architecture. The problem can be detected by a detector in the signal from the emitter associated with the user. The problem is a possible security event that the security architecture is adapted to respond to and remediate.

[0060] In step 410, a first response is inferred using a global model implemented as part of a response system that takes the problem as input. The global model includes a global model parameter set learned globally from a mapping of problems and responses for the preferences of all users (e.g., all of an entity or organization) using the security architecture. That is, the global model parameter set is learned from training data including problems and labels, including ground truth correspondences that replicate the preferences of all users using the security architecture (i.e., supervised learning). Preferences are responses that users expect to be taken to the problem. An example of a given problem is "Alice's user credentials have been compromised," and examples of responses preferred by the global community of users are "suspend the user account" and / or "shut down and take a snapshot."

[0061] In step 415, a second response is inferred using a local model implemented as part of a response system that takes the problem as input. The local model includes a local model parameter set that is locally learned from a mapping of problems and responses for a user's (e.g., entity or organization's) preferences. That is, the local model parameter set is learned from training data that includes problems and labels that include ground truth correspondences that replicate the user's preferences using a security architecture (i.e., supervised learning). The preferences are the responses that the user expects to take to the problem. An example of a given problem is "Alice's user credentials have been compromised," and an example of a user's preferred response is "increase multi-factor authentication."

[0062] In optional step 420, a third response is inferred using an intermediate model implemented as part of a response system that takes the problem as input. The third response can be inferred instead of the first response or in addition to the first and second responses. The intermediate model includes a regional model parameter set that is regionally learned from a problem-to-response mapping for the preferences of a subgroup of users (e.g., entities or organizations within a given industry) using a security architecture. That is, the regional model parameter set is learned from training data that includes labels and problems, including ground truth correspondences that replicate the preferences of the subgroup of users using a security architecture (i.e., supervised learning). The preferences are the responses that the subgroup of users expect to take to the problem. An example of a given problem is "Alice's user credentials have been compromised," and an example of a response preferred by the local community of users is "suspend the user account" and / or "increase multi-factor authentication."

[0063] In step 425, a criterion is used to evaluate the first and second correspondences (and optionally, the third response). In some cases, the criterion includes weighting the global model and the local model (and optionally, the intermediate model). The model weights may be dynamically changed as the model state changes (e.g., as local models are trained and validated, model weights may be increased). For example, when a new user first comes online, the user's local model may have little or no training, so the global model may be assigned a higher weight than the local model (optionally, the intermediate model may be assigned a middle or highest weight). As a result, the inferred correspondence of the global model and / or the intermediate model may become the final correspondence based on the ranking of the weighted correspondences from each model.

[0064] In some cases, the criteria include a confidence score of the inferred correspondence from each of the global model and the local model (optionally, an intermediate model). The confidence score may be computed according to any known process (e.g., softmax, a prediction function / operation, a Bayesian network, etc. are all known processes that can be implemented to estimate the confidence of a model's prediction). For example, the global model may provide an inferred correspondence with a confidence score of 80%, and the local model may provide an inferred correspondence with a confidence score of 85%. As a result, the inferred correspondence of the local model may be the final correspondence based on a ranking of the confidence scores of the inferred correspondence from each model. In some cases, the criteria include (1) a confidence score associated with each of the first and second correspondences (optionally, a third response) and (2) a weight associated with each of the global model and the local model (optionally, an intermediate model).

[0065] In step 430, the response system determines a final response to the problem based on the evaluation of the first and second responses (and optionally, the third response). For example, if the ranking of models by weight indicates that the global model response is the best response (e.g., the highest ranked model based on the weights), the first response is determined to be the final response. Alternatively, if the ranking of models by confidence score indicates that the local model response is the best response (e.g., the highest ranked response based on the confidence score), the second response is determined to be the final response. Alternatively, if the ranking of models by confidence score factored by weight indicates that the local model response is the best response (e.g., the highest ranked response / model based on the confidence score factored by the weights), the second response is determined to be the final response.

[0066] In step 435, the accuracy of the final correspondence is evaluated by the correspondence system. The accuracy is evaluated based on a comparison of the final correspondence with a ground truth correspondence that the user would prefer for the problem. If the final correspondence matches the ground truth correspondence based on the comparison, the accuracy of the final correspondence is determined to be acceptable. If the final correspondence does not match the ground truth correspondence based on the comparison, the accuracy of the final correspondence is determined to be unacceptable. The ground truth may be provided manually by the user and / or automatically by a monitoring / analysis unit of the correspondence system (e.g., obtained from a repository). A match can be determined in several ways, such as an exact match, a substantially exact match, or a partial match. In some cases, the match is an exact match. In further or alternative examples, the match is a substantially exact match or at least a partial match.

[0067] In step 440, in response to determining that the accuracy rate is unacceptable, the response system (1) generates a label for the problem (the label includes the ground truth correspondence), (2) stores the label including the ground truth and the problem in a local data store (repository), and (3) in step 475, selects a correspondence from the set of correspondences based on the ground truth rather than the final correspondence. The correspondence is adapted to take one or more actions to respond to the problem. For example, if the ground truth is "suspend the user account," the response system selects the suspend user account correspondence to take the action of suspending the user account, resolving the problem "Alice's user credentials have been compromised."

[0068] In step 445, the corresponding system begins training (i.e., active learning) the local model with local training data from the local repository. The local training data includes labels, including ground truth and problems. Training can be initiated continuously, such that the local model parameters of the local model continually learn from the local training data. Alternatively, training can be batched or scheduled for retraining, such that the local model parameters of the local model periodically learn from the local training data. The process can start with

[0069] In step 450, in response to determining that the accuracy rate is unacceptable, the response system (1) stores the label including the ground truth and the problem in a general data store or places the label including the ground truth and the problem in a data queue for evaluation by the administrator, (2) receives from the administrator either a response of taking no action on the label including the ground truth and the problem or a response of training a global model using the label including the ground truth and the problem, (3) in response to taking no action, deletes the label including the ground truth and the problem from the data store or data queue, and (4) in response to training a global model, stores the label including the ground truth and the problem in a global repository.

[0070] In step 455, the response system begins training (i.e., active learning) the global model with global training data from the global repository. The global training data includes labels, including ground truth and problems. Training can be initiated continuously, such that the global model parameters of the global model continually learn from the global training data. Alternatively, training can be initiated in a batch or scheduled process for retraining, such that the global model parameters of the global model periodically learn from the global training data.

[0071] In step 460, in response to determining that the accuracy rate is acceptable, the correspondence system (1) generates a label for the problem (the label includes the final correspondence), and (2) stores the label including the final correspondence and the problem in the local data store and the global data store.

[0072] In step 465, the response system begins training (i.e., active learning) the global model with global training data from the global repository. The global training data includes labels, including ground truth and problems. Training can be initiated continuously, such that the global model parameters of the global model continually learn from the global training data. Alternatively, training can be initiated in a batch or scheduled process for retraining, such that the global model parameters of the global model periodically learn from the global training data.

[0073] In step 470, the response system begins training (i.e., active learning) the local model with local training data from the local repository. The local training data includes labels, including ground truth and problems. Training can be initiated continuously, such that the local model parameters of the local model continually learn from the local training data. Alternatively, training can be initiated in a batch or scheduled process for retraining, such that the local model parameters of the local model periodically learn from the local training data.

[0074] In step 475, a response part is selected from the set of response parts by the response system based on the final response. The response part is adapted to take one or more actions to respond to the problem. For example, if the final response is "suspend user account," the response system selects the suspend user account response part to take the action of suspending the user account, thereby resolving the problem "Alice's user credentials have been compromised."

[0075] In step 480, one or more actions are taken to address the problem. For example, the user account suspension response unit may take the action of temporarily suspending the user account, and then issue a "suspend" command. Resolve the "User credentials for this account have been compromised" issue.

[0076] Exemplary System As mentioned above, Infrastructure as a Service (IaaS) is a special type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may also provide a variety of services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) that accompany these infrastructure components. These services can therefore be policy-based, allowing IaaS users to enforce policies that drive load balancing and maintain application availability and performance.

[0077] In some cases, IaaS customers may access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log in to an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including load balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0078] In most cases, the cloud computing model requires the participation of a cloud provider, which can be, but does not have to be, a third-party service that specializes in providing (e.g., offering, renting, selling) IaaS. Also, an entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0079] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a provisioned application server or similar. It can also include the process of provisioning the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Thus, the customer will be responsible for handling the deployment of the operating system, middleware, and / or application (e.g., on top of self-service virtual machines (which can be spun up on demand)).

[0080] In some examples, IaaS provisioning may refer to obtaining a computer or virtual host for use and also installing the necessary libraries or services on it. In most cases, deployment does not include provisioning, which may need to be done first.

[0081] In some cases, IaaS provisioning presents two distinct challenges: first, the initial challenge of provisioning the initial set of infrastructure before you can run anything; and second, the challenge of deploying the existing infrastructure after everything is provisioned. One challenge is evolving the infrastructure (e.g. adding new services, modifying services, removing services, etc.). In some cases, these two challenges may be addressed by allowing the configuration of the infrastructure to be defined declaratively: one or more configuration files can define the infrastructure (e.g. which components are needed and how they interact). The entire topology of the infrastructure can then be defined declaratively (e.g. which resources depend on which resources and how those resources work together). In some cases, once the topology is defined, it is possible to generate workflows that create and / or manage the different components described in the configuration files.

[0082] In some examples, the infrastructure may have many elements connected to each other. For example, there may be one or more virtual private clouds (VPCs) (e.g., possibly on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, one or more security group rules may also be provisioned that define how security is configured for the network and one or more virtual machines (VMs). Other infrastructure elements may also be provisioned, such as load balancers, databases, etc. The infrastructure will evolve over time as more infrastructure elements are required and / or added.

[0083] In some cases, intermittent deployment techniques may be employed to enable infrastructure code deployment across various virtual computing environments. In addition, the described techniques enable management of the infrastructure within these environments. In some examples, a service team may write code that requires deployment to one or more, often many, different production environments (e.g., across various different geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure onto which the code will be deployed must first be installed. In some cases, provisioning may be done manually, and once the infrastructure is provisioned, resources may be provisioned using provisioning tools and / or development tools may be used to deploy the code.

[0084] FIG. 5 is a block diagram 500 illustrating an example IaaS architecture pattern according to at least one embodiment. A service operator 502 is communicatively coupled to a secure host tenancy 504. The secure host tenancy 504 may include a virtual cloud network (VCN) 506 and a secure host subnet 505. In some examples, the service operator 502 may employ one or more client computing devices. The client computing devices may be palm-sized portable devices (e.g., iPhones, mobile phones, iPads, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass head-mounted displays) that run software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, PalmOS, and that support internet, email, SMS (Short Message Service), Blackberry, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, including personal and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing devices may be, for example, but are not limited to, workstation computers running various commercially available UNIX or UNIX-like operating systems, including various GNU / Linux operating systems such as Google Chrome OS. Alternatively, or in addition, the client computing devices may be other electronic devices capable of communicating over a network and / or the Internet that have access to VCN 506, such as thin client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox gaming consoles with or without Kinect gesture input devices), and / or personal messaging devices.

[0085] VCN 506 may include a local peering gateway (LPG) 510. LPG 510 may be communicatively coupled to a secure shell (SSH) VCN 512 via an LPG 510 included in SSH VCN 512. SSH VCN 512 may include an SSH subnet 514, which may be communicatively coupled to a control plane VCN 516 via an LPG 510 included in control plane VCN 516. SSH VCN 512 may also be communicatively coupled to a data plane VCN 518 via LPG 510. Control plane VCN 516 and data plane VCN 518 may be included in a service tenancy 519. Service tenancy 519 may be owned and / or operated by an IaaS provider.

[0086] The control plane VCN 516 may include a control plane DMZ (Demilitarized Zone) tier 520 that functions as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based server operations may be limited, which may help limit security breaches. Additionally, the DMZ tier 520 may include one or more LB (Load Balancer) subnet(s) 522, a control plane App Tier 524 may include App subnet(s) 526, and a control plane Data Tier 528 may include DB (Database) subnet(s) 530 (e.g., front-end DB subnet(s) and / or back-end DB subnet(s)). The LB subnet(s) 522 included in the control plane DMZ tier 520 can be communicatively coupled to the app subnet(s) 526 included in the control plane app tier 524 and to an internet gateway 534 that may be included in the control plane VCN 516, and the app subnet(s) 526 can be communicatively coupled to the DB subnet(s) 530 included in the control plane data tier 528 and to a service gateway 536 and a NAT (Network Address Translation) gateway 538. The control plane VCN 516 can include the service gateway 536 and the NAT gateway 538.

[0087] The control plane VCN 516 may include a data plane mirror Aplitier 540. The data plane mirror Aplitier 540 may include app subnet(s) 526. The app subnet(s) 526 included in the data plane mirror Aplitier 540 may include a VNIC (Virtual Network Interface Controller) 542. The VNIC 542 may run a computing instance 544. The computing instance 544 may communicatively couple the app subnet(s) 526 of the data plane mirror Aplitier 540 to the app subnet(s) 526. The app subnet(s) 526 may be included in the data plane Aplitier 546.

[0088] The data plane VCN 518 is connected to the data plane Appli Tier 546 and the data plane The data plane data tier 550 may include a data plane DMZ tier 548 and a data plane data tier 550. The data plane DMZ tier 548 may include LB subnet(s) 522. The LB subnet(s) 522 may be communicatively coupled to app subnet(s) 526 of the data plane application tier 546 and to an internet gateway 534 of the data plane VCN 518. The app subnet(s) 526 may be communicatively coupled to a service gateway 536 of the data plane VCN 518 and a NAT gateway 538 of the data plane VCN 518. The data plane data tier 550 also includes DB subnet(s) 530. The DB subnet(s) 530 may be communicatively coupled to the app subnet(s) 526 of the data plane application tier 546.

[0089] The internet gateway 534 of the control plane VCN 516 and the internet gateway 534 of the data plane VCN 518 can be communicatively coupled to a metadata management service 552. The metadata management service 552 can be communicatively coupled to the public internet 554. The public internet 554 can be communicatively coupled to a NAT gateway 538 of the control plane VCN 516 and the NAT gateway 538 of the data plane VCN 518. The service gateway 536 of the control plane VCN 516 and the service gateway 536 of the data plane VCN 518 can be communicatively coupled to cloud services 556.

[0090] In some examples, the service gateway 536 of the control plane VCN 516 or the service gateway 536 of the data plane VCN 518 can make API (application programming interface) calls to the cloud service 556 without traversing the public internet 554. The API calls from the service gateway 536 to the cloud service 556 can be one-way. That is, the service gateway 536 can make an API call to the cloud service 556, and the cloud service 556 can send the requested data to the service gateway 536. However, the cloud service 556 would not initiate the API call to the service gateway 536.

[0091] In some examples, a secure host subnet 504 can be directly connected to an otherwise isolated service tenancy 519. The secure host subnet 508 can communicate with an SSH subnet 514 through an LPG 510. The LPG 510 can enable bidirectional communication through an otherwise isolated system. By connecting the secure host subnet 508 to the SSH subnet 514, the secure host subnet 508 can be given access to other entities within the service tenancy 519.

[0092] The control plane VCN 516 may enable users of the service tenancy 519 to install or provision desired resources. The desired resources provisioned in the control plane VCN 516 may be deployed or used in the data plane VCN 518. In some examples, the control plane VCN 516 may be isolated from the data plane VCN 518, and the data plane mirror Aplita 540 of the control plane VCN 516 may communicate with the data plane Aplita 546 of the data plane VCN 518 via a VNIC 542 that may be included in the data plane mirror Aplita 540 and a VNIC 542 that may be included in the data plane Aplita 546.

[0093] In some examples, users, or customers, of the system may make requests, for example, CRUD (Create, Read, Update, and Delete) operations, over the public internet 554. The public internet 554 may be used by the metadata management The request may be communicated to the service 552. The metadata management service 552 may communicate the request to the control plane VCN 516 through an internet gateway 534. The request may be received by LB subnet(s) 522 included in the control plane DMZ tier 520. The LB subnet(s) 522 may determine that the request is valid, and in response to this determination, the LB subnet(s) 522 may send the request to app subnet(s) 526 included in the control plane app tier 524. If the request is validated and requires a call to the public internet 554, the call to the public internet 554 may be sent to a NAT gateway 538 that can make the call to the public internet 554. The memory desired to be stored by the request may be stored in the DB subnet(s) 530.

[0094] In some examples, data plane mirror AppliTia 540 can facilitate direct communication between control plane VCN 516 and data plane VCN 518. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in data plane VCN 518. Via VNIC 542, control plane VCN 516 can make these changes, updates, or other suitable modifications to the configuration for resources included in data plane VCN 518 by communicating directly with those resources.

[0095] In some embodiments, the control plane VCN 516 and the data plane VCN 518 can be included in the service tenancy 519. In this case, a user or customer of the system need not own or operate either the control plane VCN 516 or the data plane VCN 518. Instead, an IaaS provider may own or operate both the control plane VCN 516 and the data plane VCN 518, which can be included in the service tenancy 519. This embodiment enables network isolation, preventing users or customers from interacting with the resources of other users or other customers. This embodiment also allows users or customers of the system to store databases privately without having to rely on the public internet 554, which may not have the desired level of storage security.

[0096] In other embodiments, the LB subnet(s) 522 included in the control plane VCN 516 can be configured to receive signals from the service gateway 536. In this embodiment, the control plane VCN 516 and the data plane VCN 518 can be configured by the IaaS provider's customers to be called without calling the public internet 554. Customers of the IaaS provider may desire this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and stored in the service tenancy 519, isolated from the public internet 554.

[0097] FIG. 6 is a block diagram 600 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 602 (e.g., service operator 802 in FIG. 5 ) can be communicatively coupled to a secure host tenancy 604 (e.g., secure host tenancy 804 in FIG. 5 ). The secure host tenancy 604 can include a VCN (Virtual Cloud Network) 606 (e.g., VCN 506 in FIG. 5 ) and a secure host subnet 608 (e.g., secure host subnet 508 in FIG. 5 ). The VCN 606 can include an LPG (Local Peering Gateway) 610 (e.g., LPG 510 in FIG. 5 ). The LPG 610 can be communicatively coupled to an SSH (Secure Shell) VCN 612 (e.g., SSH VCN 512 in FIG. 5 ) via the LPG 510 included in the SSH VCN 612. VCN 612 can include an SSH subnet 614 (e.g., SSH subnet 514 in FIG. 5 ), which can be communicatively coupled to a control plane VCN 616 (e.g., control plane VCN 516 in FIG. 5 ) via an LPG 610 included in control plane VCN 616. Control plane VCN 616 can be included in a service tenancy 619 (e.g., service tenancy 519 in FIG. 5 ), and data plane VCN 618 (e.g., data plane VCN 518 in FIG. 5 ) can be included in a customer tenancy 621, which can be owned or operated by a user or customer of the system.

[0098] The control plane VCN 616 may comprise a control plane DMZ tier 620 (e.g., control plane DMZ tier 820 in FIG. 5 ) that may include LB subnet(s) 622 (e.g., LB subnet(s) 822 in FIG. 5 ), a control plane Apli tier 624 (e.g., control plane Apli tier 824 in FIG. 5 ) that may include app subnet(s) 626 (e.g., app subnet(s) 826 in FIG. 5 ), and a control plane data tier 628 (e.g., control plane data tier 828 in FIG. 5 ) that may include DB (database) subnet(s) 630 (e.g., similar to DB subnet(s) 830 in FIG. 5 ). The LB subnet(s) 622 included in the control plane DMZ tier 620 can be communicatively coupled to the app subnet(s) 626 included in the control plane app tier 624 and to an internet gateway 634 (e.g., internet gateway 834 in FIG. 5 ) that can be included in the control plane VCN 616, and the app subnet(s) 626 can be communicatively coupled to the DB subnet(s) 630 included in the control plane data tier 628, to a service gateway 636 (e.g., service gateway in FIG. 5 ), and to a network address translation (NAT) gateway 638 (e.g., NAT gateway 838 in FIG. 5 ). The control plane VCN 616 can include the service gateway 636 and the NAT gateway 638.

[0099] The control plane VCN 616 may include a data plane mirror Apri Tier 640 (e.g., data plane mirror Apri Tier 840 of FIG. 5 ). The data plane mirror Apri Tier 640 may include app subnet(s) 626. The app subnet(s) 626 included in the data plane mirror Apri Tier 640 may include VNICs (Virtual Network Interface Controllers) 642 (e.g., VNICs 542) on which compute instances 644 can run (e.g., similar to compute instances 544 of FIG. 5 ). The compute instances 644 can facilitate communication between the app subnet(s) 626 of the data plane mirror Apri Tier 640 and the app subnet(s) 626 that may be included in the data plane Apri Tier 646 (e.g., data plane Apri Tier 546 of FIG. 5 ) via the VNICs 642 included in the data plane mirror Apri Tier 640 and the VNICs 642 included in the data plane Apri Tier 646.

[0100] An internet gateway 634 included in the control plane VCN 616 can be communicatively coupled to a metadata management service 652 (e.g., metadata management service 552 in FIG. 5 ). The metadata management service 652 can be communicatively coupled to the public internet 654 (e.g., public internet 554 in FIG. 5 ). The public internet 654 can be communicatively coupled to a NAT gateway 638 included in the control plane VCN 616. A service gateway 636 included in the control plane VCN 616 can be communicatively coupled to cloud services 656 (e.g., cloud services 556 in FIG. 5 ).

[0101] In some examples, the data plane VCN 618 can be included in the customer tenancy 621. In this case, the IaaS provider may provide a control plane VCN 616 for each customer, and the IaaS provider may install a unique compute instance 644, included in the service tenancy 619, for each customer. Each compute instance 644 may enable communication between the control plane VCN 616, included in the service tenancy 619, and the data plane VCN 618, included in the customer tenancy 621. The compute instance 644 enables resources provisioned in the control plane VCN 616, included in the service tenancy 619, to be deployed or used in the data plane VCN 618, included in the customer tenancy 621.

[0102] In another example, an IaaS provider customer may have a database housed in customer tenancy 621. In this example, control plane VCN 616 may include data plane mirror Ap Tier 640, which may include app subnet(s) 626. Data plane mirror Ap Tier 640 may reside in data plane VCN 618, but data plane mirror Ap Tier 640 may not be housed in data plane VCN 618. That is, data plane mirror Ap Tier 640 may have access to customer tenancy 621, but data plane mirror Ap Tier 640 may not reside in data plane VCN 618 or be owned or operated by the IaaS provider customer. Data plane mirror Ap Tier 640 may be configured to call data plane VCN 618, but may not be configured to call entities contained in control plane VCN 616. A customer may wish to deploy or use resources in a data plane VCN 618 provisioned in a control plane VCN 616, and the data plane mirror appli tier 640 can facilitate the customer's desired deployment or use of resources.

[0103] In some embodiments, an IaaS provider's customers can apply filters to the data plane VCN 618. In this embodiment, the customer can determine which data plane VCNs 618 are accessible, and the customer may restrict access from the data plane VCN 618 to the public internet 654. The IaaS provider may not be able to apply filters or control the data plane VCN 618's access to any external networks or databases. Customer filtering and control of the data plane VCN 618 in the customer tenancy 621 can help isolate the data plane VCN 618 from other customers and the public internet 654.

[0104] In some embodiments, cloud services 656 can be called by the service gateway 636 to access services that may not reside on the public internet 654, the control plane VCN 616, or the data plane VCN 618. The connection between the cloud services 656 and the control plane VCN 616 or the data plane VCN 618 may be disconnected or continuous. The cloud services 656 may reside on different networks owned or operated by the IaaS provider. The cloud services 656 may be configured to receive calls from the service gateway 636 and may not be configured to receive calls from the public internet 654. The cloud services 656 may be isolated from other cloud services 656, and the control plane VCN 616 may be isolated from cloud services 656 that are not in the same region as the control plane VCN 616. For example, the control plane VCN 616 may be located in "Region 1," and cloud service "Deployment 5" may be located in Region 1 and Region 2. If a call to Deployment 5 is made by service gateway 636 included in control plane VCN 616 located in Region 1, the call may be sent to Deployment 5 in Region 1. In this example, control plane VCN 616, or Deployment 5 in Region 1, may not be communicatively coupled to or in communication with Deployment 5 in Region 2.

[0105] 7 is a block diagram 700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 702 (e.g., service operator 502 of FIG. 5 ) can be communicatively coupled to a secure host tenancy 704 (e.g., secure host tenancy 504 of FIG. 5 ). The secure host tenancy 704 can include a virtual cloud network (VCN) 706 (e.g., VCN 506 of FIG. 5 ) and a secure host subnet 708 (e.g., secure host subnet 508 of FIG. 5 ). The VCN 706 can include an LPG 710 (e.g., LPG 510 of FIG. 5 ) that can be communicatively coupled to an SSH VCN 712 (e.g., SSH VCN 512 of FIG. 5 ) via an LPG 710 included in the SSH VCN 712. SSH VCN 712 can comprise an SSH subnet 714 (e.g., SSH subnet 514 in FIG. 5 ), which can be communicatively coupled to a control plane VCN 716 (e.g., control plane VCN 516 in FIG. 5 ) via an LPG 710 included in the control plane VCN 716, and to a data plane VCN 718 (e.g., data plane 518 in FIG. 5 ) via an LPG 710 included in the data plane VCN 718. The control plane VCN 716 and the data plane VCN 718 can be included in a service tenancy 719 (e.g., service tenancy 519 in FIG. 5 ).

[0106] The control plane VCN 716 may include a control plane DMZ tier 720 (e.g., control plane DMZ tier 520 of FIG. 5 ), which may include LB (load balancer) subnet(s) 722 (e.g., LB subnet(s) 522 of FIG. 5 ), a control plane Aplicity tier 724 (e.g., control plane Aplicity tier 524 of FIG. 5 ), which may include app subnet(s) 726 (e.g., similar to app subnet(s) 526 of FIG. 5 ), and a control plane data tier 728 (e.g., control plane data tier 528 of FIG. 5 ), which may include DB subnet(s) 730. The LB subnet(s) 722 included in the control plane DMZ tier 720 can be communicatively coupled with the app subnet(s) 726 included in the control plane app tier 724 and can be communicatively coupled with an internet gateway 734 (e.g., internet gateway 534 in FIG. 5 ) that can be included in the control plane VCN 716, and the app subnet(s) 726 can be communicatively coupled with the DB subnet(s) 730 included in the control plane data tier 728 and can be communicatively coupled with a service gateway 736 (e.g., service gateway in FIG. 5 ) and a NAT (Network Address Translation) gateway 738 (e.g., NAT gateway 538 in FIG. 5 ). The control plane VCN 716 can include the service gateway 736 and the NAT gateway 738.

[0107] The data plane VCN 718 may comprise a data plane A-tier 746 (e.g., data plane A-tier 546 of FIG. 5 ), a data plane DMZ tier 748 (e.g., data plane DMZ tier 548 of FIG. 5 ), and a data plane data tier 750 (e.g., data plane data tier 550 of FIG. 5 ). The data plane DMZ tier 748 may comprise LB subnet(s) 722. The LB subnet 722 can be communicatively coupled to trusted app subnet(s) 760 and untrusted app subnet(s) 762 of the data plane app tier 746, and can be communicatively coupled to an internet gateway 734 included in the data plane VCN 718. The trusted app subnet(s) 760 can be communicatively coupled to a service gateway 736 included in the data plane VCN 718, a NAT gateway 738 included in the data plane VCN 718, and DB subnet(s) 730 included in the data plane data tier 750. The untrusted app subnet(s) 762 can be communicatively coupled to the service gateway 736 included in the data plane VCN 718 and the DB subnet(s) 730 included in the data plane data tier 750. The data plane data tier 750 may include the DB subnet(s) 730. The DB subnet(s) 730 can be communicatively coupled to the service gateway 736 included in the data plane VCN 718.

[0108] The untrusted app subnet(s) 762 can include one or more primary VNICs 764(1)-(N). The VNICs 764(1)-(N) can be communicatively coupled to tenant VMs (virtual machines) 766(1)-(N). Each tenant VM 766(1)-(N) can be communicatively coupled to a respective app subnet 767(1)-(N). The app subnets 767(1)-(N) can be included in a respective container egress VCN 768(1)-(N). The container egress VCNs 768(1)-(N) can be included in a respective customer tenancy 770(1)-(N). Each auxiliary VNIC 772(1)-(N) can facilitate communication between the untrusted app subnet(s) 762 included in the data plane VCN 718 and the app subnet included in the container egress VCN 768(1)-(N). Each container egress VCN 768(1)-(N) may include a NAT gateway 738 that may be communicatively coupled to the public internet 754 (e.g., public internet 554 in FIG. 5).

[0109] The internet gateway 734 included in the control plane VCN 716 and the internet gateway 734 included in the data plane VCN 718 can be communicatively coupled to a metadata management service 752 (e.g., metadata management system 552 of FIG. 5 ). The metadata management service 752 can be communicatively coupled to the public internet 754. The public internet 754 can be communicatively coupled to a NAT gateway 738 included in the control plane VCN 716 and the NAT gateway 738 included in the data plane VCN 718. The service gateway 736 included in the control plane VCN 716 and the service gateway 736 included in the data plane VCN 718 can be communicatively coupled to cloud services 756.

[0110] In some embodiments, data plane VCN 718 can be integrated with customer tenancy 770. This integration may be useful or desirable for an IaaS provider's customer in some cases, such as when they want support when running code. A customer may provide code for execution that may be destructive, communicate with other customer resources, or cause undesirable effects. Accordingly, the IaaS provider may determine whether to run code that the customer has provided to the IaaS provider.

[0111] In some examples, a customer of an IaaS provider may request that the IaaS provider grant temporary network access to the ability to attach to a data plane tier app 746. The code to perform this function may run in VMs 766(1)-(N), and the code may not be configured to run elsewhere on the data plane VCN 718. Each VM 766(1)-(N) may be connected to one customer tenancy 770. Each container 771(1)-(N) contained in a VM 766(1)-(N) may be connected to a single customer tenancy 770. ) may be configured to execute this code. In this case, there may be two isolations (e.g., containers 771(1)-(N) may be running code that may be contained in at least VMs 766(1)-(N) that are contained in untrusted app subnet(s) 762). The two isolations may help prevent erroneous or unwanted code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 771(1)-(N) may be communicatively coupled to customer tenancy 770 and configured to send or receive data from customer tenancy 770. Containers 771(1)-(N) may not be configured to send or receive data from other entities in data plane VCN 718. Once the code execution is complete, the IaaS provider may kill or discard containers 771(1)-(N).

[0112] In some embodiments, trusted app subnet(s) 760 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet(s) 760 may be communicatively coupled to DB subnet(s) 730 and may be configured to perform CRUD operations on DB subnet(s) 730. Untrusted app subnet(s) 762 may be communicatively coupled to DB subnet(s) 730, although in this embodiment, the untrusted app subnet(s) may be configured to perform read operations on DB subnet(s) 730. Containers 771(1)-(N) that may be included in each customer's VMs 766(1)-(N) and that may execute code from the customer may not be communicatively coupled to DB subnet(s) 730.

[0113] In other embodiments, the control plane VCN 716 and the data plane VCN 718 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 716 and the data plane VCN 718. However, communication may occur indirectly through at least one method. The LPG 710 may be established by an IaaS provider that can facilitate communication between the control plane VCN 716 and the data plane VCN 718. In another example, the control plane VCN 716 or the data plane VCN 718 can invoke a cloud service 756 via a service gateway 736. For example, a call from the control plane VCN 716 to the cloud service 756 can include a request for a service that can communicate with the data plane VCN 718.

[0114] FIG. 8 is a block diagram 800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 802 (e.g., service operator 502 of FIG. 5 ) can be communicatively coupled to a secure host tenancy 804 (e.g., secure host tenancy 504 of FIG. 5 ). Secure host tenancy 804 can comprise a virtual cloud network (VCN) 806 (e.g., VCN 506 of FIG. 5 ) and a secure host subnet 808 (e.g., secure host subnet 508 of FIG. 5 ). VCN 806 can comprise an LPG 810 (e.g., LPG 510 of FIG. 5 ) that can be communicatively coupled to an SSH VCN 812 (e.g., SSH VCN 512 of FIG. 5 ) via an LPG 810 included in SSH VCN 812. SSH VCN 812 can include an SSH subnet 814 (e.g., SSH subnet 514 in FIG. 5 ), which can be communicatively coupled to a control plane VCN 816 (e.g., control plane VCN 516 in FIG. 5 ) via an LPG 810 included in the control plane VCN 816, and to a data plane VCN 818 (e.g., data plane 518 in FIG. 5 ) via an LPG 810 included in the data plane VCN 818. The control plane VCN 816 and the data plane VCN 818 are managed by a service tenancy 819 (e.g., It can be included in the service tenancy 519).

[0115] The control plane VCN 816 may comprise a control plane DMZ tier 820 (e.g., control plane DMZ tier 520 of FIG. 5 ), which may include LB subnet(s) 822 (e.g., LB subnet(s) 522 of FIG. 5 ), a control plane Apli tier 824 (e.g., control plane Apli tier 524 of FIG. 5 ), which may include app subnet(s) 826 (e.g., app subnet(s) 526 of FIG. 5 ), and a control plane data tier 828 (e.g., control plane data tier 528 of FIG. 5 ), which may include DB subnet(s) 830 (e.g., DB subnet(s) 1030 of FIG. 10 ). The LB subnet(s) 822 included in the control plane DMZ tier 820 can be communicatively coupled with the app subnet(s) 826 included in the control plane app tier 824 and can be communicatively coupled with an internet gateway 834 (e.g., internet gateway 534 in FIG. 5 ) that can be included in the control plane VCN 816, and the app subnet(s) 826 can be communicatively coupled with the DB subnet(s) 830 included in the control plane data tier 828 and can be communicatively coupled with a service gateway 836 (e.g., service gateway in FIG. 5 ) and a NAT (network address translation) gateway 838 (e.g., NAT gateway 538 in FIG. 5 ). The control plane VCN 816 can include the service gateway 836 and the NAT gateway 838.

[0116] Data plane VCN 818 may comprise a data plane A-Tier 846 (e.g., data plane A-Tier 546 in FIG. 5 ), a data plane DMZ tier 848 (e.g., data plane DMZ tier 548 in FIG. 5 ), and a data plane data tier 850 (e.g., data plane data tier 550 in FIG. 5 ). Data plane DMZ tier 848 may comprise LB subnet(s) 822. LB subnet(s) 822 may be communicatively coupled to trusted app subnet(s) 860 (e.g., trusted app subnet(s) 1060 in FIG. 10 ) and untrusted app subnet(s) 862 (e.g., untrusted app subnet(s) 1062 in FIG. 10 ) of data plane A-Tier 846, and may be communicatively coupled to an Internet gateway 834 included in data plane VCN 818. Trusted app subnet(s) 860 can be communicatively coupled to service gateway 836 included in data plane VCN 818, NAT gateway 838 included in data plane VCN 818, and DB subnet(s) 830 included in data plane data tier 850. Untrusted app subnet(s) 862 can be communicatively coupled to service gateway 836 included in data plane VCN 818 and DB subnet(s) 830 included in data plane data tier 850. Data plane data tier 850 may include DB subnet(s) 830. DB subnet(s) 830 can be communicatively coupled to service gateway 836 included in data plane VCN 818.

[0117] The untrusted app subnet(s) 862 may include primary VNICs 864(1)-(N). The VNICs 864(1)-(N) may be communicatively coupled to tenant VMs (virtual machines) 866(1)-(N) present in the untrusted app subnet(s) 862. Each tenant VM 866(1)-(N) may execute code in a respective container 867(1)-(N) and may be communicatively coupled to the app subnet 826. The app subnet 826 may be included in a data plane app tier 846. The data plane app tier 846 may be included in a container egress VCN 868. Each auxiliary VNIC 872(1)-(N) may be included in a data plane VCN 818 and the app subnets included in the container egress VCN 868. The container egress VCN 868 can include a NAT gateway 838 that can be communicatively coupled to the public internet 854 (e.g., public internet 554 in FIG. 5).

[0118] The internet gateway 834 included in the control plane VCN 816 and the internet gateway 834 included in the data plane VCN 818 can be communicatively coupled to a metadata management service 852 (e.g., metadata management system 552 of FIG. 5 ). The metadata management service 852 can be communicatively coupled to the public internet 854. The public internet 854 can be communicatively coupled to a NAT gateway 838 included in the control plane VCN 816 and the NAT gateway 838 included in the data plane VCN 818. The service gateway 836 included in the control plane VCN 816 and the service gateway 836 included in the data plane VCN 818 can be communicatively coupled to cloud services 856.

[0119] In some examples, the pattern illustrated by the architecture of block diagram 800 in FIG. 8 may be considered an exception to the pattern illustrated by the architecture of block diagram 700 in FIG. 7 , and the pattern illustrated by the architecture of block diagram 800 may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., because regions are isolated). Each customer can access containers 867(1)-(N) contained in VMs 866(1)-(N) in real time. The containers 867(1)-(N) can be configured to call respective auxiliary VNICs 872(1)-(N) contained in app subnet(s) 826 of the data plane app tier 846, which can be included in a container egress VCN 868. The auxiliary VNICs 872(1)-(N) can send the call to a NAT gateway 838. The NAT gateway 838 can send the call to the public Internet 854. In this example, containers 867(1)-(N) that customers can access in real time can be isolated from control plane VCN 816 and isolated from other entities included in data plane VCN 818. Containers 867(1)-(N) can also be isolated from other customer resources.

[0120] In another example, a customer can invoke cloud service 856 using containers 867(1)-(N). In this example, the customer can execute code in containers 867(1)-(N) that requests a service from cloud service 856. Containers 867(1)-(N) can send the request to auxiliary VNICs 872(1)-(N). Auxiliary VNICs 872(1)-(N) can send the request to a NAT gateway. The NAT gateway can send the request to public internet 854. Public internet 854 can send the request to LB subnet(s) 822 included in control plane VCN 816 via internet gateway 834. In response to determining that the request is valid, the LB subnet(s) can send the request to app subnet(s) 826. The app subnet(s) 826 can send the request to cloud service 856 via service gateway 836.

[0121] It should be understood that the illustrated IaaS architectures 500, 600, 700, 800 may have components other than those shown. Additionally, the illustrated embodiments are only a few examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may have more components than those shown, fewer components than those shown, a combination of two or more components, or a different configuration or arrangement of components.

[0122] In certain embodiments, an IaaS system may include a suite of application, middleware, and database service offerings delivered to customers on a self-service, subscription-based basis in an elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is OCI (Oracle Cloud Infrastructure), offered by the assignee of the present application.

[0123] 9 is a diagram illustrating an example of a computer system 900 on which various embodiments of the present disclosure may be implemented. System 900 may be used to implement any of the computer systems described above. As shown, computer system 900 includes a processing unit 904 that communicates with several peripheral subsystems via a bus subsystem 902. These peripheral subsystems may include a processing acceleration unit 906, an I / O subsystem 908, a storage subsystem 918, and a communication subsystem 924. The storage subsystem 918 includes a tangible computer-readable storage medium 922 and a system memory 910.

[0124] Bus subsystem 902 provides a mechanism for allowing the various components and subsystems of computer system 900 to communicate with each other as intended. While bus subsystem 902 is illustrated as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 902 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a Mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0125] Processing unit 904, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of computer system 900. Processing unit 904 may include one or more processors. These processors may include single-core or multi-core processors. In particular embodiments, processing unit 904 may be implemented as one or more independent processing units 932 and / or 934, each including a single-core or multi-core processor. In other embodiments, processing unit 904 may be implemented as a quad-core processing unit formed by incorporating two dual-core processors on a single chip.

[0126] In various embodiments, the processing unit 904 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any time, some or all of the program code being executed may reside in the processor(s) 904 and / or in the storage subsystem 918. With appropriate programming, the processor(s) 904 may provide the various functions described above. The computer system 900 may further include a processing acceleration unit 906. The processing acceleration unit 906 may include a DSP (digital signal processor) and / or a special purpose processor, etc.

[0127] The I / O subsystem 908 includes user interface input devices and and face output devices. User interface input devices may include pointing devices such as keyboards, mice, or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor, which allows a user to control and interact with input devices such as a Microsoft Xbox® 360 game controller through a natural user interface using gesture and voice commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detection device, which detects a user's eye activity (e.g., "blinking" while taking a photo and / or making a menu selection) and translates eye gestures into input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) by voice commands.

[0128] User interface input devices also include, but are not limited to, 3D (three-dimensional) mice, joysticks or pointing sticks, game pads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and ultrasound devices. User interface input devices may also include audio input devices, such as MIDI keyboards and digital musical instruments.

[0129] User interface output devices may include a display subsystem, indicator lights, or non-visual display devices such as audio output devices. The display subsystem may be a flat-panel display such as one using a CRT (cathode ray tube), LCD (liquid crystal display), or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all manner of devices and mechanisms for outputting information from computer system 900 to a user or to another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey textual, graphical, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, drawing devices, audio output devices, and modems.

[0130] Computer system 900 may be configured with a storage subsystem 918 that includes software elements that are illustrated as currently residing in system memory 910. System memory 910 may store program instructions that are loadable and executable on processing unit 904, as well as data generated during the execution of these programs.

[0131] Depending on the configuration and type of computer system 900, the system memory 910 may be volatile memory (such as RAM (random access memory)) and / or non-volatile memory (such as ROM (read only memory), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 904. In some implementations, the system memory 910 may be static RAM (SRAM) or DRRAM (DRAM). The system memory 910 may include multiple different types of memory, such as dynamic RAM (RAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 900, such as during startup, may typically be stored in ROM. Also, by way of example, the system memory 910 includes application programs 912, which may include client applications, a web browser, mid-tier applications, a relational database management system (RDBMS), and the like; program data 914; and an operating system 916. By way of example, the operating system 916 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, and the like), and / or mobile operating systems, such as iOS, Windows® Phone, Android® OS, BlackBerry® 9 OS, and Palm® OS operating systems.

[0132] Additionally, storage subsystem 918 may provide a tangible, computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the functionality described above may be stored in storage subsystem 918. These software modules or instructions may be executed by processing unit 904. Storage subsystem 918 may also provide a repository for storing data used in accordance with the present disclosure.

[0133] Storage subsystem 900 may also include a computer-readable storage medium reader 920 that may be further coupled to a computer-readable storage medium 922. In conjunction with system memory 910, and optionally in combination with system memory 910, computer-readable storage medium 922 may comprehensively represent remote, local, fixed, and / or removable storage devices, as well as storage media for containing, storing, transmitting, and retrieving computer-readable information on a temporary and / or even permanent basis.

[0134] The computer-readable storage medium 922 containing the code or portions of code may include any suitable medium known or used in the art, including, but not limited to, storage and communication media, such as volatile and nonvolatile, removable and non-removable media, implemented in any method and technology for storing and / or transmitting information. This may include tangible computer-readable storage media, such as RAM, ROM, EEPROM (Electronically Erasable Programmable ROM), flash memory, or other memory technology, CD-ROM, DVD (Digital Versatile Disk), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage, or other tangible computer-readable media. This may also include intangible computer-readable media, such as data signals, data transmissions, or other media that can be used to transmit the desired information and that can be accessed by computing system 900.

[0135] By way of example, the computer readable storage medium 922 may include a hard disk drive that reads from or writes to fixed, nonvolatile magnetic media, a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk, and a CD-ROM drive. The computer system 900 may also include an optical disk drive that reads from or writes to removable, nonvolatile optical disks or other optical media, such as ROM, DVDs, and Blu-Ray® disks. The computer-readable storage medium 922 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer-readable storage medium 922 may also include solid-state drives (SSDs) based on nonvolatile memory, such as flash memory-based solid-state drives (SSDs), enterprise flash drives, and solid-state ROM; volatile memory-based SSDs, such as solid-state RAM, dynamic RAM, static RAM, and DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. These disk drives and their associated computer-readable media may provide nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 900.

[0136] The communications subsystem 924 provides an interface to other computer systems and networks. The communications subsystem 924 serves as an interface for receiving data from the computer system 900 and transmitting data from the computer system 900 to other systems. For example, the communications subsystem 924 may enable the computer system 900 to connect to one or more devices over the Internet. In some embodiments, the communications subsystem 924 may support a variety of technologies (e.g., cellular technology, 3G, 4G, or EDGE (Enhanced Data Rates For The communications subsystem 924 may include a Radio Frequency (RF) transceiver component, a Global Positioning System (GPS) receiver component, and / or other components for accessing wireless voice or / and data networks (using next generation data network technologies such as IEEE 802.28 (Global Evolution), WiFi (IEEE 802.28 family of standards), other mobile communications technologies, or any combination thereof. In some embodiments, the communications subsystem 924 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0137] In some embodiments, the communications subsystem 924 may receive incoming communications in the form of structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc., on behalf of one or more users who may utilize the computer system 900.

[0138] As an example, the communications subsystem 924 may be configured to receive data feeds 926 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, RSS (Rich Site Summary) feeds, and / or real-time updates from one or more third-party information sources.

[0139] Additionally, the communications subsystem 924 may be configured to receive data in the form of a continuous data stream, which may include an event stream 928 of real-time events and / or event updates 930 that essentially has no apparent end, continuous or infinite. Applications may include, for example, sensor data applications, tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like.

[0140] The communications subsystem 924 may also be configured to output structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 900.

[0141] The computer system 900 may be one of a variety of types, including a palm-sized portable device (e.g., an IPhone® mobile phone, an IPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or other data processing system.

[0142] Due to the ever-changing nature of computers and networks, the illustrated description of computer system 900 is illustrative only. Many other configurations are possible having more or fewer components than the illustrated system. For example, customized hardware may be used, and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods for implementing the various embodiments.

[0143] While specific embodiments have been described, various modifications, substitutions, alternative configurations, and equivalents are encompassed within the scope of the present disclosure. The embodiments are not limited to operation in one particular data processing environment, but can freely operate in multiple data processing environments. In addition, while the embodiments have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the described sequence of transactions and steps. Various features and aspects of the above-described embodiments may be used individually or jointly.

[0144] Furthermore, while embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. Various processes described herein may be implemented on the same processor or on any combination of different processors. Thus, where a component or module is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor) to perform the operation, or any combination thereof. Processes may communicate using various techniques, including, but not limited to, conventional techniques for inter-process communication; different paired processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0145] The specification and drawings are to be appropriately regarded as illustrative rather than strict, but it will be apparent that additions, subtractions, deletions and other modifications and alterations may be made thereto without departing from the broader spirit and scope of the appended claims. Accordingly, while specific embodiments of the present disclosure have been described, These are not limiting and various modifications and equivalents are intended to fall within the scope of the appended claims.

[0146] In the context of describing the disclosed embodiments (particularly in the context of the appended claims), the use of the terms "a," "an," and "the," and terms of similar referents, should be construed to include both the singular and the plural, unless inconsistent. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise noted. The term "connected" should be construed as including, attached to, or joined together, partly or wholly, even if there is something intervening. The recitation of ranges of values ​​herein is merely a shortcut for referring individually to each individual value within the range, unless otherwise indicated herein, and each individual value is included in the specification as if it were individually stated. All methods described herein can be performed in any suitable order, unless inconsistent. Any examples or exemplary language (e.g., "such as") provided herein are merely intended to further clarify embodiments and do not limit the scope of the disclosure, unless specifically claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0147] Disjunctive language, such as the phrase "at least one of X, Y, or Z," should be understood to be within the scope of common usage to indicate that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless specifically indicated otherwise. Thus, such disjunctive language is generally not intended to imply that a particular embodiment requires, and does not require, at least one of X, at least one of Y, or at least one of Z.

[0148] Preferred embodiments of the present disclosure, including the best mode known for carrying out the disclosure, are described herein. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art will be able to employ such variations as appropriate, and the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Additionally, any combination of the above-described elements in any possible variations thereof is included in this disclosure, unless expressly indicated herein.

[0149] All references, including publications, patent applications, and patents, cited in this specification are incorporated by reference herein to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and set forth in its entirety herein.

[0150] While aspects of the disclosure have been described in the foregoing specification with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or jointly. Moreover, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and accompanying drawings are to be regarded as illustrative rather than restrictive.

Claims

1. 1. A method comprising: A response system of the security architecture obtains a detected problem in a signal from an emitter associated with the user; inferring a first correspondence using a global model implemented as part of the correspondence system that takes the problem as an input, the global model including a set of global model parameters learned globally from a mapping of problems to correspondences for all user preferences using the security architecture, the method further comprising: inferring a second correspondence using a local model implemented as part of the correspondence system that takes the problem as input, the local model including a set of local model parameters learned locally from a mapping of problems and correspondences of the user's preferences, the method further comprising: the correspondence system includes evaluating the first correspondence and the second correspondence using criteria, the criteria including (1) a confidence score associated with each of the first correspondence and the second correspondence, and (2) a weight associated with each of the global model and the local model, the method further comprising: the response system determining a final response to the problem based on the evaluation of the first response and the second response; the response system selecting a response unit from a set of response units based on the final response, the response unit being adapted to take one or more actions to respond to the problem.

2. and, prior to selecting the correspondence, evaluating a accuracy rate of the final correspondence using the correspondence system, the accuracy rate being evaluated based on a comparison of the final correspondence with a ground truth correspondence that the user would prefer for the problem, and determining the accuracy rate of the final correspondence as acceptable if the final correspondence matches the ground truth correspondence based on the comparison, and determining the accuracy rate as unacceptable if the final correspondence does not match the ground truth correspondence based on the comparison; The method further includes, in response to determining that the accuracy rate is unacceptable, generating labels for the problem, the labels including the ground truth correspondences; The method further includes, in response to determining that the accuracy rate is unacceptable, storing the labels, including the ground truth and the problem, in a local data store; selecting the correspondence from among the set of correspondences based on the ground truth rather than the final correspondence; The method further includes, in response to determining that the accuracy rate is acceptable, generating a label for the problem, the label including the final correspondence; The method further includes, in response to determining that the accuracy rate is acceptable, storing the label including the final response and the problem in the local data store and the global data store; and selecting the correspondence from among the set of correspondences based on the final correspondence.

3. In response to the accuracy rate being determined to be unacceptable, storing the labels including the ground truth and the problem in a global data store or data queue for evaluation by a curator; A response in which no action is taken for the label including the ground truth and the problem, or a response in which no action is taken for the label including the ground truth and the problem receiving from the administrator one of the responses of training the global model; In response to the response being a no-action response, removing the label including the ground truth and the problem from the data store or the data queue; 3. The method of claim 2, further comprising: storing the label including the ground truth and the problem in the global repository in response to the correspondence being a correspondence to train the global model.

4. training the global model with global training data from the global repository, the global training data including the labels including the final response and the problem; training the local model with local training data from the local repository, the local training data including the labels including the final response and the problem.

5. The method of claim 2 , further comprising training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problems.

6. training the global model with global training data from the global repository, the global training data including the labels including the ground truth correspondences and the problems; training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problems.

7. The method of claim 1 , further comprising the response system taking the one or more actions to respond to the problem.

8. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising: A response system of the security architecture obtains a detected problem in a signal from an emitter associated with the user; inferring a first correspondence using a global model implemented as part of the correspondence system that takes the problem as an input, the global model including a set of global model parameters learned globally from a mapping of problems to correspondences for all user preferences using the security architecture, the process further comprising: inferring a second correspondence using a local model implemented as part of the correspondence system that takes the problem as input, the local model including a set of local model parameters learned locally from a mapping of problems and correspondences of the user's preferences, the process further comprising: the correspondence system includes evaluating the first correspondence and the second correspondence using criteria, the criteria including (1) a confidence score associated with each of the first correspondence and the second correspondence, and (2) a weight associated with each of the global model and the local model, and the process further includes: the response system determining a final response to the problem based on the evaluation of the first response and the second response; and the response system selecting a response unit from a set of response units based on the final response, the response unit being adapted to take one or more actions to respond to the problem.

9. The process may further include: and before selecting the correspondence, using the correspondence system to evaluate a accuracy rate of the final correspondence, the accuracy rate being evaluated based on a comparison between the final correspondence and a ground truth correspondence that the user would prefer for the problem, and determining the accuracy rate of the final correspondence as acceptable if the final correspondence matches the ground truth correspondence based on the comparison, and determining the accuracy rate as unacceptable if the final correspondence does not match the ground truth correspondence based on the comparison; The process further includes, in response to determining that the accuracy rate is unacceptable, generating labels for the problem, the labels including the ground truth correspondences; The process further includes, in response to determining that the accuracy rate is unacceptable, storing the labels, including the ground truth and the problem, in a local data store; selecting the correspondence from among the set of correspondences based on the ground truth rather than the final correspondence; The process further includes, in response to determining that the accuracy rate is acceptable, generating a label for the problem, the label including the final correspondence; The process further includes, in response to determining that the accuracy rate is acceptable, storing the label including the final response and the problem in the local data store and the global data store; and selecting the correspondence from among the set of correspondences based on the final correspondence.

10. The process may further include: In response to the accuracy rate being determined to be unacceptable, storing the labels including the ground truth and the problem in a common data store or data queue for evaluation by a curator; receiving from the curator either a response of taking no action on the label including the ground truth and the problem, or a response of training the global model using the label including the ground truth and the problem; In response to the response being a no-action response, removing the label including the ground truth and the problem from the data store or the data queue; and storing the label including the ground truth and the problem in the global repository in response to the correspondence being a correspondence to train the global model.

11. The process may further include: training the global model with global training data from the global repository, the global training data including the labels including the final response and the problem; 10. The non-transitory computer-readable memory of claim 9, further comprising training the local model with local training data from the local repository, the local training data including the labels including the final response and the problem.

12. The process may further include:

10. The non-transitory computer-readable memory of claim 9, further comprising training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problem.

13. The process may further include: training the global model with global training data from the global repository, the global training data including the labels including the ground truth correspondences and the problems; and training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problem.

14. The process may further include: The non-transitory computer-readable memory of claim 8 , wherein the response system comprises taking the one or more actions to respond to the problem.

15. 1. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising: A response system of the security architecture obtains a detected problem in a signal from an emitter associated with the user; inferring a first correspondence using a global model implemented as part of the correspondence system that takes the problem as an input, the global model including a set of global model parameters learned globally from a mapping of problems to correspondences for all user preferences using the security architecture, the process further comprising: inferring a second correspondence using a local model implemented as part of the correspondence system that takes the problem as input, the local model including a set of local model parameters learned locally from a mapping of problems and correspondences of the user's preferences, the process further comprising: the correspondence system includes evaluating the first correspondence and the second correspondence using criteria, the criteria including (1) a confidence score associated with each of the first correspondence and the second correspondence, and (2) a weight associated with each of the global model and the local model, and the process further includes: the response system determining a final response to the problem based on the evaluation of the first response and the second response; The response system includes selecting a response unit from a set of response units based on the final response, the response unit being adapted to take one or more actions to respond to the problem.

16. The process may further include: and before selecting the correspondence, using the correspondence system to evaluate a accuracy rate of the final correspondence, the accuracy rate being evaluated based on a comparison between the final correspondence and a ground truth correspondence that the user would prefer for the problem, and determining the accuracy rate of the final correspondence as acceptable if the final correspondence matches the ground truth correspondence based on the comparison, and determining the accuracy rate as unacceptable if the final correspondence does not match the ground truth correspondence based on the comparison; The process further includes, in response to determining that the accuracy rate is unacceptable, generating labels for the problem, the labels including the ground truth correspondences; The process further includes, in response to determining that the accuracy rate is unacceptable, storing the labels, including the ground truth and the problem, in a local data store; selecting the correspondence from among the set of correspondences based on the ground truth rather than the final correspondence; The process further includes, in response to determining that the accuracy rate is acceptable, generating a label for the problem, the label including the final correspondence; The process further includes, in response to determining that the accuracy rate is acceptable, storing the label including the final response and the problem in the local data store and the global data store; and selecting the correspondence from among the set of correspondences based on the final correspondence.

17. The process may further include: In response to the accuracy rate being determined to be unacceptable, storing the labels including the ground truth and the problem in a common data store or data queue for evaluation by a curator; receiving from the curator either a response of taking no action on the label including the ground truth and the problem, or a response of training the global model using the label including the ground truth and the problem; In response to the response being a no-action response, removing the label including the ground truth and the problem from the data store or the data queue; and storing the label including the ground truth and the problem in the global repository in response to the correspondence being a correspondence to train the global model.

18. The process may further include: training the global model with global training data from the global repository, the global training data including the labels including the final response and the problem; 17. The non-transitory computer-readable memory of claim 16, further comprising training the local model with local training data from the local repository, the local training data including the label comprising the final response and the problem.

19. The process may further include:

17. The non-transitory computer-readable memory of claim 16, further comprising training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problem.

20. The process may further include: training the global model with global training data from the global repository, the global training data including the labels including the ground truth correspondences and the problems; and training the local model with local training data from the local repository, the local training data including the labels including the ground truth correspondences and the problem.

Images