Computer automated processing using rule augmented machine learning

Abstract

Various examples are directed to systems and methods for performing computer- automated processes using trained machine learning (ML) models. A computing system can access first event data describing a first event. The computing system can execute a first ML model to determine a ML characterization of the first event using the first event data. The computing system can also apply a first set of rules to the first event data to generate a rule characterization of the first event. The computing system can determine an output characterization of the first event based at least in part on the rule characterization of the first event and determine to deactivate the first set of rules based at least in part on the ML characterization of the first event.

Description

Background Technology

[0001] Computing systems are used to automate various processes in order to reduce or eliminate the human labor required to perform these processes. Attached Figure Description

[0002] The disclosure is illustrated by way of example and not limitation in the following figures.

[0003] Figure 1 This is a diagram illustrating an example of a computing system environment used to perform computer-automated processes using one or more machine learning (ML) models supplemented by one or more sets of rules.

[0004] Figure 2 This is a diagram illustrating an example of a workflow for performing a computer automation process using an ML model supplemented by one or more sets of rules.

[0005] Figure 3 This is a flowchart illustrating an example of a processing flow that can be executed by a computing system to apply one or more proactive rules to an event characterization request.

[0006] Figure 4 This is a flowchart illustrating an example of a processing flow that can be executed by a computing system to apply one or more reactive rules.

[0007] Figure 5 This is a flowchart illustrating an example of a processing flow that can be executed by a computing system to apply one or more complementary rules to an event characterization request.

[0008] Figure 6 This is a diagram illustrating an example of the processing flow.

[0009] Figure 7 This illustrates methods for determining when a trained ML model is properly trained to handle untrained events. Figure 5 The flowchart shows the alternative layout of a portion of the processing flow.

[0010] Figure 8 It shows the use Figure 7 The alternative operation shown Figure 5 A diagram illustrating an example of the processing flow.

[0011] Figure 9 This is a diagram illustrating an example of a workflow that can be used to supplement ML characterization by simultaneously employing both pre-emptive and reactive rules.

[0012] Figure 10This is a diagram illustrating an example of an in-memory database management system that can be used to implement, for example, table permission management as described in this article.

[0013] Figure 11 It is shown Figure 10 A diagram illustrating an example of an index server in an in-memory database management system.

[0014] Figure 12 It is shown Figure 10 A diagram illustrating an example of request processing and execution control in a database management system.

[0015] Figure 13 This is a block diagram illustrating an example of the software architecture of a computing device.

[0016] Figure 14 It is a block diagram of a machine in the form of an example computer system, within which instructions can be executed to cause the machine to perform any one or more of the methods discussed herein. Detailed Implementation

[0017] The following description includes illustrative systems, methods, techniques, instruction sequences, and computer program products for implementing illustrative embodiments. In this description, numerous specific details are set forth for purposes of explanation in order to provide an understanding of various embodiments of the subject matter of the invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the invention can be practiced without these specific details. Generally, well-known examples of instructions, protocols, structures, and techniques are not shown in detail.

[0018] Computer-automated processes can leverage machine learning (ML) models to improve the accuracy of event representation in a way that reduces (or in some cases, eliminates) the need for human intervention. The ML model is trained to represent incoming events. For example, the ML model can classify or score incoming events. The classification and / or scores are then used to select the next action in the computer-automated process.

[0019] Consider an example of performing ML-based computer automation in an enterprise resource planning (ERP) computing system to support the selection of candidates for open job positions. Each candidate for a position represents an event represented using an ML model. The ML model is trained to characterize candidates using training data, such as data describing candidates for previous positions and their determined desirability or suitability. The candidate representations generated by the ML model can be used to determine which candidates should proceed to the next stage of consideration, such as an interview. The ML model can characterize candidates by generating classifications (e.g., continue or not continue) and / or by generating scores for candidates in one or more categories.

[0020] Consider another example of implementing ML-based computer automation in an information technology (IT) computing system to manage service requests directed to the IT help desk. Service requests can represent events. A trained ML model characterizes an incoming service request, for example, by assigning a corresponding next action to it. The next action could include, for example, modifying software or hardware settings, ordering a piece of hardware, sending an automation command to a user, etc. In some examples, the ML model can be trained using training data derived from previous IT help desk service requests.

[0021] Consider another example of using an ML model to estimate the runtime of a software code segment. The event can be the execution of the software code segment or the execution of a planned event. The trained ML model characterizes the execution of the software code segment or the execution of a planned event by assigning estimated runtimes to the execution or the planned event. Next steps may include, for example, modifying the operation of the system, allocating processing resources (towards or away from the execution of the software code), or other actions.

[0022] The use of ML models in computer automation processes offers significant advantages by generating accurate representations of events, allowing the automation process to continue, in whole or in part, with reduced or even eliminated human intervention. For example, a computer automation process for candidate screening could automatically select candidates for interviews or other more detailed reviews. Furthermore, a computer automation process for handling IT help desk service requests could resolve all or most of the service requests without human help desk expert intervention.

[0023] Beyond the advantages of using ML models in computer automation processes, several challenges exist. For example, an ML model is only as good as the training data used to train it. As a result, computer automation processes using ML models can be vulnerable when the ML model encounters untrained events (referred to as untrained events in this paper) outside the problem space covered by the training data. Untrained events can occur when an ML model is presented with events that are not described (or not adequately described) by the training data on which the ML model was trained. Because the ML model was not trained to handle untrained events, it either fails to generate representations or generates incorrect representations. If no representations are generated (or not generated with sufficient confidence), the computer automation process may trigger manual intervention from a human user, meaning the process is no longer automated. If incorrect representations are generated, the ML model may trigger incorrect next actions, which could lead to incorrect results and also require human intervention.

[0024] Untrained events can also occur if an ML model is expected to represent a type of event in a way that differs from how the events are represented by the training data used by the ML model. In this example, the event, or type of event, becomes an untrained event when the expected representation changes. For instance, returning to the candidate / resume example, if an open position is in a new domain or new geographic location that is not reflected in previous training data, the ML model might be trained with historical candidate data that incorrectly represents candidates in the new domain or new geographic location. Incorrect representations may require human intervention to detect and correct.

[0025] When encountering untrained events during ML-based computer automation, the ML model can be retrained using new training data reflecting the untrained events and their appropriate representations to correctly handle them. However, retraining takes time to accumulate new training data reflecting the untrained events and to retrain the ML model to correctly represent them. Manual intervention may be required to correctly represent the untrained events before retraining is complete. For example, a human user might detect the untrained events and manually set the output to the desired representation. Manual intervention may also be needed to determine when the ML model is appropriately retrained for untrained events or the type of untrained event, making further manual intervention unnecessary. This can lead to increased use of expensive human resources, longer process times, and reduced service level agreement (SLA) compliance.

[0026] The various examples described herein address these and other problems by executing a computer-automated process that supplements an ML model with one or more rule sets. Upon receiving an incoming event, the computational system executing the computer-automated process executes the ML model to generate an ML representation of the incoming event. The computational system also executes a detection rule set that includes rules for identifying one or more untrained events. If the result of the detection rule set indicates that the incoming event is not an untrained event (e.g., the ML model has been appropriately trained to represent the incoming event), the computational system returns the ML representation of the event as an output representation and executes subsequent steps of the computer-automated process based on the ML representation.

[0027] If the result of the detection rule set indicates that the event is an untrained event, the computational system applies one or more event representation rules associated with the untrained event to generate a rule representation. The rule representation can be returned as the output representation of the input event, and subsequent steps of the computer-automated process can be performed based on the rule representation. In other examples described herein, rule representations and ML representations are combined to generate a supplementary representation of the untrained event, and the supplementary representation is used to perform subsequent steps of the computer-automated process.

[0028] In various examples, the ML representation of untrained events is used to determine when an ML model has been properly retrained to correctly represent untrained events. For example, untrained events received by an ML model can be used as new training data for retraining the ML model to correctly represent future untrained events of the same type. The ML representation of untrained events can be compared to untrained event criteria. If the ML representation meets the untrained event criteria, it can indicate that the ML model has been adequately trained to represent untrained events and similar types of events. When this occurs, rules used to detect and / or represent previous untrained events can be deactivated. For example, when a rule used to detect and / or represent previous untrained events is deactivated, that rule may not be applied to potential untrained events in the future.

[0029] Figure 1 This is a diagram illustrating an example of a computing system environment 100 for performing a computerized automation process using one or more ML models 111A, 111B supplemented by one or more rule sets. The computing system environment 100 includes various components, including an automation process engine 102, an untrained event coordinator 104, an ML engine 106, and a rule engine 108. Components 102, 104, 106, and 108 can be implemented using one or more computing devices (such as, for example, server computing devices, user computing devices, etc.). In some examples, some or all of components 102, 104, 106, and 108 are used as a database management system (such as those referenced herein). Figures 10-13 It is executed as part of the in-memory database management system 1000.

[0030] Automation process engine 102 executes computer-automated processes. For example, automation process engine 102 may be or include an ERP computing system or other computing system that implements a candidate review process. In some examples, as described herein, automation process engine 102 is or includes an IT computing system for implementing an IT help desk process. In other examples, automation process engine 102 may execute other computer-automated processes.

[0031] The automation process engine 102 is configured to access incoming events and request event representations. Accessing incoming events may include receiving data describing the incoming event from another system, retrieving data describing the incoming event from the automation process engine 102's data storage, etc. Upon receiving an event representation, the automation process engine 102 uses the received representation to select the next step in the computer automation process. For example, when the automation process engine 102 performs a candidate screening process, it may receive a candidate's resume. The automation process engine 102 may request a representation of the resume and then generate a recommended next action for the candidate. When the automation process engine 102 performs an IT help desk process, it may receive events including service request messages. The automation process engine 102 may request a representation of the service request message and perform the next action based on that representation.

[0032] ML engine 106 executes one or more ML models 111A, 111B. ML engine 106 may receive one or more representation requests. A representation request includes a set of features describing an incoming event. The set of features describing an event may be referred to as a feature set or feature array. For example, the feature set for a candidate resume event may include information from the resume and / or other information about the candidate. The feature set for a help desk service request event may include details contained in the service request message, and so on. Upon receiving a representation request, the ML engine executes one or more ML models 111A, 111B to generate an ML representation of the incoming event. The ML representation is returned as an output representation to automation process engine 102.

[0033] In some examples, ML engine 106 is also configured to train and / or retrain ML models 111A, 111B. As described herein, ML engine 106 may receive untrained event data, for example, from untrained event coordinator 104. The untrained event data may include a set of features describing the desired representation of the untrained events. ML engine 106 applies the untrained event data as new training data to retrain one or more of ML models 111A, 111B. With the receipt of additional untrained event data, ML models 111A, 111B can be better trained to represent the untrained events until, as described herein, it is determined that the previously untrained incoming events can be accurately represented by ML model 111A or 111B.

[0034] In some examples, ML model 111A is the deployed ML model, and ML model 111B is the training ML model. ML engine 106 uses the deployed ML model 111A to generate ML representations of events that are not untrained events. The training ML model 111B can be retrained with new training data describing untrained events. In some examples, ML engine 106 uses the training ML model 111B to generate ML representations of untrained events, for example, to determine the progress of the training ML model 111B towards being trained to accurately represent untrained events. When a new version of ML model 111B is properly trained to handle untrained events, the ML model 111B can be deployed as described herein.

[0035] The ML models 111A and 111B executed by the ML engine 106 can be or include any suitable ML model trained to represent the incoming events, including supervised and unsupervised learning models. In some examples, the ML models 111A and 111B can be or include classification models such as logistic regression models, Naive Bayes models, K-nearest neighbor models, decision tree models, random forest models, support vector machine (SVM) models, etc.

[0036] Rule engine 108 is configured to apply one or more rules 109. Rule engine 108 may apply one or more detection rules and / or one or more representation rules. Detection rules are applied to determine whether the event indicated by the event representation request is an untrained event. Representation rules are applied to a set of features describing the incoming event to generate a representation of the incoming event.

[0037] In some examples, rule engine 108 stores one or more rules 109 at rule repository 110. In some examples, rule engine 108 executes at a database management system, such as using the methods described herein. Figures 10-13 The in-memory database management system is described below. Therefore, rule store 110 can be implemented at the database level. In some examples, rule 109 can be developed by one or more rule developer users and deployed to rule engine 108 (e.g., for use and storage at rule store 110).

[0038] As described herein, rule engine 108 can also be configured to selectively activate or deactivate detection rules and / or representation rules, for example, in response to instructions from untrained event coordinator 104. For instance, when training or retraining the ML model 111B to accurately represent untrained events, rule engine 108 can be instructed to deactivate the corresponding detection rules and / or representation rules. Therefore, applying detection rules to subsequent incoming events received from automated process engine 102 can indicate that the incoming event is not untrained, and thus, the ML representation of the event can be returned to automated process engine 102 as an output representation.

[0039] In some examples, rule engine 108 is configured to manage detection rules and / or characterization rules for more than one type of untrained event. Detection rules can then include multiple sets of one or more rules, each set of rules used to detect different types of untrained events. Consider again the example of automation process engine 102 performing an IT help desk service. One type of untrained event might correspond to a new problem of a certain type in a first software application, while another type of untrained event might correspond to a different type of new problem (e.g., in a second software application). Yet another type of untrained event might correspond to a new problem in a common hardware component, and so on. In this example, detection rules could include a set of one or more rules developed to detect new problems in the first software application, a set of one or more rules developed to detect new problems in the second software application, a set of one or more rules developed to detect new problems in a common hardware component, and so on.

[0040] The sets of detection rules for different types of untrained events can be activated or deactivated separately. For example, ML models 111A or 111B can be appropriately retrained to represent the first type of untrained events, but not the second type. In this example, the set of detection rules for detecting the first type of untrained events can be deactivated, while the set of rules for detecting the second type of untrained events can remain active.

[0041] In some examples, Rule Engine 108 also applies sets of representation rules specific to different untrained event types. Returning to the IT Helpdesk service example above, it develops one or more sets of representation rules to represent a new problem in a first software application, one or more sets of representation rules to represent a new problem in a second software application, one or more sets of representation rules to represent a new problem in a common hardware component, and so on.

[0042] In some examples, detection and / or representation rules may be common across different untrained event types. For instance, a sample detection rule could be applied to detect untrained events of both type 1 and type 2. Furthermore, representation rules may exist that are applied to represent events of more than one type. When common representation or detection rules exist between different untrained event types, disabling such a common rule for one event type may not prevent that common rule from being applied to untrained events of different types.

[0043] Figure 1 Environment 100 also includes an untrained event coordinator system 104. Figure 1 In the example environment 100, the automation process engine 102 is configured to direct calls to event representations to an untrained event coordinator 104. The untrained event coordinator 104 manages the ML engine 106 and the rules engine 108 to generate event representations using ML models 111A, 111B of the ML engine 106, supplemented by rules 109 of the rules engine 108, as described herein. The resulting event representations are returned as output representations to the automation process engine 102. The automation process engine 102 uses the output representations to select and / or execute the next action of the automation process.

[0044] In some examples, the untrained event coordinator 104 is configured to store outcome data, such as untrained event ML representations and untrained event rule representations, at outcome store 112. The outcome data can also be provided to the ML engine 106 for use as training data.

[0045] Upon receiving an event representation request from the automated process engine 102, the untrained event coordinator 104 sends a request to the rule engine 108, requesting the rule engine 108 to execute a set of one or more detection rules to determine whether the first event is an untrained event. The request to the rule engine 108 may include some or all of the feature set describing the incoming event.

[0046] The untrained event coordinator 104 can also send a request for a corresponding event representation for an incoming event to the ML engine 106. The ML engine 106 executes one or more of ML models 111A and 111B to generate an ML representation of the first event and returns it to the untrained event coordinator 104. In some examples, the ML model executed to determine the ML representation depends on whether the incoming event is an untrained event. If the incoming event is not an untrained event, the untrained event coordinator 104 can request the ML engine 106 to use the deployed ML model 111A to generate the ML representation. If the incoming event is an untrained event, the untrained event coordinator 104 can request the ML engine 106 to use the trained ML model 111B to generate the ML representation.

[0047] If the application of the detection rule indicates that the incoming event is not an untrained event, the untrained event coordinator 104 returns the ML representation as the output representation of the incoming event to the automation process engine 102. If the application of the detection rule indicates that the incoming event is an untrained event, the untrained event coordinator 104 requests the rule engine 108 to apply representation rules to generate a rule representation of the incoming event. The rule representation can be returned to the automation process engine 102 as the output representation of the untrained event. In some examples, both the rule representation and the ML representation are used to determine supplementary representations that are returned to the automation process engine 102 as output representations.

[0048] The untrained event coordinator 104 can also be programmed to determine when ML models 111A and 111B are appropriately trained to handle untrained events without relying on rule representations. For example, when an untrained event is detected, representation rules are used to determine the representation of the incoming event provided to the automation process engine 102. However, data from the incoming events can be stored (e.g., at the results repository 112) and used as training data for training ML model 111B. As the untrained event coordinator 104 receives and processes more untrained events, more training data is generated, allowing ML model 111B to be trained in a way that allows the ML model to become increasingly accurate in representing untrained events.

[0049] The untrained event coordinator 104 is programmed to determine whether the trained ML model 111B has been properly trained to represent untrained events using the ML representations generated for detected untrained events. When the trained ML model 111B has been properly trained to represent untrained events, the untrained event coordinator 104 can deploy the trained model 111B as a newly deployed model 111A and deactivate some or all of the untrained event detection rules associated with the untrained events. In this way, incoming events that can be properly represented by the ML engine 106 are no longer detected as untrained events. Therefore, the ML representation of such events can be returned to the automation process engine 102 as the output representation of the incoming events.

[0050] exist Figure 1In the example, the untrained event coordinator 104 includes components 114, 116, and 118 for processing three types of rules for handling untrained events. Preemptive rules are considered at the reactive rule component 114. Reactive rules are processed at the reactive rule component 116. Supplementary rules are processed by the supplementary rule component 118. The untrained event coordinator 104 can be configured to implement any combination of the three rule components 114, 116, and 118, individually or in combination. The different rule types differ in, for example, how and when detection rules are applied and, for example, how rule representations are determined.

[0051] The responsive rule component 114 uses responsive detection rules to identify various types of untrained events. When an incoming event characterization event is received from the automation process engine 102, responsive detection rules are applied to a set of features describing the incoming event to determine whether the incoming event is an untrained event. Consider again the example of the automation process engine 102 implementing a candidate review process. Example responsive detection rules might ask whether a resume event points to an open position in a new field, whether a candidate possesses a listed skillset in a new field, whether a resume event points to an open position in a new geographic location, etc. Also consider an example of the automation process engine 102 implementing an IT help desk process. Example responsive detection rules might ask an incoming service request whether it identifies a known new problem and / or symptoms associated with a known new problem. In some examples, responsive detection rules may include rules or sets of rules for detecting different types of untrained events, as described herein.

[0052] The reactive rule component 114 may include a reactive rule controller 120 that manages various aspects of the reactive rule processing flow. For example, when the untrained event coordinator 104 receives a call for an event representation, the reactive rule controller 120 may provide the rule engine 108 with a feature set of the incoming event, along with a request to apply the currently active reactive rule set. If the currently active reactive rule set indicates that the incoming event is an untrained event, the reactive rule controller 120 may send an additional request to the rule engine 108 to apply the representation rule set to the feature set of the incoming event to generate a rule representation of the incoming event. The reactive rule controller 120 may return the rule representation as an output representation of the incoming event to the automation process engine 102.

[0053] The responsive rule controller 120 can also be configured to determine whether the trained ML model 111B has been properly trained to handle future instances of incoming untrained events (or similar events). The responsive rule controller 120 obtains ML representations of incoming untrained events generated using the trained ML model 111B. The responsive rule controller 120 determines whether the ML representations of the incoming rule events meet the untrained event criteria. The untrained event criteria can be the accuracy level of the ML representations and can be applied to a single untrained event and / or a set of multiple untrained events. For example, if ML models 111A and 111B are classification models, the untrained event criteria can be met if the trained ML model 111B has correctly classified the untrained event type in more than X% of previous instances in which the trained ML model 111B has been asked to represent the untrained event type. In another example, ML models 111A and 111B can provide confidence scores indicating the accuracy of the ML representations. In some examples, if the confidence score of the ML representation generated by training ML model 111B is higher than a threshold level, the untrained event criterion is met. When the untrained event criterion for an untrained event or an untrained event type is met, the responsive rule controller 120 instructs the rule engine 108 to deactivate the responsive detection rule associated with the untrained event type.

[0054] Reactive rule component 116 uses reactive rules to identify untrained events. Reactive rules are applied using ML representations generated by a deployed ML model 111A. For example, as described herein, ML model 111A can be configured to generate confidence scores indicating the accuracy of the ML representations. If the confidence score of the ML representation is below a threshold level, the untrained event coordinator 104 can be configured not to return the ML representation to the automation process engine 102. Instead, the untrained event coordinator 104 can apply one or more reactive detection rules to the ML representation and / or feature set of the corresponding incoming event. If the set of untrained reactive detection rules indicates that the incoming event is an untrained event, the untrained event coordinator 104 can apply one or more representation rules to generate a rule representation of the event. The rule representation can be returned to the automation process engine 102 as an output representation of the incoming event.

[0055] The reactive rule component 116 may include a reactive rule controller 122 that manages various aspects of the reactive rule processing flow. For example, upon receiving an indication that the ML representation of an incoming event has a confidence score below a threshold, the reactive rule controller 122 may request the rule engine 108 to apply a set of reactive detection rules to the incoming event. If the application of the reactive detection rule set indicates that the incoming event is an untrained event, the reactive rule component 116 may request the rule engine 108 to apply one or more representation rules to generate a rule representation of the incoming event. The rule representation may be provided to the automated process engine 102 as a representation of the incoming event.

[0056] The reactive rule controller 122 can also be configured to determine whether the deployed ML model 111A has been properly trained to handle future instances of incoming untrained events (or similar events). For example, with reactive rules, the training model 111B can be deployed iteratively as it is trained on available training data. The reactive rule controller 122 can, for example, compare the number of incoming events with ML representations below a threshold level to the number of incoming events of a given type that are identified as untrained events by the reactive detection rule. If the percentage of untrained events of a given type detected is below the threshold level, the reactive rule controller 122 can determine that the deployed ML model 111A has been properly trained to handle untrained events, and the reactive detection rule for untrained events can be deactivated.

[0057] The supplementary rule component 118 uses rule representations and ML representations, employing supplementary rules to represent untrained events. For example, the supplementary rule component 118 can be used to include ML representations of values ​​(such as scores). One or more supplementary detection rules can be applied to the ML representation and / or feature set of an incoming event to detect untrained events. When an untrained event is detected, one or more representation rules can be applied based on the event's ML representation and the event-based rule representation to generate a representation of the incoming event.

[0058] Consider the candidate resume example above again. The ML representation of a candidate resume event can include experience scores, which are numerical indicators of a candidate's experience relevant to the open position. It can be determined that the experience scores generated by ML models 111A and 111B underweight Type I experience. Supplementation detection rules can identify incoming resume events that indicate candidates with Type I experience. Representation rules can be applied to determine the amount of supplementation based on Type I experience. Supplementation representations of incoming events are determined by combining the supplementation amount and experience scores (e.g., using addition, multiplication, cascading, etc.).

[0059] Consider another example of an automated process based on estimated runtime of a software code segment. In this example, the incoming event could be a call to software code. The ML representation can be or include the estimated runtime of the called code. In this example, a supplementary detection rule can identify calls to software code with new features indicating that the event is not trained, such as, for example, the call is being made by a new component, or the call includes input parameters for which ML models 111A and 111B are not adequately trained to model. Supplementary representation rules can be applied to determine the supplementary runtime of the software code, which can be negative or positive. The supplementary representation of an incoming event can be determined by combining the supplementary amount with the estimated runtime indicated by the ML representation. The automated process engine 102 can use the supplementary representation, for example, to determine when to initialize or initiate one or more subsequent steps of the computer-automated process.

[0060] The supplementary rule component 118 may include a supplementary rule controller 124 that manages various aspects of the proactive rule processing flow. For example, the supplementary rule controller 124 may be provided with an ML representation of an incoming event generated by the ML engine 106 and / or a corresponding feature set of the incoming event. The supplementary rule controller 124 may request the rule engine 108 to apply one or more supplementary detection rules to the ML representation and / or feature set. If an untrained event is detected, the supplementary rule controller 124 may request the rule engine 108 to apply one or more supplementary representation rules to generate supplementary values ​​for the incoming event. The supplementary values ​​for the incoming event may be combined with the ML representation to generate a supplementary representation, which is returned to the automated process engine 102 as an output representation of the incoming event.

[0061] The supplementary rule controller 124 can also be configured to determine whether the trained ML model 111B has been properly trained to handle future instances of incoming untrained events (or similar events). In an example using the supplementary rule component, the supplementary representation can be determined using the ML representation generated by the deployed ML model 111A. The supplementary rule controller 124 can use the second ML representation generated by the trained ML model 111B to determine whether the trained ML model 111B has been properly trained to represent untrained events. For example, if the difference between the supplementary representation of the incoming event and the second ML representation generated by the trained ML model 111B is less than a threshold, the supplementary rule controller 124 can determine that the trained ML model 111B has been properly trained for untrained events. The supplementary rule controller 124 can instruct the ML engine 106 to deploy the trained ML model 111B, and can also instruct the rule engine 108 to disable a set of one or more supplementary detection rules corresponding to untrained events (or similar untrained events).

[0062] Figure 2This is a diagram illustrating an example of a workflow 200 for performing a computer automation process using an ML model supplemented by one or more rule sets. Workflow 200 includes a horizontal axis indicating time and three channels indicating the behavior of a computing system, such as computing system environment 100. The "Rules in Use" channel indicates when a set of rules is being used for untrained events or untrained event types. The "Deployed Model" channel indicates when a model is deployed to characterize an event and which model is deployed to characterize the event. The "Model Training in Progress" channel indicates when the model is being retrained and which model is being retrained.

[0063] Initially in workflow 200, ML model V1 is the deployed model, and no untrained events are active. At time T1, an untrained event or untrained event type becomes active. For example, a new type of event may begin to be rendered. Furthermore, in some examples, it may become desirable to change how previously trained events are represented. When a new event becomes active at time T1, untrained event detection rules can be in place, as described herein, to detect the new event, and untrained event representation rules can be in place to generate rule representations of the event. When an incoming event is identified as a new event using the untrained event detection rules, the computational system can return a representation as or based on the rule representation.

[0064] At time T2, the computational system begins training the training model V2α. Training can be performed using training data derived from untrained events received between times T1 and T2. At time T3, the computational system may optionally begin training a second training model V2β. For example, training model V2β can be based on training model V2α and can be trained using training data from untrained events received between times T1 and T3. The final version V2 of the training model can be trained starting at time T4 using training data from untrained events received between times T1 and T4. When the computational system determines that training model V2 has been properly trained to represent untrained events, at time T5, model V2 is deployed, and untrained events are deactivated, for example, by removing the set of untrained event detection rules corresponding to the untrained events.

[0065] Figure 3 This is a flowchart illustrating an example of a processing flow 300, as described herein, that can be executed by a computing system to apply one or more pre-defined rules. For example, processing flow 300 can be... Figure 1 The proactive rule component 114 and / or the proactive rule controller 120 execute in response to an event representation request from the automation process engine 102.

[0066] At operation 302, the computing system receives information from computer-automated processes (such as from...). Figure 1 The automated process engine 102 receives an event representation request. The representation request includes a feature set of the incoming event. At operation 304, the computational system applies an ML model to generate an ML representation of the incoming event. At operation 306, the computational system applies one or more pre-emptive detection rules to determine whether the incoming event is an untrained event. As described herein, pre-emptive detection rules can be applied to the feature set describing the incoming event to detect whether the incoming event is an untrained event.

[0067] If the application of one or more pre-defined detection rules at operation 306 indicates that the incoming event is not an untrained event, the computation system returns the ML representation determined at operation 304 as the output representation of the incoming event at operation 308, and waits for the next representation request at operation 314.

[0068] If one or more pre-defined detection rules applied at operation 306 indicate that the incoming event is an untrained event, the computational system generates a rule representation of the incoming event at operation 310. The rule representation is generated by applying one or more representation rules to, for example, a set of features describing the incoming event. At operation 312, the computational system returns the rule representation determined at operation 310 as an output representation and waits at operation 314 for the next event representation request to be received.

[0069] While performing operations 302, 304, 306, 308, 310, 312, and 314, the computational system can periodically execute operation 316 to determine whether the ML model has been trained to accurately represent unknown events. In some examples, as part of processing flow 300, operation 316 is executed for each untrained event processed. In other examples, as part of processing flow 300, operation 316 is executed after processing a certain number of untrained events. In some examples, as part of processing flow 300, operation 316 is executed after processing a certain number of untrained events of a specific type. In some examples, operation 316 is executed periodically (e.g., every hour, every ten minutes, etc.).

[0070] At operation 318, the computational system determines whether one or more ML representations of untrained events are within a threshold of the expected representations for those events. The untrained events considered at operation 318 can be all untrained events or all untrained events of a specific type. For example, the expected representations can be rule representations. Comparisons can be made on a single untrained event and / or multiple untrained events of a common type. If the ML representation is not within the threshold of the expected representations, the computational system can, for example, re-execute operation 318 on the next execution of operation 316. If the ML representation is within the threshold of the expected representations, then at operation 320, the computational system can deactivate one or more rules associated with the untrained event type. Deactivation can include deactivating a set of one or more pre-emptive detection rules associated with the untrained event type and / or deactivating one or more representation rules associated with the untrained event type.

[0071] Figure 4 This is a flowchart illustrating an example of a processing flow 400, as described herein, that can be executed by a computing system to apply one or more reactive rules to an event representation request. For example, processing flow 400 could be... Figure 1 The reactive rule component 116 and / or reactive rule controller 122 execute in response to an event representation request from the automated process engine 102.

[0072] At operation 402, the computing system receives information from computer-automated processes (such as from...). Figure 1 The automated process engine 102 receives an event representation request. The representation request includes a set of features for the incoming event. At operation 404, the computational system applies an ML model to generate an ML representation of the incoming event. At operation 406, the computational system determines whether the accuracy of the ML representation is greater than a threshold. In some examples, the accuracy of the ML representation is based on a confidence score generated by the ML model. If the confidence score generated by the ML model is greater than a confidence threshold, the accuracy of the ML representation is sufficient. If the accuracy of the ML representation is greater than the threshold, the ML representation is returned as an output representation at operation 408, and the computational system waits for the next event representation request at operation 418.

[0073] If the accuracy of the ML representation fails to meet the threshold at operation 406, then at operation 410, the computational system applies one or more reactive detection rules to determine whether the incoming event is an untrained event. If the reactive detection rule does not indicate that the incoming event is an untrained event, the computational system executes a manual exception handling routine at operation 414 to handle the representation request. For example, if the accuracy of the ML representation is insufficient at operation 406, and the reactive detection rule does not identify the incoming event as an untrained event at operation 410, a manual intervention can be initiated to determine an appropriate representation for the event.

[0074] If the reactive detection rule does indeed indicate that the incoming event is an untrained event, then at operation 412, the computational system can apply one or more representation rules to generate a rule representation of the incoming event. At operation 416, the rule representation can be returned as an output representation, and at operation 418, the computational system can wait for the next event representation request.

[0075] While performing operations 402, 404, 406, 408, 410, 412, 414, 416, and 418, the computational system can periodically execute operation 416 to determine whether the ML model has been trained to accurately represent unknown events. In some examples, as part of processing flow 400, operation 420 is executed for each untrained event processed. In other examples, as part of processing flow 400, operation 420 is executed after processing a certain number of untrained events. In some examples, as part of processing flow 400, operation 420 is executed after processing a certain number of untrained events of a specific type. In some examples, operation 420 is executed periodically (e.g., every hour, every ten minutes, etc.).

[0076] At operation 422, the computational system determines whether the ratio of manually generated anomalous events to the untrained events detected at operation 410 is greater than a threshold. This action can be performed relative to all untrained events or relative to all untrained events of a specific type. Furthermore, in some examples, operation 422 is performed relative to untrained events received within a given time period. If the ratio of manually generated anomalous events to untrained events is greater than a specific threshold, it can indicate that fewer untrained events are occurring, and the ML model is being retrained to handle untrained events (or related types of untrained events). Therefore, at operation 424, the computational system can deactivate the set of rules used to determine the untrained events or event types considered at operation 422. If the ratio of manually generated anomalous events to untrained events is not greater than the threshold, no change can be made, and operation 422 can be re-executed, for example, after a predetermined time, a predetermined number of events, etc.

[0077] Figure 5 This is a flowchart illustrating an example of a processing flow 500, as described herein, that can be executed by a computing system to apply one or more supplementary rules to an event representation request. For example, processing flow 500 could be... Figure 1 The supplementary rule component 118 and / or supplementary rule controller 124 are used to execute this.

[0078] At operation 502, the computing system receives information from computer-automated processes (such as from...). Figure 1 The automated process engine 102 receives an event representation request. The representation request includes a feature set of the incoming event. At operation 504, the computational system applies an ML model to generate an ML representation of the incoming event. At operation 506, the computational system applies one or more supplementary detection rules to detect whether the event is an untrained event. If the supplementary detection rules do not indicate an untrained event, at operation 508, the computational system returns the ML representation generated at operation 504 as an output representation and waits for the next event representation request at operation 516.

[0079] If the application of the supplementary detection rule indicates that the incoming event is an untrained event, then at operation 510, the computational system applies one or more supplementary representation rules to the feature set of the incoming event to generate supplementary values ​​for the incoming event. The supplementary values ​​are combined with all or part of the ML representation of the incoming event to generate a supplementary representation of the incoming event. Combinations may include adding the supplementary values ​​and the ML representation, multiplying the supplementary values ​​and the ML representation, concatenating the supplementary values ​​and the ML representation, etc. At operation 512, the supplementary ML representation is returned as the output representation.

[0080] At operation 514, the computational system uses a trained ML model (such as a second version of ML model 111B) to generate an updated ML representation of the incoming event. In this example, as described herein, the ML model used at operation 504 may not be updated with new training data. As described herein, the ML model applied at operation 514 to generate the updated ML representation can be updated with new training data generated from untrained events. As described herein, the updated ML representation can be stored for use, and at operation 516, the computational system can wait for the next event representation request.

[0081] While performing operations 502, 504, 506, 508, 510, 512, 514, and 516, the computing system can periodically execute operation 518 to determine whether the trained ML model has been trained to accurately represent unknown events. In some examples, as part of processing flow 500, operation 518 is executed for each untrained event processed. In other examples, as part of processing flow 500, operation 518 is executed after processing a certain number of untrained events. In some examples, as part of processing flow 500, operation 518 is executed after processing a certain number of untrained events of a specific type. In some examples, operation 518 is executed periodically (e.g., every hour, every ten minutes, etc.).

[0082] At operation 520, the computational system determines the difference or delta between the updated ML representation generated by the trained ML model and the expected representation of the considered event or event type. For example, the expected representation could be a supplementary representation. The computational system determines whether the delta is less than a threshold. The comparison at operation 520 can be on all untrained events or on untrained events of a common type. If the delta is not less than the threshold, operation 520 can be re-evaluated, for example, scheduled or otherwise expected to be executed next time. If the delta at operation 520 is less than the threshold, it indicates that the trained ML model has been appropriately trained to handle untrained events (e.g., untrained events of the type considered at operation 520). Therefore, at operation 522, the computational system can deactivate one or more supplementary detection rules used to identify untrained events of the type considered at operation 520 and / or untrained events.

[0083] Figure 6 Figure 600 illustrates an example of the operation of processing flow 500. Figure 600 includes a horizontal axis indicating time and a vertical axis indicating the representation values ​​of events. For example, values ​​could be experience scores, the runtime of a software segment, etc. ML representation line 602 indicates the ML representation of an incoming untrained event. Representation value 604 indicates the desired representation value for the untrained event. To achieve the desired representation value, the computing system applies one or more supplementary representation rules to determine a supplementary value 608. In this example, supplementary value 608 is added to the ML representation to generate a supplementary representation at the desired representation value 604.

[0084] Line 606 represents the updated ML representation generated by the updated model. The updated ML representation starts at the level of ML representation value 602 generated by the ML model. As the trained ML model is retrained over time, the updated ML representation begins to approach the desired representation value 604. When (e.g., for untrained event types) the increment between the updated ML representation and the desired representation value 604 is less than a threshold, the computation system deactivates the corresponding supplementary detection rule and deploys the trained ML model as a new ML model.

[0085] Figure 7 This is a flowchart illustrating an alternative arrangement for operation 418, used to determine when a trained ML model is appropriately trained to handle untrained events. At operation 702, the computational system determines whether the difference between the ML representation and the desired representation (e.g., indicated by supplementary representations) is less than a first threshold. If the difference or the difference is less than the first threshold, then at operation 704, the computational system may deactivate one or more supplementary detection rules for the untrained event or event type considered at operation 702 and deploy the updated model.

[0086] If the increment at operation 702 is not less than a first threshold, then at operation 706, the computation system can determine whether the increment is less than a second threshold. The second threshold can be greater than the first threshold. If the increment is not less than the second threshold, the computation system returns to operation 702, for example, on the next execution of operation 518. If the increment is less than the second threshold, then at operation 708, the computation system can deploy the trained ML model and modify the supplementary representation rules to return different (e.g., smaller) supplementary values. In some examples, the version of the trained ML model continues to be trained with new training data from untrained events, as a new trained ML model. On the next run of operation 518, the increment is determined relative to the currently deployed trained ML model. When the increment is less than the first threshold, the new updated ML model is deployed, and the relevant detection rules are deactivated.

[0087] Figure 8 It shows the use Figure 7 Figure 800 shows an example of the operation of the alternative operation processing flow 500. Figure 800 includes a horizontal axis indicating time and a vertical axis indicating the representation value of an event. For example, the value could be an experience score, the runtime of a software segment, etc. ML representation value 802 indicates the initial ML representation of the incoming untrained event. Desired representation value 804 indicates the desired representation value of the untrained event. To achieve the desired representation value, the computing system applies one or more supplementary representation rules to determine a first supplementary value 808A.

[0088] The updated ML representation value 806 represents the updated ML representation generated by the updated model. The updated ML representation value 806 starts at the level of the ML representation value 802 generated by the ML model. As the trained ML model is retrained over time, the updated ML representation begins to approach the desired representation value 804. When the increment between the updated ML representation and the desired representation value 804 is less than a second threshold, the trained ML model is deployed as indicated by the updated model 810. The supplementary representation rule is modified to generate a new supplementary value 808B, which is added to the output of the now deployed updated model 810 to generate a supplementary representation at the desired representation value 804. The new trained ML model 812 continues to be retrained and approaches the desired representation 804. When the increment between the new trained ML model 812 and the desired representation 804 is less than a first threshold, the relevant detection rule is deactivated, and the new trained ML model 812 is deployed.

[0089] Figure 9 This diagram illustrates an example of a workflow 900 that can be used to supplement ML representations with both proactive and reactive rules. The computing system receives an incoming event 902. Incoming event 902 can be described by a set of features. At operation 904, the computing system applies a set of one or more proactive detection rules. In the IT help desk example, the proactive detection rule evaluation at operation 904 may include scanning service requests for error codes associated with a new feature or component, and returning the information to the IT team responsible for that new feature or feature. Simultaneously, at operation 906, the currently deployed ML model can be invoked to generate an ML representation of the incoming event 902.

[0090] At operation 908, the computational system determines whether the application of the responsive detection rule at operation 904 indicates that the incoming event 902 is an untrained event. If the application of the responsive detection rule at operation 904 does indeed indicate that the incoming event 902 is an untrained event, then the processing of the incoming event 902 can be transferred to rule-based processing 920, which can apply one or more representation rules to generate a rule representation of the incoming event 902. The rule representation can be returned as an output representation.

[0091] If the pre-emptive detection rule applied at operation 908 does not indicate that the incoming event 902 is an untrained event, then at operation 912, the computation system determines the accuracy of the ML representation, for example, based on the confidence score generated from the ML representation. If the accuracy of the ML representation is higher than a threshold level, the processing of the incoming event 902 is transferred to standard processing 924, which can return the ML representation of the event as the output representation.

[0092] If the accuracy of the ML representation is not higher than a threshold level, a set of reactive detection rules is applied at operation 914. At operation 916, the computational system determines whether the reactive detection rules indicate that the incoming event 902 is an untrained event. If the reactive detection rules indicate that the incoming event 902 is an untrained event, the processing is transferred to rule-based processing 920, which determines a rule representation of the incoming event 902 and returns the rule representation of the incoming event 902 as an output representation. If the reactive detection rules indicate that the incoming event 902 is not an untrained event, the processing of the incoming event 902 can be transferred to manual exception handling 922, which may involve human intervention to characterize the incoming event 902. For example, transferring the incoming event 902 to manual exception handling 922 may include inputting the incoming event 902 into a manual processing queue for human user processing.

[0093] Workflow 900 also illustrates comparison operations 910 and 918. At comparison operation 910, the computational system compares the performance of the ML model and rule-based processing 920 for reactive untrained events, for example, as described herein with reference to operation 316. At comparison operation 916, the computational system compares the performance of the ML model relative to reactive detection rules, for example, as described herein with reference to operation 420. If it is determined at operation 910 or operation 918 that the trained ML model is appropriately trained to represent untrained events, the trained ML model can be deployed, and the untrained event detection rules can be modified so that they no longer indicate untrained events or types of untrained events under consideration.

[0094] Figure 10 This is a diagram illustrating an example of an in-memory database management system 1000. The in-memory database management system 1000 demonstrates what can be used to implement... Figure 1 This is an example arrangement of some or all of the various components of environment 100. In-memory databases primarily store data in main memory (such as random access memory). This differs from databases that primarily employ disk storage mechanisms. In some examples, the database management system 1000 may be or include examples of the HANA system from SAP SE in Walldorf, Germany. Although this paper describes various features of carbon footprint optimization in the context of in-memory databases, carbon footprint optimization can generally be performed on any suitable database management system.

[0095] The in-memory database management system 1000 can be coupled to one or more client applications 1002A, 1002B. In some examples, client applications 1002A, 1002B are used to provide computer automation processes to client users. Client applications 1002A, 1002B can communicate with the in-memory database management system 1000 through a variety of different protocols, including Structured Query Language (SQL), Multidimensional Expressions (MDX), Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST), and Hypertext Markup Language (HTML). For example, the representation request described herein can be made by one or more client applications 1002A, 1002B, where various components of the environment 100 are used to respond to the representation request implemented at the database management system 1000.

[0096] Figure 10 Studio 1004, which can be used to perform modeling by accessing the in-memory database management system 1000, is also shown. In some examples, Studio 1004 can allow complex analysis to be performed on data extracted not only from real-time event data and windows, but also from stored database information. In some examples, as described herein, carbon footprint data can be generated by the studio. In some examples, Studio 1004 is configured to implement the ML engine 106 and / or rule engine 108 described herein.

[0097] The in-memory database management system 1000 may include several different components, including an index server 1006, an XS engine 1008, a statistics server 1010, a preprocessor server 1012, and a name server 1014. These components may operate on a single computing device or may be distributed across multiple computing devices (e.g., separate servers). The index server 1006 contains the actual data and the engine for processing that data. The index server 1006 may also coordinate and utilize other servers.

[0098] Statistics server 1010 collects information about status, performance, and resource consumption from all other server components. Statistics server 1010 can be accessed from studio 1004 to obtain the status of various alarm monitors.

[0099] The preprocessor server 1012 is used to analyze text data and extract information on which text search capabilities are based.

[0100] Name server 1014 stores information about the database topology. This is used in distributed systems where database instances reside on different hosts. Name server 1014 knows where components are running and which data resides on which server. In the example embodiment, a separate enqueue server can operate as described above with reference to enqueue servers (specifically, with reference to creating and managing lightweight enqueue sessions).

[0101] Figure 11 This is a diagram illustrating an example of index server 1006. Specifically, it is described in more detail... Figure 10 The index server 1006 includes a connection and session management component 1100, which is responsible for creating and managing sessions and connections for database clients (e.g., client applications 1002A, 1002B). Once a session is established, the client can communicate with the database system using SQL statements. For each session, a set of session parameters 1102 can be maintained, such as auto-commit, current transaction isolation level, etc. Users (e.g., system administrators, developers) can be authenticated by the database system itself (e.g., by logging in with login information such as username and password using authentication component 1104), or authentication can be delegated to an external authentication provider such as a Lightweight Directory Access Protocol (LDAP) directory.

[0102] Client requests (such as contractual requests) can be analyzed and executed by a set of components aggregated into request processing and execution control 1106. SQL processor 1108 examines the syntax and semantics of client SQL statements and generates a logical execution plan. MDX is a language for querying and manipulating multidimensional data stored in an Online Analytical Processing (OLAP) cube. Thus, an MDX engine 1110 can be provided to allow the parsing and execution of MDX commands. Planning engine 1112 allows applications to perform basic planning operations at the database layer. One such operation is creating a new version of a dataset as a copy of an existing dataset, while applying filters and transformations.

[0103] Computation engine 1114 implements various SQL scripts and planning operations. Computation engine 1114 creates logical execution plans for computational models derived from SQL scripts, MDX, plans, and domain-specific models. Such logical execution plans may include, for example, decomposing the model into operations that can be processed in parallel. Data is stored in relational store 1116, which implements a relational database in main memory. Each SQL statement can be processed within the context of a transaction. New sessions are implicitly assigned to new transactions. Transaction manager 1118 coordinates database transactions, controls transaction isolation, and tracks running and ending transactions. When a transaction is committed or rolled back, transaction manager 1118 notifies the involved engines of such events so that the involved engines can perform the necessary actions. Transaction manager 1118 also works with persistence layer 1120 to implement atomic and durable transactions.

[0104] The authorization manager 1122 is invoked by other database system components to check whether a user has the specified permissions to perform the requested operation. The database system allows permissions to be granted to users or roles. Permissions grant the right to perform specified operations on specified objects.

[0105] The persistence layer 1120 ensures that the database recovers to its most recently committed state after a restart, and that transactions are either fully executed or fully rolled back. To achieve this efficiently, the persistence layer 1120 uses a combination of pre-written logs, shadow paging, and savepoints. The persistence layer 1120 also provides a page management interface 1124 for writing and reading data to and from the detached disk storage device 1126, and includes a logger 1128 for managing the transaction log. When data is written via the persistence interface, log entries can be written implicitly by the persistence layer 1120 or explicitly using the log interface.

[0106] Figure 12 This is a diagram illustrating an example of request processing and execution control 1106. The diagram depicts this in more detail. Figure 11The request processing and execution control is 1106. The SQL processor 1108 includes an SQL parser 1200 that parses SQL statements and generates a logical execution plan 1202, which is then passed to an SQL optimizer 1204. The SQL optimizer 1204 optimizes the logical execution plan 1202 and converts it into a physical execution plan 1206, which is then passed to an SQL executor 1208. The computation engine 1114 implements various SQL scripts and planning operations and includes a computation engine optimizer 1210 for optimizing operations, a computation engine executor 1212 for executing operations, and an inherent computation engine arithmetic unit 1214, an L arithmetic unit 1216, and an R arithmetic unit 1218.

[0107] The L infrastructure 1220 includes several components that assist in the operation of the L process, including the L runtime (system mode) 1222, the L compiler 1224, and the L runtime (user mode) 1226.

[0108] In view of the above-described implementation of the subject matter, this application discloses the following list of examples, wherein a feature of a single example, a combination of more than one feature of the example, and optionally a combination with one or more features of one or more other examples are other examples that also fall within the scope of disclosure of this application:

[0109] Example:

[0110] Example 1 is a method for performing a computer-automated process using a trained machine learning (ML) model, the method comprising: accessing first event data describing a first event; executing a first ML model by one or more hardware processors to determine an ML representation of the first event using the first event data; applying a first set of rules to the first event data by one or more hardware processors to generate a rule representation of the first event; determining an output representation of the first event by one or more hardware processors based at least in part on the rule representation of the first event; and determining a deactivation first set of rules by one or more hardware processors based at least in part on the ML representation of the first event.

[0111] In Example 2, the subject of Example 1 may optionally include: applying a set of pre-training detection rules to the first event data before applying the first set of rules to the first event data to determine that the first event is an untrained event.

[0112] In Example 3, any one or more of the topics in Examples 1-2 may optionally include: determining that the confidence level of the ML representation of the first event is less than a threshold level; applying a set of reactive detection rules to the first event data to determine that the first event is not an untrained event; and in response to determining that the first event is not an untrained event, sending an instruction for the first event to a manual processing queue.

[0113] In Example 4, any one or more of the topics in Examples 1-3 may optionally include: determining that the confidence of the ML representation of the first event is less than a confidence threshold; and applying a set of reactive detection rules to the first event data to determine that the first event is an untrained event, wherein the application of the first set of rules is in response to determining that the first event is an untrained event.

[0114] In Example 5, the subject of Example 4 may optionally include: receiving second event data describing a second event for representation; executing a first ML model to determine an ML representation of the second event using the second event data; determining that the confidence of the ML representation of the second event is less than a confidence threshold; applying a set of reactive detection rules to the first event data to determine that the first event is not an untrained event; and directing the first event to a manual anomaly process.

[0115] In Example 6, the subject of Example 5 may optionally include: determining that the deactivation first rule set is based at least in part on the ratio of events identified as untrained events using the reactive detection rule set to events identified as not untrained events using the reactive detection rule set.

[0116] In Example 7, the subject of any one or more of Examples 1-6 may optionally include: an ML representation including a first score of the first event, a rule representation including supplementary values, and the method further includes combining the supplementary values ​​and the ML representation to determine an output representation of the first event.

[0117] In Example 8, the subject of any one or more of Examples 1-7 may optionally include: a first ML model being a trained ML model retrained using training data derived from multiple untrained events, the method further comprising: performing a deployed ML model to generate a deployed ML model representation of a first event, the deployed ML model representation including a score of the first event; and determining an output representation of the first event at least in part by combining a rule representation of the first event and the deployed ML model representation of the first event.

[0118] In Example 9, the subject of Example 8 may optionally include: determining the deactivation of the first set of rules includes: determining that the difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold.

[0119] In Example 10, the subject of any one or more of Examples 8-9 may optionally include: determining that the difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold; deploying a first ML model in response to determining that the difference between the ML representation of the first event and the expected representation of the first event is less than the first threshold; training a second version of the first ML model; and determining that the difference between at least one ML representation generated by the second version of the first ML model and the expected representation is less than a second threshold, the second threshold being less than the first threshold.

[0120] Example 11 is a system for performing a computer-automated process using a trained machine learning (ML) model, the system comprising: at least one computing device including a processor and memory, the at least one computing device being programmed to perform operations including: accessing first event data describing a first event; executing a first ML model to determine an ML representation of the first event using the first event data; applying a first set of rules to the first event data to generate a rule representation of the first event; determining an output representation of the first event based at least in part on the rule representation of the first event; and determining a deactivation of the first set of rules based at least in part on the ML representation of the first event.

[0121] In Example 12, the subject of Example 11 may optionally include an operation that further includes: applying a set of pre-training detection rules to the first event data to determine that the first event is an untrained event before applying the first set of rules to the first event data.

[0122] In Example 13, any one or more of the topics in Examples 11-12 may optionally include operations that further include: determining that the confidence level of the ML representation of a first event is less than a threshold level; applying a set of reactive detection rules to the first event data to determine that the first event is not an untrained event; and in response to determining that the first event is not an untrained event, sending an instruction for the first event to a manual processing queue.

[0123] In Example 14, any one or more of the topics in Examples 11-13 may optionally include operations that further include: determining that the confidence of the ML representation of a first event is less than a confidence threshold; and applying a set of reactive detection rules to the first event data to determine that the first event is an untrained event, the application of the first set of rules being in response to determining that the first event is an untrained event.

[0124] In Example 15, the subject of any one or more of Examples 11-14 may optionally include: the ML representation includes a first score of the first event, the rule representation includes supplementary values, and the operation further includes combining the supplementary values ​​and the ML representation to determine the output representation of the first event.

[0125] In Example 16, the subject of any one or more of Examples 11-15 may optionally include: a first ML model being a trained ML model retrained using training data derived from multiple untrained events, and the operation further including: executing a deployed ML model to generate a deployed ML model representation of a first event, the deployed ML model representation including a score of the first event; and determining an output representation of the first event at least in part by combining a rule representation of the first event and a deployed ML representation of the first event.

[0126] In Example 17, the subject of Example 16 may optionally include: determining the deactivation of the first set of rules includes: determining that the difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold.

[0127] In Example 18, the topics of any one or more of Examples 16-17 may optionally include operations that further include: determining that the difference between the ML representation of a first event and the desired representation of the first event is less than a first threshold; deploying a first ML model in response to determining that the difference between the ML representation of the first event and the desired representation of the first event is less than the first threshold; training a second version of the first ML model; and determining that the difference between at least one ML representation generated by the second version of the first ML model and the desired representation is less than a second threshold, the second threshold being less than the first threshold.

[0128] Example 19 is a non-transitory machine-readable medium having instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations including: accessing first event data describing a first event; executing a first ML model to determine an ML representation of the first event using the first event data; applying a first set of rules to the first event data to generate a rule representation of the first event; determining an output representation of the first event based at least in part on the rule representation of the first event; and determining to deactivate the first set of rules based at least in part on the ML representation of the first event.

[0129] In Example 20, the subject of Example 19 may optionally include an operation that further includes: applying a set of pre-training detection rules to the first event data to determine that the first event is an untrained event before applying the first set of rules to the first event data.

[0130] Figure 13 This is a block diagram 1300 illustrating an example of a software architecture 1302 for a computing device. Architecture 1302 can be used in conjunction with various hardware architectures, such as those described herein. Figure 13This is merely a non-limiting example of a software architecture, and many other architectures can be implemented to facilitate the functionality described herein. A representative hardware layer 1304 is shown, which can represent, for example, any of the computing devices described above. In some examples, hardware layer 1304 can be configured according to... Figure 13 It is implemented through the architecture of computer systems.

[0131] Representative hardware layer 1304 includes one or more processing units 1306 having associated executable instructions 1308. The executable instructions 1308 represent executable instructions of software architecture 1302, including implementations of the methods, modules, subsystems, and components described herein, and may also include memory and / or storage modules 1310, which also have executable instructions 1308. Hardware layer 1304 may also include other hardware indicated by other hardware 1312 representing any other hardware of hardware layer 1304 (such as other hardware shown as part of architecture 1302).

[0132] exist Figure 13 In the example architecture, software architecture 1302 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 1302 may include layers such as operating system 1314, libraries 1316, frameworks / middleware 1318, applications 1320, and presentation layer 1344. Operationally, applications 1320 and / or other components within a layer can call API calls 1324 via the software stack and, in response to API calls 1324, access responses, return values, etc., as indicated by message 1326. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide a framework / middleware layer 1318, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0133] Operating system 1314 can manage hardware resources and provide public services. Operating system 1314 may include, for example, kernel 1328, services 1330, and drivers 1332. Kernel 1328 can act as an abstraction layer between hardware and other software layers. For example, kernel 1328 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Service 1330 can provide other public services to other software layers. In some examples, service 1330 includes interrupt services. Interrupt services can detect the receipt of an interrupt and, in response, cause architecture 1302 to suspend its current processing and execute an interrupt service routine (ISR) when the interrupt is accessed.

[0134] Driver 1332 can be responsible for controlling or interfacing with underlying hardware. For example, depending on the hardware configuration, driver 1332 may include a display driver, a camera driver, etc. Drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, NFC drivers, audio drivers, power management drivers, etc.

[0135] Library 1316 can provide common infrastructure that can be utilized by application 1320 and / or other components and / or layers. Library 1316 typically provides functionality that allows other software modules to perform tasks in a way that is easier than directly interfaced with the underlying operating system 1314 functions (e.g., kernel 1328, services 1330, and / or drivers 1332). Library 1316 may include system 1334 libraries (e.g., the C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Furthermore, library 1316 may include API libraries 1336, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL frameworks that can be used to render 2D and 3D content on a display), database libraries (e.g., SQLite that provides various relational database functionalities), network libraries (e.g., WebKit that provides web browsing functionality), etc. Library 1316 may also include a wide variety of other libraries 1338 to provide a variety of other APIs to application 1320 and other software components / modules.

[0136] Framework 1318 (sometimes referred to as middleware) provides higher-level common infrastructure that can be utilized by Application 1320 and / or other software components / modules. For example, Framework 1318 can provide various graphical user interface (GUI) functionalities, advanced resource management, advanced location services, etc. Framework 1318 can provide a wide range of other APIs that can be utilized by Application 1320 and / or other software components / modules, some of which may be specific to a particular operating system or platform.

[0137] Application 1320 includes built-in application 1340 and / or third-party application 1342. Examples of representative built-in applications 1340 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1342 may include any of the built-in applications 1340 as well as a wide variety of other applications. In a particular example, a third-party application 1342 (e.g., used by an entity from a vendor different from the platform-specific provider of Android) may be an application that utilizes Android. TM Or iOS TMApplications developed using a Software Development Kit (SDK) can run on mobile operating systems such as iOS. TM Android TM , Mobile software on a telephone or other mobile computing device operating system. In this example, a third-party application 1342 may invoke API calls 1324 provided by a mobile operating system such as operating system 1314 to facilitate the functionality described herein.

[0138] Application 1320 can utilize built-in operating system functions (such as kernel 1328, services 1330, and / or drivers 1332), libraries (such as system 1334, API library 1336, and other libraries 1338), and frameworks / middleware 1318 to create user interfaces for interacting with the system's user. Alternatively or additionally, in some systems, user interaction may occur through a presentation layer (such as presentation layer 1344). In these systems, the application / module "logic" can be decoupled from the aspects of the application / module that interact with the user.

[0139] Some software architectures utilize virtual machines. Figure 13 In the example, this is illustrated by virtual machine 1348. The virtual machine creates a software environment in which applications / modules can execute as if they were running on a hardware computing device. The virtual machine is hosted by a host operating system (operating system 1314) and typically (though not always) has a virtual machine monitor 1346, which manages the operation of the virtual machine and its interface with the host operating system (i.e., operating system 1314). Software architectures execute within virtual machine 1348, such as operating system 1350, libraries 1352, frameworks / middleware 1354, applications 1356, and / or presentation layers 1358. These layers of the software architecture executing within virtual machine 1348 may be the same as or different from the corresponding layers previously described.

[0140] Modules, components and logic

[0141] Specific embodiments herein are described as including logic or multiple components, modules, or mechanisms. Modules may constitute software modules (e.g., code embodied in (1) a non-transitory machine-readable medium or (2) a transmitted signal) or hardware-implemented modules. Hardware-implemented modules are tangible units capable of performing specific operations and may be configured or arranged in a specific manner. In example embodiments, one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware processors may be hardware-implemented modules configured by software (e.g., an application or application portion) to operate to perform the specific operations described herein.

[0142] In various embodiments, the hardware-implemented module can be implemented mechanically or electronically. For example, a hardware-implemented module may include dedicated circuitry or logic permanently configured to perform a specific operation (e.g., as a dedicated processor, such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)). A hardware-implemented module may also include programmable logic or circuitry temporarily configured by software to perform a specific operation (e.g., as contained within a general-purpose processor or another programmable processor). It will be understood that the decision to implement a hardware-implemented module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured (e.g., software-configured) circuitry may be driven by cost and time considerations.

[0143] Therefore, the term "hardware-implemented module" should be understood to include tangible entities, which are physically constructed and permanently configured (e.g., hardwired) or temporarily or provisionally configured (e.g., programmed) to operate and / or perform the specific operations described herein in a particular manner. Considering embodiments where hardware-implemented modules are provisionally configured (e.g., programmed), each of the hardware-implemented modules does not need to be configured or instantiated at any given time. For example, in cases where the hardware-implemented modules include a general-purpose processor configured using software, the general-purpose processor can be configured at different times as correspondingly different hardware-implemented modules. The software can accordingly configure the processor, for example, constituting a specific hardware-implemented module in one time instance and different hardware-implemented modules in different time instances.

[0144] Hardware-implemented modules can provide information to and receive information from other hardware-implemented modules. Therefore, the described hardware-implemented modules can be considered communicatively coupled. When multiple such hardware-implemented modules exist simultaneously, communication can be achieved through signal transmission (e.g., through appropriate circuitry and buses connecting the hardware-implemented modules). In embodiments where multiple hardware-implemented modules are configured or instantiated at different times, such communication between hardware-implemented modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware-implemented modules. For example, one hardware-implemented module can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Subsequently, another hardware-implemented module can access the memory device at a later time to retrieve and process the stored output. Hardware-implemented modules can also initiate communication with input or output devices and are capable of manipulating resources (e.g., collections of information).

[0145] The various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, the modules mentioned herein may include processor-implemented modules.

[0146] Similarly, the methods described herein can be implemented at least in part by processors. For example, at least some of the operations of the methods can be executed by one or more processors or modules implemented by processors. The execution of a particular operation can be distributed across one or more processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, one or more processors may be located in a single location (e.g., in a home environment, office environment, or server farm), while in other embodiments, processors may be distributed across multiple locations.

[0147] One or more processors may also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service (SaaS)”. For example, at least some of the operations may be executed by a set of computers (as an example of a machine that includes processors), and these operations may be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0148] Electronic devices and systems

[0149] The example embodiments may be implemented in digital electronic circuits, or in computer hardware, firmware, or software, or in a combination thereof. The example embodiments may be implemented using a computer program product, such as a computer program tangibly embodied in an information carrier (e.g., in a machine-readable medium executed or controlled by a data processing apparatus (e.g., a programmable processor, computer, or a plurality of computers)).

[0150] Computer programs can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, subroutines, or other units suitable for use in a computing environment. Computer programs can be deployed to execute on one or more computers, distributed at a single site or across multiple sites and interconnected via a communication network.

[0151] In the example embodiments, the operations may be performed by one or more programmable processors executing a computer program to perform the function by manipulating input data and generating output. The method operations may also be performed by dedicated logic circuitry (e.g., an FPGA or ASIC), and the apparatus of the example embodiments may be implemented as dedicated logic circuitry.

[0152] Computing systems may include clients and servers. Clients and servers are generally geographically isolated and typically interact via a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. In embodiments of deploying programmable computing systems, it will be understood that both hardware and software architectures are worth considering. Specifically, it will be understood that the choice of implementing a particular function in permanently configured hardware (e.g., ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or in a combination of permanently and temporarily configured hardware can be a design choice. The hardware (e.g., machines) and software architectures that can be deployed in various example embodiments are listed below.

[0153] Example machine architecture and machine-readable media

[0154] Figure 14 This is a block diagram of a machine in the example form of a computer system 1400, within which instructions 1424 can be executed to cause the machine to perform any or more of the methods discussed herein. In alternative embodiments, the machine operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, network router, switch, or bridge, or any machine capable of executing (sequentially or otherwise) instructions specifying the actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0155] Example computer system 1400 includes a processor 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), main memory 1404, and static memory 1406, which communicate with each other via a bus 1408. Computer system 1400 may also include a video display unit 1410 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). Computer system 1400 also includes an alphanumeric input device 1412 (e.g., a keyboard or touch-sensitive display), a user interface (UI) navigation (or cursor control) device 1414 (e.g., a mouse), a disk drive unit 1416, a signal generation device 1418 (e.g., a speaker), and a network interface device 1420.

[0156] Machine-readable media

[0157] The disk drive unit 1416 includes a machine-readable medium 1422 on which one or more sets of data structures and instructions 1424 (e.g., software) are stored, which embody or be utilized by any one or more of the methods or functions described herein. The instructions 1424 may also reside wholly or at least partially within main memory 1404 and / or processor 1402 during execution thereon by the computer system 1400, wherein main memory 1404 and processor 1402 also constitute the machine-readable medium 1422.

[0158] Although machine-readable medium 1422 is shown as a single medium in the example embodiment, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more instructions 1424 or data structures. The term "machine-readable medium" should also be understood to include any tangible medium capable of storing, encoding, or carrying machine-executable instructions 1424 and enabling a machine to perform any one or more methods of this disclosure, or any tangible medium capable of storing, encoding, or carrying data structures utilized by or associated with such instructions 1424. Therefore, the term "machine-readable medium" should be understood to include, but is not limited to, solid-state memory and optical and magnetic media. Specific examples of machine-readable medium 1422 include non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0159] transmission medium

[0160] Instructions 1424 can also be sent or received on communication network 1426 using a transmission medium. Instructions 1424 can be sent using network interface device 1420 and any of a variety of well-known transmission protocols (e.g., HTTP). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, mobile phone networks, conventional telephone (POTS) networks, and wireless data networks (e.g., WiFi and WiMax networks). The term "transmission medium" should be understood to include any intangible medium capable of storing, encoding, or carrying instructions 1424 for machine execution, and includes digital or analog communication signals or other intangible media that facilitate such software communication.

[0161] Although embodiments have been described with reference to specific exemplary examples, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader spirit and scope of this disclosure. Therefore, the specification and drawings are to be considered illustrative rather than limiting. The accompanying drawings, which form a part of this document, illustrate specific embodiments in which the subject matter can be practiced in an illustrative rather than limiting manner. The illustrated embodiments have been described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments can be utilized and derived therefrom, thereby allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, the detailed description should not be considered limiting, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

[0162] Therefore, although specific embodiments have been shown and described herein, it should be understood that any arrangement intended to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon reading the foregoing description.

Claims

1. A method for performing a computer-automated process using a trained machine learning (ML) model, the method comprising: Use at least one set of rules to determine that the first ML model was not trained to characterize the first event type; Modify the first ML model; Access the first event data that describes the first event type; The first ML model is executed by one or more hardware processors to determine the ML representation of the first event using the first event data; One or more hardware processors apply a first set of rules to the first event data to generate a rule representation of the first event; The output representation of the first event is determined by one or more hardware processors based at least in part on the rule representation of the first event; as well as The deactivation of the first set of rules is determined by one or more hardware processors based at least in part on the ML representation of the first event.

2. The method according to claim 1, further comprising: Before applying the first set of rules to the first event data, a set of pre-training detection rules is applied to the first event data to determine that the first event is an untrained event.

3. The method according to claim 1, further comprising: Access the second event data of the second event type, which is different from the first event type; Execute the first ML model to determine the ML representation of the second event using the second event data; The confidence level of the ML representation of the second event is less than the threshold level; The reactive detection rule set is applied to the first event data to determine that the second event is not an untrained event; as well as In response to determining that the second event is not an untrained event, an indication of the second event is sent to the manual processing queue.

4. The method according to claim 1, further comprising: Access the second event data of the second event type, which is different from the first event type; Execute the first ML model to determine the ML representation of the second event using the second event data; The confidence level of the ML representation of the second event is determined to be less than the confidence threshold; as well as The reactive detection rule set is applied to the second event data to determine that the second event is an untrained event, and the application of the first rule set is in response to determining that the first event is an untrained event.

5. The method according to claim 4, further comprising: Receive third event data describing the third event used for characterization; Execute the first ML model to determine the ML representation of the third event using the third event data; The confidence level of the ML representation of the third event is less than the confidence threshold; The reactive detection rule set is applied to the third event data to determine that the third event is not an untrained event; as well as Direct the third event to a manual exception procedure.

6. The method of claim 5, wherein determining the first set of deactivation rules is based at least in part on the ratio of events identified as untrained events using the reactive detection rule set to events identified as not untrained events using the reactive detection rule set.

7. The method of claim 1, wherein the ML representation includes a first score of a first event, the rule representation includes supplementary values, and the method further comprises: Combine supplementary values ​​and ML representations to determine the output representation of the first event.

8. The method of claim 1, wherein the first ML model is a trained ML model retrained using training data derived from multiple untrained events, the method further comprising: The deployed ML model is executed to generate a deployed ML model representation of the first event, which includes a score for the first event; as well as The output representation of the first event is determined at least in part by combining the rule representation of the first event and the ML model representation of the deployment of the first event.

9. The method according to claim 8, wherein determining the deactivation first rule set comprises: The difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold.

10. The method of claim 8, further comprising: The difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold. In response to the fact that the difference between the determined ML representation of the first event and the expected representation of the first event is less than a first threshold, the first ML model is deployed; The second version of the first ML model was trained; as well as The difference between at least one ML representation generated by the second version of the first ML model and the expected representation is determined to be less than a second threshold, which is less than a first threshold.

11. A system for performing a computer-automated process using a trained machine learning (ML) model, the system comprising: At least one computing device including a processor and memory, said at least one computing device being programmed to perform operations including: Use at least one set of rules to determine that the first ML model was not trained to characterize the first event type; Modify the first ML model; Access the first event data that describes the first event type; Execute the first ML model to determine the ML representation of the first event using the first event data; The first set of rules is applied to the first event data to generate a rule representation of the first event; The output representation of the first event is determined at least in part based on the rule representation of the first event; and The deactivation first rule set is determined at least in part based on the ML representation of the first event.

12. The system according to claim 11, further comprising: Before applying the first set of rules to the first event data, the pre-training detection rule set is applied to the first event data to determine whether the first event is an untrained event.

13. The system according to claim 11, further comprising: Access the second event data of the second event type, which is different from the first event type; Execute the first ML model to determine the ML representation of the second event using the second event data; The confidence level of the ML representation of the second event is less than the threshold level; The reactive detection rule set is applied to the first event data to determine that the second event is not an untrained event; as well as In response to determining that the second event is not an untrained event, an indication of the second event is sent to the manual processing queue.

14. The system of claim 11, further comprising: Access the second event data of the second event type, which is different from the first event type; Execute the first ML model to determine the ML representation of the second event using the second event data; The confidence level of the ML representation of the second event is determined to be less than the confidence threshold; as well as The reactive detection rule set is applied to the second event data to determine that the second event is an untrained event. The application of the second rule set is in response to the determination that the second event is an untrained event.

15. The system of claim 11, wherein the ML representation includes a first score of a first event, the rule representation includes supplementary values, and the operation further includes: Combine supplementary values ​​and ML representations to determine the output representation of the first event.

16. The system of claim 11, wherein the first ML model is a trained ML model retrained using training data derived from multiple untrained events, the operation further comprising: The deployed ML model is executed to generate a deployed ML model representation of the first event, which includes a score for the first event; as well as The output representation of the first event is determined at least in part by combining the rule representation of the first event and the ML representation of the deployment of the first event.

17. The system of claim 16, determining the deactivation first set of rules includes: The difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold.

18. The system of claim 16, further comprising: The difference between the ML representation of the first event and the expected representation of the first event is less than a first threshold. In response to the fact that the difference between the determined ML representation of the first event and the expected representation of the first event is less than a first threshold, the first ML model is deployed; The second version of the first ML model was trained; as well as The difference between at least one ML representation generated by the second version of the first ML model and the expected representation is determined to be less than a second threshold, which is less than a first threshold.

19. A non-transitory machine-readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform an operation, the operation comprising: Use at least one set of rules to determine that the first ML model was not trained to characterize the first event type; Modify the first ML model; Access the first event data that describes the first event type; Execute the first ML model to determine the ML representation of the first event using the first event data; The first set of rules is applied to the first event data to generate a rule representation of the first event; The output representation of the first event is determined at least in part based on the rule representation of the first event; as well as The deactivation first rule set is determined at least in part based on the ML representation of the first event.

20. The non-transitory machine-readable medium of claim 19, further comprising: Before applying the first set of rules to the first event data, the pre-training detection rule set is applied to the first event data to determine whether the first event is an untrained event.

Images