Automatically execute computer actions in response to meeting machine learning-based conditions

Abstract

Implementations involve automatically performing one or more computer actions in response to the satisfaction of one or more machine learning (ML)-based conditions. Some implementations involve determining which ML-based conditions to render in an automated interface and / or how to render the machine learning-based conditions in an automated interface. Those implementations can result in a reduction (or even no) of the amount of user input required to define the action(s) ...

Description

Background Technology

[0001] Various techniques have been proposed for automating computer actions in response to the fulfillment of rule-based conditions. For example, techniques have been proposed for automatically forwarding emails sent to a first email address to another email address when one or more rule-based conditions are met. For example, rule-based conditions can include: the email was sent from a specific email address, the email was sent from a specific email domain, the email subject includes certain terms(s), and / or other rule-based conditions(s).

[0002] Automatically executing computer actions in response to the fulfillment of rule-based conditions can reduce (or eliminate) the additional user input required at the client device to perform these actions. Furthermore, automatic execution can conserve various client device resources that would otherwise require such user input at the client device, leading to the activation and / or higher power levels of the client device's display and / or other components.

[0003] However, rule-based conditions can have various drawbacks on their own. As an example, rule-based conditions must be manually defined with a large amount of user input, which can require extended interaction with the client device and corresponding extended use of various resources on the client device.

[0004] As another example, rule-based conditions can often be defined too narrowly, leading to under-triggering—or too broadly, leading to over-triggering. Under-triggering can cause (multiple) corresponding automatic actions to fail to execute in many cases where they should, resulting in the need for (multiple) user inputs (and consequently, the utilization of client device resources) in those cases. Over-triggering can cause corresponding actions to execute in many cases where they should not, resulting in unnecessary use of computational and / or network resources in those cases.

[0005] Furthermore, under-triggered and over-triggered conditions can lead to a manual redefinition of rule-based conditions in an attempt to mitigate either. Similar to defining rule-based conditions, redefining them can also result in the activation and / or higher power states of (multiple) components of the client device, extending the duration. Summary of the Invention

[0006] The implementations disclosed herein relate to automatically performing one or more computer actions in response to the satisfaction of one or more machine learning (ML)-based conditions (also referred to herein as "ML-based conditions"). ML-based conditions determine whether a condition is satisfied or not based on analysis of predicted outputs (e.g., probability values, value vectors), which are generated by processing corresponding data using an ML model employing the ML-based conditions. Various ML-based conditions and corresponding ML models can be generated and utilized. For example, a first ML-based condition could be "electronic communication with an action item," and a corresponding first ML model could be used to process features of the electronic communication to generate an output indicating whether the electronic communication "has an action item." Similarly, for example, a second ML-based condition could be "electronic communication requiring immediate attention," and a corresponding second ML model could be used to process features of the electronic communication to generate an output indicating whether the electronic communication "requires immediate attention." Further details regarding example ML models and their training are provided herein.

[0007] Some implementations involve determining which ML-based conditions to render in an automation interface and / or how to render these ML-based conditions in an automation interface. An automation interface is an interface capable of providing user input to define multiple computer actions and multiple action conditions (e.g., multiple ML-based conditions and optionally multiple rule-based conditions) that, when met, result in the automatic execution of the multiple computer actions. As used herein, automation interfaces encompass workflow interfaces. Implementations that determine which ML-based conditions to render and / or how to render these ML-based conditions can reduce the amount of user input required to define the action conditions for the multiple computer actions (or even eliminate the need for user input). Those implementations can additionally or alternatively reduce the duration of the interaction defining the multiple action conditions, which can reduce the duration for which components of the client device used to interact with the automation interface are active and / or active in a higher power state.

[0008] Some implementations additionally or alternatively involve training machine learning models that are used to evaluate whether ML-based conditions have occurred based on (e.g., based only on or fine-tuned based on) user-specific and / or organization-specific training data. When the trained machine learning model is used to determine whether to perform (multiple) computer actions for the user and / or organization, those implementations can mitigate (or eliminate) over-triggered and / or under-triggered events. Those implementations can additionally or alternatively mitigate the computational and / or network inefficiencies associated with over-triggered and / or under-triggered events.

[0009] In some implementations involving determining which ML-based conditions to render in the automation interface and / or how to render them, the determination is based at least in part on one or more computer actions already defined by the user via the automation interface. In other words, it is possible to render different ML-based conditions for different computer actions in the automation interface and / or to present ML-based conditions in different ways for different computer actions.

[0010] For example, when only a first computer action has been defined in the automation interface: the first ML-based condition can be presented with content and / or display characteristics indicating that it is more relevant than the second ML-based condition; the first ML-based condition can be pre-selected, while the second ML-based condition cannot; and / or the first ML-based condition can be presented without the second ML-based condition. On the other hand, when only a second computer action has been defined in the automation interface: the second ML-based condition can be presented with content and / or display characteristics indicating that it is more relevant than the first ML-based condition; the second ML-based condition can be pre-selected, while the first ML-based condition cannot; and / or the second ML-based condition can be presented without the first ML-based condition.

[0011] More generally, ML-based conditions that are more likely to be applicable to the defined multiple computer actions can be presented in a way that allows them to be selected more quickly and / or with less user input (or even no user input). These technical advantages are particularly impactful for ML-based conditions that can be described using semantic descriptors (e.g., “email with action items”), which, without the techniques disclosed herein, would be difficult for users to ascertain the applicability of their computer actions to be automatically performed by the ML-based conditions. Therefore, the implementations disclosed herein can help guide users to more relevant ML-based conditions during user interaction with an automated interface, while optionally still providing the user with ultimate control over the selected ML-based conditions(s).

[0012] As mentioned above, in determining which ML-based conditions to render and / or how to render them, this determination can be based at least in part on one or more computer actions defined by the user via an automation interface. In some implementations of those implementations, the determination is based on a corresponding metric for each of the ML-based conditions, wherein each metric is specific to the ML-based condition and the computer action(s). The metric for the ML-based condition(s) for the computer action(s) can be determined before or in response to the selection of the computer action(s).

[0013] For example, for each ML-based condition, at least one corresponding metric can be generated based on user-defined automated computer actions(s). For instance, when generating a metric for a given ML-based condition, past occurrences of the computer actions(s)(s) can be identified, where these past occurrences were user-initiated rather than automatically executed. Past occurrences can be past occurrences of the user or past occurrences of a user group (e.g., users of the user's employer, including the user). For a given ML-based condition, corresponding data for each of the past occurrences can be processed using a given ML model to generate corresponding predicted values ​​based on the corresponding data. Any function that can determine the metric for a given ML model based on the predicted values ​​can then be used to determine whether and / or how the ML-based condition is presented. For example, the metric can be used to present, highlight, or automatically select "good" (metric-based) ML-based conditions for(s)(s) and / or tone down / suppress "bad" (metric-based) ML-based conditions(s).

[0014] As a specific example, suppose a user provides (multiple) user inputs via an automated interface to define the following computer actions: "Forward email to jon@exampleurl.com" (e.g., the email address of the user's administrative assistant); and "Move to the 'Action Items' folder." The (multiple) user inputs can define the (multiple) computer actions through free-form input and / or selections from pre-formed computer actions (e.g., from dropdown lists, radio buttons, etc.). Further suppose the ML-based conditions are: (1) "Email with action items"; (2) "Email requiring immediate attention"; (3) "Email with customer questions"; and (4) "Email with positive sentiment." Each of the ML-based conditions can include a corresponding trained ML model used to process the features of the email and generate output indicating whether the corresponding ML-based condition is satisfied. A subset of past emails (e.g., the user who provided the input and / or other users) can be identified: forwarded to "Administrative Assistant" (e.g., forwarded to jon@exampleurl.com or, if this relationship is known), and moved to the 'Action Items' folder. Emails (e.g., their characteristics) can each be processed using an ML model of ML-based conditions to determine that 90% meet ML-based condition (1) and less than 10% meet ML-based conditions (2)-(4). Therefore, ML-based condition (1) can be: the most prominently presented condition as a suggested condition; automatically selected as a condition (requiring user confirmation); and / or presented with the indication "90%". Alternatively, ML-based conditions (2)-(4) can be suppressed or presented less prominently, or have indications that the ML-based condition might be "bad" (e.g., with an indication of its corresponding percentage). As will be understood from the specific examples above, the metrics will differ for other selected automated computer actions(s)—causing different suggestions / displays for those other computer actions(s). Furthermore, the processing to determine the selected metrics for the selected computer actions(s)(s) can be pre-executed or can be performed in response to the selection.

[0015] As mentioned above, some implementations additionally or alternatively involve training an ML model for ML-based conditions based on (e.g., based solely on, or finely tuned based on) user-specific and / or organization-specific training data. As an example, suppose the ML-based condition is "electronic communications requiring immediate attention." A corresponding ML model for the user can be trained by generating positive training instances based on past electronic communications (of a specific type or any of several types) that the user responded to within one hour of receiving the data, and based on those positive training instances. Alternatively, a corresponding ML model for the user can be trained by generating negative training instances based on past electronic communications that the user responded to outside of one hour of receiving the data, optionally conditioned on those electronic communications that the user also viewed within one hour of receiving the data. Thus, the ML model can be customized to identify electronic communications that typically elicit a rapid response for a given user (e.g., within one hour of receiving the data—or other criteria). The corresponding ML model can optionally be based on a model pre-trained according to similar training instances of other user interactions. Types of electronic communication include, for example, email, Rich Communication Services (RCS) messages, Short Message Services (SMS) messages, Multimedia Messaging Services (MMS) messages, Over-the-Top (OTT) chat messages, social network messages, voice communication (e.g., telephone calls, voicemail), audio-video communication, calendar invitations, etc.

[0016] The above description is provided as an overview of only some of the embodiments disclosed herein. Those embodiments and other embodiments are described in more detail herein.

[0017] Various embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform methods, such as one or more methods described herein. Other embodiments may include a system comprising a memory and one or more hardware processors operable to execute instructions stored in the memory to perform methods such as one or more methods described herein.

[0018] It should be understood that all combinations of the foregoing concepts and other concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0019] Figure 1A An example environment is illustrated that allows for the implementation of the methods disclosed herein.

[0020] Figure 1B Depicting the display Figure 1A Examples of implementation processes for how various components can interact.

[0021] Figure 2A , Figure 2B , Figure 2C and Figure 2D Each illustration shows an example of an automation interface, which is customized based on a corresponding metric, which is also illustrated and is based on (multiple) corresponding computer actions defined via the automation interface.

[0022] Figure 3 A flowchart illustrating example methods according to various embodiments disclosed herein is provided.

[0023] Figure 4 A flowchart illustrating another example method according to various embodiments disclosed herein is provided.

[0024] Figure 5 An example architecture of a computer system is schematically depicted. Detailed Implementation

[0025] Figure 1A An example environment in which the embodiments disclosed herein can be implemented is illustrated. This example environment includes a client device 110 and an automated action system 118. The automated action system 118 can be implemented in one or more servers, for example, communicating via a network (not depicted). The automated action system 118 is an example of a system in which the technologies and / or systems, components, and technologies described herein can be implemented and interfaced. Although various components are illustrated and described as being implemented by the automated action system 118 in one or more servers remote from the client device 110, one or more components can be additionally or alternatively implemented on the client device 110 (in whole or in part).

[0026] Users can interact with the automated action system 118 via client device 110. Other computer devices can communicate with the automated action system 118, including but not limited to the user's (multiple) other client devices, other users' other client devices, and / or one or more servers that have cooperated with the provider of the automated action system 118 to implement services. However, for the sake of brevity, the example is described in the context of client device 110.

[0027] Client device 110 communicates with automated action system 118 via a network, such as a local area network (LAN) or a wide area network (WAN) such as the Internet (one or more such networks are typically indicated at 117). Client device 110 can be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (optionally with a display) operating a voice-interactive personal digital assistant (also referred to as an "automatic assistant"), or a user's wearable device including a computing device (e.g., a user's watch with a computing device, glasses with a computing device, a wearable music player). Additional and / or alternative client devices can be provided.

[0028] Client device 110 may include various software and / or hardware components. For example, in FIG1, client device 110 includes multiple user interface (UI) input devices 112 and multiple output devices 113. The multiple UI input devices 112 may include, for example, multiple microphones, touchscreens, keyboards (physical or virtual), mice, and / or multiple other UI input devices. A user of client device 110 may use one or more of the multiple UI input devices 112 to provide input to the automation interface described herein. For example, selection of elements of the automation interface may be in response to touch input pointing to an element (via touchscreen), mouse or keyboard selection pointing to an element, voice input identifying an element (detected via multiple microphones), and / or gesture input pointing to an element (e.g., non-touch gestures detected via multiple visual components). The multiple output devices 113 may include, for example, touchscreens or other displays, multiple speakers, and / or multiple other output devices. A user of client device 110 may use one or more of the multiple output devices 113 to consume the output of the automation interface described herein. For example, a display can be used to view the visual components of an automation interface and / or (multiple) speakers can be used to listen to the audio components of an automation interface. The automation interface can be, for example, visual only, audiovisual, or audio only.

[0029] Client device 110 can also execute various software. For example, in the embodiment described in FIG1, client device 110 executes one or more applications 114. The applications 114 can include, for example, auto-assistant applications, web browsers, messaging applications, email applications, cloud storage applications, video conferencing applications, calendar applications, chat applications, etc. One or more of the applications 114 can at least selectively interface with the automated action system 118, for example, by defining the computer actions to be automatically executed and defining action conditions (e.g., multiple ML-based conditions and / or multiple other conditions) that cause the automatic execution of the computer actions when met. Furthermore, the same and / or multiple different applications 114 can be used to view the results of the automatic execution of the computer actions. For example, the web browser and / or auto-assistant application of application 114 can be used to interface with the automated action system 118. As another example, the applications 114 can include automated action applications dedicated to interacting with the automated action system 118. As yet another example, (multiple) applications 114 may include a first application (e.g., an email application) that can interface with the automatic action system 118 to define computer actions to be performed automatically in relation to the first application; a second application (e.g., a chat application) that can interface with the automatic action system 118 to define computer actions to be performed automatically in relation to the second application, and so on.

[0030] The automatic action system 118 includes a graphical user interface (GUI) engine 120, a measurement engine 122, a past occurrence engine 124, an assignment engine 126, and an automatic action engine 128.

[0031] GUI engine 120 controls an automation interface rendered via one of the applications 114 on client device 110. This automation interface is an interface through which user input (e.g., via one or more UI input devices of UI input device 112) is provided to define multiple computer actions and multiple action conditions (e.g., multiple ML-based conditions and optionally multiple rule-based conditions), which, when satisfied, lead to the automatic execution of the computer actions. As described herein, in various embodiments, GUI engine 120 is able to determine which ML-based conditions(s) will be rendered in the automation interface and / or how machine learning-based conditions(s) will be rendered in the automation interface.

[0032] In some implementations where the GUI engine 120 determines which ML-based conditions(s) to render in the automation interface and / or how to render them, this determination is at least in part based on one or more computer actions(s) already defined by the user via the automation interface. In other words, the GUI engine 120 is capable of rendering different ML-based conditions(s) for different computer actions(s) in the automation interface and / or of presenting ML-based conditions(s) differently for different computer actions(s). Generally, the GUI engine 120 is capable of presenting ML-based conditions(s) that may be more suitable for the defined computer actions(s) in a way that allows them to be selected more quickly and / or with less user input(s) (or even no user input).

[0033] In many implementations where the GUI engine 120 determines, at least in part, which ML-based conditions and / or how to render them based on one or more computer actions defined by the user via an automated interface, the GUI engine 120 makes multiple determinations based on metrics from the metric engine 122.

[0034] Metric engine 122 can interface with past occurrence engine 124. This allows for the identification of past occurrences of computer actions(s)(s) defined via a user interface from past data database 154. Past occurrences identified by past occurrence engine 124 are all user-initiated occurrences. In other words, the past computer actions are not executed automatically, but rather in response to one or more manual user inputs. Past occurrences can be past occurrences that are interfaced by a user with an automated interface, or they can be past occurrences of a user group (optionally including users). Past occurrences can be used in the techniques described herein, depending on the approval of the user(s)(s) who initiated the past computer action(s). In the case of past occurrences via a user group, this user group can optionally be a user-based group, and the users interacting with the automated interface all belong to the employer's public enterprise account and / or have(s) other common characteristics (e.g., all have the same title assigned by the employer, all have the same workgroup assigned by the employer, etc.). In some implementations, the user group(s) are selected based on the automated interface used to define the computer action(s) and(s) associated action conditions(s), which are then applied to all users in that group. For example, an automation interface can include interface elements that can define multiple computer actions and multiple action conditions for a single user or a group of users.

[0035] As an example, if the computer action of "making the document available offline" is defined for an automated computer action in a cloud-based storage environment, then the past occurrence engine 124 can identify past occurrences of "making the document available offline" in the cloud-based storage environment. Each of the past occurrences is identified in response to user input, such as right-clicking the corresponding document in the cloud-based storage interface and selecting "available offline" from the menu revealed in response to the right-click. The past occurrence engine 124 can identify all past occurrences, or only a subset of past occurrences (e.g., only 50 occurrences or other threshold numbers). The data in the identified past data database 154 can include various features and can depend on the features required by the metric engine 122 (described in more detail below). For example, for an action that makes a document available offline, features can include those indicating: the time the document was created; the size of the document; the duration of viewing the document; the duration of editing the document; the document's title (e.g., Word2Vec or other embeddings of the title); multiple images of the document (e.g., multiple embeddings of multiple images of the document); terms included in the document (e.g., Word2Vec or other embeddings of multiple first sentences of the document); the folder where the document is stored; the document type (e.g., PDF, spreadsheet, word processing document) and / or multiple other features.

[0036] After past occurrence engine 124 has identified past occurrence data for (multiple) computer actions, metric engine 122 is capable of generating at least one corresponding metric for each of the multiple available ML-based conditions associated with (multiple) computer actions. In some implementations, when generating a metric for the ML-based condition, metric engine 122 processes each instance of data using one of the ML models 152A-N corresponding to the ML-based condition to generate a corresponding predicted output. Metric engine 122 is then capable of generating a metric for the ML-based condition based on the predicted output from the processing using the corresponding ML model. For example, each predicted output can be a probability measure (e.g., 0 to 1), and the metric can be based on the number of predicted outputs that satisfy a threshold probability measure indicating that the ML-based condition is satisfied (e.g., a threshold probability measure of 0.7 or other probabilities). For example, the metric can be based on the percentage of the number of predicted outputs that satisfy the threshold probability measure divided by the total number of predicted outputs. Additional and / or alternative metrics can be generated, such as metrics that define the mean and / or median probability measures of all predicted outputs and / or the standard deviation (optionally excluding outliers) of the probability measures of all predicted outputs.

[0037] For example, and continuing with the working example, suppose we have an ML-based condition for a corresponding ML model 152G, "Important Document". Metrics engine 122 is capable of (individually) processing instances of past data 1-N using ML model 152G to generate N individual instances of a predicted output indicating probabilities 1-N. Metrics engine 122 is then capable of generating at least one metric based on probabilities 1-N. This metric typically indicates the frequency with which the ML-based condition for "Important Document" will be considered satisfied based on the corresponding instances of past data 1-N. In other words, the metric provides an indication of the frequency with which the ML-based condition will be considered satisfied in cases where a user (or user group, including the user) manually performs computer actions defined in an automated interface.

[0038] Based on instances of processing past data using other models in ML-based models that correspond to other conditions in ML-based conditions, metric engine 122 can similarly generate metrics for other conditions in ML-based conditions related to computer actions. For example, metric engine 122 can generate metrics for other conditions in ML-based conditions related to cloud-based storage devices (e.g., conditions with appropriate input parameters corresponding to cloud-based storage domains). For example, ML-based conditions can include certain ML-based conditions that correspond only to emails, certain other ML-based conditions that correspond only to documents in cloud-based storage devices (which can include emails and / or other documents), certain ML-based conditions that correspond to video conferencing, and / or certain other ML-based conditions that can be applied to other domains (or even multiple domains).

[0039] After the metric engine 122 generates metrics, the GUI engine 120 can render ML-based conditions in a metric-dependent manner (initially or after updating). For example, the GUI engine 120 can use metrics to render, highlight, or automatically select "good" (metric-based) ML-based conditions and / or dilute / suppress "bad" (metric-based) ML-based conditions for (multiple) actions. Similarly, for example, the GUI engine 120 can additionally or alternatively provide indications of metrics and ML-based conditions. Some non-limiting examples of automated interfaces that can be rendered by the GUI engine 120 based on metrics are provided below. Figures 2A-2D The diagram is shown below.

[0040] Users of client device 110 can further interact with the automation interface via one or more UI input devices 112 to select ML-based conditions and / or (multiple) other action conditions (e.g., rule-based conditions or (multiple) other non-ML-based conditions) rendered for an action—and / or provide confirmation user input indicating confirmation of user selection (and / or automatically pre-selected) conditions for (multiple) computer actions defined via the automation interface. In some embodiments, GUI engine 120 is capable of providing user interface elements that enable users to define multiple conditions via UI input device 112. In some of those embodiments, the user interface elements may optionally enable users to define whether all conditions need to be satisfied for the computer action to be executed automatically, or alternatively whether only any subset needs to be satisfied to cause one or more computer actions to be executed automatically. Each subset includes one or more action conditions.

[0041] In response to confirmation of user input, the allocation engine 126 can allocate in the automatic action database 156 the computer action(s) to be automatically executed, the action conditions of the computer action(s), and the identifier (e.g., account identifier) ​​of the user (or user group) whose computer action will be automatically executed in response to the occurrence of the action conditions(s).

[0042] Following the allocation in the automatic action database 156, the automatic action engine 128 can utilize appropriate permissions and monitor the satisfaction of multiple conditions for multiple users based on the allocation in the automatic action database 156. If the automatic action engine 128 determines that the multiple conditions for multiple users are satisfied, the automatic action engine can cause multiple computer actions to be executed. For example, and continuing with the example, suppose the computer action "make document available offline" is defined with the condition "important document". In this case, the automatic action engine 128 can use one of the corresponding ML models 152A-N to process the features of the document for multiple users, and automatically make the document available offline (e.g., make the document locally downloaded to the corresponding client device) if the predicted output indicates that the ML-based condition is satisfied. Document features can be processed periodically or irregularly in response to document creation, document modification, document opening, document closing, or in response to multiple other conditions to determine whether the ML-based condition is satisfied.

[0043] In some implementations, the automatic action engine 128 is capable of interfaceing with one or more additional systems 130 to determine whether one or more action conditions are met and / or to automatically perform one or more computer actions. For example, for the action "make my office light blink" with an ML-based condition of "urgent email", the automatic action engine 128 is capable of interfaceing with one of the additional systems(s) controlling the "office light" to blink the office light in response to determining that the ML-based condition is met.

[0044] Briefly switch to Figure 1B The illustration shows an example process flow that demonstrates some implementations of how the various components of the client device 110 and the automated motion system 118 can interact in various implementations.

[0045] exist Figure 1B In this context, client device 110 is used to interact with an automation interface rendered by GUI engine 120 to define one or more computer actions 201. The computer actions 201 are provided to the pasting engine 124. As a running example, the computer actions(s) 201 can be used in a video conferencing domain and can be "saving a recorded copy of the video conference".

[0046] Past occurrence engine 124 interfaces with past data database 154 to identify past occurrence data 203. Past occurrence data 203 includes instances of past data, where each instance corresponds to a user-initiated occurrence of computer action(s) 201. Continuing as an example, past occurrence data 203 is capable of including an instance of each user-initiated "Save a copy of the video conference recording" (e.g., in response to manually selecting the "Save a copy of the recording" interface element at the end of the video conference). Each instance of data may include various characteristics, such as those indicating the following: day of the week in the video conference, the time of the video conference, the duration of the video conference, the topics(s) discussed in the video conference (e.g., determined from the recording copy and / or agenda), the name of the video conference, and / or(s) other features. Past occurrence data 203 is provided to metrics engine 122.

[0047] The metric engine 122 is then able to process instances of data using ML models 152A-N, which correspond to ML-based conditions relevant to the video conferencing domain. Based on the predictions generated for each of the relevant ML models 152A-N, the metric engine 122 generates at least one metric for each ML-based condition 205.

[0048] GUI engine 120 is then able to render (initially or after updating) the GUI generated based on metric 207 (i.e., based on metric 205). For example, GUI 207 can omit (multiple) ML-based conditions with poor metrics, make (multiple) ML-based conditions with good metrics more prominent than other conditions with worse metrics, and / or preselect (multiple) ML-based conditions with good metrics. GUI 207 is rendered in an automation interface, and the user can interact with the automation interface via client device 110 to select (multiple) action conditions, modify (multiple) preselected action conditions, and / or confirm the selected (automatic or manual) (multiple) action conditions.

[0049] Once the selected action(s) are confirmed, the GUI engine 120 provides the computer(s) and action(s) 209 to the allocation engine 126. The allocation engine 126 stores entries including the computer(s) and action(s) 209, and optionally, identifiers of the computer(s) and action(s) 209 for the user account to which they are being defined, in the automatic action database 156.

[0050] The automatic action engine 128 is capable of monitoring the satisfaction of action conditions based on assignments in the automatic action database 156, and, if determined to be satisfied, causing the execution of the (multiple) computer actions. For example, and continuing with the example, the automatic action engine 128 is capable of processing features of a user's subsequent video conferences using an ML-based conditional ML model for the (multiple) action conditions. If the processing generates a predicted output that satisfies a threshold, the automatic action engine 128 is capable of determining that the ML-based conditions are satisfied and thus automatically storing a recorded copy of the video conference. In some implementations, the automatic action engine 128 interfaces with one or more additional systems 130 to determine whether one or more action conditions are satisfied and / or to automatically execute one or more computer actions.

[0051] The determination of past data 203 and the generation of metrics for each ML-based condition 205 are in Figure 1BThe action 201 is illustrated as being executed in response to user input defining the computer action 201. However, in various implementations, previously generated data 203 and / or metrics for each ML-based condition 205 can be determined (i.e., preemptively) before the user input defining the computer action 201. In those implementations, various metrics can be pre-generated for various computer actions, and these metrics can each be optionally specific to a user or user group (e.g., organization). Therefore, in those implementations, the GUI generated based on metrics 207 can be rendered more quickly in response to user input defining the computer action 201.

[0052] Turn again Figure 1A The diagram also illustrates the training data engine 133, the training data database 158, and the training engine 136.

[0053] Training data engine 133 generates training instances for inclusion in training data database 158 for training ML models 152A-N. It should be understood that each training instance will be specific to only a single ML model within ML models 152A-N. Training data engine 133 generates training instances for training ML models 152A-N and / or for fine-tuning / personalizing (to a user or user group) one or more ML models within ML models 152A-N.

[0054] In some implementations, and with the permission of the associated user, the training data engine 133 automatically generates training data based on instances of past data from the past data database 154. As an example, suppose an ML model 152C is being trained (or fine-tuned) to predict whether an email meets the ML-based condition of "emails that need immediate attention." For such an ML-based condition, the training data engine 133 can generate positive training instances based on past data identifying past emails that were replied to by the user within one hour of receipt. For example, each training instance can include a training instance input with a positive label (e.g., "1") and a training instance output, the training instance input including features of such emails. Alternatively, for such an ML-based condition, the training data engine 133 can also be conditioned on past data identifying past emails that were responded to by the user outside of one hour of receipt, optionally conditioned on those emails that the user also viewed within one hour of receipt. For example, each training instance can include a training instance input and a training instance output with a negative label (e.g., "0"), the training instance input including features of such emails. The training data database 158 can additionally or alternatively include training instances labeled based on human review.

[0055] Training engine 136 utilizes training instances from training data database 158 in training ML models 152A-N. For example, training engine 136 can utilize training instances corresponding to ML model 152A in training ML model 152A, and can utilize training instances corresponding to ML model 152B in training ML model 152B, etc. As described herein (e.g., Figure 4 In some implementations, training engine 136 can train an ML model for a user or organization based on user- or organization-specific training instances. For example, training data engine 133 can automatically generate training instances for the ML model using past data 154 for the user or organization. In some implementations of those implementations, the ML model can be a pre-trained ML model based on similar training instances based on interactions with other users, and training for the user or organization can occur after pre-training. In other implementations, the ML model can be trained only based on user- or organization-specific training instances. Training engine 136 can store the ML model trained for the user or organization and an identifier indicating the user or organization. This identifier can then be used to process corresponding data using the ML model trained for the user or organization, instead of other models trained globally for the same ML-based conditions or for (multiple) other users or (multiple) other organizations.

[0056] refer to Figure 2A , Figure 2B , Figure 2C and Figure 2D The illustration shows examples of client device 110 presenting different automation interfaces. Each automation interface is based on a corresponding metric, and each of the corresponding metrics is also illustrated (above the illustration of client device 110) and is based on (a plurality of) corresponding computer actions defined via the automation interface.

[0057] Initial turn Figure 2A The user has already interacted with the Action Definition section 281 of the automation interface to define the action "Forward to jon@exampleurl.com" as an automated email action. Figure 2A In the example interface, the user has already selected "Forward to" from a dropdown menu that includes various email-related actions such as "Move to," "Reply," and "Send Notification to." The user has further provided, for example, an email address "jon@exampleurl.com" via a virtual keyboard.

[0058] The past data engine 124 (Figure 1) can be used to identify past user-initiated actions (e.g., initiated by a user connected to client device 110) that forwarded the corresponding email to "jon@exampleurl.com". Furthermore, the measurement engine 122 (Figure 1) can generate, based on the past data, the data illustrated in the figure. Figure 2A The metric 250A is located on the client device 110. Generated from past data based on action 282A, metric 250A is specific to action 282A. Metric 250A includes: metric 0.5 for ML model 152A (corresponding to the ML-based condition "new email with action item"); metric 0.9 for ML model 152B (corresponding to the ML-based condition "new email requiring immediate attention"); metric 0.1 for ML model 152C (corresponding to the ML-based condition "new email with customer question"); and metric 0.1 for ML model 152D (corresponding to the ML-based condition "new email with positive sentiment"). Each metric indicates the percentage of past emails forwarded to "jon@exampleurl.com" that are considered to satisfy the corresponding ML-based condition.

[0059] Based on metric 250A, multiple ML-based condition definition sections 283 are generated to include an indication 284BA for the ML-based condition "New emails requiring immediate attention," which is most prominent (positioned at the "top" of the ML-based conditions) based on having the "best metric" (0.9) and is pre-selected based on having a metric that meets a threshold (e.g., greater than 0.85). Further, based on the metric, section 283 is generated to include an indication 284AA for the ML-based condition "New emails with action items," which is positioned at the next most prominent position based on having the "second best metric" (0.5). Still further, based on the metric, section 283 is generated to include an indication 284CA for the ML-based condition "New emails containing customer questions," which is positioned at the next most prominent position based on having the "third best metric" (0.15). Finally, based on the metric, part 283 was generated to include an indicator 284DA that includes the ML-based condition "new emails with positive sentiment," which is positioned at the least prominent position based on having the "worst metric" (0.1). Each of indicators 284BA, 284AA, 284CA, and 284DA also illustrates the indicators of their metrics (90%, 50%, 15%, and 10%).

[0060] If the pre-selection of instruction 284BA is satisfied, the user can define the ML-based condition "new email requiring immediate attention" for action 282A by selecting submission interface element 288. The single selection can be, for example: touch input detected and pointed at submission interface element 288 on the touchscreen of client device 110; voice input detected and identified via microphone(s) of client device 110 (e.g., voice input of "Submit," "Select Submit Button," or "Done"); selection of submission interface element 288 via mouse paired with client device 110; or a touchless gesture pointed at submission interface element 288 and detected via radar or camera sensors of client device 110. Therefore, in this case, no user input is required to define the ML-based condition. Instead, only confirmation input is needed to select submission interface element 288, which results in action 282A and the ML-based condition "new email requiring immediate attention" being defined. Alternatively, users can interact with the automation interface to define additional or alternative ML-based conditions or even non-ML-based conditions (not illustrated for simplicity). Interaction with the automation interface can be performed through one or more of a variety of input modalities, such as touch, voice, gestures, keyboard, mouse, and / or other input modalities.

[0061] Next, turn to Figure 2B The user has interacted with the Action Definition section 281 of the automation interface to define the "Move to Action Item" action as an automated email action. Figure 2B In the example interface, the user has already selected "Move To" from a drop-down menu that includes various email-related actions, and the location of the "action item" (e.g., virtual folder location) has been further provided, for example, via a virtual keyboard.

[0062] In the past, engine 124 (Figure 1) can be used to identify past data of user-initiated actions (e.g., initiated by a user connected to client device 110) that moved the corresponding email to an "action item". Furthermore, metric engine 122 (Figure 1) can generate data based on this past data. Figure 2BThe metric 250B shown above the client device 110 is generated based on past data related to action 282B and is specific to action 282B. Metric 250B includes: metric 0.7 for ML model 152A (corresponding to the ML-based condition "new email with action item"); metric 0.5 for ML model 152B (corresponding to the ML-based condition "new email requiring immediate attention"); metric 0.2 for ML model 152C (corresponding to the ML-based condition "new email containing customer questions"); and metric 0.1 for ML model 152D (corresponding to the ML-based condition "new email with positive sentiment"). Each metric indicates the percentage of past emails moved to "action item" that are considered to satisfy the corresponding ML-based condition.

[0063] Based on metric 250B, multiple ML-based condition definition sections 283 are generated to include an instruction 284AB for the ML-based condition "new email with action item," which is positioned most prominently (at the "top" of the ML-based condition) based on its "best metric" (0.7). However, in Figure 2B In the example, indicator 284BA was not pre-selected because its metric (0.7) failed to meet a threshold (e.g., greater than 0.85). Further, based on the metric, part 283 was generated to include indicator 284BB, which is positioned next in prominence based on its "second-best metric" (0.5), according to the ML-based condition "new emails requiring immediate attention." Additionally, based on the metric, part 283 was generated to include indicator 284CB, which is positioned next in prominence based on its "third-best metric" (0.2), according to the ML-based condition "new emails containing customer questions." Finally, based on the metric, part 283 was generated to include indicator 284DB, which is positioned least prominence based on its "worst metric" (0.1), according to the ML-based condition "new emails with positive sentiment." Each of indicators 284AB, 284BB, 284CB, and 284DB also illustrates the indicators for their metrics (70%, 50%, 20%, and 10%).

[0064] Users can interact with the automation interface to define (multiple) ML-based conditions or even (multiple) non-ML-based conditions (not illustrated for simplicity).

[0065] Next, turn to Figure 2CThe user has interacted with the Action Definition section 281 of the automation interface to define the "Share with Patent Group" action as an automated cloud storage action. When executed automatically, the "Share with Patent Group" action automatically shares the corresponding document stored in the cloud storage device with the user accounts assigned to the "Patent Group" (thus making the corresponding document viewable and / or editable by those user accounts). Figure 2C In the example interface, the user has already selected an action from a drop-down menu that includes various cloud storage-related actions.

[0066] In the past, engine 124 (Figure 1) could be used to identify past data of user-initiated actions (e.g., actions initiated by a user connected to client device 110) that shared corresponding documents with the "patent group". Furthermore, measurement engine 122 (Figure 1) can generate data based on this past data. Figure 2C The metric 282C is illustrated above the client device 110. Metric 250C is generated from past data based on action 282C and is specific to action 282C. Metric 250C includes: metric 0.0 for ML model 152G (corresponding to the ML-based condition "time-sensitive"); metric 0.4 for ML model 152H (corresponding to the ML-based condition "important document"); and metric 0.9 for ML model 152I (corresponding to the ML-based condition "practice group relevant"). Each metric indicates the percentage of past documents shared with the "patent group" that are considered to satisfy the corresponding ML-based condition.

[0067] Based on metric 250C, the ML-based condition definition section 283 is generated to include an ML-based condition "practice group related" indication 284HC, which is most prominent based on having the "best metric" (0.9). Further, in Figure 2C In the example, indicator 284HC is preselected based on its metric (0.9) satisfying the preselection threshold. Further, based on the metric, section 283 is generated to include indicator 284IC, which is positioned as the next most prominent based on its "second-best metric" (0.4). Even further, based on the metric, section 283 is generated to omit any indicator for the ML-based condition "time-sensitive," which is based on its metric (0.0) failing to satisfy a display threshold (e.g., threshold 0.1).

[0068] If the pre-selection of instruction 284HC is satisfied, the user can use a single selection submission interface element 288 to define an ML-based condition for action 282C: "New emails that need immediate attention." Alternatively, the user can interact with the automation interface to define additional or alternative ML-based conditions(s) or even non-ML-based conditions(s) (not illustrated for simplicity).

[0069] Next, turn to Figure 2D The user has interacted with the Action Definition section 281 of the automation interface to define the actions "Make Offline Available" and "Add to Task List" as automated cloud storage actions. These actions, when executed automatically, make the corresponding documents stored in the cloud storage device available offline (e.g., downloaded locally to the client device) and add document-related information (e.g., title and links) to the task list (e.g., in a separate application). Figure 2D In the example interface, the user has already selected an action from a drop-down menu that includes various cloud storage-related actions.

[0070] The past data engine 124 (Figure 1) can be used to identify past user-initiated actions (e.g., those initiated by a user interfaced with client device 110) that make a document available offline and add it to a task list. Furthermore, the metric engine 122 (Figure 1) can generate metrics based on this past data. Figure 2D The metric 282D is illustrated above the client device 110. Metric 250D is generated from past data based on action 282D and is specific to action 282D. Metric 250D includes: metric 0.95 for ML model 152G (corresponding to the ML-based condition "time-sensitive"); metric 0.2 for ML model 152H (corresponding to the ML-based condition "important document"); and metric 0.3 for ML model 152I (corresponding to the ML-based condition "practice group relevant"). Each metric indicates the percentage of past documents that are "available offline" and "added to the task list" and are considered to satisfy the corresponding ML-based condition.

[0071] Based on metric 250D, multiple ML-based condition definition parts 283 are generated to include an ML-based condition "practice group related" indicator 284GD, which stands out based on having the "best metric" (0.95). Further, in Figure 2D In the example, indicator 284GD is preselected based on its metric (0.95) meeting the preselection threshold. Further, based on the metric, part 283 is generated to include indicators 284HD and 284ID, which are ML-based indicators categorized as "important document" and "relevant to practice groups," and are not preselected based on their worse metric (0.2) being positioned less prominent and their metric failing to meet the preselection threshold.

[0072] If the pre-selection of instruction 284GD is satisfied, the user can use a single selection submission interface element 288 to define an ML-based condition for action 282D: "New emails that need immediate attention." Alternatively, the user can interact with the automation interface to define additional or alternative ML-based conditions(s) or even non-ML-based conditions(s) (not illustrated for simplicity).

[0073] Figures 2A-2D The diagrams illustrate specific ML-based conditions and computer actions. However, those diagrams are provided as examples only, and it should be understood that the techniques disclosed herein can be combined with a variety of ML-based conditions and / or computer actions. As an example, the ML-based condition “instructing a new calendar event for a client meeting” can result in multiple computer actions such as “adding a 24-hour reminder before the calendar event” and “scheduling an hour on my calendar to prepare for the event.” As another example, the ML-based condition “chat message, email, or voicemail with a potential new customer” can include multiple computer actions such as “adding an e-reminder ‘reply to the potential new customer’” and “adding contact information to the CRM.”

[0074] For reference Figure 3 The flowchart describes an example method 300 for implementing selected aspects of this disclosure. For convenience, the operations in the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems. For example, the operations may be performed at client device 110 and / or at automated action system 118. Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0075] In box 352, the system receives one or more instances of user interface input via an automation interface, which defines one or more computer actions to be performed automatically in response to one or more action conditions. For example, the system can receive multiple instances of user interface input via user interaction with the automation interface.

[0076] In box 354, the system identifies data associated with a past user-initiated occurrence of (multiple) computer actions. For example, the system can identify data associated with a past user-initiated occurrence of (multiple) computer actions initiated by a user who provided the user interface input in box 352 and / or by a group in which the user is a member. As another example, the system can identify data associated with a past user-initiated occurrence of (multiple) computer actions initiated by individual users within a user group, who may not have any specific relationship to the user.

[0077] In box 356, the system selects an ML model for an ML-based condition among multiple action conditions. For example, the system can select an ML-based model for an ML-based condition related to (e.g., sharing a domain with) the multiple computer actions defined in box 352.

[0078] In box 358, the system generates predictions based on instances of data processed using an ML-based conditional ML model (from box 356) (from box 354). For example, the system can generate predictions (directly or indirectly) indicating the probability that instances of data will satisfy ML-based conditions represented by the ML model.

[0079] In box 360, the system determines whether there is more data to be processed. If so, the system returns to box 358 and generates another prediction based on another instance of the data. If not, the system proceeds to box 362.

[0080] In box 362, the system generates one or more metrics for ML-based conditions based on the predictions from the iterations of box 358 performed using an ML-based conditional ML model. For example, when the predictions are probabilities, the system is able to generate metrics based on the generated probabilities.

[0081] In box 364, the system determines whether another ML model exists that is related to the computer actions(s) defined in box 352 and has not yet been used in the iterations(s) of box 358. If yes, the system returns to box 356 and selects another ML model, then executes boxes 358, 360, and 362 based on the other ML model. If not, the system proceeds to box 366.

[0082] In box 366, the system renders the ML-based conditions based on metrics of the ML-based conditions determined in the iteration of box 362. For example, the system can use metrics to render, highlight, or automatically select “good” (metric-based) ML-based conditions and / or dilute / suppress “bad” (metric-based) ML-based conditions for (multiple) computer actions. Similarly, for example, the system can additionally or alternatively provide indications of metrics and ML-based conditions.

[0083] In box 368, the system, in response to a confirmation input received at the automation interface, assigns multiple ML-based conditions to multiple computer actions in box 352. The multiple ML-based conditions may be those selected upon receiving the confirmation input (based on user input or pre-selection without modification). Multiple non-ML-based conditions (e.g., rule-based) may be additionally or alternatively defined via the automation interface and, if they are ML-based conditions, assigned. The assignment of the multiple ML-based conditions to the multiple computer actions in box 352 may be user- or organization-specific, and, after assignment, may result in the automatic execution of computer actions in response to the satisfaction of the multiple ML-based conditions.

[0084] Although boxes 354, 356, 358, 360, 362, and 364 are illustrated between boxes 352 and 366, in various implementations, those boxes can be executed before boxes 352 and 366. For example, those boxes can be executed based on past data from multiple users for computer actions to generate corresponding metrics before box 352 occurs. Then, in response to box 352, the system can proceed directly to box 366 and use the corresponding metric when executing box 366.

[0085] For reference Figure 4 The flowchart describes an example method 400 for implementing selected aspects of this disclosure. For convenience, the operations in the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems. For example, the operations may be performed at client device 110 and / or at automated action system 118. Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0086] In box 452, the system identifies one or more criteria for actions based on ML-based conditions. For example, if the ML-based condition is “emails that need immediate attention,” one or more criteria could include replying to the email within one hour (or other threshold) of receiving the email. Similarly, for example, if the ML-based condition is “important document,” one or more criteria could include interacting with the document (e.g., viewing and / or editing) at least a threshold number of times (optionally in terms of duration).

[0087] In box 454, the system determines an instance of user or organization's data based on each of the instances associated with an action(s) that meets one or more criteria. For example, if one or more criteria include replying to an email within one hour (or other threshold) of receiving an email, then each instance of the data can include the characteristic of the corresponding email replied within one hour. Similarly, for example, if one or more criteria include interacting with a document at least a threshold number of times, then each instance of the data can include the characteristic of the corresponding document interacted with at least the threshold number of times.

[0088] In box 456, the system uses instances of the data from positive training instances when training a custom ML model for ML-based conditions. For example, the system can utilize features of instances of the data as input to positive training instances and can assign positive labels as outputs to positive training instances. The system can further train the custom ML model based on the positive training instances. The custom ML model can optionally be a model pre-trained on other training instances prior to training in box 456, including those not based on data from users or organizations.

[0089] In box 458, the system receives user input via an automation interface defining multiple computer actions and multiple action conditions for those actions, including ML-based conditions. For example, the user input can be provided via the automation interface described herein.

[0090] In box 460, the system uses a custom ML model to determine whether to automatically perform (multiple) computer actions. The system uses the custom ML model based on determining that the user interface input in box 458 originates from a user or organization. In other words, the system uses a custom ML model based on user interface input from box 458 and on user- or organization-specific training instances to determine whether ML-based conditions are met. The system may automatically perform (multiple) computer actions in response to determining that ML-based conditions are met (and optionally based on the satisfaction of one or more other action conditions).

[0091] Figure 5 This is a block diagram of an example computer system 510. Computer system 510 typically includes at least one processor 514 that communicates with multiple peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow users to interact with computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0092] User interface input device 522 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphic digitizer, a scanner, a touchscreen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of means and methods for inputting information into computer system 510 or a communication network.

[0093] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating visible images. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is intended to encompass all possible types of means and methods for outputting information from computer system 510 to a user or another machine or computer system.

[0094] Storage subsystem 524 stores the programming and data structures that provide functionality for some or all of the modules described herein. For example, storage subsystem 524 may include selected aspects of the logic that perform the methods described herein.

[0095] These software modules are typically executed by processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem can include multiple memories, including main random access memory (RAM) 530 for storing instructions and data during program execution and read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 524 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by the file storage subsystem 524 within the storage subsystem 524 or in other machines accessible by processor(s) 514.

[0096] Bus subsystem 512 provides a mechanism for enabling various components and subsystems of computer system 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0097] Computer systems 510 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 5The description of the computer system 510 depicted herein is intended only as a concrete example for illustrating some implementation methods. Many other configurations of the computer system 510 may have... Figure 5 The computer system depicted in the text has more or fewer components.

[0098] In situations where the systems described herein collect or may use personal information about users, users may be given the opportunity to control whether a program or function collects user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content is received from content servers that may be more relevant to the user. Similarly, before storing or using certain data, it may be processed in one or more ways to remove personally identifiable information. For example, a user's identity may be processed to the point that it is impossible to determine the user's personally identifiable information, or the user's geographic location may be generalized at the point where geographic location information is obtained (e.g., city, ZIP code, or state level) to make it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0099] In some implementations, a method is provided that includes receiving instances of user interface input(s) directed to an automation interface, wherein the instances of user interface input(s) define one or more computer actions to be automatically performed in response to satisfying one or more action conditions defined via the automation interface. The method further includes identifying correspondence data associated with multiple past occurrences of the one or more computer actions. The multiple past occurrences may optionally be user-initiated and non-automatically executed. The method further includes generating a correspondence metric for each of a plurality of machine learning-based conditions based on the correspondence data. Each correspondence metric indicates the frequency at which a corresponding machine learning-based condition among the plurality of machine learning-based conditions would be considered satisfied based on the correspondence data. The method further includes rendering an identifier for a given machine learning-based condition among the plurality of machine learning-based conditions at the automation interface. Rendering the identifier for the given machine learning-based condition is based on the correspondence metric for the given machine learning-based condition, and / or the content and / or display characteristics of the identifier are based on the correspondence metric for the given machine learning-based condition. The method further includes, in response to receiving confirmation of assigning the given machine learning-based condition as one of the one or more computer actions: assigning the given machine learning-based condition as one of the action conditions in one or more computer-readable media.

[0100] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0101] In some implementations, the content of the identifier is based on the corresponding metric, and the content includes a visual display of the corresponding metric.

[0102] In some implementations, the display characteristics of the identifier are based on a corresponding metric, and the display characteristics include the size of the identifier and / or the position of the identifier in the automation interface.

[0103] In some implementations, an identifier for a given machine learning-based condition is rendered based on the corresponding metric for that given machine learning-based condition satisfying a display threshold.

[0104] In some implementations, the method further includes preventing any identifiers of other machine learning-based conditions among a plurality of machine learning-based conditions from being rendered at the automation interface, wherein the prevention is based on a corresponding metric for the other machine learning-based condition. For example, the prevention could be based on a corresponding metric failing to meet a display threshold and / or failing to meet a threshold relative to a metric for the other machine learning-based condition (e.g., only the N machine learning-based conditions with the best metric can be rendered).

[0105] In some implementations, the method further includes pre-selecting an identifier for a given machine learning-based condition as one of the action conditions in the automation interface, based on a corresponding metric for that given machine learning-based condition. In some of those implementations, it is confirmed that another user interface input assigning the given machine learning-based condition to one or more computer actions is a selection of additional interface elements that appears without altering the pre-selection of the given machine learning-based condition. Pre-selecting the identifier for the given machine learning condition ensures that it satisfies a pre-selected threshold based on the corresponding metric and / or a threshold relative to a metric for other machine learning-based conditions (e.g., based on the metric being the best among all metrics).

[0106] In some implementations, generating a corresponding metric for a given machine learning-based condition based on corresponding data includes: processing the corresponding data using a given machine learning model for the machine learning-based condition to generate a plurality of corresponding values; and generating a metric based on the plurality of corresponding values. In some implementations of those implementations, the plurality of corresponding values ​​are probabilities, and generating the metric includes generating the metric based on the probabilities.

[0107] In some embodiments, the method further includes receiving additional user interface input defining one or more rule-based conditions, and in response to the additional user interface input, assigning one or more rule-based conditions as additional action conditions in one or more computer-readable media, the satisfaction of which leads to the automatic execution of one or more computer actions. In those embodiments, confirming the assignment of the given machine learning-based condition to one or more computer actions also confirms the assignment of one or more rule-based conditions. In some versions of those embodiments, both the one or more rule-based conditions and the given machine learning-based condition are assigned to be satisfied to lead to the automatic execution of one or more computer actions. In some other versions of those embodiments, the given machine learning-based condition, when satisfied alone, leads to the automatic execution of one or more computer actions.

[0108] In some implementations, identifying corresponding data includes identifying corresponding data based on whether the corresponding data is for a user providing user interface input or for an organization in which the user is a verified member.

[0109] In some implementations, one or more actions include modifying corresponding content, transmitting the corresponding content to one or more recipients other than the user, and / or presenting a push notification of the corresponding content to the user. In some versions of those implementations, the corresponding content is corresponding electronic communication. For example, the corresponding electronic communication could be an email, chat message, or voicemail (e.g., its transcription). In some other or alternative versions, the method further includes, after assigning a given machine learning-based condition as one of the action conditions: receiving given content of the corresponding content; determining that the given machine learning-based condition is satisfied; and automatically performing one or more actions based on the determination that the given machine learning-based condition is satisfied. Determining that the given machine learning-based condition is satisfied can include processing features of the given content using a given machine learning model for the given machine learning-based condition to generate a value, and determining that the given machine learning-based condition is satisfied based on that value.

[0110] In some implementations, corresponding data that identifies multiple past occurrences of one or more computer actions and corresponding metrics generated based on the corresponding data both appear before one or more instances receiving user interface input.

[0111] In some implementations, a method is provided that includes: identifying one or more criteria for actions indicative of machine learning-based conditions, given a machine learning-based condition; identifying a corresponding instance of data for a user or organization based on each of instances of data associated with one or more corresponding computer actions that satisfy one or more criteria; and using the corresponding instance of user or organization-specific data and a positive label as a positive training instance in training a customized machine learning model for the machine learning-based condition. The method further includes, after training the customized machine learning model, (1) receiving one or more instances of user interface input to an automation interface, wherein the one or more instances of user interface input define one or more computer actions to be automatically performed in response to the satisfaction of one or more action conditions, and the one or more action conditions include machine learning-based conditions; and (2) using the customized machine learning model, based on one or more instances of user interface input from another user of the user or organization, and based on the machine learning-based conditions included in the defined one or more action conditions, to determine whether one or more computer actions are satisfied to determine whether one or more computer actions are automatically performed.

[0112] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0113] In some implementations, the method further includes, before receiving one or more instances of user interface input: identifying one or more negative criteria for actions that do not indicate machine learning-based conditions, given a machine learning-based condition; identifying instances of corresponding negative data for a user or organization based on each of instances of data associated with one or more corresponding computer actions that satisfy one or more negative criteria; and using the instances of corresponding negative data and negative labels as negative training instances in training a customized machine learning model.

[0114] In some implementations, one or more criteria include responding to electronic communications for a threshold duration.

[0115] In some implementations, a method is provided that includes receiving one or more instances of user interface input directed to an automation interface. The one or more instances of user interface input define one or more computer actions to be automatically performed in response to the satisfaction of one or more action conditions defined via the automation interface. The method further includes rendering an identifier of a given machine learning-based condition among a plurality of machine learning-based conditions at the automation interface. Rendering the identifier of the given machine learning-based condition is based on one or more computer actions, and / or the content and / or display characteristics of the identifier are based on one or more computer actions. The method further includes responding to receiving another user interface input confirming the assignment of the given machine learning-based condition as one of the one or more computer actions: assigning the given machine learning-based condition as one of the action conditions in one or more computer-readable media.

[0116] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0117] In some embodiments, the method further includes identifying correspondence data associated with multiple past occurrences of one or more computer actions; and generating a correspondence metric for each of the machine learning-based conditions based on the correspondence data. In those embodiments, an identifier for a given machine learning-based condition is rendered based on one or more computer actions, with content and / or display characteristics based on the correspondence metric and / or identifier for the given machine learning-based condition, and based on the one or more computer actions, with its content and / or display characteristics based on the correspondence metric for the given machine learning condition. In some embodiments of those embodiments, past occurrences are user-initiated and non-automatically executed, and / or the correspondence metrics each indicate the frequency with which a corresponding machine learning-based condition among the multiple machine learning-based conditions based on the correspondence data will be considered satisfied.

[0118] In some implementations, before receiving one or more instances of user interface input defining one or more computer actions, an identifier given machine learning-based conditions is initially rendered at the automation interface with initial content and / or display characteristics. In some implementations of those implementations, rendering the identifier includes rendering the identifier with content and / or display characteristics that differ from the initial content and / or display characteristics.

Claims

1. A method implemented by one or more processors, the method comprising: Receive one or more instances of user interface input to an automation interface, wherein the one or more instances of user interface input define one or more computer actions to be automatically performed in response to the satisfaction of one or more action conditions defined via the automation interface; Identifying corresponding data associated with multiple past occurrences of the one or more computer actions; Based on the corresponding data, a corresponding metric is generated for each of a plurality of machine learning-based conditions, wherein each corresponding metric indicates the frequency with which a corresponding machine learning-based condition among the plurality of machine learning-based conditions will be considered satisfied based on the corresponding data; The identifier of a given machine learning-based condition among the plurality of machine learning-based conditions is rendered at the automation interface. Wherein, the identifier of the given machine learning-based condition is rendered based on the corresponding metric for the given machine learning-based condition, and / or Wherein, the content and / or display characteristics of the identifier are based on the corresponding metric for the given machine learning-based condition; Another user interface input in response to receiving confirmation to assign the given machine learning-based condition to the one or more computer actions: The given machine learning-based condition assignment is used as one of the action conditions in one or more computer-readable media.

2. The method according to claim 1, wherein, The content of the identifier is based on the corresponding metric, and the content includes a visual display of the corresponding metric.

3. The method according to claim 1, wherein, The display characteristics of the identifier are based on the corresponding metric, and the display characteristics include the size of the identifier and / or the position of the identifier in the automation interface.

4. The method according to claim 1, wherein, The identifier of the given machine learning-based condition is rendered based on the corresponding metric for the given machine learning-based condition satisfying a display threshold.

5. The method of claim 1, further comprising: Prevent any identifier of any of the additional machine learning-based conditions among the plurality of machine learning-based conditions from being rendered at the automation interface, wherein the prevention is based on the corresponding metric for the additional machine learning-based condition failing to meet the display threshold.

6. The method of claim 1, further comprising: Based on the fact that the corresponding metric for the given machine learning-based condition satisfies a pre-selected threshold, the identifier of the given machine learning-based condition in the automation interface is pre-selected as one of the action conditions. Specifically, the selection of another interface element that confirms the assignment of the given machine learning-based condition to the one or more computer actions is a selection of other interface elements that appears without changing the pre-selected other interface inputs of the given machine learning-based condition.

7. The method according to claim 1, wherein, Generating the corresponding metric for the given machine learning-based condition based on the corresponding data includes: The corresponding data is processed using a given machine learning model tailored to the machine learning-based conditions to generate multiple corresponding values; and The metric is generated based on the multiple corresponding values.

8. The method according to claim 7, wherein, The plurality of corresponding values ​​are probabilities, and generating the metric includes generating the metric based on the probabilities.

9. The method of claim 1, further comprising: Receive additional user interface input that defines one or more rule-based conditions; The other user interface input confirms the assignment of the given machine learning-based condition to the one or more computer actions, and confirms the assignment of the one or more rule-based conditions; and Further includes responding to the other user interface input: The one or more rule-based conditions are assigned in one or more computer-readable media as additional action conditions that, when satisfied, lead to the automatic execution of the one or more computer actions.

10. The method according to claim 9, wherein, The one or more rule-based conditions and the given machine learning-based conditions are assigned to be satisfied in order to cause the automatic execution of the one or more computer actions.

11. The method according to claim 9, wherein, The given machine learning-based conditions, when individually satisfied, lead to the automatic execution of the one or more computer actions.

12. The method according to claim 1, wherein, Identifying the corresponding data includes identifying the corresponding data based on whether the corresponding data is for a user who provided the user interface input or for an organization where the user is a verified member.

13. The method according to claim 1, wherein, The one or more computer actions include: Modify the corresponding content, transmit the corresponding content to one or more recipients other than the user, and / or cause a push notification of the corresponding content to be presented to the user.

14. The method according to claim 13, wherein, The corresponding content refers to the corresponding electronic communication.

15. The method of claim 13 or 14, further comprising, after assigning the given machine learning-based condition as one of the action conditions: Receive the given content corresponding to the content; Determine whether the given machine learning-based conditions are met, wherein, Determining whether the given machine learning-based conditions are met includes: The features of the given content are processed using a given machine learning model tailored to the given machine learning conditions to generate values; and Based on the value, determine whether the given machine learning-based condition is satisfied; and The one or more actions are automatically performed based on determining that the given machine learning-based conditions are met.

16. The method according to claim 1, wherein, The corresponding data that identifies the multiple past occurrences associated with the one or more computer actions and the corresponding metric generated based on the corresponding data both appear before one or more instances receiving the user interface input.

17. A method implemented by one or more processors, the method comprising: For a given machine learning-based condition, identify one or more criteria for actions indicative of the machine learning-based condition; Each instance of user or organization data is associated with one or more corresponding computer actions that meet one or more of the criteria to determine the corresponding instance of the data. The corresponding instance and positive label of the data specific to the user or the organization are used as positive training instances for training the customized machine learning model based on the machine learning conditions. Receive one or more instances of user interface input directed to the automation interface, wherein the one or more instances of user interface input are defined as follows: One or more computer actions to be automatically executed in response to the fulfillment of one or more action conditions, and One or more action conditions, including the machine learning-based conditions; The one or more instances based on user interface input originate from the user or another user within the organization, and are included in one or more defined action conditions based on the machine learning-based conditions: The custom machine learning model is used to determine whether the conditions for the one or more computer actions are met in determining whether to automatically execute the one or more computer actions.

18. The method of claim 17, further comprising, before receiving one or more instances of the user interface input: For the given machine learning-based condition, identify one or more negative criteria for actions that do not indicate the machine learning-based condition; The corresponding instance of negative data for the user or the organization is determined by associating each of the instances of the data with one or more corresponding computer actions that satisfy one or more negative criteria. The corresponding instances of the negative data and their negative labels are used as negative training instances for training the custom machine learning model.

19. The method according to claim 17 or claim 18, wherein, The one or more criteria include responding to electronic communications within a threshold duration.

20. A method implemented by one or more processors, the method comprising: Receive one or more instances of user interface input to an automation interface, wherein the one or more instances of user interface input define one or more computer actions to be automatically performed in response to the satisfaction of one or more action conditions defined via the automation interface; The identifier of a given machine learning-based condition among multiple machine learning-based conditions is rendered at the automation interface. Wherein, the identifier of the given machine learning-based condition is rendered based on the one or more computer actions, and / or The content and / or display characteristics of the identifier are based on the one or more computer actions; In response to receiving confirmation of assigning the given machine learning-based condition to the one or more computer actions, another user interface input is provided: The given machine learning-based condition assignment is used as one of the action conditions in one or more computer-readable media.

21. The method of claim 20, further comprising: Identifying corresponding data associated with multiple past occurrences of the one or more computer actions, the past occurrences being user-initiated and not executed automatically; Based on the corresponding data, a corresponding metric is generated for each of the machine learning-based conditions, wherein each corresponding metric indicates the frequency with which the corresponding machine learning-based condition among the plurality of machine learning-based conditions will be considered satisfied based on the corresponding data; The identifier of the given machine learning-based condition is rendered based on the corresponding metric for the given machine learning-based condition based on the one or more computer actions.

22. The method of claim 20, further comprising: Identifying corresponding data associated with multiple past occurrences of the one or more computer actions, the past occurrences being user-initiated and not executed automatically; Based on the corresponding data, a corresponding metric is generated for each of the machine learning-based conditions, wherein each corresponding metric indicates the frequency with which the corresponding machine learning-based condition among the plurality of machine learning-based conditions will be considered satisfied based on the corresponding data; Wherein, the content and / or display characteristics of the identifier are based on the one or more computer actions, and the content and / or display characteristics are based on the corresponding metric for the given machine learning-based condition.

23. The method of claim 20, wherein, Before receiving one or more instances of the user interface input defining the one or more computer actions, the identifier given the machine learning-based conditions is initially rendered at the automation interface with initial content and / or display characteristics, wherein rendering the identifier includes rendering the identifier with the content and / or display characteristics, and wherein the content and / or display characteristics are different from the initial content and / or display characteristics.

24. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 23.

25. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 23.

26. A system comprising one or more processors for performing the method according to any one of claims 1 to 23.

Images