Enhanced Processing for Communication Workflows Using Machine Learning Techniques

Abstract

The present disclosure relates generally to using machine learning techniques to evaluate communication workflows consisting of tasks. More specifically, the present disclosure relates to systems and methods for generating predictions of task outcomes of a communication workflow, generating recommendations of one or more tasks to add to a partial communication workflow to complete the communication workflow, and generating a vector representation of the communication workflow.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This disclosure claims priority to U.S. patent application Ser. No. 16 / 800,869, filed Feb. 25, 2020, for “Enhanced Processing For Communication Workflows Using Machine-Learning Techniques,” which is incorporated by reference in its entirety.

[0002] Technical Field The present disclosure relates generally to using machine learning techniques to evaluate communication workflows consisting of tasks. More specifically, the present disclosure relates to systems and methods for using machine learning or artificial intelligence to generate predictions of task outcomes of a communication workflow, generate recommendations of one or more tasks to add to a partial communication workflow to complete the workflow, and generate a vector representation of the communication workflow. [Background technology]

[0003] background A workflow may be configured to trigger the sending of a communication to a user at a certain time. For example, code may be executed to trigger one or more communications to a user device via a communication channel at a given time. Multiple communications via different communication channels may be coordinated in a workflow to engage an individual user. However, evaluating the likelihood of a task resulting in a target action being performed by a user is a technical challenge at a big data scale due to the complexity of workflows. The decision of which tasks to include in a workflow is often made manually without leveraging previous workflows. Furthermore, given the complexity of task configurations in various workflows, evaluating workflows against each other is inefficient and computationally burdensome. Summary of the Invention

[0004] overview Some aspects and features of the present disclosure relate to processing a communication workflow using machine learning or artificial intelligence. The communication workflow may include an ordered sequence of one or more tasks. Executing the communication workflow may include executing the one or more tasks in an ordered sequence. For example, executing a task of the communication workflow may include causing a server to send a communication (e.g., a digital message) to a target group of user devices over a communication channel. The communication may be configured to enable a user operating a user device of the target group to perform one or more actions (e.g., select a link included in the communication). An indication of the action performed at the user device may be received at a cloud-based application. Once the task of the communication workflow is executed, the cloud-based application may determine a task result of the communication workflow. The task result may represent an overall performance metric of the communication workflow. For example, the overall performance metric may include a percentage of user devices (out of the target group of user devices) that performed a target action (e.g., selected a link included in the communication) aggregated across all tasks of the communication workflow.

[0005] To illustrate by way of non-limiting example only, a communication workflow may include a sequence of two tasks; a first task is an email communication task and a (subsequent) second task is a text message task. The email task may include sending an email communication to a target group of mobile devices. The text message task may later include sending a text message (e.g., Short Message Service (SMS)) to a subset of the target group of mobile devices. If the email communication is received at the target group of mobile devices, the email communication may include a selectable link. If the text message is received at the subset of the target group of mobile devices, the text message may also include a selectable link. The cloud-based application may receive an indication of whether a link was selected at the target group of mobile devices through the email communication or through the text message. The cloud-based application may determine a task result of the communication workflow (e.g., an overall performance metric for both tasks) as a percentage of users who selected the link through either the email communication or the text message. As another example, the cloud-based application may add up a partial send count (e.g., the number of communications sent) for each of the email communication task and the text message task of the communication workflow. The cloud-based application may also add up partial link selection counts (e.g., the number of selected links) for each of the email communication tasks and the text message tasks. The cloud-based application may then determine the task results of the communication workflow by determining a ratio of the total link selection count to the total transmission count.

[0006] According to some aspects described herein, task outcomes of a communication workflow can be predicted by generating a composite feature vector that represents the communication workflow and inputting the composite feature vector into a machine learning model. The machine learning model may be trained using previously executed communication workflows and the corresponding task outcomes of those previously executed communication workflows. Additionally, some aspects and features may relate to using machine learning techniques to generate recommendations of tasks to add to a partial communication workflow to complete the partial communication workflow. A set of tasks to complete the partial communication workflow can be recommended by inputting the composite feature vector of the partial communication workflow into a machine learning model and generating an output representing tasks that resulted in a particular value of task outcome of the previously executed communication workflow (e.g., a value of the task outcome above a threshold). Some aspects and features may also relate to using a neural network (e.g., a Graph2Vec model) to generate a vector representation of the communication workflow in a computationally efficient manner. The vector representation can be input into various machine learning models to perform various functions. According to some aspects of the present disclosure, the vector representation may represent the entire communication workflow, including the structure, tasks, and task content.

[0007] In some implementations, the cloud-based application may generate a composite feature vector representing individual previously executed communication workflow tasks. The composite feature vector may be generated using a multi-dimensional machine learning feature vector (e.g., singular value decomposition (SVD)). The composite feature vector can be input into a machine learning model to generate an output predicting a new communication workflow task outcome (e.g., an overall performance metric or overall rate of target actions being performed by a user receiving a communication) or an output representing a recommendation of tasks to include in a new partial or incomplete communication workflow.

[0008] In some examples, the communication workflow may be a marketing campaign orchestration (MCO). The communication workflow may include an ordered sequence of tasks. The tasks may be configured to send digital communications to a group of target user devices (e.g., audiences). In some examples, a composite feature vector may be used as a standardized representation of a communication workflow that represents the tasks, parameters, target users, and content of the communication workflow. To generate the composite feature vector, a multi-dimensional machine learning feature vector may be generated. Each task of the ordered sequence of tasks in the communication workflow may be represented by a task vector (e.g., feature vector). A group of target user devices targeted to receive a communication as a result of the execution of a task may be represented by a group vector. In some examples, the structure of the communication workflow and the component task vectors may be combined to generate a composite feature vector representing the communication workflow.

[0009] The ordered sequence of tasks included in a communication workflow may represent the structure of the workflow. The structure of each communication workflow may include, for example, one or more nodes (e.g., a start node, a split node, and an end node) or one or more stages (e.g., a task such as sending an email to a set of user devices) connected to each other by one or more branches. A branch may include a sequence flow of tasks (e.g., a filter task, a send task, or a delay task). For example, a node may represent a split task in which a target group of user devices is split into two or more subgroups. In some examples, the structure of a communication workflow may include a root node (e.g., a start task indicating the start of a communication workflow), intermediate nodes (e.g., a communication task that, when executed, sends a communication such as an email to a user device), and leaf nodes (e.g., an end node indicating the end of a communication workflow). Each tree branch may include a sequence of one or more tasks. A composite feature vector can be generated to represent the overall structure of the communication workflow, the tasks, and the content of the communication of each task.

[0010] In some implementations, the cloud-based application may input the composite feature vector of the communication workflow into a machine learning model to generate an output that predicts a task outcome of the communication workflow. The task outcome may represent the performance of the communication workflow, including, for example, a click-through rate or a conversion rate of a target group of users targeted by the communication workflow. In some implementations, given a new partial communication workflow (e.g., one that is under development and not yet completed), the composite feature vector of the previously executed partial parts of the workflow and the composite feature vector of the new partial communication workflow can be evaluated using the machine learning model. According to these implementations, the machine learning model may generate an output that represents a recommendation of one or more tasks to add to the new partial communication workflow to complete the new partial communication workflow.

[0011] In some implementations, the communication workflow may be analyzed as a graph using machine learning techniques (e.g., Graph2Vec) to learn a rooted subgraph for each node in the graph. The machine learning techniques may generate a vector representation of the entire workflow without separating the structure of the communication workflow from the composite feature vector of the communication workflow. The vector representation, according to these implementations, may generate an n-dimensional vector that represents the entire communication workflow including the structure, tasks, and task content. As a non-limiting example, a neural network of the Graph2Vec model may be run to learn the rooted subgraph embedding of the communication workflow in an unsupervised manner. The resulting vector representation may be input to a machine learning model to generate predictions of task outcomes for new communication workflows or to generate recommendations of tasks to add to partial communication workflows to complete the partial communication workflow, as described above.

[0012] In some implementations, a system of one or more computers can be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action during operation. One or more computer programs can be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the data processing device to perform an action. One general aspect includes a computer-implemented method, the method including: accessing a workflow including one or more tasks arranged in a sequential order, the workflow configured to facilitate interaction with a set of user devices, each task of the one or more tasks including executable code that, when executed, performs a function (e.g., split, filter, send) associated with the set of user devices, the workflow being associated with one or more parameters (e.g., vocabulary for topic modeling, content images, etc.) characterizing each task of the one or more tasks of the workflow, the method further including generating a composite feature vector representing the workflow, the composite feature vector being generated using feature vectors for each task of the one or more tasks of the workflow, and The feature vector for each task of the one or more tasks is generated by performing one or more machine learning techniques using the one or more parameters characterizing the task, the method further comprising inputting the composite feature vector of the workflow to a trained machine learning model, the trained machine learning model being trained to generate a prediction of a performance value of the workflow, the trained machine learning model being trained using a training dataset representing one or more previously executed workflows and corresponding previous performance values ​​of each of the one or more previously executed workflows, and the method further comprises generating an output using the trained machine learning model, the output comprising a prediction of the performance value of the workflow.Other embodiments of this aspect include corresponding computer systems, devices, and executable code stored on a non-transitory computer-readable storage medium, each configured to perform the operations of the method.

[0013] Implementations may include one or more of the following features: The computer-implemented method further includes determining a structure of the workflow, the structure represented by a plurality of nodes of a tree structure, two nodes of the plurality of nodes of the tree structure being connected by one or more stages, and each task of the one or more tasks of the workflow corresponding to a node of the plurality of nodes or a stage of the one or more stages, and the method further includes evaluating the training data set to determine whether the structure of the workflow matches a structure of at least one previously executed workflow of the one or more previously executed workflows. The computer-implemented method further includes determining that the structure of the workflow matches (e.g., is the same as) a structure of a group of previously executed workflows of the one or more previously executed workflows of the training data set. The computer-implemented method may also include comparing the composite feature vector of the workflow to a composite feature vector of each previously executed workflow of the group of previously executed workflows in a domain space. The computer-implemented method may also include selecting a subgroup of the group of previously executed workflows based on a threshold and a result of the comparison. The computer-implemented method may also include identifying a previous performance value for each previously executed workflow of the subgroup of previously executed workflows. The computer-implemented method may also include generating the output predicting the performance value of the workflow by determining a combination of the previous performance values ​​associated with the subgroup of previously executed workflows. The computer-implemented method further includes determining that the structure of the workflow matches a structure of a group of previously executed workflows of the one or more previously executed workflows of the training dataset.During generation of the training dataset, one or more subsets of previously executed workflows are determined, each subset of the one or more subsets corresponding to two or more previously executed workflows sharing a common structure. The computer-implemented method may also include performing a clustering operation (k-means clustering) on ​​each subset of the one or more subsets of previously executed workflows, each subset of previously executed workflows being associated with one or more clusters of previously executed workflows, the one or more clusters being formed by performing the clustering operation on the composite feature vector of the subset of previously executed workflows in a domain space, each previously executed workflow included in a cluster being associated with a composite feature vector corresponding to the composite feature vector of other previously executed workflows in that cluster, the method further including generating the output predicting the performance value of the workflow, the generating may be performed by identifying a particular subset of the one or more subsets of the training dataset whose corresponding structure corresponds to the structure of the workflow. The computer-implemented method may also include assigning the workflow to a particular one of the one or more clusters corresponding to the particular subset based on a comparison of the composite feature vector of the workflow to a feature vector of each of the one or more clusters. The computer-implemented method may also include determining a combination of the previous performance values ​​associated with the previously executed workflows associated with the particular cluster to which the workflow is assigned.The computer-implemented method further includes determining that the structure of the workflow matches a structure of a group of previously executed workflows of the one or more previously executed workflows of the training dataset, wherein during generation of the training dataset, one or more subsets of previously executed workflows are identified, each subset of the one or more subsets corresponding to two or more previously executed workflows sharing a common structure. The computer-implemented method also includes a supervised machine learning model is trained for each subset of the one or more subsets of the previously executed workflows, the method further includes generating the output predicting the performance value of the workflow, wherein the generating may be performed by identifying a particular subset of the one or more subsets of the training dataset whose corresponding structure matches the structure of the workflow, inputting the composite feature vector of the workflow into the supervised machine learning model corresponding to the particular subset, and generating the output predicting the performance value of the workflow. The computer-implemented method further includes determining that the structure of the workflow does not match a structure of any previously-executed workflow of the one or more previously-executed workflows of the training data set, and generating the output predicting the performance value of the workflow, wherein the generating is performed by segmenting each previously-executed workflow of the one or more previously-executed workflows of the training data set into a plurality of previously-executed sub-workflows. The computer-implemented method may also include comparing the structure of the workflow to each sub-workflow of the plurality of sub-workflows of each previously-executed workflow of the one or more previously-executed workflows of the training data set.The computer-implemented method may also include identifying a group of previously executed sub-workflows from the plurality of previously executed sub-workflows that match the structure of the workflow. The computer-implemented method may also include comparing the composite feature vector of the workflow to a composite feature vector of each previously executed sub-workflow of the group of previously executed sub-workflows in a domain space. The computer-implemented method may also include selecting a subgroup of the group of previously executed sub-workflows based on a threshold and a result of the comparison. The computer-implemented method may also include determining a predicted performance value of the workflow based on the previous performance values ​​associated with the selected subgroup of previously executed sub-workflows. The computer-implemented method may further include the workflow being defined by a list that includes metadata describing the structure of the workflow. The computer-implemented method may also include each task included in the workflow. The computer-implemented method may also include each feature vector representing a task of the one or more tasks of the workflow. The computer-implemented method may also include when the list is parsed to perform one or more functions. An implementation of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0014] In some implementations, a system of one or more computers may be configured to perform particular operations or actions by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs may be configured to perform particular operations or actions by including instructions that, when executed by a data processing device, cause the data processing device to perform the actions.One general aspect includes a computer-implemented method, the method including providing an interface to enable a user to define a workflow, the workflow being configurable to include a set of tasks facilitating interaction with a set of user devices, each task of the set of tasks including executable code that, when executed, performs a function associated with the set of user devices (e.g., splitting, filtering, sending), the method further including receiving, at the interface, an input corresponding to a selection of one or more tasks from the set of tasks, the one or more tasks defining a partial workflow, the partial workflow being associated with one or more parameters (e.g., vocabulary for topic modeling, content images, etc.) characterizing each task of the one or more tasks of the partial workflow, the method further including identifying a structure of the partial workflow, the structure of the partial workflow being associated with one or more Each of the one or more tasks of the node and / or one or more stages of the partial workflow corresponds to a node of the one or more nodes or a stage of the one or more stages, the method further includes generating a composite feature vector representing the partial workflow, the composite feature vector being generated using the feature vector of each of the one or more tasks of the workflow, the feature vector of each of the one or more tasks being generated by performing one or more machine learning techniques using the one or more parameters characterizing the task (e.g., constructing a machine learning feature vector based on parameters of the node and / or task of the new partial workflow), the method further includes accessing a set of previously executed partial workflows, each previously executed partial workflow of the previously executed partial workflows being represented by a structure, a composite feature vector, and a performance value. The computer-implemented method also includes selecting a subset of the set of previously executed partial workflows, the subset of previously executed partial workflows sharing the same structure as the partial workflow.The computer-implemented method also includes determining, from a subset of the previously-executed partial workflows, one or more previously-executed partial workflows that are similar to the partial workflow, wherein the similarity is based on a comparison between a composite feature vector of each previously-executed partial workflow of the subset and the composite feature vector of the partial workflow, the method further includes generating a recommendation for completing the partial workflow, the recommendation including one or more recommended tasks for completing the partial workflow, the one or more recommended tasks being selected from one or more remaining tasks of a previously-executed partial workflow of the one or more previously-executed partial workflows that share a same structure as the partial workflow and are determined to be similar to the partial workflow, the selection being based on the previous performance values ​​of the one or more previously-executed partial workflows that share the same structure as the partial workflow and are determined to be similar to the partial workflow, and the method further includes displaying the recommendation on the interface. Other embodiments of this aspect include corresponding computer systems, apparatus, and executable code or instructions (e.g., computer program products) stored on a non-transitory computer-readable storage medium, each configured to perform the operations of the method.

[0015] Implementations may include one or more of the following features: The computer-implemented method, wherein comparing the composite feature vector of each previously executed partial workflow from a subset of previously executed partial workflows that share the same structure as the partial workflow with the composite feature vector of the partial workflow includes: for each previously executed partial workflow, calculating a distance in domain space between the composite feature vector of the previously executed partial workflow and the composite feature vector of the partial workflow; comparing the distance to a threshold; and determining that the previously executed partial workflow is similar to the partial workflow if the distance is less than or equal to the threshold and determining that the previously executed partial workflow is not similar to the partial workflow if the distance is greater than the threshold. The computer-implemented method further includes ordering the one or more previously executed partial workflows that share the same structure as the partial workflow and that are determined to be similar to the partial workflow by ordering the one or more previously executed partial workflows from highest performance value to lowest performance value. The computer-implemented method further includes receiving an additional input at the interface corresponding to a selection of one or more recommended tasks for completing the partial workflow. The computer-implemented method may also include, in response to receiving the additional input, sequentially adding the one or more recommended tasks to the one or more tasks to represent a complete workflow. The computer-implemented method further includes receiving an additional input at the interface, the additional input corresponding to a selection of at least one recommended task of the one or more recommended tasks and at least one task not included in the one or more recommended tasks, wherein the selection of the at least one recommended task and the at least one task not included in the one or more recommended tasks completes the partial workflow.The computer-implemented method may also include, in response to receiving the additional input, sequentially adding the at least one recommended task and the at least one task not included in the one or more recommended tasks to the one or more tasks to represent a complete workflow. The computer-implemented method defines a partial or complete workflow by a list including metadata describing the structure of the workflow, each task included in the workflow, and each feature vector representing a task of the set of tasks included in the workflow. The computer-implemented method further includes parsing the list to perform one or more functions. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0016] In some implementations, a system of one or more computers can be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action during operation. One or more computer programs can be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the data processing device to perform an action. One general aspect includes a computer-implemented method, the method including: accessing a workflow including one or more tasks arranged in a sequential order, the workflow configured to facilitate interaction with a set of user devices, each task of the one or more tasks including executable code that, when executed, performs a function (e.g., split, filter, send) associated with the set of user devices, the workflow being associated with one or more parameters (e.g., vocabulary for topic modeling, content images, etc.) characterizing each task of the one or more tasks of the workflow, the method further including defining a tree structure of the workflow, the tree structure being configured to facilitate interaction with a set of user devices, the workflow including one or more tasks arranged in a sequential order, the workflow being configured to facilitate interaction with a set of user devices, each task of the one or more tasks including executable code that, when executed, performs a function (e.g., split, filter, send) associated with the set of user devices, the workflow being associated with one or more parameters (e.g., vocabulary for topic modeling, content images, etc.) characterizing each task of the one or more tasks of the workflow, the method further including defining a tree structure of the workflow, the tree structure being configured to facilitate interaction with a set of user devices, the workflow including one or more tasks arranged in a sequential order ... The tree structure includes a plurality of nodes and one or more stages, two nodes of the plurality of nodes of the tree structure are connected by at least one stage of the one or more stages, and each task of the one or more tasks of the workflow corresponds to a node of the plurality of nodes or a stage of the one or more stages, the method further includes inputting the tree structure to a machine learning model (e.g., a graph-to-vector model), the inputting of the tree structure to the machine learning model results in sampling one or more sub-trees of the tree structure of the workflow, and the method further includes generating a vector representation of the tree structure of the workflow. Other embodiments of this aspect include corresponding computer systems, devices, and executable code stored on a non-transitory computer-readable storage medium, each configured to perform operations of the method.

[0017] Implementations may include one or more of the following features: The computer-implemented method further includes accessing a set of previously executed workflows, defining a tree structure for each previously executed workflow of the set of previously executed workflows, inputting the tree structure for each previously executed workflow of the set of previously executed workflows into a machine learning model, generating a vector representation representing the tree structure for each previously executed workflow of the set of previously executed workflows, and storing the vector representation of each previously executed workflow in a training dataset. In the computer-implemented method, the vector representation of the tree structure is generated without evaluating vector representations of individual nodes of the plurality of nodes or individual stages of the one or more stages of the tree structure. In the computer-implemented method, each node of the plurality of nodes of the tree structure or each stage of the one or more stages is associated with metadata used to characterize the task or portion of the workflow. In the computer-implemented method, the metadata associated with each node or stage represents a function performed by a task associated with the node or stage, respectively. In the computer-implemented method, the machine learning technique is a neural network configured to learn an embedding of the tree structure of the workflow. In the computer-implemented method, the evaluation of the one or more nodes of the tree structure further includes extracting one or more rooted subtree structures for each node of the plurality of nodes or each stage of the one or more stages of the tree structure of the workflow, each rooted subtree structure of the one or more rooted subtree structures including a set of nodes one or more hops away from the node or stage. The computer-implemented method may also include training the machine learning model using the one or more rooted subtree structures. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0018] This specification refers to the accompanying drawings, in which the use of like reference numbers in different drawings is intended to indicate the same or similar elements. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram illustrating an example of a network environment in accordance with certain aspects of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating another example of a network environment in accordance with some aspects of the present disclosure. [Diagram 3] 1 illustrates an example process flow for generating a prediction of a task outcome of a communication workflow according to some aspects of the present disclosure. [Figure 4] 1 illustrates an example process flow for generating task recommendations to complete a partial communication workflow in accordance with some aspects of the present disclosure. [Diagram 5] 1 is a flowchart illustrating an example process for generating a prediction of a task outcome of a communication workflow, in accordance with some aspects of the present disclosure. [Figure 6] 1 is a flowchart illustrating an example process for generating task recommendations to complete a partial communication workflow in accordance with some aspects of the present disclosure. [Figure 7] 1 is a flowchart illustrating an example process for generating a vector representation of a communication workflow in accordance with some aspects of the present disclosure. [Figure 8] 1 is a simplified diagram illustrating a distributed system for implementing one embodiment. [Figure 9] FIG. 1 is a simplified block diagram illustrating one or more components of the system environment. [Figure 10] FIG. 1 illustrates an exemplary computer system upon which various embodiments of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Detailed Description The cloud-based application may include a data management platform hosted on a cloud network. The data management platform may be configured to enable a user to perform certain functions on data records. For example, the data records may be user profiles that include one or more data fields. The data fields of the data records may include values ​​that characterize a user associated with the user profile. The data fields may be generated based on previous interactions, such as interactions between a browser operated by the user and a web server.

[0021] The communication workflow may be executed by a cloud-based application (or possibly by one or more servers outside the cloud network) to process a particular data record. As a non-limiting example, the communication workflow may be configured to execute a coordinated sequence of one or more tasks for a target group of a user profile (e.g., send a digital message involving the user). The task, when executed by the server, may cause the server to send a communication to a user device associated with the user profile via a communication channel. The content of the communication and the communication channel selected to send the communication may be determined based on the configuration of the task and the communication workflow. Including a target group of user devices in a communication workflow may cause the task included in the workflow to send a communication to each user device of the target group via a communication channel. The configuration of the task may define the content of the communication and the type of communication channel selected to facilitate sending the communication. Non-limiting examples of communication channels include SMS messaging, email, phone call, pop-up messages while browsing a web page, and other suitable communication channels.

[0022] As the complexity of communication workflows increases, the configuration and execution of communication workflows is a technical challenge. As a practical application, some aspects and features of the present disclosure generate vector representations of communication workflows and increase the computational efficiency of comparing various new communication workflows with previously executed communication workflows to predict the performance of new communication workflows and recommend tasks for completing partial communication workflows. Thus, some aspects and features of the present disclosure improve the technology of cloud-based data management platforms by enabling them to compare and evaluate various communication workflows in a computationally efficient manner using vectors (e.g., composite feature vectors, or vectors generated using a graph-to-vector model) to represent the structure and content of communication workflows.

[0023] Some implementations of the present disclosure include a cloud-based application configured to generate a vector representation of a communication workflow. The vector representation may be generated by evaluating the tasks and task content of the communication workflow using machine learning techniques such as Latent Dirichlet Allocation (LDA) and Word2Vec models for topic modeling. A feature vector may be generated for each task included in the communication workflow. The feature vectors of various tasks of the communication workflow may be concatenated to generate a composite feature vector. The composite feature vector may then be input into a machine learning model to generate a prediction of the task outcome of the communication workflow. The composite feature vector may also be input into the machine learning model to generate a recommendation of one or more tasks for inclusion in the communication workflow to complete the workflow.

[0024] FIG. 1 is a block diagram illustrating an example of a network environment 100 for processing data records using a data management platform such as a cloud-based application 120. Cloud network 110 may be any cloud-based network including one or more cloud-based servers and one or more cloud-based databases. The one or more cloud-based servers of cloud network 110 may include at least one application server hosting cloud-based application 120 along with at least one database server. For example, cloud-based application 120 may be a data management platform configured to perform functions associated with one or more data sets of the data records. In some implementations, the data records may represent a user profile including one or more data fields. Each data field of the one or more data fields may include a value generated in response to the user device interacting with a native application or a web server during a previous interaction. For example, when a user device accesses a web server to load a web page on the user device, the web server may extract one or more user features associated with a request from the user device to load the web page (e.g., the location of the outgoing request to load the web page). Various user features may be collected over time in the user profile.

[0025] The client system 130 may include a network of one or more servers and databases associated with a client (e.g., an entity such as a business). In some implementations, the network in the client system 130 may include a web server hosting one or more web pages or an application server hosting native applications run by or associated with the client. The user device 140 may interact with the client system 130 by sending a communication to the web server of the client system 130. For example, the communication may be a request to load a web page on a browser running on the user device 140. Similarly, the user device 150 may send a communication to the client system 130 requesting to load a web page on a browser of the user device 150. It will be appreciated that any number of user devices and any type of computing device (e.g., desktop computers, tablet computers, electronic kiosks, etc.) may communicate with the client system 130.

[0026] In some implementations, the client system 130 may define one or more workflows for communicating with a user device. A communication workflow may be an orchestrated sequence of one or more tasks, such as an MCO. A task may include executable code that, upon execution by the server, causes the client system 130 to send a communication to a user device over a communication channel. The content of the communication and the particular communication channel selected to send the communication may be defined by the executable code of the task. For example, a task may be a digital message designed to engage a particular target user. A workflow orchestrates a sequence of one or more tasks for an individual user profile over the course of a defined time interval. A workflow may play an effective role in engaging a user in relation to a particular product or service. As an illustrative example, the client system 130 may store at least two workflows. A first workflow may include two tasks for the user device 140, and a second workflow may include three tasks for the user device 150. The first workflow may include a first task of sending a communication to the user device 140 using a text messaging communication channel at a first time and a second task of sending a communication to the user device 140 using an email communication channel at a later second time. The second workflow may include a first task of sending a communication to the user device 150 using an email communication channel at a first time, a second task of sending a communication to the user device 150 using a social media communication channel (e.g., by sending a message to the user's account using a social media platform) at a later second time, and a third task of sending a communication to the user device 150 using another email communication channel at a later third time. The communication received at the user device may enable a user operating the user device to perform one or more actions. For example, the communication may include a selectable link. The user may or may not select the link.When a user performs an action, the client system 130 may detect the action because a signal may be sent back to the client system 130 (or the signal is not received within a prescribed period of time). In some implementations, the client system 130 may send a signal to the cloud-based application 120 hosted on the cloud network 110. The signal may include a response received from the user device in response to receiving the communication. Continuing with the above example, the client system 130 may send a signal to the cloud-based application 120 when the user device 140 responds to each of the first task and the second task, and similarly, when the user device 150 responds to each of the first task, the second task, and the third task. In other implementations, the cloud-based application 120 may execute the tasks of the workflow instead of the client system 130 executing the task execution.

[0027] In some implementations, the client system 130 may access the cloud-based application 120 to create, design, or generate a communication workflow. For example, the cloud-based application 120 may generate interface data that causes an interface to be displayed on an end user associated with the client system 130. The interface may allow the user to create an ordered sequence of tasks and save the ordered sequence as a communication workflow. Each task may be configured to perform a function such as splitting a target group of user devices into subgroups or sending a communication to a target group of user devices. The cloud-based application 120 may allow the end user associated with the client system 130 to generate a composite feature vector that represents the communication workflow. The composite feature vector can then be input into a trained machine learning model to generate an output that predicts task outcomes of the communication workflow (e.g., the overall performance or predicted success of all tasks in the communication workflow). The cloud-based application 120 may also input the composite feature vector into the trained machine learning model to generate recommendations of one or more tasks to add or include in the communication workflow to complete the workflow (if the communication workflow is a partial workflow). The cloud-based application 120 may also run one or more neural networks to learn a sub-graph embedding of the communication workflow to convert the communication workflow into a vector representation.

[0028] 2 is a block diagram illustrating another example of a cloud network 110 according to some aspects of the disclosure. A cloud-based application 120 may include a network of cloud-based servers and databases. In some implementations, the network of cloud-based application 120 may include a workflow structure generator 210, a feature vector generator 220, a workflow representation system 230, a workflow performance predictor 240, a partial workflow predictor 250, and a machine learning model 260. The machine learning model 260 may be trained using a training dataset 270.

[0029] The workflow structure generator 210 may be any server, processor, and / or database configured to store and potentially execute code to generate a structure for a communication workflow. The workflow structure generator 210 may receive input from a user accessing the cloud-based application 120. The input may represent a selection by the user of one or more tasks to include in a new communication workflow. The one or more tasks may be arranged in an ordered sequence determined by the user. The ordered sequence of tasks may represent a communication workflow. When the input is received, the workflow structure generator 210 may evaluate the ordered sequence of tasks to generate a structure for the communication workflow. The structure may be a tree structure including multiple nodes. Each node may represent a task. As an illustrative example, the workflow structure generator 210 may generate the structure by searching through the communication workflow for a starting task (e.g., a task that starts the communication workflow). The workflow structure generator 210 may create a hierarchical data structure and use the starting task as the root node. The workflow structure generator 210 may continue searching the communication workflow for the next task, which may be an email task (e.g., a task to send an email to a user device). The workflow structure generator 210 may add a stage (e.g., a child node) representing the email task to the root node of the hierarchical data structure. The next task may be a split task in which a target group of user devices is split into two or more subgroups. The workflow structure generator 210 may add another node to the hierarchical data structure to represent the split task. The workflow structure generator 210 may continue adding stages (e.g., nodes) to the hierarchical data structure until the communication workflow reaches an end task (e.g., a task that completes the communication workflow). The resulting hierarchical data structure may represent the structure of the communication workflow. The generated structure may be stored in the workflow structure generator 210.

[0030] The feature vector generator 220 may be any server, processor, and / or database configured to generate a task vector for each task of a communication workflow. For example, U.S. Ser. No. 16 / 286,297, filed Feb. 26, 2019, which is incorporated by reference in its entirety for all purposes, describes a process for generating vectors representing the tasks of a communication workflow. The feature vector generator 220 may then, for example, concatenate the task vectors of the communication workflow. The resulting concatenation of the task vectors may represent a composite feature vector of the communication workflow.

[0031] In some implementations, the feature vector generator 220 may generate a representation of the content and metadata of the tasks included in the communication workflow. As an illustrative example, for an email task, the feature vector generator 220 may generate a representation of various parameters of the email communication. The parameters may include numerical and categorical variables of metadata associated with the email (e.g., product category, task type, etc.). In this example, the representation of the email metadata may be left as is, scaled, dimensionally reduced, or otherwise processed. The parameters may also include the email subject and email body. In this case, natural language processing (NLP), word frequency matrices, word frequency-inverse document frequency (TF-IDF), topic modeling, etc. may be performed to convert the text into a vector representation of the text. The parameters may also include the email content (e.g., image data). In this case, the feature vector generator 220 may convert the raw image vectors into task vectors, for example using dimensionality reduction techniques (e.g., non-negative matrix factorization (NMF), singular value decomposition (SVD), principal component analysis (PCA), and other suitable techniques), or as intermediate outputs of a neural network.

[0032] The feature vector generator 220 may also generate one or more vectors representing one or more target groups of user devices that will receive one or more communications of the communication workflow. As a non-limiting example, assuming that each user device included in a target group of user devices is represented by a feature vector (e.g., a vector representation of the characteristics of the user device), the target group may be summarized by defining the target group vector as the average (e.g., centroid) of the feature vectors of the members of the target group. However, the disclosure is not limited to this technique. By way of example, the communication workflow may include an email communication task and a text message task. The email communication task may be configured to send an email to each of the first target group of user devices. The text message task may be configured to send a text message to each of the second target group of user devices. In this case, the feature vector generator 220 may generate a vector for each of the first and second target groups of user devices.

[0033] Workflow representation system 230 may be any server, processor, and / or database configured to generate a vector representation of a communicating workflow by executing a neural network to learn a graph embedding of the communicating workflow. For example, workflow representation system 230 may input the structure of the communicating workflow (e.g., as determined by workflow structure generator 210) into a Graph2Vec model to generate an n-dimensional vector representation of the communicating workflow.

[0034] The workflow performance predictor 240 may be any server, processor, and / or database configured to receive as input a vector representation of a communication workflow and generate as output a prediction of a task outcome of the communication workflow. In some implementations, the vector representation is a composite feature vector generated by the feature vector generator 220. In other implementations, the vector representation is an n-dimensional vector representation of the communication workflow generated by the workflow representation system 230. The workflow performance predictor 240 may input the vector representation of the communication workflow to a machine learning model (e.g., a supervised learning model trained on labeled data such as known overall performance values ​​of previously executed workflows). The machine learning model may be a supervised, semi-supervised, or unsupervised (e.g., k-means clustering) model. The output of the machine learning model may represent a prediction of the performance of the communication workflow. In some examples, the overall performance of the communication workflow may be represented by a task outcome, which represents a rate of target user-performed actions in response to receiving a communication (e.g., conversion rate, click rate of selecting a link included in a digital message, open rate of opening a web page from a link included in a communication, etc.).

[0035] It will be appreciated that an action may not necessarily be an action associated with a link contained in the communication. A user device may receive a communication with a link, but the user may independently access a web page to purchase a good or service. Although a target action may be selection of a link, purchasing a good or service may nevertheless be considered a target action if, for example, the link may have navigated the user to a web page that enabled the user to purchase the good or service.

[0036] The workflow performance predictor 240 may generate an output predicting a task outcome, which may be a value representing a predicted rate of a target action performed by a user in response to receiving a communication across all tasks in the communication workflow. The task outcome may be predicted by evaluating known task outcomes of previously executed communication workflows (e.g., calculating an average of known task outcomes of previously executed communication workflows that have the same structure as the new communication workflow and are determined to be similar in vector space). The previously executed communication workflow may be a communication workflow in which the task was previously executed. The previously executed communication workflow may have previously engaged communications or communications sent to a target group of user devices. The previously executed communication workflows may be stored in the training dataset 270. The cloud-based application 120 may store task outcomes associated with each previously executed communication workflow.

[0037] In some implementations, the workflow performance predictor 240 may identify a set of previously executed communication workflows that have the same structure as that of the communication workflow. Within the set of previously executed communication workflows, the workflow performance predictor 240 may identify one or more previously executed communication workflows that are similar to the communication workflow. Similarity may be measured in a multi-dimensional space of composite feature vectors of previously executed communication workflows. For example, a previously executed communication workflow of a given structure whose corresponding composite feature vector is within a threshold distance from the composite feature vector of the communication workflow may be determined to be similar to the communication workflow. In some examples, the task results of a communication workflow may be predicted based on a combination (e.g., average, weighted average, etc.) of known task results of previously executed communication workflows determined to be similar to the communication workflow. In some examples, a confidence interval around the predicted task results of the communication workflow may be constructed based on a calculated variance of known task results of previously executed communication workflows determined to be similar to the communication workflow and to have the same structure as the new communication workflow.

[0038] In some implementations, the previously executed communication workflows may be segmented by structure. For a given structure, the composite feature vectors of the grouped previously executed communication workflows may be evaluated using machine learning techniques such as k-means clustering to form one or more clusters of previously executed communication workflows. The clustering may be performed based on the composite feature vectors of the previously executed communication workflows. The workflow performance predictor 240 may then assign the communication workflow to a cluster based on a comparison of the composite feature vector of the communication workflow with the composite feature vectors associated with each cluster. After the communication workflow is assigned to a cluster, the task outcome of the communication workflow may be predicted based on a combination (e.g., average, weighted average, etc.) of known task outcomes of the previously executed communication workflows in the cluster. In some examples, a confidence interval around the predicted task outcome of the communication workflow may be constructed based on the calculated variance of the known task outcomes of the previously executed communication workflows in the cluster.

[0039] In some implementations, previously executed communication workflows may be segmented by structure. For each workflow structure, the workflow performance predictor 240 may train a machine learning model (e.g., supervised learning classification or regression) using a composite feature vector of previously executed communication workflows having that structure. The composite feature vector of the communication workflow may be input to the machine learning model associated with the structure of the communication workflow to generate a prediction of the task outcome. In some examples, a confidence interval around a predicted task outcome of the communication workflow may be determined based on a particular model type (e.g., linear regression, random forest regression, etc.).

[0040] In some examples, none of the previously executed communication workflows may share the same structure as the communication workflow. The workflow performance predictor 240 may identify a set of previously executed communication workflow partial parts that have the same structure as the communication workflow. The workflow performance predictor 240 may then evaluate the composite feature vectors of the set of previously executed communication workflow partial parts to determine which partial parts are similar to the communication workflow using the techniques described above (e.g., identifying composite feature vectors within a threshold distance from the composite feature vector of the communication workflow). Similar to the techniques described above, known task results of the set of previously executed workflow partial parts may be combined (e.g., averaged) to determine a prediction of the task results of the communication workflow. In some implementations, the communication workflow may be segmented into partial workflows such that each partial workflow corresponds to the structure of a previously executed communication workflow. Using the techniques described above, the task results of each partial workflow can be determined based on known task results of previously executed communication workflows that share the same structure as the partial workflow. The predicted task outcomes of each partial workflow may then be combined (eg, averaged or summed) to determine the predicted task outcomes of the complete communication workflow.

[0041] The partial workflow predictor 250 may be any server, processor, and / or database configured to generate recommendations of one or more tasks to complete a communication workflow. The cloud-based application 120 may generate an interface that allows a user to create a communication workflow. When a user creates a partial communication workflow (e.g., the user has not yet set an end node), the partial workflow predictor 250 may be configured to generate a recommendation to the user of which tasks to add to complete the partial communication workflow. Given a new partial communication workflow, the partial workflow predictor 250 may generate a composite feature vector of the new partial communication workflow based on the techniques described above. The partial workflow predictor 250 may then identify a set of partial parts of previously executed communication workflows that have the same structure as the new partial communication workflow. The partial workflow predictor 250 may determine the partial parts of previously executed communication workflows that are similar to the new partial communication workflow based on a comparison of the composite feature vectors in the domain space, as described above. The partial workflow predictor 250 may rank a set of previously executed communication workflow partial parts that are similar to the new partial communication workflow in descending order based on the known task results of the previously executed communication workflows. The partial workflow predictor 250 may then recommend an entire previously executed communication workflow for one or more of the previously executed communication workflow partial parts that are ranked highest. The recommendation may include the remaining tasks of the previously executed communication workflow partial parts.

[0042] The machine learning models 260 may be any server, processor, and / or database configured to generate, train, or execute machine learning or artificial intelligence models. For example, the machine learning models may be generated using one or more machine learning algorithms, such as ensembles of multi-label classifiers (e.g., supervised or unsupervised learning), artificial neural networks (including backpropagation, Boltzmann machines, etc.), Bayesian statistics (e.g., Bayesian networks or knowledge bases), learning-to-rank techniques, logistic model trees, decision tree models, support vector machines, informative fuzzy networks, hidden Markov models, hierarchical clustering (unsupervised), self-organizing maps, clustering techniques, and other suitable machine learning techniques (supervised, semi-supervised, or unsupervised).

[0043] The feature vector generator 220 may separate the structural representation of the communication workflow (e.g., a tree-based representation of the communication workflow including "nodes" and "branches") from the vector representation of the communication workflow (e.g., a concatenation of task vectors and vectors representing target groups of user devices). Additionally, the machine learning model 260 may generate a model for each structure of the previously executed communication workflow (e.g., one machine learning model may be generated and trained on performance values ​​for each individual structure of the communication workflow, one pre-computed cluster set for each structure, etc.). The separate structural representation may be used to select a subset of the training dataset 270 of previously executed communication workflows that have the same structure as the new communication workflow (e.g., one that has not yet been executed or one that is currently being executed). However, it will be appreciated that in some implementations, an alternative approach may be to generate one or more lists (e.g., a single list) of metadata describing the entire communication workflow. The cloud-based application 120 may execute code to parse through the list to determine a complete description of the communication workflow, including the structure of the communication workflow, the tasks included in the communication workflow, the task features of each task included in the communication workflow, the composite feature vector of the communication workflow, and other suitable information items. In some implementations, the list may be a data structure representing a communication workflow. For example, the list of previously executed communication workflows may be automatically parsed to identify previously executed communication workflows that have the same structure as a given communication workflow. The list may also be automatically parsed to detect previously executed communication workflows that are similar to the given communication workflow (e.g., (i) have the same structure, (ii) are similar in the multidimensional domain space of the task vectors of the constituent tasks, etc.). The cloud-based application 120 may evaluate the list as a composite communication workflow unit vector that can serve as an input for a tree-based machine learning model (e.g., training and applying a random forest model).

[0044] To illustrate by way of non-limiting example only, a communication workflow may be composed of four tasks. The structure of the communication workflow may include four "nodes" (e.g., "start", "split", "end branch 1" and "end branch 2"). This list of communication workflows may be represented by the following data structure: [ num_nodes = 4, node_type = "start", num_tasks = 3, task_type = "filter", filter_type, filter_input = [<target_group_vector for “target group 0”> ], task_type = “email_communication”, communication_input = [<target_group_vector for “target group 1”> ], communication_task_vector = [<concatenated task content and context task vector of “task 1”> ], task_type = “delay”, delay_input = [<target_group_vector for “target group 1”> ], delay_value = T1, node_type = “split”, split_type, split_input = [<target_group_vector for “target group 1”> ], num_split_outputs = 2, num_tasks_split_output_1 = 0, node_type = “end”, end_input = [<target_group_vector for “target group 2”> ], num_tasks_split_output_2 = 1, task_type = “sms_communication”, communication_input = [<target_group_vector for “target group 3”> ],communication_task_vector = [<concatenated task content and context task vector of “task 2”> ], task_type = “end”, end_input = [<target_group_vector for “target group 3”> ] ]. ,

[0045] 3 illustrates an example process flow 300 for generating a prediction of a task outcome (e.g., overall performance) of a communication workflow, in accordance with some aspects of the present disclosure. The process flow 300 may be performed, at least in part, by, for example, any of the components described in FIGS. 1-2. Additionally, the process flow 300 may be performed to generate an output that predicts a task outcome of a new communication workflow.

[0046] The process flow 300 may begin when a user interacts with an interface provided by the cloud-based application 120 to create a new communication workflow 310. As an illustrative example, the new communication workflow 310 may include an ordered sequence of tasks 1-5, where task 1 is a start task that initiates the workflow, task 2 is a split task that splits a target group of user devices into two subgroups, task 3 may be a communication task that sends an email to a subgroup of user devices associated with task 3, task 4 may be an end node that terminates the workflow, task 5 may be a communication task that sends a text message to another subgroup of user devices associated with task 5, and task 6 may be an end node that terminates the workflow. The cloud-based application 120 may use the workflow structure generator 210 to generate a structure for the new communication workflow 310.

[0047] The new communication workflow 310 may be input to the workflow performance predictor 240 to generate a predicted performance value. The predicted performance value may represent a predicted task outcome of the new communication workflow 310, such as a percentage of users who perform a target action in response to receiving a communication triggered by a task included in the workflow, such as task 3 or 5. The workflow performance predictor 240 may receive the composite feature vector of the new communication workflow 310 generated by the feature vector generator 220 and input the composite feature vector to a trained machine learning model stored in the machine learning model 260. In response to receiving the composite feature vector of the new communication workflow 310, the trained machine learning model may generate an output of a predicted performance of the new communication workflow 310.

[0048] 4 illustrates an example process flow 400 for generating task recommendations for inclusion in a partial communication workflow, in accordance with some aspects of the present disclosure. Process flow 400 may be performed, at least in part, by any of the components described in Figures 1-2, for example. Additionally, process flow 400 may be performed to generate recommendations for one or more tasks to include in the partial communication workflow.

[0049] The process flow 400 may begin when a user interacts with an interface provided by the cloud-based application 120 to create a new partial workflow 410. As an illustrative example, the new partial workflow 410 may include an ordered sequence of tasks 1-3, where task 1 is a start task that initiates the workflow, task 2 is a split task that splits a target group of user devices into two subgroups, and task 3 is a communication task that sends emails to the subgroup of user devices associated with task 3. The cloud-based application 120 may use the workflow structure generator 210 to generate the structure of the new partial workflow 410.

[0050] The new partial workflow 410 may be input to the partial workflow predictor 250 to generate recommendations of one or more tasks that would complete the new partial workflow 410. The partial workflow predictor 250 may receive the composite feature vector of the new partial workflow 410 generated by the feature vector generator 220. The partial workflow predictor 250 may identify a set of previously executed partial parts of communication workflows stored in the training dataset 270 that have the same structure as the new partial workflow 410. The partial workflow predictor 250 may then use the similarity detection techniques described above to determine one or more partial parts of the previously executed communication workflows that are similar to the composite feature vector of the new partial workflow 410. The partial workflow predictor 250 may rank the partial parts of the previously executed communication workflows that are determined to be similar by performance. For example, a top performing partial part of the previously executed communication workflow may be selected. The remaining tasks of the selected partial parts of the previously executed communication workflow may be recommended to complete the new partial workflow 410. As an illustrative example, task 4 and task 5 may be recommended for inclusion in a new partial workflow 410 to generate a complete workflow 420 .

[0051] 5 is a flow chart illustrating an example of a process 500 for generating an output predicting a task outcome of a communication workflow according to some aspects of the disclosure. In some implementations, the process 500 may be performed by any of the components described in FIGS. 1-2. For example, the process 500 may be performed by any of the cloud-based application 120 and corresponding subcomponents. Additionally, as a non-limiting example, the process 500 may be performed to generate a prediction of a value representing a predicted task outcome of executing a communication workflow for a target group of user devices using machine learning techniques.

[0052] For example, process 500 may begin at block 510, where cloud-based application 120 may receive or access a communication workflow that includes an ordered sequence of one or more tasks. The ordered sequence of tasks may represent a structure, such as a hierarchical tree structure. Cloud-based application 120 may generate an interface that allows a user to create a communication workflow by selecting tasks and arranging those tasks in an ordered sequence.

[0053] At block 520, the feature vector generator 220 may evaluate the selected tasks and the ordered sequence of tasks to generate a composite feature vector representing the communication workflow. For example, the feature vector generator 220 may generate vectors to represent each task using techniques such as Word2Vec, topic modeling, singular value decomposition (SVD), neural networks for image representation (e.g., in an example where the tasks are messages containing images), and other suitable techniques. The feature vector generator 220 may then generate a composite feature vector based on a combination of the vectors generated for each task.

[0054] At block 530, the workflow performance predictor 240 may input the composite feature vector of the communication workflow into a trained machine learning model stored in the machine learning model 260. In some examples, the trained machine learning model may be a supervised learning model, such as a linear regression model, a support vector machine (SVM), a logistic regression, a naive Bayes, a decision tree model, a k-nearest neighbor model, a neural network, similarity learning, and other suitable models. The supervised learning model may be trained using labeled data. For example, each previously executed communication workflow may be labeled with its corresponding performance value (e.g., click rate) based on its known task results. The supervised learning model may be trained to classify or predict the performance value of the communication workflow. At block 540, the workflow performance predictor 240 may generate an output corresponding to a prediction of the performance value of the communication workflow.

[0055] 6 is a flow chart illustrating an example of a process 600 for generating recommendations of tasks to include in a new partial communication workflow, according to some aspects of the disclosure. In some implementations, the process 600 may be performed by any of the components described in FIGS. 1-2. For example, the process 600 may be performed by the cloud-based application 120 and any of the corresponding subcomponents. Additionally, as a non-limiting example, the process 600 may be performed to generate recommendations to a user of one or more tasks to add to a new partial communication workflow.

[0056] For example, process 600 may begin at block 610, where cloud-based application 120 may generate and facilitate display of an interface that allows a user to define a communication workflow. At block 620, the user may navigate the interface to select tasks and arrange the selected tasks in an ordered sequence that represents an incomplete or partial communication workflow. The interface may receive user input corresponding to the partial communication workflow. The partial communication workflow may be associated with one or more parameters that characterize each task of the one or more tasks of the partial communication workflow. For example, the parameters may include vocabulary for topic modeling, content images of the digital message, and other suitable characteristics of the task.

[0057] In block 630, the workflow structure generator 210 may evaluate the partial communication workflow to identify or generate a structure for the partial communication workflow. For example, the structure may be a hierarchical tree structure including a root node, one or more intermediate nodes, and one or more leaf nodes. The root node may represent a start task that initiates the partial communication workflow. The intermediate nodes may be any type of task, such as a split task that splits a target group of user devices into two or more subgroups, a filter task that filters a target group of user devices, a communication task that sends a communication to a target group of user devices, etc. The leaf node may be an end node that represents an end of the communication workflow. The partial communication workflow may not yet include an end node because the workflow is partial or incomplete. In some examples, the structure of the partial communication workflow may be represented by one or more nodes and / or one or more stages. Each task of the one or more tasks of the partial workflow may correspond to a node of the one or more nodes or a stage of the one or more stages.

[0058] At block 640, the feature vector generator 220 may evaluate the selected tasks and the ordered sequence of tasks to generate a composite feature vector representing the partial communication workflow. For example, the feature vector generator 220 may generate vectors to represent each task using techniques such as Word2Vec, topic modeling, singular value decomposition (SVD), neural networks for image representation (e.g., in an example where the tasks are messages containing images), and other suitable techniques. The feature vector generator 220 may then generate a composite feature vector based on a combination of the vectors generated for each task.

[0059] In block 650, the partial workflow predictor 250 may access a set of previously executed partial workflows of the training dataset 270. Each previously executed partial workflow of the set of previously executed partial workflows may be represented by a structure, a composite feature vector, and a performance value (e.g., a task outcome). In block 660, the partial workflow predictor 250 may select a subset of the set of previously executed partial workflows. The selected subset of previously executed partial workflows may share the same structure as the partial workflow. The partial workflow predictor 250 may determine one or more previously executed partial workflows similar to the partial workflow from the subset of previously executed partial workflows. The similarity may be determined based on a comparison between the composite feature vector of each previously executed partial workflow of the subset and the composite feature vector of the partial workflow, or any other similarity determination technique described above with respect to FIG. 2.

[0060] At block 670, the partial workflow predictor 250 may generate recommendations of one or more recommended tasks to complete the partial workflow. The one or more recommended tasks may be selected from one or more remaining tasks of a previously executed partial workflow of the one or more previously executed partial workflows that share the same structure as the partial workflow and are determined to be similar to the partial workflow and have the highest performance value. The selection of the recommended tasks may be based on previous performance values ​​of the one or more previously executed partial workflows that share the same structure as the partial workflow and are determined to be similar to the partial workflow.

[0061] FIG. 7 is a flow chart illustrating an example of a process 700 for generating a vector representation of a communication workflow according to some aspects of the disclosure. In some implementations, the process 700 may be performed by any of the components described in FIGS. 1-2. For example, the process 700 may be performed by any of the cloud-based applications 120 and corresponding subcomponents. Further, as a non-limiting example, the process 700 may be performed to generate a vector representation of a communication workflow using a graph-based learning model (e.g., Graph2Vec). The graph-based learning model may use a neural network to generate a multi-dimensional vector representation of the structure and tasks of the communication workflow. The vector representation of the communication workflow as generated by the process 700 may be input to the workflow performance predictor 240 and the partial workflow predictor 250 to generate performance predictions and task recommendations, respectively.

[0062] For example, the process 700 may begin at block 710, where the cloud-based application 120 may generate a graph for each previously executed workflow in the training data set 270. Given a graph representing the structure of the communication workflow, the workflow representation system 230 may sample several rooted subgraphs around different nodes of the graph. At block 720, the workflow representation system 230 may train an n-dimensional neural network using the sampled rooted subgraphs. In some examples, the rooted subgraphs may include a representation of a subgraph where a node is selected as a root node, and one or more branches may be learned from the selected node. Nodes of the tree structure of the communication workflow may be iteratively selected as root nodes for the neural network to learn various branches that arise from each rooted node. At block 730, training the neural network may include learning a subgraph embedding for each node of each previously executed communication workflow in the training data set 270. Furthermore, learning the subgraph embeddings using the neural network may transform the nodes and stages of the tree structure representing the communication workflow into a vector space while preserving the structure and metadata of the tree structure. A vector representation may be generated for each previously executed communication workflow in the training dataset 270 using the graph-based learning model. The vector representations stored in the training dataset 270 may then be evaluated against the vector representations of new communication workflows to perform the methods and processes described herein.

[0063] 8 shows a simplified diagram of a distributed system 800 for implementing one embodiment. In the illustrated embodiment, the distributed system 800 includes one or more client computing devices 802, 804, 806, and 808 configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., over one or more networks 810. A server 812 may be communicatively coupled to remote client computing devices 802, 804, 806, and 808 over network 810.

[0064] In various embodiments, server 812 may be adapted to execute one or more services or software applications provided by one or more of the system's components. In some embodiments, these services may be provided to users of client computing devices 802, 804, 806, and / or 808 as web-based or cloud services, or under a software-as-a-service (SaaS) model. Users operating client computing devices 802, 804, 806, and / or 808 may then utilize one or more client applications to interact with server 812 to utilize the services provided by these components.

[0065] In the illustrated configuration, the software components 818, 820, and 822 of the system 800 are shown as being implemented on the server 812. In other embodiments, one or more of the components of the system 800 and / or the services provided by these components may also be implemented by one or more of the client computing devices 802, 804, 806, and / or 808. A user operating a client computing device may then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 800. Thus, the illustrated embodiment is an example of a distributed system for implementing the system of the embodiments, and is not intended to be limiting.

[0066] The client computing devices 802, 804, 806, and / or 808 may be portable handheld devices (e.g., iPhone®, cellular telephone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head mounted display), running software such as Microsoft Windows Mobile®, and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, and may support Internet, email, short message service (SMS), Blackberry®, or other enabled communication protocols. The client computing devices may be general purpose personal computers, including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing devices may be workstation computers running any of the various commercially available UNIX® or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS. Alternatively, or in addition, client computing devices 802, 804, 806, and 808 may be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over network 810.

[0067] Although the example distributed system 800 is shown with four client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the server 812.

[0068] The network 810 in the distributed system 800 may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, and the like. By way of example only, the network 810 may be a local area network (LAN), such as one based on Ethernet, Token Ring, and the like. The network 810 may be a wide area network and the Internet. It may include virtual networks, including, but not limited to, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (e.g., networks operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 suite of protocols, Bluetooth, and / or any other wireless protocol); and / or any combination of these and / or other networks.

[0069] Servers 812 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configurations and / or combinations. In various embodiments, servers 812 may be adapted to execute one or more services or software applications described in the preceding disclosure. For example, servers 812 may correspond to servers for executing the processes described above in accordance with certain embodiments of the present disclosure.

[0070] Server 812 may run an operating system, including any of those mentioned above, as well as any commercially available server operating system. Server 812 may also run any of a variety of other server applications and / or mid-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA® server, a database server, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM® (International Business Machines), and the like.

[0071] In some implementations, server 812 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 802, 804, 806, and 808. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 812 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 802, 804, 806, and 808.

[0072] Distributed system 800 may also include one or more databases 814 and 816. Databases 814 and 816 may reside in a variety of locations. By way of example, one or more of databases 814 and 816 may reside on a non-transitory storage medium local to (and / or resident on) server 812. Alternatively, databases 814 and 816 may be remote from server 812 and communicate with server 812 via a network-based or dedicated connection. In one set of embodiments, databases 814 and 816 may reside within a storage area network (SAN). Similarly, any necessary files to perform the functions ascribed to server 812 may be stored locally on server 812 and / or remotely, as appropriate. In one set of embodiments, databases 814 and 816 may include relational databases, such as those provided by Oracle, adapted to store, update, and retrieve data in response to SQL-formatted commands.

[0073] 9 is a simplified block diagram of one or more components of a system environment 900 that may provide services provided by one or more components of an embodiment system as cloud services, according to one embodiment of the disclosure. In the illustrated embodiment, the system environment 900 includes one or more client computing devices 904, 906, and 908 that may be used by a user to interact with a cloud infrastructure system 902 that provides cloud services. The client computing devices may be configured to run a client application, such as a web browser, a proprietary client application (e.g., Oracle Forms), or some other application, that may be used by a user of the client computing device to interact with the cloud infrastructure system 902 to use the services provided by the cloud infrastructure system 902.

[0074] It should be understood that the illustrated cloud infrastructure system 902 may have other components than those shown. Additionally, the illustrated embodiment is only one example of a cloud infrastructure system that may incorporate embodiments of the present invention. In some other embodiments, cloud infrastructure system 902 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of components.

[0075] The client computing devices 904, 906, and 908 may be devices similar to those described above for 802, 804, 806, and 808.

[0076] Although the exemplary system environment 900 is shown with three client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, may interact with the cloud infrastructure system 902.

[0077] Network 910 may facilitate communication and exchange of data between clients 904, 906, and 908 and cloud infrastructure system 902. Each network may be any type of network familiar to those skilled in the art capable of supporting data communications using any of a variety of commercially available protocols, including those described above for network 810.

[0078] Cloud infrastructure system 902 may include one or more computers and / or servers, which may include those described above for server 812.

[0079] In an embodiment, the services provided by a cloud infrastructure system may include a host of services made available to users of the cloud infrastructure system on demand, such as online data storage and backup solutions, web-based email services, hosted office suites and document collaboration services, database processing, managed technical support services, etc. The services provided by the cloud infrastructure system may be dynamically scaled to meet the needs of its users. A particular instantiation of a service provided by the cloud infrastructure system is referred to herein as a "service instance." In general, any service made available to users from a cloud service provider's system over a communications network such as the Internet is referred to as a "cloud service." Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from the customer's own on-premise servers and systems. For example, the cloud service provider's system may host an application, and a user may order and use that application on demand over a communications network such as the Internet.

[0080] In some examples, services in a computer network cloud infrastructure may include protected computer network access to storage, hosted databases, hosted web servers, software applications, or other services provided to users by cloud vendors or known in the art. For example, a service may include password-protected access to remote storage on the cloud over the Internet. As another example, a service may include a web services-based hosted relational database and a scripting language middleware engine for private use by networked developers. As another example, a service may include access to an email software application hosted on a cloud vendor's website.

[0081] In one embodiment, cloud infrastructure system 902 may include a suite of application, middleware, and database service offerings that are self-service, subscription-based, elastically scalable, reliable, highly available, and securely delivered to customers. An example of such a cloud infrastructure system is the Oracle Public Cloud offered by the present assignee.

[0082] In various embodiments, cloud infrastructure system 902 may be adapted to automatically provision, manage, and track customer subscriptions to services provided by cloud infrastructure system 902. Cloud infrastructure system 902 may provide cloud services through different deployment models. For example, the services may be provided under a public cloud model, where cloud infrastructure system 902 is owned by an organization that sells cloud services (e.g., owned by Oracle) and the services are made available to the general public or to companies in different industries. As another example, the services may be provided under a private cloud model, where cloud infrastructure system 902 operates only for a single organization and may provide services to one or more entities within that organization. Cloud services may also be provided under a community cloud model, where cloud infrastructure system 902 and the services provided by cloud infrastructure system 902 are shared by several organizations within an associated community. Cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

[0083] In some embodiments, the services offered by cloud infrastructure system 802 may include one or more services offered under a Software as a Service (SaaS) category, a Platform as a Service (PaaS) category, an Infrastructure as a Service (IaaS) category, or other category of services including hybrid services. A customer may order one or more services offered by cloud infrastructure system 902 via a subscription order. Cloud infrastructure system 902 then performs processing to provide the services in the customer's subscription order.

[0084] In some embodiments, the services provided by the cloud infrastructure system 902 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system via a SaaS platform. The SaaS platform may be configured to provide cloud services that fall under the SaaS category. For example, the SaaS platform may provide the ability to build and deliver a suite of on-demand applications on an integrated development and deployment platform. The SaaS platform may manage and control the underlying software and infrastructure to provide the SaaS services. By utilizing the services provided by the SaaS platform, customers can utilize applications that run on the cloud infrastructure system. Customers can obtain application services without having to purchase separate licenses and support. A variety of different SaaS services may be provided. Examples include, but are not limited to, services that provide solutions for sales performance management, enterprise integration, and business flexibility for large organizations.

[0085] In some embodiments, platform services may be provided by the cloud infrastructure system through a PaaS platform. The PaaS platform may be configured to provide cloud services that fall under the PaaS category. Examples of platform services may include, but are not limited to, services that allow an organization (such as Oracle) to consolidate existing applications on a shared common architecture, and the ability to build new applications that leverage the shared services provided by the platform. The PaaS platform may manage and control the underlying software and infrastructure to provide the PaaS services. Customers can obtain the PaaS services provided by the cloud infrastructure system without the customer having to purchase separate licenses and support. Examples of platform services include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), and the like.

[0086] By utilizing the services offered by the PaaS platform, customers can employ programming languages ​​and tools supported by the cloud infrastructure system and also control the deployed services. In some embodiments, the platform services offered by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware services), and Java cloud services. In one embodiment, the database cloud services may support a shared services deployment model that allows organizations to pool database resources and provide database-as-a-service to customers in the form of a database cloud. The middleware cloud services may provide a platform for customers to develop and deploy various cloud applications, and the Java cloud services may provide a platform for customers to deploy Java applications in the cloud infrastructure system.

[0087] A variety of different infrastructure services may be provided by the IaaS platform in a cloud infrastructure system that facilitates management and control of underlying computing resources, such as storage, network, and other fundamental computing resources, for customers that use the services provided by the SaaS and PaaS platforms.

[0088] In an embodiment, cloud infrastructure system 902 may also include infrastructure resources 930 for providing resources used to provide various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources 930 may include a pre-integrated and optimized combination of hardware, such as servers, storage, and networking resources to run the services offered by the PaaS and SaaS platforms.

[0089] In some embodiments, resources in cloud infrastructure system 902 may be shared by multiple users and dynamically reallocated per demand. In addition, resources may be allocated to users in different time periods. For example, cloud infrastructure system 930 may allow a first set of users in a first time period to utilize resources in the cloud infrastructure system for a specified number of hours, and then reallocate the same resources to another set of users in a different time period, thereby maximizing utilization of the resources.

[0090] In an embodiment, several internal shared services 932 may be provided that are shared by different components or modules of cloud infrastructure system 902 and by services provided by cloud infrastructure system 902. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0091] In particular embodiments, cloud infrastructure system 902 may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one embodiment, cloud management functions may include capabilities for provisioning, managing, tracking customer subscriptions received by cloud infrastructure system 902, and the like.

[0092] In one embodiment, as shown, the cloud management functionality may be provided by one or more modules, such as an order management module 920, an order orchestration module 922, an order provisioning module 924, an order management and monitoring module 926, and an identity management module 928. These modules may include or be provided using one or more computers and / or servers, which may be general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0093] In example operation 934, a customer using a client device, such as client device 904, 906, or 908, may interact with cloud infrastructure system 902 by requesting one or more services offered by cloud infrastructure system 902 and placing an order for a subscription to one or more services offered by cloud infrastructure system 902. In an embodiment, the customer may access cloud user interfaces (UIs), cloud UI 912, cloud UI 914, and / or cloud UI 916, and place the subscription order via these UIs. Order information received by cloud infrastructure system 902 in response to the customer placing the order may include information identifying the customer and the one or more services offered by cloud infrastructure system 902 to which the customer wishes to subscribe.

[0094] After an order is placed by a customer, the order information is received via the cloud UI 912, 914 and / or 916.

[0095] In operation 936, the order is stored in order database 918. Order database 918 can be one of several databases operated by cloud infrastructure system 918 and in conjunction with other system elements.

[0096] At operation 938, the order information is forwarded to the order management module 920. In some examples, the order management module 920 may be configured to perform billing and accounting functions related to the order, such as validating the order and booking the order once verified.

[0097] At operation 940, information regarding the order is communicated to the order orchestration module 922. The order orchestration module 922 may utilize the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some examples, the order orchestration module 922 may use the services of the order provisioning module 924 to orchestrate the provisioning of resources to support the subscribed services.

[0098] In certain embodiments, the order orchestration module 922 enables management of processes associated with each order and applies logic to determine whether the order should proceed to provisioning. In operation 942, upon receiving a new subscription order, the order orchestration module 922 sends a request to the order provisioning module 924 to allocate resources and configure the resources needed to fulfill the subscription order. The order provisioning module 924 enables the allocation of resources for the services ordered by the customer. The order provisioning module 924 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 900 and the physical implementation layer used to provision resources to provide the requested services. Thus, the order orchestration module 922 may be insulated from implementation details, such as whether services and resources are actually provisioned on the fly or pre-provisioned and allocated / allocated only on demand.

[0099] In operation 944 , once the services and resources have been provisioned, a notification of the rendered services may be sent by the order provisioning module 924 of the cloud infrastructure system 902 to the customer on the client device 904 , 906 and / or 908 .

[0100] At operation 946, the customer's subscription order may be managed and tracked by the order management and monitoring module 926. In some cases, the order management and monitoring module 926 may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system up and down time.

[0101] In an embodiment, cloud infrastructure system 900 may include an identity management module 928. Identity management module 928 may be configured to provide identity services, such as access management and authorization services in cloud infrastructure system 900. In some embodiments, identity management module 928 may control information about customers who wish to utilize services provided by cloud infrastructure system 902. Such information may include information authenticating the identities of such customers and information describing what actions those customers are authorized to perform on various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). Identity management module 928 may also include management of descriptive information about each customer, as well as how and who can access and modify that descriptive information.

[0102] 10 illustrates an exemplary computer system 1000 upon which various embodiments of the present invention may be implemented. The system 1000 may be used to implement any of the computer systems described above. As shown in the figure, the computer system 1000 includes a processing unit 1004 that communicates with several peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and a system memory 1010.

[0103] Bus subsystem 1002 provides a mechanism for allowing the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0104] The processing unit 1004 may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers) and controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core processors or multi-core processors. In particular embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, with a single-core processor or a multi-core processor included in each processing unit. In other embodiments, the processing unit 1004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0105] In various embodiments, the processing unit 1004 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed may reside in the processor 1004 and / or in the storage subsystem 1018. Through suitable programming, the processor 1004 may provide various functionality as described above. The computer system 1000 may further include a processing acceleration unit 1006, which may include a digital signal processor (DSP), special purpose processor, etc.

[0106] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, a motion sensing and / or gesture recognition device such as a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and speech commands. User interface input devices may also include an eye gesture recognition device such as a Google Glass® blink detector that detects eye activity from a user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gesture as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® Navigator) via voice commands.

[0107] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, medical ultrasound machines, etc. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, etc.

[0108] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0109] Computer system 1000 may also include a storage subsystem 1018 that includes software elements currently shown as located in system memory 1010. System memory 1010 may store program instructions loadable and executable on processing unit 1004, as well as data generated during the execution of these programs.

[0110] Depending on the configuration and type of computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 1004. In some implementations, the system memory 1010 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1000, such as during start-up, may typically be stored in ROM. By way of example and not limitation, the system memory 1010 also illustrates application programs 1012, program data 1014, and an operating system 1016, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like. By way of example, operating system 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS operating systems.

[0111] The storage subsystem 1018 may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the functionality described above may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with the present invention.

[0112] The storage subsystem 1018 may also include a computer readable storage medium reader 1020 that may be further connected to a computer readable storage medium 1022. Along with, and optionally in combination with, the system memory 1010, the computer readable storage medium 1022 may collectively represent remote, local, fixed, and / or removable storage devices plus storage media for containing, storing, transmitting, and retrieving computer readable information on a temporary and / or more permanent basis.

[0113] The computer readable storage medium 1022 containing the code or portions of code may also include any suitable medium known or used in the art, including, but not limited to, storage media and communication media, such as volatile and non-volatile, removable and non-removable media implemented in any manner or technology for the storage and / or transmission of information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable medium. This may also include non-tangible computer readable media, such as data signals, data transmissions, or any other medium that may be used to transmit the desired information and that may be accessed by the computing system 1000.

[0114] By way of example, the computer readable storage medium 1022 may include hard disk drives that read and write to non-removable non-volatile magnetic media, magnetic disk drives that read and write to removable non-volatile magnetic disks, and optical disk drives that read and write to removable non-volatile optical disks or other optical media, such as CD ROMs, DVDs, and Blu-Ray® disks. The computer readable storage medium 1022 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer readable storage medium 1022 may also include flash memory-based SSDs, enterprise flash drives, solid state drives (SSDs) based on non-volatile memory such as solid state ROM, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1000.

[0115] The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for sending and receiving data between other systems and the computer system 1000. For example, the communications subsystem 924 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1024 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 1202.11 family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1024 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0116] In some embodiments, the communications subsystem 1024 may also receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. on behalf of one or more users who may be using the computer system 1000.

[0117] As an example, the communications subsystem 1024 may be configured to receive data feeds 1026 in real time from users of social networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0118] In addition, the communications subsystem 1024 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1028 of real-time events and / or event updates 1030, which may be continuous or infinite in nature with no explicit termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0119] The communications subsystem 1024 may also be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1000.

[0120] The computer system 1000 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0121] Due to the ever-changing nature of computers and networks, the description of the computer system 1000 shown in the figures is intended only as a specific example. Many other configurations are possible having more or fewer components than the system shown. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other aspects and / or methods for implementing various embodiments.

[0122] In the foregoing specification, aspects of the invention have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the invention is not limited thereto. The various features and aspects of the invention described above may be used individually or together. Moreover, the embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method comprising: accessing a communication workflow including one or more tasks arranged in a sequential order, the communication workflow configured to facilitate interaction with a set of user devices, each task of the one or more tasks including executable code that, when executed, performs a function associated with the set of user devices, the communication workflow associated with one or more parameters characterizing each task of the one or more tasks of the communication workflow, the method further comprising: generating a composite feature vector representing the communication workflow, the composite feature vector being generated using a feature vector for each task of the one or more tasks of the communication workflow, the feature vector for each task of the one or more tasks being generated by performing one or more machine learning techniques using the one or more parameters characterizing the task, the method further comprising: inputting the composite feature vector of the communication workflow into a trained machine learning model, the trained machine learning model being trained to generate a prediction of a performance value of the communication workflow, the trained machine learning model being trained using a training dataset representing one or more previously executed communication workflows and corresponding previous performance values ​​of each of the one or more previously executed communication workflows, the method further comprising:

22. A computer-implemented method comprising: generating an output using the trained machine learning model, the output predicting a performance value of the communication workflow.

2. The method further comprises: determining a structure of the communication workflow, the structure being represented by a plurality of nodes of a tree structure, two nodes of the plurality of nodes of the tree structure being connected by one or more stages, and each task of the one or more tasks of the communication workflow corresponding to a node of the plurality of nodes or a stage of the one or more stages, the method further comprising:

2. The computer-implemented method of claim 1, comprising evaluating the training data set to determine whether the structure of the communication workflow matches a structure of at least one previously executed communication workflow of the one or more previously executed communication workflows.

3. The method further comprises: determining that the structure of the communication workflow matches a structure of a group of previously-executed communication workflows of the one or more previously-executed communication workflows of the training data set; comparing said composite feature vector of said communication workflow with composite feature vectors of each previously executed communication workflow of said group of previously executed communication workflows in a domain space; selecting a subgroup of the group of previously executed communication workflows based on a threshold and a result of the comparison; identifying a previous performance value for each previously-executed communication workflow of said subgroup of previously-executed communication workflows; and generating the output predicting the performance value of the communicating workflow by determining a combination of the previous performance values ​​associated with a subgroup of the previously executed communicating workflows.

4. The method further comprises: determining that the structure of the communication workflow matches a structure associated with a group of previously executed communication workflows of the one or more previously executed communication workflows of the training data set, wherein during generation of the training data set: one or more subsets of previously executed communication workflows are determined, each subset of the one or more subsets corresponding to two or more previously executed communication workflows sharing a common structure; a clustering operation is performed on each subset of the one or more subsets of previously executed communication workflows, each subset of previously executed communication workflows being associated with one or more clusters of previously executed communication workflows, the one or more clusters being formed by performing the clustering operation using the composite feature vector associated with the subset of previously executed communication workflows in a domain space, each previously executed communication workflow included in a cluster being associated with a composite feature vector that corresponds to the composite feature vectors of other previously executed communication workflows in that cluster, the method further comprising: generating the output predicting the performance value of the communication workflow, said generating comprising: identifying a particular subset of the one or more subsets of the training data set whose structure corresponds to the structure of the communication workflow; assigning the communication workflow to a particular one of the one or more clusters corresponding to the particular subset based on a comparison of the composite feature vector of the communication workflow to the composite feature vectors associated with each cluster of the one or more clusters; and determining a combination of the previous performance values ​​associated with the previously executed communicating workflows that is associated with the particular cluster to which the communicating workflow is assigned.

5. The method further comprises: determining that the structure of the communication workflow matches a structure associated with a group of previously executed communication workflows of the one or more previously executed communication workflows of the training data set, wherein during generation of the training data set: one or more subsets of previously executed communication workflows are identified, each subset of the one or more subsets corresponding to two or more previously executed communication workflows sharing a common structure; A supervised machine learning model is trained for each subset of the one or more subsets of previously executed communication workflows, the method further comprising: generating the output predicting the performance value of the communication workflow, said generating comprising: identifying particular subsets of the one or more subsets of the training data set whose corresponding structure matches the structure of the communication workflow; inputting the composite feature vector of the communication workflow into the supervised machine learning model corresponding to the particular subset; and generating the output predicting the performance value of the communication workflow.

6. The method further comprises: The structure of the communication workflow is determined based on any previously executed communication workflow of the one or more previously executed communication workflows of the training data set. and generating the output predicting the performance value of the communication workflow, said generating comprising: segmenting each previously executed communication workflow of the one or more previously executed communication workflows of the training data set into a plurality of previously executed sub-workflows; comparing the structure of the communication workflow with each sub-workflow of the plurality of sub-workflows of each previously executed communication workflow of the training data set; identifying a group of previously executed sub-workflows among the plurality of previously executed sub-workflows that match the structure of the communication workflow; comparing the composite feature vector of the communicating workflow with a composite feature vector of each previously executed sub-workflow of the group of previously executed sub-workflows in a domain space; selecting a sub-group of the group of previously executed sub-workflows based on a threshold and a result of the comparison; and determining a predicted performance value of the communicating workflow based on the previous performance values ​​associated with the selected sub-group of previously executed sub-workflows.

7. The communication workflow includes: a structure of the communication workflow; Each task included in the communication workflow; 7. The computer-implemented method of claim 1, wherein the communication workflow is defined by a list including metadata describing the communication workflow and each feature vector representing a task of the one or more tasks, the list of metadata being analyzed to perform one or more functions.

8. 1. A system comprising: one or more processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more processors, cause the one or more processors to perform operations, the operations including: accessing a communication workflow including one or more tasks arranged in a sequential order, the communication workflow configured to facilitate interaction with a set of user devices, each task of the one or more tasks including executable code that, when executed, performs a function associated with the set of user devices, the communication workflow associated with one or more parameters characterizing each task of the one or more tasks of the communication workflow, the operation further comprising: generating a composite feature vector representing the communication workflow, the composite feature vector being generated using a feature vector for each task of the one or more tasks of the communication workflow, the feature vector for each task of the one or more tasks being generated by performing one or more machine learning techniques using the one or more parameters characterizing the task, the operations further comprising: inputting the composite feature vector of the communication workflow into a trained machine learning model, the trained machine learning model being trained to generate a prediction of a performance value of the communication workflow, the trained machine learning model being trained using a training dataset representing one or more previously executed communication workflows and corresponding previous performance values ​​of each of the one or more previously executed communication workflows, the operations further comprising: and generating an output using the trained machine learning model, the output predicting a performance value of the communication workflow.

9. The operation further comprises: determining a structure of the communication workflow, the structure being represented by a plurality of nodes in a tree structure, two nodes of the plurality of nodes of the tree structure being connected by one or more stages, each task of the one or more tasks of the communication workflow corresponding to a node of the plurality of nodes or a stage of the one or more stages, the operations further comprising:

10. The system of claim 8, further comprising: evaluating the training data set to determine whether the structure of the communication workflow matches a structure of at least one previously executed communication workflow of the one or more previously executed communication workflows.

10. The operation further comprises: determining that the structure of the communication workflow matches a structure of a group of previously-executed communication workflows of the one or more previously-executed communication workflows of the training data set; comparing said composite feature vector of said communication workflow with composite feature vectors of each previously executed communication workflow of said group of previously executed communication workflows in a domain space; selecting a subgroup of the group of previously executed communication workflows based on a threshold and a result of the comparison; identifying a previous performance value for each previously-executed communication workflow of said subgroup of previously-executed communication workflows; and generating the output predicting the performance value of the communication workflow by determining a combination of the previous performance values ​​associated with a subgroup of the previously executed communication workflows.

11. The operation further comprises: determining that the structure of the communication workflow matches a structure associated with a group of previously executed communication workflows of the one or more previously executed communication workflows of the training data set, wherein during generation of the training data set: one or more subsets of previously executed communication workflows are determined, each subset of the one or more subsets corresponding to two or more previously executed communication workflows sharing a common structure; a clustering operation is performed on each subset of the one or more subsets of previously executed communication workflows, each subset of previously executed communication workflows being associated with one or more clusters of previously executed communication workflows, the one or more clusters being formed by performing the clustering operation using the composite feature vector associated with the subset of previously executed communication workflows in a domain space, each previously executed communication workflow included in a cluster being associated with a composite feature vector that corresponds to the composite feature vectors of other previously executed communication workflows in that cluster, the operation further comprising: generating the output predicting the performance value of the communication workflow, said generating comprising: identifying a particular subset of the one or more subsets of the training data set whose structure corresponds to the structure of the communication workflow; the composite feature vector of the communication workflow and each class of the one or more clusters assigning the communication workflow to a particular one of the one or more clusters corresponding to the particular subset based on a comparison of the composite feature vector associated with the particular subset to the composite feature vector associated with the particular subset; and determining a combination of the previous performance values ​​associated with the previously executed communication workflows that are associated with the particular cluster to which the communication workflow is assigned.

12. The operation further comprises: determining that the structure of the communication workflow matches a structure associated with a group of previously executed communication workflows of the one or more previously executed communication workflows of the training data set, wherein during generation of the training data set: one or more subsets of previously executed communication workflows are identified, each subset of the one or more subsets corresponding to two or more previously executed communication workflows sharing a common structure; A supervised machine learning model is trained for each subset of the one or more subsets of the previously executed communication workflows, the operations further comprising: generating the output predicting the performance value of the communication workflow, said generating comprising: identifying particular subsets of the one or more subsets of the training data set whose corresponding structure matches the structure of the communication workflow; inputting the composite feature vector of the communication workflow into the supervised machine learning model corresponding to the particular subset; and generating the output predicting the performance value of the communication workflow.

13. The operation further comprises: determining that the structure of the communication workflow does not match a structure of any previously-executed communication workflow of the one or more previously-executed communication workflows of the training data set; generating the output predicting the performance value of the communication workflow, said generating comprising: segmenting each previously executed communication workflow of the one or more previously executed communication workflows of the training data set into a plurality of previously executed sub-workflows; comparing the structure of the communication workflow with each sub-workflow of the plurality of sub-workflows of each previously executed communication workflow of the training data set; identifying a group of previously executed sub-workflows among the plurality of previously executed sub-workflows that match the structure of the communication workflow; comparing the composite feature vector of the communicating workflow with a composite feature vector of each previously executed sub-workflow of the group of previously executed sub-workflows in a domain space; selecting a sub-group of the group of previously executed sub-workflows based on a threshold and a result of the comparison; and determining a predicted performance value of the communication workflow based on the previous performance values ​​associated with a subgroup of the selected previously executed sub-workflows.

14. The communication workflow includes: a structure of the communication workflow; Each task included in the communication workflow; and each feature vector representing a task of the one or more tasks of the communication workflow, the list of metadata being analyzed to perform one or more functions.

15. A program for causing a system to execute the method according to any one of claims 1 to 7.

Images