Directed trajectories through a communication decision tree using iterative artificial intelligence

Abstract

The present disclosure relates to directed trajectories through a communication decision tree using iterative artificial intelligence. Embodiments relate to configuring artificial intelligence (AI) decision nodes throughout a communication decision tree. The decision nodes can support continuous iteration of the AI ​​model to dynamically define iterative data corresponding to the trajectory through the tree.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of U.S. Application No. 16 / 007,677, filed on June 13, 2018, which claims the benefit of and priority to U.S. Provisional Application No. 62 / 566,026, filed on September 29, 2017. Each of these applications is incorporated herein by reference in its entirety for all purposes. Background Art

[0003] Technological advances have improved the accessibility and complexity of various types of communication channels. In addition, data storage and network advances have increased capacity, making it possible to store increasing amounts (and types) of data at the data source for potential transmission. Therefore, data sources can be located to deliver various types of data across any one of multiple data channels at many potential times. When considering multiple related content delivery rather than a single distribution, the array of content delivery options will explode. Content providers often configure one or more static rules to indiscriminately provide the same content to each data ingestor through the same communication channel. Although (one or more) communication specifications may be different between receiving different data requests, (one or more) rules can be configured to respond to data requests indiscriminately and consistently. Although this approach provides configuration simplicity and deterministic operation, it cannot react to the potential variability between groups of data ingestors, and therefore may handle requests suboptimally. Summary of the invention

[0004] In some embodiments, a computer-implemented method is provided. A data structure representing a communication decision tree is accessed, and the communication decision tree is configured to dynamically define individual trajectories through the communication decision tree using machine learning techniques to indicate a series of communication specifications. The communication decision tree includes a set of branch nodes. Each branch node of the set of branch nodes corresponds to an action point configured to identify the direction of a given trajectory. At a first time, it is detected that the trajectory through the communication decision tree has reached a first branch node of the set of branch nodes. The trajectory is associated with a specific user. In response to detecting that the trajectory has reached the first branch node, first learned data generated by processing first user data using machine learning techniques is retrieved. The first user data includes user attributes for other user sets. Further in response to detecting that the trajectory has reached the first branch node, one or more specific user attributes associated with the specific user are retrieved, one or more first communication specifications are identified based on the first learned data and the one or more specific user attributes, and the first content is transmitted to a user device associated with the specific user according to the one or more first communication specifications. At a second time after the first time, it is detected that the trajectory through the communication decision tree has reached a second branch node of the set of branch nodes. In response to detecting that the trajectory has reached the second branch node, second learned data generated by processing second user data using machine learning techniques is retrieved. The second user data includes at least some user attributes not included in the first user data. In further response to detecting that the trajectory has reached a second branch node, one or more second communication specifications are identified based on the second learned data and at least some of the one or more specific user attributes, and the second content is transmitted to the user device according to the one or more second communication specifications.

[0005] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided. The computer program product may include instructions configured to cause one or more data processors to perform some or all of the operations of one or more methods disclosed herein.

[0006] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of the operations of one or more methods disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Illustrative embodiments of the present invention are described in detail below with reference to the following drawings:

[0008] Figure 1 A block diagram of an interactive system is shown.

[0009] Figure 2A template for generating an email communication is shown.

[0010] Figure 3 A template for generating an app notification communication is shown.

[0011] Figure 4 A representation of a communication decision tree is shown.

[0012] Figure 5 An example of a trace corresponding to a user device and extending through a communication decision tree is illustrated.

[0013] Figure 6 An exemplary interface for defining a communication decision tree is shown.

[0014] Figure 7 An exemplary parameter definition interface for a switch icon is shown.

[0015] Figure 8 Another exemplary parameter definition interface is shown that includes options for implementing biases toward or against various content in a communication.

[0016] Fig. 9 Another exemplary parameter definition interface is shown that includes options for sending communications using various communication channels to effectuate a bias toward or against.

[0017] Fig.10 A flow chart is shown of a process for using machine learning to direct a trajectory through a communication decision tree according to some embodiments of the invention.

[0018] Fig.11 A flow chart illustrating a process for defining a machine learning based communication decision tree using an interface supporting locatable visual elements.

[0019] Fig.12 An exemplary progressive filtering decision tree according to some embodiments of the present invention is shown.

[0020] Fig.13 An exemplary representation of data processing occurring during implementation of a progressive filtering decision tree according to some embodiments of the present invention is shown.

[0021] Fig.14 A flow chart of a process for iteratively advancing a trajectory through a decision tree using a machine learning model according to some embodiments of the invention is shown. DETAILED DESCRIPTION

[0022] In some embodiments, a system and method for reusing machine learning data to facilitate iterative identification of communication specifications is provided. More specifically, a communication decision tree including a set of nodes is generated. Each node may correspond to, for example, a detected event or a branch node, which corresponds to a communication specification decision and is connected to a plurality of next nodes representing communications corresponding to one or more specific communication specifications. Each individual track through the communication decision tree may correspond to an individual user and / or one or more specific user devices. Each individual track may extend over a specific subset of a node set, wherein the nodes in the subset represent specific actions initiated by a user and / or initiated at a specific user device in one or more specific devices, specific characteristics of communications sent to a specific user device in one or more specific devices; and / or decisions to be made regarding the specifications of upcoming communications. For example, the specifications of the upcoming communications may indicate when it will be sent, which device it will be sent to, which type of communication channel it will be sent through, and / or which type of content it will include. In some cases, natural language processing may be used to assign one or more categories to each of the one or more content objects sent in a training set and / or to each of the one or more content objects available for transmission. Then, the communication specification may identify specific categories of content to be sent.

[0023] Each communication specification decision may be made based on current data corresponding to a user and / or a particular user device, a machine learning model, and one or more learned parameters for the machine learning model. The parameter(s) may be learned based on user data associated with a set of other users, and for each user in the set of other users, the user data indicates a set of one or more attributes and / or events of the other user (e.g., characteristics of actions initiated by the user or communications sent to the user). The parameter(s) may be further learned based on a trajectory target (e.g., identified by a client) corresponding to a particular node within the communication decision tree and / or a particular action initiated by a user.

[0024] The communication decision tree can be configured to include multiple branch nodes so that multiple communication specification decisions can be made for a single trajectory. In some cases, the same type of machine learning algorithm (e.g., a supervised regression algorithm) is used to make each decision. However, the algorithm can be configured differently at each branch node so that the branch node is different with respect to (for example) the input type processed for each trajectory and / or the learned parameters for processing the input corresponding to the trajectory. In various cases, algorithms for different branch nodes can be trained to optimize the same or different variables (e.g., based on the identification of the same target node or different target nodes). Not only can the branch node change with respect to the input type that the algorithm is configured to process, but the type of profile data that can be used to potentially be processed for a given user can also change (e.g., due to interaction monitoring, profile data can accumulate over time). In addition, the learned data associated with any given node can change over time (due to continuous and / or repeated learning).

[0025] As an example, a user's track can be initialized upon detecting that profile data corresponding to the user includes information of at least a predefined set of fields. One or more web servers can be used to collect profile data over one or more sessions associated with the user and / or retrieve profile data from a remote data source. In some cases, the user device automatically detects at least some of the profile data and transmits it to the (one or more) web servers (e.g., via header information that is automatically populated in the communication, which identifies, for example, a unique device identifier, a MAC address, a browser type, a browser version, an operating system type, an operating system version, a device type, a language to which the device is set, etc.). In some cases, the communication includes data representing user input (e.g., text entered into a web form, link selection, page navigation, etc.), which can then be recorded as profile data.

[0026] Initializing the trajectory may include identifying a first node within a communication decision tree, the first node may include a first branch node. The first decision node may correspond to a decision regarding identifying which of a plurality of content objects to transmit to a user's device within an email communication (e.g., identifying various groups of items and / or information associated with a website). The first decision node may also correspond to a decision regarding when to send an email within a two-day period. A decision may be made based on first learned data indicating—for a particular type of user—what types of objects and / or communication times are most likely to lead to a target outcome. For example, the target outcome may include an occurrence of a user activating a link within an email to visit a page on a website and / or a user interacting with the website in a manner corresponding to a conversion (e.g., purchasing an item represented on the website), and the first learned data may indicate predictors regarding which of the three content objects will be more effective in leading to the target outcome including whether the user most frequently uses a mobile device (relative to a laptop or computer), the user's age, and prior email interaction indications regarding which types of content objects the user clicked on the link.

[0027] Once the email is sent, the trajectory can be extended to the node representing the content sent until the next event is detected. The next event can include (for example) activating a link within the website, thereby indicating that the user is engaged in a current session with the website. After detecting this event, the trajectory can be extended to a second decision node to determine how to configure the requested web page on the website (for example, whether to include dynamic content objects and / or how to arrange various content objects). In this example, the second learned data indicates which configurations are most likely to lead to the same target result for a particular type of user. For example, the second learned data can indicate that the predictors of which of the four configurations is more effective in leading to the target result include whether the user uses a mobile device most frequently (relative to a laptop or computer), the browser type, the current location of the user's device, and the current time at the user's location. Once the web page is sent (configured according to the decision made at the second decision node), the trajectory can be extended to the node representing the configuration of the sent web page. After detecting various user-initiated, system-initiated, or external events (for example, a predefined time interval since a previous event), the trajectory can continue to extend.

[0028] In this example, the target result remains the same in multiple decisions. However, rather than a static workflow that identifies the actions to be performed - or even a complete user-specific sequence that determines the actions to be performed - the technology disclosed herein bases decisions about individual actions on current profile data, current learned data, and current event detection. Machine learning is iteratively performed throughout the life cycle of a specific trajectory to identify piecemeal actions to be performed. This approach can promote high utilization of data (e.g., the expansion and / or evolution of learned data and / or profile data can be utilized in mid-process decisions), which can facilitate achieving goals. In addition, this approach enables changes (e.g., initiated by the client) to the definition and / or constraints of machine learning techniques to act quickly (e.g., because the change can still affect the trajectory that has been initiated). For example, the client can change the goal from conversion to retaining the user device on the website for at least a threshold session duration. Modified parameters for machine learning models associated with various branch nodes can be determined and immediately implemented to affect previously initiated trajectories that subsequently reach (one or more) nodes.

[0029] Communication decisions (and / or partial guidance by communication decision trees) can be based on anonymous or partially anonymous data, any one of which or both can be constructed from anonymous, partially anonymous or non-anonymous data provided by one or more providers or clients. For example, a remote user data management system can receive partially anonymous or non-anonymous data from one or more data providers, and can blur or eliminate fields in individual records and / or can aggregate field values ​​between user subgroups to comply with data privacy rules according to data privacy rules. As described herein, anonymous or partially anonymous data is data that has been stripped of PII and / or aggregated so that individual data values ​​cannot exceed a certain probability and are associated with a specific person or user. Therefore, anonymous or partially anonymous data can lack or blur enough data values ​​to prevent a specific person from being identified as a specific user or to prevent a specific person from being identified as a user with at least a threshold probability. For example, anonymous or partially anonymous data can lack a name, email address, IP address, physical address and / or telephone number from profile data. Anonymous or partially anonymous data can include or exclude certain demographic data, such as age, city, occupation, etc. In some cases, anonymous or partially anonymous data is useful for collecting in order to comply with privacy rules, regulations and laws, while also being able to process some of these data. Anonymous or partially anonymous data may include information collected from a device based on IP address range, zip code, date, category of prior online queries, race, gender, age, purchase history and / or browsing history, etc., which may have been collected pursuant to various privacy policies and laws that restrict the flow of personally identifiable information (PII).

[0030] In some cases, anonymous or partially anonymous data is used to generate and / or update learned data (e.g., one or more parameters) associated with individual machine learning configurations. This type of training does not (for example) require or benefit from data fields such as contact information, so these fields of the data record can be stripped. As another example, one or more subpopulations can be generated based on the value of a particular field, and the specific value of that field can thereafter be replaced with an identifier of the subpopulation.

[0031] In some cases, profile data corresponding to a particular user for whom a decision is being made includes anonymous or partially anonymous data. For example, the system may detect that a trajectory has reached a branch node and request data from a user data management system (e.g., using a device identifier or other identifier associated with the trajectory). The system may return profile data that includes, for example, one or more specific non-anonymous field values, one or more field values ​​that have been generalized (e.g., assigned to a category), and / or eliminated field values. Non-anonymous field data may be included in the profile data when, for example, such field values ​​are supplied (e.g., after having been collected using a data collection feature built into a web page and / or via a transmission from a client) or (e.g., collected using a data collection feature) or otherwise made accessible to the client for whom a decision is being made (e.g., via a data sharing agreement). The system may also return demographic data indicating relationships between field values ​​(e.g., which itself may be learned and / or may evolve over time), which may be used to estimate the value or category of a missing field value.

[0032] Figure 1A block diagram of an interactive system 100 is shown. The machine learning data platform 105 may include one or more cloud servers and may be configured to receive user data from one or more client systems 110. The user data may include anonymous or partially anonymous user data (stored in an anonymous user data repository 115) and / or secure client useful user data (stored in a secure client useful user data repository 120), some of which may be less anonymous or non-anonymous than anonymous user data. When secure client useful user data is received, it may be securely stored in association with an identifier of the client so that other clients cannot get access to the data. The data may be stored in a multi-tenant cloud storage system so that multiple clients can log in to a central location to access a server or a collection of servers, but where specific access to the data is controlled depending on which client has authenticated to the cloud storage system. Anonymous or partially anonymous user data may or may not be specifically configured for various clients (e.g., depending on what data the client supplies and / or a data sharing agreement associated with the client). Thus, the profile data populator 122 at the machine learning data platform 105 can generate profile data corresponding to one or more individual users of a particular client, and can customize which field values ​​are included in the profile data for the individual client.

[0033] In some cases, the profile data populator 122 enhances the profile data set to supplement the client useful user data with partially anonymous user data, which can (e.g., aggregated) define client-specific learned data for a given user (stored in the client-specific learned data repository 130). For example, data from a profile in the client useful data can be mapped to one or more data sets in the anonymous or partially anonymous user data so that a richer data set can be used in the machine learning analysis. The mapping can be done using overlapping data (e.g., IP address (if included in the anonymous or partially anonymous user data), purchase time, pseudo-random user identifier assigned by the client, etc.).

[0034] The machine learning model configurator 123 can configure a given machine learning model based on, for example, an identified target outcome, available training data, constraints identified by one or more clients, and / or potential actions indicated by a communication decision tree and / or by a client. Configuring a machine learning model can include defining one or more parameters for a particular instance of the model (e.g., an instance is associated with a particular branch node, client, and / or time period).

[0035] Each parameter may indicate a relationship and / or correlation between user attributes (stored in the learned parameter data store 125). The parameter(s) may include a weight indicating how a first user attribute predicts a second user attribute and / or the degree to which the second user attribute predicts the second user attribute, where the second user attribute corresponds to an indication of whether and / or the degree to which a target outcome occurred. The weights may be defined along a discrete or continuous range of values ​​and / or may be binary.

[0036] As an example, the parameter(s) may indicate which attributes in a set of attributes predict future occurrences of a particular type of conversion event. For example, it may be determined that having visited a web page associated with the tag "travel" more than twice in the last month is a predictor of purchasing a piece of luggage. As another example, it may be determined that having visited a movie review web page within a given date is a predictor of later purchasing an online rental of a movie. Indirect associations and trends may also be learned, such as recognizing that there is an inverse correlation between a user's age and the average time spent online per day. Each parameter may be associated with the strength and / or confidence of a relationship, optionally with a sequence of associations between the data points collected and the conclusion being made, where each association in the sequence carries some probability that the data at the start of the association is accurate for what it says and some other probability that the association itself is accurate.

[0037] The configuration and / or generation of client-specific parameters may be performed but not necessarily using profile data available to the client. The client-specific parameter(s) may be, for example, a modified version of the parameter(s) generated using anonymous or partially anonymous profile data.

[0038] Various machine learning techniques can be used to generate the learned data. For example, machine learning techniques can use decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, or rule-based machine learning. In some cases, machine learning techniques include set techniques that learn inter-set weights to apply to results generated from various underlying technologies (such as two or more of those previously mentioned). Inter-set weights can be identified based on, for example, accuracy, speed, and / or resource utilization associated with the underlying technology.

[0039] Training a machine learning technique (to identify one or more parameters) can include identifying how a set of observed inputs (e.g., content of marketing emails, content of promotions, and / or configurations of websites) relate to a set of corresponding outputs (e.g., outcomes, such as the presence or absence of certain conversion events for marketing emails, corresponding promotions, and / or corresponding website configurations). These observed observations can be used to identify modeled relationships and / or trends with the goal of predicting candidate factual information that has not yet occurred (e.g., a predicted next input to be received or a predicted output based on certain inputs) based on the factual information that led to the candidate factual information. Each prediction can carry a confidence or probability, and a chain of predictions has a combined confidence or probability.

[0040] Thus, the machine learning model configurator 123 can identify model parameters for a specific client system 110 based on, for example, target results, client-specific profile data, and / or machine learning techniques. Client-specific learned data can be selectively shared with client systems that have been provided with underlying client useful profile data. The client system 110 may include a system that hosts one or more websites, hosts one or more apps, and / or enables emails to be sent. For example, the client system 110 may include a web server 135 that receives and responds to HTTP requests for pages on one or more domains, and an email server 140 that delivers emails to a user's email address. The client system 110 may also or alternatively include an app server 145 to receive and respond to requests received via an application executed on a user device. Thus, one or more servers at the client system 110 may be configured to detect requests from one or more user devices 150-1, 150-2 and / or trigger the transmission of content to one or more user devices 150-1, 150-2. The user devices 150-1, 150-2 may include, for example, computers, smart phones, tablets, and the like. It can be appreciated that, in various situations, a single user device may be associated with a single user or more than one user. In addition, a single user may be associated with a single user device or more than one user device.

[0041] Web server 135 and / or app server 145 may store indications of requests for content (e.g., web pages or app pages) from content library 153 as user data in client-managed user data repository 150. The stored data may include automatically detected information (e.g., request time) as well as information included in the request (e.g., device identifier, IP address, requested web page, input entered by the user, etc.). Storing the data may include updating a profile to include the data. Web server 135, email server 140, and / or app server 145 may also store data in client-managed user data repository 150 that indicates which content is distributed to a particular user device (e.g., by identifying a transmission time, a user device identifier, (one or more) content object identifiers, and / or a type of communication).

[0042] The client system 110 may send at least a portion of the user data from the client-managed user data repository 150 to the machine learning data platform 105, which may store it in the secure client useful user data repository 120. The transmission(s) may occur periodically, during a request for client-specific learned data, at regular time intervals, etc. In some cases, the client system 110 at least partially anonymizes some or all of the user data (e.g., by omitting or obfuscating the values ​​of at least some fields) before sending the user data to the machine learning data platform (e.g., causing such user data to be stored at the platform as anonymous or partially anonymous user data). In some cases, the data is not at least partially anonymous, such that the data is either stored in the secure client useful user data repository 120 or at least partially anonymized at the machine learning data platform 105. For some data sets, the anonymous or partially anonymous data is received from a third party after being stripped of PII and stored by the client system 110 without access to the non-anonymous data. In some embodiments, the anonymous or partially anonymous data is locally anonymous or partially anonymous. In these embodiments, a website may run an embedded script on its website that, when executed, collects anonymous or partially anonymous data about a user's visit to the website. The script may only collect information that can be collected and stored in the data cloud without knowing the user's personal information, which ensures that the user's identity can never be inferred beyond a certain probability.

[0043] The client system 110 may store the machine learning data in the machine learning data repository 155. In some cases, the machine learning data includes an indication of one or more decisions made for a given trajectory at a branch node, one or more content specifications identified using a communication decision tree and / or one or more parameters. The machine learning data may be requested, received, and / or derived from data from the machine learning platform 105. For example, in some cases, the machine learning model configurator 123 causes parameters generated for and / or applicable to a client to be sent to the client system 110. As another example, the machine learning model implementer 157 may apply a machine learning model configured with specific parameters to specific profile data to identify one or more specific communication specifications to define a communication action to be taken for a client corresponding to the profile data (and / or a next node of the trajectory). The machine learning model implementer 157 may then cause the identified communication action and / or indication of the next node to be sent in association with an identifier of the trajectory, user, and / or user device.

[0044] Identifying the next node and / or (one or more) communication specifications can include running a machine learning model (associated with the current branch node) using specific profile data and one or more learned parameters. The results can indicate (for example) which of the various content presentation characteristics is associated with a high (e.g., above a threshold) or highest probability of causing a specific target result (e.g., a target conversion). In some cases, the analysis includes identifying one or more content presentation characteristics associated with the highest probability of causing a specific conversion target result. In some cases, the analysis balances the probability of causing a specific conversion result with a predefined cost metric associated with the various content presentation characteristics.

[0045] In some cases, running the machine learning model using the parameters (e.g., at the machine learning data platform 105 or the client system 110) may include, for example, performing a regression analysis using the profile data and the parameters to generate a number that can be compared to one or more thresholds. The one or more thresholds may define two or more ranges (e.g., an open range or a closed range), each corresponding to a particular next node and / or communication action. In some cases, running the machine learning model using the parameters may include processing at least a portion of the profile data and at least a portion of the parameters to produce a result that can be compared (e.g., via calculating a difference, calculating a cost using a cost function, etc.) to each of a set of reference data variables (e.g., a single value, a vector, a matrix, a time series, etc.) - each associated with a particular next node and / or communication action and each potentially defined at least in part based on the parameters. A node or communication associated with a reference data variable for which a comparison indicates the closest correspondence may be selected.

[0046] The dynamic content generator 147 may trigger the presentation of the content object according to the selected communication specification(s). To generate the appropriate instructions, the dynamic content generator 147 may first identify what communication channel to use to send the object, the type of object to send, the version of the content to send, and / or when the content object is to be sent. The identification may be determined based on, for example, implementation results of a machine learning model, a configuration of the machine learning model (e.g., which may constrain potential options regarding one or more of these options), and / or one or more parameters.

[0047] Dynamic content generator 147 may identify the type of communication to send (e.g., an email, SMS message, pop-up browser window, or pushed app alert), which may notify, for example, web server 135, email server 140, and / or app server 145, which one to send the communication. The identification may be made explicitly (e.g., based on machine learning results, parameters, and / or machine learning model configuration) or implicitly (e.g., because the selected content object is of a particular type).

[0048] Identifying a content object may include selecting or generating a new content object from a set of existing content objects. The content object may include, for example, a web page, an object within a web page, an image, a text message, an email, an object within an email, and / or text. In some cases, the result of executing a configured machine learning model on the profile data identifies a specific content object. In some cases, the result identifies the characteristics of the content (e.g., having a specific metadata category) and / or identifies a specific technique for selecting content. For example, the result may indicate that a "tool" item will be characterized in the content object and / or that the communication will include four content objects corresponding to four different (but unspecified) categories. In this case, the dynamic content generator 147 may, for example, select from a set of potential content objects using a selection technique, which is indicated, for example, via the results of a machine learning implementation, via parameters, and / or via predefined settings. For example, the selection technique may indicate that the selection technique will include a pseudo-random selection technique, a technique for identifying recently added objects, a technique for identifying the highest conversion object within a set of potential content objects (e.g., having one or more attributes indicated in the machine learning results).

[0049] In some cases, times at which communications are to be sent are explicitly identified (e.g., based on machine learning results, parameters, and / or machine learning model configurations). For example, a time range can be defined as starting at the current time and ending at a maximum time identified by the client. The model can evaluate a set of regularly spaced potential transmission times within the time range. (In some cases, each potential transmission time is considered multiple times in combination with other potential specifications (such as content categories or communication channels)). The machine learning model results can identify transmission times associated with the highest probability of resulting in a target outcome. Notably, if a combination of specifications is considered, the transmission times can include times in the combination associated with the highest probability. In some cases, communications are sent immediately after the next request for content (e.g., corresponding to a given website or app) is received from a user device associated with the machine learning results, or according to a predefined transmission schedule.

[0050] In some cases, each specification corresponding to the communication is identified before or at the time of sending the communication (e.g., during a task and / or using a machine learning model, machine learning configuration, parameters, client rules, etc.). Thus, all or some client-controlled configuration of the communication and / or the communication transmission can be performed before the communication is transmitted. In some cases, at least one specification corresponding to the communication is identified after the communication is sent (e.g., during a task and / or using a machine learning model, machine learning configuration, parameters, client rules, etc.). Thus, at least some client-controlled configuration of the communication and / or the communication transmission can be performed after the communication is transmitted. Thus, such post-transmission configuration can be based on learned data and / or profile data that was not available before the communication was transmitted. For example, additional profile data corresponding to a user can become available between a first time an email is sent and a second time the email is opened and rendered. The sent email may include a script that is executed when the email is to be rendered. The script may cause a request to be issued to identify device properties, such as a layout and / or application type. The script may pass these properties and a request for content to be presented to the server. Thus, the server may select content and / or identify one or more display configurations based on specific rendering information, current profile data, and / or current parameters to guide selection of specific content.

[0051] As an additional or alternative example, a communication may contain one or more references or links to a page that, when opened (e.g., in a web browser), renders content for display. The page targeted by the link may include some content determined by the machine learning engine before or at the time of generating the communication. The page may also be configured to include content to be selected or generated when a request to render the page is detected (e.g., when a script detects activation of a link) and / or when the page is being generated or rendered (e.g., as indicated by executing a script as part of loading the page). In some cases, a script in the email identifies the content configuration at rendering time or when rendering is requested. In some cases, a script executed on the linked page identifies the content configuration.

[0052] As an example, a client system may provide online purchase of food delivery. It may be detected that a particular user viewed the menu of a given restaurant at 2 p.m. The client system may retrieve a set of user attributes from the user's profile data from its client-managed user data. The client-specific learned data may indicate that if an email including a discount code is sent to the user in the evening, the user will have a 76% chance of making a purchase from the restaurant (e.g., compared to lower probabilities associated with other types of communications and other times). In response to determining that the 76% chance is above the 65% threshold for sending the discount threshold, the email server 140 sends an email to the user device. The email includes a script that, when executed, identifies the restaurant and discount to be presented. The user opens the email at 10 a.m. the next day. The code is executed to request the restaurant and discount from the client system. Thereafter, the client system has received the updated public learned relevant data. The client system inputs the time, the user's location (when she is now at work), and the prior purchase information into a decision tree constructed based on the learned data. It is determined that the discount is 10% (e.g., to maintain a threshold likelihood of conversion) and the restaurant will be a deli close to the user's workplace (e.g., to maximize the likelihood of conversion), while if the user opened the email the night before, different user attributes and learned data would result in a 15% discount (e.g., to maintain a threshold likelihood) from an Indian restaurant near the user's home (e.g., to maximize the likelihood). The email includes a link to order from the deli. When the user clicks on the link, the web server determines what content to present - specifically, which food items to recommend. These recommendations are based even more on recently updated public learned related data, which indicates that salads and sandwiches should be recommended over soups and entrees because the former choices have been popular recently (expected to be popular due to warmer weather). Thus, this example illustrates how content presentation can be dynamically customized for a given user based on recently learned data and user attributes.

[0053] The machine learning data platform 105 can generate updated client data based on, for example, any communication received from a user device (e.g., in response to a workflow action). For example, the updated client data can include one or more new fields generated based on data in a header or payload of a received communication, an indication of whether (e.g., and when) a particular event was detected, and / or the current or last stage of a workflow to which the profile is assigned. The machine learning data platform 105 can utilize the updated client data (e.g., along with a corresponding profile identifier) ​​to the client system 110, which can store the updated data in a client-specific learned data repository 165. The client system 110 can, but need not, store the updated data separately from the underlying profile(s).

[0054] It will be appreciated that in some cases, some or all of the machine learning data platform may be incorporated within the client system 110. In some cases, the client system 110 communicates with the machine learning data platform during iterations of the communication decision tree. For example, the client system 110 (e.g., a web server 135 or an app server 145 at the client system 110) may detect a tag in a request for web content or app content received from a user device (e.g., included in a URL), where the tag indicates its association with a machine learning-based workflow. The client system 110 may then alert the machine learning model implementer 157 of the request so that the trajectory may be updated appropriately.

[0055] The machine learning data platform, client system 110, and user devices 150-1, 150-2 may communicate via a network 160, which may include, for example, the Internet, a local area network, a wide area network, etc. It will be appreciated that various alternatives to the depicted and described embodiments may be considered. For example, some or all of the machine learning may be performed at the client system 110. The client system 110 may periodically receive anonymous or partially anonymous user data for processing using machine learning techniques.

[0056] Other techniques for using and configuring communication decision trees are described in detail in U.S. Application No. 16 / 007,762, filed on June 13, 2018, entitled “Methods and Systems for Configuring Communication Decision Trees based on Connected Positionable Elements on Canvas” and U.S. Application No. 16 / 007,787, filed on June 13, 2018, entitled “Machine-Learning Based Processing of De-Obfuscated Data for Data Enrichment”. Each of these applications is incorporated herein by reference in its entirety for all purposes.

[0057] Figure 2 and Figure 3 Interfaces 200 and 300 for configuring templates 202 and 302 for communication are illustrated, which are configured to be partially configured upon detection of a rendering process or at rendering time.Configuration may include executing a configured machine learning model using a current learned model configuration and current profile data. Figure 2 The template 202 shown in FIG. 1 includes a template for generating an email communication, and Figure 3 The template 302 shown in FIG. 1 includes a template for generating an app notification communication.

[0058] Template 202 includes static text (e.g., text 205) and interactive features (e.g., button 210). Template 202 also represents a specific layout in which three items will be represented linearly above text 205. Template 202 also includes dynamic components (e.g., dynamic text 215 and dynamic image 220) that are configured to be recognized when rendering of an email is requested or occurs. Therefore, when sending an email communication, the static components can be sent together with code that is configured to (upon detection of a request to render an email) locally recognize at least a portion of the current profile data, request at least a portion of the current profile data, request recognition of the dynamic components, receive or retrieve the dynamic components (e.g., using the current profile data, current anonymous or partially anonymous data, and / or current learned parameters to recognize), and / or generate a complete email based on the template and the dynamic components. The generated email can then be presented.

[0059] Template 302 includes a static layout and multiple dynamic text components (e.g., a dynamic title portion 310). Template 302 can be configured to be sent together with a script that facilitates dynamic identification of each dynamic text component. For example, the script can - after detecting a request to present a notification (e.g., in response to opening an app, clicking a notification app element, etc.) - locally identify at least a portion of the current profile data, request at least a portion of the current profile data, request identification of a dynamic text component, receive or retrieve a dynamic text component (e.g., using current profile data, current anonymous or partially anonymous data, and / or currently learned parameters to identify) and / or generate a complete notification based on the template and dynamic text component. The generated notification can then be presented. Interface 300 shows an example of a dynamically generated notification 315, which includes a static layout and specific dynamic text.

[0060] Figure 4 A representation of a communication decision tree 400 is shown. Communication decision tree 400 includes a start node 405 at which each trajectory begins. A particular trajectory may be initiated (in this example) upon detecting that a user has completed two particular actions (e.g., initiated two website sessions, purchased two items from a website, navigated to at least two web pages on a website, etc.).

[0061] The communication decision tree 400 includes three branch nodes 410, 415, and 420—each branch node branches to connect to three nodes representing three different actions. The trajectory can automatically and immediately extend from the initial node 405 to the first branch node 410, which triggers the first decision to be made. Specifically, the first decision can include identifying a communication channel to be used to send an alert of a website feature. The alert can include an automatically presented static header indicating (for example) that a product or discount is (generally) available in association with the website. The alert can also be associated with dynamic content (e.g., specifically identifying one or more products and / or discounts) to be identified at the second branch node 415 after detecting a request to open the notification.

[0062] The first branch node 410 is connected to a first action node 425a representing an email communication channel, a second action node 425b representing an SMS message communication channel, and a third action node 425c representing an app-based communication channel (where notifications will be pushed to and / or by applications installed on the user's device).

[0063] A first decision may be made using a machine learning model configured based on one or more first parameters. One or more first parameters may be dynamically determined based on anonymous and / or partially anonymous user data and / or client-specific data. For example, anonymous and / or partially anonymous user data may indicate—for each of various user subgroups (as defined based on one or more user attributes)—how effective the transmission of an alert sent via each of three types of communication channels is for triggering a user to initiate a session on a corresponding website (e.g., as determined based on the tracking link within the alert) and complete a transaction during the session. Anonymous and / or partially anonymous user data may correspond to many different websites with one or more specific characteristics. Client-specific data may include data tracked by a given client for a specific website of interest, and may include data specifically identifying each user to whom various alerts and results are sent. Therefore, client-specific data may be richer in some respects relative to anonymous and / or partially anonymous data, but the number of users represented in the client-specific data may be less than the number of users represented in the anonymous and / or partially anonymous data. In addition, client-specific data may lack relevant attribute combinations. For example, a given client may not have previously used application-based alerts, which may reduce the accuracy with which a machine learning model can predict the potential impact of such alerts.

[0064] The machine learning model (configured with the first parameter) can use profile data associated with the track to determine a communication channel for the user. The profile data can include profile data collected by the client (e.g., using metadata, cookies, and / or input associated with previous HTML requests from a user device associated with the track). The profile data can also include other profile data requested and received from a remote user profile data repository, which can collect and manage profile data from multiple web hosts, clients, etc.

[0065] After the communication channel is identified, the trace is extended to the corresponding action node (425a, 425b or 425c). The alert is then sent using the corresponding communication channel. The alert can be configured to automatically identify limited content after detecting a request to open the alert and extend the trace to the second branch node 415. A decision can then be made at the second branch node 415 to determine the specific content to be presented in the body of the alert.

[0066] Thus, the second branch node 415 is connected to a first notification content node 430a representing content identifying products that the user has recently viewed on the website, a second notification content node 430b representing content identifying four of the products that were viewed most on the website (across users) in the past week, and a third notification content node 430c representing content including a discount identifier. The second decision can be made using a machine learning model configured based on one or more second parameters. Thus, in some (but not all) cases, the general type of machine learning model used to make decisions at various branch nodes can be the same, but the specific configuration (e.g., indicating weights to be assigned to various user attributes that should be considered at all and / or target results) can be different.

[0067] One or more second parameters may be dynamically determined based on anonymous and / or partially anonymous user data and / or client-specific data. However, each of the anonymous and / or partially anonymous user data and / or client-specific data may have changed since the first decision was made, which may result in differences between the first and second parameters. In addition, the potential actions considered at the second branch node 415 are different from the actions considered at the first branch node 410. Therefore, the first and second configurations may be different. In addition, the processed profile data may be different between the first and second branch nodes. For example, an application associated with the client may have been installed on the user device between the processing performed at the first and second branch nodes (e.g., so that application-based notifications are an option at the second branch node but not an option at the first branch node).

[0068] After the content is identified, the trace is extended to the corresponding content node (430a, 430b or 430c). The corresponding content is then sent to the user device so that it can be presented at the user device.

[0069] The content may include one or more tracking links to a web page at the website. After detecting that the tracking link has been activated, the track may extend to a third branch node 420. A decision may then be made at the third branch node 420 to determine the specific content to be presented on the requested web page.

[0070] Thus, third branch node 420 is connected to a first web page content node 435a, which represents content identifying four representative products—each representative product is associated with a different category; a second web page content node 435b, which represents content identifying four representative products—each representative product is associated with the same category; and a third web page content node 435c, which represents content identifying a single product that is predicted to be of interest to a given user based on previous web page interaction data.

[0071] A third decision may be made using a machine learning model configured based on one or more third parameters. The (one or more) third parameters may be different from the (one or more) first parameters and / or (one or more) second parameters as a result of temporary changes to anonymous and / or partially anonymous user data, client-specific data, and / or as a result of differences in potential actions. In addition, the profile data processed at the third branch node 420 may be different from the profile data processed at the first branch node 410 and / or the second branch node 415 (e.g., as a result of detecting new metadata in communications from the user device and / or receiving information corresponding to the profile from a remote system).

[0072] After identifying the content, the trace is extended to the corresponding content node (435a, 435b or 435c). The corresponding content is then sent to the user device so that it can be presented within the corresponding web page at the user device.

[0073] It will be recognized that although Figure 4 The communication decision tree 400 depicted in shows that a single decision is made at each communication stage (when a notification is to be sent, when the body of the notification is to be presented, and when a web page is to be presented), but a machine learning model may instead be used to make multiple decisions. For example, at branch node 410, a decision may be made as to what communication channel to use and when to send the notification (e.g., by identifying a time within a time period or a time in a set of potential times). As another example, a separate decision may be made before or after the communication channel decision to identify the transmission time. Thus, a machine learning model may be configured to generate multiple outputs, or multiple machine learning models may have multiple configurations (each configuration corresponding to different parameters and / or hyperparameters, each parameter and / or hyperparameter being trained separately and / or each producing a separate type of output).

[0074] Figure 5 An example of a trajectory 500 corresponding to a user device and extending through a communication decision tree 400 is illustrated. In this case, the machine learning result made at the first branch node 410 indicates that the email communication channel will be used to send the notification, causing the trajectory 500 to extend to the first action node 425a. The email notification is then sent to the user device. A request for email content is detected, indicating that the user is attempting to view the email, causing the trajectory 500 to extend to the second branch node 415. There, a decision is made to include content containing a discount logo in the email. Therefore, the trajectory 500 extends to the third notification content node 430c, and the corresponding content is sent to the user device.

[0075] A request for a web page corresponding to the target link within the email is then detected, causing the trace 500 to extend to the third branch node 420. A machine learning result is generated indicating that the web page will include content identifying four representative products, each associated with a different category. Therefore, the trace 500 extends to the first email content node 435a, where the corresponding web page content is sent to the user device.

[0076] In the depicted example, decisions at the first branch node, the second branch node, and the third branch node are indicated as being made at 5 p.m. on the first day, 12 p.m. on the second day, and 6 p.m. on the second day. The corresponding actions are then performed immediately. It will be appreciated that the action times may also be determined based on machine learning model execution, client rules, or other techniques.

[0077] It should also be appreciated that identifying a machine learning based decision may include implementing one or more additional constraints and / or factors. Alternatively or additionally, the machine learning based decision may be further modified based on one or more additional constraints and / or factors. For example, U.S. Application No. 14 / 798,293, filed on July 13, 2015 (which is incorporated herein by reference in its entirety for all purposes) further details additional techniques for dynamically identifying communication characteristics, which may also be combined with the machine learning techniques disclosed herein.

[0078] In some embodiments, a system and method for utilizing a canvas to facilitate configuration of a machine learning implementation sequence to partially define a communication exchange is provided. More specifically, a canvas is utilized that accepts the positioning and connection of an individual switch visual element to a corresponding set of communication visual elements. A communication decision tree can then be generated based on the set of positioned and connected visual elements. The canvas can be configured to accept the identification of one or more communication specifications associated with the communication visual element for each communication visual element. Each switch visual element can represent a machine learning technique (associated with specific parameters learned through training) for selecting a specific communication visual element from a set of communication visual elements connected to the switch visual element. The canvas can be configured to accept (e.g., for each switch visual element or in general) an identification of a target result (e.g., representing an event or communication initiated by a user), which can guide the machine learning selection. Therefore, a specific communication visual element selected using a machine learning technique can correspond to a communication specification (e.g., which can be represented as an event visual element in a communication decision tree) that is predicted to be relatively likely to lead to a target result.

[0079] A machine learning model may be defined for each represented switch visual element. The machine learning model may be trained using previous trajectories associated with other communication decision trees (e.g., but utilizing other communication decision trees having communication visual elements that correspond to communication specifications that are similar or identical to the communication specifications represented by the communication visual elements in the model being trained). For example, the model may be trained by determining—for trajectories that are routed so as to trigger communications having a particular communication specification—what subsequent user-initiated events those trajectories represent (e.g., and what portion of the trajectory represents the occurrence of a target outcome identified by the client). The model may also or alternatively be trained using trajectories because their occurrence is associated with the generated communication decision trees.

[0080] In some cases, a model may be trained using a data set reflecting previous events (e.g., through the trajectory of the same or different communication decision tree and / or other indications of a sequence of events), and augmented with new data. The new data may have been recently available (e.g., via newly received form input or metadata detection), but may correspond to variable types that are estimated to be static or predictably changing. For example, if the user's age is identified at time X, then the user's age at time X-3 years may be calculated, while the accuracy of retrospective estimates of interest or location variables over extended time periods may be less reliable. Training may then determine whether the various attributes represented in the new data are predictive of whether a particular event will occur.

[0081] The interface can be configured to accept instructions about biases applied at various machine learning stages. For example, for a given switch element connected to a particular set of communication visual elements, a client can interact with a slider control visually associated with a first visual element to indicate that path selection will be promoted (or weakened) toward the first visual element. Metadata fed into the machine learning model can be set based on the interaction to enable corresponding biases. In some cases, metadata may correspond to unlearned hyperparameters, which are then used to adjust or constrain learned parameters (e.g., weights). In some cases, metadata is used to define post-processing adjustments to be applied to the results generated by the machine learning model. In some cases, when the training data corresponding to the communication specification represented by the element is relatively low (e.g., in general and / or associated with a given communication stage), the client or system implements a bias toward a given communication visual element.

[0082] In some cases, the interface may enable the client to define the structure of the communication decision tree and / or - for each decision node - one or more hyperparameters of the machine learning model to be executed at the node. It should be noted that the machine learning model can be defined based on one or more hyperparameters and one or more parameters. Each of the one or more hyperparameters includes variables that are not learned via training of the machine learning model, and the one or more parameters include one or more variables learned via training of the machine learning model. Thus, the interface can be configured to allow the client to specify hyperparameters that indicate (for example) the number of branch nodes, the actions corresponding to each branch connected to each branch node, other inter-node connections, one or more constraints to be observed during execution of an individual machine learning model, and the like.

[0083] Figure 6 An exemplary interface 600 for defining a communication decision tree is illustrated. Specifically, the interface includes a canvas 605 on which representations of various nodes can be positioned and connected. The interface 600 can include a collection of icons that can be selected and positioned on the canvas 605 to represent specific sequential operations. The collection of icons can include a start icon 610 that represents the start of the communication decision tree. The start icon 610 can be associated with a configuration logic that can receive a definition of a condition that, when satisfied, indicates that a trajectory through the communication decision tree is to be initiated.

[0084] The set of icons may also include an end icon 615. A communication decision tree may be defined to indicate that a given trajectory is complete upon reaching the end icon 615. The client may then connect action definition icons and / or event detection icons between the located start icon 610 and the located end icon 615 to represent various operations and evaluations to be performed during trajectory observation.

[0085] The action definition icon included in the icon set can be a switch icon 620. The switch icon 620 corresponds to a branch node, at which a branch is selected or "switched to". The selection can be made using a configured machine learning model and profile data. In many cases, the switch icon 620 is connected to multiple potential paths. The potential path can intersect with another icon (e.g., a communication icon, an event detection icon, other switch icons, and / or an end icon).

[0086] Exemplary communication icons include an email icon 625 indicating that an email is to be sent to the user device, a text message icon 630 indicating that a text or SMS message is to be sent to the user device, and an app message icon 635 indicating that an alert is to be indicated via an app installed on the user device. In some cases, the potential paths indicate that no action is taken (via the absence of a communication icon). In the depicted canvas, the positioned switch icon is connected to three paths: two email paths (e.g., associated with different content and / or transmission times) and a no-action path.

[0087] The event detection icons included in the icon set may include a target detection icon 637, which indicates that an event corresponding to a target result for one or more machine learning techniques has been detected. The target detection icon 637 and / or another event detection icon may indicate, for example, that a notification has been opened, a target link included in the notification has been activated, a user device associated with the track has initiated a session with the website, a product (e.g., any product or a specific product) has been purchased on the website, additional profile information corresponding to the track has been provided, and the like.

[0088] Interface 600 may include a connection tool 640 that can be used to connect multiple icons in a directed manner. Each connection may indicate that the communication decision tree is configured to allow the track to extend across the connected nodes in the indicated direction. However, each connection may be associated with a condition so that the track extends across the connection only when the condition is met. For example, the connection may be configured so that the condition is met when a branch node (connected at one end of the connection) makes a determination to perform an action represented by a communication icon (connected at the other end of the connection). As another example, the condition may be configured to be met after a specific type of interaction associated with a user device associated with the track is detected.

[0089] Each action definition icon can be associated with one or more action parameters that define the detailed action to be performed when the track reaches the icon. For example, the parameter definition interface can be presented as part of interface 600 after clicking the icon and / or the parameter definition interface can be opened in a pop-up window when right-clicking and / or double-clicking the icon.

[0090] In some cases, each action definition icon and / or event detection icon corresponds to a widget or code segment that can be executed independently. Canvas 605 can be used as a communication structure so that a result produced by a widget (e.g., an indication from a machine learning model corresponding to a switch icon that a communication is to be sent according to a particular communication specification) can be used for another widget (e.g., a widget corresponding to a communication icon corresponding to a particular communication specification). Thus, canvas 605 can extend the track and orchestrate various aspects of a communication exchange in response to widget results.

[0091] Although not in Figure 6 605, but it should be appreciated that in some cases, multiple switch icons 620 can be positioned on canvas 605. Each switch icon 620 can correspond to a separate instance of a machine learning model that can be configured and operated separately.

[0092] Figure 7 An exemplary parameter definition interface 700 for a switch icon is illustrated. The parameter definition interface 700 includes a field for a stage label that accepts text input. The text input can then be displayed next to the associated icon in the interface for defining a communication decision tree. A description can also be entered via the text input, which can be displayed, for example, in the interface for defining a communication decision tree in response to detecting a single click or double click associated with the icon.

[0093] For switch icons configured to identify selections or action specifications and / or configured to implement machine learning models, the parameter definition interface 700 may include fields for defining target results. For example, a drop-down menu may identify a set of events that are being tracked and can be used to identify a target result. The target result may include an action initiated at a user device, a notification initiated by the system, etc. For example, the target result may include detecting that a link within a communication utilized by a user device is clicked, a communication utilized by a user device is opened, a communication results in a purchase associated with the user device (i.e., a conversion), a chat session is initiated, a form is completed, etc.

[0094] For switch icons configured to identify selections or action specifications and / or configured to implement machine learning models, the parameter definition interface 700 may also include one or more fields indicating potential results to be identified. For example, the interface 700 includes fields corresponding to three paths or branches extending from the icon. In this case, the stage label name of another action definition icon is identified for each path. In some cases, after detecting that the switch is connected to one or more other icons at the interface to define a communication decision tree, the path information is automatically updated at the parameter definition interface 700. It should also be recognized that the parameter definition interface 700 may include options for adding additional paths, removing paths, etc.

[0095] In some cases, one of the paths may be identified as a default path. The trajectory may then generally be routed to the default path unless, for example, the machine learning model predicts that another path will have at least a threshold degree of greater probability of leading to the target outcome, traversing the other path will produce additional data for the path with a threshold value (e.g., as indicated by a predicted improvement in confidence of subsequent predictions), etc. In some cases, whether the default path is selected depends on the confidence associated with the probability of the target outcome occurring (e.g., unless another path is predicted to have at least a 60% probability of leading to the target outcome and the probability has at least a 50% confidence).

[0096] In some cases, the switch icon can be configured to select a path and / or a next action (or lack thereof) and a time to extend the path to the next icon (e.g., and perform any next action). The time can be selected from a plurality of times and / or along an open or closed continuum. In the depicted example, the parameter definition interface 700 includes a maximum time at which the trajectory is extended to the next action definition icon. Thus, here, the trajectory will be extended no later than one day after the trajectory reaches the switch icon, unless the decision logic executed in association with the switch icon indicates that another time period is sufficiently more favorable (e.g., due to a higher probability of resulting in a target outcome and / or increased training data).

[0097] Machine learning techniques and / or other selection techniques can be configured to identify the path associated with the highest probability of leading to the target result from among multiple potential paths. In some cases, the technology further introduces a certain degree of noise and / or variability, so that a path that is considered suboptimal is occasionally selected to facilitate continued training of the underlying model.

[0098] In some cases, a client may have reasons to introduce a bias toward or against the selection of a particular path. For example, a particular path may be expensive (e.g., computationally and / or economically) to use relative to another path. As another example, a particular path may have high availability relative to another path. As yet another example, a client may desire to quickly obtain information about the efficacy of a given path in order to inform subsequent resource allocation decisions.

[0099] Therefore, the parameter definition interface 700 may include one or more options to achieve a bias toward or against an individual path. In the depicted example, a slider is provided for each path. When the slider is positioned toward the right "promote" side, the path selection technology can be adjusted to bias toward the corresponding path. When the slider is positioned toward the left "constraint" side, the path selection technology can be adjusted to oppose the corresponding path bias. This promotion and / or constraint option can have imposed restrictions so that (for example) the effect of moving the slider to the leftmost constraint position is not to prevent the corresponding path from being selected. Such restrictions can allow the machine learning model to continue to collect data related to various options and continue to modify one or more parameters through learning. When there are only two options, a single interface component can be provided to identify the relative bias toward one option relative to another option. At the same time, when there are more than two options, option-specific promotion / constraint options can provide more intuitive control.

[0100] Figure 8 Another parameter definition interface 800 is shown, which includes options for implementing biases toward or against various content in the representation communication. In the depicted example, nine content items are represented (each content item represents a corresponding product). A slider is provided in visual association with the representation of each content item. When the slider is positioned toward the right "promote" side, the content selection (e.g., it can correspond to the selection between multiple paths representing different content or can correspond to selecting content after identifying a communication channel) can be adjusted to bias toward the corresponding content item. When the slider is positioned toward the left "constraint" side, the path selection technique can be adjusted to oppose the corresponding item bias.

[0101] In the depicted example, the slider is positioned in the leftmost position. This triggers the "never serve" option. In some cases, if the "never serve" option is not selected, the first content item may still be selected at least occasionally.

[0102] Based on the relative deviations indicated by the sliders and the historical communication counts, the system can predict the number of times that an individual content item will be represented on a given day. Thus, as the client moves one or more sliders, interface 800 can automatically update an estimated count of the number of times an individual content item will be presented (e.g., per day) given the slider positions.

[0103] It will be appreciated that different types of biases may also be identified and implemented. For example, one or more sliders may be provided to indicate a bias related to the time at which a communication is sent. For example, a slider may indicate the degree to which a decision is biased toward immediate transmission (and / or toward another time, such as at an upper limit).

[0104] Implementing a bias (e.g., toward or against one type of communication channel, toward or against representing a particular type of content, toward or against sending communications at a particular time, etc.) may include, for example, modifying one or more weights and / or one or more thresholds in a machine learning model. In some cases, implementing a bias includes performing post-processing (e.g., redistributing a portion of the results to improve the degree to which the distribution of communication attributes matches a target distribution indicated based on the bias(es).

[0105] Fig. 9 Yet another parameter definition interface 900 is shown, which includes options for biasing toward or against using various communication channels to send communications. In the depicted example, three communication channels are represented: email, app-based notifications, and SMS messages. A slider is provided in visual association with the representation of each channel. When the slider is positioned toward the right "promote" side, the content transmission can be adjusted to bias toward using the corresponding type of channel. When the slider is positioned toward the left "constrain" side, the path selection technology can be adjusted to bias against the corresponding channel.

[0106] Interface 900 also shows a time series representation that indicates the number of communications that have been sent using each channel in a recent time period, and also indicates the number of communications scheduled for sending using each channel across an upcoming time period. The current time is represented by a vertical line. Communications can be scheduled based on a selection technique that selects between multiple potential transmission times (e.g., which can be included in the same or different machine learning model relative to the technique for selecting a communication channel). Thus, a client can view the load of scheduling across various channels and determine whether to adjust any bias set for or against a channel.

[0107] Fig.10A flow chart of a process 1000 for using machine learning to guide a trajectory through a communication decision tree according to some embodiments of the present invention is shown. Process 1000 begins at box 1005, where a data structure representing a communication decision tree is accessed. The communication decision tree can be configured to dynamically define each trajectory through the communication decision tree using machine learning techniques to indicate a series of communication specifications. More specifically, the communication decision tree can include a collection of nodes. In response to detecting an event indicating that such an extension will occur, a given trajectory can be extended across nodes. The event can include (for example) detecting a specific type of action or communication from a user event, or can include identifying a specific decision (corresponding to a node identification) at a trajectory management system or a machine learning data platform. The collection of nodes can include a collection of branch nodes. Each branch node in the collection of branch nodes can correspond to an action point, which is configured to identify the direction of a given trajectory and / or identify a specific action to be initiated at a trajectory management system or a machine learning data platform. The branch node can be configured to use a configured machine learning model to identify a direction or action.

[0108] At box 1010, it is detected that the trajectory (associated with a specific user and / or a specific user device) has been extended to a branch node that reaches a communication decision tree. The specific user can be a user in a set of users included in a target group or target audience group of the communication recipient (e.g., each user is associated with one or more predefined attributes identified by the client). The target group or target audience of the communication recipient does not need to be (although it may be) statically defined. For example, it can represent a dynamic set corresponding to a profile representing each of one or more predefined attributes at various points in time. As a result of detecting a specific type of event initiated at a user device (e.g., a communication indicating that the user device is participating in a session at a website associated with the client, a communication indicating that the user has completed a profile form submission, etc.) and / or as a result of completing an action initiated by a specific system, the trajectory can have been extended to a branch node.

[0109] At block 1015, the learned data generated by processing other user data is retrieved. The other user data may correspond to data associated with at least a portion of a target group of target audiences and / or communication recipients. The learned data may include data generated when training machine learning techniques. It will be appreciated that training may occur during a separate time relative to the use of machine learning techniques to guide one or more trajectories, or the training and utilization of machine learning techniques may be performed concurrently. The other user data may include trajectory data associated with one or more trajectories through the same or different communication decision trees. For example, the other user data may indicate which of at least a portion of the target group of communication recipients the corresponding trajectory reaches a target node in the communication decision tree specified in the predefined trajectory target (e.g., indicating the success of the workflow). The target node may represent, for example, interaction with content, conversion, or response to communication. As another alternative or additional example, the other user data may indicate which of at least a portion of the target group the corresponding trajectory reaches a pre-identified node representing an undesirable result (e.g., lack of response to communication, lack of conversion, or lack of interaction with content). Other user data may indicate profile data and / or attributes corresponding to one or more users, and may also indicate various events detected and / or initiated in association with various tracks. Thus, for example, other user data may indicate the probability of detecting a particular type of event (e.g., identified by the client as a target outcome) when various scenarios exist.

[0110] At block 1020, one or more user attributes associated with the user corresponding to the trace (detected as extending to the branch node) are retrieved. The user attribute(s) may include, for example, the type of user device; the geographic location of the user device; the type of browser used at the user device; the operating system used at the user device; a partial or complete history of interactions between the user device and a particular website; interactions between the user device and one or more other websites; cookie data associated with the user device; historical data indicating the type of notifications (e.g., types of emails, text messages, and / or app messages) opened at the user device that resulted in activation of included links, etc. The one or more specific user attributes may be collected and / or retrieved locally and / or requested and received from a remote source.

[0111] At box 1025, one or more communication specifications are identified based on the learned data and one or more user attributes. For example, the learned data may include one or more parameters of a machine learning model (e.g., a regression model). The machine learning model may also be defined based on one or more hyperparameters. The machine learning model may then be configured to process (one or more) user attributes using (one or more) parameters, (one or more) hyperparameters, and / or underlying structures. The results of the implementation of the model may identify the most successful choice predicted to achieve the target result from a plurality of available options. A plurality of available options may correspond to (for example) different types of communication channels to be used, different types of content to be sent, and / or different transmission timings. In some cases, a plurality of available options share one or more other communication specifications.

[0112] At block 1030, transmission of content to a user device associated with the track is triggered. The content transmission is performed according to one or more communication specifications.

[0113] At block 1035, determine whether the trace has been extended to reach another branch node within the communication decision tree. The determination may include determining, for example, whether a threshold amount of time has passed since the last communication was sent to a particular user (or corresponding device); the particular user interacted with the last communication sent to the particular user (or corresponding device) and / or the particular user interacted with the target content, regardless of whether such interaction with the target content was a result of the last communication sent to the particular user (or corresponding device). In some cases, each of two or more of these determinations is associated with criteria for a different branch node. Block 1035 may include identifying which other branch node the trace has been extended to.

[0114] If it is determined that the trajectory has been extended to reach another branch node, then process 1000 returns to box 1015 and repeats boxes 1015-1035. However, repeated iterations of box 1015 may include retrieving different learned data generated by processing other user data (e.g., possibly but not necessarily combined with at least some user data). Different learned data (e.g., having the same or different values ​​and / or the same or different types of parameters and / or hyperparameters) may have been generated using the same or different configurations of machine learning techniques. Repeated iterations of box 1020 may include retrieving at least one other user attribute. Repeated iterations of box 1025 may include identifying at least one other communication specification (and / or from a different set of potential communication specifications) based on different learned data and at least one other user attribute. The same or different types of underlying models may be used to identify at least one other communication specification. And repeated iterations of box 1030 may include triggering another transmission of other content based on at least one other communication specification.

[0115] When it is determined that the track has not been extended to reach another branch node, the process 1000 proceeds to block 1040 to determine whether the track is complete. The determination can be made by determining whether the current end of the track is associated with a track that lacks an extended connection. If it is determined that the track is complete, then processing of the track can be terminated. If it is determined that the track is not complete, then the process 1000 can return to block 1035 to wait for a determination that the track has reached another branch node (e.g., as a result of a user-initiated action or external event).

[0116] Thus, process 1000 facilitates repeated use of differently configured machine learning models to identify specifications corresponding to different stages in a communication exchange. At different stages, the model can use different profile data (e.g., values ​​of different fields or values ​​that have changed over time) and / or different model parameters (e.g., learned based on different inputs and / or outputs associated with the model and / or learned based on temporal changes). This iterative application of the machine learning model facilitates dynamically directing the communication exchange of individual users.

[0117] Fig.11 The flow chart of the process 1100 for defining a communication decision tree based on machine learning using an interface that supports positionable visual elements is shown.Process 1100 starts at box 1105, and it is useful that the interface including a visual element set and a canvas for element positioning is useful.Each visual element set in the visual element set can be positionable on the canvas.For example, the interface can be configured to allow the user to click on the representation of the visual element, and--while maintaining the click--the cursor is dragged to another position on the canvas to drop the visual element on this another position.As another example, it is possible to select to represent (for example, via clicking or double-clicking), and another input (for example, another click or double-clicking) received when the cursor is located at another position can make the visual element be positioned at this another position.

[0118] The visual element set may include an action definition visual element set. Each action definition visual element of the action definition visual element set may be a specific action to be performed when a given trajectory has extended to the action definition visual element. The action definition visual element set may include a switch visual element, which represents a decision action (e.g., made using a machine learning model) that uses machine learning techniques to identify communication specifications. The action definition visual element set may also include a communication visual element set. Each communication visual element in the communication visual element set may represent a specific communication specification (e.g., the type of communication channel, specific content, transmission time, etc.). The visual element set may also include a connection visual element, which is configured to connect multiple positioned visual elements in a directional manner. Each positioned visual element in the multiple positioned visual elements may correspond to the action definition visual element in the action definition visual element set. The directional connection may indicate the order in which the specific actions represented by the multiple positioned visual elements will occur.

[0119] At block 1110, an update to the canvas is detected. The updated canvas may include a switch visual element at a first position within the canvas, a first communication visual element of a set of communication visual elements at a second position within the canvas, and a second communication visual element of the set of communication visual elements at a third position within the canvas. The first communication visual element may represent a first specific communication specification, and the second communication visual element may represent a second specific communication specification.

[0120] The updated canvas may also include a set of connected visual elements. Each connected visual element in the set of connected visual elements may include an instance of the connected visual element. A first connection in the set of connected visual elements may be positioned to connect the switch visual element to the first communication visual element. A second connection in the set of connected visual elements may be positioned to connect the switch visual element to the second communication visual element. The set of connected visual elements may indicate that potential results of performing the machine learning technique at the switch visual element include a first result of triggering a communication transmission having a first specific communication specification and a second result of triggering a communication transmission having a second specific communication specification.

[0121] At box 1115, a specific communication decision tree is defined based on the updated canvas. At box 1120, it is detected that a given trajectory associated with the specific profile data has extended to a specific decision action represented by the switch visual element. In response to the detection, at box 1125, machine learning techniques (configured with learned parameter data and / or static data) are used to process the specific profile data to produce a machine learning result. The learned parameter data may include data learned during separate or ongoing training of the machine learning model based on a set of trajectories associated with other users and / or associated with the same or different communication decision trees. Processing the specific profile data using machine learning techniques can indicate which of the first and second specific communication specifications will be applied to the content transmission.

[0122] Thus, at block 1130, the content is sent to the user device associated with the track. The transmission is performed according to one of the first and second specific communication specifications as indicated in the machine learning results. For example, the first and second communication visual elements may correspond to different types of communication channels. Block 1125 may then include identifying one of the two types of communication channels, and the content may be sent via the identified channel.

[0123] Thus, the canvas facilitates the configuration of defining a communication decision tree. However, the client need not define communication exchanges that apply to all users and / or that only include one or more deterministic rules. Rather, the interface generally supports options for identifying various communication specifications, the order of communication events, and / or constraints. Machine learning techniques can then be used to automatically and dynamically generate canonical communication specifications. This approach can facilitate configuring the communication system to adhere to client priorities, but can allow the communication system to dynamically adapt to the characteristics of a particular user, resource load, recent interaction patterns, etc.

[0124] It will be appreciated that variations of the disclosed technology are contemplated. For example, a branch node may use another type of artificial intelligence model that is not a machine learning model to select a communication specification to be used for communication. As another example, an interface may be configured to accept a selection of a specific type or a more general type of artificial intelligence model to be used for a trajectory phase corresponding to a switch element. As yet another example, an interface may be configured to allow an indication of what data (e.g., in terms of corresponding to one or more communication decision trees, one or more time periods, and / or one or more user population segments) will be used to train a machine learning model corresponding to one, more than one, or all switch elements located on a canvas.

[0125] It will be appreciated that the technology disclosed herein can be used to support various types of decision trees. For example, the nodes in the tree and / or the visual elements represented on the canvas can (in some cases) correspond to the elements generally associated with the logic of evaluating whether a given condition is met (e.g., detecting a specific type of inter-device communication, a non-client associated application indicating that an action has been performed, a specific time has passed), and, after detecting that it is met, a specific action is performed. For a subset of nodes and / or visual elements, conditional specific actions can include executing a machine learning model based on profile data to select from a connected set of nodes (or visual elements) to continue so that another specific action associated with the selected node (or visual element) can be performed. For example, trajectory path selection based on machine learning can be integrated into an If ThisThen That environment. Branches can (for example) identify different applications for storing data, rather than connecting branch nodes to nodes that identify communication specifications. Therefore, a decision framework can be established to enable artificial intelligence applets and / or plug-ins to communicate with one or more other applets or return through a canvas.

[0126] In some cases, the technique may involve identifying a worker to whom a request to perform a specific job is to be sent. The technique may also or alternatively include, for example, identifying a worker to whom a request to participate in an inquiry process is to be made and / or one or more configurations for the request (e.g., to perform a specific job and / or to participate in an inquiry process).

[0127] In some cases, identification may be performed during a period corresponding to an iteration through a decision tree. A decision tree need not include finite and / or consistent potential decisions at each decision node. For example, a decision tree may include a progressive filtering decision tree so that a set of workers is iteratively reduced at each of two or more decision nodes. The set of potential decisions may correspond to all potentially reduced subsets of workers corresponding to a given decision. A particular decision node may, but need not, constrain or define the number of workers to be included in the reduced subset of workers (e.g., indicating that the set will be reduced to a subset including 5 workers, will be reduced by 80%, or will be reduced to between 2 and 9 workers). The final output of the decision tree may include, for example, the identification of a single worker, a defined number of workers, and / or may not be numerically limited.

[0128] The decision tree may include one or more chance nodes that may reflect, for example, responses received from one or more workers. For example, after executing a decision node, a request may be sent to a worker, and the response (or lack of response) may indicate whether the worker accepted the request. In some cases, if a worker rejects and / or does not accept the request (e.g., within a predetermined duration), the worker is filtered out of the current pool of workers being considered.

[0129] In some cases, the decision tree represents a decision corresponding to a job offer (e.g., accepting an offer corresponding to a specific position and / or occupation and / or for employment with a specific employer). A job offer may involve one or more documents, files, or data sets that describe a job, an ideal candidate, a decision timeline, compensation, location, etc. Each worker may include an identifier corresponding to a person who submits a resume and / or application to apply for one or more specific jobs, one or more specified types of jobs, and / or general jobs. One or more first decision nodes may be related to decisions involving which (or which) workers are invited to an inquiry process (e.g., an in-person, telephone, or webex interview) with an employer, recruiter, etc. One or more second decision nodes may be related to decisions involving which (or which) workers are provided with a job offer (e.g., a request to perform work). One or more third decision nodes may be related to requests for configurations (e.g., compensation, vacation, location, and / or other job offer details) that involve identifying the execution of work. As with respect to a particular example, a first decision node may be configured to output a ranking across a set of workers that corresponds to a probability of being offered an interview for a job, and a second decision node may be configured to output a ranking across a set of workers (or a subset thereof) that corresponds to a probability of being offered a job. As another particular example, a first decision node may be configured to generate an output corresponding to the likelihood that a worker will accept an offer for a job (e.g., relative to not accepting an offer regardless of any possible offer configurations), and a second decision node may be configured to generate an output corresponding to what package will be needed (e.g., defining salary, bonus, and / or compensation details) to attract the worker. The decision node outputs may be presented or sent concurrently or simultaneously or at separate times.

[0130] It will be appreciated that decisions may, but need not, be made automatically at each decision node regarding the progression of a given trajectory. For example, rather than filtering a set of workers (or a subset thereof) to a reduced set, an automated process (e.g., relying on execution of a machine learning model) may output ratings of the workers and / or a score for each worker. The output(s) may be presented and / or sent (e.g., to a user device) to facilitate a decision by a user indicating the filtering and / or selection that is actually to occur. The filtering and / or selection may be communicated (e.g., via user input provided at a portal interface) to further adjust the decision tree trajectory.

[0131] Filtering, ranking, and / or scoring can be based on one or more optimization variables, which can be used consistently across multiple or all decisions in a decision tree, or can be defined specifically for each decision node (e.g., and different across some or all decision nodes). The optimization variables can relate to decisions made by workers (e.g., the probability that a worker will accept a request to participate in an inquiry process and / or the probability that a worker will accept a request to perform work), decisions made regarding subsequent requests (e.g., the probability that a request to participate in an inquiry process will be sent and / or the probability that a request to perform work will be sent), and / or performance (e.g., the probability that a worker will continue to perform work within a predefined time interval, the probability of achieving a high third-party assessment of the worker's performance, and / or the probability of receiving a high stock award or bonus based on work performance).

[0132] In some cases, the rating of workers is performed based on performance data (e.g., a filtered subset of an initial set of workers). For example, a machine learning model can be trained based on training data including worker attributes and performance data. Thus, attributes of each worker from a plurality of workers (e.g., represented in a filtered subset of an initial set of workers) and job data can be input into the model, and the output can indicate the rating of a plurality of workers based on a predicted performance evaluation of each worker in the plurality of workers (e.g., a predicted performance corresponding to the predicted performance evaluation if the worker is assigned to the job). The output(s) can include, for example, a numerical rating (e.g., assigning a number to each worker), a ranking (e.g., including an ordered identification of the workers), a numerical or categorical score for each worker (e.g., enabling the workers to be ranked based on the score), etc. Training the model can result in, for example, learning one or more weights that associate various worker attributes with performance predictions. In some cases, the model is trained so that performance predictions are constrained based on job specifications in addition to worker attributes.

[0133] The performance data used to train the model may include, for example, numerical, categorical, and / or textual data. In some cases, the performance data may include and / or may be processed to produce metrics along a scale. Performance data may be received from one or more work owners (e.g., employers or reviewers) at one or more times (e.g., at regular times (such as annually), in response to pull requests, or as push reports). The training data may include attributes associated with the worker at (or about) the time when the work (performance is being evaluated for the work) is assigned to the worker and / or the worker's current attributes (e.g., current in terms of the time when the performance evaluation is being performed). The attributes at a given time may be determined based on the normative data set available to the worker at that time. In some cases, the stored normative data set may be updated based on the passage of time to derive attributes for training. For example, if the normative data set includes a years of experience attribute and the attribute is initially stored or last updated at time t minus x, then this attribute may be updated to add x to the attribute (e.g., if it is assumed that the worker performs the relevant work during x time). In some cases, performance data is only included in the training data set if, for example, one or more associated worker attributes have been updated within a predefined time period.

[0134] Fig.12 An exemplary progressive filtering decision tree 1200 is shown according to some embodiments of the present invention. The decision tree may evaluate a set of workers associated with a particular job. In some cases, the evaluation includes evaluating a set of worker files (each worker file representing and / or corresponding to a particular worker) and / or job files (e.g., representing jobs and / or job offers). A worker file may represent and / or include a worker specification data structure (e.g., resume data). A job file may represent and / or include a job specification data structure (e.g., job posting data).

[0135] At decision node 1205, a determination is made as to which workers are included in the set of workers. The determination may be made based on a collection of files or data representations corresponding to the set of workers and an evaluation of files or data representations corresponding to the jobs. Specifically, each file or data representation (corresponding to a worker or job) may be processed and transformed into a feature data set. For example, the feature data set may include an array, matrix, or vector including a value at each element. The value may represent the degree to which the corresponding file or data representation matches, identifies, or otherwise corresponds to a given feature.

[0136] In some cases, each feature is defined as a specific space within the word space. As a simple example, a feature can represent a cluster of words (e.g., which can include synonyms and / or other corresponding words), and a weight can (but need not) be assigned to each word. As another example, a feature can represent a distribution. Words can be located across a multidimensional word space based on, for example, their semantic similarity to other words and / or the probability of their occurrence in documents, files, paragraphs, field values, and / or sentences identical to each word in one or more other words. Distribution can be defined for each dimension in one, multiple, or all dimensions, for example, by a center or centroid and a weight (and / or tilt).

[0137] A feature data set can be generated using data from a canonical data structure, which can include unstructured data (e.g., a word set of variable length in terms of character or word count). The data can be processed to identify different words that can be further processed. For example, the further processing can include converting each word into a stem (e.g., removing endings such as "ing", "ed", etc.). In addition, various words (e.g., stop words, articles, prepositions, conjunctions, etc.) can be removed from the word set.

[0138] The (e.g., processed) word set can be input to a machine learning model or other artificial intelligence model to identify a feature data set. As an example, a machine learning model including a neural network language model can be used to calculate the feature data set. In some cases, the relative ordering of words in the corresponding worker specification data structure affects the feature assignment. Exemplary types of machine learning models include those identified in Le, Quoc. "Distributed Representations of Sentences and Documents" Proceedings of the 31st International Conference on Machine Learning, Beijing, China, 2014. JMLR: W&CP Volume 32, which is available at the following URL: https: / / cs.stanford.edu / ~quocle / paragraph_vector.pdf, which is incorporated herein by reference in its entirety for all purposes. For example, doc2vec can be used to transform a text input including a word set into a feature data set.

[0139] As another example, the value of a feature can be calculated by, for example, generating a weight for each word in the processed word set based on the feature's specific space and averaging the weights. As yet another example, the value of a feature can be calculated by counting the number of occurrences of a phrase associated with any feature within the data from a canonical data structure (e.g., and normalizing the counts based on other counts or the total number of words in the data).

[0140] Individual feature data sets may be stored (e.g., for individual workers or individual jobs). Each feature data set may be stored in association with an identifier of the worker or job and / or other information (e.g., one or more key-value pairs from a corresponding canonical data structure, other information from a corresponding canonical data structure, raw data from a corresponding canonical data structure). In some cases, the feature data set(s) and associated information may be stored in a Solr database.

[0141] Determining which workers to include in the set of workers may include comparing a feature data set for each worker (e.g., included in the plurality of workers) to a feature data set for the job (e.g., and selecting a subset of the plurality of workers based on the comparison). The plurality of workers may include, for example, workers identified as being available to potentially accept the job (e.g., via a submission specification data structure). The plurality of workers may be dynamic in that workers may be dynamically removed from the plurality of workers (e.g., in response to a job having been assigned to the worker or a removal request being received) and / or workers may be dynamically added to the plurality of workers (e.g., in response to receiving a new worker specification data structure).

[0142] Performing a comparison between the feature data set of the worker and the feature data set of the job may include generating a comparison metric based on the feature data set. The metric may include, for example, an angle, a dot product, a correlation coefficient, etc. A subset of workers may be defined corresponding to each worker associated with a dot product or correlation coefficient (or other comparison metric) that exceeds an absolute threshold or a relative threshold. The relative threshold may be configured to identify a subset that includes a specific number or a specific percentage of workers.

[0143] At decision node 1210, the set of workers is ranked via iteration of the machine learning model, the feature data set, and the additional information. That is, the machine learning model may be executed to process inputs corresponding to and / or including the worker feature data set, the job feature data set, and the additional information. The additional information may include one or more key-value pairs from the worker specification data set (e.g., representing or indicating years of experience, education level, etc.). In some cases, the additional information (further or alternatively) includes one or more key-value pairs from the job specification data set.

[0144] Thus, in some cases, a machine learning model used in an early stage does not receive and / or process some of the worker data that is received and / or processed by a machine learning model in a later stage. For example, one or more key-value pairs for various workers may be received and / or processed by a later stage machine learning model but not an early stage machine learning model. The reverse may also be true. For example, an early stage machine may receive and / or process raw text, while a later stage machine learning model may receive and / or process a processed version of the raw text instead of the raw text itself.

[0145] Using different types of machine learning models, different types of inputs across different stages of decision tree iterations, and / or different optimization variables can provide technical advantages. In some cases, using different types of inputs promotes efficient processing. For example, using a processed version of the worker data instead of the original data in a later stage can take advantage of previous data processing and learning, which can be performed to emphasize or select useful (or potentially useful) information rather than repeating processing. In addition, because the early stage processing can (but does not need to) learn from a larger and / or more diverse dataset, the learned dimensionality reduction processing can be more robust relative to the later stage learning when relying on early stage learning.

[0146] In some cases, selectively using some data in later stages can provide a balance of leveraging rich worker datasets while also promoting efficient processing and resource consumption. For example, large groups of workers may be associated with tens of thousands or millions of key-value pairs. The volume of the space of potential worker attributes grows exponentially as each additional category of key-value pairs is considered. At the same time, the size of the potential worker pool can be reduced across stages so that the absolute impact of including one or more key-value pairs in later stages is less than the impact of including them in earlier stages. That is, in some cases, early stage calculations can be designed to be relatively simple compared to later stage calculations to increase the efficiency and practical likelihood of performing these steps by a computer.

[0147] A machine learning model may include a model configured to output ratings of various inputs (a reference job specification data structure given a job characteristics dataset). For example, a machine learning model may use learning to rate algorithms.

[0148] The machine learning model can be implemented to use supervised, semi-supervised, or reinforcement learning. As an example, the machine learning model can use LambdaRank (e.g., implemented using xgboost). The machine learning model can be trained using communications corresponding to representations of the degree to which various worker data (e.g., worker specification data structures) correspond to various job data (e.g., job specification data structures). It will be appreciated that other types of machine learning models that do not output ratings for individual workers can be used. For example, the machine learning model can be configured to generate an output that indicates, for each of some or all of a set of workers, a match score representing the degree of match between worker data (e.g., a worker feature data set and / or additional information associated with the worker) and job data (e.g., a job feature data set and / or additional information associated with the job).

[0149] In some cases, the output of the execution of the machine learning model can also be processed. For example, post-processing can include identifying a subset of workers associated with ratings below a predetermined rating threshold and / or identifying a subset of workers associated with a match score above a predetermined score threshold.

[0150] At decision node 1215, a subset of the set of workers is identified. This identification may include filtering the set of workers based on the ratings (or other results) from decision node 1210 to identify workers to whom to send requests to participate in the query process.

[0151] In some cases, identification is performed automatically. For example, a rule may indicate that a predetermined number of sets with the lowest (best) ratings will be included in the subset, or that each worker associated with a score exceeding a predefined threshold will be included in the subset.

[0152] In some cases, identification is performed based on the receipt of a communication corresponding to the identification of the worker to be included in the subset. The communication may have been generated based on and in response to an input received at a user device (e.g., a recruiter's device). For example, after a rating has been identified at node 1210, a communication indicating the rating associated with the corresponding worker may be generated. The communication may be configured so that an interface is updated and / or generated and also presented at the user device, wherein the interface includes some or all ratings. The interface may also include and / or utilize information about the rated worker. For example, the interface may include a link associated with each rating space to view a file or screen showing a specification data set for the worker. The interface may include one or more input options (e.g., (one or more) check boxes, (one or more) drop-down menus, (one or more) links, (one or more) drag-and-drop interfaces, (one or more) (one or more) radio buttons, etc.). Interactions with (one or more) options may be automatically detected and translated into identifications of specific workers to be included in the subset.

[0153] As an illustration, the interface can include a table with each row representing a different worker. One column can indicate the worker's rating, while other columns can include other information about the worker (e.g., field value pairs, worker identifiers, etc.). A splitting element (e.g., a thick line, which can also be movable) can be used to indicate the workers to be included in the subset (e.g., only those above the splitting element), and the interface can accept drag-and-drop element movements and reorder rows. As another illustration, the interface can include visual objects (e.g., small boxes) that identify worker-specific information and the worker's rating. Each visual object can include or can be spatially associated with a check box. A selected check box can indicate that the worker will be included in the subset.

[0154] In some cases, a request to participate in the inquiry process is automatically sent (e.g., emailed or indicated in the worker's portal to the work assignment system) to each worker in the subset. In some cases, the identities of the workers in the subset are displayed or sent (e.g., along with key information identifying contact information for the workers in the subset, such as an email address or phone number) to facilitate external communication of the request to participate in the inquiry process. The request may include, for example, work information (e.g., job title and employer), a location associated with the inquiry process (e.g., a street address), and a time (or time options) for the inquiry process.

[0155] In some cases, responses to requests are tracked. For example, when a request is sent as an email or via a worker portal, one or more response options can be provided to indicate whether the worker accepts the request and / or whether the worker rejects the request. As another example, a communication can be received from a user indicating a response associated with each worker in a subset. In some cases, determining that no response has been received from a given worker within a predefined time period (e.g., starting at the time the request was sent) can result in generating a virtual response that the request was rejected. The subset can also be filtered to remove identification of the worker(s) that rejected the request.

[0156] In some cases, decision node 1210 is not included in decision tree 1200. Identification of workers to be included in a subset may be performed using iterations of a machine learning model. The machine learning model may process a worker specification data structure (e.g., corresponding to a set of workers), a job specification data structure, and additional information (e.g., key-value pairs associated with a set of workers and / or key-value pairs associated with a job). The machine learning model may be configured, for example, to output a rating, score, or binary identifier (corresponding to whether or not to be included in a subset).

[0157] At decision node 1220, one or more work configurations are identified for the work request, and at decision node 1225, a decision is made as to which worker(s) to send the request to perform the work. In various cases, the decisions made at nodes 1220 and 1225 are made concurrently; before the decision is made at node 1225, the decision is made at node 1220; or after the decision is made at node 1225. In some cases, one or both of the decisions made at blocks 1220 and 1225 are made using a machine learning model.

[0158] With respect to block 1220, the machine learning model is configured to output, for each of one, some, or all of the subset of workers, one or more job configurations that are predicted to be sufficient (and / or minimal) to cause the worker to accept the job (with that configuration(s)). The model may also be configured to output, when appropriate, a result indicating that the worker is predicted not to accept the job regardless of the configuration(s).

[0159] With respect to block 1225, the (same or different) machine learning model is configured to generate an output related to the selection of one or more workers to request to perform the work. The output may include (e.g., and as one or more output values) a rating, score, and / or estimate related to the likelihood that the offer will be accepted, the predicted work performance value, and / or the predicted retention probability or duration. The identification of which worker(s) to send the request to perform the work may be made automatically and / or based on a communication reflecting an identification generated by a remote system and / or user. For example, various data (e.g., ratings, scores, minimum specification data) may be presented on an interface that accepts a selection of one or more workers for a subset of workers to be utilized by the request. Each request may (but need not) be configured with corresponding configuration information identified in block 1220. In some cases, the request(s) are configured with a default (or consistent, such as a minimum or maximum) configuration.

[0160] Thus, decision tree 1200 provides an illustration of how to use multiple decision nodes to iteratively refine a pool of workers for work assignment. In some cases, responses to work requests are tracked (e.g., as described above with respect to potential techniques for tracking responses to requests to participate in a work inquiry process). If one, more, or all work requests are rejected (e.g., and / or not accepted within a default time period), then one or both of the decisions associated with blocks 1215, 1220, and 1225 may be repeated, but with the previously selected worker(s) eliminated from the worker(s) under consideration.

[0161] Fig.13An exemplary representation of data processing occurring during implementation of a progressive filtering decision tree according to some embodiments of the present invention is shown.

[0162] In the illustrated example, at a first decision node, a transformer 1305 (e.g., a block of code configured to transform one or more input data objects into processed versions thereof) receives a collection of worker files 1310 and one or more work files 1315. Each worker file 1310 may include a worker specification data structure that includes a set of worker specifications, and each work file 1315 may include a work specification data structure that includes a set of work specifications. It will be appreciated that in some cases, the transformer 1305 need not receive worker files and / or work files, but may instead identify worker specification data and / or work specification data to be transformed. For example, specification data may be received in one or more communications generated and sent from individual worker devices and / or other devices in response to detecting an input (e.g., corresponding to an online form input).

[0163] The transformer 1305 can be configured to transform each worker file 1310 into a worker feature dataset 1320 and each work file 1315 into a work feature dataset 1325. Each worker feature dataset 1320 and work feature dataset 1325 can include, for each feature in the feature set, a value indicating the degree to which the feature is represented in the corresponding (work or worker) file. Each feature can correspond to a portion of the semantic space 1330. In some cases, a machine learning model can define each feature using a training dataset (e.g., including unstructured data). The machine learning model can be trained via unsupervised learning and can include (for example) doc2vec. Features can be defined based on the co-occurrence of words and / or phrases and based on structural analysis (e.g., detecting word groups, sentences and / or paragraphs). In some cases, the optimization variables used during training (to identify feature sets) can include orthogonality measures between features. That is, feature sets can be defined so as to maximize the degree to which each individual feature is orthogonal and / or independent of each other feature.

[0164] Comparison component 1332 can compare each worker feature data set 1320 with each work feature data set 1325 to generate a comparison output for each pairwise combination. In some cases, the comparison output includes a cosine angle between feature data sets. In some cases, the comparison includes generating a correlation coefficient between feature data sets, generating a difference metric (e.g., mean, median, maximum, or minimum) based on each feature difference between the sets, and the like.

[0165] The first filter 1335 can process the comparison outputs to identify a subset 1340 of the set of worker feature datasets 1320 for each job feature dataset 1325. For example, the processing can include identifying a predefined number of workers from the set of workers based on the comparison outputs generated for the set of workers. To illustrate, filtering can include selecting the 100 worker feature datasets corresponding to the 100 highest comparison outputs (where high values ​​indicate strong similarity, but alternatively, when the comparison outputs are configured such that low values ​​indicate strong similarity, the 100 lowest comparison outputs can be identified). In some cases, filtering includes selecting each worker feature dataset associated with a comparison output that is above (or, in an alternative configuration of the output, below) a predefined threshold.

[0166] At the second decision node, the machine learning model 1345 may also process a subset 1340 of the set of worker feature data sets 1320 and an individual job feature data set 1325. The machine learning model 1345 may be configured to be trained via supervised learning. The machine learning model 1345 may be configured to process structured data sizes (e.g., of a pre-identified format and / or size). The machine learning model 1345 may be configured to output a value (e.g., a rating or score) indicating a correspondence with a job associated with the job feature data set 1325 for each subset 1340 of the set of worker feature data sets. Thus, the output may include a rating 1350 of the worker for each worker in one, some, or all of the subsets 1340. The machine learning model may include a learning to rating model. In some cases, the machine learning model 1345 is trained based on optimization variables corresponding to inputs received from a user. The input may identify a prior decision (e.g., identifying—regarding one or more specific workers—whether to hire and / or accept an interview, hire and / or accept a job, and / or identify an evaluation of job performance).

[0167] In addition to the feature data sets, the machine learning model 1345 may also process data from the worker files 1310 corresponding to the subset 1340 and / or the job files 1315 corresponding to the job feature data set 1325. The other data may include, for example, key-value data and / or individual form entries, and may be identified via implementation of the second filter 1352. In some cases, the other data for a given worker (or job) may be aggregated (e.g., concatenated) with the corresponding worker feature data set 1340 (or job feature data set 1325), and the concatenated data may be input to the model.

[0168] The rating 1350 can be processed at a third decision node 1355, which can (for example) further filter and reduce the subset 1340 of the worker set (e.g., to a predefined size). In some cases, the further filtering includes identifying a predefined number of workers associated with the lowest rating value. In some cases, other information (e.g., feature data sets, key-value pairs, etc.) can also be considered. In some cases, the rating is presented to the user, and the further filtering corresponds to the selection of one or more workers indicated via one or more communications from the user device. In some cases, the further filtering is performed by a machine learning model (e.g., using past worker selections to request the inquiry process to participate, past worker acceptances to participate in the inquiry process, past worker selections to work execution, past worker acceptances of work and / or past successful work executions as optimization variables). The further filtered subset of workers can include (for example) workers to whom a request to participate in the inquiry process is to be transmitted. If such a request is rejected or unanswered, the third decision node 1355 can be configured to identify another worker to be included in the further filtered subset.

[0169] At a fourth decision node 1360, the results from the query process can be evaluated to even further filter the subset of workers (e.g., to identify a predefined number of workers for a job, such as a single worker). A machine learning model can be used to make decisions (e.g., using past worker selections for job performance, past worker acceptances of jobs, and / or past successful job performances as optimization variables). The further filtered subset of workers can include, for example, workers to whom a request to perform a job is to be transmitted. If such a request is denied or unanswered, then the fourth decision node 1360 can be configured to identify another worker to be included in the further filtered subset.

[0170] At a fifth decision node 1365, work specification information can be determined on a per-worker basis. For example, with respect to each of the one or more workers, one or more work specifications can be identified that correspond to a predicted lower threshold or value predicted to be sufficient for the worker to accept the job. In some cases, decision nodes 1360 and 1365 are merged. For example, threshold specification data can be identified for each worker and transmitted to a user device, which can respond with a selection of the worker(s) to be included in a further filtered subset.

[0171] Each decision made at the third, fourth, and fifth decision nodes 1355, 1360, and 1365 may be fed back to the machine learning model 1345. Additional feedback 1370 may indicate, for example, whether the individual worker accepted the request to participate in the inquiry process and / or accepted the request to perform work and / or an indication of the quality of performance of work associated with the worker.

[0172] Fig.14 A flow chart of a process 1400 for iteratively advancing a trajectory through a decision tree using a machine learning model is shown according to some embodiments of the invention.

[0173] Process 1400 begins at block 1405, where a data structure (e.g., a data file or data set) representing a decision tree for assigning work is accessed. One or more first decision nodes of the decision tree may correspond to decisions about how to process unstructured data in worker information and / or job information to convert it into meaningful structured data. The conversion may be performed using an unsupervised machine learning model that is trained to detect various relationships and structures between (e.g.,) words and then defines a feature set for subsequently characterizing the unstructured (e.g., text) data.

[0174] At block 1410, a job characteristics dataset is generated using the machine learning model. The job characteristics dataset may include a set of values ​​that indicate characteristics of a job (via indicating the extent to which various characteristics are identified as represented in the job information). The job characteristics dataset may be generated by processing job information (e.g., job offer data) received, for example, via an online portal.

[0175] At block 1415, a worker specification dataset is generated for each worker in the worker set using the machine learning model. The worker characteristic dataset may include a set of numerical values ​​indicating characteristics of the worker. The worker characteristic dataset may be generated by processing worker information (e.g., resume data) received, for example, via an online portal. The work characteristic dataset may be generated by performing a semantic analysis of the worker information.

[0176] Generating normative datasets (for individual workers and / or for jobs) can be performed at a single decision node, or worker and job normative datasets can be generated at separate decision nodes (e.g., and possibly relying on different learned parameters).

[0177] At block 1420, it is detected that the trajectory corresponding to the iteration of the decision tree has reached another decision node. The other decision node may correspond to a decision regarding, for example, how to filter (or further filter) the set of workers and / or how to configure requests to be sent to the workers. The other decision node may be configured to make a decision based on processing the input data using a machine learning model (e.g., a different type of model than the model used to identify features of the feature data set). The machine learning model may include, for example, a neural network model.

[0178] At block 1425, learned parameters corresponding to the node are retrieved. The learned parameters may include one or more weights of the neural network. The learned parameters may have been generated during a training process.

[0179] At block 1430, an iteration of the machine learning model is performed. The machine learning model may be configured with node-specific learned parameters. Execution may include inputting a feature data set associated with the job and one, more than one, or all feature data sets associated with the worker into the machine learning model. In some cases, each feature data set associated with the worker is processed together with a feature data set associated with the job but not with feature data sets associated with other workers (which may be processed separately from feature data sets associated with the job). In some cases, all feature data sets associated with the worker and feature data sets associated with the job are processed in a given execution of the model. Results of the model may include, for example, filtering a set of workers, rating a set of workers, assigning a score to each worker of a set of workers, and the like.

[0180] Where the decision tree includes additional decision nodes that use machine learning models to make or facilitate decisions, process 1400 may return to block 1420. The models may be the same or different types across decision nodes.

[0181] At block 1435, one or more communication actions are triggered that represent one or more results generated using the machine learning results generated at the one or more decision nodes. For example, the communication may include a request sent to the worker to participate in the inquiry process or to accept the work. As another example, the communication may include an identification of one or more workers included in the subset of workers generated at the decision node. As yet another example, for each of the one or more workers, the communication may include a rating or score generated using the machine learning model that represents a prediction, for example, that the worker's specifications correspond to the job specifications, that the worker will accept the request to participate in the inquiry process, that the worker will accept the request to perform the work, or that the worker will successfully perform the work.

[0182] Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits can be shown in block diagrams to avoid obscuring the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

[0183] The implementation of the above-mentioned techniques, blocks, steps and means can be accomplished in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the above-mentioned functions and / or combinations thereof.

[0184] Moreover, it should be noted that the embodiments can be described as processes depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flow diagrams can describe operations as sequential processes, many operations can be performed in parallel or concurrently. In addition, the order of operations can be rearranged. The process terminates when its operations are completed, but may have additional steps not included in the figure. The process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination corresponds to the function returning to the calling function or main function.

[0185] In addition, embodiments can be implemented by hardware, software, scripting language, firmware, middleware, microcode, hardware description language and / or any combination thereof. When implemented with software, firmware, middleware, scripting language and / or microcode, the program code or code segment performing the necessary tasks can be stored in a machine-readable medium such as a storage medium. Code segments or machine executable instructions can represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, scripts, classes, or instructions, data structures and / or program statements. Code segments can be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, parameters, parameters and / or memory contents. Information, parameters, parameters, data, etc. can be transmitted, forwarded or sent via any suitable means (including memory sharing, message transmission, ticket transmission, network transmission, etc.).

[0186] For firmware and / or software implementations, the methods can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly implements instructions can be used to implement the methods described herein. For example, software code can be stored in a memory. The memory can be implemented within a processor or outside a processor. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, non-volatile, or other storage medium, and is not limited to any specific type of memory or the amount of memory, or the type of medium on which the memory is stored.

[0187] Moreover, as disclosed herein, the term "storage medium" may refer to one or more memories for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing or carrying (one or more) instructions and / or data.

[0188] Although the principles of the present disclosure have been described above in conjunction with specific devices and methods, it should be clearly understood that this description is only made by way of example rather than limiting the scope of the present disclosure.

Claims

1. A computer-implemented method comprising: accessing a data structure representing a progressive filtering decision tree for assigning work, wherein the progressive filtering decision tree includes an ordered set of decision nodes and is configured to perform iterative decisions corresponding to progressive filtering of workers, wherein at least one decision node of the ordered set of decision nodes of the progressive filtering decision tree: (i) corresponds to an iterative decision in the iterative decision; and (ii) is configured to identify, from the workers, an incomplete subset of the workers for performing the work, the incomplete subset of the workers being input to a subsequent decision node of the ordered set of decision nodes, and wherein, for each decision node of the ordered set of decision nodes, each of one or more trajectory extensions to the decision node triggers an intermediate decision or a final decision related to configuring a request for a worker to perform the work; initiating a progressive filtering of a decision tree of tracks associated with a set of workers for potential assignment of the job; detecting that the trajectory has reached a first decision node of an ordered set of decision nodes in a progressive filtering decision tree; retrieving a first set of node-specific learned parameters generated by training a first machine learning model with a first training data set, wherein the first set of node-specific learned parameters defines how to transform unstructured data into structured feature data; accessing a plurality of worker-associated unstructured data sets, each worker-associated unstructured data set of the plurality of worker-associated unstructured data sets corresponding to a worker in the set of workers and comprising unstructured data; and executing one or more first iterations of a first machine learning model configured with a first set of node-specific learned parameters and using the plurality of worker-associated unstructured data sets, wherein execution of the one or more first iterations of the first machine learning model generates a first result including a plurality of worker feature data sets, each of the plurality of worker feature data sets corresponding to a worker in the set of workers and indicating a degree to which the worker-associated unstructured data set corresponding to the worker represents each feature in a feature set, wherein each of the plurality of worker feature data sets is structured data, and wherein the feature set is defined based on the first set of node-specific learned parameters; identifying a subset of the set of workers for continued evaluation of the work based on the plurality of worker characteristic data sets; detecting that the trajectory has reached a second decision node of the ordered set of decision nodes in the progressive filtering decision tree; retrieving a second set of node-specific learned parameters generated by training a second machine learning model with a second training data set, wherein the second set of node-specific learned parameters defines how the structured data is transformed; performing one or more second iterations of a second machine learning model configured with a second set of node-specific learned parameters and using a subset of the plurality of worker feature data sets, wherein each worker feature data set in the subset of the plurality of worker feature data sets corresponds to a worker in the subset of the set of workers, and wherein execution of the one or more second iterations of the second machine learning model generates a second result indicating, for each worker of at least one worker in the subset of the set of workers, an estimated absolute or relative degree of correspondence between the job and the worker; and A communication action representing the second result is triggered.

2. The method of claim 1, further comprising: identifying a plurality of key-value pairs, wherein each key-value pair in the plurality of key-value pairs corresponds to a worker in the subset of the set of workers; wherein the execution of the one or more second iterations of the second machine learning model also uses the multiple key-value pairs, and wherein the execution of the one or more first iterations of the first machine learning model does not use the multiple key-value pairs.

3. The method of claim 1, further comprising: learning a first set of node-specific learned parameters using an unsupervised learning process; as well as A second set of node-specific learned parameters is learned using a supervised learning process. 4 . The method of claim 1 , wherein each of the plurality of worker feature data sets comprises a vector, a matrix, or an array, and wherein the vector, the matrix, or the array has a predefined dimension.

5. The method of claim 1, further comprising: accessing a working unstructured data set corresponding to said work; executing another iteration of the first machine learning model configured with the first set of node-specific learned parameters and using the working unstructured dataset, wherein execution of the another iteration of the first machine learning model generates a working feature dataset indicating the degree to which the working unstructured dataset represents each feature in the feature set; as well as For each worker in the set of workers, a work comparison metric is generated based on a worker feature dataset corresponding to the worker and the work feature dataset, wherein the subset of the set of workers is identified based on the work comparison metric generated for the set of workers.

6. The method of claim 1 further comprises training a first machine learning model using a first training data set to learn a first set of node-specific learned parameters, wherein the first training data set comprises other unstructured data, and wherein the first set of node-specific learned parameters associates specific semantic characteristics with experience levels, skills, or job characteristics.

7. The method of claim 1, further comprising: detecting that the trajectory has reached a third decision node of an ordered set of decision nodes in a progressive filtering decision tree; as well as Executing a third iteration of a third machine learning model configured with a third set of node-specific learned parameters and using at least one of the plurality of worker feature data sets, wherein a third result of executing the third iteration of the third machine learning model represents one or more request configurations predicted to result in accepting a request for a specific worker of the subset to perform the work, the request being configured with the one or more request configurations.

8. The method of claim 1, wherein triggering the communication action comprises transmitting a communication comprising the second result to a user device.

9. The method of claim 1, wherein: Based on the second result, identifying one or more workers from the set of workers for requesting to participate in the query process, and wherein the method further comprises: For each of the one or more workers, a request to participate in the interrogation process is transmitted to a device associated with the worker.

10. The method of claim 1, wherein: For each worker in the subset of the set of workers, the second result includes a rating of the worker or a score of the worker.

11. The method of claim 1 , further comprising: detecting that the trajectory has reached a second decision node of the ordered set of decision nodes in the progressive filtering decision tree; identifying a specific worker to whom a request to perform said work is to be provided; as well as Triggering another communication identifying or being transmitted to the particular worker.

12. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform actions comprising: accessing a data structure representing a progressive filtering decision tree for assigning work, wherein the progressive filtering decision tree includes an ordered set of decision nodes and is configured to perform iterative decisions corresponding to progressive filtering of workers, wherein at least one decision node of the ordered set of decision nodes of the progressive filtering decision tree: (i) corresponds to an iterative decision in the iterative decision; and (ii) is configured to identify, from the workers, an incomplete subset of the workers for performing the work, the incomplete subset of the workers being input to a subsequent decision node of the ordered set of decision nodes, and wherein, for each decision node of the ordered set of decision nodes, each of one or more trajectory extensions to the decision node triggers an intermediate decision or a final decision related to configuring a request for a worker to perform the work; initiating a progressive filtering of a decision tree of tracks associated with a set of workers for potential assignment of the job; detecting that the trajectory has reached a first decision node of an ordered set of decision nodes in a progressive filtering decision tree; retrieving a first set of node-specific learned parameters generated by training a first machine learning model with a first training data set, wherein the first set of node-specific learned parameters defines how to transform unstructured data into structured feature data; accessing a plurality of worker-associated unstructured data sets, each worker-associated unstructured data set of the plurality of worker-associated unstructured data sets corresponding to a worker in the set of workers and comprising unstructured data; and executing one or more first iterations of a first machine learning model configured with a first set of node-specific learned parameters and using the plurality of worker-associated unstructured data sets, wherein execution of the one or more first iterations of the first machine learning model generates a first result including a plurality of worker feature data sets, each of the plurality of worker feature data sets corresponding to a worker in the set of workers and indicating a degree to which the worker-associated unstructured data set corresponding to the worker represents each feature in a feature set, wherein each of the plurality of worker feature data sets is structured data, and wherein the feature set is defined based on the first set of node-specific learned parameters; identifying a subset of the set of workers for continued evaluation of the work based on the plurality of worker characteristic data sets; detecting that the trajectory has reached a second decision node of the ordered set of decision nodes in the progressive filtering decision tree; retrieving a second set of node-specific learned parameters generated by training a second machine learning model with a second training data set, wherein the second set of node-specific learned parameters defines how the structured data is transformed; performing one or more second iterations of a second machine learning model configured with a second set of node-specific learned parameters and using a subset of the plurality of worker feature data sets, wherein each worker feature data set in the subset of the plurality of worker feature data sets corresponds to a worker in the subset of the set of workers, and wherein execution of the one or more second iterations of the second machine learning model generates a second result indicating, for each worker of at least one worker in the subset of the set of workers, an estimated absolute or relative degree of correspondence between the job and the worker; and A communication action representing the second result is triggered.

13. The computer program product of claim 12, wherein the actions further comprise: identifying a plurality of key-value pairs, wherein each key-value pair in the plurality of key-value pairs corresponds to a worker in the subset of the set of workers; wherein the execution of the one or more second iterations of the second machine learning model also uses the multiple key-value pairs, and wherein the execution of the one or more first iterations of the first machine learning model does not use the multiple key-value pairs.

14. The computer program product of claim 12, wherein the actions further comprise: learning a first set of node-specific learned parameters using an unsupervised learning process; as well as A second set of node-specific learned parameters is learned using a supervised learning process.

15. The computer program product of claim 12, wherein each of the plurality of worker characteristic data sets comprises a vector, a matrix, or an array, and wherein the vector, the matrix, or the array has a predefined dimension.

16. The computer program product of claim 12, wherein the actions further comprise: accessing a working unstructured data set corresponding to said work; executing another iteration of the first machine learning model configured with the first set of node-specific learned parameters and using the working unstructured dataset, wherein execution of the another iteration of the first machine learning model generates a working feature dataset indicating the degree to which the working unstructured dataset represents each feature in the feature set; as well as For each worker in the set of workers, a work comparison metric is generated based on a worker feature dataset corresponding to the worker and the work feature dataset, wherein the subset of the set of workers is identified based on the work comparison metric generated for the set of workers.

17. A computer program product as described in claim 12, wherein the action also includes training a first machine learning model using a first training data set to learn a first set of node-specific learned parameters, wherein the first training data set includes other unstructured data, and wherein the first set of node-specific learned parameters associates specific semantic characteristics with experience levels, skills, or work characteristics.

18. The computer program product of claim 12, wherein the actions further comprise: detecting that the trajectory has reached a third decision node of an ordered set of decision nodes in a progressive filtering decision tree; as well as Executing a third iteration of a third machine learning model configured with a third set of node-specific learned parameters and using at least one of the plurality of worker feature data sets, wherein a third result of executing the third iteration of the third machine learning model represents one or more request configurations predicted to result in accepting a request for a specific worker of the subset to perform the work, the request being configured with the one or more request configurations.

19. The computer program product of claim 12, wherein triggering the communication action comprises transmitting a communication comprising the second result to a user device.

20. A system for iteratively advancing a trajectory through a decision tree using a machine learning model, the system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions comprising: accessing a data structure representing a progressive filtering decision tree for assigning work, wherein the progressive filtering decision tree includes an ordered set of decision nodes and is configured to perform iterative decisions corresponding to progressive filtering of workers, wherein at least one decision node of the ordered set of decision nodes of the progressive filtering decision tree: (i) corresponds to an iterative decision in the iterative decision; and (ii) is configured to identify, from the workers, an incomplete subset of the workers for performing the work, the incomplete subset of the workers being input to a subsequent decision node of the ordered set of decision nodes, and wherein, for each decision node of the ordered set of decision nodes, each of one or more trajectory extensions to the decision node triggers an intermediate decision or a final decision related to configuring a request for a worker to perform the work; initiating a progressive filtering of a decision tree of tracks associated with a set of workers for potential assignment of the job; detecting that the trajectory has reached a first decision node of an ordered set of decision nodes in a progressive filtering decision tree; retrieving a first set of node-specific learned parameters generated by training a first machine learning model with a first training data set, wherein the first set of node-specific learned parameters defines how to transform unstructured data into structured feature data; accessing a plurality of worker-associated unstructured data sets, each worker-associated unstructured data set of the plurality of worker-associated unstructured data sets corresponding to a worker in the set of workers and comprising unstructured data; and executing one or more first iterations of a first machine learning model configured with a first set of node-specific learned parameters and using the plurality of worker-associated unstructured data sets, wherein execution of the one or more first iterations of the first machine learning model generates a first result including a plurality of worker feature data sets, each of the plurality of worker feature data sets corresponding to a worker in the set of workers and indicating a degree to which the worker-associated unstructured data set corresponding to the worker represents each feature in a feature set, wherein each of the plurality of worker feature data sets is structured data, and wherein the feature set is defined based on the first set of node-specific learned parameters; identifying a subset of the set of workers for continued evaluation of the work based on the plurality of worker characteristic data sets; detecting that the trajectory has reached a second decision node of the ordered set of decision nodes in the progressive filtering decision tree; retrieving a second set of node-specific learned parameters generated by training a second machine learning model with a second training data set, wherein the second set of node-specific learned parameters defines how the structured data is transformed; performing one or more second iterations of a second machine learning model configured with a second set of node-specific learned parameters and using a subset of the plurality of worker feature data sets, wherein each worker feature data set in the subset of the plurality of worker feature data sets corresponds to a worker in the subset of the set of workers, and wherein execution of the one or more second iterations of the second machine learning model generates a second result indicating, for each worker of at least one worker in the subset of the set of workers, an estimated absolute or relative degree of correspondence between the job and the worker; and A communication action representing the second result is triggered.

Images