Data integrity optimization

Abstract

A method includes receiving interaction data indicative of a user of a user device performing a specified action, identifying a last temporal action associated with the user and an initial action associated with the user and the specified action, propagating, based on the identified last temporal and initial actions, a first attribute associated with the identified last temporal action and the specified action and a second additional attribute associated with the identified initial action and the specified action, to two or more different models, and generating, based on the first attribute and the second additional attribute, one or more visual representations of the first attribute and the second additional attribute.

Description

BACKGROUND

[0001] This specification relates to data processing and preserving data integrity when collecting and analyzing user data for modeling purposes. SUMMARY

[0002] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method that includes receiving, by one or more processors, interaction data indicative of a user of a user device performing a specified action, identifying, by the one or more processors, a last time action associated with the user and an initial action associated with the user and the specified action, generating, by the one or more processors and based on the identified last time action and the identified initial action, a first attribute associated with the identified last time action and the specified action and a second additional attribute associated with the identified initial action and the specified action, propagating, by the one or more processors, the first attribute and the second additional attribute to two or more different models, and generating, by the one or more processors and based on the first attribute and the second additional attribute, one or more visual representations of the first attribute and the second additional attribute.

[0003] In some implementations, the one or more visual representations include a first visual representation of the first attribute and a second, different visual representation of the second additional attribute. In some implementations, the second, different visual representation of the second additional attribute is visually distinct from the first visual representation of the first attribute.

[0004] In some implementations, identifying the last time action associated with the user and the initial action associated with the user and the specified action includes querying one or more interaction databases.

[0005] In some implementations, the specified action includes providing user input through a user interface element.

[0006] In some implementations, the initial action includes downloading and installing an application on the user device.

[0007] In some implementations, the method includes determining, by the one or more processors, that the identified initial action occurred within a predetermined time period before the specified action occurred.

[0008] Systems using different models can experience discrepancies due to the models using different data sets. For example, when these systems use multiple models and one model has access to data that another model does not, some systems can lose or fail to collect data that should be analyzed. This loss of data can result in discrepancies, particularly when only one model is able to collect a particular kind of data, resulting in false predictions from the other model due to the loss or inability to access the data. The following description discusses techniques that preserve data integrity by ensuring that data is provided to or accessed by a particular model to improve the accuracy of the model individually and the system as a whole. Furthermore, these techniques ensure that data is replicated to the appropriate system, improving the accuracy of the model and preventing loss of data.

[0009] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. For example, the solutions described in this specification also reduce data integrity issues by preventing the propagation of inaccuracies. In other words, the improved methods provide a model with the same data that is provided to another model, such that the data used across the models is consistent. Furthermore, these methods improve the accuracy of the models by collecting and analyzing more representative and more complete data sets than were previously used. In other words, the improved methods allow the models to consider new factors that were not previously considered. Furthermore, the methods expand the field of feedback available to the models. For example, although models such as content selection models typically use a particular type of recent activity data as feedback, the techniques described in this specification allow these models to use other types of data and / or data from a more extensive time window, such as activity that occurred within a predetermined period of time before the current time.

[0010] By improving the accuracy of these models, the computer reduces the amount of processing required to provide content that a user is less likely to be interested in or to influence the user to take a particular action. Furthermore, by preventing the propagation of inaccurate or incomplete data and / or results, the method improves the efficiency of the system processing the models by reducing the number of processing cycles required to achieve accurate results.

[0011] The method also provides a visual indication of data that was not previously considered or available to a particular model. By emphasizing the data that has been added, these methods allow a user to quickly and easily identify new data and any impact the data has when included in the model.

[0012] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram of an example environment for optimizing data integrity during data collection and analysis.

[0014] Figure 2 Data flow depicting a method for improving data completeness in modeling.

[0015] Figure 3 is a flowchart of an example method for improving data completeness in modeling of data collection and analysis.

[0016] Figure 4 is a block diagram of an example computer system.

[0017] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] The present disclosure relates to computer-implemented methods and systems that optimize data completeness when collecting and analyzing user data.

[0019] Through the exchange of information and transactions between users and content providers, providers can receive user data such as, for example, the type of content a user accesses, the time at which a user last accessed a provider’s website, and other information related to a user’s interaction with a provider and / or the provider’s website. For purposes described herein, “content” refers to digital content, including third-party content provided by a content provider. A content item refers to a specific piece of content, and includes a digital component for presentation with other content requested by a user. A system that distributes content to users can select content based on models that predict the relevance of content items to a particular user and the likelihood of a user interacting with the content, among other factors. In some cases, the system uses multiple models, but the models do not share data sets. Rather, the models are trained using specific data sets that do not represent all factors. For example, some models can not have access to specific data sets, and can be trained using incomplete data sets. Models that use incomplete data sets will produce inaccuracies compared to models that use more complete data sets, and the continued use of the results of these models in subsequent modeling, such as inputs to other models or feedback to the models themselves, only serves to propagate the inaccuracies. Existing methods do not account for specific types of missing data, or do not collect or have access to data that can improve the accuracy of the models. As described in detail throughout this specification, the innovative technologies herein allow for techniques to improve data completeness in specific systems that use multiple models to access different data sets.

[0020] In addition to the descriptions throughout this document, a user can be provided with controls to allow the user to make an election as to whether and when the systems, programs, or features described herein can enable collection of user information (e.g., information about a user's social network, social actions, or activities, a user's preferences, or a user's current location), and if the user is sent content or communications from a server that the user can elect to

[0021] Figure 1 is a block diagram of an example environment 100 for optimizing data integrity during data collection and analysis. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 104 ("electronic document server"), user devices 106, a digital component distribution system 110 (also referred to as DCDS 110), and a privacy server 120. The example environment 100 can include many different electronic document servers 104 and user devices 106.

[0022] A user device 106 is an electronic device capable of requesting and receiving resources (e.g., electronic documents) over the network 102. Example user devices 106 include personal computers, wearable devices, smart speakers, tablet devices, mobile communication devices (e.g., smart phones), smart appliances, and other devices that can send and receive data over the network 102. In some implementations, a user device can include a speaker that outputs audible information to a user and a microphone that accepts audible input (e.g., spoken language input) from a user. A user device can also include a digital assistant that provides an interactive voice interface for submitting input and / or receiving output provided in response to the input. A user device can also include a display for presenting visual information (e.g., text, images, and / or video). A user device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data over the network 102, although native applications executed by the user device 106 can also facilitate sending and receiving data over the network 102.

[0023] An electronic document is data that presents a set of content at a user device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on a mobile device, tablet, or desktop computing device, are also examples of electronic documents. An electronic document 105 ("electronic document") can be provided to a user device 106 by an electronic document server 104. For example, the electronic document server 104 can include a server that hosts a publisher's website. In this example, the user device 106 can initiate a request for a given publisher's web page, and the electronic document server 104 that hosts the given publisher's web page can respond to the request by sending machine hypertext markup language (HTML) code that initiates presentation of the given web page at the user device 106.

[0024] An electronic document can include a variety of content. For example, an electronic document 105 can include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. An electronic document can also include dynamic content that can change over time or on a per-request basis. For example, a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a tag or script that, when the given electronic document is processed (e.g., rendered or executed) by the user device 106, causes the user device 106 to request content from the data source. The user device 106 integrates the content obtained from the data source into the presentation of the given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0025] In some cases, a given electronic document can include a digital content tag or a digital content script that references the DCDS 110. In these cases, the digital content tag or the digital content script is executed by the user device 106 when the given electronic document is executed by the user device 106. Execution of the digital content tag or the digital content script configures the user device 106 to generate a request 108 for digital content that is transmitted to the DCDS 110 over the network 102. For example, the digital content tag or the digital content script can enable the user device 106 to generate a packetized data request that includes header and payload data. The request 108 can include information such as the name (or network location) of the server from which the digital content is requested, the name (or network location) of the requesting device (e.g., the user device 106), and / or information that the DCDS 110 can use to select digital content to provide in response to the request. The request 108 is transmitted by the user device 106 to a server of the DCDS 110 over the network 102 (e.g., a telecommunications network).

[0026] Request 108 can include data specifying the electronic document and characteristics of the locations where digital content can be presented. For example, data specifying a reference (e.g., a URL) to the electronic document (e.g., a web page) in which the digital content is to be presented, available locations (e.g., digital content slots) of the electronic document that can be used to present digital content, sizes of the available locations, locations of the available locations in the presentation of the electronic document, and / or media types that are eligible for presentation in the locations can be provided to DCDS 110. Data specifying keywords for selection of the electronic document ("document keywords") or entities referenced by the electronic document (e.g., people, places, or things) can also be included in request 108 (e.g., as payload data) and provided to DCDS 110 to facilitate identification of digital content items that are eligible for presentation of the electronic document.

[0027] Request 108 can also include data related to other information, such as information that has been provided by the user, geographic information indicating the state or region in which the request was submitted, or other information providing context for the environment in which the digital content will be displayed (e.g., the type of device on which the digital content will be displayed, such as a mobile device or a tablet device). User-provided information can include demographic data for the user of user device 106. For example, demographic information can include age, gender, geographic location, education level, marital status, household income, occupation, hobbies, social media data, and whether the user owns particular items, among other characteristics.

[0028] Data specifying characteristics of user device 106 can also be provided in request 108, such as information identifying the model of user device 106, the configuration of user device 106, or the size (e.g., physical size or resolution) of the electronic display (e.g., a touchscreen or a desktop display) on which the electronic document is presented. Request 108 can be transmitted, for example, over a packetized network, and request 108 itself can be formatted as packetized data having a header and payload data. The header can specify the destination of the packet and the payload data can include any of the information discussed above.

[0029] The DCDS 110 selects digital content to be presented with a given electronic document in response to receiving the request 108 and / or using information included in the request 108. In some implementations, the DCDS 110 is implemented in a distributed computing system (or environment) that includes, for example, a server and a collection of multiple computing devices that are interconnected and identify and distribute digital content in response to the request 108. The collection of multiple computing devices operate together to identify a set of digital content that is eligible to be presented in the electronic document from a corpus of millions or more available digital content. The millions or more available digital content can be indexed, for example, in a digital component database 112. Each digital content index entry can reference a respective digital content and / or include distribution parameters (e.g., selection criteria) that regulate distribution of the respective digital content.

[0030] The identification of eligible digital content can be split into multiple tasks that are then assigned among the computing devices within the collection of multiple computing devices. For example, different computing devices can each analyze different portions of the digital component database 112 to identify various digital content that has distribution parameters that match information included in the request 108.

[0031] The DCDS 110 aggregates results received from the collection of multiple computing devices and uses information associated with the aggregated results to select one or more instances of digital content to be provided in response to the request 108. In turn, the DCDS 110 can generate and transmit reply data 114 (e.g., digital data representing a reply) over the network 102 that enables the user device 106 to integrate the selected set of digital content into the given electronic document so that the selected set of digital content and content of the electronic document are presented together on a display of the user device 106.

[0032] Figure 2 is a data flow diagram of a method 200 for data collection and analysis. Operations of the method 200 are performed by various components of the system 100. For example, operations of the method 200 can be performed by components of the DCDS 110 that are in communication with the user device 106, including an activity processor 202, a last interaction database 204, an initial interaction database 206, an action database 208, a re-engagement model 210, an initial engagement model 212, and a user interface generator 214. These components of the DCDS 110 can be implemented as physical subsystems and / or software modules.

[0033] The method 200 encapsulates the above-described data integrity improvements and enables models within a content distribution system to analyze information that is not typically available to these models. Moreover, the method 200 allows a content distribution system to arrive at more accurate results when evaluating factors that influence user actions.

[0034] When multiple different content campaigns with different parameters run in parallel, it becomes more difficult to distinguish the effects of one campaign from the effects of another. Without collecting and / or analyzing data reflecting the effects of one campaign, these effects can be misrecorded or attributed to another campaign, or lost entirely.

[0035] The method 200 describes a process by which data quantifying the effects of a particular campaign can be collected and stored for use in models that previously had no access to such data, improving the accuracy and consistency of the models.

[0036] The campaign processor 202 can detect and process user actions. The campaign processor 202 can receive user interaction data and determine, based on the received data, that a particular action has occurred. In some implementations, the campaign processor 202 can also receive data indicating that a particular action has occurred. For example, the particular action can be a conversion event. A conversion event is a user action that is desired and / or specified by an entity such as a content provider. Conversion events can include, for example, navigating to a particular page, completing a purchase, interacting with a particular user interface element, downloading a particular content item, installing a particular program or application, and the like. For example, the campaign processor 202 can receive user interaction data and determine, based on the received data, that a user has completed a purchase of lives in a game application such as an application running on the user device 106. The particular action can be specified by a content provider, by an application, by the DCDS 110, and other entities. For example, a content provider that provides content items distributed by the DCDS 110 to the user device 106 can specify a particular action performed by a user that the campaign processor 202 is configured to detect.

[0037] The last interaction database 204 maintains a collection of last time interactions. In some implementations, the last interaction database 204 indexes each interaction. In some implementations, the last interaction database 204 stores an identifier or reference to each interaction. Each of these interactions is a most recently occurring interaction associated with a particular user of the user device 106. For example, the last time interaction can be a click on a link to load an application that is a particular action that the campaign processor 202 is configured to detect. For example, the interaction data can indicate an interaction type, an associated address of a website or application through which the interaction was performed, a user associated with the action, a user device associated with the action, a time and date of the interaction, or an interaction that occurred prior to the current interaction, and other information related to the interaction. The interactions can be indexed, for example, according to a user or user device 106, according to a time at which the interaction occurred, or according to a type of the interaction, among other attributes by which the interactions can be indexed.

[0038] The initial interaction database 206 maintains a collection of initial interactions. Each of these initial interactions is an interaction logged as being associated with a user performing an initial action related to a specified action that the activity processor 202 has detected. The initial action can include downloading and installing an application. For example, an initial interaction can be an interaction to which an action of a particular user of the user device 106 downloading and installing an online shopping application is attributed. In some implementations, the initial interaction database 206 indexes each interaction. In some implementations, the initial interaction database 206 stores an identifier or reference to each interaction. For example, the interaction data can indicate an interaction type, an associated address of a website or application through which the interaction was performed, a user associated with the action, a user device associated with the action, a time and date of the interaction, or an interaction that occurred prior to the current interaction, and other information associated with the interaction. Interactions can be indexed, for example, according to a user or user device 106, according to a time at which the interaction occurred, or according to a type of the interaction, among other attributes that the interactions can be indexed by.

[0039] The action database 208 maintains a collection of actions, such as actions listed as particular actions that the activity processor 202 is configured to detect. The collection of actions includes a specified action for which the last interactions stored in the last interaction database 204 and the initial interactions stored in the initial database 206 are logged and for which the actions can be attributed. As discussed above, the actions can be specified by an entity including a content provider. In some implementations, each content distribution system, such as the DCDS 110, maintains its own action database.

[0040] The re-engagement model 210 is a model that predicts a likelihood and / or efficacy of a particular activity affecting user actions after the user has performed an initial action. For example, the re-engagement model 210 considers data of interactions that occur after a user has performed an action, such as downloading and installing a game application.

[0041] The initial engagement model 212 is a model that predicts a likelihood and / or efficacy of a particular activity affecting a user performing an initial action. For example, the initial engagement model 212 considers data of interactions that occur directly before a user has performed an action, such as downloading and installing a navigation application.

[0042] The re-engagement model 210 and the initial engagement model 212 can use, for example, artificial intelligence and machine learning techniques to predict the likelihood and / or efficacy of a particular activity. The re-engagement model 210 and the initial engagement model 212 can use the predicted likelihood to generate, for example, a value associated with each activity.

[0043] The re-engagement model 210 and the initial engagement model 212 can use statistical and / or machine learning models that accept user-provided information as input. The machine learning models can use any of a variety of models such as decision trees, generative adversarial network-based models, deep learning models, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, ensemble models (e.g., using techniques such as bagging, boosting, random forests, etc.), genetic algorithms, Bayesian networks, etc., and can be trained using a variety of methods such as deep learning, association rules, inductive logic, clustering, maximum entropy classification, learning classification, etc. In some examples, the machine learning models can use supervised learning. In some examples, the machine learning models use unsupervised learning.

[0044] The user interface generator 214 generates user interfaces that display the particular action and attribute data provided to the re-engagement model 210 and the initial engagement model 212. For example, the user interface generator 214 can generate a user interface for a user of the system 100 to comment on changes to the data provided to the models 210 and 212. The user interface generator 214 can highlight data that is typically not provided to the models 210 and 212, allowing the user to quickly identify new information.

[0045] The method 200 begins at step A, where the activity processor 202 receives interaction data associated with a specified action, such as leaving a comment about a set of directions provided by a navigation application. In another example, the interaction data can indicate a user input to load an application or data. The activity processor 202 can detect that the specified action has occurred based on the interaction data.

[0046] The method 200 continues at step B, where the activity processor 202 accesses the last interaction database 204 and the initial interaction database 206. The activity processor 202 can retrieve the last interaction associated with a user of the user device 106 in the last interaction database 204. For example, the activity processor 202 can access the last interaction associated with a user of the user device 106 from the last interaction database 204, clicking on a suggested action to comment on a restaurant they visited yesterday. In addition, the activity processor 202 also considers the initial interactions associated with the user of the user device 106 in the initial interaction database 204, as opposed to existing methods for modeling activity efficacy that only consider the last time interaction or interactions performed immediately preceding the specified action. For example, the activity processor 202 can access the initial interaction associated with a user of the user device 106 and the specified action detected by the activity processor 202 in step A, downloading and installing a suggested link to a navigation application. By considering this additional initial data, the method 200 improves the accuracy of any subsequent modeling as a result of including interactions that have an impact on modeling results that were not previously considered.

[0047] Method 200 continues with step C, in which activity processor 202 generates two attribute instances and provides the attribute instances to action database 208. One attribute instance is associated with the specified action and the last interaction action retrieved from last interaction database 204. The other instance of the attribute is associated with the specified action and the initial action retrieved from initial interaction database 206. Action database 208 maintains instances of the attribute and the specified action.

[0048] In contrast to existing methods that record only one attribute instance, method 200 allows for data that was not collected or provided for analysis to also be recorded, thereby improving data integrity and accuracy of re-engagement model 210 and initial engagement model 212. In addition, content providers were previously unable to assess the true value of initial engagement activities because subsequent attributes, such as those that occur directly after a different interaction, were not reported to the initial engagement model. Method 200 allows initial engagement model 212 to take into account these subsequent interactions that were previously not collected or lost data.

[0049] In some implementations, activity processor 202 analyzes the timestamp of the initial action to determine whether the action occurred within a predetermined time period of the specified action that was detected in step A. For example, activity processor 202 can use the interaction data to determine that the initial action occurred more than 30 days before the specified action. If the initial action occurred outside of the predetermined time period of the specified action, activity processor 202 can exclude the initial action to account for the diminished effect of the initial action on the user’s action. In these cases, activity processor 202 can not generate an attribute instance for the initial action. For example, an activity that has occurred more than a threshold number of days can have occurred too long ago to have an effect on the user’s action, while an activity that occurred within the threshold number of days but before the last time action can still have an effect. By imposing a threshold number of days, the system mitigates the likelihood of attributing actions to activities that are unlikely to have an effect on the user’s action, while still including activities that are typically not considered or even recorded. For example, a suggestion to the user to download a navigation application that was provided two years ago can not have much of an effect on the user’s decision to currently review a clothing store within the application, but a suggestion to the user to download the navigation application the previous week can have an effect on the user’s decision to use the application to search for a current nearest taco location, and this effect can not be considered or even recorded by existing methods.

[0050] Method 200 continues with step D, in which action database 208 propagates the attribute instances to re-engagement model 210 and initial engagement model 212. These attribute instances are used as input to re-engagement model 210 and initial engagement model 212 to improve the models. For example, action database 208 can provide the attribute instances of re-engagement model 210 and initial engagement model 212 as positive examples of valid activity, such that models 210 and 212 learn from the examples. Both re-engagement model 210 and initial engagement model 212 are trained on this data to improve future predictions of user re-engagement or initial user engagement, respectively.

[0051] Method 200 continues with step E, in which action database 208 propagates the attribute instances to user interface generator 214. User interface generator 214 can then generate a visual representation of the provided data. For example, user interface generator 214 can generate user interface elements that highlight the provided data. In some implementations, user interface generator 214 can represent attributes associated with initial actions differently than attributes associated with last actions.

[0052] By highlighting newly collected and / or analyzed data, the data is made more visible, such that a user seeing the data can easily distinguish the new data from previously collected and used data. In some implementations, the system can report the new data separately within a reporting user interface, such that the new attribute data associated with initial actions is visually distinct and separate from attribute data associated with last time actions.

[0053] Figure 3 is a flowchart of an example method 300 for improving data completeness in modeling of data collection and analysis. In some implementations, method 300 can be performed by one or more systems. For example, method 300 can be implemented by DCDS 110, activity processor 202, last interaction database 204, initial interaction database 206, action database 208, re-engagement model 210, initial engagement model 212, and user interface generator 214 of Figures 1-2 In some implementations, process 300 can be implemented as instructions stored on a computer-readable medium, which can be non-transitory, and the instructions can cause one or more servers to perform the operations of process 300 when the instructions are executed by the one or more servers.

[0054] Method 300 begins with receiving, by one or more processors, interaction data indicating that a specified action has been performed by a user of a user device (302). For example, activity processor 202 can receive interaction data indicating that a specified action was performed by a user of user device 106. The specified action can include, for example, providing user input through a user interface element or downloading and installing an application on user device 106.

[0055] The method 300 continues with identifying, by one or more processors, a last temporal action associated with the user and an initial action associated with the user and the specified action (304). For example, the activity processor 202 can identify and receive data indicating a last temporal action— clicking a link to a suggested news article associated with the user, and an initial action— clicking a link to download and install a news application associated with the user and the specified action— providing annotations to the news article through a user interface element.

[0056] In some implementations, identifying the last temporal action associated with the user and the initial action associated with the user and the specified action includes querying one or more interaction databases. For example, the activity processor 202 can query the last interaction database 204 and / or the initial interaction database 206 to identify the last temporal action associated with the user and the initial action associated with the user and the specified action.

[0057] The method 300 continues with generating, by one or more processors and based on the identified last temporal action and the identified initial action, a first attribute associated with the identified last temporal action and the specified action, and a second additional attribute associated with the identified initial action and the specified action (306). For example, the activity processor 202 can generate a first attribute associated with the last temporal action and the specified action and a second attribute associated with the initial action and the specified action.

[0058] In some implementations, the activity processor 202 generates the second attribute in response to determining that the identified initial action occurred within a predetermined time period before the specified action. For example, the activity processor 202 can generate the second attribute in response to determining that the identified initial action occurred less than two weeks before the specified action occurred.

[0059] The method 300 continues with propagating, by one or more processors, the first attribute and the second additional attribute to two or more different models (308). For example, the action database 208 can independently propagate the two attributes to the reengagement model 210 and the initial engagement model 212.

[0060] The method 300 continues with generating, by one or more processors, one or more visual representations of the first attribute and the second additional attribute based on the first attribute and the second additional attribute (310). For example, the user interface generator 214 can generate visual representations of the two attributes.

[0061] In some implementations, the one or more visual representations include a first visual representation of the first attribute and a second, different visual representation of the second additional attribute. For example, the user interface generator 214 can generate separate visual representations of the first attribute and the second, different attribute.

[0062] In some implementations, the second different visual representation of the second additional attribute is visually distinct from the first visual representation of the first attribute. For example, the visual representations can have different colors, fonts, text sizes, media types (e.g., video vs. image), and / or audio, among other features.

[0063] Figure 4 is a block diagram of an example computer system 400 that can be used to perform the operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.

[0064] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0065] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0066] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 can include one or more network interface devices, e.g., an Ethernet card, a serial communication device, e.g., an RS-232 port, and / or a wireless interface device, e.g., an 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other input / output devices, e.g., keyboard, printer and display 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0067] Although an example manner of making and using the present application has been Figure 4Example processing systems are described herein, but implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0068] Media does not necessarily correspond to a file. Media can be stored in a portion of a file that holds other documents, in a single file that is dedicated to the document at issue, or in multiple coordinated files.

[0069] Embodiments of the subject matter and the operational described in this specification can be implemented using a digital electronic circuit, or using computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage media (or medium) for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0070] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0071] The term“data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple programmable processors, computers, systems on chips, or combinations thereof. The apparatus can include special purpose logic, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0072] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0073] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic

[0074] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0075] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0076] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0077] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0078] While this specification contains many specifics, these should not be construed as limiting the scope of any invention or application in that they can be practiced in other embodiments other than the particular embodiments described. Some features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0079] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0080] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the process depicted in the accompanying figures does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method, comprising: One or more processors receive interactive data instructing a user on a user device to perform a specified action; The one or more processors identify a last-time action associated with the user and an initial action associated with the user and the specified action, wherein the last-time action is (i) associated with the specified action and (ii) the most recent action performed by the user before the specified action is performed by the user, and wherein the initial action is an action performed by the user before the last-time action and is determined to have caused the download of the application for the specified action to occur; The one or more processors generate a first attribute associated with the identified last time action and the identified initial action, and a second attribute associated with the identified initial action and the identified initial action. The first attribute and the second attribute are propagated by the one or more processors to two or more different models, wherein the first model of the two or more different models is configured to predict the probability that a user will perform a specific action after the specified action is performed, and the second model of the two or more different models is configured to predict the probability that a user will perform an identified initial action. The one or more processors generate one or more visual representations of the first attribute and the second attribute based on the first attribute and the second attribute; Generate predictions for at least one of the two or more different models using at least one of the following: (i) the relevance of the digital component to one or more users or (ii) the likelihood that the one or more users will interact with the digital component; and The digital component is distributed to one or more users based on the prediction.

2. The method according to claim 1, wherein, The one or more visual representations include a first visual representation of the first attribute and a second different visual representation of the second attribute.

3. The method according to claim 2, wherein, The second different visual representation of the second attribute is visually different from the first visual representation of the first attribute.

4. The method according to claim 1, wherein, Identifying the last time action associated with the user and the initial action associated with the user and the specified action includes querying one or more interaction databases.

5. The method according to claim 1, wherein, The specified action includes providing user input through user interface elements.

6. The method according to claim 1, wherein, The initial actions include downloading and installing the application on the user's device.

7. The method of claim 1, further comprising determining by the one or more processors that the identified initial action occurred within a predetermined time period prior to the occurrence of the specified action.

8. The method according to claim 1, wherein, The first attribute attributes the specified action to an activity associated with the last time action, and the second attribute attributes the specified action to an activity associated with the initial action.

9. The method according to claim 1, wherein, Each of the two or more different models includes a trained machine learning model.

10. The method of claim 1, further comprising: The first model is trained using the first attribute and the second attribute; as well as The second model is trained using the first attribute and the second attribute.

11. A system comprising: One or more processors; as well as One or more memory elements, said one or more memory elements including instructions that, when executed, cause said one or more processors to perform operations, said operations including: The one or more processors receive interactive data instructing a user of a user device to perform a specified action; The one or more processors identify a last-time action associated with the user and an initial action associated with the user and the specified action, wherein the last-time action is (i) associated with the specified action and (ii) the most recent action performed by the user before the specified action is performed by the user, and wherein the initial action is an action performed by the user before the last-time action and is determined to have caused the download of the application for the specified action to occur; The one or more processors generate a first attribute associated with the identified last time action and the identified initial action, and a second attribute associated with the identified initial action and the identified initial action. The first attribute and the second attribute are propagated by the one or more processors to two or more different models, wherein the first model of the two or more different models is configured to predict the probability that a user will perform a specific action after the specified action is performed, and the second model of the two or more different models is configured to predict the probability that a user will perform an identified initial action. The one or more processors generate one or more visual representations of the first attribute and the second attribute based on the first attribute and the second attribute; Generate predictions for at least one of the two or more different models using at least one of the following: (i) the relevance of the digital component to one or more users or (ii) the likelihood that the one or more users will interact with the digital component; and The digital component is distributed to one or more users based on the prediction.

12. The system according to claim 11, wherein, The one or more visual representations include a first visual representation of the first attribute and a second different visual representation of the second attribute.

13. The system according to claim 12, wherein, The second different visual representation of the second attribute is visually different from the first visual representation of the first attribute.

14. The system according to claim 11, wherein, Identifying the last time action associated with the user and the initial action associated with the user and the specified action includes querying one or more interaction databases.

15. The system according to claim 11, wherein, The specified action includes providing user input through user interface elements.

16. The system according to claim 11, wherein, The initial actions include downloading and installing the application on the user's device.

17. The system of claim 11, wherein the operation further comprises determining, by the one or more processors, that the identified initial action occurs within a predetermined time period prior to the occurrence of the specified action.

18. A non-transitory computer storage medium encoded with instructions, said instructions, when executed by a distributed computing system, causing the distributed computing system to perform operations including: One or more processors receive interactive data instructing a user on a user device to perform a specified action; The one or more processors identify the last time action associated with the user and the initial action associated with the user and the specified action, wherein, The last time action is (i) related to the specified action and (ii) the most recent action performed by the user before the specified action was performed by the user, wherein the initial action was performed by the user before the last time action and is determined to have caused the download of the application for which the specified action occurred; The one or more processors generate a first attribute associated with the identified last time action and the identified initial action, and a second attribute associated with the identified initial action and the identified initial action. The first attribute and the second attribute are propagated by the one or more processors to two or more different models, wherein the first model of the two or more different models is configured to predict the probability that a user will perform a specific action after the specified action is performed, and the second model of the two or more different models is configured to predict the probability that a user will perform an identified initial action. The one or more processors generate one or more visual representations of the first attribute and the second attribute based on the first attribute and the second attribute; Generate predictions for at least one of the two or more different models using at least one of the following: (i) the relevance of the digital component to one or more users or (ii) the likelihood that the one or more users will interact with the digital component; and The digital component is distributed to one or more users based on the prediction.

19. The non-transitory computer storage medium according to claim 18, wherein, The one or more visual representations include a first visual representation of the first attribute and a second different visual representation of the second attribute.

20. The non-transitory computer storage medium according to claim 19, wherein, The second different visual representation of the second attribute is visually different from the first visual representation of the first attribute.

Images