Contribution increment machine learning model

Abstract

Methods, systems, and computer programs encoded on a computer storage medium for training and using machine learning models are disclosed. The method includes creating a model representing a relationship between user attributes, content exposures, and a level of execution of a target action using organic exposure data specifying one or more organic exposures experienced by a particular user before the particular user performs a specified target action within a specified time, and third-party exposure data specifying one or more organic exposures experienced by the particular user before the particular user performs the specified target action. The third-party exposure data specifies a third-party exposure of a specified type of digital component to a particular user within a specified time period. Using the model, an incremental execution level is determined at which an action time when a particular user performs a target action is attributable to each third-party exposure. Based on the incremental execution, a transmission criterion for at least some of the digital components that the particular user is exposed is modified.

Description

[0001] This application is a divisional application for a patent application with an application date of December 5, 2019, application number 201980053188.7, and invention name “Contribution Incremental Machine Learning Model”. Technical Field

[0002] This specification involves data processing and machine learning models. Background Art

[0003] Users engage in a variety of online activities, and each of these activities results in the user being exposed to different information. A user's subsequent online activities may be influenced by their previous activities and the information they were exposed to. However, it is difficult to assess the amount of influence each previous activity has on subsequent activities. Summary of the invention

[0004] In general, an innovative aspect of the subject matter described in this specification can be embodied in methods, which include the following operations: creating a model that represents the relationship between user attributes, content exposure, and the execution level of a specified target action; obtaining organic exposure data that specifies one or more organic exposures experienced by a specific user before the specific user performs a specified target action within a specified time, wherein the organic exposure is neither exposure to a specified type of digital component nor the execution of a specified target action; obtaining third-party exposure data that specifies third-party exposure of a specified type of digital component to a specific user within a specified time period, wherein for each third-party exposure, the third-party exposure data includes an exposure time that specifies when the third-party exposure occurred; using the model to determine the incremental execution level attributable to each third-party exposure at the action time when the specific user performs the specified target action; and modifying the transmission standard of at least some digital components based on the incremental execution level of the third-party exposure of at least some digital components to which the specific user is exposed. Other embodiments of this aspect include corresponding methods, apparatus, and computer programs encoded on a computer storage device configured to perform the actions of the method. These and other embodiments may each optionally include one or more of the following features.

[0005] The method may include performing an ablation experiment to obtain a set of control results for a group of control users who are not exposed to a specific set of digital components, the control results specifying, for each specific control user in the group of control users, whether the specific control user performed a specified action; and collecting exposure results for a group of exposed users who are not included in the group of control users, the exposure results specifying, for each exposed user in the group of exposed users, whether the exposed user performed a specified action.

[0006] The method may include creating the model by utilizing a machine learning framework to create the model using user attributes, control outcomes for each particular control user, and exposure outcomes for each exposed user.

[0007] The method may further include determining an incremental execution level attributable to each third-party exposure, including: for each third-party exposure: determining a difference between an exposure time of the third-party exposure and an action time when a specified target action occurs; based on the difference between the exposure time of the third-party exposure and the action time when the specified target action occurs, determining a residual amount of execution level contribution from the third-party exposure remaining at the action time; and attributing the residual amount of execution level to the third-party exposure.

[0008] The method may include determining, for each different type of third-party exposure, a decay function that specifies a decay rate of an execution level contribution remaining over time; and for each third-party exposure, determining a residual amount of the execution level contribution from the third-party exposure remaining at an action time based on the decay function and a difference between an exposure time and an action time for the third-party exposure.

[0009] The method may include modifying transmission criteria for at least some of the digital components, including adjusting the transmission criteria for particular digital components in proportion to the magnitude of the incremental execution level attributable to the third party exposure of the particular digital components.

[0010] The method may include adjusting a transfer standard, including disabling a particular transfer standard having an incremental execution level of a specified size less than a third party exposure attributable to the particular transfer standard.

[0011] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A machine learning model can be trained to delineate between the residual effects of various events on the results at any point in time after the various events occur. The quantification of the residual effect of each previous event at the time of the result can represent a portion of the result attributable to each previous event. Even if the events may occur at different times, this quantification of the residual effect can directly compare the impact of each event on the final result. The model discussed herein can quantify the incremental impact of each event (e.g., content exposure) relative to the baseline trend of the user reaching the target result. The data collection described herein is performed in such a way that it makes it possible to train a machine learning model to characterize the inherent probability that the user will reach the target result, and it is also possible to train the machine learning model to quantify the initial impact of the event on the inherent probability of the user, and the change of the initial impact over time after the event. The transmission standard of the digital component can be modified based on the output of the machine learning model to modify when or how the digital component is transmitted over the network.

[0012] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example environment in which online content is distributed.

[0014] Figure 2 is a graph illustrating example effects of content exposure on performance of a target action over time.

[0015] Figure 3 is a graph of an example decay of the impact of content exposure over time.

[0016] Figure 4 is a block diagram of an example model apparatus that implements the baseline model.

[0017] Figure 5 is a block diagram showing an ablation experiment.

[0018] Fig. 6A is a graph showing the incremental impact of exposure to a single digital component.

[0019] Figure 6B is a graph illustrating the incremental impact of exposure to multiple digital components.

[0020] Figure 7 is a flow chart of an example process for modifying a transmission standard of a digital component based on an incremental impact of the digital component.

[0021] Figure 8 is a block diagram of an example computer system that may be used to perform the operations described herein. DETAILED DESCRIPTION

[0022] Users connected to the Internet are exposed to a variety of digital content (e.g., search results, web pages, digital components, news articles, social media posts, audio information output by digital assistant devices). Some of these exposures to content may contribute to the user performing a specified target action. For example, a user exposed to a web page about an endangered species may subscribe to (sign up) a newsletter related to helping save the endangered species, where subscribing to the newsletter can be considered a specified target action. Similarly, a user exposed to information about a specific type of mobile device may eventually acquire the specific type of mobile device, where acquiring the mobile device can be considered a target action. Examples of target actions may also include registering / signing up for a service in a website, adding an item to an online shopping cart, downloading a white paper, or acquiring a product.

[0023] It may be difficult to determine how much each exposure of content to a user contributes to the user's performance of a subsequent target action. For example, assume that a user searches for "sports car", views the search results returned in response to the submission of the search query "sports car", is exposed to digital components depicting a specific brand of sports car, and visits various websites that provide information about sports cars. Further assume that the user in this example subsequently submits an online request to obtain information about obtaining the specific brand of sports car. In this example, the amount of contribution of each of these online activities to the user's subsequent submission of a request to obtain information about obtaining the specific brand of sports car is content that cannot be directly observed, and from the perspective of a third party (i.e., someone other than the user), as the time between the online activity and the subsequent user action increases, the contribution amount is difficult to determine specifically. However, the online content distribution system can use any information about the relative contribution of each online activity and content exposure that can be derived to more efficiently and effectively present information relevant to the user to the user, and also allow third parties to understand how the content they produce and distribute affects subsequent user actions.

[0024] This article discusses techniques for determining the relative contributions of different content exposures to a user performing a specified target action. For example, the techniques discussed herein create a model that is able to determine the impact of each exposure to online content as it relates to the user's subsequent performance of a specified target action. In addition, the techniques discussed herein enable the system to determine the incremental contribution of each exposure remaining when the specified target action is performed.

[0025] More specifically, the techniques described herein determine a baseline performance level of a specified target action attributable to a user's organic online activity, and can determine the incremental impact of a user's exposure to digital components injected into an online resource by a third party on the user's performance of the target action.

[0026] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component can be stored electronically as a single file or a collection of files in a physical memory device, and a digital component can take the form of a video file, an audio file, a multimedia file, an image file, or a text file, and include advertising information, such that an advertisement is a type of digital component. As used herein, the phrase digital component refers to a discrete unit of content that is distinct from an organic content exposure, which is referred to as an organic event and is discussed in detail below.

[0027] Figure 1is a block diagram of an example environment 100 in which content is distributed. As described in more detail below, users are exposed to various online content in environment 100, and techniques discussed throughout this document can determine the incremental impact of each of these exposures on the user performing a specified target action.

[0028] Example environment 100 includes network 102, such as a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof. Network 102 connects content server 104, user device 106, digital component server 108, and digital component distribution system 110 (also referred to as component distribution system).

[0029] An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on mobile devices, tablet devices, or desktop computing devices, are also examples of electronic documents. Content server 104 can provide electronic documents to user device 106. For example, content server 104 can include a server that hosts a publisher website. In this example, user device 106 can initiate a request for a given publisher web page, and content server 104 that hosts the given publisher web page can respond to the request by sending machine-executable instructions that initiate presentation of the given web page at client device 106.

[0030] In another example, content server 104 may include an app server from which user device 106 may download the app. In this example, user device 106 may download the files required to install the app at user device 106 and then execute the downloaded app locally. The app may present organic content, e.g., content specified by the developer of the app, and in some cases, may also present one or more digital components (e.g., content created / distributed by a third party) obtained from digital component server 108 and inserted into the app when the app is executed at user device 106.

[0031] An electronic document may include a variety of content. For example, an electronic document may include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. An electronic document may also contain dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document may maintain data sources for populating various portions of an electronic document. In this example, a given electronic document may include a tag or script that causes the user device 106 to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by the user device 106. The user device 106 integrates the content obtained from the data source into a given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0032] In some cases, a given electronic document may include a digital component tag or a digital component script that references the digital component distribution system 110. In these cases, the digital component tag or the digital component script is executed by the client device 106 when the given electronic document is processed by the user device 106. The execution of the digital component tag or the digital component script configures the user device 106 to generate a request for a digital component 112 (referred to as a "component request"), which is transmitted to the digital component distribution system 110 via the network 102. For example, the digital component tag or the digital component script may enable the user device 106 to generate a packet data request including header and payload data. The component request 112 may include event data that specifies characteristics such as the name (or network location) of the server from which the digital component is requested, the name (or network location) of the requesting device (e.g., the user device 106), and / or information that the digital component distribution system 110 may use to select one or more digital components to be provided in response to the request. The component request 112 is transmitted by the user device 106 to a server of the digital component distribution system 110 via the network 102 (e.g., a telecommunications network).

[0033] The component request 112 may include event data specifying other event characteristics, such as the characteristics of the requested electronic document and the location of the electronic document where the digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., a web page) in which the digital component will be presented, available locations of the electronic document that can be used to present the digital component, the size of the available locations, and / or the media types that are suitable for presentation in these locations may be provided to the digital component distribution system 110. Similarly, event data specifying keywords ("document keywords") or entities (e.g., people, places, or things) associated with the electronic document that are referenced by the electronic document may also be included in the component request 112 (e.g., as payload data) and provided to the digital component distribution system 110 to facilitate identification of digital components that are suitable for presentation with the electronic document. Event data may also include search queries submitted from the user device 106 to obtain search results pages, and / or data specifying search results and / or text, audible, or other visual content included in the search results.

[0034] The component request 112 may also include event data related to other information, such as information that has been provided by a user of the user device, geographic information indicating the state or region where the component request is submitted, or other information that provides an environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the type of device at which the digital component will be displayed, such as a mobile device or a tablet device). The component request 112 may be transmitted, for example, over a packet network, and the component request 112 itself may be formatted as packet data having a header and payload data. The header may specify the destination of the packet, and the payload data may include any of the information discussed above.

[0035] A component distribution system 110 including one or more digital component distribution servers selects a digital component to be presented with a given electronic document in response to receiving a component request 112 and / or using information included in the component request 112. In some embodiments, the digital component is selected in less than one second to avoid errors that may be caused by delayed selection of the digital component. For example, a delay in providing a digital component in response to a component request 112 may cause a page loading error at a user device 106, or cause some parts of the electronic document to remain unfilled, even after other parts of the electronic document are presented at a client device 106. In addition, as the delay in providing the digital component to the client device 106 increases, when the digital component is transmitted to the client device 106, it is more likely that the electronic document will no longer be presented at the client device 106, thereby negatively affecting the user's experience of the electronic document. In addition, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided, the delay in providing the digital component may cause the transmission of the digital component to fail.

[0036] To facilitate searching for electronic documents, environment 100 may include a search system 150 that identifies electronic documents by crawling and indexing the electronic documents (e.g., indexing based on the content of the crawled electronic documents). Data about electronic documents may be indexed based on the electronic documents associated with the data. Indexed copies of the electronic documents, and optionally cached copies, are stored in a search index 152 (e.g., hardware memory device(s)). Data associated with an electronic document is data representing content included in the electronic document and / or metadata of the electronic document.

[0037] The user device 106 may submit a search query to the search system 150 via the network 102. In response, the search system 150 accesses the search index 152 to identify electronic documents related to the search query. The search system 150 identifies the electronic documents in the form of search results and returns the search results to the user device 106 in a search results page. Search results are data generated by the search system 150 that identify electronic documents that are responsive to (e.g., related to) a particular search query and include active links (e.g., hypertext links) that cause a client device to request data from a specified location in response to a user interaction with the search results. An example search result may include a web page title, a portion of a text snippet or image extracted from a web page, and a URL of a web page. Another example search result may include a title of a downloadable application, a text snippet describing the downloadable application, an image depicting a user interface of the downloadable application, and / or a URL pointing to a location from which the application can be downloaded to the user device 106. Another example search result may include a title of a streaming media, a text snippet describing the streaming media, an image depicting the content of the streaming media, and / or a URL pointing to a location from which the streaming media can be downloaded to the client device 106. Similar to other electronic documents, the search results page may include one or more slots in which digital components (e.g., advertisements, video clips, audio clips, images, or other digital components) may be presented. As described above, when a user interacts with an online resource available through the Internet, various online content may be presented to the user. The online content may generally be classified as organic content or digital components. Organic content is content specified and / or provided by the owner or administrator of the online resource in which the content is presented. Examples of organic content include search results provided by a search engine and content presented in a web page provided by a publisher. In each of these examples, the content presented is specified by the entity providing the online resource and is therefore considered to be first-party content. For example, a search engine identifies online resources relevant to a submitted search query, generates search results identifying these relevant resources, and generates a search results page (in the domain of the search engine), which includes search results generated by the search engine. Therefore, the search results are generated by the search engine and presented in a search results page generated by the search engine, thereby making the search results first-party content and therefore organic content. Similarly, when a user visits a particular web page, that web page will include content that is specified and / or generated by the publisher of that web page, which is also considered first-party content, also making it organic content for the purposes of this discussion.

[0038] For the purpose of discussion, digital components are considered third-party content because the digital components are created and / or provided by an entity different from the entity that provides the online resource on which the digital components are presented. In the context of a search results page, a digital component that includes third-party content can be a digital component (e.g., weather data, stock data, or advertisements) that is selected for inclusion in an online resource when presenting the online resource. For example, when generating a search results page, a third party (e.g., a domain different from the search engine domain) can select a digital component (e.g., presenting current weather conditions, stock prices, or advertisements) and provide the digital component for presentation in the search results page. As described above, a digital component presented with the search results page can be selected by an entity other than the entity that provides the search results page based at least in part on a search query submitted by a user. In the context of a web page provided by a publisher (e.g., a blog, a news web page, a weather web page, a stock information web page), when a client device requests a web page, a digital component provided by a third party different from the publisher of the web page can be selected to be presented in the web page. For example, a digital component selected for presentation with a given web page can be selected based on the organic content of the given web page and / or the characteristics of the user accessing the given web page (e.g., interests, profile information, etc.).

[0039] Each exposure to content may have an impact on a user's future online (or offline) activities. For example, a user who sees content (e.g., a review, news article, or advertisement) related to a particular brand of shoes may be more likely to acquire that particular brand of shoes than if the user had not experienced that exposure. In some cases, it may be advantageous to be able to quantify the impact of different content exposures because they are associated with the user's subsequent performance of some specified target action.

[0040] In some embodiments, the target action may be specified by the digital component provider. For example, the digital component provider specifies the target action as one or more of the following: the user downloads a white paper, navigates to at least a given depth of a website, views at least a certain number of web pages, spends at least a predetermined amount of time on a website or web page, completes a website registration process, subscribes to a digital service, adds an item to a shopping cart, or purchases a product. When the user performs the specified target action, the execution of the specified target action may be referred to as a conversion.

[0041] There is typically a series of exposures to online content before the user performs a specified target action. For example, assume that user 160 is interested in a particular camera and wants to learn more about the camera. Further assume that a digital component provider that distributes digital components containing information about the particular camera has specified the target action as acquisition of the particular camera.

[0042] In this example, user 160 may search for information about a particular camera on user device 106 by submitting a search query to search system 150 via network 102. Search system 150 identifies search results in response to the search query and returns the search results to user device 106 for display, which is considered exposure to organic content about the particular camera to user 160 (e.g., assuming that a digital component about the particular camera is not presented on the search results page). User 160 viewing the search results at user device 106 may visit websites 180, 182, and 184 (e.g., by clicking on several search results), each of which contains information about the particular camera. Each of these visits to websites by user 160 may also be considered exposure to organic content to user 160. Assuming that user 160 ultimately acquires the particular camera (i.e., performs a specified target action), each of these exposures to organic content (referred to as organic events) will contribute to the user performing the specified target action, and the relative contributions of these organic exposures may be quantified as described in more detail below.

[0043] In the above example, assume that before performing the specified target action, user 160 was exposed to organic content about a particular camera, but user 160 was not exposed to digital components (e.g., digital components of a specified type) about the particular camera. Exposure to digital components may also contribute to the user's performance of the specified target action. For example, assume that before performing the specified target action, user 160 performs another search. Further assume that, in response to the search, search system 150 returns a search results page 186 that includes search results (e.g., SR1 and SR2), and digital component server 108 provides digital components 188 about the particular camera for presentation with the search results.

[0044] In this example, when user 160 subsequently performs a specified target action (e.g., acquires a particular camera), the user's exposure to digital component 188 will also contribute to the performance of the specified target action. However, when digital component exposure is associated with the user's subsequent performance of the specified target action, it is not easy to see, cannot be directly observed, and is difficult to determine the level of influence of digital component exposure. In addition, when organic exposure and exposure to digital components are associated with influencing the user's subsequent performance of the target action, it is not easy to clearly detail the contribution of organic exposure and the contribution of exposure to digital components from only the raw data associated with content exposure. Therefore, it may be difficult to effectively and efficiently distribute content to users, especially when it is associated with digital components.

[0045] To determine the impact of content exposure when the content exposure is associated with the user performing a specified target action, the environment 100 may include a model device 130 that is configured to evaluate the content exposures and determine the level of contribution of each of these content exposures to the user's subsequent performance of the specified target action. This information can be used to determine the performance of digital components of a specified type distributed by the component distribution system 110, which can be used to improve the relevance of content presented to the user, for example, by modifying the transmission criteria that controls when, where, or how the digital components are transmitted for presentation to the user.

[0046] As described in more detail below, the model device 130 is configured to implement data collection techniques that enable the model device 130 to learn relationships between attributes of a user and baseline execution levels of specified target actions (e.g., the level at which users with certain attributes perform specified target interactions). These relationships can be referred to as baseline action models, which can output baseline execution levels based on the attributes input to the system. The baseline execution model can be a stand-alone model, or can be incorporated into a more complex model structure that also considers other data, which is described in more detail below.

[0047] The baseline performance level represents the performance level of the specified target action without the user being exposed to the specified type of digital component. For example, a specific baseline performance level can be created to represent the rate at which users acquire a specific type of shoes when they are not exposed to the digital component distributed for the seller of the specific type of shoes. In a specific example, the baseline performance metric can indicate the portion of users with a specific set of attributes that will acquire a specific type of shoes without the users being exposed to advertisements for the specific type of shoes.

[0048] The data collection techniques implemented by the model device 130 also enable the model device 130 to model the impact of various content exposures over time as they relate to users' subsequent performance of a specified target action. For example, the model device 130 can create a model that quantifies an initial change in the portion of users who perform a specified target action immediately after being exposed to a particular type of content (e.g., organic content or a digital component), and the model represents the decay of the initial change over time (e.g., toward a baseline performance metric), which is described in more detail below. This ability to detail between a baseline performance level and the residual impact of various content exposures over time enables the model device 130 to determine the incremental impact of each content exposure remaining when the specified target action is performed, thereby providing an improved attribution model relative to traditional attribution models, which are unable to detail the relative contribution of each content exposure remaining when the specified target action is performed.

[0049] Figure 2 is an example graph 200 illustrating an example effect of a content exposure digital component on the performance of a target action over time. Figure 2 , at time t0, the execution level is PL1, which can be assumed to be the baseline execution level of a group of users with a given set of attributes. The execution level can be represented in a variety of ways. In some embodiments, the execution level at any given time represents the user's action level, for example, the portion of the specified user who will perform the specified target action, or the frequency with which the user performs the specified target action. In some embodiments, the execution level can be represented as the probability that the user will perform the specified target action. In some cases, the execution level is standardized to a bounded numerical range (e.g., 0-1, 0-10, 0-100, etc.).

[0050] At time t1, content exposure occurs ("Exposure A" 210), which causes an immediate increase in the execution level caused by the content exposure. Specifically, at exposure A, the execution level changes from PL1 to PL2. This change in execution level indicates that as a result of exposure A 210, the level at which the user performs the specified target action changes from PL1 to PL2. For the purpose of example, assume that exposure A 210 indicates that the user is exposed to search results for a specific type of shoes. In this example, Figure 1 The modeling device 130 indicates that exposure to the search results leads to an increase in the performance of the target action (e.g., the acquisition of a particular type of shoe).

[0051] After the initial change in execution level at t1, the effect of exposure A 210 on execution level begins to decay and returns to PL1, as shown by the curve between t1 and t2 in graph 200. This decay represents the fact that, over time, the contribution of a particular content exposure to the user's subsequent execution of the target action tends to decrease. The decay rate for each particular type of content exposure can be determined based on data collected by model device 130, which is discussed in more detail below.

[0052] exist Figure 2 In the example shown, at time t2, the execution level has returned to PL1, which is the baseline execution level, indicating that at t2, exposure A 210 is no longer considered to contribute to performing the target action. At time t2, a second exposure ("exposure B" 220) occurs, which changes the execution level from PL1 to PL3, indicating that exposure B caused the user to perform the target action at a higher level relative to the baseline level (e.g., PL1) at which the user performs the target action. After t2, the increased execution level contributed by exposure B 220 begins to decay according to a decay function determined for a particular type of content exposure. Continuing with the example above, exposure B 220 can be, for example, a user exposure to a digital component created by a manufacturer of a particular type of shoe.

[0053] At t3, the contribution of exposure B 220 to the execution level has dropped below PL3 to PL4, but has not been dropped back to PL1, thus indicating that at t3, exposure B 220 is still considered to contribute to the user's execution of the target action. Therefore, the merit of performing the target action at time t3 can be given to exposure B 220, which is discussed in more detail below. In this example, another exposure ("exposure C" 230) occurs, which changes the execution level again, and specifically increases the execution level to PL5. Exposure C 230 can be an organic exposure or an exposure of a digital component. In both cases, the initial level of the execution increase caused by exposure C 230 will be the difference between PL4 and PL5, that is, the difference between the execution level (e.g., PL4) remaining immediately before exposure C 230 and the execution level (e.g., PL5) as a result immediately after exposure C. As shown, PL5 includes the contribution from exposure B 220 and exposure C 230 that have not yet fully decayed. Therefore, correct attribution of their respective contributions to the execution of the target action at t3 will require determining the respective incremental levels of execution that combine to provide execution level PL5. The determination of the incremental level of execution of a content exposure at any given time is discussed in more detail below.

[0054] After t3, the execution level begins to decay (e.g., decrease) again, but this time, the decay is not only due to the decay that has been determined for exposure C 230, but also due to the decay of the execution level of exposure B 220 remaining at t3, which is discussed in more detail below. For example, assume that the target action is performed by the user at t4 after t3. In this example, at t4, the execution level has returned to PL4, which is higher than the baseline execution level PL1. Therefore, one or more content exposures that occurred between t0 and t3 are still considered to contribute to the execution of the target action 240 at t4. For example, the relative contribution of each content exposure remaining at t4 can be determined based on the change in execution level caused by the content exposure, the decay function of the content exposure, and the amount of time elapsed from the content exposure and the execution of the target action 240, which is discussed in more detail below.

[0055] The above discussion illustrates how the execution level of a target action can change based on content exposure, and how this can change the overall execution level. A mathematical relationship can be used to represent and / or quantify the overall execution level at any given time. In some embodiments, the execution level of a given action is represented by the following formula:

[0056] X(t)=X I (t)+X S (t)

[0057] Where X(t) represents the total execution level at time t, X I(t) is the baseline execution level without any exposure to the content, and X s (t) is the improvement in execution level due to exposure to the content. In some embodiments, the improvement in execution level due to exposure to the content is modeled as:

[0058] X s (t) = a S X S (t-Δt)+b S U S (t)

[0059] Among them, a S is the decay rate of the execution level, b S is an immediate increase in the execution level, and U S (t) is the exposure to the digital component at time t. Historical data can be used to determine a S and b S The values ​​of historical data such as the time when the specified content was exposed and the time when the target action occurred after the exposure of the content.

[0060] Figure 3 3 is a graph 300 illustrating an example decay of the effect of content exposure 330 over time. For example, assume that user 160 performs an Internet search and is exposed to search results, as shown by exposure 330, which results in an increase in the performance level. As described above, this increased performance level will decay over time, for example, according to a decay function created for the search result exposure.

[0061] In some embodiments, the decay of the execution level is modeled using a piecewise constant function (or another piecewise function, such as a piecewise linear function). For example, the decay of the execution level during time interval 350 is considered as short-term decay, the decay of the execution level during time interval 360 is considered as medium-term decay, and the decay of the execution level during time interval 370 is considered as long-term decay. In other embodiments, the piecewise constant function can decompose the decay process into finer time intervals, thereby modeling the decay at a finer level. In other embodiments, other methods can be used to estimate the continuous decay function. Using a piecewise constant function can reduce the computational resources required to determine the decay and enable the decay to be calculated more quickly.

[0062] In such an embodiment, the time after content exposure is modeled as the sum of smaller time intervals. For example, in order to model the decay of the execution level between time t1 and t4, the time between t1 and t4 is modeled as the sum of time intervals t1-t2, t2-t3, and t3-t4. It is assumed that each time interval has a corresponding decay, or more specifically, a corresponding decay rate. Based on the decay rate of the corresponding time interval, the execution level before the specific time interval can be used to calculate the execution level after the specific time interval. In some embodiments, the decay rate can be set by the digital component provider or any third party interested in generating a baseline model. In some embodiments, the decay rate can be determined by other machine learning models based on the user's past activities using multiple learnable parameters. In other embodiments, the decay rate can be a parameter of the baseline model learned during the training process.

[0063] It should be noted that the performance level after exposure to the content is determined by the baseline model. In some embodiments, the baseline model can determine that the performance level after exposure to specific content is likely to remain the same or increase based on the learned parameters of the baseline model, the specific content, the user, and the training data collected to train the baseline model.

[0064] Figure 4 is a block diagram of an example system 400 configured to create a model 450 that a model device 130 can use to attribute the performance of a target action to content exposures that occur over time. In a simple form, the model can be a baseline model that accepts user attributes as input and outputs a baseline execution level based on the user attributes. The model can be extended to accept inputs related to organic exposures that occurred before the target action was performed (e.g., the types of organic exposures and the timing of these exposures) and output various information, such as the relative contribution of each organic exposure to the performance of the target action when the target action occurred. The model can also be extended to accept inputs related to digital component exposures that occurred before the target action was performed (e.g., the type of digital component and the exposure timing) and output information specifying the relative contribution of each digital component to the performance of the target action when the target action occurred. The following discussion relates to the creation of a baseline model.

[0065] System 400 includes a data collector 410 that obtains / prepares training samples for training model 450. For example, model 450 may be trained using data collected for a group of users who performed a target action and another group of users who did not perform the target action. For example, assume that the target action is that the user downloads / installs a native application. In this example, the system may identify a previous download / installation of the native application and obtain data on content exposure that a specified user experienced before downloading / installing the native application. The system may also obtain data on content exposure that a specified user experienced after the user did not download / install the native application.

[0066] The data collected in such an embodiment includes user data representing exposure to all digital components before performing a target action (e.g., application download / installation). In some embodiments, model 450 learns the relationship between user attributes and execution levels based on exposure to various digital components. In other embodiments, model 450 is configured to learn the relationship between user attributes and execution levels independently of exposure to a specified type of digital component. For example, model 450 can learn the relationship between user attributes and execution levels based on historical exposure data of users corresponding to users who were not exposed to one or more specified types of digital components (e.g., digital components provided by a specific entity) before the user performed the target action. This allows model 450 to determine the baseline execution level of the user based on the user's attributes. In such an embodiment, ablation experiments are used to generate training samples, which allow the collection of event data (e.g., organic exposure data) representing user exposure to organic content while the user is not exposed to third-party digital components, and allow the collection of training samples including third-party exposure data representing user exposure to digital components provided by a third party. In other embodiments, training samples generated from users who have not been exposed to one or more specified types of digital components can be used to train model 450 without explicitly performing ablation experiments. For example, search system 150 and component distribution system 110 may record all search requests from user 160 and the digital components presented to user 160. Using this data, a training sample may be generated representing users who were not presented with one or more specified types of digital components and therefore must be exposed to specified digital components.

[0067] Figure 5 It shows that it can be Figure 4A block diagram of an ablation experiment implemented by the ablation experiment device 415 of the embodiment. The ablation experiment is performed using a group of users 510. In such an embodiment, a group of control users 520 is created for a specified period of time, during which one or more specified types of digital components are not presented to the group of control users 520, thereby preventing exposure to the digital components being evaluated. For example, the group of control users 520 can be marked as being part of a control group of a specific entity that distributes digital components, and in this example, including users in the control group can prevent these users from being exposed to digital components distributed by the specific entity, thereby preventing these digital components from affecting subsequent actions performed by users in the control group 520.

[0068] The ablation experiment also defines a group of exposed users 550 within a specified time period. Unlike the group of control users, the group of exposed users 550 is presented with a specified type of digital component, thereby ensuring that the group of exposed users 550 is exposed to a specified type of digital content (e.g., a digital component distributed by a specific entity). For example, the group of users 510 includes the group of control users 520, which includes users 1-6. The group of users 510 also includes a group of exposed users 550, which includes users 7-12. Each user 1-6 in the control user group can experience organic exposure, but will not experience third-party exposure to a specified type of digital component (e.g., a digital component provided by a specific entity and / or related to a specific topic, object, product, or service). Users 7-12 in the group of exposed users 550 are exposed to a specified type of digital component 565 (e.g., a digital component provided by a specific entity).

[0069] In some embodiments, the users in the group of control users 520 and the group of exposed users 550 are randomly selected from the users in the group 510. In other embodiments, certain conditions can be used to control the random selection process. For example, a baseline model can be trained to determine the performance level of users in a specific age group. In such an embodiment, the ablation experiment device 415 can select users from a specific age group in the group of users 510, and then randomly assign the selected users to the group of control users 520 or the group of exposed users 550.

[0070] In some embodiments, exposure data (e.g., organic exposure data and / or third-party exposure data) is collected for the group of control users and the group of exposed users. For example, the exposure data collected for user 4 in the group of control users 520 may specify that user 4 was exposed to a first website 532 and a second website 534, both of which are organic exposures. According to the collected data, after being exposed to the first website 532 and the second website 534, user 4 performed a specified target action 538. Continuing with this example, the exposure data may also indicate that during a specified time period, user 6 did not perform a specified target action 538 after being exposed to the first website 532 and the third website 536. The exposure data may be used in a training sample that includes an identifier of the exposure type (e.g., organic or third-party), the time of the exposure, the duration of the exposure, user attributes, and whether the specified target action was performed within the specified time period.

[0071] Similarly, from the set of exposed users 550, user 10 performs a specified target action 538 after initial exposure to first website 532 and subsequent third-party exposure 565 (e.g., exposure to a digital component provided by a particular entity and / or related to a particular topic, service, or product). In this example, user 12 does not perform a specified target action 538 after initial exposure to second website 534 and subsequent third-party exposure 565.

[0072] return Figure 4 After collecting exposure data (e.g., organic exposure data and third-party exposure data) as described above, the exposure data is processed to generate training samples for training model 450. For example, preprocessor 420 generates each training sample based at least in part on (i) a user descriptor or identifier, (ii) a timestamp of the exposure, (ii) an event descriptor or exposure descriptor, (iv) a timestamp of the next chronological exposure, (v) a count of customers performing a specified target action that occurred within an interval defined by the time of the exposure and the next exposure, and (vi) any other suitable features. In some embodiments, multiple processors can be used to train model 450. For example, multiple workers 430, 432, and 434 are used to train model 450.

[0073] In some embodiments, the model 450 is designed using a model in which the parameters of the model characterize the expected increase in the number of actions of the user over a given time interval. This is referred to herein as the intensity rate (λ). In such embodiments, the model may be defined as follows:

[0074]

[0075] Among them, Y i(t) is a counting process, which represents the execution of the target action by counting the number of conversions of the user within time t, where X is a feature set.

[0076] In some embodiments, the intensity at time t is modeled as a function of a set of time-dependent features and is assumed to be piecewise constant over each interval. In such an embodiment, the logarithm of the average intensity rate for each exposure interval of the user is modeled as

[0077] log(λ ij (t|X ij (t) = x ij (t)) = log(λ 0 (t))+β T .x ij (t)+η i

[0078] Among them, t∈[t j ,t j+1 ), i is the index of the user, j is the exposure that does not lead to the user performing the target action, η i is the user-level random effect of user i, and t j is the start time of exposure j.

[0079] In some embodiments, each interval [t j ,t j+1 ) will take the following form:

[0080]

[0081] Among them, Δt ij =t j+1 -t j And ij is the segmented increment of the number of specified target actions of the user over a given time interval. In some embodiments, the baseline increment λ of the number of specified target actions 0 (t) may be constant. In other embodiments, a piecewise step function is used to estimate λ 0 (t). In such an embodiment, the above model can be estimated using Poisson regression, where the offset is given by the logarithmic length of time in each interval.

[0082] In some embodiments, the model 450 is validated using simulation results, real-world data, and cross-validation. For example, the validator 460 can simulate user exposure to digital components and performance of specified target actions using known values ​​of the learnable parameters of the model 450. These known parameters are compared to the learned parameters to measure the goodness of the training process. In another embodiment, the model predictions can be compared to real-world data representing exposure and specified target actions for model accuracy.

[0083] In some embodiments, parties such as providers of particular digital components may be interested in the effectiveness of particular digital components. For example, the provider may wish to measure the impact of exposure to a particular digital component on a user's subsequent performance of a target action. In such embodiments, the difference between the contribution of the digital component exposure to the total level of performance determined for the user and the determined attenuation of the digital component exposure corresponds to the effectiveness (e.g., incremental impact) of exposure to the particular digital component in causing performance of the target action.

[0084] Fig. 6A 600 is a graph showing the incremental impact of exposure to a single digital component on the execution of a target action. In this example, it is assumed that the user experiences digital component exposure 660 at time t and then performs a specified target action at time t+1. As shown, before time t, the user's execution level is initially at a baseline execution level 630 determined based on the user's attributes. At time t, the user experiences digital component exposure 660, which leads to an increase in the execution level. Between time t and time t+1 when the user performs the specified target action, the execution level attributable to digital component exposure 660 decays according to a decay function corresponding to a specific type of digital component to which the user is exposed at time t. As seen in graph 600, at time t+1, the remaining execution level 650 includes contributions from the baseline execution level 630 and the incremental execution level 640 attributable to the digital component exposure at time t (e.g., their sum). The incremental execution level 640 generally represents the impact of digital component exposure 660 on the execution of a specified target action at time t+1, which can be regarded as a measure of the effectiveness of digital component exposure 660 in causing the user to perform a specified target action.

[0085] As previously described, performance of a specified target action can be attributed to multiple different exposures (e.g., organic exposures and / or third-party exposures), as well as a baseline execution level determined for the user. As described above, the baseline execution level represents the user's tendency to perform the specified target action in the absence of exposure to the digital component and / or organic exposure. Each component exposure can also contribute incrementally to the execution level that existed at the time of the target action. Therefore, the execution level when performing the target action can be attributable to the baseline execution level determined for the user, and the incremental execution level contribution remaining for each exposure at the time of the target action. In the event that a user experiences multiple different content exposures, an incremental amount of execution level can be assigned to each exposure based on the difference between the initial execution level contribution provided by the exposure and the amount of decay that occurs between the time of that exposure and the time when the target action occurs.

[0086] Figure 6B is a graph 665 showing the incremental impact of exposure to multiple digital components. Initially, the execution level is at a baseline execution level 690 determined using user attributes. When the user experiences exposure A 670, the execution level rises and then begins to decay over time. The user then experiences exposure B 675, which causes the execution level to rise and then decay again over time.

[0087] At time t after exposure B 675, the user performs the target action (e.g., conversion). In this example, the execution level decays to a final execution level 695. In this example, the final execution level 695 includes incremental contributions from both exposure 670 and exposure 675. For illustrative purposes, regarding the remaining incremental impact of these two exposures, it is assumed that exposure B 675 did not occur, but the target action still occurred at time t. In this example, in the absence of exposure B 675, the remaining execution level at time t will decay to 697. In this case, the incremental execution 680 attributed to exposure B 675 is the difference between execution levels 695 and 697, so that a portion of the merit of exposure B 675 for performing the target action can be given based on the difference between execution levels 695 and 697.

[0088] Continuing with the example, the incremental contribution 685 of exposure A 670 at time t may be accounted for by considering the difference between the baseline execution level 690 and the execution level 697 that would have existed if exposure A 670 had occurred but exposure B 675 had not occurred. For example, the portion of the incremental execution level 685 attributable to exposure A 670, and therefore the merit for performing the target action, is the portion of the execution level at time t that exceeds the baseline execution level 690 that existed for the user independent of any exposure to the digital component (or other content).

[0089] In some embodiments, attributing incremental execution to content exposure can be based on an intensity rate. In such embodiments, the incremental execution is based on the difference between the intensity rates before and after exposure. In such embodiments, the intensity at which the user performs a given target action can be expressed as:

[0090]

[0091] where $ refers to executing the specified target action, S and L are short-term and long-term constants, k is the exposure type, and i k is a counter of the exposure type. In such an embodiment, λ $ It can be expressed as:

[0092] λ $ =λ 0 ×λ 1 ×…×λ j ×…×λ n

[0093] Where j = 1, 2, ..., n represents the jth exposure among the total n exposures. Then, the incremental execution can be calculated sequentially using the following formula:

[0094]

[0095] Where j = 0 and

[0096]

[0097] In some embodiments, one or more entities may be interested in knowing which digital components to present to a user to influence the user's execution level for a specified target action. In such embodiments, a model representing the relationship between content exposure and the execution level of a specified action can be used to determine which digital component to present to a user to increase the likelihood that the user will perform the target action. For example, user 160 uses user device 106 to perform an online search for a specific product. Search system 150 provides digital components containing information about a specific product while providing search results to user 160 through user device 106. In some embodiments, model device 130 uses model 450 to determine which digital components will be displayed with the search results so that the execution level associated with obtaining the product increases. In some embodiments, when it is inferred that the digital component may not result in a significant increase in the execution level after exposure, model device 130 may determine based on user attributes not to present certain digital components to user 160.

[0098] Figure 7Flowchart of an example process for modifying the transmission standard of a digital component based on the incremental impact of the digital component. The operations of process 700 may be performed by one or more data processing devices or computing devices (such as the model device 130 discussed above). The operations of process 700 may also be implemented as instructions stored on a non-transitory computer-readable medium. The execution of the instructions may cause one or more data processing devices or computing devices to perform the operations of process 700. The operations of process 700 may also be implemented by a system comprising one or more data processing devices or computing devices, and a memory device storing instructions that cause one or more data processing devices or computing devices to perform the operations of process 700. As discussed in more detail below, process 700 builds and utilizes a model that can provide a user's baseline performance level and the relative contribution of each content exposure to the user's performance of a specified target action.

[0099] An ablation experiment is performed to obtain control result data and exposure result data (710). The ablation experiment is performed by dividing users into a group of control users and a group of exposed users. The group of control users are those users who will not be exposed to the specific type of digital component being analyzed. For example, assume that data is being collected to determine the relative impact of exposure to digital components (e.g., advertisements) distributed by the manufacturer of a specific camera. In this example, the ablation experiment can utilize a flag to specify that each user in the group of control users is not eligible to receive digital components for a specific camera distributed by the manufacturer of the specific camera. In some cases, the specific type of digital component can be limited based on the media type (or other appropriate characteristics). For example, the group of control users can be prevented from seeing video digital components provided by the manufacturer of the digital camera, but can be eligible for exposure to non-video digital components.

[0100] Ablation experiments enable the system to obtain a set of control results for a group of control users who are not exposed to a specific set of digital components. In some embodiments, the control results specify, for each specific control user in the group of control users, whether the specific control user performed a specified action. For example, historical online activity data for each control user in the group of control users can be evaluated to determine whether the control user performed a specified target action, the activities performed by the control user (e.g., submitting a search query), and the content exposure experienced by the control user. This information can be processed and / or used as a training example for creating a model that can determine the incremental contribution of each content exposure relative to the user's baseline execution level. More specifically, the control results for the control users and the corresponding data collected can be used to learn the baseline execution level of users with a specific set of characteristics and / or the impact of organic exposure on the execution level, which is independent of any impact of a specified type of digital component on the control user's action because the control user will not be exposed to a specific type of digital component.

[0101] The ablation experiment can also enable the system to collect exposure results for a set of exposed users that are not included in the set of control users. In some embodiments, the exposure results specify, for each exposed user in the set of exposed users, whether the exposed user performed a specified action. This information can be processed and / or used as training examples for creating a model that can determine the incremental contribution of each third-party content exposure relative to the user's baseline execution level.

[0102] A model is created based on the control result data and the exposure result data (720). The model can be created (e.g., trained) based on the data collected during the ablation experiment and using the mathematical relationships discussed throughout this article. For example, a machine learning framework can be used to create a model that uses the user's attributes, the control results for each specific control user, and the exposure results for each exposed user. In some embodiments, the model represents the relationship between user attributes, content exposure, and the execution level of a specified target action. A model can be generated to accept a set of user attributes for a specific user, a set of content exposure (e.g., organic exposure and / or third-party exposure), the time when the content exposure occurred, and the time when the specific user performed the specified target action as input. A model can be generated to output information including, for example, a baseline execution level for a specific user based on user attributes, and an incremental contribution of content exposure experienced by a specific user before performing a specified target action.

[0103] Organic exposure data for a particular user who performed a specified target action is obtained (730). In some embodiments, the organic exposure data specifies one or more organic exposures experienced by the particular user that resulted in (i.e., prior to) the particular user performing the specified target action within a specified time period. The organic exposure is neither exposure to a digital component of a specified type (e.g., third-party exposure to a digital component that is not provided by a third party) nor the performance of the specified target action. An example of an organic exposure is exposure to search results generated by a search engine and presented to the particular user in response to the particular user submitting a search query. Another example organic exposure is exposure to content from a web page to which the user navigates, independent of exposure to or interaction with a third-party digital component. For example, a particular user may manually enter a web page address in a browser to navigate to the web page. This would be considered an organic exposure to the content.

[0104] Obtain third-party exposure data for a specific user who performed a specified target action (740). In some embodiments, the third-party exposure data specifies third-party exposure of a specified type of digital component to the specific user when (i.e., before) the specific user performed the specified target action within a specified time period. The third-party exposure data may include an exposure time for each third-party exposure, the exposure time specifying when the third-party exposure occurred. Examples of third-party exposures include content provided by a third party that is presented in any search result page, a web page provided by a publisher different from the third party, or a native application provided by an app developer different from the publisher.

[0105] An incremental execution level attributable to each third-party exposure is determined using a model (750). For example, organic exposure data and third-party exposure data may be input into the model, and the model may output an incremental execution level attributable to each third-party exposure. The incremental execution level may be determined at the action time when a particular user performs a specified target action. The incremental execution level may be determined as the residual amount of the execution level contribution from the third-party exposure remaining at the action time. For example, for each third-party exposure, the difference between the exposure time of the third-party exposure and the action time when the specified target action occurs may be determined. This difference between the exposure time and the action time may be input into a decay function of the third-party exposure, which decays the amount by which the execution level contribution provided by the specified third-party exposure decays between the exposure time and the action time.

[0106] The amount of contribution remaining at the action time (e.g., by subtracting the amount of decay experienced between the exposure time and the action time from the initial performance level contribution provided by the third-party exposure) is considered to be the residual amount of the performance level contribution from the third-party exposure remaining at the action time that can be attributed to the third-party exposure. A different decay function can be determined for each different type of third-party exposure based on data obtained during the ablation experiment, and each decay function will specify a decay rate at which the performance level contribution changes over time.

[0107] The model uses both organic exposure data and third-party exposure data so that the contribution of organic exposure can be removed from the total execution level remaining at the action time. Similarly, the model considers (e.g., removes) the contribution of the baseline execution level from the total execution level remaining at the action time, thereby isolating the contribution of third-party exposure.

[0108] Based on the incremental execution level attributed to the exposure of the digital component, the transmission criteria of at least some digital components are modified (760). In some cases, when the incremental execution level attributed to the exposure of the digital component is high (e.g., higher than other available digital components), the transmission criteria of the digital component can be modified to increase the frequency of transmitting the digital component to the user. In some cases, when the incremental execution level attributed to the exposure of the digital component is low (e.g., lower than other available digital components), the transmission criteria of the digital component can be modified to reduce the frequency of transmitting the digital component to the user. In some embodiments, the transmission criteria of a specific digital component can be adjusted in proportion to the size of the incremental execution level attributed to the third-party exposure of the specific digital component. In some embodiments, when the incremental execution level of the third-party exposure caused by the transmission of the digital component based on the specific transmission criteria is less than a specified size, the specific transmission criteria (e.g., keyword) can be disabled. For example, assume that the keyword "boot" is used to distribute digital components of a specific third-party content provider, and the incremental execution level attributed to the third-party exposure triggered by the keyword boot is less than a specified minimum acceptable level. In this example, the keyword boot can be disabled so that it will not trigger the distribution of the digital component of the third-party content provider.

[0109] Figure 8 8 is a block diagram of an example computer system 800 that can be used to perform the above operations. System 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. Each of components 810, 820, 830, and 840 can be interconnected, for example, using a system bus 850. Processor 810 is capable of processing instructions for execution within system 800. In one embodiment, processor 810 is a single-threaded processor. In another embodiment, processor 810 is a multi-threaded processor. Processor 810 is capable of processing instructions stored in memory 820 or on storage device 830.

[0110] The memory 820 stores information within the system 800. In one embodiment, the memory 820 is a computer readable medium. In one embodiment, the memory 820 is a volatile memory unit. In another embodiment, the memory 820 is a non-volatile memory unit.

[0111] The storage device 830 can provide mass storage for the system 800. In one embodiment, the storage device 830 is a computer-readable medium. In various embodiments, the storage device 830 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0112] The input / output device 840 provides input / output operations for the system 800. In one embodiment, the input / output device 840 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 370. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top television client device, etc.

[0113] Despite Figure 8 An example processing system is described in the specification, but the embodiments and functional operations of the subject matter described in this specification may be implemented in other types of digital electronic circuitry or computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them.

[0114] An electronic document (referred to simply as a document for simplicity) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0115] The subject matter and the embodiments of the operation described in this specification can be implemented in digital electronic circuits, or in computer software, firmware or hardware or a combination of one or more of them, including the structures disclosed in this specification and their equivalent structures. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium (or multiple media) to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium can be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium can be the source or destination of the computer program instructions encoded with an artificially generated propagation signal. The computer storage medium can also be one or more separate physical components or media (for example, multiple CDs, disks or other storage devices) or included therein.

[0116] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0117] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may include dedicated logic circuits, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0118] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including in the form of a stand-alone program or in a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to the target, or in multiple collaborative files (e.g., files storing some portions of one or more modules, subroutines, or codes). A computer program may be deployed to execute on one or more computers that are located at one site or distributed between multiple sites and connected by a communication network.

[0119] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the device can be implemented as the dedicated logic circuit.

[0120] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. In general, a computer will include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or can be operatively connected to one or more mass storage devices to receive data from it or transfer data to it or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few). Devices suitable for storing computer program instructions and data can include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0121] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to a user, and a keyboard and pointing device through which the user may provide input to the computer, such as a mouse or trackball. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending files to and receiving files from a device used by the user; for example, by sending a web page to a browser on a user's client device in response to a request received from the browser.

[0122] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface and a browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0123] A computing system may include a client and a server. In general, the client and the server are remote from each other and usually interact through a communication network. The relationship between the client and the server is formed by computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from it). Data generated at the client device (e.g., the result of the user interaction) can be received from the client device at the server.

[0124] Although this specification contains many specific implementation details, these details should not be interpreted as limiting any invention or the scope of the claim, but should be interpreted as a description of the features of a particular embodiment specific to a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in a single embodiment in combination. On the contrary, the various features described in the case of a single embodiment may also be implemented in multiple independent embodiments or in any suitable sub-combination. In addition, although features may be described as working in certain combinations as described above and initially requested as such, in some cases, one or more features from the requested combination may be excluded from the combination, and the requested combination may involve a sub-combination or a variant of a sub-combination.

[0125] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order or requiring all illustrated operations to be performed to obtain desirable results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, but should be understood that the described program components and systems can generally be integrated together or packaged into multiple software products in a single software product.

[0126] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method comprising: creating, by one or more computing devices, a model representing relationships between user attributes, content exposure, and performance levels of specified target actions; obtaining content exposure data specifying content exposure experienced by a particular user; determining, using the model and the content exposure data, an incremental execution level attributable to each content exposure at the time of the action when the specific user performed the specified target action; as well as Modifying, by the one or more computing devices, a transmission standard of at least some digital components to which the particular user is exposed based on the incremental execution level.

2. The method according to claim 1, further comprising: performing an ablation experiment to obtain a set of control results for a set of control users that were not exposed to the set of specific digital components, the control results specifying, for each specific control user in the set of control users, whether the specific control user performed a specified action; Exposure results for a group of exposed users not included in the group of control users are collected, the exposure results specifying, for each exposed user in the group of exposed users, whether the exposed user has performed a specified action.

3. The method according to claim 2, wherein: Creating the model includes utilizing a machine learning framework to create the model using user attributes, control results for each specific control user, and exposure results for each exposed user.

4. The method according to claim 1, wherein: Determining the incremental execution level includes: For each third-party exposure in the Content Exposure: determining a difference between an exposure time of the third party exposure and an action time when the specified target action occurs; determining a residual amount of the performance level contribution from the third-party exposure remaining at the action time based on a difference between an exposure time of the third-party exposure and an action time when the specified target action occurs; and A residual amount of the performance level is attributed to the third party exposure.

5. The method according to claim 4, further comprising: For each different type of third-party exposure, determining a decay function that specifies a decay rate of the remaining execution level contribution over time; as well as For each third-party exposure, a residual amount of the performance level contribution from the third-party exposure remaining at the action time is determined based on the decay function and a difference between the exposure time of the third-party exposure and the action time.

6. The method according to claim 4, wherein: Modifying the transmission criteria of at least some of the digital components includes adjusting the transmission criteria of the particular digital components in proportion to the magnitude of the incremental performance level attributable to the third party exposure of the particular digital components.

7. The method according to claim 6, wherein: Adjusting the transmission standard includes disabling a particular transmission standard having a specified size that is less than an incremental execution level of third party exposure attributable to the particular transmission standard.

8. A system comprising: a data store storing one or more evaluation rules; as well as One or more data processors configured to interact with the one or more evaluation rules and perform operations including: creating, by one or more computing devices, a model representing relationships between user attributes, content exposure, and performance levels of specified target actions; obtaining content exposure data specifying content exposure experienced by a particular user; determining, using the model and the content exposure data, an incremental execution level attributable to each content exposure at the time of the action when the specific user performed the specified target action; as well as Modifying, by the one or more computing devices, a transmission standard of at least some digital components to which the particular user is exposed based on the incremental execution level.

9. The system according to claim 8, wherein: The one or more data processors are configured to perform operations including: performing an ablation experiment to obtain a set of control results for a set of control users that were not exposed to the set of specific digital components, the control results specifying, for each specific control user in the set of control users, whether the specific control user performed a specified action; Exposure results for a group of exposed users not included in the group of control users are collected, the exposure results specifying, for each exposed user in the group of exposed users, whether the exposed user has performed a specified action.

10. The system of claim 9, wherein: Creating the model includes utilizing a machine learning framework to create the model using user attributes, control results for each specific control user, and exposure results for each exposed user.

11. The system according to claim 8, wherein: Determining the incremental execution level includes: For each third-party exposure in the Content Exposure: determining a difference between an exposure time of the third party exposure and an action time when the specified target action occurs; determining a residual amount of the performance level contribution from the third-party exposure remaining at the action time based on a difference between an exposure time of the third-party exposure and an action time when the specified target action occurs; and A residual amount of the performance level is attributed to the third party exposure.

12. The system according to claim 11, wherein: The one or more data processors are configured to perform operations including: For each different type of third-party exposure, determining a decay function that specifies a decay rate of the remaining execution level contribution over time; as well as For each third-party exposure, a residual amount of the performance level contribution from the third-party exposure remaining at the action time is determined based on the decay function and a difference between the exposure time of the third-party exposure and the action time.

13. The system according to claim 11, wherein: Modifying the transmission criteria of at least some of the digital components includes adjusting the transmission criteria of the particular digital components in proportion to the magnitude of the incremental performance level attributable to the third party exposure of the particular digital components.

14. The system according to claim 13, wherein: Adjusting the transmission standard includes disabling a particular transmission standard having a specified size that is less than an incremental execution level of third party exposure attributable to the particular transmission standard.

15. A non-transitory computer readable medium storing instructions which, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: creating, by one or more computing devices, a model representing relationships between user attributes, content exposure, and performance levels of specified target actions; obtaining content exposure data specifying content exposure experienced by a particular user; determining, using the model and the content exposure data, an incremental execution level attributable to each content exposure at the time of the action when the specific user performed the specified target action; as well as Modifying, by the one or more computing devices, a transmission standard of at least some digital components to which the particular user is exposed based on the incremental execution level.

16. The non-transitory computer readable medium of claim 15, wherein: The instructions cause the one or more data processing devices to perform operations including: performing an ablation experiment to obtain a set of control results for a set of control users that were not exposed to the set of specific digital components, the control results specifying, for each specific control user in the set of control users, whether the specific control user performed a specified action; Exposure results for a group of exposed users not included in the group of control users are collected, the exposure results specifying, for each exposed user in the group of exposed users, whether the exposed user has performed a specified action.

17. The non-transitory computer readable medium of claim 16, wherein: Creating the model includes utilizing a machine learning framework to create the model using user attributes, control results for each specific control user, and exposure results for each exposed user.

18. The non-transitory computer readable medium of claim 15, wherein: Determining the incremental execution level includes: For each third-party exposure in the Content Exposure: determining a difference between an exposure time of the third party exposure and an action time when the specified target action occurs; determining a residual amount of the performance level contribution from the third-party exposure remaining at the action time based on a difference between an exposure time of the third-party exposure and an action time when the specified target action occurs; and A residual amount of the performance level is attributed to the third party exposure.

19. The non-transitory computer readable medium of claim 18, wherein: The instructions cause the one or more data processing devices to perform operations including: For each different type of third-party exposure, determining a decay function that specifies a decay rate of the remaining execution level contribution over time; as well as For each third-party exposure, a residual amount of the performance level contribution from the third-party exposure remaining at the action time is determined based on the decay function and a difference between the exposure time of the third-party exposure and the action time.

20. The non-transitory computer readable medium of claim 18, wherein: Modifying the transmission criteria of at least some of the digital components includes adjusting the transmission criteria of the particular digital components in proportion to the magnitude of the incremental performance level attributable to the third party exposure of the particular digital components.

Images