Model Orchestrator

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for determining attributes of unattributed outcomes across various content channels. The method includes receiving, by a model orchestrator, outcome data representing a set of unattributed outcomes, where each unattributed outcome has no observed attributes for an exposure of a predefined set of exposures. Attribute data representing a set of modeled attributes from each outcome model of a plurality of outcome models is received by the model orchestrator, where each set of modeled attributes includes respective measurements between one or more unattributed outcomes and one or more exposures. The respective measurements are updated based on one or more criteria to determine one or more updated attributes, where each updated attribute indicates a new attribute of a respective outcome from the set of unattributed outcomes for a corresponding exposure of the predefined set of exposures.

Description

[Technical field]

[0001] The present specification generally relates to data processing and techniques for aligning models across various data channels. [Background technology]

[0002] In a network environment such as the Internet, content providers can provide information for display in electronic or digital form. Digital content can be displayed in a variety of electronic forms. For example, digital content can be provided on a web page or within a defined execution environment (e.g., an iframe) on a web page, integrated into a presentation of query results, provided on an application interface, or provided in other suitable digital forms. Digital content can be video clips, audio clips, multimedia clips, images, text, or other suitable digital content.

[0003] Users participate in various online activities, and each of these activities exposes the user to different information. For example, a user's action of reading or viewing digital content can be referred to as the user's exposure to the particular digital content. Subsequent online activities, such as a user's downloading and installing an application, can be influenced by previous activities and the information the user was exposed to. The successful completion of an event by a user (e.g., the completion of the user's subsequent online activity) is typically referred to as an "outcome" or "conversion." Models can be used to determine or predict which online activities contributed to any given outcome, and different models are typically implemented to operate independently on data obtained from various data channels. Summary of the Invention [Means for solving the problem]

[0004] As described throughout this specification, various techniques can be used to coordinate the work performed by models implemented on various data channels. For example, in situations where various models operate on various data streams from various channels, a model orchestrator can be implemented to fill in data gaps that may exist between the various data channels and / or the various models. In a particular example, assume that model A is implemented to attribute observed outcomes to online activities specified in a data stream obtained from data channel 1, and model B is implemented to attribute observed outcomes to online activities specified in another data stream obtained from data channel 2. In this example, model A and / or model B may not have sufficient information to accurately attribute one or more of the observed outcomes to online activities because privacy settings or other data restrictions may prevent model A and / or model B from obtaining information from one of the data channels. This can result in unattributed observations and / or misattributed observations, in either case producing a suboptimal system that outputs incorrect or incomplete solutions.

[0005] For at least these reasons, existing techniques are generally inaccurate and even misleading for predicting attribution of results without metadata showing a clear path back to exposure, especially when exposure occurs through various content channels as represented by data contained in various data streams. Such types of results are also referred to herein as "unattributed results." To name just a few, the content channels may include at least one or more of a video streaming platform, a social messaging platform, an audio broadcasting platform, a search engine webpage, a retail webpage, a virtual store for a software application, or other suitable content channels. In some implementations, the content channels may have one or more dedicated result models for determining attribution for the channels.

[0006] When users are exposed to different digital content through various content channels that do not communicate user information to each other, one existing technique uses multiple outcome models to determine attributions for unattributed outcomes. However, such techniques may generate inconsistent attributions, which may also lead to a suboptimal attribution system. For example, a first outcome model may be configured to predict attributions to different exposures A, B, and C, and a second outcome model may be configured to predict attributions to different exposures B, D, and F. In the case of unattributed outcomes, the first outcome model may determine that the unattributed outcome is attributable to exposure A, while the second outcome model may determine that the unattributed outcome is attributable to another exposure, e.g., exposure F. The attributions generated from the first and second outcome models are inconsistent. This may occur because each outcome model is designed and / or trained exclusively for a particular content channel (or exclusively for a particular set of predefined exposures), and therefore may not have a "global view" of all exposures to which the outcome is attributable.

[0007] The techniques described herein can solve the above problems faced by existing techniques. A system implementing the described techniques is configured to determine attributes of a set of unattributed outcomes that could not be attributed with sufficient confidence by outcome models implemented in each of the various content channels. More specifically, the system receives data representing a set of unattributed outcomes (referred to as "outcome data") that do not have a clear path back to a corresponding exposure (or no attribution to a corresponding exposure in a set of predefined exposures has been observed). The system then receives data representing modeled attributes (referred to as "attribute data"). In some implementations, each modeled attribute indicates an attribute of the unattributed outcome for a type of exposure covered by the corresponding outcome model.

[0008] The modeled attributes are generated by a plurality of outcome models, each of which generates a respective set of modeled attributes for at least a portion of the set of unattributed outcomes, each set of outcome models including a respective measurement between a respective portion of the unattributed outcomes and a corresponding exposure.

[0009] In some implementations, each measurement may include a probability distribution indicating the likelihood of one or more outcomes not being attributed to one or more corresponding exposures. Note that the likelihood mapping between outcomes and exposures may be arbitrary, such as, for example, a one-to-one mapping, a one-to-many mapping, or a many-to-one mapping. For example, the probability distribution may indicate the likelihood that an unattributed outcome is due to one or more corresponding exposures. As another example, the probability distribution may indicate the likelihood that one or more unattributed outcomes are due to the corresponding exposures.

[0010] The system uses one or more criteria to update each measurement in a set of modeled attributes received from a plurality of outcome models. To update each measurement, the system is configured to determine one or more sets of modeled attributes, each having a measurement for at least the uncommonly attributed outcomes. The system obtains a respective value from the measurements determined by the outcome models. The value may be a real number representing the likelihood that the uncommonly attributed outcome is attributable to the respective exposure, or a specific exposure to which the uncommonly attributed outcome is attributable. The system determines whether to update the value based on the one or more criteria. If the one or more criteria are met, the system updates the value to resolve discrepancies in attributes determined between the various outcome models for the uncommonly attributed outcomes.

[0011] The system determines one or more updated attributes based on the respective updated measurements, each of the updated attributes indicating a respective attributed result resulting from a corresponding exposure in the set of predetermined exposures.

[0012] Optionally, the system determines whether one or more modeled attributes differ from the updated attributes, and in response to a determination that the one or more modeled attributes differ from the updated attributes, the system updates the one or more modeled attributes to match the updated attributes. In some implementations where the outcome model does not cover the type of outcome exposure predicted by the updated attributes, the system can retract modeled attributes for unattributed outcomes generated by the outcome model. The term "retract" generally refers to invalidating a modeled attribute for future reference and / or not storing a modeled attribute generated from an outcome model.

[0013] The one or more criteria used to update each measurement value can be determined by user input or system pre-setting according to different attribute requirements. In some implementations, the one or more criteria can include a result volume threshold of the result model. For example, if the result model is assigned a number of unattributed results that meets the result volume threshold, the system can decide to update the value of the measurement value generated by the result model.

[0014] In some implementations, the one or more criteria may include attributes associated with the unattributed results. For example, the attributes may be a geographic location associated with the unattributed results, which may optionally be used to compare the geographic location to a geographic location associated with an exposure predicted by the modeled attributes. As another example, the attributes may be a time value associated with the unattributed results, which may optionally be used to compare the time the corresponding user was exposed to an exposure predicted by the modeled attributes. Additionally, the attributes associated with the unattributed results may include a type of device and / or operating system on which the results occur. Examples of device types may include types of smartphones, computers, smart tablets, smart televisions, smart watches, or other device types, respectively. Examples of operating system types may include types of Windows, MacOS, Android, Linux, and other operating systems. Additionally, the attributes associated with the unattributed results may include information related to the provider providing the digital content and the type of digital content provided. The digital content type may include text, images, videos, or other digital content types.

[0015] In some implementations, the one or more criteria may include a similarity threshold between the unattributed result and the exposure predicted by the modeled attributes. For example, the system may generate a similarity value for the unattributed result and the exposure by determining a distance between the result's features in the embedding space and the exposure. The system may compare the similarity value to the similarity threshold to determine whether the one or more criteria are met.

[0016] In some implementations, the one or more criteria may include a threshold number of remaining unattributed results in the set of unattributed results. Alternatively or additionally, any one of the above criteria may depend on a threshold number of remaining unattributed results in the set of unattributed results. For example, if the number of remaining unattributed results meets the threshold number of remaining unattributed results, the system does not update the parameters of the modeled attributes. As another example, if the number of remaining unattributed results meets the threshold number of remaining unattributed results, at least one of the above criteria may be relaxed to different degrees according to different requirements, for example, the similarity threshold may be half the original threshold, or the geographic location criterion may be ignored.

[0017] Other embodiments of this aspect include corresponding methods, apparatus, and computer programs, the computer programs being configured to perform the above-mentioned actions of the methods and encoded on a computer storage device.

[0018] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages: The described techniques improve accuracy and efficiency for determining attribution of unattributed results to exposures among a set of predefined exposures. Particularly for results completed by a user exposed to one or more digital content from one or more different channels, the system may not include metadata indicating a clear path back to the exposure of the result. This is due to privacy policies, for example, where user identification information may not be accessible when one or more digital content is displayed to the user from the different channels. In these cases, the described techniques do not expend time and computational resources to generate modeled attributes, since the described techniques are efficient by determining the attributes using modeled attributes generated from one or more outcome models. Rather, the described techniques determine the attributes by modifying or adjusting respective measurements in the modeled attributes based on one or more criteria. The one or more criteria are determined to resolve potential discrepancies in attributes predicted by multiple outcome models (e.g., one outcome is attributed to multiple different exposures by multiple outcome models). The one or more criteria are further designed to consider information from a global perspective, i.e., attributes associated with the outcome and global information from all outcome models, such that the attribution generated by the system is more accurate than attribution predicted by existing techniques. Additionally, the techniques described herein address the problem of outcome attribution in situations where the model created to perform attribution is unable to properly attribute based on the information available to the model.For example, if the data required to attribute a given outcome with sufficient confidence is not available for the individual models, the model orchestrator device may implement the techniques described herein by collecting outputs from each of the various models and attributing the given outcome based on the various model outputs and other data available to the model orchestrator (e.g., not available for the different models), thereby reducing the number of unattributed outcomes while complying with data sharing restrictions.

[0019] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram of an example system for determining attributes of unattributed results. [Diagram 2] FIG. 2 is a block diagram of an example model orchestrator included in an example system. [Diagram 3] 1 is a flow diagram of an example process for determining attributes of unattributed results. [Figure 4] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Like reference numbers and designations in the various drawings indicate like elements.

[0022] Attributing outcomes to prior activity is a common application of models. However, a model's ability to properly attribute outcomes to prior activity is limited in part by the data available when training and running the model. In the context of online activity, there are privacy policies (and other restrictions) that prevent data from one content channel from being shared with another, but a user's online activity that occurred prior to any given outcome may span many different content channels. This lack of cross-channel data limits the ability of a given model to output attributes based on data from any given content channel. Thus, many outcomes go unattributed.

[0023] For example, a user connected to the Internet is exposed to various digital content, such as, for example, search results, web pages, news articles, social media posts, audio information output by a digital assistant device, video streams, digital content associated with one or more of the aforementioned digital content, or other suitable digital content. Some of these exposures to digital content may contribute to the user performing or completing a designated target action as a result (also referred to as a conversion, as described above). For example, a user exposed to digital content, e.g., a user viewing or being shown a digital component about furniture from a content channel, may purchase the furniture in a retail platform. In this example, purchasing the furniture generally refers to a user action performed or an event completed by the user in response to being exposed to digital content from a content channel. In other words, the result of purchasing the furniture can be attributed to the digital content displayed to the user. Similarly, a user exposed to another digital component related to a video streaming service may eventually download and install an application for the video streaming service, in which case the download and installation of the application is a result or completion of an event and is attributed to the other digital content.

[0024] The term "exposure" throughout this specification generally refers to an event in which a user, viewer, or viewer views or hears digital content provided by a content provider. The digital content may include at least one or more of a video clip, an audio clip, a multimedia clip, an image, text, or other suitable digital content. The digital content may be provided in a variety of forms, such as, for example, on a web page, on a banner within a web page, integrated into a representation of a query result, provided on an application interface, or other suitable digital form. In some implementations, an exposure event includes one or more user engagements with the provided digital content. User engagement may include positive or negative user interactions with the digital content, for example, positive interactions including tapping or clicking on a tricky component within the digital content, or negative interactions including a user scrolling past or closing the digital content presentation. In some implementations, successful exposure includes at least one positive user interaction with the provided digital content.

[0025] An interactive component can be, for example, a Uniform Resource Locator (URL) that, after being interacted with by the user, directs the user to a particular profile associated with that URL. In some cases, the term "exposure" may also be referred to as an "impression." For simplicity, in the following description, an exposure with a positive user interaction may also be referred to as a "user click." Note that the terms "result" or "conversion" as used throughout this specification do not typically include a user click on an exposure that does not result in the completion of an event (e.g., downloading a software application, completing a transaction, or other event). However, in some circumstances, a result may include a user click without considering subsequent user activity.

[0026] The term "content channel" throughout this specification generally refers to a service, platform, and / or interface that can provide digital content to a viewer or viewer. Examples of content channels include application interfaces, web page interfaces, video streaming services, audio streaming services, email services, online search services, navigation services, online shopping platforms, social messaging platforms, social streaming platforms, or other suitable services and / or interfaces. In some implementations, the various content channels can include services, platforms, and / or interfaces that are managed or owned by different parties. For example, a content channel can include a first social messaging platform that is managed or owned by a first party and a second social messaging platform that is managed or owned by a second party that is different from the first party. In some circumstances, restrictions may prevent data from one content channel from being shared with a device or system that processes data from another content channel.

[0027] In some cases, a party may manage or run multiple content channels to provide digital content to viewers. The party may further request one or more attribution models designed and / or trained to determine attribution of results to exposures provided through multiple content channels. Generally, the attribution models may analyze metadata associated with results and one or more exposures from various channels to determine the exposure to which the results should be attributed. In such situations, limitations or technical constraints may prevent data from one content channel from being shared with a device or system that processes data from another content channel.

[0028] In an environment where multiple content channels or platforms can provide digital content to a client device (or a viewer associated with the client device), it can be difficult to determine the exposure from which content channel an outcome is attributable. This is typically because the user / device identifier (e.g., anonymized string) may not be accessible or the user / device identifier may be lost, encrypted, or deleted between the various content channels. First, when a user is exposed to digital content through a content channel, the user / device identifier may not be accessible for various reasons, e.g., due to one or more particular user's privacy policies and / or user activity (e.g., the user is not logged into a user account or does not have a registered account associated with the content channel). Furthermore, data representing the user / device identifier is generally not shared or communicated between the various content channels, especially when the various content channels are managed or owned by different parties. Thus, a user / device identifier obtained by a first content channel cannot be accessed by a second content channel.

[0029] As a simple example, a user may be exposed to digital content on a first platform operated by a first party, and then later exposed to the same digital content on a second platform operated by a second party. The first platform collects data representing a user / device identifier, but does not communicate that identifier to the second platform. The second platform would not otherwise have access to the user / device identifier. The user completes an event (e.g., a conversion occurs) that is associated with the digital content on another platform managed by the second party. The result is attributed to the digital content on the second platform. However, from the second party's perspective, the attribution model for the content channel managed by the second party does not have the metadata used to establish a path that tracks the result to the exposure on the second platform.

[0030] Some results have data indicating the exposure to which the result was attributed, also referred to as "observed results" or "observed attributes." Observed attributes typically include data indicating a completed result based on a user's positive interaction with the exposure. For example, a user clicks or taps on a URL embedded in digital content, which displays a profile in which the result occurs. However, as noted above, not all results have data indicating a clear path to track exposure. Results without data indicating associated exposure are referred to as "unattributed results," as noted above. In some implementations, one or more result models of a content channel are implemented to predict attributes of unattributed results of a content channel. The predicted attributes generated by the result models are generally referred to as "modeled attributes."

[0031] Described herein are techniques for determining attribution of unattributed results from various content channels. In particular, the described techniques include a central attribution engine configured to coordinate operations between various models and process unattributed results based on different sets of modeled attributes, each set of modeled attributes predicted by a respective outcome model for a respective content channel. Each set of modeled attributes may further include a respective measurement indicating a relationship between the unattributed results and a corresponding exposure through a corresponding content channel for which the results are specifically designed to be attributable. Herein, the central attribution engine is also referred to as a model orchestrator.

[0032] The described techniques can analyze the modeled predictions based on information associated with multiple different channels and use the global information to adjust or modify measurements of a set of modeled predictions. In some implementations, the described techniques can determine one or more criteria based on the global information and adjust measurements in the modeled predictions based on the one or more criteria. The described techniques can use the adjusted or modified measurements to determine updated attributes for these unattributed results. In some implementations, the system can also update the modeled attributes based on the updated attributes to avoid conflicts in attribution, e.g., to prevent one result from being modeled as being attributable to multiple exposures. Because the central attribution engine or model orchestrator obtains global information across the various outcome models of the various content channels, the attributions determined for the unattributed results are more accurate than those determined by the local outcome models.

[0033] In situations where the systems described herein collect and / or use personal information about the user, the user may be provided with an opportunity to enable / disable or control programs or features that may collect and / or use personal information (e.g., information about the user's social networks, social actions or activities, the user's preferences, or the user's current location). Additionally, certain data may be processed in one or more ways before being stored or used such that personally identifiable information associated with the user is removed. For example, the user's identity may be anonymized so that the user's personally identifiable information cannot be determined, or the user's geographic location from which the location information is obtained (e.g., to city, zip code, state level, etc.) may be generalized so that the user's specific location cannot be determined.

[0034] 1 is a block diagram of an example system 100 for determining attribution of unattributed results. The example system 100 includes a model orchestrator 140 and is an example of an attribution system implemented on one or more computers (e.g., nodes or machines) at one or more locations, which may implement the systems, components, and techniques described below. Some of the components of system 100 may be implemented as computer programs configured to run on one or more computers.

[0035] As shown in FIG. 1, the system 100 can include one or more attribution engines 120a-120n. An attribution engine can include one or more processors at one or more locations configured to determine attributes of results. In some implementations, each attribution engine 120a-120n can be assigned to a particular content channel to determine attributes of results completed in the particular content channel. For example, a first attribution engine 120a is assigned to a first channel (e.g., a first social messaging platform), a second attribution engine 120b is assigned to a second channel (e.g., a first video streaming service), and an n-th attribution engine 120n is assigned to an online search service. Other content channel examples are described in more detail above.

[0036] Further, each attribution engine 120a-n may include a respective result model 130a-n that is dedicated to determining results for a content channel assigned to the corresponding attribution engine 120a-n. In other words, the result model of the attribution engine 120 does not have a complete picture of exposures to which results may be attributed across various content channels. Rather, the result models 130a-n generally cover predefined exposures associated with the corresponding content channels. For example, the attribution engine 120a may include a result model 130a, which may determine result attributes for a first set of predefined exposures (e.g., exposures A, B, and C) associated with the corresponding content channel (e.g., channel 1 and channel 2). The attribution engine 120d may include a result model 130d, which may determine result attributes for a second set of predefined exposures (e.g., exposures C, G, and K) associated with the content channel (e.g., channel 2, channel 6, and channel 7).

[0037] In general, the attribution engines 120a-n receive the observation data 110 as input and generate respective model outputs 135a-n that include observed and / or modeled attributes for the outcome based on the observation data 110. The observation data 110 can include metadata that stores the outcome and a "footprint" associated with the outcome. The term "footprint" generally refers to a user's historical activity or interactions with various exposures that may affect the outcome completed by the user. For example, the observation data 110 can include data indicating that for a first outcome, the user clicked or tapped on a URL while browsing digital content, and the user subsequently completed a purchase on the webpage provided to the user by the URL. In this case, the observation data 100 can include an identifier of the user click or tap (e.g., a click ID) and associate the click ID with a corresponding identifier of the outcome event (e.g., an outcome ID). As described above, an outcome having observation data 110 that indicates an exposure to which the outcome should be attributed is referred to as an "observed attribute."

[0038] In some circumstances, the observation data 110 may include data indicative of multiple exposures along a user's "footprint" to a result. For example, before a result is completed by an entity, the entity interacts with a first digital content through a first content channel (e.g., clicked on the digital content on a search service webpage) and interacts with a second digital content through a second content channel (e.g., watched an entire video advertisement while watching a streamlined video on a video streaming platform). The attribution engines 120e assigned to the two content channels perform operations on the associated result models 130e to predict to which exposure the result should be attributed. As noted above, this type of result is also referred to as "modeled attribution."

[0039] In some cases, the observation data 110 may not include a clear path to track one or more exposures to a result for various reasons. In some situations, due to certain privacy policies, data such as a user / device identifier for tracking user activity or interaction is removed from the observation data 110 provided to the system 100 or encrypted before the observation data 110 is provided to the system 100 such that the system 100 cannot determine user activity or interaction to determine attribution. In some situations, data representing a user / device identifier is not communicated across various content channels, so that the result model 130a of the first channel does not know that the user who completed the result positively interacted with the digital content on the second channel, and the second channel does not share information with the first channel. In some implementations, data representing a user / device identifier may not be observed at all, and the user who completed the result may forget to log into their user account when they were exposed to the first digital content and then completed the result on the retail platform managed by the attribution engine 120m. In the above situations, a result that does not have observation data linking one or more exposures to the result is referred to as an "unattributed result." Outcome models may not produce sufficiently accurate attributions and may produce misleading and / or inconsistent attributions for unattributed outcomes.

[0040] For unattributed results that do not have an established link or connection between the result and the exposure, the result models 130a-130n can predict attributes of the result based on the predefined exposures of the content channels managed by the corresponding result models. To predict the attributes, the example result models can generate a measure between the unattributed result and the exposures in the predefined set of exposures of the result models. The measure can include a likelihood that the one or more unattributed results are attributable to one or more exposures in the content channels of the result models. Note that the predicted connection or mapping need not be a one-to-one mapping. Instead, the mapping or connection can be a many-to-one mapping or a one-to-many mapping. For example, a result model for content channels A, B, C can determine that, without data representing a user / device identifier, with a 50% probability, the first unattributed result is attributable to an exposure from content channel A, with a 10% probability, the second unattributed result is attributable to an exposure from content channel B, and with a 20% probability, the third unattributed result is attributable to an exposure from content channel C. As another example, the outcome model may determine that, without data representing a user / device identifier, there is a 40% probability that a first unattributed result is attributable to exposure from content channel A, a 5% probability that a first unattributed result is attributable to exposure from content channel B, and a 15% probability that a first unattributed result is attributable to exposure from content channel C. As another example, without data representing a user / device identifier, the outcome model may determine that there is a 70% probability that a first unattributed result is attributable to exposure from content channel A, a 21% probability that a second unattributed result is attributable to exposure from content channel A, and a 3% probability that a third unattributed result is attributable to exposure from content channel A. Once the outcome model determines attributes for the unattributed results, for simplicity, the unattributed results are also referred to as "modeled attributes." The attribution engines 120a-n output the modeled attributes to corresponding model outputs 135a-n.

[0041] The system 100 includes a model orchestrator 140 configured to solve the above-mentioned problems. More specifically, the model orchestrator 140 can efficiently determine accurate attribution for unattributed results based on global information collected from multiple attribution engines 120a-120n.

[0042] As shown in FIG. 1, upon receiving one or more requests 150 to determine attributes of the unattributed results 150, the model orchestrator 140 can process the input unattributed results 145 to generate output data 155 having attributes determined for the unattributed results 145. In some implementations, the requests 150 can be received from an entity that uses one or more services provided by the system 100 (e.g., a service for determining attributes for completed results in a corresponding content channel). The entity can send the requests 150 using, for example, a client device. It should be noted that the model orchestrator 140 does not need to receive a request to process the unattributed results. Rather, the model orchestrator 140 can perform processing based on a schedule or otherwise automatically process the unattributed results without first receiving a request.

[0043] In some implementations, the unattributed results may be automatically processed by the system 100 based on one or more predefined criteria. For example, one example of a criterion may be a threshold number of unattributed results. If the system 100 determines that the total number of unattributed results is equal to or greater than the threshold number, the system 100 may automatically generate a request 150 to the model orchestrator 140 to determine attributes for the unattributed results 145.

[0044] In some implementations, the requests 150 may be included in the model outputs 135a-n generated from the corresponding attribution engines 120a-n. The outcome models 130a-n cannot determine the appropriate exposures to which the results should be attributed due to the lack of user-related information from the observation data 110. Thus, the outcome models 130a-n may output unattributed results included in the model outputs 135a-n. The attribution engines 120a-n may determine whether a request needs to be generated for the model orchestrator 140 to further process the unattributed results in the model outputs 135a-n. For example, the attribution engines 120a-n may determine a ratio of the number of unattributed results to all results in the model outputs 135a-n and compare the ratio to a threshold ratio. If the ratio meets the threshold ratio, the attribution engines 120a-n may generate a request for the model orchestrator 140 to process the unattributed results to be included in the model outputs 135a-n.

[0045] In general, the model orchestrator 140 processes the set of unattributed results 145 and the modeled attributes generated from the outcome models to determine attributes for the unattributed results 145. In some implementations, as described above, the model orchestrator 140 also receives the unattributed results from the model outputs 134a-n from the various attribution engines 120a-n. This technique is efficient because the model orchestrator 140 does not need to repeat the operations for modeling the attributes performed by the attribution engines 120a-n. Rather, the model orchestrator 140 is configured to adjust or update the modeled attributes based on global information of all the observation data 110 and the outcome models 130a-n. The model orchestrator 140 can adjust or modify the predicted measurements for the modeled attributes, for example, based on one or more criteria. The one or more criteria are determined or selected according to various attribution requirements or are specified by user input or within the request 150. The details of determining updated attributes for unattributed results based on modeled attributes are described in further detail in conjunction with FIG.

[0046] After determining updated attributes for the unattributed outcomes 145, the model orchestrator 140 may further retract some of the modeled attributes if they differ from the updated attributes. For example, if the modeled attributes for a first unattributed outcome indicate exposure A to which the outcome should be attributed, but the corresponding updated attributes for the first unattributed outcome indicate a different exposure, e.g., exposure B, the system 100 or the model orchestrator 140 may retract the modeled attributes. The term "retract" generally refers to invalidating a modeled attribute, severing a link or connection between an outcome and an exposure predicted by the modeled attribute, or deleting or not storing the modeled outcome in storage.

[0047] The model orchestrator 140 generates output data 155 including the updated attributes and provides the output data 155 for further operations performed by one or more downstream engines 160. The operations can include any appropriate operations depending on various attribute requirements. For example, one operation can be generating a report based on the updated attributes to shed light on a question of interest, such as how efficient or influential a particular exposure is in achieving results, which content channels generate the most results, or other questions of interest. Another exemplary operation can be adjusting various exposures presented to a viewer based on the updated attributes. In some implementations, the downstream engines 160 again include the model orchestrator 140 configured to process the remaining unattributed results included in the output data 155.

[0048] In some implementations, the output data 155 can further include one or more unattributed results 145 that the model orchestrator 140 does not determine during the current time cycle. The downstream operations can include providing the one or more unattributed results 145 to one or more of the attribution engines 120a-n or to the model orchestrator 140 for further processing during the next time cycle.

[0049] Figure 2 is a block diagram of an example model orchestrator 200 included in an example system. The example model orchestrator 200 may be equivalent to the model orchestrator 140 in Figure 1, and this example system may be equivalent to the example system 100 of Figure 1. Components of the model orchestrator 200 and operations performed by the model orchestrator 200 will now be described in more detail.

[0050] As shown in FIG. 2, the model orchestrator 200 includes a request processor 220 configured to receive and process requests 210. As described above, the requests 210 can be received from entities of a service or from model outputs 240 from various attribute engines (e.g., attribute engines 120a-n in FIG. 1). In some implementations, the operations performed by the model orchestrator 200 need not be triggered by a request, as described above. The request processor 220 can include a data structure for storing the various queues received. For example, the request processor 220 can include a queue structure (e.g., a first-in-first-out (FIFO) queue).

[0051] The model orchestrator 200 further includes a configuration engine 230 for receiving configuration data 215. The configuration data 215 may include data specifying one or more attribution engines to be registered with the model orchestrator 200 such that the model orchestrator will obtain model outputs from the registered attribution engines to process unattributed results. The configuration data 215 may further include a schedule for the model orchestrator that generally specifies a mechanism for determining updated measurements of modeled attributes and sampling exposures from the updated measurements. In some implementations, the configuration data 215 may further include a capping configuration that determines the size of the set of modeled attributes and / or a global threshold ratio of observed attributes to modeled attributes from all registered attribution engines (in other words, global capping).

[0052] The received requests 210 and configuration data 215 can be stored in storage 225 of the model orchestrator 200. In some implementations, the storage 225 can be a distributed database where each node has multiple replicas of the storage. In this way, the system can avoid unwanted data loss or damage due to hardware malfunction.

[0053] The model orchestrator 200 further includes an attribution optimizer 250 for processing unattributed results 235 based on the model outputs 240. The model outputs 240 are obtained from attribution engines registered with the model orchestrator 200. The model outputs 240 include a number of modeled attributes generated by the respective result models for the respective content channels. In some implementations, the model outputs 240 may also include unattributed results that are not processed by the attribution engines.

[0054] In some implementations, the model output 240 may include a request for sampling exposures for modeled attributes. The request may include a scored path, one or more capping configurations, and / or one or more retraction requests. A scored path typically refers to data representing a user's interaction or activity with one or more exposures, and data indicating the number of results to sample for each exposure. In some cases, a scored path may further include data associated with the result, such as data indicating whether the result is a modeled attribute or an observed attribute, whether the result has data dating back to one or more exposures, when the result was completed, and / or a geographic location associated with the result. The capping configuration includes data representing a threshold ratio between the modeled attribute and the observed attribute for each result model (i.e., local capping), similar to the global capping configuration described above. The capping configuration may further include data associated with the exposure and content channel, such as data representing the ID of the exposure or campaign, and / or data representing the device on which the digital content is provided to the viewer.

[0055] In some implementations, the model output 240 may further include a request to retract inaccurate modeled attributes, e.g., modeled attributes that differ from the updated attributes generated by the model orchestrator 200. Details of retracting modeled attributes are described below.

[0056] The attribute optimizer 250 is configured to obtain respective measurements on various sets of modeled attributes, each set of modeled attributes generated by a respective outcome model from the registered attribute engine. As described above, each measurement of the set of modeled attributes may include a probability distribution that includes multiple likelihoods of the outcome resulting from exposure through the content channel of the outcome model. The probability distribution may be used by the model orchestrator 200 to sample exposures from the set of predefined exposures for the outcome.

[0057] To generate updated measurements (e.g., updated probability distributions) for sampling exposures to attribute outcomes, the attribute optimizer analyzes each measurement from the various sets of modeled attributes and determines whether to adjust or modify one or more measurements based on one or more criteria, examples of which are described in more detail below.

[0058] For unattributed outcomes in the set of unattributed outcomes 235, model orchestrator 200 may determine a set of modeled attributes from each outcome model that performed an attribution operation on the unattributed outcome. Model orchestrator 200 analyzes information associated with the set of modeled attributes to determine whether the respective measurements for one or more of the selected set of modeled attributes need to be adjusted or modified. In response to determining that the respective measurements need to be adjusted or modified, model orchestrator 200 modifies the values ​​of the respective measurements based on one or more criteria.

[0059] For example, for an outcome of purchasing a watch on a retail platform, a first outcome model determines that the outcome is attributable to a first channel (e.g., exposure on a search engine webpage) with a 70% probability and a second channel (e.g., exposure on a video streaming platform) with a 20% probability. A second outcome model determines that the same outcome is attributable to a third channel (e.g., exposure on a social messaging platform) with a 55% probability. The model orchestrator 200 obtains two sets of modeled attributes from the two outcome models and determines which exposure and which channel the outcome of purchasing a watch should be attributable to. For example, if the model orchestrator 200 obtains data indicating that a user did not use a search engine to search for information related to watches, or that a search for such information was performed a period of time longer than a threshold period of time ago, the model orchestrator 200 can adjust the 70% probability to, for example, 30%, 20%, 5%, or other appropriate probability value. In some implementations, the model orchestrator can equalize or rescale the probability distributions from the various outcome models. For instance, taking the example above, the model orchestrator 200 may rescale the probability of a search engine to 20%, the probability of a video streaming platform to 10%, and the probability of a social messaging platform to 60%.

[0060] The model orchestrator 200 generates updated attributes based on the updated measurements. Each updated attribute indicates a respective unattributed outcome of the set of unattributed outcomes that is attributable to a corresponding exposure of the set of predefined exposures. For example, the model orchestrator 200 may select an exposure with a maximum likelihood in the updated probability distribution as an exposure to which the unattributed outcome is attributable. Alternatively, the model orchestrator 200 may determine a group of candidate exposures for the outcome according to a threshold probability value, and then select one exposure from the group of candidate exposures as an attribute for the outcome based on the respective likely values ​​of the group of candidate exposures. As another example, the orchestrator 200 may determine an exposure for the unattributed outcome such that the remaining unattributed outcomes can be minimized. In other words, the orchestrator 200 determines as many exposures as possible for the unattributed outcome based on the updated measurements.

[0061] The model orchestrator 200 may repeatedly perform the above operations on various unattributed results until a stopping point is reached. The stopping point may be determined by comparing the number of iterations or real-time execution time to a threshold. When the number of iterations or real-time execution time meets or exceeds the threshold, the model orchestrator 200 stops generating updated attributes for the unattributed results 235.

[0062] The one or more criteria used to update each measurement value may be determined by user input or system pre-settings according to various attribute requirements. For example, the one or more criteria may be specified in configuration data 215. Alternatively, the one or more criteria may be stored in storage 225.

[0063] In some implementations, the one or more criteria may include an outcome volume threshold for the outcome model. The outcome volume may typically be represented by a ratio of unattributed outcomes assigned to the outcome model to the overall set of unattributed outcomes. For example, if an outcome model is assigned too many unattributed outcomes to determine attribution, e.g., an outcome volume exceeding a predefined threshold, the model orchestrator 200 will update the values ​​of the respective measurements in the modeled outcomes of the outcome model.

[0064] In some implementations, the one or more criteria may include an attribute associated with the unattributed result. The attribute associated with the unattributed result may include a geographic location where the unattributed result is completed. Alternatively or additionally, the attribute may include a time value when the unattributed result is completed. The model orchestrator 200 may obtain data from the model output 240 indicating when and where the result is completed, compare the time value with a time value at which the exposure is presented to the user, and / or compare the geographic location with a geographic location associated with the exposure. The model orchestrator 200 may determine whether a measurement needs to be updated based on the comparison. For example, a result associated with a geographic location in California, USA may be less likely to be attributed to an exposure associated with a location in Germany. In light of this, the model orchestrator 200 may determine to update the measurement of the corresponding modeled attribute. As another example, a result completed at a time prior to the time the exposure is presented to the user may be less likely to be attributed to the exposure. In light of this, the model orchestrator 200 may determine to update the measurement of the corresponding modeled attribute.

[0065] In some implementations, the one or more criteria may include a similarity threshold between the unattributed results and the exposure predicted by the modeled attributes. The model orchestrator 200 may determine whether to update the measurements based on a comparison of the similarity threshold to the calculated similarity value. For example, the system 100 of FIG. 1 or the model orchestrator 200 may calculate a similarity value of the unattributed results and the predicted exposure from the modeled attributes by determining the distance between the features of the results in the embedding space and the exposure. The model orchestrator 200 may compare the similarity value to the similarity threshold to determine whether an update of the measurements in the modeled attributes is required. An exemplary similarity value may be a percentage ranging from zero to one, with zero representing the least similar and one representing the most similar or substantially the same, where the similarity threshold may be 0.75, 0.9, 0.95, or other suitable similarity value.

[0066] In some implementations, the one or more criteria may include a threshold number or percentage of remaining unattributed results in the set of attributed results. The threshold may be determined according to various attribute requirements. For example, the threshold percentage may be 10%, 30%, 50%, or other suitable percentage. If the remaining unattributed results are equal to or greater than the threshold, the model orchestrator 200 may decide not to update the measurements of the modeled attributes. In this way, the model orchestrator 200 may avoid unnecessary updates when the modeled attributes are not in conflict, and may resort to updating the modeled attributes to prevent potential conflicts in the modeled attributes when the unattributed results are below the threshold.

[0067] Alternatively or additionally, one or more of the above criteria may depend on a threshold number of remaining unattributed results in the set of attribute results. For example, if the number of remaining unattributed results meets the threshold number of remaining unattributed results, the model orchestrator 200 does not update the parameters of the modeled attribute. As another example, if the number of remaining unattributed results meets the threshold number of remaining unattributed results, at least one of the above criteria may be relaxed by the model orchestrator 200 to different degrees according to different requirements. For example, the similarity threshold may be doubled from the original similarity threshold.

[0068] The model orchestrator 200 may further include a retraction engine 265 for retracting modeled attributes that are inconsistent with the updated attributes. For example, for a first outcome, the updated attributes may indicate that it was due to a first exposure. However, the previously modeled attributes from the outcome model indicate that the first outcome was due to a second exposure that is different from the first exposure. The model orchestrator 200 may also remove the modeled attributes, as described above, and may also remove the connection between the outcome and the second exposure.

[0069] After processing the unattributed results 235, the model orchestrator 200 is configured to provide output data to downstream operations. The output data may include updated attributes 270 and re-traced modeled attributes 275. The updated attributes 270 may be used to generate campaign reports for the entities of the attribute service. The re-traced modeled attributes 275 may be used to modify previously generated reports and / or be fed back to the model orchestrator 200 for processing in the next time cycle.

[0070] 3 is a flow diagram of an exemplary process for determining attribution of unattributed results. For convenience, the above-described process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a system, such as the example attribution system 100 of FIG. 1, may be suitably programmed to perform the process 300.

[0071] First, the system receives a request to determine attributes for a set of unattributed results for a predefined set of exposures. The request can be received from an entity using a service provided by the system. In some implementations, the request can be included in a model output generated by a multiple results model. In some implementations, the system does not need to receive a request to perform an operation to determine attributes for unattributed results.

[0072] The system typically determines the output on demand. The output is determined based on one or more updates of modeled attributes for one or more unattributed outcomes. Details of generating the output are described below. Note that the set of predefined exposures includes exposures of various channels governed by different outcome models; in other words, the system determines attributes across the entire landscape of all exposures in the corresponding content channel.

[0073] The system receives 310 results data representing a set of unattributed results. As described above, an unattributed result in the set of unattributed results has no observed attributes to an exposure in the set of predefined exposures. Alternatively or additionally, an unattributed result does not have data indicating a clear path to track a corresponding exposure for various reasons, such as a particular privacy policy.

[0074] The system receives attribute data representing a set of modeled attributes from each outcome model of the plurality of outcome models (320). The modeled attributes of the set of modeled attributes indicate attributes of unattributed outcomes for exposures of the set of predefined exposures. Each set of the set of modeled attributes includes respective measurements between one or more of the set of unattributed outcomes and one or more of the set of predefined exposures. The measurements of the sets of modeled attributes may include probability distributions indicating one or more likelihoods of one or more unattributed outcomes being attributable to one or more corresponding exposures of the set of exposures.

[0075] The system updates (330) each measurement in the set of modeled attributes received from the plurality of outcome models based on one or more criteria. The one or more criteria may include an outcome volume threshold for the outcome model, an attribute associated with the unattributed outcome such as geographic location, a similarity threshold between the unattributed outcome and the corresponding modeled attribute, a threshold number of remaining unattributed outcomes in the set of unattributed results, or other suitable criteria. Details of exemplary criteria are described above.

[0076] The system determines one or more updated attributes based on each updated measurement (340), where each updated attribute indicates a new attribute of a respective result from the set of unattributed results for a corresponding exposure in the set of predetermined exposures.

[0077] Optionally, when determining the updated attributes, the system can retract modeled attributes that are inconsistent with the updated attributes. To retract, the system removes or invalidates previously predicted modeled attributes and updates one or more modeled attributes of the set of modeled attributes based on the one or more updated attributes.

[0078] In some implementations, the system can provide one or more updated attributes to one or more downstream operations. An example downstream operation can include generating one or more reports based on the updated attributes of entities subscribing to the attribute service. Optionally, the output generated by the system can still include one or more unattributed results, which can be returned to the system for processing in the next time cycle.

[0079] In some implementations, the system can generate one or more attribution reports based on the one or more updated attributes. The reports can include information about newly determined attributes for previously unattributed outcomes, newly updated or modified attributes for outcome attributes modeled by upstream conversion models, and / or remaining unattributed outcomes output from the model orchestrator. The reports can be in any suitable form, such as, for example, digital figures, charts, tables, images, audio, video, or digital data having other suitable forms that can be sent to another module, printed on paper, or displayed on a display.

[0080] In some implementations, the system can determine the digital content to be provided to the client device based on the one or more updated attributes. The system can further provide data including the digital content to a display of the client device for display. Alternatively, the system can determine and provide the digital content to the client device based on an attributed report, as described above.

[0081] 4 is a block diagram of an exemplary computer system 400. The exemplary computer system 400 can be used to perform the operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected using, for example, a system bus 450. The processor 410 can process instructions for execution within the system 400. In some implementations, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 can process instructions stored in the memory 420 or the storage device 430.

[0082] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In some implementations, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0083] The storage device 430 can provide mass storage for the system 400. In some implementations, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0084] The input / output device 440 provides input / output operations for the system 400. In some implementations, the input / output device 440 can include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In another implementation, the input / output device can include a driver device configured to receive input data and send output data to an external device 860, such as a keyboard, a printer, and a display device. However, other implementations, such as a mobile computing device, a mobile communication device, a set-top box television client device, etc., can also be used.

[0085] Although an exemplary processing system is illustrated in FIG. 1, the implementation and functional operations of the subject matter described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof.

[0086] The subject matter and embodiments of the operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium (or a computer storage medium) for execution by or to control the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus. The computer storage medium may be or may be included in a computer readable storage device, a computer readable storage substrate, a random access memory array or a serial access memory array or device, or one or more combinations thereof. Furthermore, although a computer storage medium is not a propagating signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage media may be or may be contained in one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0087] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices, or data received from other sources.

[0088] The term "data processing apparatus" encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination thereof. The apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may realize a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0089] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but need not. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on multiple computers located at one site, or on multiple computers distributed across multiple sites and interconnected by a communications network.

[0090] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data to generate output. The processes and logic flows may also be performed by, or apparatus implemented as, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0091] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated into another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0092] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, e.g., feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0093] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., as an application server, or includes a front-end component, e.g., a client computer with a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Exemplary communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0094] A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a client device (e.g., for the purposes of displaying the data and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., results of user interaction) can be received at the server from the client device.

[0095] Although the specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or the scope of the claimed subject matter, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the specification with respect to the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and initially claimed as such, in some cases one or more features from a claimed combination may be deleted from the combination, such that the claimed combination is directed to a subcombination or a variation of the subcombination.

[0096] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0097] In addition to the embodiments described above, the following examples are also innovative.

[0098] Example 1 is a method comprising: receiving, by a model orchestrator including one or more processors, outcome data representing a set of unattributed outcomes, where the unattributed outcomes in the set of unattributed outcomes have no observed attributes for an exposure in the set of predefined exposures; receiving, by the model orchestrator, attribute data representing a set of modeled attributes from each outcome model of a plurality of outcome models, where each set of modeled attributes comprises a respective measurement value between one or more of the set of unattributed outcomes and one or more of the set of predefined exposures; updating the respective measurements in the set of modeled attributes received from the plurality of outcome models based on one or more criteria; and determining one or more updated attributes based on the respective updated measurements, where each updated attribute indicates a new attribute of a respective outcome from the set of unattributed outcomes for a corresponding exposure in the set of predefined exposures.

[0099] Example 2 is the method of example 1, wherein the modeled attributes of the set of modeled attributes represent attributes of the outcome without attribution to an exposure of the set of predefined exposures.

[0100] Example 3 is the method of examples 1 or 2, further comprising updating one or more modeled attributes of the set of modeled attributes based on the one or more updated attributes, where the updating comprises retracting the modeled attribute such that the corresponding outcome is not attributable to an exposure indicated by the modeled attribute.

[0101] Example 4 is the method of any one of Examples 1-3, further comprising providing the one or more updated attributes to one or more downstream operations that generate a report based on at least the one or more updated attributes.

[0102] Example 5 is the method of any one of Examples 1-4, further comprising generating an attribute report based on the one or more updated attributes.

[0103] Example 6 is the method of any one of Examples 1-5, further comprising determining digital content to be provided to the client device based on the one or more updated attributes, and transmitting data including the digital content to the client device.

[0104] Example 7 is the method of any one of Examples 1-6, wherein each measurement between one or more of the set of unattributed outcomes and one or more of the set of predetermined exposures comprises a probability distribution indicating the likelihood that the unattributed outcome is attributable to the one or more corresponding exposures.

[0105] Example 8 is the method of any one of Examples 1-7, wherein each measurement between one or more of the set of unattributed outcomes and one or more of the set of predetermined exposures comprises a probability distribution indicative of the likelihood that the one or more unattributed outcomes are attributable to a corresponding exposure.

[0106] Example 9 is the method of any one of Examples 1-8, where updating each measurement in the set of modeled attributes based on the one or more criteria comprises, for an unattributed outcome in the set of unattributed outcomes, selecting a set of modeled attributes from one or more of the plurality of outcome models, each of the selected sets of modeled attributes comprising a respective measurement for the unattributed outcome; obtaining from each of the one or more of the plurality of outcome models a respective value of the measurement for the unattributed outcome; and modifying the respective value of the measurement for the unattributed outcome based on the one or more criteria.

[0107] Example 10 is the method of any one of Examples 1-9, wherein the one or more criteria comprise at least one or more of an outcome quantity threshold of an outcome model, an attribute associated with the unattributed outcome, a similarity threshold between the unattributed outcome and a corresponding exposure predicted by the modeled attribute, or a threshold number of remaining unattributed results in the set of unattributed results.

[0108] Example 11 is the method of example 10, where the attribute is a geographic location associated with the unattributed results.

[0109] Example 12 is the method of any one of Examples 1-11, further comprising receiving a request from an entity to determine attributes of the set of unattributed results for a predetermined set of exposures, and in response to receiving the request, generating an output based on one or more updated attributes corresponding to the request.

[0110] Example 13 is a system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform any one of the methods of embodiments 1 to 12.

[0111] Example 14 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing apparatus, to cause the data processing apparatus to perform any one of the methods of embodiments 1 to 12.

[0112] The foregoing has described certain embodiments of the subject matter. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0113] 100 Systems 110 Observational Data 120a~n Attribute Engine 120a First Attribute Engine 120b Second Attribute Engine 120d Attribute Engine 120e Attribute Engine 120m Attribute Engine 120n nth attribute engine 130a~n Resulting Model 130a Resulting Model 130d Resulting Model 130e Resulting Model 135a~n model output 140 Model Orchestrator 145 Unattributed Results 150 requests 155 Output Data 160 downstream engine 200 Model Orchestrator 210 request 215 Configuration Data 220 Request Processor 225 Storage 230 Configuration Engine 235 Unattributed Results 240 Model Output 250 Attribute Optimizer 265 Retraction Engine 270 Updated Attributes 275 Retraced Modeled Attributes 300 processes 400 Computer Systems 410 Processor 420 Memory 430 Storage Devices 440 Input / Output Devices 450 System Bus

Claims

1. receiving, by a model orchestrator including one or more processors, outcome data representing a set of unattributed outcomes, wherein an unattributed outcome in the set of unattributed outcomes does not have an observed attribute to an exposure in a set of predetermined exposures; receiving, by the model orchestrator, attribute data representing a set of modeled attributes from each outcome model of a plurality of outcome models, each set of modeled attributes comprising a respective measure between one or more of the set of unattributed outcomes and one or more of the set of predefined exposures, the respective measure between one or more of the set of unattributed outcomes and one or more of the set of predefined exposures comprising a probability distribution indicative of the likelihood that one or more unattributed outcomes are attributable to one or more corresponding exposures; updating the respective measurements in the set of modeled attributes received from the plurality of outcome models based on one or more criteria, the one or more criteria being determined by user input or pre-setting according to various attribute requirements; determining one or more updated attributes based on the updated respective measurements, each updated attribute indicating a new attribute of a respective result from the set of unattributed results for a corresponding exposure in the set of predetermined exposures; A computer-implemented method comprising:

2. The method of claim 1 , wherein modeled attributes of the set of modeled attributes represent attributes of outcomes without attribution to exposures of the set of predefined exposures.

3. 2. The method of claim 1 , further comprising: updating one or more modeled attributes of the set of modeled attributes based on the one or more updated attributes, wherein the updating comprises retracting a modeled attribute such that a corresponding result is not attributable to an exposure indicated by the modeled attribute.

4. 10. The method of claim 1, further comprising providing the one or more updated attributes to one or more downstream operations that generate a report based at least on the one or more updated attributes.

5. The method of claim 1 , further comprising generating an attribute report based on the one or more updated attributes.

6. determining digital content to be provided to the client device based on the one or more updated attributes; transmitting data including the digital content to the client device; The method of claim 1 , further comprising:

7. updating the respective measurements in the set of modeled attributes based on the one or more criteria, For an unattributed outcome of the set of unattributed outcomes, selecting a set of modeled attributes from one or more of the plurality of outcome models, each of the selected sets of modeled attributes comprising the respective measure values ​​for the unattributed outcome; obtaining a value for each of the measures of the unattributed outcome from each of the one or more of the plurality of outcome models; modifying the respective values ​​of the measurements for the unattributed outcomes based on the one or more criteria; The method of claim 1 , comprising:

8. 2. The method of claim 1 , wherein the one or more criteria comprise at least one or more of an outcome volume threshold for an outcome model, an attribute associated with an unattributed outcome, a similarity threshold between an unattributed outcome and a corresponding exposure predicted by a modeled attribute, or a threshold number of remaining unattributed results in the set of unattributed results.

9. The method of claim 8 , wherein the attribute is a geographic location associated with the unattributed result.

10. receiving a request from an entity to determine attributes of the set of unattributed results to the set of predefined exposures; in response to receiving the request, generating an output based on the one or more updated attributes corresponding to the request; The method of claim 1 , further comprising:

11. A system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of claims 1 to 10.

12. 11. A computer storage medium encoded with a computer program comprising instructions operable, when executed by a data processing apparatus, to cause the data processing apparatus to perform a method according to any one of claims 1 to 10.

Images