Method and system for determining the type of digital component to provide

Abstract

Methods, systems, and apparatus for determining and recommending types of digital components that a content provider can generate and provide for distribution to a client device, including a computer program encoded on a computer storage medium. In one aspect, the method can determine whether a content provider has not previously provided a first digital component of a first media type. A first user interaction dataset can be obtained and input into a machine learning model. The model can output result data related to an expected positive user action associated with the first digital component of the first media type. Based on the result data, a recommendation can be generated and provided to a content provider specifying whether the content provider should provide the first digital component of the first media type.

Description

Technical Field

[0001] This description relates generally to data processing and, more particularly, to determining and recommending types of digital components that a content provider can generate and provide for distribution to client devices. Background Art

[0002] Content providers are typically capable of generating and providing digital components of various media types (e.g., audio, video). Generating digital components may consume significant resources (e.g., computing, network, power resources). Once generated, the content platform is capable of displaying one or more digital components that may be generated and / or provided by the content provider within an application (e.g., a browser) executed on a client device. Summary of the Invention

[0003] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the steps of determining that a content provider has not previously provided a first digital component of a first media type, including determining that a content provider has previously provided digital components of one or more media types different from the first media type; obtaining a first user interaction dataset representing interactions of multiple users with the digital components provided by the content provider; inputting the first user interaction dataset into a machine learning model, wherein: the machine learning model is trained on (i) historical user interaction data of the digital components of the first media type provided by multiple other content providers and (ii) corresponding data of positive user actions associated with the digital components of the first media type, The machine learning model outputs data of expected positive user actions associated with a specific digital component of a first media type based on an input user interaction data set, and the positive user action associated with the digital component represents the user's performance of a target action after an initial user interaction with the digital component; obtains result data of expected positive user actions associated with a first digital component of the first media type from the machine learning model and based on the first user interaction data set; determines whether a specified content provider should provide a recommendation for the first digital component of the first media type based on the result data of the expected positive user actions associated with the first digital component of the first media type; and provides a recommendation to the content provider on whether the specified content provider should provide the digital component of the first media type.

[0004] Other embodiments of this aspect include corresponding methods, apparatus, and computer programs encoded on computer storage devices, the computer programs being configured to perform the actions of the methods.These and other embodiments can each optionally include one or more of the following features.

[0005] In some implementations, the first media type can include one of video, audio, image, or text.

[0006] In some embodiments, the first user interaction data can include data indicating one or more of: one or more characteristics of multiple users, one or more characteristics of multiple client devices corresponding to the multiple users, a number of user interactions with one or more digital components of the first media type, and a duration of user interactions with one or more digital components of the first media type.

[0007] In some embodiments, the machine learning model can use a gradient boosting ensemble technique.

[0008] In some implementations, the machine learning model can correspond to one of a plurality of different machine learning models, each machine learning model corresponding to a different media type.

[0009] In some embodiments, data associated with expected positive user actions for a first digital component of a first media type can include at least one of: a first data item that specifies an expected number of resources consumed in obtaining the expected positive user actions associated with the first digital component of the first media type; or a second data item that represents the expected number of positive user actions associated with the first digital component of the first type relative to the expected number of resources consumed.

[0010] In some embodiments, determining a recommendation can include: determining (1) whether the first data item satisfies a first threshold and (2) whether the second data item satisfies a second threshold; and generating a recommendation based on (1) whether the first data item satisfies the first threshold and (2) whether the second data item satisfies the second threshold, including: generating a recommendation that the specified content provider should provide a first digital component of the first media type when (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold; and generating a recommendation that the specified content provider should not provide the first digital component of the first media type when (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold.

[0011] Specific embodiments of the subject matter described in this specification can be implemented to achieve resource-efficient generation and provision of digital components. Content providers typically expend significant computing, network, and power resources to generate digital components of various media types. Where a content provider has not previously provided digital components of a particular media type (e.g., video), the content provider is typically unable to leverage the resource efficiencies associated with generating digital components of a media type previously generated / provided by the content provider. In other words, when generating a digital component of a new media type (i.e., a media type for which the content provider has not previously provided / generated a digital component), the resources required and consumed are typically greater than the resources required and consumed when generating a digital component of a media type for which the content provider has previously generated / provided a digital component. Furthermore, in such cases, since the content provider is unfamiliar with the provision and distribution of digital components of the new media type, the content provider is typically unable to estimate user interactions and / or affirmative user actions (which represent the execution of a target action by a user after the initial user interaction with the digital component) with respect to the generated digital component of the new media type. In fact, the new digital component may fail to obtain a threshold number of user interactions / affirmative user actions, which in turn may cause the digital component to be decommissioned (i.e., no longer provided to the content platform and / or removed from the stored digital component inventory). In the event that a digital component is taken out of service, the computational, power, and network resources consumed in generating, storing, and providing the digital component are effectively wasted and unrecoverable.

[0012] In contrast, the techniques described herein utilize trained models (e.g., machine learning models) to estimate user interactions and / or affirmative user actions with respect to digital components of new media types before expending resources required to generate, store, and provide such digital components. If a content provider determines that a digital component of a new media type is not estimated to receive a threshold number of user interactions and / or affirmative user actions, the content provider can redirect resources that would otherwise be used to generate and provide the digital component of the new media type to generate and provide digital components of other media types. By doing so, significant resource efficiencies and savings can be achieved in the generation and provision of digital components.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the detailed description below. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example environment in which digital content is distributed and provided for display on client devices.

[0015] Figure 2is a block diagram of an exemplary system for recommending digital components of new media types for distribution.

[0016] Figure 3 is a flow diagram of an example process for recommending a digital component of a new media type for distribution.

[0017] Figure 4 is a block diagram of an example computer system that can be used to perform the described operations. DETAILED DESCRIPTION

[0018] This specification generally describes systems and techniques for determining and recommending the generation and provision of digital components for specific media types, including media types for which content providers have not previously generated and / or provided digital components.

[0019] As outlined below and described in more detail throughout this document, the techniques described herein enable a content provider's system to estimate user interactions and / or affirm user actions with respect to digital components of new media types prior to the expenditure of resources required to generate, store, and provide such digital components.

[0020] In some embodiments, a content provider's system can include a content provider data management engine deployed with one or more machine learning models, and a recommendation generation engine that provides recommendations to administrators of the content provider regarding whether to provide digital components of new media types. As used herein, an engine includes any data processing device that includes hardware and / or software and is configured to perform one or more operations.

[0021] The content provider data management engine communicates with one or more machine learning models to estimate positive user actions expected from the provision and storage of digital components of a new media type (also referred to herein as a "first media type") (i.e., a media type for which the content provider has not previously provided / generated digital components). In some embodiments, the content provider management engine obtains interaction data with respect to the digital components provided by the content provider from a plurality of client devices. The interaction data can include, for example, data indicating one or more characteristics about the client devices and their respective environments, data regarding the nature, number, and / or duration of user interactions with one or more digital components, data regarding positive user actions that occur following an initial user interaction with a digital component of the new media type, or a combination thereof.

[0022] The content provider data management engine provides the obtained interaction data to a machine learning model (e.g., a model using a gradient boosting ensemble technique), which is trained on (i) historical user interaction data for digital components of the new / first media type provided by multiple other content providers and (ii) corresponding data of positive user actions associated with the digital components of the first media type. Based on the input interaction data, the machine learning model outputs values ​​of one or more data items associated with expected positive user actions for a particular digital component of the new / first media type. The set of data items can include, for example, data specifying an expected amount of resources consumed when distributing the digital component of the first media type to multiple users on behalf of the content provider, data specifying an expected number of positive actions performed by users after an initial user interaction with the digital component of the first media type, or a combination thereof.

[0023] Based on this data output by the machine learning model, the recommendation generation engine determines whether the specified content provider should provide a recommendation for a digital component of the new / first media type. In some embodiments, the recommendation generation engine compares the value of each data item (examples of which are described in the previous paragraph) output by the machine learning model with a corresponding predetermined threshold value for the data item. In some embodiments, if the recommendation generation engine determines that the value of each data item meets (e.g., meets or exceeds) the predetermined threshold value for the data item, the recommendation generation engine generates a recommendation that the specified content provider should provide a digital component of the new / first media type. On the other hand, if the recommendation generation engine determines that the value of one or more data items does not meet (e.g., is less than) the predetermined threshold value for the data item, the recommendation generation engine generates a recommendation that the specified content provider should not provide a digital component of the new / first media type.

[0024] References below Figure 1-4 These and additional features are described in more detail further on.

[0025] In addition to the description throughout this document, users may be provided with controls that allow them to choose whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current location), and whether the user sends content or communications from a server. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized (e.g., to a city, zip code, or state level) where location information is available so that the user's specific location cannot be determined. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0026] Figure 1 is a block diagram of an example environment 100 in which digital content is distributed and provided for presentation on client devices.

[0027] Example environment 100 includes a data communication network 105, which can include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 105 can also include any type of wired and / or wireless network, satellite network, cable network, Wi-Fi network, mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. Network 105 can utilize communication protocols, including packet-based and / or datagram-based protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP) or other types of protocols. Network 105 can also include multiple devices that facilitate network communications and / or form the network hardware foundation, such as switches, routers, gateways, access points, firewalls, base stations, repeaters or a combination thereof.

[0028] Network 105 connects client devices 110, publishers 140, websites 142, content platforms 150, and content providers 160. Example environment 100 may include many different client devices 110, publishers 140, websites 142, content platforms 150, and content providers 160.

[0029] The client device 110 is an electronic device capable of communicating over the network 105. Example client devices 110 include personal computers, mobile communication devices such as smartphones, and other devices capable of sending and receiving data over the network 105. The client device can also include a digital assistant device that accepts audio input through a microphone and outputs audio output through a speaker. When the digital assistant detects a "hot word" or "hot phrase" that activates the microphone to accept audio input, the digital assistant can be placed in a listening mode (e.g., ready to accept audio input). The digital assistant device can also include a camera and / or a display to capture images and visually present information. The digital assistant can be implemented in different forms of hardware devices, including wearable devices (e.g., watches or glasses), smartphones, speaker devices, tablet devices, or another hardware device. The client device can also include a digital media device, such as a streaming device that is plugged into a television or other display to stream video to the television.

[0030] The client device 110 typically includes applications 112, such as a web browser and / or native applications, to facilitate sending and receiving data over the network 105. A native application is an application developed for a specific platform or a specific device (e.g., a mobile device with a specific operating system). Publishers 140 can develop and provide native applications to client devices 110, such as for download. For example, in response to a user of the client device 110 entering a resource address of a resource 145 into an address bar of a web browser or selecting a link that references a resource address, the web browser can request the resource 145 from a web server hosting a website 142 of the publisher 140. Similarly, when a native application is executed on the client device 110, the native application can request application content from the publisher's remote server.

[0031] Some resources, application pages or other application content can include a digital component slot for presenting a digital component with a resource 145 or application page / presenting the digital component in a resource 145 or application page. As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another content unit). A digital component can be stored electronically in a physical memory device or in a file collection as a single file, and the digital component can take the form of a video file, an audio file, a multimedia file, an image file, or a text file. For example, a digital component can be content that is intended to supplement the content of a web page or other resource presented by an application 112. More specifically, a digital component can include digital content related to the resource content (e.g., a digital component can relate to a theme that is the same as or otherwise related to the theme / content on the web page). Therefore, providing a digital component can supplement and generally enhance a web page or application content.

[0032] When application 112 loads a resource (or application content) that includes one or more digital component slots, application 112 can request a digital component for each slot. In some embodiments, a digital component slot can include code (e.g., a script) that enables application 112 to request a digital component from a digital component distribution system, which selects the digital component and provides it to application 112 for presentation to a user of client device 110.

[0033] Content platform 150 is a computing platform capable of distributing digital components and other content. Example content platforms 150 include search engines, social media platforms, news platforms, data aggregator platforms, or other content sharing platforms. Each content platform 150 can be operated by a content platform service provider.

[0034] The content platform 150 can publish its own content and make it available. For example, the content platform 150 can be a news platform that publishes its own news articles. The content platform 150 can also present content (e.g., digital components) provided by one or more content providers 160 that are not part of the content platform 150. In the above example, the news platform can also present third-party content provided by one or more content providers 160 and / or publishers 140. As another example, the content platform 150 can be a data aggregator platform that does not publish its own content, but aggregates and presents third-party content provided by different content providers 160 and / or publishers 140. In some embodiments, the content platform 150 manages the selection and distribution of digital components on behalf of the publishers 140 and / or content providers 160.

[0035] The content platform 150 can include a supply-side platform (SSP) and a demand-side platform (DSP). Some publishers 140 use an SSP to manage the process of obtaining digital components for their resources and / or applications. An SSP is a technology platform implemented in hardware and / or software that automates the process of obtaining digital components for resources and / or applications. Each publisher 140 can have a corresponding SSP or multiple SSPs. Several publishers 140 may use the same SSP.

[0036] Content providers 160 can create (or otherwise publish) digital components that appear in digital component slots on publishers' resources and applications. Content providers 160 can use DSPs to manage the provision of their digital components for presentation in these slots. A DSP is a technology platform implemented in hardware and / or software that automates the process of distributing digital components for presentation with resources and / or applications. On behalf of content providers 160, the DSP can interact with multiple supply-side platforms (SSPs) to provide digital components for presentation with resources and / or applications from multiple different publishers 140. Generally speaking, the DSP can receive a request for a digital component (e.g., from an SSP), generate (or select) selection parameters for one or more digital components created by one or more content providers based on the request, and provide data related to the digital component (e.g., the digital component itself) and the selection parameters to the SSP. The SSP can then select the digital component for presentation on client device 110 and provide data to client device 110 that enables presentation of the digital component. As described above, in some examples, content platform 150 can include one or more of the aforementioned SSPs and / or DSPs.

[0037] In operation, the content platform 150 distributes various digital components 151 to different client devices 110 on behalf of the content providers 160. The content platform 150 can be configured to distribute digital components 151 of different media types to different content providers 160. That is, for each of the content providers 160, the content platform 150 can be configured to distribute digital components of a corresponding subset of k different media types. The k different media types can include, for example, video, text, audio, images, etc. For example, the content platform 150 may distribute digital components of a first media type (e.g., video) among k different media types to various client devices 110 on behalf of the content provider 160-2, distribute digital components of a second media type (e.g., text) and digital components of a third media type (e.g., audio) among k different media types to various client devices 110 on behalf of the content provider 160-1, and distribute digital components of a first media type (e.g., video), a second media type (e.g., text), and a fourth media type (e.g., image) among k different media types to various client devices 110 on behalf of the content provider 160-N.

[0038] In operation, the content platform 150 also receives user interaction data 111 from the various client devices 110 to which the digital components 151 are distributed. The user interaction data 111 transmitted by each client device 110 to the content platform 150 can include data indicating one or more characteristics of the corresponding client device 110 and / or the user of the corresponding client device 110 (e.g., device type, device preference information, usage information, contextual information, user profile, user interests, etc.), data indicating user interactions with one or more digital components 151 distributed to the corresponding client device 110 (e.g., data related to time spent viewing one or more digital components, selection (click, touch), etc. of one or more digital components), data indicating affirmative actions performed by the user of the corresponding client device 110 after an initial user interaction with one or more digital components 151 distributed to the corresponding client device 110 (e.g., data related to the user performing one or more predetermined interactions or other user activities after the initial user interaction with this digital component, data related to the user performing one or more predetermined interactions or other user activities with a digital component or resource after being redirected to this digital component / component based on a user interaction with another digital component, etc.), or a combination thereof. For example, an affirmative user action can include interaction with a particular digital component or web page, such as, for example, viewing a particular digital component for a particular duration after an initial interaction with the digital component (e.g., clicking or selecting), or entering and submitting data within a form presented on a web page.

[0039] Thus, after distributing one or more digital components 151 to the respective client devices 110, some portions of the user interaction data 111 may be transmitted to the content platform 150 by each client device 110, while other portions of the user interaction data 111 may be transmitted to the content platform 150 by the respective client device 110 before or after distributing the one or more digital components 151 to the respective client devices 110. Furthermore, when the content platform 150 distributes additional digital components 151 to the client devices 110 on behalf of the content providers 160, the content platform 150 may also receive additional user interaction data 111 from the client devices 110. Upon receiving the user interaction data 111, the content platform 150 may store, process, and / or utilize the user interaction data 111 to generate additional data. As described in greater detail below, in some embodiments, the content platform 150 may utilize the user interaction data 111 received from the client devices 110 to train one or more machine learning models, provide the user interaction data 111 received from the client devices 110 as input to one or more machine learning models, or both.

[0040] In some embodiments, one or more of the content platforms 150 may maintain or otherwise access one or more machine learning models, such as gradient boosted decision trees (GBDT) or ridge regression. In some examples, one or more of these machine learning models may be used in conjunction with the reference Figure 2 One or more of the machine learning models 240 described in further detail may be similar or identical. In some embodiments, the machine learning model can include a separate machine learning model for each of k different digital component media types. Thus, in at least some of these embodiments, one or more of the content platforms 150 can maintain or otherwise access k different machine learning models. For example, in some such embodiments, one or more of the content platforms 150 can maintain or otherwise access a first machine learning model corresponding to a digital component of a first media type (e.g., video), a second machine learning model corresponding to a digital component of a second media type (e.g., text), a third machine learning model corresponding to a digital component of a third media type (e.g., audio), a fourth machine learning model corresponding to a digital component of a fourth media type (e.g., image), and so on. Alternatively, in some embodiments, a single machine learning model can be implemented and the single machine learning model can correspond to digital components of different media types. For convenience, the subsequent disclosure will refer to embodiments of a separate machine learning model for each of the k different digital component media types. However, it should be understood that the same techniques apply to embodiments in which a single machine learning model is used.

[0041] Each machine learning model can be trained based on user interaction data 111 received from client devices 110 to which digital components 151 of the corresponding media type are distributed, and data specifying one or more quantities corresponding to the distribution of digital components 151 of the corresponding media type. This data specifying one or more quantities corresponding to the distribution of digital components 151 of the corresponding media type can correspond to data maintained by one or more of the content platforms 150 and / or determined by one or more of the content platforms 150 based on user interaction data 111 received from client devices 110 to which digital components 151 of the corresponding media type are distributed, data received from content providers 160, and / or other data. In some embodiments, this data specifying one or more quantities corresponding to the distribution of digital components 151 of corresponding media types can include data specifying an amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed in distributing digital components 151 of corresponding media types to client devices 110, data specifying a number of affirmative actions performed by a user of such client device 110 following an initial user interaction with one or more digital components 151 of corresponding media types, data representing a ratio between the amount of the aforementioned resources consumed and the number of the aforementioned affirmative actions performed, or a combination thereof.

[0042] For example, if one or more of the content platforms 150 have distributed one or more digital components of a first media type (e.g., video) to a first subset of client devices 110 on behalf of the content provider 160-2, and have distributed one or more digital components of the first media type (e.g., video) to a second subset of client devices 110 on behalf of the content provider 160-N, the above-mentioned first machine learning model corresponding to the digital components of the first media type (e.g., video) can be trained using user interaction data 111 received from the first and second subsets of client devices 110, as well as one or more quantities of data specifying corresponding to the distribution of the digital components of the first media type (e.g., video) to the first and second subsets of client devices 110.

[0043] In this example, the user interaction data 111 received from each client device in the first or second subset of client devices 110 can include data indicating one or more characteristics of the respective client device 110 and / or a user of the respective client device 110, data indicating user interactions with one or more digital components 151 of the first media type (e.g., video) distributed to the respective client device 110, data indicating affirmative actions performed by the user of the respective client device 110 following an initial user interaction with one or more digital components 151 of the first media type (e.g., video) distributed to the respective client device 110, or a combination thereof.

[0044] Similarly, in this example, in some embodiments, the data specifying one or more quantities corresponding to distributing digital components of a first media type (e.g., video) to the first and second subsets of client devices 110 can include data specifying an amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed in distributing digital components 151 of the first media type (e.g., video) to the first and second subsets of client devices 110, data specifying a number of affirmative actions performed by users of the first and second subsets of client devices 110 following an initial user interaction with one or more digital components 151 of the first media type (e.g., video), data representing a ratio between the amount of the aforementioned resources consumed and the number of the aforementioned affirmative actions performed, or a combination thereof.

[0045] After training, a machine learning model corresponding to a particular media type (e.g., video) can be configured to receive as input user interaction data 111 for a set of client devices 110 to which a content provider provides digital components 151 of one or more media types (e.g., one or more media types in addition to the particular media type, one or more media types including the particular media type), and generate as output data specifying one or more expected quantities corresponding to distributing the digital components of the particular media type to a particular subset of the client devices 110. In this manner, user interaction data 111 obtained in association with distributing digital components of one or more media types on behalf of a given content provider can be used to provide insight into one or more expected quantities corresponding to distribution of digital components of the particular media type, even if digital components of the particular media type have not previously been distributed by the content platform 150 on behalf of the given content provider.

[0046] In some implementations, for each of content providers 160 , one or more of content platforms 150 may determine whether the corresponding content provider has not previously provided a digital component of any of the k different media types for distribution.

[0047] In some embodiments, in response to determining that a corresponding content provider has not previously provided a digital component of, for example, a particular media type (e.g., video) for distribution, one or more of k different machine learning models utilizes user interaction data 111 obtained in association with distribution of digital components (e.g., of one or more media types in addition to the particular media type) on behalf of the corresponding content provider in conjunction with one or more of k different machine learning models to provide one or more insights into expected quantities corresponding to distribution of digital components of the particular media type on behalf of the corresponding content provider. The user interaction data 111 that can be provided as input to one or more of the above-described machine learning models can include one or more portions of the user interaction data 111 obtained from a client device, data determined based on one or more portions of the user interaction data 111, or a combination thereof (as described above).

[0048] For example, if one or more of the content platforms 150 have previously distributed one or more digital components of a second media type (e.g., text) to a particular subset of client devices 110 on behalf of the content provider 160-1, but have not previously distributed one or more digital components of a first media type (e.g., video) to any client device 110 on behalf of the content provider 160-1, one or more of the content platforms 150 may, for example, provide data indicating user interaction data 111 received from the particular subset of client devices 110 as input to the above-mentioned first machine learning model corresponding to digital components of the first media type (e.g., video) to obtain from the first machine learning model and based on the input one or more expected quantities of data specifying distribution of digital components 151 of the first media type (e.g., video) on behalf of the content provider 160-1. In this example, in some embodiments, this data specifying one or more expected quantities corresponding to distributing digital components 151 of a first media type (e.g., video) on behalf of content provider 160-1 can include data specifying an expected amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed when distributing digital components 151 of the first media type (e.g., video) to a particular subset of client devices 110 on behalf of content provider 160-1, data specifying an expected number of affirmative actions performed by users of the particular subset of client devices 110 following an initial user interaction with one or more digital components 151 of the first media type (e.g., video), data representing a ratio between the aforementioned expected amount of resources consumed and the aforementioned expected number of affirmative actions performed, or a combination thereof.

[0049] After obtaining result data from one of the k different machine learning models that specifies one or more expected quantities corresponding to a subset of client devices 110 distributing a digital component of a corresponding one of the k different media types on behalf of a given content provider, in some embodiments, one or more of the content platforms 150 may further generate a recommendation based on the result data and provide the recommendation to the given content provider. This recommendation may specify whether the given content provider should provide the digital component of the corresponding one of the k different media types for distribution. In some embodiments, this recommendation may additionally or alternatively specify at least a portion of the one or more expected quantities. In some examples, this recommendation may be presented to parties associated with the content provider via one or more of a content provider account portal, email, text message, push notification, etc.

[0050] For example, if one or more of the content platforms 150 has provided data indicating user interaction data 111 received from a particular subset of client devices 110 as input to the aforementioned first machine learning model corresponding to digital components of a first media type (e.g., video), and has obtained from the first machine learning model and based on the input one or more expected quantities of result data specifying digital components 151 corresponding to distribution of the first media type (e.g., video) on behalf of the content provider 160-1, then one or more of the content platforms 150 may, as described above, Figure 1 , a recommendation 152 is generated and provided to the content provider 160-1 as depicted in FIG. In this example, the recommendation 152 can specify whether the content provider 160-1 should provide a digital component of a first media type (e.g., video), at least a portion of one or more desired quantities, or a combination thereof.

[0051] In some embodiments, one or more of the content platforms 150 may determine whether one or more expected quantities respectively meet one or more thresholds and generate this recommendation based at least in part on the determination. For example, in response to determining that one or more expected quantities respectively meet (e.g., meet or exceed) one or more thresholds, one or more of the content platforms 150 may generate a recommendation specifying that a given content provider should provide a digital component of a corresponding one of k different media types for distribution. On the other hand, in response to determining that one or more expected quantities respectively do not meet (e.g., do not exceed) one or more thresholds, one or more of the content platforms 150 may (i) generate a recommendation specifying that a given content provider should not provide a digital component of a corresponding one of k different media types for distribution, or (ii) refrain from generating and providing the corresponding recommendation to the given content provider. As described above, in some examples, this recommendation may additionally or alternatively specify at least a portion of the one or more expected quantities and may be presented to parties associated with the given content provider via one or more of a content provider account portal, email, text message, push notification, etc.

[0052] For example, if one or more of the content platforms 150 obtains result data from a machine learning model that specifies one or more expected quantities corresponding to distributing a digital component 151 of a first media type (e.g., a video) on behalf of the content provider 160-1, and determines that the one or more expected quantities exceed one or more thresholds, then the above-mentioned recommendation 152 can specify that the content provider 160-1 should provide the digital component of the first media type (e.g., a video) for distribution. On the other hand, if one or more of the content platforms 150 determines that the one or more expected quantities do not exceed one or more thresholds, then the above-mentioned recommendation 152 can specify that the content provider 160-1 should not provide the digital component of the first media type (e.g., a video) for distribution. As described above, the recommendation 152 can specify whether the content provider 160-1 should provide the digital component of the first media type (e.g., a video), at least a portion of the one or more expected quantities, or a combination thereof.

[0053] This recommendation can advantageously enable content providers 160 to adjust the manner in which they offer digital components for distribution in order to reduce the expected amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed in distributing digital components 151 to client devices 110, increase the expected number of affirmative actions performed by users of such client devices 110 following an initial user interaction with the digital component 151, optimize the ratio between the aforementioned expected amount of resources consumed and the aforementioned expected number of affirmative actions performed, or a combination thereof.

[0054] In some implementations, one or more of the functionality described above with reference to content platform 150 can be provided through one or more other computing devices (not shown) in environment 100 and / or through communication with network 105. Furthermore, in some examples, environment 100 can include one or more relay servers or other appropriate data processing devices (not shown) that serve to relay data between various other components of environment 100 via network 105.

[0055] refer to Figure 2 Described above reference Figure 1 Additional structural and operational aspects of the components are described.

[0056] Figure 2 2 is an exemplary system 200 for recommending digital components of new media types for distribution. More specifically, system 200 includes a content provider data management engine 220 and a recommendation generation engine 250. System 200 also includes or maintains a dataset 230 and a machine learning model 240. Briefly, and as described in further detail below, system 200 receives input data 211 from various client devices and provides output data 252 to various content providers based at least in part on the input data 211.

[0057] In some embodiments, the system 200 can be included as part of the content platform 150 and configured to perform one or more of the operations provided above with reference to the content platform 150. For example, in some examples, the input data 211 and the output data 252 can include the data and the content, respectively, as described above with reference to the content platform 150. Figure 1 Similarly, in some examples, one or more of the machine learning models 240 may correspond to the user interaction data 111 and the recommendation data 152 described above. Figure 1 Thus, machine learning models 240 may include one machine learning model for each of the k different media types.

[0058] The content provider data management engine 220 obtains input data 211 from various client devices over time, which can include user interaction data provided by the various client devices in association with distributing digital components on behalf of various content providers. In some examples, such content providers may be associated with the content provider described above with reference to Figure 1 The content providers 160 described are similar or identical.

[0059] The content provider data management engine 220 stores or otherwise maintains portions of the input data 211 that correspond to / are associated with corresponding content providers as datasets 230. For example, the content provider data management engine 220 may store or maintain input data 211 obtained in association with digital components distributed on behalf of content providers similar to or the same as content provider 160-1 as dataset 230-1, store or maintain input data 211 obtained in association with digital components distributed on behalf of content providers similar to or the same as content provider 160-2 as dataset 230-2, and store or maintain input data 211 obtained in association with digital components distributed on behalf of content providers similar to or the same as content provider 160-N as dataset 230-N. Thus, dataset 230-1 may include data similar to or identical to user interaction data 111 obtained from various client devices to which digital components are distributed on behalf of content provider 160-1, dataset 230-2 may include data similar to or identical to user interaction data 111 obtained from various client devices to which digital components are distributed on behalf of content provider 160-2, and dataset 230-N may include data similar to or identical to user interaction data 111 obtained from various client devices to which digital components are distributed on behalf of content provider 160-N.

[0060] As described above, this user interaction data 111 can include data indicating one or more characteristics of the corresponding client device 110 and / or the user of the corresponding client device 110 (e.g., device type, device preference information, usage information, contextual information, etc.), data indicating user interactions with one or more digital components 151 distributed to the corresponding client device 110 (e.g., data indicating whether the user interacted with the one or more digital components and / or the manner in which the user interacted with the one or more digital components, etc.), data indicating affirmative actions performed by the user of the corresponding client device 110 after the initial user interaction with the one or more digital components 151 distributed to the corresponding client device 110, or a combination thereof.

[0061] In addition, each of the data sets 230 may also include additional data, such as data indicating a digital component provided by the corresponding content provider, data indicating one or more conditions under which the digital component provided by the corresponding content provider is distributed to the client device, data indicating the media type of the digital component previously provided for distribution by the corresponding content provider, etc. In some embodiments, the content provider data management engine 220 can generate or obtain data based at least in part on one or more of the above-mentioned data segments and / or the input data 211. In these embodiments, the content provider data management engine 220 can also store or maintain such generated or obtained data in the data set 230. The data set 230 may also include other information, such as various data segments received from the content provider, information specifying the amount of resources consumed in distributing the digital component on behalf of the content provider (e.g., power, bandwidth, cost, time, etc.), etc.

[0062] The system 200 is capable of training a machine learning model 240 using the input data 211 and / or other data included in the dataset 230. In some examples, each of the one or more machine learning models 240 can be a GBDT, or otherwise utilize one or more gradient boosting ensemble techniques to perform inference. In some embodiments, for each of the k different machine learning models 240, the system 200 is capable of identifying one or more datasets from the dataset 230 that correspond to content providers that have previously provided digital components of the media type corresponding to the corresponding machine learning model, and the system utilizes data from the one or more identified datasets to train the corresponding machine learning model. For example, for a machine learning model 2401 for a first media type (e.g., video), the system 200 can identify one or more datasets from the dataset 230 that correspond to content providers that have previously provided digital components of the first media type (e.g., video), and the system utilizes data from the one or more identified datasets to train the machine learning model for the first media type 2401.

[0063] Such training can be performed initially before the machine learning model 240 is used to generate recommendations, and can optionally be performed on an ongoing basis as the system 200 obtains more data and adds more data to the dataset 230, so as to update and improve the performance of the machine learning model 240 over time. In some implementations, one or more computing devices other than the system 200 can perform one or more of the above-described training processes and simply pass data representing the trained machine learning model 240 to the system 200. In either case, the input data 211 used to train the machine learning model 240 and / or other data included in the dataset 230 can be utilized for the purpose of training the machine learning model 240. Other data can also be used to train the machine learning model 240.

[0064] In some embodiments, once the machine learning model 240 is at least initially trained for each content provider, the content provider data management engine 220 monitors the incoming input data 211 and the data set 230 over time to determine whether a minimum amount of data corresponding to the respective content provider has been obtained.

[0065] Additionally or alternatively, at this time, for each content provider, the content provider data management engine 220 determines whether the corresponding content provider has not previously provided a digital component of any of the k different media types for distribution. In response to determining that the corresponding content provider has not previously provided a digital component of one or more of the k different media types for distribution, the content provider data management engine 220 obtains data corresponding to the corresponding content provider, which may include obtaining input data 211 and / or obtaining data from a corresponding one of the data sets 230, and providing this data as input to one or more of the k different machine learning models 240 that respectively correspond to one or more of the k different media types. In some embodiments, the content provider data management engine 220 obtains this data and provides this data as input to one or more of the k different machine learning models 240 in response to (i) determining that the corresponding content provider has not previously provided a digital component of one or more of the k different media types for distribution, and (ii) determining that a minimum amount of data corresponding to the corresponding content provider has been obtained.

[0066] For example, if the content provider data management engine 220 determines that the content provider corresponding to the dataset 230-1 has previously provided a digital component of a second media type (e.g., text) for distribution, but has not previously provided a digital component of a first media type (e.g., video) for distribution, the content provider data management engine 220 can, for example, obtain input data 211 and / or include in the dataset 230-1 and provide the above data as input to the machine learning model for the first media type 2401.

[0067] After the content provider data management engine 220 provides input data corresponding to a given content provider to one or more of the machine learning models 240, the recommendation generation engine 250 obtains result data from one or more of the machine learning models 240 generated at least in part based on the input data. In some embodiments, the result data obtained from each machine learning model 240 includes data specifying one or more expected quantities corresponding to distributing digital components of the media type corresponding to the corresponding machine learning model 240 on behalf of the given content provider. For example, in at least some of these embodiments, the result data obtained from each machine learning model 240 can include data specifying an expected amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed when distributing digital components of the media type corresponding to the corresponding machine learning model 240 on behalf of the given content provider, data specifying an expected number of affirmative actions performed on behalf of the given content provider by a user of a client device after an initial user interaction with one or more digital components of the media type corresponding to the corresponding machine learning model 240, data representing a ratio between the aforementioned expected amount of resources consumed and the aforementioned expected number of affirmative actions performed, or a combination thereof.

[0068] Based on the result data obtained from the corresponding machine learning model 240, the recommendation generation engine 250 generates a recommendation 252 based at least in part on the result data and provides the recommendation 252 to the given content provider. This recommendation 252 can specify whether the given content provider should provide a digital component of the media type corresponding to the corresponding machine learning model 240 for distribution. In some embodiments, this recommendation 252 can additionally or alternatively specify at least a portion of one or more expected quantities reflected in the result data obtained from the corresponding machine learning model 240. In some examples, the recommendation generation engine 250 provides the recommendation 252 to the given content provider via one or more of an account portal accessible to the given content provider, email, text message, push notification, etc. Thus, in some examples, the recommendation generation engine 250 generates a recommendation 252 via one or more of the following methods: Figure 1A communication network similar to or the same as the described network 105 provides recommendations 252 to a given content provider. In some implementations, the recommendations 252 themselves include digital components of one or more media types.

[0069] In some embodiments, the recommendation generation engine 250 determines whether one or more expected quantities reflected in the result data obtained from the corresponding machine learning model 240 respectively satisfy one or more thresholds, and generates a recommendation 252 based at least in part on the determination. For example, in response to determining that the one or more expected quantities respectively satisfy the one or more thresholds, the recommendation generation engine 250 can generate a recommendation 252 to indicate that the given content provider should provide for distribution a digital component of the media type corresponding to the corresponding machine learning model 240. On the other hand, in response to determining that the one or more expected quantities do not respectively satisfy the one or more thresholds, the recommendation generation engine 250 can generate (i) a recommendation 252 to indicate that the given content provider should not provide for distribution a digital component of the media type corresponding to the corresponding machine learning model 240, or (ii) refrain from generating and providing the corresponding recommendation to the given content provider.

[0070] Figure 3 is a flow chart of an example process 300 for recommending digital components of new media types for distribution. The operation of process 300 is described below as being performed by Figure 1 and 2 The operations of process 300 are performed by components of the system described and depicted in the preceding text, such as one or more content platforms 150 and one or more components of system 200. The operations of process 300 are described below for illustrative purposes only. The operations of process 300 can be performed by any suitable apparatus or system, such as any suitable data processing device. The operations of process 300 can also be implemented as instructions stored on a computer-readable medium, which can be non-transitory. Execution of the instructions causes one or more data processing devices to perform the operations of process 300.

[0071] The system determines that the content provider has not previously provided a first digital component of the first media type (310). In some embodiments, this step corresponds to one or more operations that are similar to or equivalent to the one or more operations performed in conjunction with the content provider data management engine 220 when determining that a given content provider (such as the content provider corresponding to data set 230-1) has previously provided digital components of one or more of k media types other than the first media type (e.g., video) for distribution (described above with reference to FIG). Figure 1 and 2For example purposes, the content provider corresponding to / associated with the data set 230-1 is also referred to as content provider 160-1 hereinafter. In some embodiments, the first media type corresponds to one of video, audio, image, or text.

[0072] The system obtains a first user interaction dataset representing interactions of a plurality of users with a digital component provided by a content provider (320). In some embodiments, this step may correspond to one or more operations similar to or equivalent to one or more operations performed in association with the content provider data management engine 220 obtaining input data 211 and / or data included in the dataset 230-1 from various client devices to which the digital component has been distributed on behalf of the content provider 160-1 (as described above with reference to FIG. Figure 1 and 2 In some embodiments, the first user interaction dataset includes the above reference Figure 1 and 2 The data described, user interaction data, includes data indicating one or more characteristics of multiple users and / or multiple client devices corresponding to the multiple users, and data related to the nature, number and / or duration of user interactions with one or more digital components, or a combination thereof.

[0073] The system inputs the first user interaction data set into the machine learning model (330). In some embodiments, the machine learning model is trained on (i) historical user interaction data of the digital component of the first media type provided by multiple other content providers and (ii) corresponding data of positive user actions related to the digital component of the first media type. Figure 1 and 2 As described, a positive user action associated with a digital component represents a user's performance of a target action following an initial user interaction with the digital component.

[0074] In addition, based on the input first user interaction data set, the machine learning model outputs data of expected positive user actions related to a specific digital component of the first media type based on the input user interaction data set. In some embodiments, and as described above with reference to Figure 1 and 2As described, this step corresponds to one or more operations that are similar to or equivalent to one or more operations performed in association with the content provider data management engine 220 providing input data 211 obtained in conjunction with content provider 160-1 and / or data obtained from dataset 230-1 as input to a machine learning model for the first media type 2401, the machine learning model being trained based on input data 211 obtained from the client device in association with distributing a digital component of the first media type to such client device and / or other data included in one or more datasets 230 (e.g., datasets 230-2 and 230-N) associated with one or more content providers that have previously provided digital components of the first media type for distribution, which datasets may be associated with content providers similar to or equivalent to content providers 160-2 and 160-N, respectively.

[0075] In some embodiments, the machine learning model uses a gradient boosting ensemble technique. In addition, in some embodiments, the machine learning model corresponds to one of a plurality of different machine learning models, each of which corresponds to a different media type.

[0076] The system obtains result data from the machine learning model for the expected positive user action associated with the first digital component of the first media type (340). In some embodiments, this step corresponds to one or more operations that are similar to or equivalent to the one or more operations performed in association with the recommendation generation engine 250 obtaining result data from the machine learning model for the first media type 2401 (as described above with reference to Figure 2 describe).

[0077] As described above, this result data can include data specifying one or more expected quantities corresponding to distributing a digital component of the first media type on behalf of a content provider, and in some embodiments, the data can include data specifying an expected amount of resources consumed when distributing the digital component of the first media type to multiple users on behalf of the content provider, data specifying an expected number of affirmative actions performed by multiple users after an initial user interaction with one or more digital components of the first media type, data representing a ratio between the aforementioned expected amount of resources consumed and the aforementioned expected number of affirmative actions performed, or a combination thereof.

[0078] In some embodiments, data associated with an expected positive user action for a first digital component of a first media type includes: a first data item specifying an expected amount of resources consumed upon obtaining the expected positive user action associated with the first digital component of the first media type; a second data item indicating an expected number of positive user actions associated with the first digital component of the first type relative to the expected amount of resources consumed, or both. In some examples, one or both of the first and second data items may correspond to data indicating one or more of the expected amounts.

[0079] The system determines, based on the result data, whether the specified content provider should provide a recommendation for the first digital component of the first media type (350). In some embodiments, this step corresponds to one or more operations that are similar to or equivalent to the one or more operations performed in association with the recommendation generation engine 250 generating the recommendation 252 based on the result data obtained from the machine learning model for the first media type 2401 (as described above with reference to Figure 2 describe).

[0080] For at least some embodiments, wherein the data associated with the affirmative user action in anticipation of a first digital component of a first media type includes one or both of the first and second data items described above, determining a recommendation based on the result data includes determining (1) whether the first data item satisfies a first threshold and (2) whether the second data item satisfies a second threshold. If the system determines that the first data item satisfies the first threshold and the second data item satisfies the second threshold, the system generates a recommendation that specifies that the content provider should provide the first digital component of the first media type. On the other hand, if the system determines that the first data item does not satisfy the first threshold and / or the second data item does not satisfy the second threshold, the system generates a recommendation that the content provider should not provide the first digital component of the first media type.

[0081] The system provides recommendations to the content provider (360). In some implementations, this step may correspond to one or more operations that are similar to or equivalent to the one or more operations performed in association with the recommendation generation engine 250 providing recommendations 252 to the content provider 160-1 (as described above with reference to FIG. Figure 1 and 2 describe).

[0082] Thus, in this manner, the system provides recommendations to content providers that specify whether the content providers should provide digital components of new media types (i.e., media types for which the content providers have not previously provided digital components). The recommendations can advantageously enable content providers to adjust the manner in which they provide digital components for distribution to reduce the expected amount of resources (e.g., power, bandwidth, cost, time, etc.) consumed in distributing the digital components, increase the expected number of affirmative actions performed by multiple users, optimize the ratio between the expected amount of resources consumed and the expected number of affirmative actions performed, or a combination thereof.

[0083] Figure 4 4 is a block diagram of an example computer system 400 capable of performing the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. Processor 410 can process instructions for execution within system 400. In some embodiments, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 can process instructions stored in memory 420 or on storage device 430.

[0084] Memory 420 stores information within system 400. In one embodiment, memory 420 is a computer-readable medium. In some embodiments, memory 420 is a volatile memory unit. In another embodiment, memory 420 is a non-volatile memory unit.

[0085] The storage device 430 can provide mass storage for the system 400. In some embodiments, the storage device 430 is a computer-readable medium. In various embodiments, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) over a network, or some other mass storage device.

[0086] Input / output device 440 provides the input / output operation of system 400.In some embodiments, input / output device 440 can comprise one or more in the network interface device, for example Ethernet card, serial communication device, and RS-232 port, and / or wireless interface device, for example 802.11 card.In another embodiment, input / output device can comprise driver device, and described driver device is configured to receive input data and output data is sent to peripheral device 460, for example keyboard, printer and display device.Yet, other embodiment can also be used, such as mobile computing device, mobile communication device, set-top box television client device etc.

[0087] Although already Figure 4 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof.

[0088] Embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more computer storage media for execution by, or to control the operation of, data processing equipment. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing equipment. A computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof, or can be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0089] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0090] The term "data processing equipment" encompasses all types of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple systems on a chip, or a combination of the foregoing. The equipment can include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment can also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The equipment and execution environment can implement a variety of different computing model infrastructures, such as network services, distributed computing infrastructure, and grid computing infrastructure.

[0091] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but does not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document) in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file that stores a portion of one or more modules, subroutines, or code). A computer program can be deployed to execute on one computer or on multiple computers, the multiple computers being located at one location or distributed across multiple locations and interconnected by a communications network.

[0092] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and the apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0093] For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, or be operably coupled to receive data from the one or more large-capacity storage devices or transfer data to the one or more large-capacity storage devices, or receive data from the one or more large-capacity storage devices and transfer data to the one or more large-capacity storage devices, the one or more large-capacity storage devices being, for example, magnetic disks, magneto-optical disks, or optical disks. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (for example, a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0094] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, to display information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to a request received from the web browser.

[0095] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with the implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0096] A computing system can include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., due to user interaction) can be received from the client device at the server.

[0097] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, individual features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and a claimed combination may be directed to a subcombination or a variant of a subcombination.

[0098] Similarly, although operations are depicted in a particular order in the figures, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0099] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or any sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising: determining that a content provider has not previously provided a first digital component of a first media type, including determining that the content provider has previously provided digital components of one or more media types other than the first media type; obtaining a first user interaction dataset representing interactions of a plurality of users with a digital component provided by the content provider; The first user interaction dataset is input into a machine learning model, wherein: The machine learning model is trained on (i) historical user interaction data for digital components of the first media type provided by a plurality of other content providers and (ii) corresponding data of affirmative user actions associated with the digital components of the first media type, The machine learning model outputs data of expected positive user actions associated with a specific digital component of the first media type based on an input user interaction dataset; and A positive user action associated with a digital component represents a user's performance of a target action following an initial user interaction with the digital component; obtaining, from the machine learning model and based on the first user interaction dataset, outcome data for an expected positive user action associated with the first digital component of the first media type; determining, based on the result data of the expected positive user action associated with the first digital component of the first media type, whether the designated content provider should provide a recommendation for the first digital component of the first media type; and The recommendation is provided to the content provider specifying whether the content provider should provide the digital component of the first media type.

2. The computer-implemented method of claim 1 , wherein: The first media type includes one of video, audio, image or text.

3. The computer-implemented method of claim 1 , wherein: The first user interaction data includes data indicating one or more of the following: one or more characteristics of the multiple users, one or more characteristics of the multiple client devices corresponding to the multiple users, the number of user interactions with one or more digital components of the first media type, and the duration of user interactions with the one or more digital components of the first media type.

4. The computer-implemented method of claim 1 , wherein: The machine learning model uses gradient boosting ensemble technique.

5. The computer-implemented method of claim 1 , wherein: The machine learning model corresponds to one of a plurality of different machine learning models, each machine learning model corresponding to a different media type.

6. The computer-implemented method according to any one of claims 1 to 5, wherein: The data related to the affirmative user action in anticipation of the first digital component for the first media type includes at least one of: a first data item specifying an expected number of resources to be consumed in obtaining an expected positive user action associated with the first digital component of the first media type; and A second data item representing an expected number of positive user actions associated with the first digital component of the first media type relative to the expected number of resources consumed.

7. The computer-implemented method of claim 6, wherein: Determining the recommendation includes: determining (1) whether the first data item satisfies a first threshold and (2) whether the second data item satisfies a second threshold; and Generating the recommendation based on (1) whether the first data item satisfies the first threshold and (2) whether the second data item satisfies the second threshold, includes: generating a recommendation specifying that the content provider should provide the first digital component of the first media type when (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold; and When (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold, a recommendation is generated specifying that the content provider should not provide the first digital component of the first media type.

8. The computer-implemented method of claim 7, wherein: Providing the recommendation as to whether the content provider should provide the digital component of the first media type includes providing the first data item and the second data item.

9. A system for determining a type of a digital component, comprising: one or more memory devices storing instructions; as well as One or more data processing apparatus configured to interact with the one or more memory devices and, when executing the instructions, perform operations including: determining that a content provider has not previously provided a first digital component of a first media type, including determining that the content provider has previously provided digital components of one or more media types other than the first media type; obtaining a first user interaction dataset representing interactions of a plurality of users with a digital component provided by the content provider; The first user interaction dataset is input into a machine learning model, wherein: The machine learning model is trained on (i) historical user interaction data for digital components of the first media type provided by a plurality of other content providers and (ii) corresponding data of affirmative user actions associated with the digital components of the first media type, The machine learning model outputs data of expected positive user actions associated with a specific digital component of the first media type based on an input user interaction dataset; and A positive user action associated with a digital component represents a user's performance of a target action following an initial user interaction with the digital component; obtaining, from the machine learning model and based on the first user interaction dataset, outcome data for an expected positive user action associated with the first digital component of the first media type; determining, based on the result data of the expected positive user action associated with the first digital component of the first media type, whether the designated content provider should provide a recommendation for the first digital component of the first media type; and The recommendation is provided to the content provider specifying whether the content provider should provide the digital component of the first media type.

10. The system according to claim 9, wherein: The first media type includes one of video, audio, image or text.

11. The system according to claim 9, wherein: The first user interaction data includes data indicating one or more of the following: one or more characteristics of the multiple users, one or more characteristics of the multiple client devices corresponding to the multiple users, the number of user interactions with one or more digital components of the first media type, and the duration of user interactions with the one or more digital components of the first media type.

12. The system according to claim 9, wherein: The machine learning model uses gradient boosting ensemble technique.

13. The system according to claim 9, wherein: The machine learning model corresponds to one of a plurality of different machine learning models, each machine learning model corresponding to a different media type.

14. The system according to any one of claims 9 to 13, wherein: The data related to the affirmative user action in anticipation of the first digital component for the first media type includes at least one of: a first data item specifying an expected number of resources to be consumed in obtaining an expected positive user action associated with the first digital component of the first media type; and A second data item representing an expected number of positive user actions associated with the first digital component of the first media type relative to the expected number of resources consumed.

15. The system according to claim 14, wherein: Determining the recommendation includes: determining (1) whether the first data item satisfies a first threshold and (2) whether the second data item satisfies a second threshold; and Generating the recommendation based on (1) whether the first data item satisfies the first threshold and (2) whether the second data item satisfies the second threshold, includes: generating a recommendation specifying that the content provider should provide the first digital component of the first media type when (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold; and When (1) the first data item satisfies the first threshold and (2) the second data item satisfies the second threshold, a recommendation is generated specifying that the content provider should not provide the first digital component of the first media type.

16. The system according to claim 15, wherein: Providing the recommendation as to whether the content provider should provide the digital component of the first media type includes providing the first data item and the second data item.

17. A non-volatile computer-readable storage medium storing instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: determining that a content provider has not previously provided a first digital component of a first media type, including determining that the content provider has previously provided digital components of one or more media types other than the first media type; obtaining a first user interaction dataset representing interactions of a plurality of users with a digital component provided by the content provider; The first user interaction dataset is input into a machine learning model, wherein: The machine learning model is trained on (i) historical user interaction data for digital components of the first media type provided by a plurality of other content providers and (ii) corresponding data of affirmative user actions associated with the digital components of the first media type, The machine learning model outputs data of expected positive user actions associated with a specific digital component of the first media type based on an input user interaction dataset; and A positive user action associated with a digital component represents a user's performance of a target action following an initial user interaction with the digital component; obtaining, from the machine learning model and based on the first user interaction dataset, outcome data for an expected positive user action associated with the first digital component of the first media type; determining, based on the result data of the expected positive user action associated with the first digital component of the first media type, whether the designated content provider should provide a recommendation for the first digital component of the first media type; and The recommendation is provided to the content provider specifying whether the content provider should provide the digital component of the first media type.

18. The non-volatile computer-readable storage medium of claim 17, wherein: The first media type comprises one of video, audio, image, or text, and wherein the first user interaction data comprises data indicating one or more of: one or more characteristics of the plurality of users, one or more characteristics of a plurality of client devices corresponding to the plurality of users, a number of user interactions with one or more digital components of the first media type, and a duration of user interactions with the one or more digital components of the first media type.

19. The non-volatile computer-readable storage medium of claim 17, wherein: The machine learning model uses a gradient boosting ensemble technique, and wherein the machine learning model corresponds to one of a plurality of different machine learning models, each machine learning model corresponding to a different media type.

20. The non-volatile computer-readable storage medium according to any one of claims 17 to 19, wherein: The data related to the affirmative user action in anticipation of the first digital component for the first media type includes at least one of: a first data item specifying an expected number of resources to be consumed in obtaining an expected positive user action associated with the first digital component of the first media type; and A second data item representing an expected number of positive user actions associated with the first digital component of the first media type relative to the expected amount of resources consumed.

Images