Method for identifying new audiences for content providers' content

Abstract

A method is disclosed for providing a user interface with an option for viewing new audiences to be added to a plurality of users currently designated to receive the content provider's content for display to a content provider, receiving a user selection of the option, and causing the display of information identifying the new audience, the information identifying the new audience including, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content provider's content, and an option for requesting that the corresponding audience be added to the plurality of users currently designated to receive the content provider's content.

Description

[Technical Field]

[0001] Aspects and implementations of the present disclosure relate to identifying new audiences for a content provider's content. [Background technology]

[0002] Content providers frequently select an appropriate group of users who may be interested in their content. The content provider may wish to vary the size of the group of users to include additional users or to limit the users who may be presented with the content. The content provider may not be able to easily identify users that should be included or excluded from the group of users who may be interested in the content. Summary of the Invention [Means for solving the problem]

[0003] The following summary is a simplified summary of the disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to identify key or critical elements of the disclosure, nor to delineate the scope or claims of particular implementations of the disclosure. Its sole purpose is to present some aspects of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] One aspect of the present disclosure provides a computer-implemented method, comprising the steps of: providing, for display to a content provider, a user interface with options for viewing new audiences to be added to a plurality of users currently designated to receive the content of the content provider; receiving a user selection of the options; and causing the display of information identifying the new audiences, the information identifying the new audiences comprising, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content of the content provider, and an option to request that the corresponding audience be added to the plurality of users currently designated to receive the content of the content provider.

[0005] In some implementations, the method further includes receiving a selection of an option to request that a corresponding audience be added to a plurality of users currently designated to receive the content of the content provider, and adding users from the corresponding audience to the plurality of users currently designated to receive the content of the content provider.

[0006] In some implementations, the content provider's content comprises one or more secondary media items to be presented to multiple users in association with the primary media item on the user interface of the content sharing platform.

[0007] In some implementations, the indication of the estimated number of user actions related to the content provider's content comprises an estimated increase in the number of user actions related to the content provider's content in response to the addition of a user from the corresponding audience to a plurality of users currently designated to receive the content of the content provider.

[0008] In some implementations, providing a user interface with an option to view new audiences to be added to a plurality of users currently designated to receive the content of the content provider for display to the content provider further includes collecting data identifying a number of presentations of the content of the content provider to a first group of users and a number of user actions regarding the content of the content provider by the first group of users; identifying a subgroup of users in the first group of users that belong to a second group of users not currently designated to receive the content of the content provider; and determining content-relevance metrics for the identified subgroup of users, wherein the content-relevance metrics for the identified subgroup of users are a number of presentations of the content of the content provider to the identified subgroup of users and a number of user actions regarding the content of the content provider by the identified subgroup of users. estimating content-relevance indicators for a second group of users based on the content-relevance indicators for the identified subgroup of users, wherein the estimated content-relevance indicators for the second group of users comprise an estimated number of presentations of the content provider's content to the second group of users and an estimated number of user actions regarding the content provider's content by the second group of users; predicting an increase in the number of user actions regarding the content provider's content in response to the addition of the second group of users to the first group of users based on the estimated content-relevance indicators for the second group of users; and adding the second group of users to a new audience in response to the predicted increase satisfying a threshold condition, wherein the displayed information identifying the new audience comprises an identifier of the second group of users and the predicted increase in the number of user actions regarding the content provider's content.

[0009] Another aspect of the present disclosure provides a computer-implemented method comprising: collecting content-relevance indicators for a first group of users, the content-relevance indicators for the first group of users comprising a number of presentations of the content of the content provider to the first group of users and a number of user actions regarding the content of the content provider by the first group of users; identifying a subgroup of users in the first group of users who belong to a second group of users who are not currently designated to receive the content of the content provider; determining content-relevance indicators for the identified subgroup of users based on the content-relevance indicators for the first group of users; estimating content-relevance indicators for the second group of users based on the content-relevance indicators of the identified subgroup of users; predicting an increase in the number of user actions regarding the content of the content provider in response to the addition of the second group of users to the first group of users based on the estimated content-relevance indicators of the second group of users; and providing information identifying the second group of users and the estimated increase for presentation to the content provider in response to the estimated increase satisfying a threshold condition.

[0010] In some implementations, estimating content-related metrics for the second group of users further includes determining a number of presentations of the content provider's content to a subgroup of users; determining a number of user actions regarding the content provider's content by the subgroup of users; and determining a conversion rate for the second group of users based on the number of presentations of the content provider's content to the subgroup of users and the number of user actions regarding the content provider's content by the subgroup of users.

[0011] In some implementations, determining a conversion rate for the second group of users further comprises comparing a set of characteristics associated with the subgroup of users with a set of characteristics associated with the second group of users, and the conversion rate for the second group of users is determined based on the comparison of the set of characteristics associated with the subgroup of users with the set of characteristics associated with the second group of users.

[0012] In some implementations, the increase in the number of user actions related to the content of the content provider is predicted using an estimation function.

[0013] In some implementations, the estimation function comprises one or more equations that use a plurality of parameters and a plurality of corresponding weights, the plurality of parameters comprising the number of users in the second group that are not currently designated to receive the content of the content provider, an estimated number of presentations of the content of the content provider to the second group of users, and an estimated number of user actions regarding the content of the content provider by the second group of users.

[0014] In some implementations, the method further includes determining content-related indicators for a first group of users and content-related indicators for a third group of users, where the first group of users and the third group of users were previously designated to receive content from the content provider in the past; obtaining estimated content-related indicators for the third group of users using the determined content-related indicators for the first group of users and the estimation function; and modifying a plurality of weights of the estimation function based on a comparison of the estimated content-related indicators for the third group of users and the determined content-related indicators for the third group of users.

[0015] A further aspect of the present disclosure provides a system comprising a memory and a processing device coupled to the memory, the processing device performing a method according to any aspect or embodiment described herein.

[0016] A further aspect of the present disclosure provides a non-transitory computer-readable medium comprising instructions that, upon execution by a processing device, cause the processing device to perform operations according to any aspect or embodiment described herein.

[0017] Aspects and implementations of the present disclosure will be more fully understood from the detailed description given below and from the accompanying drawings of various aspects and implementations of the present disclosure, which should not be construed as limiting the disclosure to particular aspects or implementations, but are for purposes of explanation and facilitate understanding. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an exemplary system architecture according to an implementation of the present disclosure. [Figure 2] 1 illustrates an exemplary user interface that presents a content provider with a recommendation summary for adding new audiences that should be designated to receive the content of the content provider, in accordance with some aspects of the present disclosure. [Figure 3] FIG. 10 illustrates an exemplary user interface providing details regarding a new audience that is recommended to be added to an existing audience designated to receive content from a content provider, in accordance with some aspects of the present disclosure. [Figure 4] FIG. 10 is an exemplary Venn diagram illustrating how an end user may belong to one or more audiences, according to some aspects of the present disclosure. [Figure 5] 1 is a flow diagram of an example method for providing a content provider with a user interface including new audience recommendations, according to some aspects of the present disclosure. [Figure 6]1 is a flow diagram of an example method for determining new audiences to recommend to content providers, according to some aspects of the present disclosure. [Figure 7] 1 is a flow diagram of an example method for estimating a conversion rate for a group of users not currently designated to receive content from a content provider, according to some aspects of the disclosure. [Figure 8] 1 is a flow diagram of an example method for optimizing an estimation function used to predict an increased number of user actions related to content of a content provider, according to some aspects of the present disclosure. [Figure 9] FIG. 1 is a block diagram illustrating an exemplary computer system according to an implementation of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Aspects of the present disclosure are directed to identifying new audiences for a content provider's content. A content provider typically selects several audiences that are likely to be interested in the content provider's content. Each audience may include a group of end users of a content sharing platform with one or more shared interests. End users within the selected audiences may then be presented with the content provider's content based on a probability p. The content provider's selection may be deemed successful if a large number of end users perform one or more specific actions while being presented with the content provider's content (e.g., selecting the content, subscribing to a class identified / advertised in the content, downloading a mobile application identified / advertised in the content, subscribing to the content provider's channel, etc.). After making the initial selection and seeing its initial success, the content provider may decide to focus on a different set of audiences to increase the number of end users likely to be interested in the content provider's content. Selecting an appropriate group of audiences that may be interested in the content provider's content may be a significant burden for the content provider.

[0020] Existing mechanisms that generate recommendations to content providers about new audiences to target do not include an indication of the increased number of content-related user actions that are expected to occur when a new audience is added to the group of audiences being targeted. Rather, existing mechanisms only indicate how many end users of a new audience have already performed a particular content-related action. If a content provider determines which new audience to target based solely on the number of end users in that audience who have already performed a content-related action, the audience selection may result in the content being presented to a large number of users who are not interested in the content, resulting in an inefficient use of processing resources and a decrease in user trust in the content-sharing platform.

[0021] Implementations of the present disclosure address these and other shortcomings by identifying new audiences for a content provider's content using a function that estimates the likely number of additional content-related user actions that will occur if the new audience is added to the existing audience designated to receive the content of the content provider. The estimation function uses data collected from the presentation of content to existing audiences as well as information about potential new audiences to generate an estimated number of increased content-related user actions. A subset of the best new audiences is then presented to the content provider. To generate the best estimate, data from past content presentations to existing audiences can be used offline to evaluate and improve the accuracy of the estimation function.

[0022] In some implementations, a user interface with an option to view new audiences to be added to users currently designated to receive the content of the content provider is provided for display to the content provider. Upon receiving a user selection of the option to view new audiences to be added, information identifying the new audiences is displayed. The information identifying the new audiences may include, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content of the content, and an option to request that the corresponding new audience be added to users currently designated to receive the content of the content provider. The indication of the estimated number of user actions related to the content may be in the form of an estimated increase in the number of user actions related to the content if users from the corresponding new audience are added to users currently designated to receive the content. The option to request that the corresponding new audience be added to an existing audience may be in the form of a selectable user interface (UI) element (e.g., a button, a link, etc.) presented in association with the identifier of the corresponding new audience.

[0023] In view of the estimated growth for new audiences, the content provider may decide to add one or more of the new audiences to existing audiences by selecting a UI element for each new audience to be added. In response, each selected new audience is added to the existing audiences designated to receive the content provider's content by associating, in the data store, an identifier for each selected new audience with an identifier for the content provider's content. For example, if the content is part of the content provider's advertising campaign, the data store may include information defining the advertising campaign, such as an identifier for the advertising campaign, an identifier for related content (e.g., video advertisements, audio advertisements, etc.), identifiers for one or more targeted audiences, one or more parameters for presenting the content to the targeted audiences (e.g., timing of presenting advertisements relative to primary media items, skippability settings, etc.), and one or more content-related actions (user actions performed when presented with the content) that result in conversions. When a content provider selects a new audience as discussed above, the information defining the advertising campaign in the data store is updated to include an identifier for the selected new audience, and this information can be used by the content sharing platform to present the content provider's content (e.g., secondary content related to a primary media item) to users from the selected new audience.

[0024] In some implementations, new audiences may be identified based on content-related metrics collected for existing audiences that were previously designated to receive the content of the content provider. These content-related metrics may relate to the presentation of the content provider's content to users of the existing audience (a first group of users with a shared interest) and user behavior regarding the content provider's content by the first group of users. The first group of users may include some users (a subgroup of users) that also share an interest with another group of users (a second group of users) that are not currently designated to receive the content of the content provider. For example, the first group of users may include users in the automobile market, the second group of users may include homeowners, and the subgroup of users may include users who are both homeowners and in the automobile market. The second group of users is not currently designated to receive the content of the content provider.

[0025] The content-related metrics collected for a first group of users may be used to determine content-related metrics for a subgroup of users belonging to both the first and second groups of users. The content-related metrics determined for the subgroup of users may include, for example, the number of presentations of the content provider's content to the users of the subgroup and the number of actions related to the content provider's content by the users of the subgroup. The content-related metrics determined for the subgroup of users may further be used to estimate content-related metrics for a second group of users not previously designated to receive content from the content provider. Then, using the estimated content-related metrics for the second group of users, an increase in the number of content-related user actions resulting from the addition of the second group of users to the first group of users is predicted using an estimation function. If the estimated increase exceeds a certain threshold, information identifying the second group of users and the estimated increase is provided for presentation to the content provider.

[0026] In some implementations, the estimation function includes one or more equations that use several parameters and corresponding weights. The parameters may include the number of users in the second group that are not currently designated to receive the content of the content provider, the number of presentations of the content to users in the second group, and the number of actions on the content of the content provider by users in the second group.

[0027] In some implementations, the estimation function may be optimized using an offline process that utilizes content-related indicators collected for existing audiences, such as audience A and audience B, that were previously designated to receive the content provider's content and that have received this content in the past. Audience A may be selected as a base audience, and its content-related indicators may be applied to the estimation function to predict content-related indicators for audience B. Audience B's predicted content-related indicators are then compared to audience B's collected content-related indicators to assess how accurately the estimation function can predict the audience's content-related indicators. If the difference between the two sets of indicators is large (e.g., meets the adjustment condition), the estimation function weights are adjusted (modified) to cause the estimation function to produce an accurate prediction of the content-related indicators for audience B. Alternatively, if the difference between the two sets of indicators is not large (e.g., does not meet the adjustment condition), no adjustment of the estimation function weights is performed.

[0028] Thus, aspects of the present disclosure provide content providers with a mechanism for simplifying the audience selection process by identifying potential new audiences for the content of the content provider and presenting, for each identified new audience, a predicted increase in the number of content-related user actions that would occur if the new audience were also designated to receive the content provider's content. This prevents the content provider from having to consider which additional audiences should also receive the content provider's content in order to increase the number of desired user actions related to the content provider's content. By preventing the content provider from having to consider which additional audiences should receive the content, processing resources are not wasted presenting content to users of the content sharing platform who are no longer interested in the content, user trust and / or interest in the content sharing platform is improved, and content is provided to the appropriate group of users. Furthermore, by optimizing the estimation function offline, improved accuracy is achieved in predicting the increase in the number of user actions related to the content provider's content without having to consume processing resources and network bandwidth for optimizing the estimation function during live operation of the content sharing platform.

[0029] 1 illustrates an exemplary system architecture 100 according to an implementation of the present disclosure. The system architecture 100 (also referred to herein as the “system”) includes end user devices 102A-N, a data store 110, a content sharing platform 120, one or more server machines 130-140, content provider devices 152A-N, and a third-party platform 165, each connected to a network 104.

[0030] In implementations, the network 104 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), a router, a hub, a switch, a server computer, and / or combinations thereof.

[0031] The end user devices 102A-N and the content provider devices 152A-N may each include a computing device such as a personal computer (PC), laptop, mobile phone, smartphone, tablet computer, netbook computer, network-connected television, etc. In some implementations, the end user devices 102A-N may also be referred to as "user devices" or "client devices." Each end user device may include a content viewer. In some implementations, the content viewer may be an application that provides a user interface (UI) for a user to view or upload content such as images, video items, web pages, documents, etc. For example, the content viewer may be a web browser that can access, retrieve, present, and / or navigate content (e.g., web pages such as HyperText Markup Language (HTML) pages, digital media items, etc.) served by a web server. The content viewer may render, display, and / or present the content to a user. A content viewer may also include an embedded media player (e.g., a Flash player or an HTML5 player) embedded in a web page (e.g., a web page that may provide information about products sold by an online merchant). In another example, a content viewer may be a standalone application (e.g., a mobile application or app) that allows a user to view digital media items (e.g., digital video items, digital images, e-books, etc.). According to aspects of the present disclosure, a content viewer may be a content sharing platform application through which a user records, edits, and / or uploads content for sharing on the content sharing platform 120. Thus, a content viewer may be provided to end-user devices 102A-N by the content sharing platform 120.For example, the content viewer may be an embedded media player that is embedded in a web page provided by the content sharing platform 120 .

[0032] Media items 121 may be consumed over the Internet or through a mobile device application, such as a content viewer on end-user devices 102A-N. As previously discussed, requested media items 121 may be requested by a user of content sharing platform 120 for presentation to the user. As used herein, “media,” “media item,” “online media item,” “digital media,” “digital media item,” “content,” and “content item” may include electronic files that may be executed or loaded using software, firmware, or hardware configured to present digital media items to an entity. In one implementation, content sharing platform 120 may store media items 121 using data store 110. In another implementation, content sharing platform 120 may store media items 121 or fingerprints as electronic files in one or more formats using data store 110. The media item 121 may be provided to a user, and providing the media item 121 may comprise one or more of allowing access to the media item 121, transmitting the media item 121, and / or presenting or allowing the presentation of the media item 121.

[0033] In one implementation, media item 121 is a video item. A video item is a set of sequential video frames (e.g., image frames) representing a moving scene. For example, a series of sequential video frames may be continuously captured or later reconstructed to create an animation. Video items may be provided in a variety of formats, including, but not limited to, analog video, digital video, two-dimensional video, and three-dimensional video. Furthermore, a video item may include a movie, a video clip, or any set of animated images to be displayed sequentially. In addition, a video item may be stored as a video file that includes a video component and an audio component. The video component may refer to video data in a certain video or image coding format (e.g., H.264 (MPEG-4 AVC), H.264 MPEG-4 Part 2, Graphic Interchange Format (GIF), WebP, etc.). The audio component may refer to audio data in a certain audio coding format (e.g., advanced audio coding (AAC), MP3, etc.). It may be noted that GIF may be stored as an image file (e.g., a .gif file) or may be stored as a series of images into an animated GIF (e.g., a GIF89a format). It may be noted that H.264 may be a video coding format that is, for example, a block-oriented, motion compensation-based video compression standard for recording, compressing, or distributing video content.

[0034] In some implementations, data store 110 is persistent storage capable of storing media items 121, as well as a data structure for tagging, organizing, and indexing media items 121. Data store 110 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes or hard drives, NAS, SAN, etc. In some implementations, data store 110 may be a network-attached file server, while in other embodiments, data store 110 may be some other type of persistent storage, such as an object-oriented database, a relational database, or one or more different machines that may be hosted by content sharing platform 120 via network 104.

[0035] In one implementation, the content sharing platform 120 or server machines 130-140 may be one or more computing devices (such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that may be used to provide users with access to the media items 121 and / or to provide the media items 121 to users. For example, the content sharing platform 120 may allow users to consume, upload, search, approve (“like”), disapprove (“dislike”), or comment on the media items 121. The content sharing platform 120 may also include websites (e.g., web pages) or application backend software that may be used to provide users with access to the media items 121.

[0036] In implementations of the present disclosure, a "user" may be represented as a single individual. However, other implementations of the present disclosure encompass a "user" being a set of users and / or an entity controlled by an automated source. For example, a set of individual users federated as a community in a social network may be considered a "user." In another example, an automated consumer may be an automated ingestion pipeline, such as a topic channel, of the content sharing platform 120.

[0037] The content sharing platform 120 may include multiple channels (e.g., channels A through Z). A channel may include one or more media items 121 available from a common source or media items 121 with a common topic, theme, or content. The media items 121 may be digital content selected by a user, digital content made available by a user, digital content uploaded by a user, digital content selected by a content provider, digital content selected by a broadcaster, etc. For example, channel X may include videos Y and Z. A channel may be associated with an owner, which is a user who can perform actions on the channel. Various activities may be associated with a channel based on the owner's actions, such as the owner making digital content available on the channel, the owner selecting (e.g., liking) digital content related to another channel, or the owner commenting on digital content related to another channel. Channel-related activities may be aggregated into an activity feed for the channel. Users other than the channel owner can subscribe to one or more channels of interest. The concept of "subscribing" may also be referred to as "liking," "following," "becoming a friend," etc.

[0038] The third-party platform 165 may be used to provide the video advertisements. Alternatively, the third-party platform 165 may provide other services. For example, the third-party platform 165 may be a video streaming service provider that creates a media streaming service via a communication application for users to play videos, TV shows, video clips, audio, audio clips, and movies on end-user devices 102A-N via the third-party platform 165.

[0039] In some implementations, a content provider may upload or otherwise provide media items 121 to the content sharing platform 120 (e.g., via a third-party platform 165) for presentation to one or more users. The content provider may identify an audience to which the media items 121 should be provided. The audience may be defined by specifying categories of interests shared by a group of users. A category may correspond to one or more statuses (e.g., in the market for automobiles, interested in environmentally conscious living, interested in romance movies, interested in baseball, owning property, etc.) or attributes (e.g., occupation, industry related to that occupation, size of the company employing the user, etc.) of users of the content sharing platform 120. A user may belong to a respective category if associated with a status or attribute corresponding to that category. A user may belong to a category if associated with each of the statuses or attributes corresponding to each of the categories. For example, a content provider may indicate that a media item 121 should be provided to users in the market for automobiles. Thus, a user with a status of being in the car market may be provided with the media item 121. In another example, a content provider may indicate that the media item 121 should be provided to lawyers. Thus, a user with the attribute of being a lawyer may be provided with the media item 121. In another example, a content provider may indicate that the media item 121 should be provided to users who are either in the car market or who are lawyers. Thus, a user with the status of being in the car market or the attribute of being a lawyer may be provided with the media item 121.

[0040] A user may be determined to be associated with a status or attribute corresponding to each category based on one or more weights associated with the user, where each of the one or more weights indicates a probability that the user belongs to the respective category. A user of content sharing platform 120 may create a user profile and identify certain information in the user profile (e.g., name, email address, occupation, etc.). The content sharing platform 120 may also include one or more weights in the user profile, each of which indicates a probability that the user belongs to a respective category. The user profile may further include historical data associated with each user. In some implementations, the historical data may include data provided by the user (e.g., data provided when the user creates a user profile for content sharing platform 120, data provided by the user in response to questions prompted by content sharing platform 120, etc.). In other or similar implementations, the historical data may include data collected as a result of the user's interaction with content sharing platform 120. Each user's historical data may be used to determine one or more weights associated with the user.

[0041] After a content provider indicates an audience to which a media item 121 should be presented, content-related metrics may be collected as end users are shown and interact with the content. The content-related metrics may be stored in the data store 110. The content-related metrics may include information about, for example, when a media item 121 (content) was shown to an end user, the length of time the end user viewed the media item 121, which audience the end user belongs to, whether the end user performed an action related to the content, what action the end user performed, etc.

[0042] The server machine 140 may include a new audience identification subsystem 142. The new audience identification subsystem 142 may include a content-related user behavior estimator 146 and a new audience GUI creator 144. In some implementations, the content-related user behavior estimator 146 may predict how much additional user behavior related to the content provider's content will occur if a new audience is added to the audiences already designated to receive the content. As further discussed with reference to the exemplary Venn diagram shown in FIG. 4, the content-related user behavior estimator 146 may use content-related metrics collected from audiences already designated to receive the content to make predictions about the effect of adding a new audience.

[0043] 4, a content provider may initially designate two audiences (e.g., audience B 420 and audience C 430) to receive the content. As the content is shown to end users in audience B 420 or audience C 430, content-related metrics may be collected. The metrics may include how many end users in audience B 420 were shown the content (e.g., number of impressions) and how many end users in audience B 420 performed actions related to the content (e.g., number of conversions). The metrics may also include how many end users in audience C 430 were shown the content (e.g., number of impressions) and how many end users in audience C 430 performed actions related to the content (e.g., number of conversions). An impression rate for audience B 420 may be calculated based on the number of end users in audience B 420 who were shown the content (number of impressions) compared to the total number of end users in audience B 420. A conversion rate for audience B 420 may be calculated based on the number of end users in audience B 420 who performed an action related to the content (number of conversions) compared to the number of impressions for audience B 420. For example, if audience B 420 includes 1000 end users and 430 end users were shown the content, the impression rate is 43% (i.e., 430 / 1000). If 43 of the end users who were shown the content performed an action related to the content, the conversion rate is 10% (i.e., 43 / 430). Impression rates and conversion rates for audience C 430 may be calculated similarly.

[0044] In one example, the content-related user behavior estimator 146 may predict an increased number of user actions related to the content of a content provider in response to audience A 410 being added to audience B 420 and audience C 430 as designated to receive the content. The content-related user behavior estimator 146 has content-related indicators associated with end users currently designated to receive the content (i.e., end users in audience B 420 and end users in audience C 430). Because an end user may belong to more than one audience (e.g., may be associated with more than one interest category), the content-related user behavior estimator 146 may also have content-related indicators associated with some end users in audience A 410. For example, the shaded area 440 indicates end users who are in either audience B 420 or audience C 430 (or both) and who are also in audience A 410. The content-related user behavior estimator 146 may estimate content-related metrics for end users in the shaded area 415, which indicates end users who are not currently designated to receive the content (i.e., end users who are not in audience B 420 and not in audience C 430).

[0045] The content-related user behavior estimator 146 may operate based on the assumption that the impression rates of end users in the shaded region 415, the impression rates of end users in the shaded region 440, the impression rates of end users in audience B 420, and the impression rates of end users in audience C 430 are all the same. Based on this assumption, the content-related user behavior estimator 146 may estimate the number of impressions of end users in audience A 410 by multiplying the impression rates of end users in the shaded region 440 by the total number of end users belonging to audience A 410.

[0046] In another example, the content-related user behavior estimator 146 may estimate the number of impressions for end users in audience A 410 by calculating an average impression rate for end users in the shaded area 440 using the impression rates of end users in audience B 420 and the impression rates of end users in audience C 430 and multiplying that average impression rate by the total number of end users in audience A 410.

[0047] The content-related user behavior estimator 146 may also operate based on the assumption that the conversion rate of end users in the shaded region 415 is the same as the conversion rate of end users in the shaded region 440. In some implementations, the content-related user behavior estimator 146 may calculate the conversion rate of end users in the shaded region 440 as the average of the conversion rate of end users in audience B 420 and the conversion rate of end users in audience C 430. Using these assumptions, the content-related user behavior estimator 146 may estimate an increased number of end-user actions related to the content (e.g., an increased number of conversions) that occur in response to audience A 410 being added to audience B 420 and audience C 430 as designated to receive the content provider's content. The increased number of end-user actions on the content by end users in audience A 410 (e.g., the estimated number of conversions for audience A 410) may be equal to the conversion rate of the shaded area 440 multiplied by the estimated number of impressions of end users in audience A 410 as calculated above. In some implementations, because the actions performed by users in the shaded area 440 have already been taken into account when calculating the conversion rates for users in audience B 420 and audience C 430, the increased number of end-user actions on the content by end users in audience A 410 (e.g., the estimated number of conversions for audience A 410) may be equal to the conversion rate of the shaded area 440 multiplied by the estimated number of impressions of end users in the shaded area 415 as calculated above.

[0048] In some implementations, the content-related user behavior estimator 146 may operate under the assumption that the content-related metrics of end users in the shaded region 415 are significantly different from the content-related metrics of end users in the shaded region 440. In such implementations, a trained machine learning model may be used to provide the content-related metrics of end users in the shaded region 415 based on the content-related metrics of end users in audience B 420, the content-related metrics of end users in audience C 430, characteristics of end users in audience B 420, characteristics of end users in audience C 430, and characteristics of end users in audience A 410. The machine learning model may be trained on past content-related metrics and characteristics collected for existing audiences that were previously designated to receive the content provider's content and that have received this content in the past.

[0049] In some implementations, the content-related user behavior estimator 146 uses weights to calculate estimated content-related metrics for end users in audience A 410. Initially, the weights are assigned using default values ​​and then may be adjusted to improve results. For example, the content-related user behavior estimator 146 may compare an additional set of characteristics associated with end users in the shaded region 415 with an additional set of characteristics associated with end users in the shaded region 440. Based on how similar or different the additional sets of characteristics are between the two groups of end users, the content-related user behavior estimator 146 may add a certain number of weights to the formula used to calculate estimated content-related metrics for end users in audience A 410. For example, if the difference between the additional sets of characteristics is greater than some threshold (e.g., meets an adjustment condition), the estimated number of impressions for end users in audience A 410 may be equal to the weight value multiplied by the impression rate of end users in the shaded region 440 multiplied by the total number of end users belonging to audience A 410. Weight values ​​may also be added to a formula used to calculate the estimated number of conversions for end users in audience A 410 based on the conversion rates of end users in shaded area 440. The weight values ​​used in the formula to estimate the number of impressions for end users in audience A 410 may be different from the weight values ​​used in the formula to estimate the number of conversions for end users in audience A 410.

[0050] In some implementations, the weight values ​​used by the content-related user behavior estimator 146 in the estimation equation can be adjusted using an estimation function optimizer 132 on the server machine 130. In some implementations, the estimation function optimizer 132 can be part of the same server machine 140 that estimates user-related content behavior. The estimation function optimizer 132 can adjust the weight values ​​used by the content-related user behavior estimator 146 using past content-related indicators collected for existing audiences that were previously designated to receive the content provider's content and that have received this content in the past. For example, the content-related behavior estimator 146 can use past content-related indicators from audience D and audience E. Audience D can be selected as a base audience, and the content-related indicators of audience D can be applied to the estimation function to predict content-related indicators for audience E. The predicted content-related indicators of audience E are then compared to the collected content-related indicators of audience E to evaluate how accurately the estimation function can predict the audience's content-related indicators. If the difference between the two sets of metrics is significant (e.g., the adjustment condition is met), the estimation function weights are adjusted (modified) to cause the estimation function to produce an accurate prediction of content-related metrics for audience E. Alternatively, if the difference between the two sets of metrics is not significant (e.g., the adjustment condition is not met), no adjustment of the estimation function weights is performed. The estimation function optimizer 132 can be run offline (e.g., as a back-end process that runs independently and without responding to content provider interactions or requests), and therefore does not need to consume resources for the live operation of the content sharing platform.

[0051] In some implementations, the content-related user behavior estimator 146 may calculate an estimated increase in the number of user actions related to the content provider's content for multiple audiences not currently designated to receive the content provider's content, regardless of whether these audiences overlap with existing audiences that have been or are currently targeted for the content provider's content. Alternatively, the content-related user behavior estimator 146 may calculate an estimated increase in the number of user actions related to the content provider's content only for each audience not currently designated to receive the content provider's content for which some content-related metrics have already been collected as a result of a particular end user belonging to multiple audiences.

[0052] After calculating the estimated lift associated with the new audiences, the content-related user behavior estimator 146 may select a subset of the new audiences to recommend to the content provider. For example, the content-related user behavior estimator 146 may select the audiences with the highest predicted conversion rates. In another example, the content-related user behavior estimator 146 may select the audiences predicted to generate the most conversions. The content-related user behavior estimator 146 may select all audiences whose predicted lift in conversions exceeds a threshold. In some implementations, the threshold may be selected by the content provider.

[0053] Once a subset of audiences to recommend to the content provider is selected, information associated with each new audience in the subset may be used by new audience GUI generator 144 to create a graphical user interface (GUI) to be presented to the content provider. The created GUI may include a representation of the new audience's identifier, an estimated increase in the number of end-user actions related to the content that would occur if the new audience were added to the audiences designated to receive the content provider's content, and an option to add the corresponding audience to audiences currently designated to receive the content provider's content. The content provider devices 152A-N may present the GUI 154 provided by new audience GUI generator 144. Exemplary GUIs 154 are discussed in more detail below with respect to FIGS. 2 and 3.

[0054] Further to the above, a user (end user or content provider) may be provided with controls that allow the user to exercise choice regarding both whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social behavior, content-related behavior or activities, occupation, user preferences, or the user's current location) and whether the user is sent content or communications from a server. Additionally, some data may be handled in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be handled so that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized (e.g., to the city, zip code, or state level) where location information is obtained so that the user's specific location cannot be determined. Thus, the user has control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0055] 2 shows an exemplary user interface that presents a content provider with a recommendation summary for adding new audiences to be designated to receive the content of the content provider, according to some aspects of the present disclosure. User interface 200 may include at least a side menu 202, a recommendation options UI element (e.g., a button) 204, an overall optimization score indicator 206, and a new audience recommendation card 208. User interface 200 may also include other recommendation cards and other UI components.

[0056] When a user selects the recommendation options button 204, the user interface 200 can present the content provider with an interface with one or more recommendations. The interface can include an overall optimization score indicator 206 and a new audience recommendation card 208. The overall optimization score indicator 206 can indicate to the content provider how close their content presentation configuration is to being optimal. The overall optimization score indicator 206 can represent an optimization percentage, with 100% being the highest. The new audience recommendation card 208 can include text describing the recommendation to the content provider. The new audience recommendation card 208 can include a representation of an overall estimated increase in expected conversions 210 if all of the current new audience recommendations are accepted by the content provider. The new audience recommendation card 208 can include a representation of an increase in overall optimization score 212 that would occur if all of the current new audience recommendations are accepted by the content provider. The new audience recommendation card 208 can include a view recommendations button 214. When selected, the view recommendations button 214 may cause the presentation of the example user interface 300 of Figure 3. The new audience recommendation card 208 may include an apply all button 216. When selected, the apply all button 216 may cause the new audience identification subsystem 142 to add audiences corresponding to the new audiences associated with each of the new audience recommendations to the audiences designated to receive the content provider's content.

[0057] 3 shows an exemplary user interface that provides details about new audiences recommended to be added to existing audiences designated to receive a content provider's content, according to some aspects of the present disclosure. User interface 300 may include at least a side menu 302, a recommendation options UI element (e.g., a button) 304, a new audience reach summary card 306, and a new audience recommendations table 318. User interface 300 may also include other recommendation cards and other UI components.

[0058] When a user selects the view recommendations button 214, the user interface 300 can present the content provider with an interface with one or more recommendations. The interface can include a new audience reach summary card 306 and a new audience recommendations table 318. The new audience reach summary card 306 can include a representation of the increase in overall optimization score 212 that would occur if all of the current new audience recommendations were accepted by the content provider. The new audience reach summary card 306 can include a return to recommendations button 310. When selected, the return to recommendations button 310 can cause the presentation of the example user interface 200. The new audience reach summary card 306 can include a download button 312, a reject all button 314, and an apply all button 316. When selected, the download button 312 can cause the new audience identification subsystem 142 to generate a digital file to be downloaded by the content provider. The digital file can include information about the new audience recommendations. For example, the digital file may contain some or all of the information in the new audience recommendation table 318. The reject all button 314, when selected, may cause the new audience identification subsystem 142 to remove the current new audience recommendations from presentation to the content provider. The apply all button 316, when selected, may cause the new audience identification subsystem 142 to add audiences corresponding to the new audiences associated with each new audience recommendation to the audiences designated to receive the content provider's content.

[0059] The user interface 300 may include a new audience recommendation table 318. The new audience recommendation table 318 may include one or more new audience recommendation rows 320. The new audience recommendation rows 320 may include a recommendation checkbox 322, an ad group identifier 324, a campaign identifier 326, an audience identifier 328, an indication of estimated conversion lift 330, an apply recommendation button 332, and a dismiss recommendation button 334. The recommendation checkbox 322, when selected in one or more rows, may cause the apply recommendation button 332 or the dismiss recommendation button 334 to appear in the respective row. The apply recommendation button 332, when selected, may cause the new audience identification subsystem 142 to add the audience corresponding to the new audience associated with the new audience recommendation to the audiences designated to receive the content provider's content. The reject recommendation button 334, when selected, may cause the new audience identification subsystem 142 to remove the new audience recommendation row 320 from presentation to content providers.

[0060] FIG. 5 illustrates a flow chart of an example method 500 for providing a content provider with a user interface including new audience recommendations, according to some aspects of the present disclosure. FIG. 6 illustrates a flow chart of an example method 600 for determining new audiences to recommend to a content provider, according to some aspects of the present disclosure. FIG. 7 illustrates a flow chart of an example method 700 for estimating a conversion rate for a group of users not currently designated to receive the content provider's content, according to some aspects of the present disclosure. FIG. 8 illustrates a flow chart of an example method 800 for optimizing an estimation function used to predict an increased number of user actions related to the content provider's content, according to some aspects of the present disclosure. Methods 500, 600, 700, and 800 may be performed by processing logic, which may include hardware (circuitry, dedicated logic circuitry, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. In one implementation, some or all of the operations of methods 500, 600, 700, and 800 may be performed by one or more components of system 100 of FIG. 1.

[0061] 5, at block 510, the processing logic may provide a user interface with an option to view new audiences to be added to a plurality of users currently designated to receive the content provider's content, for display to the content provider. The user interface may be similar to user interface 200 as described above.

[0062] At block 520, the processing logic may receive a user selection of an option to view a new audience to be added to the plurality of users currently designated to receive the content provider's content. This user selection may be an interaction by the content provider with the user interface 200. For example, the user selection may include the content provider clicking or selecting the view recommendations button 214 as described above.

[0063] At block 530, the processing logic may cause the display of information identifying the new audiences, where the information identifying the new audiences comprises, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content provider's content, and an option for requesting that the corresponding audience be added to the plurality of users currently designated to receive the content provider's content. The display of the information may be similar to user interface 300. For example, the audience identifier, the indication of the estimated number of user actions related to the content provider's content, and the option for requesting that the corresponding audience be added to the plurality of users currently designated to receive the content provider's content may be audience identifier 328, an indication of estimated conversion lift 330, and apply recommendation button 332, respectively, as described above.

[0064] At block 540, the processing logic may receive a user selection of an option to request that the corresponding audience be added to the plurality of users currently designated to receive the content provider's content. This user selection may be an interaction by the content provider with the user interface 300. For example, the user selection may include the content provider clicking or selecting the apply recommendation button 332 as described above.

[0065] At block 550, the processing logic may add users from the corresponding audience to the plurality of users currently designated to receive the content of the content provider.

[0066] As discussed above, Figure 6 shows a flow diagram of an example method 600 for determining new audiences to recommend to content providers according to some aspects of the present disclosure. At block 610, the processing logic may collect content-related metrics for a first group of users, the content-related metrics comprising a number of presentations of the content to the first group and a number of user actions related to the content by the first group. In one example, the first group of users may include end users in audience B 420 and audience C 430.

[0067] At block 620, the processing logic may identify a subgroup of users in the first group of users that belong to a second group of users who are not currently designated to receive the content from the content provider. For example, the second group of users who are not currently designated to receive the content from the content provider may be audience A 410, and the subgroup of users within the first group of users may include the end users in the shaded area 440.

[0068] At block 630, the processing logic may determine content-related metrics for the identified subgroup of users. The content-related metrics may include, for example, a number of impressions for the subgroup of users, a number of conversions for the subgroup of users, an impression rate for the subgroup of users, a conversion rate for the subgroup of users, etc. The content-related metrics for the subgroup may be determined according to implementations previously described.

[0069] At block 640, the processing logic may estimate content-related metrics for a second group of users based on the content-related metrics of the identified subgroup of users. The content-related metrics for the second group of users may include, for example, a number of impressions for the second group of users, a number of conversions for the second group of users, an impression rate for the second group of users, a conversion rate for the second group of users, etc. In some implementations, the content-related metrics for the second group of users may be estimated according to method 700.

[0070] At block 650, the processing logic may predict an increase in the number of user actions regarding the content of the content provider in response to adding the second group of users to the first group of users based on the estimated content-relevance indicators of the second group of users. For example, the processing logic may predict an increase in the number of user actions regarding the content of the content provider in response to adding audience A 410 to audience B 420 and audience C 430 currently designated to receive the content of the content provider according to a previously described implementation.

[0071] At block 660, as described above, the processing logic may provide information identifying a second group of users and the estimated increase for presentation to the content provider in response to the estimated increase satisfying a threshold condition.

[0072] As discussed above, FIG. 7 shows a flow diagram of an example method 700 for estimating a conversion rate for a group of users not currently designated to receive content from a content provider, according to some aspects of the present disclosure. At block 710, the processing logic may determine a number of presentations of the content provider's content to a subgroup of users. At block 720, the processing logic may determine a number of user actions regarding the content provider's content by the subgroup of users. At block 730, the processing logic may determine a conversion rate for a second group of users based on the number of presentations of the content provider's content to the subgroup of users and the number of user actions regarding the content provider's content by the subgroup of users. As described above, in some implementations, the processing logic may operate based on the assumption that the conversion rate for the second group of users is the same as the conversion rate for the subgroup of users. Alternatively, the processing logic may operate based on the assumption that the conversion rate for the second group of users is significantly different from the conversion rate for the subgroup of users, in which case a trained machine learning model or a comparison of additional characteristics of the groups, as described above, may be used to determine the conversion rate for the second group of users.

[0073] As discussed above, FIG. 8 shows a flow diagram of an example method 800 for optimizing an estimation function used to predict an increased number of user actions related to a content provider's content, according to some aspects of the present disclosure. At block 810, the processing logic may determine content-related metrics for a first group of users and a third group of users. Both the first group of users and the third group of users may have previously been designated to receive the content provider's content in the past and may have previously received the content. The content-related metrics for each group may include, for example, the number of presentations of the content provider's content to the group, the number of user actions related to the content provider's content by the group, the impression rate for the group, the conversion rate for the group, etc.

[0074] In block 820, the processing logic may estimate content-relevance indicators for a third group of users using the determined content-relevance indicators for the first group of users and an estimation function. The estimation function may be the same function used to estimate the content-relevance indicators in block 640.

[0075] At block 830, the processing logic may modify the weights of the estimation function based on a comparison of the estimated content-relevance indicators for the third group of users and the determined content-relevance indicators for the third group of users. As described above, the weights of the estimation function may be modified to reduce the difference between the estimated content-relevance indicators for the third group of users and the determined content-relevance indicators for the third group of users.

[0076] FIG. 9 is a block diagram illustrating an exemplary computer system according to an implementation of the present disclosure. The computer system 900 may be a server machine 130-140, an end-user device 102A-N, or a content provider device 152A-N of FIG. 1. The machine may operate as a server or endpoint machine in an endpoint-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a server, a network router, a switch, or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine is shown, the term “machine” shall be interpreted to include any collection of machines that, individually or collectively, execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein.

[0077] The exemplary computer system 900 includes a processing device (processor) 902, a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 916, which communicate with each other via a bus 630.

[0078] Processor (processing device) 902 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), or the like. More specifically, processor 902 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. Processor 902 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. Processor 902 may include processing logic 922 for performing operations discussed herein (e.g., to identify new audiences for a content provider's content and / or to provide recommendations regarding identified new audiences for presentation to the content provider).

[0079] Computer system 900 may further include a network interface device 908. Computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), input devices 912 (e.g., a keyboard, an alphanumeric keyboard, a motion-sensing input device, a touch screen), cursor control devices 914 (e.g., a mouse), and signal generation devices 918 (e.g., a speaker).

[0080] The data storage device 916 may include a non-transitory machine-readable storage medium 924 (also a computer-readable storage medium) on which one or more sets of instructions 926 (e.g., for identifying new audiences for content of a content provider and / or for providing recommendations regarding identified new audiences for presentation to content providers) that embody any one or more of the methods or functions described herein. The instructions may also reside, completely or at least partially, in the main memory 904 and / or the processor 902 during execution by the computer system 900, with the main memory 904 and the processor 902 also constituting machine-readable storage media. The instructions may further be transmitted or received over the network 920 via the network interface device 908.

[0081] In one implementation, instructions 926 include instructions for identifying new audiences for the content of the content provider and / or for providing recommendations regarding the identified new audiences for presentation to the content provider. While the exemplary implementation illustrates computer-readable storage medium 924 (machine-readable storage medium) as being a single medium, the terms “computer-readable storage medium” and “machine-readable storage medium” should be interpreted to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of instructions. The terms “computer-readable storage medium” and “machine-readable storage medium” should also be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing a machine to perform any one or more of the methods of the present disclosure. Accordingly, the terms “computer-readable storage medium” and “machine-readable storage medium” should be interpreted to include, but are not limited to, solid-state memory, optical media, and magnetic media.

[0082] Throughout this specification, a reference to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with an implementation is included in at least one implementation. Thus, appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification can, but do not necessarily, refer to the same implementation, depending on the context. Furthermore, in one or more implementations, particular features, structures, or characteristics may be combined in any suitable manner.

[0083] To the extent that the terms "include," "including," "have," "containing," variations thereof, and other similar terms are used in either the detailed description or the claims, these terms are intended to be as inclusive as the term "comprise," as open transitional terms, without excluding any additional or other elements.

[0084] In this application, terms such as “component,” “module,” and “system” are generally intended to refer to a computer-related entity, whether hardware (e.g., a circuit), software, or a combination of hardware and software, or an entity relating to an operable machine having one or more specific functions. For example, a component may be, but is not limited to, a process running on a processor (e.g., a digital signal processor), a processor, an object, an executable file, a thread of execution, a program, and / or a computer. By way of example, both an application running on a controller and the controller may be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed across two or more computers. Furthermore, a “device” may be in the form of specially designed hardware, general-purpose hardware specialized by the execution of software that enables the hardware to perform specific functions (e.g., generating points of interest and / or descriptors), software on a computer-readable medium, or a combination thereof.

[0085] The foregoing systems, circuits, modules, etc. have been described with respect to interactions between several components and / or blocks. It will be understood that such systems, circuits, components, blocks, etc. may include those components or designated subcomponents, portions of the designated components or subcomponents, and / or additional components, according to various permutations and combinations of the foregoing. Subcomponents may also be implemented as components that are not contained in a parent component (hierarchical) but are communicatively coupled to other components. In addition, it should be noted that one or more components may be combined into a single component that provides a collective functionality or may be divided into several separate subcomponents, and that any one or more intermediate layers, such as a management layer, may be provided to communicatively couple such subcomponents to provide integrated functionality. Any component described herein may also interact with one or more other components not specifically described herein but known to those skilled in the art.

[0086] Moreover, the words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the words "example" or "exemplary" is intended to present the concept in a concrete manner. In this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X utilizes A or B" is intended to mean any of the natural inclusive permutations. That is, "X utilizes A or B" is satisfied under any of the foregoing cases if X utilizes A, if X utilizes B, or if X utilizes both A and B. Additionally, the articles "a" and "an," as used in this application and the appended claims, should generally be construed to mean "one or more," unless otherwise specified or unless it is clear from the context that the singular form is intended.

[0087] Finally, implementations described herein include collections of data describing users and / or user activities. In one implementation, such data is collected only when the user consents to the collection of this data. In some implementations, the user is prompted to explicitly authorize data collection. Additionally, users may opt in to or opt out of participating in such data collection activities. In one implementation, the collected data is anonymized before any analysis is performed to obtain statistical patterns, so that the identity of the user cannot be determined from the collected data. [Explanation of symbols]

[0088] 100 System Architecture 102 End User Devices 104 Network 110 Datastore 120 Content Sharing Platform 121 Media Items 130 Server Machine 132 Estimation Function Optimizer 140 Server Machine 142 New Audience Identification Subsystem 144 New Audience GUI Creator 146 Content-related user behavior estimator 152 Content provider device 154 New Audience GUI 165 Third-Party Platforms 200 User Interface 202 Side Menu 204 Recommended Optional UI Elements 206 Global Optimization Score Indicator 208 New Audience Recommendation Cards 210 expected conversions 212 Overall Optimization Score 214 View Recommendations Button 216 Apply All button 300 User Interface 302 Side Menu 304 Recommended Option UI Elements 306 New Audience Reach Summary Card 310 Back to Recommendations button 312 Download Button 314 Reject All Button 316 Apply All button 318 New Audience Recommendation Table 320 New Audience Recommendations 322 Recommended Checkboxes 324 Ad Group Identifier 326 Campaign Identifier 328 Audience Identifier 330 Indicators of Estimated Conversion Increase 332 Recommended Apply Button 334 Recommendation Rejection Button 410 Audience A 415 Shaded area 420 Audience B 430 Audience C 440 Shaded area 900 Computer Systems 902 Processing Device 904 Volatile Memory 906 Non-volatile memory 908 Network Interface Device 910 Video Display Unit 912 Alphanumeric Input Device 914 Cursor Control Device 916 Data storage device 918 Signal Generating Device 920 Network 922 Processing Logic Circuit 924 Computer-readable storage medium 926 command 930 Bus

Claims

1. 1. A computer-implemented method comprising: providing a user interface for display to a content provider with an option to view new audiences to be added to a plurality of users currently designated to receive content from said content provider; receiving a user selection of said option; and causing the display of information identifying the new audience, wherein the information identifying the new audience comprises, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content of the content provider, and an option to request that the corresponding audience be added to the plurality of users currently designated to receive the content of the content provider.

2. receiving a selection of the option to request that the corresponding audience be added to the plurality of users currently designated to receive the content of the content provider; and adding users from the corresponding audience to the plurality of users currently designated to receive the content of the content provider.

3. The method of claim 1 , wherein the content of the content provider comprises one or more secondary media items to be presented to the plurality of users in association with a primary media item on a user interface of a content sharing platform, the secondary media items comprising advertisements related to the primary media item.

4. 3. The method of claim 2, wherein the indication of the estimated number of user actions related to the content of the content provider comprises an estimated increase in the number of user actions related to the content of the content provider in response to the addition of the user from the corresponding audience to the plurality of users currently designated to receive the content of the content provider.

5. providing the user interface for display to the content provider with an option to view new audiences to be added to the plurality of users currently designated to receive the content of the content provider, further comprising: collecting data identifying a number of presentations of the content of the content provider to a first group of users and a number of user actions related to the content of the content provider by the first group of users; identifying a subgroup of users in the first group of users that belong to a second group of users who are not currently designated to receive the content from the content provider; determining content-relevance metrics for the identified subgroup of users, the content-relevance metrics for the identified subgroup of users comprising a number of presentations of the content of the content provider to the identified subgroup of users and a number of user actions related to the content of the content provider by the identified subgroup of users; estimating content-relevance indicators for the second group of users based on the content-relevance indicators for the identified subgroup of users, the estimated content-relevance indicators for the second group of users comprising an estimated number of presentations of the content of the content provider to the second group of users and an estimated number of user actions related to the content of the content provider by the second group of users; predicting an increase in the number of user actions related to the content of the content provider in response to the addition of the second group of users to the first group of users based on the estimated content-relevance metrics for the second group of users; and adding the second group of users to the new audience in response to the predicted increase satisfying a threshold condition, wherein the displayed information identifying the new audience comprises an identifier of the second group of users and the predicted increase in the number of user actions related to the content of the content provider.

6. 1. A computer-implemented method comprising: collecting content-related metrics for a first group of users, the content-related metrics for the first group of users comprising a number of presentations of content from a content provider to the first group of users and a number of user actions related to the content from the content provider by the first group of users; identifying a subgroup of users in the first group of users that belong to a second group of users who are not currently designated to receive the content from the content provider; determining content-relevance metrics for the identified subgroup of users based on the content-relevance metrics for the first group of users; estimating content-relevance metrics for the second group of users based on the content-relevance metrics of the identified subgroup of users; predicting an increase in the number of user actions related to the content of the content provider in response to adding the second group of users to the first group of users based on the estimated content-relevance metrics of the second group of users; and in response to the estimated increase satisfying a threshold condition, providing information identifying the second group of users and the estimated increase for presentation to the content provider.

7. 7. The method of claim 6, wherein the content of the content provider comprises one or more secondary media items to be presented to multiple users in association with a primary media item on a user interface of a content sharing platform, the secondary media items comprising advertisements related to the primary media item.

8. The step of estimating the content-relevance metrics for the second group of users further comprises: determining a number of presentations of the content of the content provider to the subgroup of users; determining a number of user actions by the subgroup of users regarding the content of the content provider; and determining a conversion rate for the second group of users based on the number of presentations of the content of the content provider to the subgroup of users and the number of user actions related to the content of the content provider by the subgroup of users.

9. The step of determining the conversion rate for the second group of users further comprises:

9. The method of claim 8, further comprising comparing a set of characteristics associated with the subgroup of users with a set of characteristics associated with the second group of users, wherein the conversion rate for the second group of users is determined based on a comparison of the set of characteristics associated with the subgroup of users with the set of characteristics associated with the second group of users.

10. The method of claim 6 , wherein the increase in the number of user actions related to the content of the content provider is predicted using an estimation function.

11. 11. The method of claim 10, wherein the estimation function comprises one or more equations using a plurality of parameters and a plurality of corresponding weights, the plurality of parameters comprising a number of users in the second group who are not currently designated to receive the content from the content provider, an estimated number of presentations of the content from the content provider to the second group of users, and an estimated number of user actions regarding the content from the content provider by the second group of users.

12. determining content-relevance metrics for the first group of users and content-relevance metrics for a third group of users, the first group of users and the third group of users having previously been designated to receive the content from the content provider; using the determined content-relevance metrics of the first group of users and the estimation function to obtain estimated content-relevance metrics of the third group of users; and modifying the weights of the estimation function based on a comparison of the estimated content-relevance indicators for the third group of users and the determined content-relevance indicators for the third group of users.

13. Memory and a processor, coupled to the memory, for performing operations, the operations comprising: providing a user interface with an option to view new audiences to be added to a plurality of users currently designated to receive content from said content provider for display to said content provider; receiving a user selection of said option; and causing the display of information identifying the new audience, wherein the information identifying the new audience comprises, for each new audience, an audience identifier, an indication of an estimated number of user actions related to the content of the content provider, and an option to request that the corresponding audience be added to the plurality of users currently designated to receive the content of the content provider.

14. The operation further comprises: receiving a selection of the option to request that the corresponding audience be added to the plurality of users currently designated to receive the content of the content provider; and adding users from the corresponding audience to the plurality of users currently designated to receive the content from the content provider.

15. 15. The system of claim 14, wherein the indication of the estimated number of user actions regarding the content of the content provider comprises an estimated increase in the number of user actions regarding the content of the content provider in response to the addition of the user from the corresponding audience to the plurality of users currently designated to receive the content of the content provider.

16. providing the user interface with an option to view new audiences to be added to the plurality of users currently designated to receive the content of the content provider for display to the content provider, further comprising: collecting data identifying a number of presentations of the content of the content provider to a first group of users and a number of user actions related to the content of the content provider by the first group of users; identifying a subgroup of users in the first group of users that belong to a second group of users who are not currently designated to receive the content from the content provider; determining content-relevance metrics for the identified subgroup of users, the content-relevance metrics for the identified subgroup of users comprising a number of presentations of the content of the content provider to the identified subgroup of users and a number of user actions related to the content of the content provider by the identified subgroup of users; estimating content-relevance indicators for the second group of users based on the content-relevance indicators for the identified subgroup of users, the estimated content-relevance indicators for the second group of users comprising an estimated number of presentations of the content of the content provider to the second group of users and an estimated number of user actions related to the content of the content provider by the second group of users; predicting an increase in a number of user actions related to the content of the content provider in response to adding the second group of users to the first group of users based on the estimated content-relevance indicators for the second group of users; and adding the second group of users to the new audience in response to the predicted increase satisfying a threshold condition, wherein the displayed information identifying the new audience comprises an identifier of the second group of users and the predicted increase in the number of user actions related to the content of the content provider.

17. 1. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform operations, the operations including: collecting content-related metrics for a first group of users, the content-related metrics for the first group of users comprising a number of presentations of content from a content provider to the first group of users and a number of user actions related to the content from the content provider by the first group of users; identifying a subgroup of users in the first group of users that belong to a second group of users who are not currently designated to receive the content from the content provider; determining content-relevance metrics for the identified subgroup of users based on the content-relevance metrics for the first group of users; estimating content-relevance metrics for the second group of users based on the content-relevance metrics of the identified subgroup of users; predicting an increase in a number of user actions related to the content of the content provider in response to adding the second group of users to the first group of users based on the estimated content-relevance metrics of the second group of users; and in response to the estimated increase satisfying a threshold condition, providing information identifying the second group of users and the estimated increase for presentation to the content provider.

18. estimating the content-relevance metrics for the second group of users further comprises: determining a number of presentations of the content of the content provider to the subgroup of users; determining a number of user actions by the subgroup of users regarding the content of the content provider; and determining a conversion rate for the second group of users based on the number of presentations of the content of the content provider to the subgroup of users and the number of user actions regarding the content of the content provider by the subgroup of users.

19. Determining the conversion rate for the second group of users further comprises:

20. The computer-readable storage medium of claim 18, further comprising comparing a set of characteristics associated with the subgroup of users to a set of characteristics associated with the second group of users, wherein the conversion rate for the second group of users is determined based on a comparison of the set of characteristics associated with the subgroup of users to the set of characteristics associated with the second group of users.

20. 20. The computer-readable storage medium of claim 17, wherein the increase in the number of user actions related to the content of the content provider is predicted using an estimation function.

21. 21. The computer-readable storage medium of claim 20, wherein the estimation function comprises one or more equations using a plurality of parameters and a plurality of corresponding weights, the plurality of parameters comprising a number of users in the second group who are not currently designated to receive the content from the content provider, an estimated number of presentations of the content from the content provider to the second group of users, and an estimated number of user actions regarding the content from the content provider by the second group of users.

22. The operation further comprises: determining content-related metrics for the first group of users and content-related metrics for a third group of users, wherein the first group of users and the third group of users were previously designated to receive the content from the content provider; using the determined content-relevance indicators of the first group of users and the estimation function to obtain estimated content-relevance indicators of the third group of users; and modifying the weights of the estimation function based on a comparison of the estimated content-relevance indicators for the third group of users and the determined content-relevance indicators for the third group of users.

Images