Digital video analysis

Abstract

The present disclosure relates to digital video analysis. In one aspect, a method includes receiving data indicative of one or more seed video groups each including one or more seed videos. Data indicative of one or more keywords is received. A set of candidate video groups each including one or more candidate videos is identified. For each candidate video group in the set of candidate video groups, a common interaction score and a topicality score are determined. A subset of candidate video groups is selected based on the common interaction score and the topicality score of each candidate video group. Data indicative of the subset of candidate video groups is provided for presentation.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Application No. 63 / 079,377, filed September 16, 2020. The disclosure of the foregoing application is hereby incorporated by reference in its entirety. BACKGROUND

[0003] This specification relates to a data processing system and analyzing a digital video group.

[0004] Some online video platforms enable users to create channels of multiple digital videos that are viewable by other users of the platform. Users can subscribe to a video channel in order to receive videos published by the channel. For example, a user’s feed can include videos recently published by channels to which the user is subscribed. Channels of online video platforms are analogous to shows or programs of traditional media, and individual video content of the channel is analogous to episodes of the show or program. SUMMARY

[0005] In general, one aspect of the subject matter described in this specification can be embodied in methods that include receiving data indicative of one or more seed video groups each including one or more seed videos; receiving data indicative of one or more keywords; identifying a set of candidate video groups each including one or more candidate videos; for each candidate video group in the set of candidate video groups: determining a co-interaction score representing a measure of frequency of interaction by users that interacted with the one or more seed videos of the one or more seed video groups with the one or more videos of the candidate video group; and determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group; selecting a subset of the candidate video groups based on the co-interaction score and the topicality score for each candidate video group; and providing data indicative of the subset of the candidate video groups for display. Other implementations of this aspect include corresponding apparatuses, systems, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

[0006] These and other implementations can each optionally include one or more of the following features. Some aspects include receiving, from a computing system of a digital component provider, data indicative of a user selection of a given candidate video group in the subset of candidate video groups; including the given candidate video group in a video group package; and distributing, to a client device, a digital component of the digital component provider for presentation with at least one of the one or more videos in the given candidate video group.

[0007] In some aspects, determining the co-interaction score for each candidate video group includes using collaborative filtering to determine the co-interaction score for each candidate video group. In some aspects, determining the co-interaction score for each candidate video group includes determining a number of user interactions with one or more videos of the candidate video group performed by users that also interacted with videos of one or more seed video groups.

[0008] In some aspects, the one or more seed video groups include one or more positive seed video groups and one or more negative seed video groups. Determining the co-interaction score for each candidate video group can include determining a first number of user interactions with one or more videos of the candidate video group performed by users that also interacted with videos of the one or more positive seed video groups; determining a second number of user interactions with one or more videos of the candidate video group performed by users that also interacted with videos of the one or more negative seed video groups; and determining the co-interaction score for the candidate video group based on the first number and the second number. Each seed video group and each candidate video group can include a respective video channel of a video sharing platform. Each user interaction can include a subscription to one of the video channels.

[0009] In some aspects, the co-interaction score for each candidate video group is based on a similarity between a pattern of user interactions with one or more seed videos of the one or more seed video groups and a pattern of user interactions with one or more videos of the candidate video group. Each pattern of user interactions can be based on at least one of: a display time of the videos, a frequency at which each video is displayed, or a subscription to the video group.

[0010] In some aspects, each candidate video includes one or more annotations selected based on at least one of: (i) content of the candidate video, (ii) a title of the candidate video, (iii) a description of the candidate video, or (iv) a user comment posted about the candidate video. Determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group includes determining a number of the one or more videos in the candidate video group that have an annotation that includes at least one of the one or more keywords. Determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group can include determining a ratio between (i) a number of the one or more videos in the candidate video group that have an annotation that includes at least one of the one or more keywords and (ii) a number of the one or more videos in the candidate video group that do not have an annotation that includes at least one of the one or more keywords. Some aspects include receiving data indicating a set of one or more negative keywords; for each candidate video group, determining an anti-topicality score representing a measure of topicality between the one or more negative keywords and the one or more videos of the candidate video group; and filtering, from the set of candidate video groups, each candidate video group that has an anti-topicality score that satisfies a threshold.

[0011] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages. The seed-based video analysis techniques described in this document can identify, from thousands or millions of different video groups (e.g., video channels), video groups that have particular characteristics that would otherwise be impossible or impractical to identify using allow lists or categories. For example, these techniques can identify video groups that express particular emotions, aesthetics, user tastes, and / or nuanced topics (e.g., recent or upcoming events).

[0012] These techniques provide a scalable solution for identifying, selecting, and packaging video groups that have particular characteristics (e.g., particular common interaction patterns, emotions, aesthetics, and / or topicality), without requiring users to browse many (e.g., thousands or millions) individual videos or video channels, which reduces the computational burden placed on servers that provide information about each video group, and reduces the network bandwidth consumed by transmitting such information to client computing systems. These computational and network bandwidth savings can be very substantial when aggregated across many users (e.g., thousands or millions).

[0013] Using seed video groups, for example, in conjunction with refinement techniques that allow users to select positive video groups and / or negative video groups and / or keywords, video groups with specific characteristics can be accurately selected. By accurately selecting video groups using these techniques, the system can conserve network resources and associated overhead by not transmitting videos and / or other content to client devices that will not be viewed by users. For example, by accurately packaging video groups that are targeted for a package, videos and / or content associated with video groups that are not targeted for the package can be prevented from being distributed to users interested in videos that are targeted for the package, and computing and network resources associated with transmission of these videos and / or content can be conserved for other tasks.

[0014] The various features and advantages of the above-described subject matter will be more apparent from the following detailed description taken in conjunction with the accompanying drawings. Additional features and advantages will be apparent from the description that follows, and from the novel features described herein, which are pointed out with particularity. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram of an environment in which a video platform provides access to videos.

[0016] Figure 2 is a flow diagram illustrating an example process for selecting a target video group and providing data about the target video group.

[0017] Figure 3 is an illustration of an example user interface that enables a user to identify seed videos and keywords.

[0018] Figure 4 is an illustration of an example user interface that enables a user to refine a set of video groups based on one or more candidate video groups.

[0019] Figure 5 is an illustration of an example user interface that displays information about a video group and enables a user to refine the video group based on keywords.

[0020] Figure 6 is an illustration of an example user interface that enables a user to refine a video group based on keywords.

[0021] Figure 7 is an illustration of an example user interface that enables a user to refine a video group based on keywords.

[0022] Figure 8 is a flow diagram illustrating an example process for refining a set of candidate video groups for selection by a user.

[0023] Figure 9 is a block diagram of an example computer system.

[0024] In the various drawings, like reference numbers and designations indicate like elements. DETAILED DESCRIPTION

[0025] In general, this document describes systems and techniques for identifying video groups using seed video groups and keywords. Each video group can include one or more digital videos. Seed keywords can be keywords that represent various concepts or attributes of a video group that in turn define a theme of a given video group. The system can identify candidate video groups based at least in part on a common interaction score and / or a topicality score, the common interaction score representing a measure of how frequently a user interacts directly with one or more seed videos of a seed video group, with one or more videos of a set of candidate video groups, or with one or more video groups of a set of candidate video groups, the topicality score representing a measure of topicality between one or more keywords and one or more videos of a candidate video group.

[0026] The system can provide a user interface that enables a user to specify seed video groups and keywords. The user interface can include user interface controls that enable a user to refine identified video groups, e.g., by selecting positive and / or negative video groups and / or selecting positive and / or negative keywords. Positive and negative keywords can be used to compute collective topicality scores and / or collective anti-topicality scores for video groups. Positive and negative seed video groups can be used to compute common interaction scores to determine video groups that are more likely to have common interactions with positive seeds and less likely to have common interactions with negative seeds. The system can use user selections to update common interaction scores and / or topicality scores for candidate video groups and use the updated scores to identify an updated set of candidate video groups. In this way, a user can iteratively refine a list using video groups and / or keywords that align with a target for a package of candidate video groups.

[0027] The system can then enable a content provider to provide content to videos of video groups selected to be included in a video group package. For example, a content provider (or other entity) can link a digital component with a video group package such that when a video is played at a user's client device, the digital component is provided for display, e.g., presentation, with the video of the video group.

[0028] Figure 1 is a block diagram of an environment 100 in which a video platform 130 provides access to videos. The example environment 100 includes a data communication network 105, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. The network 105 connects client devices 110, the video platform 130, and computing systems of content providers 160. The example environment 100 can include many different client devices 110 and content providers 160.

[0029] Client devices 110 are electronic devices that are capable of communicating over network 105. Example client devices 110 include personal computers, mobile communication devices (e.g., smart phones), and other devices that can send and receive data over network 105. Client devices can also include digital media devices such as streaming devices that plug into a television or other display to stream video to the television.

[0030] Client devices 110 can also include digital assistant devices that accept audio input through a microphone and output audio output through a speaker. When a digital assistant detects a "hotword" or "hotphrase" that activates the microphone to accept audio input, the digital assistant can be placed into a listening mode (e.g., ready to accept audio input). Digital assistant devices can also include a camera and / or a display to capture images and visually display information, e.g., video. Digital assistants can be implemented in different forms of hardware devices, including a wearable device (e.g., a watch or glasses), a smart phone, a speaker device, a tablet device, or another hardware device.

[0031] Client devices 110 typically include applications, such as web browsers and / or native applications, to facilitate sending and receiving data over network 105. Native applications are applications developed for a particular platform or a particular device (e.g., a mobile device with a particular operating system). Client devices 110 can include video application 112, which can be a native application for playing digital video or a web browser that plays digital video of web pages.

[0032] Content providers 160 can create and publish content for display at client devices 110. For example, content providers 160 can create content to display with video played at client devices 110. The content can include video content that is displayed before, during, or after another video is displayed at client devices 110. The content can include image, text, or video content that is displayed within an electronic resource that also includes video. For example, video application 112 can display a video channel or feed to a user. The video channel or feed can include multiple videos. In addition to displaying the videos, the web page or application that displays the video channel or feed can include additional content that is displayed on the screen with the videos, e.g., adjacent to the videos. Similarly, content can be displayed on the screen with individual videos.

[0033] The content can include digital components. As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component can be stored electronically in a physical memory device as a single file or a collection of files, and a digital component can take the form of a video file, an audio file, a multimedia file, an image file, or a text file, and include advertising information such that an advertisement is a type of digital component. For example, a digital component can be content intended to supplement a web page or other resource displayed by the application 112. More specifically, a digital component can include digital content that is related to the resource content (e.g., a digital component can be related to the same topic as the web page content or related to a related topic). Thus, the provision of a digital component can supplement and generally enhance the web page or application content.

[0034] The video platform 130 provides videos for display at the client devices 110. The video platform 130 includes a video distribution server 132, a video analytics server 134, a video packaging server 136, and a content distribution server 138. Each server can be implemented using one or more computer systems, such as the computer system 900 of FIG. 9. Although the content distribution server 138 is shown as part of the video platform 130, the content distribution server 138 can be part of a separate system and / or operated by a different party than the video platform 130. Figure 9

[0035] In some implementations, the video platform 130 is an online video sharing platform. For example, the videos can include videos created by users and uploaded to the video platform 130. Users of the video platform 130 can create video channels each including one or more videos. A video channel is a type of video group. A video group can include a single video or a collection of videos. Other users of the video platform 130 can subscribe to a video channel in order to receive videos published by the channel. In some implementations, no subscription can be required to watch videos of a channel, but a subscription can be used to aggregate videos for a particular user in the user's feed. In other examples, a subscription can be required for some video channels. The videos are stored in a video data store 142, such as one or more hard drives, flash memory, etc.

[0036] ​The video distribution server 132 can provide videos for display at the client device 110. For example, the video distribution server 132 can receive a request for a video from the video application 112 and provide the requested video in response to the request. In another example, a user can navigate to their subscriptions and the video distribution server 132 can provide videos (or video control user interface elements that enable the user to start a video) for display in the subscriptions user interface. In yet another example, a user can navigate to a video channel and the video distribution server 132 can provide videos of the video channel (or video control user interface elements for the videos) for display in the video channel user interface. If the user interacts with a video control user interface element, e.g., by selecting a play control, the video distribution server 132 can stream the video to the video application 112 for display at the client device 110.

[0037] The video packaging server 136 enables content providers 160, users of the video platform 130, or users of the content distribution server 138 (e.g., administrators) to create packages of video groups. The content providers 160 can then link content (e.g., digital components) to the video group packages so that the content is displayed with the videos of the video groups of the video group packages. For example, a content provider 160 can create a video group package and link one or more digital components to the video group package. In another example, a user of the video platform 130 or the content distribution server 138 can create a video group package and make the video group package available to content providers, e.g., in exchange for a fee.

[0038] The video packaging server 136 can interact with the video analytics server 134 to identify and suggest candidate video groups for inclusion in a video group package based on input from users (e.g., from content providers 160, users of the video platform 130, or the content distribution server 138). The input can include data that identifies seed video groups and / or seed keywords (also referred to as keywords for brevity).

[0039] A seed video group can include one or more videos of the video platform 130. For example, a seed video group can include individual videos, a video channel, or other groups of multiple videos. The seed video groups enable users to define or signal the type of video groups that the user wants to have a similar user interaction pattern with the video groups in the video group package. The video packaging server 136 can enable users to select positive seed video groups and negative seed video groups, as described below.

[0040] Keywords can be used to define the theme of a video bundle. For example, a user can enter or select keywords based on the theme of a video bundle. As described in more detail below, the video platform 130 can provide a user interface that suggests channels and keywords that can be used to refine a video bundle. The video bundling server 136 can enable a user to specify positive keywords (e.g., theme keywords) and negative keywords (e.g., anti-theme keywords), as described below.

[0041] The video analysis server 134 can identify candidate video groups based on the seed video and / or the keywords. In some implementations, the video analysis server 134 can determine one or more scores for each video group in a set of video groups based on the seed video group and / or the keywords, and select candidate video groups based on the scores.

[0042] For example, the video analysis server 134 can determine, for each video group, a co- interaction score based on the seed video of the seed video group and the videos of the candidate video group. The co-interaction score can represent how likely a user that interacted with the seed video of the seed video group will have similar interactions with the candidate videos of the candidate video group. As described in more detail below, the co-interaction score can be determined using a collaborative filtering technique. For example, the co-interaction score can represent a measure of how frequently a user that interacted with the seed video of the seed video group interacts with the videos of the video group.

[0043] If a negative seed video group is used, the co-interaction score of a candidate video group can reflect or measure the likelihood that the candidate video group will have co-interactions with the positive seed video group and the candidate video group will not have co-interactions with the negative video seed group. That is, the co-interaction score can be used to determine video groups that are more likely to have co-interactions with the positive video watch group and less likely to have co-interactions with the negative seed video group.

[0044] The video analysis server 134 can also determine, for each video group, a topicality score that represents a measure of topicality between the selected keywords and the videos of the video group. The topicality score can be based on the annotations assigned to the videos of the video group and the specified keywords. For example, the topicality score can be based on, e.g., equal to or proportional to, a number of videos in the video group that are assigned annotations that match at least one of the specified keywords. In another example, the topicality score can be based on, e.g., proportional to, a ratio between a number of videos in the video group that are assigned annotations that match at least one of the specified keywords and a number of videos in the video group that do not have annotations that include at least one of the specified keywords. In some implementations, the video analysis server 134 can also determine, for each video group, an anti-topicality score in a similar manner but using negative keywords instead of positive keywords.

[0045] The video analysis server 134 can analyze the videos to assign annotations to the videos. For example, the video analysis server 134 can assign a keyword as an annotation to each video based on the content of the video, the title of the video, the description of the video, user comments posted on the video platform 130 about the video, the title of co-watched videos (e.g., videos watched by the same user as the video), the description of co-watched videos, search queries that resulted in display of the video, and / or other appropriate content related to the video.

[0046] In a particular example, the video analysis server 134 can analyze the images of a video using computer vision and / or machine learning techniques to determine what the video is about (e.g., the subject matter of the video) and / or to identify entities (e.g., people, objects, etc.) in the images. The video analysis server 134 can assign the entities as annotations for the video. In another example, the video analysis server 134 can compare the identified entities to knowledge graph entities of a knowledge graph and assign any matching knowledge graph entities as annotations for the video. A knowledge graph is an entity graph in which each entity is represented by a node and edges between entities indicate that the entities are related.

[0047] The video analysis server 134 can similarly identify entities in the audio of a video and associated text (e.g., the title, description, etc. discussed above), compare those entities to knowledge graph entities of a knowledge graph, and assign any matching knowledge graph entities as annotations for the video. The video analysis server 134 can store the annotations for each video in the video information data store 144, e.g., one or more hard drives, flash memory, etc.

[0048] The video packaging server 136 can select multiple groups of videos as candidate groups of videos for video group packages and provide data for the groups of videos to the computing system. A user can then select from the candidate groups of videos and / or refine the candidate groups of videos based on additional seed groups of videos and / or keywords until the candidate groups of videos are satisfactory, as described in more detail below. The video packaging server 136 can store data specifying video group packages, e.g., including data indicating each group of videos in a video group package, in the video information data store 144.

[0049] The video packaging server 136 can enable users to assign digital components to a video package. For example, a user can select digital components for display with the videos of a video group in a video package and assign them to the video package. In a particular example, the content provider 160 can create a video package for a particular context in which the content provider 160 wants to display digital components. The content provider 160 can then assign digital components to the created video package. The video packaging server 136 can store data specifying the digital components assigned to the video package in the video information data store 144. The digital components or data indicating that the digital components are to be downloaded by the client device 110 can be stored in the content data store, e.g., one or more hard drives, flash memory, etc.

[0050] When a video is provided for display at a client device 110, the content distribution server 138 can select content to display with the video. The selection can be based in part on the digital components assigned to the video group. For example, if the video distribution server 132 is updating the user interface of the video application 112 on the client device 112 to display a given video group, e.g., a given video channel, the content distribution server 138 can select the digital components assigned to the given video group for display by the video application 112, e.g., within the user interface with the videos of the given video group. In some implementations, additional criteria, such as selection parameters (e.g., bids), can be used to select the digital components.

[0051] Figure 2 is a flowchart illustrating an example process 200 for selecting a target video group and providing data about the target video group. The operations of process 200 can be implemented, for example, by the video platform 130. The operations of process 200 can also be implemented as instructions stored on one or more computer-readable media, which can be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of process 200.

[0052] Data indicating one or more seed video groups is received (202). The video platform can provide a user interface that enables a user to input a seed video group or select a seed video group from a set of video groups. For example, if the video platform is an online video platform, the user interface can enable a user to input an address, e.g., a uniform resource locator (URL), for a seed video group. In another example, the user interface can enable a user to search for video groups and select a video group from the search results as a seed video group. Each seed video group can include one or more videos. For example, a seed video group can be an individual video or a video channel that includes multiple videos.

[0053] Data indicative of one or more keywords is received (204). The video platform can provide a user interface that enables a user to input a keyword or select a keyword from a set of keywords. For example, the video platform 130 can enable a user to search for keywords, e.g., based on a topic, and the user can select a keyword from the search results. The keyword can be a knowledge graph entity of the knowledge graph.

[0054] A set of candidate video groups is identified (206). The candidate video groups can be video groups that are eligible for selection for inclusion in a video group package. Each candidate video group can include one or more videos. For example, a candidate video group can be an individual video or a video channel that includes multiple videos. The candidate video groups can be the same as or different from the seed video group. For example, both types of video groups can be selected from the same set of video groups, e.g., video groups of an online video sharing platform.

[0055] A co-interaction score is determined for each candidate video group (208). The co- interaction score can be determined using a collaborative filtering technique. For example, the co- interaction score for a candidate video group can represent a measure of how frequently users that interact with the seed videos of the seed video group also interact with the videos of the candidate video group. The co-interaction score can be based on similar interactions between users with respect to both types of video groups.

[0056] The interactions used to determine the co-interaction score can include one or more types of user interactions. In some implementations, when the video groups are video channels, the user interactions can be subscriptions. For example, the co-interaction score for a candidate video channel represents a measure of how frequently users that subscribe to the seed video channel also subscribe to the candidate video channel. The co-interaction score can be based on the number of users that subscribe to the seed video channel that also subscribe to the candidate video channel. In another example, the co-interaction score can be based on a ratio between (i) the number of users that subscribe to both the candidate video channel and at least one seed video channel and (ii) the total number of users in the population of users that subscribe to at least one seed video channel or the candidate video channel. In these examples, a candidate video channel that has more users that also subscribe to one or more seed video channels can have a higher score than a candidate video channel that has fewer users that also subscribe to one or more seed video channels.

[0057] In another example, if the group of videos is individual videos, the user interaction can be a view of the video. For example, the common interaction score for a candidate video can be based on, for example, equal to or proportional to the number of users who both watched the candidate video and watched at least one seed video. In another example, the common interaction score for a candidate video can be based on, for example, equal to or proportional to the number of seed videos that have been viewed by users who also watched the candidate video. In yet another example, the common interaction score for a candidate video can be based on a ratio between (i) the number of users who watched both the candidate video and at least one seed video and (ii) the total number of users in the population of users who watched at least one seed video or the candidate video.

[0058] In some implementations, the common interaction score for a group of candidate videos is based on a similarity between a pattern of user interactions with videos of the group of candidate videos and a pattern of user interactions with videos of each group of seed videos. The patterns can be based on a subscription to the group of videos, a frequency with which a user watches videos in the group, and a duration of time a user watches videos in the group of videos.

[0059] To determine the common interaction score, the video platform can determine an aggregate pattern for each candidate video. The aggregate pattern for a group of candidate videos can be based on or include an average duration of time users view videos of the group of candidate videos, a frequency with which users who watch videos of the group of candidate videos return to watch videos of the group of candidate videos, and / or a percentage of users who subscribe to the group of candidate videos. The aggregate pattern for each group of seed videos can include the same or similar pattern information. The video platform can then compare the pattern of the group of candidate videos to the pattern of each group of seed videos, can determine a common interaction score based on a similarity between the pattern of the group of candidate videos and the pattern of each group of seed videos.

[0060] In some implementations, the common interaction score for a group of candidate videos can be based on, for example, a positive group of seed videos and a negative group of seed videos selected by a user. The positive group of seed videos is a group of videos that the selected group of candidate videos should be similar to. The negative group of seed videos is a group of videos that the selected group of candidate videos should not be similar to. That is, a user can select a positive group of seed videos that are similar to groups of videos that the user wishes to include in a package of groups of videos. The user can similarly select a negative group of seed videos that are similar to groups of videos that the user does not wish to include in the package of groups of videos.

[0061] In this example, the common interaction score for a candidate video group can have a positive correlation with the similarity of the user interactions and the positive seed video group (e.g., based on subscriptions, viewing durations, frequency and / or patterns of users viewing the videos, as described above). Similarly, the common interaction score for a candidate video group can have a negative correlation with the similarity of the user interactions and the positive seed video group (e.g., based on subscriptions, viewing durations, frequency and / or patterns of users viewing the videos, as described above).

[0062] A topicality score is determined for each candidate video group (210). The topicality score for a candidate video group can represent a measure of topicality between one or more keywords and the videos of the candidate video group. As described above, each video can include annotations based on the video content, text associated with the video, etc. The topicality score for a candidate video can be based on, for example, being equal to or proportional to the number of videos in the video group that are assigned annotations matching at least one of the keywords. In another example, the topicality score can be based on, for example, being proportional to a ratio between the number of videos in the video group that are assigned annotations matching at least one of the keywords and the number of videos in the video group that do not have annotations including at least one of the keywords. In another example, the topicality score for a candidate video can be based on, for example, being equal to or proportional to the number of annotations of videos in the candidate video group that match at least one of the keywords.

[0063] In some implementations, an anti-topicality score is also determined for each candidate video group. For example, the keywords can include positive keywords (e.g., topic keywords) and negative keywords (e.g., anti-topic keywords). A user can select positive keywords that reflect topics (or entities) of video groups that the user wishes to include in the video group package. Similarly, the user can select negative keywords that reflect topics (or entities) of video groups that the user does not wish to include in the video group package.

[0064] The video platform can use the positive keywords to compute the topicality score and the negative keywords to compute the anti-topicality score. For example, the topicality score can be based on the number of videos in the video group that are assigned annotations matching at least one of the positive keywords. The anti-topicality score can be based on the number of videos in the video group that are assigned annotations matching at least one of the negative keywords. Other ways of computing the topicality score described above can also be used, using the positive keywords for the topicality score and the negative keywords for the anti-topicality score.

[0065] A subset of the candidate video groups is selected (212). The subset of candidate video groups can be selected based on the co-interaction score for each candidate video group and the topicality score for each candidate video group. For example, the video platform can compute a total score for each candidate video group based on the co-interaction score and the topicality score for the candidate video group. In this example, the video platform can normalize the two scores to a particular score range and add the two scores together. The video platform can then select a specified amount of candidate video groups with the highest total scores to include in the subset.

[0066] In some implementations, the video platform can select candidate video groups to include in the subset based on the co-interaction score, the topicality score, and the anti-topicality score for each candidate video group. In this example, the video platform can filter each candidate video group from the candidate video groups that has an anti-topicality score that meets, e.g., that meets or exceeds, a threshold. In another example, the video platform can determine a total score for a candidate video based on the three scores, e.g., by normalizing the scores, adding the co-interaction score to the topicality score, and subtracting the anti-topicality score from the total.

[0067] Data indicating the subset of candidate video groups is provided for display (214). For example, the video platform can update the user interface of the user’s computing system to display identifiers of the video groups in the subset and information about the video groups, e.g., the associated scores, the number of subscribers to the candidate video groups, etc.

[0068] Figures 3 to 8 FIGURE 13 illustrates a user interface sequence that enables a user to create a video group package. The user interface enables the user to specify a seed video group and keywords to obtain an initial subset of candidate video groups selected based on the seed video group and the keywords. The user can then refine the candidate video groups by interacting with user interface controls displayed by the refinement user interface.

[0069] For example, the user can promote a candidate video group to a positive seed video group, demote a candidate video group to a negative seed video group, promote a tagged keyword to a positive keyword, and / or demote a tagged keyword to a negative keyword. The video platform 130 can update the scores for the video groups based on these refinements, select updated candidate video groups based on the refinements, and provide data for the updated candidate video groups for display to the user. The user can then make further refinements, e.g., in an iterative process, until the video group package is assembled with video groups that meet the user’s needs.

[0070] Figure 3is an illustration of an example user interface 300 that enables a user to identify a seed video and keywords. The user interface 300 includes a seed area 310 that includes a seed video group label 312 and a keyword label 314. When the seed video group label 312 is selected, the user can enter data identifying a seed video group into a seed input area 316. For example, the user can enter, e.g., type or copy and paste, a URL of the seed video group, a title of the seed video group, or other data identifying the seed video group. The user interface 300 can also allow the user to group the seed video group into a positive seed video group and a negative seed video group.

[0071] Similarly, when the keyword label 314 is selected, the user can enter keywords into the seed input area 316. In some implementations, the video platform 130 can update the user interface 300 to display suggested keywords based on the seed video group. For example, when the user selects the keyword label 314, the video platform can identify keywords based on the seed video group and populate the seed input area 316 with at least some of the identified keywords. The video platform 130 can identify the keywords based on annotations of the videos in the seed video group. For example, the video platform 130 can select a specified number of most frequently occurring annotations of the videos in the seed video group as keywords. The user can then select positive and / or negative keywords from the keyword suggestions.

[0072] Figure 4 is an illustration of an example user interface 400 that enables a user to refine a set of video groups based on one or more candidate video groups. For example, the video platform 130 can transition from the user interface 300 of Figure 3 to the user interface 400 (e.g., by updating the user interface 300) after the user selects a seed video group and keywords. Prior to the transition, the video platform 130 can, for example, use the process 200 of Figure 2 to identify a subset of candidate video groups to display in the user interface 400.

[0073] The user interface 400 includes a pack refinement user interface element 410 that enables the user to make a selection that refines the subset of candidate video groups that will be included in a video group pack. The pack refinement user interface element 410 includes a commonality label 420, a topicality label, and an anti-topicality label.

[0074] The illustration shows a common interaction label 420. This label displays information about some of the identified candidate video groups. Specifically, a first list 421 of video groups with the highest common interaction scores is shown, and a second list 428 of video groups with the lowest common interaction scores is shown. The candidate video groups shown in this label can include a subset of video groups identified based on seed video groups and keywords specified by the user.

[0075] For each video group in the list, the first list 421 includes a title 422 for the video group (e.g., based on the use of...). Figure 3 The system includes a shared interaction score 423 for the specified seed video group and a user interface control 424 that allows the user to refine the seed video group. Specifically, for each candidate video group, the user interface control 424 includes a first user interface control 425 that allows the user to add the candidate group as a positive seed video group to the set of seed video groups. For example, if the user interacts with the first user interface control 425 for candidate video group 129, such as by selecting the control 425, the video platform 130 can add candidate video group 129 as a positive seed video group. This allows the video platform 130 to use candidate video group 129 as a positive seed video group, for example, to identify candidate video groups with a similar shared interaction pattern to candidate video group 129.

[0076] For each candidate video group, user interface control 424 includes a second user interface control 426 that enables the user to add the candidate group as a negative seed video group to the set of seed video groups. For example, if the user interacts with the second user interface control 426 for candidate video group 129, such as by selecting the control 426, the video platform 130 can add candidate video group 129 as a negative seed video group. This allows the video platform 130 to use candidate video group 129 as a negative seed video group, for example, to identify candidate video groups that have a different common interaction pattern than candidate video group 129. The second list 428 includes similar information (e.g., common interaction score) and user interface controls that enable the user to perform similar actions for video groups with the lowest common interaction scores.

[0077] User interface 400 also includes a filtering control 427 that allows the user to filter video groups based on a common interaction score. For example, filtering control 427 allows the user to filter candidate video groups using a minimum common interaction score. Filtering control 427 can be used to filter candidate video groups from those identified based on a seed video group and keywords specified by the user. In another example, filtering control 427 can filter candidate video groups displayed in user interface 400 without filtering candidate video groups from the identified video groups, for example, solely for display purposes.

[0078] The user interface 400 also includes a refresh channel control 430 that can be used to update the set of candidate videos based on any user refinements, e.g., using the filter control 427 or the user interface control 424. If the user interacts with the refresh channel control 430, e.g., selects the control 430, the video platform can identify an updated set of candidate videos based on the updated set of seed video groups (e.g., the original set of seed video groups specified by the user and any seed video groups selected using the user interface control 424), the keywords (e.g., the original keywords specified by the user), any filter settings made using the filter control 427 (if it affects the selection of candidate video groups). For example, the video application 112 can provide this information (or just the updated information) to the video platform 130 in response to the user’s interaction with the refresh channel control 430. The video platform 130 can then provide data for the updated set of candidate videos to the video application 112 and update the user interface 400 to display the candidate videos with the highest and lowest common interaction scores in the updated set of candidate videos.

[0079] The user interface 400 also includes a save and review control 440 that enables the user to save the set of video packs that currently include the set of candidate videos. For example, if the user is finished refining, the user can interact with the save and review control 440 to save the set of video packs. This indicates the selection of the set of candidate videos as the target video groups for the set of video packs. The user can then assign a digital component to the set of video packs or make the set of video packs available to other users.

[0080] The user interface 400 can also allow the user to interact with each video group, e.g., select each video group, to view more information about the video group. For example, if the video application 112 or the video platform 130 receives data indicating that the user interacted with a video group, the video application 112 of the video platform 130 can update the user interface 400 to display a video group information element, e.g., as shown in Figure 5 .

[0081] Figure 5 is an illustration of an example user interface 500 that displays information about a video group and enables the user to refine the video group based on keywords. The user interface 500 includes a video group information element 510 that displays information about a selected video group (in this example, “Video Group 129”). This information can include a title of the video group 511, a number of subscribers to the video group 512, and video information elements 520, 530, and 540 for a portion of the videos in the video group.

[0082] Video information elements 520 include an image 525 of the video, which can be a screenshot from the video or another image representative of the video. Video information elements 520 also include a title 523 of the video, and a user interface control 521 that enables the user to add keywords for the video to a set of keywords used to identify a set of candidate video groups for the video group package.

[0083] As described above, the keywords displayed for the video user interface control 521 can be the annotated keywords assigned to the video. For example, the keywords can include knowledge graph entities identified in the video and / or in text associated with the video (e.g., text in a title, user comments, etc.).

[0084] The user interface control 521 can enable the user to add keywords as positive keywords or negative keywords to the set of keywords. For example, a first user interaction with the user interface control 521 for a keyword can specify the keyword to be added as a positive keyword, as shown by user interface control 521A. A second user interaction with the user interface control for the keyword can adjust the keyword from a positive keyword to a negative keyword, as shown by user interface control 521B. A third user interaction with the user control 521 that currently specifies a negative keyword can adjust the keyword from a negative keyword to an unselected keyword. Video information elements 530 and 540 can include similar information and user interface controls for their respective videos as video information elements 520.

[0085] After any refinements using the video group information elements 510, the user can return to the user interface 400. At the user interface 400, the user can use the refresh channel control 430 to update the candidate groups, which will update the set of candidate video groups based on the keyword selections made using the video group information elements 510. That is, the video application 112 can send to the video platform 130 the set of current seed video groups, the updated set of keywords, and any other settings (e.g., filter settings). The video platform 130 can then provide to the video application 112 data for the updated set of candidate videos, and update the user interface 400 to display the candidate videos with the highest and lowest common interaction scores in the updated set of candidate videos.

[0086] Figure 6 is an illustration of an example user interface 600 that enables a user to refine video groups based on keywords. The user interface 600 includes a package refinement user interface element 610 that enables the user to make selections that refine a subset of candidate video groups to be included in a video group package. The package refinement user interface element 610, which can be the same as or similar to the package refinement user interface element 410, includes a common interaction label, a topicality label 620, and an anti-topicality label.

[0087] In this illustration, a topicality tab 620 is shown. In this tab, information about some of the identified candidate video groups is shown. Specifically, a first list 621 of video groups with the highest topicality scores is shown, and a second list 628 of video groups with the lowest topicality scores is shown. The candidate video groups shown in this tab can include a portion of a subset of video groups identified based on the seed video group and keywords specified by the user, including any refinements to the seed video group and / or keywords using other user interfaces described in this document.

[0088] For each video group in the list, the first list 621 includes a title 622 for the video group, a topicality score 623 for the video group (e.g., computed based on the current set of keywords), and a user interface control 624 that enables the user to refine the set of keywords. In this example, for each candidate video group in the list, the user interface 600 can include a user interface control 624 for a specified number of keywords (e.g., three in this example, although other numbers are possible) selected from the annotations for the videos in the video group. For example, the video platform 130 can determine, for each annotation in the candidate video group, a total number of videos in the candidate video group that include the annotation. The video platform 130 can then update the user interface 600 to display a respective user interface control 624 for the keyword with the highest total number of videos.

[0089] The user interface controls 624 can be the same as or similar to the user interface controls 521 of Figure 5 For example, the user interface controls 624 can enable the user to add a keyword to the set of keywords as a positive keyword or a negative keyword. For example, a first user interaction with a user interface control 624 for a keyword can specify that the keyword is to be added as a positive keyword. A second user interaction with the user interface control 624 for the keyword can adjust the keyword from a positive keyword to a negative keyword. A third user interaction with the user interface control 624 that currently specifies a negative keyword can adjust the keyword from a negative keyword to an unselected keyword. The second list 626 includes similar information (e.g., topicality scores) and user interface controls that enable the user to perform similar actions for the video groups with the lowest topicality scores.

[0090] The user interface 600 also includes a filter control 627 that enables the user to filter the set of video groups based on the topic scores. For example, the filter control 627 enables the user to filter the set of candidate video groups using a minimum topic score. The filter control 627 can be used to filter the set of candidate video groups from the set of candidate video groups identified based on the set of seed video groups and keywords specified by the user. In another example, the filter control 627 can filter the set of candidate video groups displayed in the user interface 600 without filtering the set of candidate video groups from the set of identified video groups, e.g., for display purposes only.

[0091] The user interface 600 also includes a refresh channel control 630 that can be used to update the set of candidate video groups based on any user refinements, e.g., using the filter control 627 or the user interface control 624. If the user interacts with the refresh channel control 630, e.g., selects the refresh channel control 630, the video platform 130 can identify an updated set of candidate videos based on the current set of seed video groups (e.g., based on the originally specified seed videos and any refinements), the current set of keywords (e.g., the originally keywords and any refinements using the user interface 500 or the user interface 600), any filtering settings using the filter control 627 if it affects the selection of the set of candidate video groups. For example, the video application 112 can provide this information (or just the updated information) to the video platform 130 in response to the user’s interaction with the refresh channel control 630. The video platform 130 can then provide data for the updated set of candidate videos to the video application 112 and update the user interface 600 to display the candidate videos with the highest and lowest topic scores in the updated set of candidate videos.

[0092] The user interface 600 also includes a save and review control 640 that enables the user to save the set of video groups currently including the set of candidate video groups as a video pack. For example, if the user is finished refining, the user can interact with the save and review control 640 to save the video group pack. This indicates the selection of the set of candidate video groups as the target video groups for the video group pack. The user can then assign a digital component to the video group pack or make the video group pack available to other users.

[0093] Figure 7 is an illustration of an example user interface 700 that enables the user to refine the video groups based on keywords. This user interface 700 is similar to the user interface 600 of Figure 6 but displays information about anti-topicity instead of topicity. As described above, the topicity score for a set of candidate video groups can be based on positive keywords, and the anti-topicity score can be based on negative keywords.

[0094] The user interface 700 includes a package refinement user interface element 710 that enables the user to make selections that refine the candidate video group subsets that will be included in the video group package. Package refinement user interface elements 710 that are the same as or similar to the package refinement user interface element 710 include the commonality tag, the topicality tag, and the anti-topicality tag 720.

[0095] In this illustration, the anti-topicality tag 720 is shown. In this tag, information about some of the identified candidate video groups is shown. Specifically, a first list 721 of video groups with the highest anti-topicality scores is shown, and a second list 728 of video groups with the lowest anti-topicality scores is shown. The candidate video groups shown in this tag can include a portion of a subset of the video groups identified based on the seed video group and keywords specified by the user, including any refinements to the seed video group and / or keywords using other user interfaces described in this document.

[0096] For each video group in the list, the first list 721 includes a title 722 for the video group (e.g., computed based on the current set of keywords), an anti-topicality score 723 for the video group, and a user interface control 724 that enables the user to refine the set of keywords. In this example, for each candidate video group in the list, the user interface 700 can include a user interface control 724 for a specified number of keywords (e.g., three in this example, although other numbers are possible) selected from the annotations for the videos in the video group. For example, the video platform 130 can determine, for each annotation in the candidate video group, a total number of videos in the candidate video group that include the annotation. The video platform 130 can then update the user interface 700 to display the respective user interface control 724 for the keyword with the highest total number of videos.

[0097] The user interface control 724 can be the same as or similar to the user interface control 624 of Figure 6 , e.g., the user interface control 724 can enable the user to add a keyword to the set of keywords as a positive keyword or a negative keyword. For example, a first user interaction with the user interface control 724 for a keyword can specify that the keyword is to be added as a positive keyword, as shown by user interface control 724A. A second user interaction with the user interface control 724 for the keyword can adjust the keyword from a positive keyword to a negative keyword, as shown by user interface control 724B. A third user interaction with the user control 724 that currently specifies a negative keyword can adjust the keyword from a negative keyword to an unselected keyword. The second list 726 includes similar information (e.g., anti-topicality scores) and user interface controls that enable the user to perform similar actions for the video groups with the lowest anti-topicality scores.

[0098] The user interface 700 also includes a filter control 727 that enables the user to filter the set of candidate video groups based on the anti-topicity scores. For example, the filter control 727 enables the user to filter the set of candidate video groups using a maximum anti-topicity score. The filter control 727 can be used to filter the set of candidate video groups from the set of candidate video groups identified based on the seed video groups and keywords specified by the user. In another example, the filter control 727 can filter the set of candidate video groups displayed in the user interface 700 without filtering the set of candidate video groups from the identified set of video groups, e.g., for display purposes only.

[0099] The user interface 700 also includes a refresh channel control 730 that can be used to update the set of candidate video groups based on any user refinements, e.g., using the filter control 727 or the user interface controls 724. If the user interacts with the refresh channel control 730, e.g., selects the refresh channel control 730, the video platform 130 can identify an updated set of candidate videos based on the current set of seed video groups (e.g., based on the originally specified seed videos and any refinements), the current set of keywords (e.g., the originally keywords and any refinements using the user interfaces 500, 600, and / or 700), any filtering settings using the filter control 727 if it affects the selection of the set of candidate video groups. For example, the video application 112 can provide this information (or just the updated information) to the video platform 130 in response to the user’s interaction with the refresh channel control 730. The video platform 130 can then provide data for the updated set of candidate videos to the video application 112 and update the user interface 700 to display the candidate videos with the highest and lowest topicity scores in the updated set of candidate videos.

[0100] The user interface 700 also includes a save and review control 740 that enables the user to save the set of video groups currently including the set of candidate video groups as a video pack. For example, if the user has completed refinements, the user can interact with the save and review control 740 to save the video group pack. This indicates the selection of the set of candidate video groups as the target video groups for the video group pack. The user can then assign a digital component to the video group pack or make the video group pack available to other users.

[0101] Figure 8 is a flowchart illustrating an example process 800 for refining a set of candidate video groups for selection by a user. The operations of process 800 can be implemented, for example, by the video platform 130. The operations of process 800 can also be implemented as instructions stored on one or more computer-readable media, which can be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of process 800.

[0102] A user interface enabling a user to select seed video groups and keywords is provided (802). For example, the video platform 130 can generate or update a user interface of the video application 112 of the client device 110 to display the user interface. The user interface can be the user interface 300 of FIG. 6. The user can interact with the user interface to specify one or more seed video groups and one or more keywords (e.g., define a topicality) that will be used to select a set of initial candidate video groups to include in the video group package. After the seed video groups and keywords are specified, the video application 112 can send data to the video platform 130 indicating the seed video groups and keywords. Figure 3

[0103] The user interface is updated to display the set of candidate videos and user interface controls (804). The video platform 130 can select the set of candidate videos based on the seed video groups and keywords. For example, the video platform 130 can use the process 200 of FIG. 7 to select the candidate video groups. Figure 2

[0104] The video platform can then update the user interface to display at least a subset of the candidate videos and user interface controls. For example, the video platform 130 can provide data to the video application 112 indicating the set of candidate videos and information about the candidate video groups (e.g., topicality scores, anti-topicality scores, co-interaction scores, subscription counts, etc.). Depending on the user interface currently displayed at the video application 112, the video application 112 can display a particular subset of candidate videos. For example, if the co-interaction tab is being displayed, the user interface can display a subset of candidate videos based on the co-interaction scores, as shown in FIG. 8. Figure 4

[0105] The user interface controls enable the user to refine the set of candidate videos, e.g., based on the seed video groups and / or keywords. For example, as shown in FIG. 8, the user interface controls 424 enabling the user to add positive and / or negative seed video groups can be displayed. In another example, as shown in FIG. 9, the user interface controls 624 enabling the user to add positive and / or negative keywords can be displayed. The user can navigate between the tabs in the user interface of FIGS. 8 and 9 to add and / or remove seed video groups (e.g., positive and / or negative seed video groups) and / or keywords (e.g., positive and / or negative keywords). Figure 4 Figure 6 Figures 3 to 8

[0106] ​​​​​​Data indicating user interactions with the given user interface controls is received (806). For example, the video application 112 can detect user interactions with the user interface controls and provide data indicating the interactions to the video platform 130. The data can indicate the seed video groups or keywords corresponding to the user interface controls, and whether the seed video groups or keywords were added or removed. If added, the data can include data indicating that the seed video groups or keywords were added as positive or negative seed video groups or keywords. In some implementations, the video application 112 can provide the data for each user interface control that was interacted with in response to a user request, e.g., in response to a user’s interaction with a refresh channel control.

[0107] The user interface is updated to display the updated set of candidate videos (808). The video platform 130 can select the updated set of candidate videos based on the current set of seed video groups and the current set of keywords. The current set of seed video groups and the current set of keywords can be based on the initial seed video groups and keywords specified in operation 802 and any refinements (e.g., additions or removals) made in operation 806. The video platform 130 can use the process 200 to select the updated set of candidate video groups based on the current set of seed video groups and the keywords. Figure 2 The process 200 selects an updated set of candidate video groups based on a current set of seed video groups and keywords.

[0108] The video platform 130 can then update the user interface at the video application 112 to display at least a subset of the updated set of candidate video groups. For example, the video platform 130 can provide data indicating the updated set of candidate videos and information about the candidate video groups (e.g., topicality scores, anti-topicality scores, co-interaction scores, subscription counts, etc.) to the video application 112. Depending on the user interface currently displayed at the video application 112, the video application 112 can display a particular subset of candidate videos. For example, if a topicality tab is being displayed, the user interface can display a subset of candidate videos based on the topicality scores, as shown in FIG. 7B. Figure 6

[0109] Data indicating a selection of one or more candidate video groups as target video groups is received (810). The user can perform one or more iterations of refinement using operations 806 and 808. When the set of candidate video groups is acceptable to the user, the user can select a candidate video group for the video group package, e.g., by interacting with a save and check control. The user can then check the target video group and its associated information. The user can also assign content (e.g., digital components) to the target video group and / or make the video group package available to other users.

[0110] Figure 9 ​is a block diagram of an example computer system 900 that can be used to perform the operations described above. The system 900 includes a processor 910, a memory 920, a storage device 930, and an input / output device 940. Each of the components 910, 920, 930, and 940, for example, can be interconnected, for example, using a system bus 950. The processor 910 is capable of processing instructions for execution within the system 900. In some implementations, the processor 910 is a single-threaded processor. In another implementation, the processor 910 is a multi-threaded processor. The processor 910 is capable of processing instructions stored in the memory 920 or on the storage device 930.

[0111] The memory 920 stores information within the system 900. In one implementation, the memory 920 is a computer-readable medium. In some implementations, the memory 920 is a volatile memory unit. In another implementation, the memory 920 is a non-volatile memory unit.

[0112] The storage device 930 is capable of providing mass storage for the system 900. In some implementations, the storage device 930 is a computer-readable medium. In various different implementations, the storage device 930 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network (e.g., a cloud storage device) by multiple computing devices, or some other large capacity storage device.

[0113] The input / output device 940 provides input / output operations for the system 900. In some implementations, the input / output device 940 can include one or more network interface devices, for example, an Ethernet card, a serial communication device, e.g., an RS-232 port, and / or a wireless interface device, e.g., an 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to peripheral devices 960, for example, a keyboard, a printer and a display device. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0114] Although an example processing system has been described in Figure 9 Embodiments of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of the same.

[0115] Embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter disclosed in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0116] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored or received in one or more computer-readable storage devices or received from other sources.

[0117] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones of the same. The apparatus can include special purpose logic, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model architectures, such as web services, distributed computing, and grid computing architectures.

[0118] A computer program, which can also be referred to or be included in a computer program product, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0119] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) and / or by programmable processors

[0120] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both, the elements being located within a single device. A computer can generally also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0121] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0122] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0123] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0124] While this specification contains many specifics, these should not be construed as limiting the scope of any inventions and / or patents-issuable claims, but as describing features that can be part of particular embodiments. Some features that are described in the context of separate embodiments can also be implemented in combination, and vice versa. Conversely, various features that are described in the context of a single embodiment can also be implemented separately or in any appropriate sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0125] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems are typically integrated in a single software product or packaged into multiple software products.

[0126] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the process depicted in the accompanying figures does not necessarily require the particular order or sequential order illustrated, or that all illustrated operations be performed, to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method comprising: receiving data indicative of one or more seed video groups each comprising one or more seed videos; receiving data indicative of one or more keywords; identifying a set of candidate video groups each comprising one or more candidate videos; for each candidate video group in the set of candidate video groups: determining a common interaction score representing a measure of a frequency with which users that interact with the one or more seed videos in the one or more seed video groups interact with the one or more videos in the candidate video group; and determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group, wherein the topicality score is based on a match between one or more keywords designated as positive keywords and annotations of videos in the candidate video group; selecting a subset of the candidate video groups based on the common interaction score and the topicality score for each candidate video group; providing data indicative of the subset of the candidate video groups for display.

2. The computer-implemented method of claim 1, further comprising: receiving, from a computing system of a digital component provider, data indicative of a user selection of a given candidate video group in the subset of the candidate video groups; including the given candidate video group in a video group package; and distributing, to a client device, a digital component of the digital component provider for display with at least one of the one or more videos in the given candidate video group. Determining the common interaction score for each candidate video group comprises determining the common interaction score for each candidate video group using collaborative filtering. Determining the common interaction score for each candidate video group comprises determining a number of user interactions with the one or more videos in the candidate video group performed by users that also interact with videos in the one or more seed video groups.

3. The computer-implemented method of claim 1, wherein, 5. The computer-implemented method of claim 1, wherein:

4. The computer-implemented method of claim 1, wherein, the one or more seed video groups comprise one or more positive seed video groups and one or more negative seed video groups; Determining a common interaction score for each candidate video group comprises: determining a first number of user interactions with the one or more videos in the candidate video group performed by users that also interact with videos in the one or more positive seed video groups; determining a second number of user interactions with the one or more videos in the candidate video group performed by users that also interact with videos in the one or more negative seed video groups; and determining the common interaction score for the candidate video group based on the first number and the second number.

6. The computer-implemented method of claim 4, wherein: each seed video group and each candidate video group comprises a respective video channel of a video sharing platform; and each user interaction comprises a subscription to one of the video channels. ​ ​ 7. The computer-implemented method of claim 1, wherein, The common interaction score for each candidate video group is based on a similarity between a pattern of user interactions with the one or more seed videos in the one or more seed video groups and a pattern of user interactions with the one or more videos in the candidate video group.

8. The computer-implemented method of claim 7, wherein, Each pattern of user interactions is based on at least one of: a display time of a video, a frequency of each video being displayed, or a subscription to a video group.

9. The computer-implemented method of claim 1, wherein: each candidate video includes one or more tags selected based on at least one of: (i) content of the candidate video, (ii) a title of the candidate video, (iii) a description of the candidate video, or (iv) a user comment posted about the candidate video; and determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group includes determining a number of the one or more videos in the candidate video group that have a tag including at least one of the one or more keywords.

10. The computer-implemented method of claim 9, wherein, determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group includes determining a ratio between (i) a number of the one or more videos in the candidate video group that have a tag including at least one of the one or more keywords and (ii) a number of the one or more videos in the candidate video group that do not have a tag including at least one of the one or more keywords.

11. The computer-implemented method of claim 9, further comprising: receiving data indicating a set of one or more negative keywords; for each candidate video group, determining an anti-topicality score representing a measure of topicality between the one or more negative keywords and the one or more videos of the candidate video group; and filtering, from the set of candidate video groups, each candidate video group having an anti-topicality score that satisfies a threshold.

12. A system comprising: one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving data indicating one or more seed video groups each including one or more seed videos; receiving data indicating one or more keywords; identifying a set of candidate video groups each including one or more candidate videos; for each candidate video group in the set of candidate video groups: determining a common interaction score representing a measure of a frequency with which users that interact with the one or more seed videos in the one or more seed video groups interact with the one or more videos in the candidate video group; and determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group, wherein the topicality score is based on a match between one or more keywords designated as positive keywords and annotations of videos in the candidate video group; selecting a subset of the candidate video groups based on the common interaction score and the topicality score of each candidate video group; and providing data indicative of the subset of the candidate video groups for display. The operations include:

13. The system of claim 12, wherein, receiving, from a computing system of a digital component provider, data indicative of a user selection of a given candidate video group in the subset of the candidate video groups; including the given candidate video group in a video group package; and distributing, to a client device, a digital component of the digital component provider for display with at least one of the one or more videos in the given candidate video group. Determining the common interaction score for each candidate video group includes determining the common interaction score for each candidate video group using collaborative filtering.

14. The system of claim 12, wherein, Determining the common interaction score for each candidate video group includes determining a number of user interactions with the one or more videos in the candidate video group performed by users that also interacted with videos in the one or more seed video groups.

15. The system of claim 12, wherein, 16. The system of claim 12, wherein: the one or more seed video groups include one or more positive seed video groups and one or more negative seed video groups; Determining a common interaction score for each candidate video group includes: determining a first number of user interactions with the one or more videos in the candidate video group performed by users that also interacted with videos in the one or more positive seed video groups; determining a second number of user interactions with the one or more videos in the candidate video group performed by users that also interacted with videos in the one or more negative seed video groups; and determining the common interaction score for the candidate video group based on the first number and the second number.

17. The system of claim 16, wherein: each seed video group and each candidate video group includes a respective video channel of a video sharing platform; and each user interaction includes a subscription to one of the video channels. The common interaction score for each candidate video group is based on a similarity between a pattern of user interactions with the one or more seed videos in the one or more seed video groups and a pattern of user interactions with the one or more videos in the candidate video group.

18. The system of claim 12, wherein, Each pattern of user interactions is based on at least one of: a display time of videos, a frequency of each video being displayed, or a subscription to a video group.

19. The system of claim 18, wherein, 20. A computer-readable medium carrying instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving data indicative of one or more seed video groups each including one or more seed videos; receiving data indicative of one or more keywords; determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group, wherein the topicality score is based on a match between one or more keywords designated as positive keywords and annotations of videos in the candidate video group; selecting a subset of the candidate video groups based on the common interaction score and the topicality score of each candidate video group; and providing data indicative of the subset of the candidate video groups for display. identifying a set of candidate video groups each comprising one or more candidate videos; for each candidate video group in the set of candidate video groups: determining a common interaction score representing a measure of how frequently users that interact with the one or more seed videos in the one or more seed video groups interact with the one or more videos in the candidate video group; and determining a topicality score representing a measure of topicality between the one or more keywords and the one or more videos in the candidate video group, wherein the topicality score is based on a match between one or more keywords designated as positive keywords and annotations of videos in the candidate video group; selecting a subset of the candidate video groups based on the common interaction score and the topicality score for each candidate video group; and providing data indicative of the subset of the candidate video groups for display.

Images