Video Clustering and Analysis

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for improving content analysis. The method includes obtaining videos uploaded by a video publisher. A concept index is generated for each given video. The concept index is generated based on at least (i) concepts conveyed by one or more objects depicted in the video and (ii) a level of salience of the concepts in the given video. Multiple video groups are created based on the concept indexes of the videos. Each given video group includes two or more different videos, each having a specified level of similarity to other videos in the given video group. Insights are generated for the multiple video groups based on data obtained via a feedback loop. Distribution of at least one video is modified based on the insights for the given video group including the at least one video.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Israel Application No. 287859, filed November 5, 2021, which is incorporated herein by reference in its entirety. [Background technology]

[0002] This specification relates to data processing and analysis of videos. The amount of online videos has grown over the years, and anyone with a computer can now upload video content. Identifying groups of similar videos for analysis is difficult due to the different ways in which videos can be used to convey similar concepts. Summary of the Invention [Means for solving the problem]

[0003] In general, one inventive aspect of the subject matter described in this specification may be embodied in a method including the actions of: obtaining, by a service device, videos uploaded by a video publisher; generating, by the service device, a concept index for each given video, where the concept index is generated based on at least (i) concepts conveyed by one or more objects shown in the video and (ii) a level of salience of the concept in the given video; creating, by the service device, a plurality of video groups based on the concept indexes of the videos, where each given video group of the plurality of video groups is created to include two or more different videos, each having a specified level of similarity to other videos in the given video group; generating, by the service device, insights about the plurality of video groups based on data obtained through the feedback loop; and altering, by the service device, a manner in which at least one video is distributed over a network based on the insights about the given video group including at least one video.

[0004] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs encoded on computer storage devices and configured to perform the actions of the methods.

[0005] These and other embodiments may each optionally include one or more of the following features: Generating a concept index for each given video may include obtaining, for each given one of the videos, one or more knowledge graphs for multiple portions of the given video, each of the knowledge graphs representing one or more concepts conveyed by the given video, and determining, for each given knowledge graph, a presence share indicative of a level of salience of a concept represented by the given knowledge graph, wherein the concept index is generated based, at least in part, on a number of instances of the given knowledge graph in the given video and a total presence share of the given knowledge graph over the length of the video.

[0006] Generating a concept index for each given video may include, for each given knowledge graph obtained for the given video, summing the presence shares of the given knowledge graphs over the length of the given video, determining a number of portions of the given video that are described by the given knowledge graphs, and generating a concept index for the given video based on a ratio of the summed presence shares for the number of portions.

[0007] The method may include, for each given knowledge graph obtained for a plurality of videos uploaded by a video publisher, generating an inverse document frequency measure for the given knowledge graph based on a total number of the plurality of videos represented by the given knowledge graph, and applying the generated inverse document frequency to a total presence share of the given knowledge graph for each of the plurality of videos.

[0008] The method may include the steps of selecting a pair of videos from among the plurality of videos; generating a count of shared knowledge graphs between the pair of videos, where each shared knowledge graph is a given knowledge graph, which is one of the knowledge graphs representing each video of the pair of videos; and for each particular shared knowledge graph between the pair of videos, deriving a shared similarity score of the particular shared knowledge graph based on a minimum presence share of the knowledge graph with respect to either video of the pair of videos; and deriving a possible similarity score of the particular shared knowledge graph based on a maximum presence share of the knowledge graph with respect to either video of the pair of videos.

[0009] The method may include generating a dissimilarity count based on a number of dissimilar knowledge graphs representing only one of the videos in the video pair, calculating a dissimilarity score for the video pair based on a concept index of the dissimilar knowledge graphs for each video in the video pair, and generating one or more clustering factors for the video pair based on the dissimilarity scores, possible similarity scores, and shared similarity scores.

[0010] Creating multiple groups of videos may include, for each pair of videos, comparing a clustering factor of the pair of videos to a specified level of similarity, including a first pair of videos whose clustering factor meets the specified level in the same group, and excluding a third video whose clustering factor for the first pair of videos does not meet the specified level from the same group.

[0011] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: Unlike traditional similarity techniques, the techniques described herein allow a computer to take substantially different videos and cluster these different videos together with the goal of determining conceptual similarities between these different videos and improving the computer's ability to evaluate groups of videos together. This leads to a more efficient analysis system because, rather than having to evaluate all videos in a cluster (also called a group), aggregate information about a representative subset of videos and / or clusters can be evaluated at run-time, allowing the computer to provide real-time responses to queries that could not be achieved if all of the videos had to be evaluated to provide a response to the query. Furthermore, the techniques described herein prevent errors in similarity determinations that occur when two videos may have very few (e.g., one or two) features in common, but the similarity distance between those features is very small. In these situations, the similarity of a few common features may outweigh all of the dissimilarity between the videos, thereby resulting in an erroneous determination of video similarity.

[0012] The techniques described herein also reduce the amount of time required for a computer to learn a delivery strategy for video (or other content). Legacy computer systems require a fairly long learning period to determine when delivery of content to a client device will result in efficient use of service-side resources (e.g., network bandwidth, server core, etc.) needed to deliver the content and / or efficient use of client-side resources (e.g., battery consumption, processor cycles, data plan utilization, etc.) needed to present the content. This learning period can be as long as a week or more, during which delivery of content often results in wasted server-side resources (e.g., server core utilization) and wasted client-side resources (e.g., battery depletion) while the computer system learns the difference between favorable and inopportune times to deliver the content. However, using the techniques described herein, the learning period may be dramatically reduced, if not eliminated, thereby enabling the computer system to more effectively utilize both server-side and client-side resources compared to using legacy learning period approaches. This improvement in resource utilization is achieved by first determining which cluster a new video (or other content) belongs to, and then leveraging aggregate information from videos in the cluster to initialize the delivery of the new video without requiring a lengthy learning phase.

[0013] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an exemplary online environment. [Diagram 2] FIG. 2 is a block diagram showing concept extraction. [Diagram 3] FIG. 2 is a block diagram illustrating video clustering by video similarity and concept similarity between pairs of videos. [Figure 4] 1 is a flowchart of an example process for using video similarity to provide insight into groups of videos and change how the videos are distributed. [Diagram 5] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference numbers and designations in the various drawings indicate like elements.

[0016] This specification describes techniques for clustering videos based on the overall concept conveyed by different multimedia content, despite differences between the different multimedia content that may prevent the different multimedia content (e.g., a combination of images, videos, and / or audio) from being identified as similar. Using online videos as an example, the resolution, coloring, spoken language, order of elements, and other aspects of two videos may be dramatically different, but these two videos may still convey a similar concept, and these two videos should be clustered together for analysis of videos that convey the same / similar concept. However, using conventional techniques, these two videos may not be clustered together for analysis purposes because conventional computers cannot determine conceptual similarity in videos (or other multimedia content) when individual videos (or other multimedia content) differ in terms of their presentation attributes and / or the way they convey a concept through different video (or other multimedia) presentations.

[0017] Unlike older similarity techniques, the techniques described herein allow a computer to take substantially different videos, determine conceptual similarities between these different videos, and cluster these different videos together in order to improve the computer's ability to evaluate groups of videos together. As described throughout this specification, the evaluation of a group of videos can be used to provide insight into the concepts on which new videos should be developed, determine representative videos and / or aggregate information for each cluster (e.g., concept group) that can be used to reduce subsequent processing of videos in the group, and accelerate optimization of content delivery rules to efficiently utilize server-side and client-side computing resources.

[0018] As will be described in more detail, the techniques generally involve extracting conceptual information from videos (or other content) and clustering the videos based on conceptual similarity, regardless of how the conceptual information is conveyed (e.g., by different people, in different languages, or in different orders). Once the videos are clustered according to concepts, the videos can be analyzed according to concepts to provide insights about the different concepts. A visual mapping of the videos can also be created to visually convey information about closely related concepts and closely related video groups. The visual mapping can be trimmed, thereby improving the usefulness of the map in limited display areas by selecting a representative video for each video group and de-duplicating videos that are substantially the same except for differences in resolution, color scaling, compression techniques, etc. The representative video can be selected in a variety of ways (e.g., similarity to other videos, feedback data, etc.). A visualization of the video mapping can be output to a user, and the mapping can include interactive features that allow the user to obtain information about each video group through interaction with the mapping. For example, a user can click on a representative image of a video group and be provided with information about the videos in that group. Similarly, a user can interact with the mapping (e.g., zoom in / out) to drill down into a video grouping that may reveal further subgroups of videos, or to consider broader grouping level information.

[0019] Throughout this specification, video is used as an example of multimedia content to which the described techniques can be applied in order to provide more specific examples and specific use cases. However, the techniques described herein can be applied to any form of content, and the use of video as an example should not be considered limiting. In other words, the term video can generally be substituted for the term content throughout this specification.

[0020] 1 is a block diagram of an exemplary online environment 100. The exemplary environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 104, a user device 106, a digital component server 108, and a service device 110. The exemplary environment 100 may include a number of different electronic document servers 104, user devices 106, digital component servers 108, and service devices 110.

[0021] A client device 106 is an electronic device capable of requesting and receiving resources over the network 102. Exemplary client devices 106 include personal computers, mobile communication devices, digital assistant devices, and other devices capable of sending and receiving data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data over the network 102, although native applications executed by the client device 106 may also facilitate sending and receiving data over the network 102.

[0022] A digital assistant device includes a device with a microphone and a speaker. A digital assistant device is generally capable of receiving voice input and can respond with content using audible feedback and present other audible information. In some situations, the digital assistant device also includes a visual display or is in communication with a visual display (e.g., by wireless or wired communication). When a visual display is present, feedback or other information may also be provided visually. In some situations, the digital assistant device can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices registered to the digital assistant device.

[0023] An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Native applications (e.g., "apps"), such as applications installed on a mobile, tablet, or desktop computing device, are also examples of electronic documents. Electronic documents can be provided to a client device 106 by an electronic document server 104 ("electronic document server"). For example, an electronic document server 104 can include a server that hosts a publisher website. In this example, a client device 106 can initiate a request for a given publisher web page, and the electronic server 104 that hosts the given publisher web page can respond to the request by sending machine-executable instructions that initiate the presentation of the given web page at the client device 106.

[0024] In another example, electronic document server 104 may include a video server from which client device 106 can download videos (e.g., user-created videos or other videos). In this example, client device 106 may download files needed to play the video in a web browser or a native application configured to play videos.

[0025] An electronic document can include a variety of content. For example, an electronic document can include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. An electronic document can also include dynamic content that can change over time or on a per-request basis. For example, a publisher of a given electronic document can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script that causes the client device 106 to request content from the data source when the given electronic document is processed (e.g., rendered or executed) by the client device 106. The client device 106 integrates the content retrieved from the data source into the given electronic document to create a composite electronic document that includes the content retrieved from the data source.

[0026] In some situations, a given electronic document may include a digital component script that references the service apparatus 110 or a particular service provided by the service apparatus 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request 112 for a digital component (referred to as a "component request") that is sent over the network 102 to the digital component delivery system 110. For example, the digital component script may enable the client device 106 to generate a packetized data request that includes a header and payload data. The component request 112 may include event data that specifies characteristics such as the name (or network location) of the server from which the digital component is being requested, the name (or network location) of the requesting device (e.g., the client device 106), and / or information that the service apparatus 110 can use to select one or more digital components or content to be provided in response to the request. The component request 112 is sent by the client device 106 over the network 102 (eg, a telecommunications network) to a server of the service apparatus 110 .

[0027] The component request 112 may include event data specifying the electronic document being requested and other event features, such as characteristics of the location of the electronic document where the digital component can be presented. For example, event data specifying a reference (e.g., a URL) to the electronic document (e.g., a web page) where the digital component is to be presented, available locations of the electronic document available for presenting the digital component, the size of the available locations (e.g., a portion of a page or a duration within a video), and / or media types eligible for presentation at the locations may be provided to the digital component delivery system 110. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities referenced by the electronic document (e.g., a person, place, or thing) may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate identification of digital components eligible for presentation with the electronic document. The event data may also include a search query submitted from the client device 106 to obtain a search result page (e.g., presenting general search results or video search results).

[0028] The component request 112 may also include event data relating to other information, such as information provided by a user of the client device, geographic information indicating the state or region in which the component request was submitted, the language setting of the client device, or other information that provides context about the environment in which the digital component will be displayed (e.g., the time of the component request, the day of the week of the component request, the type of device on which the digital component will be displayed, such as a mobile device or tablet device). The component request 112 may be transmitted, for example, over a packetized network, and the component request 112 itself may be formatted as packetized data having a header and payload data. The header may specify a destination for the packet, and the payload data may include any of the information described above.

[0029] In response to receiving the component request 112 and / or using the information contained in the component request 112, the service device 110 selects a digital component (e.g., a video file, an audio file, an image, text, and combinations thereof, all of which may take the form of advertising or non-advertising content) to be presented with a given electronic document. In some implementations, the digital component is selected in less than a second to avoid errors that may be caused by delays in the selection of the digital component. For example, a delay in providing the digital component in response to the component request 112 may result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document have been presented at the client device 106. Also, as the delay in providing the digital component to the client device 106 increases, the likelihood increases that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby adversely affecting the user's experience with the electronic document. Furthermore, delays in providing the digital component may result in failure of delivery of the digital component if, for example, the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0030] In some implementations, the service apparatus 110 is implemented in a distributed computing system including, for example, a server and a set of computing devices 114 interconnected to identify and deliver digital components in response to requests 112. The set of computing devices 114 work together to identify a set of digital components eligible to be presented in an electronic document from among a corpus of millions of available digital components (DC1-x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference a corresponding digital component and / or can include delivery parameters (DP1-DPx) that contribute to (e.g., trigger, condition, or limit) the delivery / transmission of the corresponding digital component. For example, the delivery parameters can contribute to (e.g., trigger) the transmission of the digital component by requiring that a component request include at least one criterion that matches (e.g., exactly or at some pre-specified level of similarity) one of the delivery parameters of the digital component.

[0031] The identification of eligible digital components can be segmented into a number of tasks 117a-117c, which are then allocated among the computing devices in the set 114 of computing devices. For example, different computing devices in the set 114 can each analyze different portions of the digital component database 116 to identify various digital components having delivery parameters that match the information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and return (e.g., transmit) results of the analysis (Res 1-Res 3) 118a-118c to the digital component delivery system 110. For example, the results 118a-118c provided by each of the computing devices in the set 114 can identify a subset of digital components eligible for delivery in response to a component request and / or a subset of digital components having certain delivery parameters. Identifying the subset of digital components may include, for example, comparing the event data to delivery parameters and identifying the subset of digital components having delivery parameters that match at least some characteristics of the event data.

[0032] The service apparatus 110 aggregates the results 118a-118c received from the set of computing devices 114 and uses information associated with the aggregated results to select one or more digital components to be provided in response to the component request 112. For example, the service apparatus 110 can select a set of highest-ranking digital components (one or more digital components) based on the results of one or more content evaluation processes. The service apparatus 110 can then generate and transmit over the network 102 response data 120 (e.g., digital data representing the response) that enables the client device 106 to incorporate the set of highest-ranking digital components into a given electronic document, such that the set of highest-ranking digital components and the content of the electronic document are presented together on a display of the client device 106.

[0033] In some implementations, the client device 106 executes instructions included in the response data 120 to configure and enable the client device 106 to retrieve a set of highest-ranking digital components from one or more digital component servers. For example, the instructions in the response data 120 may include a network location (e.g., a uniform resource locator (URL) and a script that causes the client device 106 to send a server request (SR) 121 to the digital component server 108 to retrieve the given highest-ranking digital components from the digital component server 108. In response to the request, the digital component server 108 identifies the given highest-ranking digital components specified in the server request 121 (e.g., in a database that stores a plurality of digital components) and sends digital component data (DC data) 122 to the client device 106 that presents the given highest-ranking digital components in an electronic document at the client device 106.

[0034] The service device 110 can utilize various techniques to evaluate the suitability of various different digital components available to be transmitted in response to a given component request (e.g., an individual component request). For example, the service device 110 can compare the suitability scores of the various different digital components and select one or more of the digital components having the highest suitability scores as the digital components to be transmitted to the client device 106 in response to the given component request. In some situations, the initial suitability score can be determined based on one or more factors. For example, one provider (P1) of a video clip (VC1) can provide a delivery criterion X for VC1, and a different provider (P2) of a different video clip (VC2) can provide a different delivery criterion Y. In this example, assume that the component request simply requests that one digital component be presented with a particular web page or with a particular video. To select which of the two video clips to provide, the service device 110 can rank VC1 and VC2 based on their respective eligibility scores, which may be determined based on comparing the delivery criteria provided by P1 and P2 with the information included in the request 112. In some implementations, the set of delivery criteria that is most similar to the information in the request 112 will have the highest eligibility score and thus be ranked highest. The service device 110 can select the highest ranked video clip to be sent to the client device in response to the component request 112.

[0035] In some situations, the eligibility score is increased (or changed) based on one or more other factors. For example, the service device 110 may generate an adjusted eligibility score for a digital component based on the initial eligibility score for the digital component and quality factors of the digital component.

[0036] The quality factor of a given digital component can quantify the likelihood that the given digital component is a suitable digital component to be provided in response to a given component request. In some implementations, the quality factor is determined based on one or more features specified by the event data. More specifically, the service device 110 can input one or more features from the event data (e.g., geographic information and / or terms from electronic documents) into a machine learning system, which outputs a predicted distribution result, which can be used as the quality factor. The predicted distribution result can be expressed, for example, as a predicted interaction rate for the digital component in the context of the current component request (e.g., click rate, play rate, or another measure of interaction with the digital component).

[0037] Once the quality factor is obtained, the quality factor can be applied to the initial eligibility score to arrive at an adjusted eligibility score. For example, the adjusted eligibility score (AES) can be the product of the initial eligibility score and the quality factor (e.g., AES=quality factor*initial eligibility score). In some circumstances, the adjusted eligibility scores of various different components can be used to rank the digital components (e.g., from highest to lowest), and one or more of the highest ranked digital components can be selected to be transmitted in response to the component request.

[0038] The effectiveness of delivered content (e.g., videos), which can also serve as a proxy for whether server-side and / or client-side computing resources are justified, can be evaluated based on user responses to the content delivered to client devices. Tracking user responses for individual pieces of content (e.g., individual videos) can provide insight into which pieces of content result in effective utilization of computing resources needed to deliver the content, but such evaluation does not provide insight into why certain pieces of content are better for delivery (e.g., make better use of limited server-side and / or computer-side resources and / or elicit a more positive user response). To provide this type of deeper insight into why certain pieces of content are better for delivery, an aggregate analysis of multiple similar pieces of content needs to be performed, and the pieces of content need to be grouped together in a meaningful way.

[0039] One way to group pieces of content is by their conceptual similarity. Conceptual similarity, as used throughout this specification, refers to the similarity between concepts conveyed by two or more pieces of content. As described in more detail below, conceptual similarity between two pieces of content can be determined based on the level of agreement between the concepts conveyed by the two pieces of content, and the level of agreement can be determined regardless of the manner in which those concepts are conveyed.

[0040] For example, assume that Video 1 (V1) is a 30-second animated video that shows a cat chasing a mouse for 25 seconds of the video and presents the text "Call Exterminator X at 555-555-5555" for the remaining 5 seconds of the video, and assume that Video 2 (V2) is a 30-second live video that does not contain the presented text but shows the cat chasing a mouse and the mouse diving to safety. In this example, the two videos may be considered very similar based on the majority of each video showing the cat chasing a mouse, even though there may be significant differences in the concept of a cat chasing a mouse and the way in which this concept is conveyed (e.g., animated vs. live). If these two videos are grouped together along with other videos that target the concept of a cat chasing a mouse, the aggregate information (e.g., user response information) can be used to effectively evaluate the effectiveness of the videos targeting the concept of a cat chasing a mouse, regardless of the differences in the way those concepts are conveyed.

[0041] As described in more detail below, the effectiveness of videos targeting each concept can be delineated for each user group, and insights about concepts to be used in videos (or other content) can be generated based on the target audience and the effectiveness of different concepts when presented to the target audience. Furthermore, as described below, grouping and evaluating videos in this manner can enable a computer system to more quickly learn optimal delivery parameters for a new video based on insights determined for a group of existing videos that belong to the same cluster (video group) as the new video. The ability to learn optimal delivery parameters more quickly reduces the waste of resources that would be incurred by using a traditional learning period, which can take days or weeks in some cases, and also reduces the computing resources required for a computer to learn the delivery parameters.

[0042] The service device 110 includes a concept evaluation device ("CEA") 150 that includes one or more data processors. The CEA 150 is configured to cluster videos (and other content) based on concept similarity and evaluate the resulting video groups (or other content groups) to determine the effectiveness of different video concepts. This evaluation can be used to surface insights regarding the use of certain concepts depending on the target audience (e.g., the group of users that the content is aimed at). As described below, these insights can be communicated visually in a table format, a graph format, or another format. Furthermore, these insights can be used to recommend concepts for new videos, provide concept feedback for groups of existing videos, and initialize distribution rules for new videos based on the concepts that those new videos are aimed at.

[0043] FIG. 2 is a block diagram 200 illustrating concept extraction from a video. The block diagram 200 includes a video player 202 on which the video is presented. Although the video player 202 is shown as part of a web page, the video player 202 may be implemented in a native application (e.g., in a mobile or tablet device) or in another suitable device, such as a streaming video player that connects to or is integrated into a television. Additionally, the concept extraction described below may occur independent of the video player 202, which is included merely to help explain the concept extraction. Additionally, although the concept extraction is described as being performed by the service device 110, the concept extraction may occur in a portion of the service device, such as the CEA 150, or another suitable device.

[0044] A video available to be played by the video player 202 has a play time, also referred to as the length of the video. For example, the play timer 204 of the video player 202 indicates that the play time of the currently loaded video is one minute. To evaluate the context of the video, the service device 110 (e.g., by the CEA 150) can determine the context of the video over the length of the video. For example, the service device 110 can select different timestamps over the length of the video from which the context of the video is extracted. In this example, the video is segmented into four different segments ("quartiles") by the selection of timestamps of 0, 15, 30, 45, and 60 seconds, identified by reference numbers 206, 208, 210, 212, and 214, respectively. This quartile segmentation is used to illustrate the concept of timestamp selection while avoiding cluttering the diagram. However, in practice, the service device 110 can select timestamps as finely or coarsely as necessary. In some implementations, timestamps are selected for every second of the video duration, and in some implementations, the service device 110 may select timestamps that are less than one second (eg, 1 / 6th of a second).

[0045] Each selected timestamp is a representation of a portion of the video that is evaluated to determine the concept being conveyed at that point in the video. In some implementations, the service device 110 examines the video frames presented at each timestamp to determine the context of the video at that timestamp. Returning to the previous example, the service device 110 can evaluate the video at the beginning of the video (e.g., 0 seconds), 15 seconds, 30 seconds, 45 seconds, and the end of the video (e.g., 1:00).

[0046] The evaluation of the video at each timestamp may include identifying any objects presented by the video, attributes of those objects, and / or relationships between the objects presented by the video. For example, the video presented on the video player 202 includes a person 216, a chair 218, a table 220, and two additional people. In this example, the service device 110 may use object detection techniques known in the art to identify these objects, their relative positions, and attributes such as color, pattern, or other visual attributes.

[0047] Once the service device 110 obtains information about the objects shown by the video at a given timestamp, the service device 110 can generate a knowledge graph that represents the concepts conveyed by the video at the given timestamp. As used herein, a knowledge graph is a representation of relationships between unique entities, and the knowledge graph may be stored in one or more data stores. Each node may represent a different unique entity, and the nodes may be connected by graph edges (e.g., logical links) that represent relationships between the entities. The knowledge graph may be implemented, for example, in a graph form or a data structure that includes data representing each node and data representing relationships between each node.

[0048] Returning to the example above, the knowledge graph of the video shown in the video player 202 may include separate nodes for the person 216, the chair 218, the table 220, and two additional people. Each of these nodes may be connected by graph edges that represent relationships between the entities. For example, the node representing the chair 218 may be linked by an edge labeled "sitting" with a node representing the person sitting in the chair to represent that the video shows the person sitting in the chair 218. Similarly, the node representing the table 220 may also be linked by an edge labeled "rests hands" with a node representing the person sitting in the chair to represent that the person sitting in the chair 218 has his hands resting on the table 220. Other attributes of the video may similarly be represented by the knowledge graph, such as color, text, or other attributes. Of course, the edges between the nodes need not include labels, but rather may simply indicate some relationship between the nodes.

[0049] For a given video, the service device 110 can create multiple knowledge graphs. For example, the service device 110 can create a different knowledge graph for each timestamp selected for a given video. In the above example, the service device 110 generates five different knowledge graphs 222, 224, 226, 228, and 230. The service device 110 stores each of the knowledge graphs in a knowledge graph database 232 for further processing. For example, as described below, the service device 110 can use the knowledge graphs to evaluate concept similarity between videos and cluster the videos based on concept similarity.

[0050] FIG. 3 is a block diagram 300 illustrating video clustering by video similarity and concept similarity between pairs of videos. The block diagram 300 shows the service device 110 receiving as input two sets of knowledge graphs for processing. More specifically, the service device 110 receives a video 1 knowledge graph (V1KG) 302 and a video 2 knowledge graph (V2KG) 304. V1KG 302 includes multiple knowledge graphs created based on objects presented at different selected timestamps in video 1, and V2KG includes multiple knowledge graphs created based on objects presented at different selected timestamps in video 2. Knowledge graph 306 is an exemplary knowledge graph visualization that may be included in V1KG 302, and knowledge graph 308 is an exemplary knowledge graph visualization that may be included in V2KG 304. Each of knowledge graphs 306 and 308 may represent portions of videos presenting similar content as frames of the video presented in the video player 202 of FIG. 2.

[0051] For example, knowledge graph 306 includes node 310 representing a person, node 312 representing a chair, and node 314 representing a table. These nodes 310, 312, and 314 may represent, for example, chair 218, desk 220, and a person sitting in chair 218, as shown by Figure 2. Similarly, knowledge graph 308 includes nodes 316, 318, and 320, which represent a person, a chair, and a table, respectively. As explained further below, the person, chair, and table represented by nodes 316, 318, and 320 may be different people, tables, and chairs than those shown in video player 202 of Figure 2, or they may represent the same person, table, and chair.

[0052] The knowledge graph 306 also includes a node 322 representing a well-known actor, "Celebrity X," as well as nodes 324 and 326 representing the colors brown and green, respectively. The knowledge graph also includes edges connecting the nodes and specifying the relationship types between the nodes. For example, the knowledge graph 306 includes an edge 328 between node 310 and node 312, indicating that there is a relationship between the person and the chair. In this example, the edge 328 has a label indicating the relationship "sitting" with an arrow pointing to the node 312, which represents the chair. Thus, this edge indicates that the person and the chair are related by the person sitting in the chair, as shown in FIG. 2. Similarly, the knowledge graph 206 includes an edge 330 connecting the nodes 310 and 314, and having a label "hands placed on"; thus, the edge 330 represents the person placing both hands on the table, as shown by FIG. 2. The knowledge graph 306 also includes an edge 332 between nodes 301 and 322, indicating that the person represented by node 310 is Celebrity X and is an actor. Edge 334 is between node 312 and node 326 and indicates that the chair is green, and edge 336 is between node 314 and node 324 and indicates that the table is brown.

[0053] Similar to knowledge graph 306, knowledge graph 308 also includes node 338, which represents the color brown. However, knowledge graph 308 does not include nodes that represent celebrity X or the color green. Knowledge graph 308 also includes edges that connect the nodes of knowledge graph 308 and represent relationships between the nodes. For example, edge 340 connecting nodes 316 and 318 represents a person sitting in a chair, and edge 342 connecting nodes 316 and 320 indicates that a person has both feet up on the desk. Additionally, edges 344 and 346 indicate that the color of both the table and chair are brown.

[0054] The service device 110 processes the sets of knowledge graphs (eg, V1KG 302 and V2KG 304) to determine conceptual similarities between the videos and to cluster the videos based, for example, on the similarities of the knowledge graphs in each of the sets.

[0055] As part of the process, the service device 110 generates a concept index 1 for video 1 (348) and also generates a concept index 2 for video 2 (350). The concept index for each video represents the importance of each concept shown by each video. The concept index for a given video may be generated based on the concepts conveyed by the objects shown in the given video and the level of salience of each of the concepts over the length of the video. In some implementations, the concept index for a given video may be generated, for example, using a set of knowledge graphs obtained for the given video. For example, referring to FIG. 3, the service device 110 may generate a concept index 2 using the set of knowledge graphs 304 for video 2 (350).

[0056] More specifically, the service device 110 can examine all of the knowledge graphs in the set 304 to identify, for each knowledge graph, the total presence share of the knowledge graph over the length of the video 2. Because each knowledge graph represents a concept conveyed by a given video, the total presence share of a particular knowledge graph can represent the total presence share of the corresponding concept conveyed over the length of the video. In some implementations, the total presence share of a particular knowledge graph over the length of the video can be determined by aggregating the presence share of each knowledge graph at each timestamp of the video being evaluated.

[0057] For example, as described with reference to FIG. 2, assuming five timestamps are selected, the total presence share of a particular knowledge graph in this example may be the sum of the presence shares of that particular knowledge graph at each of the five timestamps (e.g.,

[0058]

number

[0059] , where ts represents a timestamp). The output obtained by summing the presence shares over the length of the video (e.g., at each of the selected timestamps) is typically a numerical value that can be combined with other information to create a concept index, as described further below. Referring to FIG. 3, the service device 110 can determine a total presence share of the knowledge graph 308 by summing the presence shares of the knowledge graph 308 for each of the evaluated timestamps in the video 2.

[0060] The presence share of a particular knowledge graph may be based on a number of factors, such as the portion of the frame occupied by the object represented by the knowledge graph, the location of the object within the frame, or other factors corresponding to the salience of the object within the frame. Generally, the presence share of a knowledge graph representing an object will increase as the salience of the object within the frame increases. For example, when an object is larger or occupies a larger portion of the frame, the presence share of the knowledge graph representing that object will increase. The presence share assigned to each knowledge graph at each timestamp may be a number between 0 (minimum salience) and 1 (maximum salience), although other suitable scales (e.g., 0 to 10 or 0 to 100) may be used.

[0061] As part of the concept index generation, the service device 110 can also determine the total number of timestamps at which each knowledge graph representing each concept is found in a given video. In some implementations, the service device 110 can determine the total number of timestamps of a video at which a particular knowledge graph is found by searching the knowledge graphs acquired at each timestamp and determining whether a particular knowledge graph was collected for that timestamp, and increment a counter for each instance of a particular knowledge graph found at a timestamp. For example, as described with reference to FIG. 2, if five timestamps are selected, the service device can search the knowledge graphs collected at each timestamp to determine whether any of the collected knowledge graphs matched the particular knowledge graph, and increment a counter for each timestamp that has a knowledge graph that matches the particular knowledge graph. The value of the counter after all timestamps for a given video have been searched represents the total number of timestamps (also called the number of parts) of a given video that are described by a particular knowledge graph.

[0062] The service device 110 can combine, for a given video, the total presence share of a particular knowledge graph over the length of the given video and the total number of timestamps in the given video where the particular knowledge graph is detected to arrive at a concept value that corresponds to the level of salience of the concept represented by the particular knowledge graph over the length of the given video. In some implementations, the concept value can be calculated by taking the ratio of the total presence share for a particular knowledge graph representing a concept to the total number of timestamps in the given video where the particular knowledge graph is detected. The generalized ratio can take the form expressed by relationship (1).

[0063]

number

[0064] however, Concept Value Kgi is the concept value of the concept represented by the i-th knowledge graph, Summed Presence SharesKGi is the summed presence shares of the i-th knowledge graph for a given video, Total Instances Kgi is the total number of timestamps (or parts) of a given video for which the i-th knowledge graph is detected.

[0065] In some implementations, each concept value generated for a given video may be considered a concept index. In some implementations, the concept index for a given video includes concept values ​​for each of multiple different concepts conveyed by the video and represented by different knowledge graphs. For example, when the concept index for a video includes multiple different concept values ​​for different concepts, each value of the concept index may correspond to a different concept and a knowledge graph representing that concept.

[0066] The service device 110 generates a knowledge graph inverse document frequency ("KGIDF") measure based on the occurrence of different knowledge graphs across all (or a subset) of the videos to be clustered (352). The KGIDF may be used to adjust the inverse document frequency of individual knowledge graphs in each video. All KGIDFs for a video may be generated, for example, using relationship (2).

[0067]

number

[0068] where |{v∈#of videos:kgi∈v}| is the number of videos in every KGi, where KGi is the i-th knowledge graph.

[0069] The service device 110 applies KGIDF to the set of knowledge graphs 302 for video 1 (354) and to the set of knowledge graphs 304 for video 2 (356). The application of KGIDF to the set of knowledge graphs results in an adjusted inverse document frequency for each of the knowledge graphs. The adjusted KGIDF prevents errors that occur when two videos have very few common knowledge graphs. For example, assume that two videos only share one common attribute (e.g., purple), but the common attribute is very similar (e.g., has a small similarity distance). In this situation, some similarity techniques will result in a very high similarity, despite the fact that the videos may not be similar in other ways at all. In other words, common attribute similarity using some similarity techniques (e.g., similarity=1 / sqrt(distance)) may weight one attribute similarity more heavily than dissimilarity, resulting in erroneous similarity determinations.

[0070] In some implementations, the KGIDF is applied to the set of knowledge graphs according to relationship (3). adj value {video j|KGi} := value {video j|KGi} *idf KGi (3) however, adj value {video j|KGi} is the refinement value of the i-th knowledge graph for the j-th video, value {video j|KGi} is the total presence share of the i-th knowledge graph over the length of the j-th video, idf KGi is the KGIDF of the i-th knowledge graph.

[0071] For example, the service device 110 may generate an adjusted value for the knowledge graph 308, sometimes referred to as an adjusted presence share, by multiplying the total presence share of the knowledge graph 308 in video 2 by the KGIDF of the knowledge graph 308 across all of the videos analyzed by the service device 110.

[0072] The service device 110 derives (e.g., determines or calculates) a similarity score for the shared knowledge graph between the pair of videos (358). With reference to FIG. 3, the service device 110 can use the set of knowledge graphs 302 and the set of knowledge graphs 304 to determine a similarity score for a video pair including video 1 and video 2. For example, the service device 110 can derive a shared similarity score for the pair of videos and possible similarity scores for the pair of videos. In some implementations, a separate shared similarity score is derived for each knowledge graph found in both videos of the pair (e.g., video 1 and video 2). Similarly, the service device can derive a separate possible similarity score for each of the knowledge graphs found in both videos of the pair.

[0073] The shared similarity score refers to a similarity score that is based on the minimum amount of concept salience between a pair of videos. For example, the shared similarity score can represent the minimum total presence share of a particular knowledge graph in either of the videos of the pair. In other words, the service device 110 can determine which of the videos of the pair (e.g., video 1 or video 2) has a lower adjustment value for a particular knowledge graph, determined, for example, using relationship (3), and select the lower adjustment value for the particular knowledge graph as the shared similarity score of the pair of videos. This determination can be made, for example, using relationship (4). SS_KGt (video 1, video 2) = min(adj value {video1|KGt} , adj value {video2 |KFt} ) (4) however, SS_KGt (video 1, video 2) is the shared similarity between video1 and video2 for the i-th knowledge graph, adj value {video1|KGt} is the adjustment value (see relation 3) for the i-th knowledge graph in video 1, adj value {video2|KFt} is the adjustment value (see relation 3) for the i-th knowledge graph in video 2.

[0074] The possible similarity score refers to a similarity score that is based on the maximum amount of concept salience of the pair of videos. For example, the possible similarity score can represent the maximum total presence share of a particular knowledge graph in either of the videos of the pair. In other words, the service device 110 can determine which of the videos of the pair (e.g., video 1 or video 2) has a higher adjustment value for a particular knowledge graph, determined, for example, using relationship (3), and select the higher adjustment value for the particular knowledge graph as the possible similarity score of the pair of videos. This determination can be made, for example, using relationship (5). PS_KGi (video 1, video 2) = max(adj value {video 1|KGi} , adj value {video2 |KGi} ) (5) however, PS_KGi (video 1, video 2) is the possible similarity between video1 and video2 for the i-th knowledge graph, adj value {video 1|KGi} is the adjustment value (see relation 3) for the i-th knowledge graph in video 1, adj value {video2 |KGi} is the adjustment value (see relation 3) for the i-th knowledge graph in video 2.

[0075] The service device 110 derives (e.g., determines or calculates) a dissimilarity score of the dissimilar knowledge graphs in each video of the video pair (358). With reference to FIG. 3, the service device 110 can use the set of knowledge graphs 302 and the set of knowledge graphs 304 to determine a dissimilarity score of a video pair including video 1 and video 2. In some implementations, the service device 110 can generate a dissimilarity score of a video pair by summing the concept values ​​of the dissimilar similarity graphs in each of the videos of the pair. For example, the service device 110 can identify the knowledge graph of video 1 that is not included in video 2, and the knowledge graph of video 2 that is not included in video 1. In this example, the service device can sum the concept indexes of the dissimilar knowledge graphs of video 1 and video 2, and output the sum as a dissimilarity score of the video pair including video 1 and video 2.

[0076] Another dissimilarity measure that may be determined by the service device 110 is the total number of dissimilar knowledge graphs in each video of a video pair, which is called a dissimilarity count, which may be determined by incrementing a counter when a knowledge graph identified in one of the videos of a pair, where the service device 110 does not find a match, is not found in the other video of the pair.

[0077] The service device clusters the videos based on the similarity and dissimilarity scores (362). The clustering creates multiple groups of videos, each of which includes videos that have at least a specified level of similarity. For example, one group created by clustering includes multiple videos, each of which has a specified level of similarity with one or more other videos in the group. Details regarding how the similarity and dissimilarity measures are used to create the groups are described in more detail with reference to FIG. 4.

[0078] 4 is a flowchart of an example process 400 for using video similarity to provide insight into a group of videos and modify how the videos are distributed. The operations of process 400 may be performed, for example, by the service apparatus 110 (and / or the CEA 150) of FIG. 1. The operations of process 400 may also be implemented using instructions encoded on one or more computer-readable media, which may be non-transitory. When the instructions are executed by one or more data processing apparatuses (e.g., one or more computing devices), the instructions cause the one or more data processing apparatuses to perform the operations of process 400.

[0079] A video is acquired by the service device (402). In some implementations, the acquired video is acquired when the video is uploaded by a video publisher. A video publisher may be a user who uploads a video to a video sharing site or an entity that creates a video for presentation with other content. For example, the acquired video may be in the form of a digital component provided for distribution with other content, as described with reference to FIG. 1.

[0080] One or more concept indexes are generated for each given video acquired (404). The one or more concept indexes may include a separate concept index generated for each given knowledge graph acquired for the given video. In some implementations, the concept index for a given video is generated based on at least (i) the concepts conveyed by one or more objects shown in the given video, and (ii) the level of salience of the concepts in the given video. As described above with reference to FIG. 3, the one or more concepts conveyed by one or more objects shown in the given video may be represented by one or more knowledge graphs acquired for multiple portions of the given video (e.g., at multiple different timestamps). The one or more knowledge graphs may be acquired, for example, through an evaluation of the video and / or from a database that stores previously generated knowledge graphs, as described with reference to FIG. 2.

[0081] In some implementations, a concept index may be generated by evaluating the presence of a knowledge graph in a video. For example, a particular concept index may be generated for a concept represented by a particular knowledge graph based on the number of times the knowledge graph is detected in the video and the total presence share of the knowledge graph over the length of the video, as described in detail with reference to FIG. 3. More specifically, the concept index for a particular knowledge graph may be determined as the ratio of (i) the total presence share of the particular knowledge graph over the length of the video and (ii) the total number of instances of the particular knowledge graph in a given video.

[0082] As explained above, the presence share at a given timestamp in a video indicates the level of salience of one or more concepts represented by a given knowledge graph at that timestamp of the video, which also indicates the level of salience of one or more concepts conveyed by the video. In some implementations, the total presence share of a given knowledge graph (and corresponding concepts) is determined by summing the presence shares of the given knowledge graph over the length of the video. Meanwhile, several portions of a video described by a given knowledge graph may be determined based on several timestamps (or portions) of the video where a particular knowledge graph is found.

[0083] An inverse document frequency is generated and applied for each given knowledge graph (406). The inverse document frequency for a given knowledge graph is generated, for example, by determining the frequency of the given knowledge graph among all knowledge graphs obtained from the video being evaluated. In some implementations, the inverse document frequency for a given knowledge graph may be generated based on the total number of videos represented by the given knowledge graph. In other words, the inverse document frequency for a given knowledge graph may be based on how many of the videos have a set of knowledge graphs that includes the given knowledge graph. In some implementations, the inverse document frequency for a given knowledge graph may be determined using relationship (2) described with reference to FIG. 3.

[0084] The inverse document frequency of a given knowledge graph may be applied to a by, for example, multiplying the inverse document frequency with the total presence share of the given knowledge graph in a particular video. For example, relationship (3) described with reference to FIG. 3 may be used to apply the inverse document frequency. In some implementations, separate inverse document frequencies are generated for each different knowledge graph, and the total presence share of each given knowledge graph in each video may be adjusted using the inverse document frequency generated for the given knowledge graph. In other words, the generated inverse document frequency is applied to the total presence share of the given knowledge graph for each of the multiple videos.

[0085] A pair of videos is selected from among the multiple captured videos 408. In some implementations, the pair of videos may be selected randomly (or pseudo-randomly).

[0086] A similarity measure is derived for the pair of videos (410). In some implementations, deriving the similarity measure includes deriving a shared similarity score for a shared knowledge graph between the pair of videos and deriving possible similarity scores for the shared knowledge graphs. A similarity score may be generated for each particular shared knowledge graph between the pair of videos. In some implementations, the shared knowledge graph is a knowledge graph found in each video of the pair of videos. For example, if a matching knowledge graph is also identified in video 2, then the particular knowledge graph identified in video 1 may be considered the shared knowledge graph. It should be noted that a matching knowledge graph may be, but is not necessarily, an exact match. In other words, a matching knowledge graph may be a knowledge graph that has one or more of the same nodes that represent one or more objects. The level of match required to consider two knowledge graphs as matching may vary depending on the application.

[0087] The shared similarity score may be derived based on the minimum presence share of the shared knowledge graph in either video of the pair of videos. For example, as described above with reference to FIG. 3, the shared similarity score of a particular shared knowledge graph may be the minimum adjusted presence share (e.g., an adjusted value) of the particular shared knowledge graph in either video of the pair, as shown in relationship (4).

[0088] The possible similarity score may be derived based on the maximum presence share of the shared knowledge graph in either video of the pair of videos. For example, as described above with reference to FIG. 3, the possible similarity score of a particular shared knowledge graph may be the maximum adjusted presence share (e.g., adjusted value) of the particular shared knowledge graph in either video of the pair, as shown in relation (5).

[0089] The derivation of the similarity measure may also include generating a count of shared knowledge graphs between a pair of videos. Each shared knowledge graph is a given knowledge graph, one of the knowledge graphs representing each video of the pair of videos. As explained above, a shared knowledge graph may be identified in one of the videos by finding a matching knowledge graph for a given knowledge graph identified in the other video of the pair. The count of shared knowledge graphs may be generated, for example, by incrementing (or possibly counting) a counter each time a given knowledge graph of one video of the pair is considered to match a knowledge graph of the other video. In some situations, the counter may be incremented only once for every match of a given knowledge graph. In other situations, the counter may be incremented for every instance of a match of a given knowledge graph of one video with multiple instances of a matching knowledge graph of the other video.

[0090] One or more dissimilarity measures are derived for the pair of videos (412). In some implementations, deriving the dissimilarity measures includes generating a dissimilarity count and calculating a dissimilarity score. The dissimilarity measures may be derived, for example, using dissimilar knowledge graphs from each of the pair of videos. For example, while the similarity measures described above were generated using matching knowledge graphs for the pair of videos, the dissimilarity measures are generated using knowledge graphs from each video for which no matching knowledge graph was found for the other video of the pair. As can be appreciated, regardless of the criteria used to determine a match, there is a set of matching knowledge graphs used to derive the similarity measures and a set of dissimilar knowledge graphs (or non-matching knowledge graphs) used to determine the dissimilarity measures.

[0091] The generation of the dissimilarity count can be generated based on the number of dissimilar knowledge graphs identified for the pair of videos. The number of dissimilar knowledge graphs can be, for example, the total number of knowledge graphs in both videos where no matching knowledge graph was identified in the other video of the pair. The number of dissimilar knowledge graphs can be, for example, generated by incrementing a counter each time a knowledge graph match in one video is not found in the other video, and the number of dissimilar knowledge graphs can be the value of the counter after all knowledge graphs of the two videos have been analyzed to find a match. In other words, the dissimilarity count can be the number of dissimilar knowledge graphs that represent only one of the videos of the pair of videos.

[0092] A dissimilarity score for a video pair may be calculated based on the concept indexes of the dissimilar knowledge graphs of each video of the video pair. For example, a concept index may be identified for each dissimilar knowledge graph for each video of the pair. These concept indexes for each of the dissimilar knowledge graphs may then be summed to obtain a dissimilarity score for the two videos of the pair.

[0093] Multiple groups of videos are created (414) based on the similarity and dissimilarity measures. In some implementations, each given group of videos is created to include only videos that have at least a specified level of similarity with other videos in the given group. For example, to be included in a given group, a given video may be required to satisfy a set of similarity conditions with respect to each other video in the group. The set of similarity conditions may relate to the similarity and dissimilarity measures described above and may be used to ensure that all of the videos relate to conceptually similar concepts.

[0094] The creation of the groups can be based (directly or indirectly) on, for example, the concept indexes of the videos. For example, a dissimilarity measure is derived based on the concept indexes of the directly dissimilar knowledge graphs, and a similarity measure is derived based on the presence shares that are also used to determine the concept indexes.

[0095] In some implementations, the creation of a group may depend on one or more of the following factors: 1) The number of shared knowledge graphs, denoted by n_similar, which refers to the number of matching knowledge graphs between a pair of graphs. 2) The number of dissimilar knowledge graphs, denoted by m_dissimilar, which refers to the number of knowledge graphs in each video of the pair that have no match in the other video of the pair. 3) The sum of possible similarities, denoted by total_possible_similarity, which refers to the mathematical sum of the possible similarity measures for all shared knowledge graphs of a video pair. 4) The sum of shared similarities, denoted by shared_similarity_mass, which refers to the mathematical sum of the shared similarity measures for all shared knowledge graphs of a pair. 5) The sum of dissimilarity scores, denoted as dissimilarity_mass, which is the sum of the dissimilarity scores for all dissimilar knowledge graphs of a video pair. 6) The dissimilarity share, denoted by dissimilarity_share, where dissimilarity_share = dissimilarity_mass / shared_similarity_mass. 7) The total mass, represented by total_mass, where total_mass = dissimilarity_mass + total_possible_similarity. 8) The share of dissimilarity, represented by share_of_dissimilarity, where share_of_dissimilarity = dissimilarity_share / total_mass. 9) The share of realized similarity, represented by share_of_realized_similiarity, where share_of_realized_similiarity = shared_similarity_mass / total_mass. 10) The share of possible similarity, represented by share_of_possible_similarity, where share_of_possible_similarity = total_possible_similarity / total_mass. 11) The comparison score, denoted by comparison_score, where comparison_score = shared_similarity_mass / dissimilarity_mass. 12) The creative approach similarity score, represented by creative_approach_similarity_score, where creative_approach_similarity_score = total_possible_similarity / dissimilarity_mass.

[0096] In some implementations, one or more of these factors are used to describe the relationship (e.g., similarity) between each pair of videos being evaluated, and each pair of videos in the retrieved videos is characterized by these factors. The factors used to describe the relationship between pairs of videos can be input to a clustering algorithm, which groups the videos together based on these factors.

[0097] In some implementations, a group of videos may be visually presented in a graphical user interface. To reduce clutter in the graphical user interface and to improve processing time, a group of videos may be deduplicated to remove videos from the display that are identical. The deduplication process may, for example, identify pairs of videos in a given group that have a shared similarity above a threshold and / or a dissimilarity below a specified level that is indicative of the videos being substantially identical.

[0098] Insights are generated for the multiple video groups (416). In some implementations, insights for each given video group are determined based on data obtained through the feedback loop. For example, user reactions to presented videos may be obtained through the feedback loop and logged with a reference to the presented video and / or the group that contains the video. The reactions may be collected for each group and used to provide insights about the group of videos, concepts conveyed by the group of videos, and other attributes of the group of videos.

[0099] The feedback loops can be implemented, for example, as scripts that are triggered when a user interacts with the video (e.g., clicking on the video), as pings that are automatically generated during the playback of the video to report watch time, or using other mechanisms. With these feedback loops, the user is not asked to separately report their reaction to the video presented to them. Instead, the reaction (e.g., positive or negative) can be inferred based on data collected using the various feedback loops.

[0100] Insights generated based on data collected about multiple different video groups can include, for example, identification of groups of videos that receive higher levels of positive feedback data compared to other groups of videos that receive lower levels of positive feedback data. Once groups with higher levels of positive feedback data are identified, a video publisher can be provided with information about the types of videos that receive higher levels of positive feedback data, allowing the video publisher to incorporate similar features in new videos. Similarly, a video publisher can be provided with information about groups of videos that receive lower levels of positive feedback data, allowing the video publisher to omit similar features from new videos.

[0101] In some implementations, insights may be created for each viewer type. For example, the feedback data may be segmented based on any type of viewer characteristic (e.g., device type, interest group, time of day, geographic region, etc.) and insights may be generated for each viewer type or viewer segment. When the data is segmented in this manner, insights (e.g., as described above) may be provided to video publishers based on the types of audiences for which they are creating videos.

[0102] The insights generated can extend beyond those used by video publishers to create videos that receive higher levels of positive feedback. In some implementations, the insights can be used to significantly reduce the amount of time required to train a computer to efficiently and effectively distribute new videos that are uploaded. For example, when a new video is uploaded, the new video can be evaluated and clustered as described throughout this specification. When a new video is assigned to a cluster, insights about the videos in that cluster can be used to generate initial distribution criteria for the new video based on the known similarity of the new video to other videos in the cluster. By generating initial distribution criteria in this manner and distributing the new video using the initial distribution criteria generated based on insights of other videos in the cluster, the training period required by existing systems (which can be as long as a week or more) is significantly reduced or eliminated, and the computer system can much more quickly fine-tune the distribution criteria for the new video to arrive at an optimal set of distribution criteria.

[0103] The manner in which the at least one video is distributed is altered (418) based on the insight. In some implementations, altering the manner in which the at least one video is distributed can include adjusting distribution criteria for the video. For example, assume that the initial target audience for a given video in a cluster is for hockey fans in Atlanta, GA, but the group of videos in which the given video is included has low feedback data for that group but high feedback data for hockey fans in Atlanta, GA. In this example, the distribution criteria for the given video can be adjusted to increase the likelihood that the given video is presented to an audience of hockey fans in Atlanta, GA. In another example, the time at which the video is distributed or the content with which the video is distributed can be adjusted based on the insight generated for each of the groups. Additionally, attributes of the video itself can be altered to increase the likelihood that the feedback data received is more positive.

[0104] 5 is a block diagram of an exemplary computer system 500 that may be used to perform the operations described above. The system 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 may be interconnected, for example, using a system bus 550. The processor 510 is capable of processing instructions for execution within the system 500. In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi-threaded processor. The processor 510 is capable of processing instructions stored in the memory 520 or on the storage device 530.

[0105] The memory 520 stores information within the system 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory 520 is a volatile memory unit. In another implementation, the memory 520 is a non-volatile memory unit.

[0106] The storage device 530 is capable of providing mass storage for the system 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0107] The input / output device 540 provides input / output operations to the system 400. In one implementation, the input / output device 540 may include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., an RS-232 port, and / or a wireless interface device, e.g., an 802.11 card. In another implementation, the input / output device may include a driver device configured to receive input data and send output data to the peripheral device 560, e.g., a keyboard, a printer, and a display device. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0108] Although an exemplary processing system is illustrated in FIG. 5, the implementation and functional operations of the subject matter described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and structural equivalents thereof, or in any combination or combinations of one or more of them.

[0109] This specification refers to a service device 110. As used herein, a service device 110 is one or more data processing devices that perform operations to facilitate the delivery of content over a network. The service device 110 is shown as a single block in the block diagram. However, while the service device 110 can be a single device or a single set of devices, this disclosure contemplates that the service device 110 can also be a group of devices or even multiple different systems that communicate to provide various content to the client device 106. For example, the service device 110 can encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, or any other service.

[0110] An electronic document (for simplicity's sake, referred to simply as a document) may, but need not, correspond to a file: a document may be stored as part of a file that holds other documents, in a single file dedicated to that document, or in multiple cooperating files.

[0111] The subject matter and embodiments of the operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in any combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, the computer storage medium is not a propagating signal, and the computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagating signal. A computer storage medium may also be, or be contained in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0112] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0113] The term "data processing apparatus" encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the above. The apparatus may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing infrastructures, and grid computing infrastructures.

[0114] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but need not. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0115] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, or devices implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0116] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing actions according to the instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Moreover, a computer can be incorporated into another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few examples. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0117] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending documents to a device used by the user and receiving documents from the device, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0118] An embodiment of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0119] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to client devices (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., results of user interaction) may be received at the server from the client device.

[0120] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as working in some combinations and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0121] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0122] Above, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0123] 100 Online Environment 102 Network 104 Electronic Document Server 106 User Devices 108 Digital Component Server 110 Service equipment 112 Component Requests 114 Set of computing devices 116 Digital Component Database 117 Tasks 118 results 120 Response data 121 Server Request (SR) 122 Digital component data (DC data) 150 Concept Evaluation Apparatus (CEA) 202 Video Player 204 Playback Timer 216 people 218 Chair 220 Tables / Desks 400 Systems 500 Computer Systems 510 Processor 520 Memory 530 Storage Devices 540 Input / Output Devices 560 Peripheral Devices

Claims

1. 1. A method implemented by a data processing apparatus, comprising: obtaining, by a service device, a video uploaded by a video publisher; generating, by the service device, a concept index for each given video, the concept index being generated based on at least (i) concepts conveyed by one or more objects shown in the video, and (ii) a level of salience of the concepts in the given video; creating, by the service device, a plurality of video groupings based on the conceptual index of the videos, where each given video group of the plurality of video groupings is created to include two or more different videos, each having a specified level of similarity with other videos of the given video group; generating, by the service device, insights about the plurality of video groups based on data obtained via a feedback loop; modifying, by the service device, how at least one video is distributed over a network based on the insight about the given video group that includes the at least one video; A method comprising:

2. generating said concept index for each given video, For each given one of the videos, obtaining one or more knowledge graphs for a plurality of portions of the given video, each of the knowledge graphs representing one or more concepts conveyed by the given video; determining, for each given knowledge graph, a presence share indicative of the level of salience of the concept represented by the given knowledge graph, the concept index being generated based at least in part on the number of instances of the given knowledge graph in the given video and the total presence share of the given knowledge graph over the length of the video; 2. The method of claim 1, comprising:

3. generating said concept index for each given video, For each given knowledge graph obtained for the given video, summing the presence share of the given knowledge graph over the length of the given video; determining a number of portions of the given video that are described by the given knowledge graph; generating the concept index for the given video based on a ratio of the aggregated presence shares to the number of portions; 3. The method of claim 2, comprising:

4. For each given knowledge graph obtained for a number of videos uploaded by the video publisher, generating an inverse document frequency for the given knowledge graph based on a total number of the plurality of videos represented by the given knowledge graph; applying the generated inverse document frequency to a total presence share of the given knowledge graph for each of the plurality of videos; 4. The method of claim 3, further comprising:

5. selecting a pair of videos from among the plurality of videos; generating a count of shared knowledge graphs between said pairs of videos, each shared knowledge graph being a given knowledge graph, one of said knowledge graphs representing each video of said pair of videos; For each particular shared knowledge graph between said pair of videos, deriving a shared similarity score for the particular shared knowledge graph for any video of the pair of videos based on a minimum presence share in the knowledge graph; deriving a possible similarity score for the particular shared knowledge graph for any video of the pair of videos based on a maximum presence share in the shared knowledge graph; 5. The method of claim 4, further comprising:

6. generating a dissimilarity count based on the number of dissimilar knowledge graphs that represent only one of the videos of the pair of videos; calculating a dissimilarity score for the pair of videos based on the concept indexes of the dissimilar knowledge graphs for each video of the pair of videos; generating one or more clustering factors for the pair of videos based on the dissimilarity scores, the possible similarity scores, and the shared similarity scores; 6. The method of claim 5, further comprising:

7. The step of creating the plurality of video groups, For each pair of videos, comparing the clustering factor of the pair of videos to the specified level of similarity; including in the same group a first pair of videos whose clustering factor meets the specified level; removing a third video from the same group whose clustering factor for the first pair of videos does not meet the specified level; 7. The method of claim 6, comprising:

8. 1. A system comprising: A memory device; One or more processors configured to interact with the memory device and execute instructions, the instructions causing the one or more processors to perform operations, the operations comprising: Obtaining a video uploaded by a video publisher; generating a concept index for each given video, the concept index being generated based on at least (i) concepts conveyed by one or more objects shown in the video, and (ii) a level of salience of the concepts in the given video; creating a plurality of video groupings based on the conceptual index of the videos, where each given video group of the plurality of video groupings is created to include two or more different videos, each having a specified level of similarity with other videos of the given video group; generating insights about the plurality of video groups based on data obtained via a feedback loop; and modifying how at least one video is distributed over a network based on the insight about the given video group that includes the at least one video; a processor; A system comprising:

9. generating said concept index for each given video, For each given one of the videos, obtaining one or more knowledge graphs for a plurality of portions of the given video, each of the knowledge graphs representing one or more concepts conveyed by the given video; determining, for each given knowledge graph, a presence share indicative of the level of salience of the concept represented by the given knowledge graph, the concept index being generated based, at least in part, on a number of instances of the given knowledge graph in the given video and a total presence share of the given knowledge graph over the length of the video; The system of claim 8, comprising:

10. generating said concept index for each given video, For each given knowledge graph obtained for the given video, summing the presence shares of the given knowledge graph over the length of the given video; determining a number of portions of the given video that are described by the given knowledge graph; generating the conceptual index for the given video based on a ratio of the aggregated presence shares to the number of portions; The system of claim 9, comprising:

11. The instructions cause the one or more processors to perform operations, the operations comprising: For each given knowledge graph obtained for a number of videos uploaded by the video publisher, generating an inverse document frequency for the given knowledge graph based on a total number of the plurality of videos represented by the given knowledge graph; applying the generated inverse document frequency to a total presence share of the given knowledge graph for each of the plurality of videos; The system of claim 10 further comprising:

12. The instructions cause the one or more processors to: selecting a pair of videos from among the plurality of videos; generating a count of shared knowledge graphs between the pairs of videos, each shared knowledge graph being a given one of the knowledge graphs representing each video of the pair of videos; For each particular shared knowledge graph between said pair of videos, deriving a shared similarity score for the particular shared knowledge graph for any video of the pair of videos based on a minimum presence share in the knowledge graph; deriving a possible similarity score for the particular shared knowledge graph for any video of the pair of videos based on a maximum presence share in the shared knowledge graph; The system of claim 11 , further comprising:

13. The instructions cause the one or more processors to perform operations, the operations comprising: generating a dissimilarity count based on the number of dissimilar knowledge graphs that represent only one of the videos of the pair of videos; calculating a dissimilarity score for the pair of videos based on the concept indexes of the dissimilar knowledge graphs for each video of the pair of videos; generating one or more clustering factors for the pair of videos based on the dissimilarity scores, the possible similarity scores, and the shared similarity scores; The system of claim 12 further comprising:

14. creating said plurality of video groups, For each pair of videos, comparing the clustering factor of the pair of videos to the specified level of similarity; including in the same group a first pair of videos whose clustering factor meets the specified level; excluding a third video from the same group whose clustering factor for the first pair of videos does not meet the specified level; The system of claim 13, comprising:

15. One or more computer readable storage media storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations, the operations including: Obtaining a video uploaded by a video publisher; generating a concept index for each given video, the concept index being generated based on at least (i) concepts conveyed by one or more objects shown in the video, and (ii) a level of salience of the concepts in the given video; creating a plurality of video groupings based on the conceptual index of the videos, where each given video group of the plurality of video groupings is created to include two or more different videos, each having a specified level of similarity with other videos of the given video group; generating insights about the plurality of video groups based on data obtained via a feedback loop; and modifying how at least one video is distributed over a network based on the insight about the given video group that includes the at least one video; and 23. A computer-readable storage medium comprising:

16. generating said concept index for each given video, For each given one of the videos, obtaining one or more knowledge graphs for a plurality of portions of the given video, each of the knowledge graphs representing one or more concepts conveyed by the given video; determining, for each given knowledge graph, a presence share indicative of the level of salience of the concept represented by the given knowledge graph, the concept index being generated based, at least in part, on a number of instances of the given knowledge graph in the given video and a total presence share of the given knowledge graph over the length of the video; 16. The computer-readable storage medium of claim 15, comprising:

17. generating said concept index for each given video, For each given knowledge graph obtained for the given video, summing the presence shares of the given knowledge graph over the length of the given video; determining a number of portions of the given video that are described by the given knowledge graph; generating the conceptual index for the given video based on a ratio of the aggregated presence shares to the number of portions; 20. The computer-readable storage medium of claim 16, comprising:

18. The instructions cause the one or more processors to perform operations, the operations comprising: For each given knowledge graph obtained for a number of videos uploaded by the video publisher, generating an inverse document frequency for the given knowledge graph based on a total number of the plurality of videos represented by the given knowledge graph; applying the generated inverse document frequency to a total presence share of the given knowledge graph for each of the plurality of videos; 20. The computer-readable storage medium of claim 17, further comprising:

19. The instructions cause the one or more processors to perform operations, the operations comprising: selecting a pair of videos from among the plurality of videos; generating a count of shared knowledge graphs between the pairs of videos, each shared knowledge graph being a given one of the knowledge graphs representing each video of the pair of videos; For each particular shared knowledge graph between said pair of videos, deriving a shared similarity score for the particular shared knowledge graph for any video of the pair of videos based on a minimum presence share in the knowledge graph; deriving a possible similarity score for the particular shared knowledge graph for any video of the pair of videos based on a maximum presence share in the shared knowledge graph; 20. The computer-readable storage medium of claim 18, further comprising:

20. The instructions cause the one or more processors to perform operations, the operations comprising: generating a dissimilarity count based on the number of dissimilar knowledge graphs that represent only one of the videos of the pair of videos; calculating a dissimilarity score for the pair of videos based on the concept indexes of the dissimilar knowledge graphs for each video of the pair of videos; generating one or more clustering factors for the pair of videos based on the dissimilarity scores, the possible similarity scores, and the shared similarity scores; 20. The computer-readable storage medium of claim 19, further comprising:

Images