Video clustering and analysis

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for improving content analysis. One method includes obtaining videos uploaded by video publishers. A concept index is generated for each given video. The concept index is generated based on at least (i) a concept conveyed by one or more objects depicted in the video and (ii) a level of saliency of the concept in the given video. Based on the concept index of the videos, a plurality of video groups is created. Each given video group includes two or more different videos, each of which has a specified level of similarity to the other videos in the given video group. Based on data obtained through a feedback loop, insights are generated about the plurality of video groups. Based on the insights for a given video group that includes at least one video, distribution of the at least one video is modified.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Israeli application No. 287859, filed on November 5, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This manual covers video data processing and analysis. Background Technology

[0004] The volume of online videos is growing year by year, and now anyone with a computer can upload video content. Because similar concepts can be conveyed in different ways using video, it is difficult to identify groups of similar videos to be analyzed. Summary of the Invention

[0005] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: obtaining videos uploaded by video publishers by a service device; generating a concept index for each given video by the service device, wherein the concept index is generated based at least on (i) a concept conveyed by one or more objects depicted in the video and (ii) the salience level of said concept in the given video; creating multiple video groups by the service device based on the concept index of the videos, wherein each given video group of the multiple video groups is created to include two or more distinct videos, each of said two or more distinct videos having a specified similarity level with other videos in the given video group; generating insights about the multiple video groups by the service device based on data obtained through a feedback loop; and modifying the manner in which at least one video is distributed over a network by the service device based on the insights about a given video group including at least one video.

[0006] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs configured to perform the actions of the method and encoded on a computer storage device.

[0007] These and other embodiments may optionally include one or more of the following features. Generating a concept index for each given video may include: for each given video in the video: obtaining one or more knowledge graphs for multiple portions of the given video, wherein each of the knowledge graphs represents one or more concepts conveyed by the given video; and determining, for each given knowledge graph, an existence share indicating the salience level of the concepts represented by the given knowledge graph, wherein the concept index is generated based at least in part on the number of instances of the given knowledge graph in the given video and the total existence share of the given knowledge graph over the video length.

[0008] Generating a concept index for each given video may include: for each given knowledge graph obtained for the given video: summing the existence shares of the given knowledge graph over the length of the given video; determining the number of parts of the given video described by the given knowledge graph; and generating a concept index for the given video based on the ratio of the summed existence shares to the number of said parts.

[0009] The method may include: for each given knowledge graph obtained from multiple videos uploaded by a video publisher: generating an inverse document frequency metric for the given knowledge graph based on the total number of videos represented by the given knowledge graph; and for each of the multiple videos, applying the generated inverse document frequency to the total presence share of the given knowledge graph.

[0010] The method may include: selecting a pair of videos from the plurality of videos; generating a count of shared knowledge graphs between the pair of videos, wherein each shared knowledge graph is a given knowledge graph representing each video in the pair of videos; and for each specific shared knowledge graph between the pair of videos: deriving a shared similarity score for the specific shared knowledge graph based on the minimum presence share of the knowledge graph of any video in the pair of videos, and deriving a probable similarity score for the specific shared knowledge graph based on the maximum presence share of the knowledge graph of any video in the pair of videos.

[0011] The method may include: generating a dissimilarity count based on the number of dissimilarity knowledge graphs representing only one of the videos in the pair; calculating a dissimilarity score for the pair of videos based on the concept index of the dissimilarity knowledge graph for each video in the pair; and generating one or more clustering factors for the pair of videos based on the dissimilarity score, the possible similarity score, and the shared similarity score.

[0012] Creating multiple video groups may include: for each pair of videos, comparing the clustering factor of the pair of videos with a specified similarity level; including the first pair of videos in the same group whose clustering factor meets the specified level; and excluding a third video from the same group whose clustering factor does not meet the specified level relative to the first pair of videos.

[0013] Specific embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. Unlike conventional similarity techniques, the techniques discussed herein enable a computer to take substantially different videos, determine the conceptual similarity between these different videos, and cluster these different videos together to improve the computer's ability to evaluate video groups together. This results in a more efficient analysis system because instead of evaluating each video in a cluster (also called a group), representative videos of the cluster and / or subsets of aggregated information can be evaluated at runtime, enabling the computer to provide real-time responses to queries, which is not possible in cases where all videos need to be evaluated to provide a response to a query. Furthermore, when two videos may have very few (e.g., one or two) common features, but the similarity distance between these features is very small, the techniques discussed herein prevent errors that occur in similarity determination using conventional similarity techniques. In these cases, the weight of the similarity of a few common features may outweigh all the dissimilarity between videos, leading to incorrect determination of video similarity.

[0014] The techniques discussed in this paper also reduce the amount of time required for computers to learn distribution strategies for videos (or other content). Traditional computer systems require a considerable learning period to determine whether delivering content to client devices will result in efficient use of server-side resources (e.g., network bandwidth, server cores, etc.) and / or client-side resources (e.g., battery consumption, processor cycles, data usage, etc.) required for content presentation. This learning period can be a week or longer, and during this period, content delivery often results in wasted server-side resources (e.g., server core usage) and wasted client-side resources (e.g., battery consumption) as the computer system learns the difference between good and bad times for delivering content. However, using the techniques described in this paper, the learning period can be significantly reduced, even if not eliminated, thereby enabling computer systems to utilize both server-side and client-side resources more efficiently compared to traditional learning-period approaches. This improvement in resource utilization is achieved by initially determining the cluster to which a new video (or other content) belongs and initializing the delivery of the new video using aggregated information from videos within that cluster, without requiring a long learning phase.

[0015] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0016] Figure 1 This is a block diagram of an example online environment.

[0017] Figure 2 This is a flowchart illustrating concept extraction.

[0018] Figure 3 This is a diagram illustrating video similarity between a pair of videos and video clustering based on conceptual similarity.

[0019] Figure 4 This is a flowchart of an example process that uses video similarity to provide insights into video grouping and to modify how videos are distributed.

[0020] Figure 5 This is a block diagram of an example computer system.

[0021] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0022] This specification describes a technique for clustering videos based on the overall concept conveyed by different multimedia content (e.g., a combination of images, videos, and / or audio), despite differences between the various multimedia contents that would otherwise prevent them from being identified as similar. Using online videos as an example, two videos may differ significantly in resolution, colorization, speech, element order, and other aspects, yet they may still convey similar concepts, making them worthy of clustering for analysis of videos conveying the same / similar concepts. However, using conventional techniques, these two videos might not be clustered together for analytical purposes because, traditionally, computers cannot determine conceptual similarity in videos (or other multimedia content) when individual videos (or other multimedia content) differ in their presentation attributes and / or in how different videos (or other multimedia content) present the concepts they convey.

[0023] Unlike traditional similarity techniques, the techniques discussed in this paper enable computers to acquire substantially different videos, determine the conceptual similarity between these different videos, and cluster these different videos together to improve the computer's ability to evaluate video groupings together. As discussed throughout this specification, the evaluation of video groupings can be used to provide insights into concepts for which new videos should be developed, identify representative videos and / or aggregate information for each cluster (e.g., concept groups), which can be used to reduce subsequent processing of videos within a group and accelerate the optimization of content distribution rules to effectively utilize server-side and client-side computing resources.

[0024] As discussed in more detail, the techniques typically involve extracting conceptual information from videos (or other content) and clustering the videos based on conceptual similarity, regardless of how the conceptual information is conveyed (e.g., by different people, in different languages, or in different orders). Once the videos are clustered according to concepts, they can be analyzed on a per-concept basis to provide insights into the different concepts. Visual maps of the videos can also be created to visually convey information about closely related concepts and closely related video groups. The visual maps are de-cluttered by selecting representative videos for each video group and deduplicating videos that are substantially the same except for differences in resolution, color scaling, compression techniques, etc., thereby improving the usefulness of the map in a limited display area. Representative videos can be selected in various ways (e.g., similarity to other videos, feedback data, etc.). The visualization of the video map can be output to the user, and the map can include interactive features that allow the user to obtain information about each video group through interaction with the map. For example, a user can click on a representative image of a video group and be provided with information about the videos in that group. Similarly, users can interact with the map (e.g., zoom in / out) to delve deeper into video groups, which can reveal additional subgroups of the video or view information at a wider grouping level.

[0025] Throughout this specification, video is used as an example of multimedia content to which the techniques discussed can be applied, to provide more concrete examples and specific use cases. However, the techniques discussed herein can be applied to any form of content, so the use of video for illustrative purposes should not be considered limiting. In other words, throughout this specification, the term "video" can generally be replaced with the word "content."

[0026] Figure 1 This is a block diagram of an example online environment 100. Example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 102 connects to an electronic document server 104, user equipment 106, a digital component server 108, and a service device 110. Example environment 100 may include many different electronic document servers 104, user equipment 106, digital component servers 108, and service devices 110.

[0027] Client device 106 is an electronic device capable of requesting and receiving resources via network 102. Example client device 106 includes personal computers, mobile communication devices, digital assistive devices, and other devices capable of sending and receiving data via network 102. Client device 106 typically includes user applications (such as web browsers) to facilitate sending and receiving data via network 102; however, local applications executed by client device 106 can also facilitate sending and receiving data via network 102.

[0028] Digital assistive devices include those with microphones and speakers. They are typically capable of receiving input via voice and responding with content using auditory feedback, and can present other auditory information. In some cases, digital assistive devices also include or communicate with a visual display (e.g., via a wireless or wired connection). When a visual display is present, feedback or other information can also be provided visually. In some cases, digital assistive devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices registered with the digital assistive device.

[0029] An electronic document is data that presents a set of content at client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., "apps") such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents may be provided to client device 106 by electronic document server 104 ("Electronic Doc Server"). For example, electronic document server 104 may include a server hosting a publisher's website. In this example, client device 106 may initiate a request for a given publisher's web page, and electronic server 104 hosting the given publisher's web page may respond to the request by sending machine-executable instructions to render the given web page at client device 106.

[0030] In another example, the electronic document server 104 may include a video server from which the client device 106 can download videos (e.g., user-created videos or other videos). In this example, the client device 106 may download the files required to play the video in a web browser or local application configured to play the video.

[0031] Electronic documents can include a variety of content. For example, an electronic document may include static content (e.g., text or other specified content) that is inherent in the electronic document itself and / or does not change over time. Electronic documents may also include dynamic content that can change over time or based on each request. For example, the publisher of a given electronic document may maintain a data source for populating multiple sections of the electronic document. In this example, a given electronic document may include a script that causes the client device 106 to request content from the data source when the given electronic document is processed (e.g., rendered or executed). The client device 106 integrates the content obtained from the data source into the given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0032] In some cases, a given electronic document may include a digital component script referencing service device 110 or a specific service provided by service device 110. In these cases, when client device 106 processes the given electronic document, client device 106 executes the digital component script. Execution of the digital component script configures client device 106 to generate a request for digital component 112 (referred to as a "component request"), which is transmitted via network 102 to digital component distribution system 110. For example, the digital component script may enable client device 106 to generate a packet data request including header and payload data. Component request 112 may include event data specifying characteristics such as the name (or network location) of the server from which it requests the digital component, the name (or network location) of the requesting device (e.g., client device 106), and / or information that service device 110 may use to select one or more digital components, or other content provided in response to the request. Component request 112 is transmitted by client device 106 via network 102 (e.g., a telecommunications network) to the server of service device 110.

[0033] Component request 112 may include event data specifying other event characteristics, such as the characteristics of the requested electronic document and the location of the electronic document in which the digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., a webpage) in which the digital component will be presented, the available location of the electronic document in which the digital component can be presented, the size of the available location (e.g., a portion of a page or the duration within a video), and / or the type of media eligible to be presented at that location may be provided to the digital component distribution system 110. Similarly, event data specifying keywords (“document keywords”) associated with an electronic document or entities (e.g., people, places, or things) referenced by the electronic document may also be included in component request 112 (e.g., as payload data) and provided to service device 110 to identify digital components eligible to be presented with the electronic document. Event data may also include search queries submitted from client device 106 to obtain a search results page (e.g., a search results page presenting general search results or video search results).

[0034] Component request 112 may also include event data related to other information, such as information already provided by the user of the client device, geographic information indicating the state or region from which the component request was submitted, the language settings of the client device, or other information providing context about the environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the device type such as a mobile device or tablet device that will display the digital component). Component request 112 may be transmitted, for example, via a packet network, and component request 112 itself may be formatted as packet data with a header and payload data. The header may specify the destination of the packet, while the payload data may include any of the information described above.

[0035] In response to receiving component request 112 and / or using the information included in component request 112, service device 110 selects a digital component (e.g., video file, audio file, image, text, and combinations thereof, all of which may take the form of advertising or non-advertising content) to be presented along with a given electronic document. In some implementations, the digital component is selected in less than one second to avoid errors that may be caused by delayed selection of the digital component. For example, a delay in providing the digital component in response to component request 112 may cause a page loading error at client device 106, or cause parts of the electronic document to remain unfilled even after other parts of the electronic document have been presented at client device 106. Furthermore, as the delay in providing the digital component to client device 106 increases, it is likely that the electronic document will no longer be presented at client device 106 when the digital component is delivered, thus negatively impacting the user experience of the electronic document. Additionally, for example, if the electronic document is no longer presented at client device 106 when the digital component is provided, the delay in providing the digital component may cause the delivery of the digital component to fail.

[0036] In some implementations, service device 110 is implemented as a distributed computing system comprising, for example, a server and a collection 114 of multiple computing devices interconnected and responding to request 112 to identify and distribute digital components. The collection 114 of multiple computing devices operates together to identify a set of digital components eligible for presentation in an electronic document from a corpus of millions of available digital components (DC1-x). For example, millions of available digital components in a digital component database 116 may be indexed. Each digital component index entry may reference the corresponding digital component and / or include distribution parameters (DP1-DPx) that facilitate (e.g., trigger, regulate, or restrict) the distribution / transmission of the corresponding digital component. For example, distribution parameters may facilitate (e.g., trigger) the transmission of a digital component by requiring the component request to include at least one criterion that matches (e.g., an exact match or has a pre-specified level of similarity) one of the distribution parameters of the digital component.

[0037] The identification of eligible digital components can be segmented into multiple tasks 117a-117c, which are then distributed among computing devices within a set 114 of multiple computing devices. For example, different computing devices in set 114 can each analyze different portions of the digital component database 116 to identify various digital components having distribution parameters that match the information included in component request 112. In some implementations, each given computing device in set 114 can analyze different data dimensions (or sets of dimensions) and pass (e.g., transmit) the results of the analysis (Res 1-Res 3) 118a-118c back to the digital component distribution system 110. For example, the results 118a-118c provided by each computing device in set 114 can identify a subset of digital components eligible for distribution in response to a component request and / or a subset of digital components having specific distribution parameters. Identifying a subset of digital components may include, for example, comparing event data with distribution parameters and identifying a subset of digital components having distribution parameters that match at least some characteristics of the event data.

[0038] Service device 110 aggregates results 118a-118c received from a collection 114 of multiple computing devices and uses information associated with the aggregated results to select one or more digital components to be provided in response to request 112. For example, service device 110 may select a set of top-ranked digital components (one or more digital components) based on the results of one or more content evaluation processes. Furthermore, service device 110 may generate and transmit response data 120 (e.g., digital data representing a response) via network 102, enabling client device 106 to integrate the set of top-ranked digital components into a given electronic document, such that the set of top-ranked digital components and the content of the electronic document are displayed together on the display of client device 106.

[0039] In some implementations, client device 106 executes instructions included in response data 120 that configure and enable client device 106 to obtain the set of highest-ranked digital components from one or more digital component servers. For example, the instructions in response data 120 may include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes client device 106 to send a server request (SR) 121 to digital component server 108 to obtain the given highest-ranked digital component from digital component server 108. In response to this request, digital component server 108 identifies the given highest-ranked digital component specified in server request 121 (e.g., within a database storing multiple digital components) and sends digital component data (DC data) 122 to client device 106, which presents the given highest-ranked digital component in an electronic document at client device 106.

[0040] Service device 110 can utilize various techniques to evaluate the eligibility of different digital components that can be delivered in response to a given component request (e.g., a single component request). For example, service device 110 can compare the eligibility scores of various different digital components and select one or more digital components with the highest eligibility scores as the digital components to be delivered to client device 106 in response to a given component request. In some cases, the initial eligibility score can be determined based on one or more factors. For example, a provider (P1) of a video clip (VC1) can provide distribution criteria X for VC1, while different providers (P2) of different video clips (VC2) can provide different distribution criteria Y. For the purposes of this example, assume that the component request only requests one digital component to be presented with a specific webpage or a specific video. To select which of the two video clips to be delivered, service device 110 can rank them based on their respective eligibility scores, which can be determined based on a comparison of the distribution criteria provided by P1 and P2 with the information included in request 112. In some implementations, the set of distribution criteria most similar to the information in request 112 will have the highest eligibility score and therefore be ranked highest. Service device 110 can respond to component request 112 by selecting the highest-ranked video clip to be sent to the client device.

[0041] In some cases, the eligibility score is enhanced (or varied) based on one or more other factors. For example, service device 110 may generate an adjusted eligibility score for a digital component based on its initial eligibility score and its quality factor.

[0042] A quality factor for a given digital component can quantify the likelihood that a given digital component is the appropriate digital component to be provided in response to a given component request. In some implementations, the quality factor is determined based on one or more features specified by event data. More specifically, service device 110 may input one or more features from event data (e.g., geographic information and / or terms from an electronic document) into a machine learning system whose output can be used as a predicted distribution result for the quality factor. The predicted distribution result may, for example, be represented as a predicted interaction rate (e.g., click-through rate, playback completion rate, or another measure of interaction with the digital component) of the digital component in the context of the current component request.

[0043] Once the quality factor is obtained, it can be applied to the initial eligibility score to arrive at the adjusted eligibility score. For example, the adjusted eligibility score (AES) can be the product of the initial eligibility score and the quality factor (e.g., AES = quality factor). Initial qualification score). In some cases, adjusted qualification scores for various different digital components can be used to rank the digital components (e.g., from highest to lowest), and one or more highest-ranked digital components can be selected to be delivered in response to a component request.

[0044] The effectiveness of distributed content (e.g., video) can be evaluated based on user responses to content delivered to client devices, and this can also serve as an indicator of the rationality of server-side and / or client-side computing resource usage. While tracking user responses to individual parts of content (e.g., individual videos) can provide insights into which parts of the content result in efficient use of the computing resources required to deliver the content, such an evaluation does not provide insights into why some parts of the content are better suited for distribution (e.g., better utilizing limited server-side and / or client-side computing resources and / or eliciting more positive user responses). To provide this type of deeper insight into why some parts of the content are better suited for distribution, aggregate analysis of multiple similar parts of the content needs to be performed, which necessitates that said parts of the content be grouped together in a meaningful way.

[0045] One way to group content into sections is according to their conceptual similarity. As used throughout this document, conceptual similarity refers to the similarity between the concepts conveyed by two or more sections of content. As described in more detail below, the conceptual similarity between two sections of content can be determined based on the level of matching between the concepts conveyed by those two sections, and can be determined independently of how those concepts are conveyed.

[0046] For example, suppose Video 1 (V1) is a 30-second animated video in which 25 seconds depict a cat chasing a mouse, while the remaining 5 seconds present the text “Call 555-555-5555 to call pest control agent X.” Also suppose Video 2 (V2) is a 30-second live video that does not include any presented text but depicts a cat chasing a mouse, with the mouse jumping to safety. In this example, based on the fact that most of each video depicts a cat chasing a mouse, the two videos can be considered very similar to the concept of a cat chasing a mouse, despite significant differences in how the concept is conveyed (e.g., animation vs. live). If these two videos are grouped with other videos pointing to the concept of a cat chasing a mouse, aggregated information (e.g., user response information) can be used to effectively evaluate the effectiveness of videos pointing to the concept of a cat chasing a mouse, regardless of differences in how those concepts are conveyed.

[0047] As discussed in detail below, the effectiveness of videos targeting each concept can be mapped on a per-user-group basis, allowing insights into concepts that should be used in videos (or other content) to be generated based on the target audience and the effectiveness of different concepts when presented to that audience. Furthermore, as described below, grouping and evaluating videos in this way enables computer systems to learn optimal distribution parameters for new videos more quickly, based on insights determined from a pre-existing set of videos belonging to the same cluster (or video group) as the new video. This ability to learn optimal distribution parameters more quickly reduces the waste of resources that would result from using traditional learning periods (which could take days or weeks) and also reduces the computational resources required for the computer to learn the distribution parameters.

[0048] Service unit 110 includes a concept evaluation unit (“CEA”) 150, which includes one or more data processors. CEA 150 is configured to cluster videos (and other content) based on concept similarity and evaluate the resulting video groups (or other content groups) to determine the effectiveness of different video concepts. This evaluation can be used to present insights into the use of certain concepts based on a target audience (e.g., a group of users to whom content will be directed). As described below, these insights can be visually communicated in tabular, chart, or other formats. Furthermore, these insights can be used to recommend concepts for new videos, provide conceptual feedback on groupings of existing videos, and initialize distribution rules for new videos based on the concepts these new videos refer to.

[0049] Figure 2 This is a block diagram 200 illustrating concept extraction from video. Block diagram 200 includes a video player 202 in which the video is presented. The video player 202 is shown as part of a webpage, but it could be implemented as a native application (e.g., on a mobile or tablet device) or in another suitable device (e.g., a streaming video player plugged into or embedded in a TV). Furthermore, the concept extraction discussed below can be performed independently of the video player 202; therefore, the video player 202 is included only to aid in the description of concept extraction. Additionally, concept extraction is discussed as being performed by the service device 110, but it could be performed on a part of the service device (e.g., CEA 150) or on another suitable device.

[0050] The video available for playback by video player 202 has a playback duration, also referred to as the video length. For example, playback timer 204 in video player 202 indicates that the playback duration of the currently loaded video is one minute. To evaluate the context of the video, service device 110 (e.g., via CEA 150) can determine the video context across the video length. For example, service device 110 can select different timestamps across the video length to extract the video context. In this example, the video is segmented into four distinct segments (“quartiles”) by selecting timestamps at 0, 15, 30, 45, and 60 seconds, identified by labels 206, 208, 210, 212, and 214, respectively. This quartile segmentation is used to illustrate the concept of timestamp selection while preventing confusion in the accompanying figures. However, in practice, service device 110 can select timestamps finely or coarsely as needed. In some implementations, timestamps are selected at one second of the video duration, and in some implementations, service device 110 can select sub-second timestamps (e.g., 1 / 6 of a second).

[0051] Each of the selected timestamps is an indication of a portion of the video that will be evaluated to determine the concept being conveyed in the video at that point. In some implementations, the service device 110 examines the video frame presented at each timestamp to determine the context of the video at that timestamp. Returning to the previous example, the service device 110 may evaluate the video at the beginning (e.g., 0 seconds), at 15 seconds, at 30 seconds, at 45 seconds, and at the end (e.g., 1:00).

[0052] The evaluation of the video at each timestamp may include identifying any objects presented in the video, the attributes of those objects, and / or the relationships between the objects presented in the video. For example, the video presented in video player 202 includes person 216, chair 218, table 220, and two other people. In this example, service device 110 may use object detection techniques known in the art to identify these objects, their relative positions, and attributes such as color, pattern, or other visual attributes.

[0053] Once the service device 110 obtains information about the objects depicted in the video at a given timestamp, it can generate a knowledge graph representing the concepts conveyed by the video at that given timestamp. As used in this document, the knowledge graph is a representation of relationships between unique entities, and it can be stored in one or more data stores. Each node can represent a distinct unique entity, and nodes can be connected by graph edges (e.g., logical links) representing relationships between entities. The knowledge graph can be implemented, for example, graphically or as a data structure that includes data representing each node and data representing relationships between each node.

[0054] Returning to the example above, the knowledge graph depicting the video in video player 202 could include individual nodes for each of person 216, chair 218, table 220, and two other people. Each of these nodes could be connected by graph edges representing relationships between entities. For example, the node representing chair 218 could be linked to the node representing the person sitting in the chair by an edge labeled "sitting," indicating the fact that the video depicts the person sitting in chair 218. Similarly, the node representing table 220 could be linked to the node representing the person sitting in the chair by an edge labeled "hands on," indicating the fact that the person sitting in chair 218 has their hands on table 220. Other attributes of the video could be similarly represented by the knowledge graph, such as color, text, or other attributes. Of course, the edges between nodes do not need to include any labels; they can simply indicate some kind of relationship between the nodes.

[0055] For a given video, service device 110 can create multiple knowledge graphs. For example, service device 110 can create a different knowledge graph for each timestamp selected for a given video. In the example above, service device 110 will generate five different knowledge graphs 222, 224, 226, 228, and 230. Service device 110 stores each knowledge graph in knowledge graph database 232 for further processing. For example, as described below, service device 110 can use the knowledge graphs to evaluate conceptual similarity between videos and cluster the videos based on conceptual similarity.

[0056] Figure 3 This is a block diagram 300 illustrating video similarity between a pair of videos and video clustering based on conceptual similarity. Block diagram 300 shows that the service device 110 receives a set of two knowledge graphs for processing as input. More specifically, the service device 110 is receiving a video 1 knowledge graph (V1KG) 302 and a video 2 knowledge graph (V2KG) 304. V1KG 302 includes multiple knowledge graphs created based on objects presented at different selected timestamps within video 1, and V2KG includes multiple knowledge graphs created based on objects presented at different selected timestamps within video 2. Knowledge graph 306 is a visualization of an example knowledge graph that can be included in V1KG 302, and knowledge graph 308 is a visualization of an example knowledge graph that can be included in V2KG 304. Each of knowledge graphs 306 and 308 can represent a portion of a video, which is presented with Figure 2 The video player 202 presents content similar to the video frames.

[0057] For example, knowledge graph 306 includes a node 310 representing a person, a node 312 representing a chair, and a node 314 representing a table. These nodes 310, 312, and 314 can represent, for example, a chair 218, a table 220, and a person sitting in chair 218, such as... Figure 2 As shown. Similarly, knowledge graph 308 includes nodes 316, 318, and 320, which represent a person, a chair, and a table, respectively. As discussed further below, the person, chair, and table represented by nodes 316, 318, and 320 can be related to... Figure 2 The different people, tables, and chairs depicted in the video player 202, or they may represent the same people, tables, and chairs.

[0058] Knowledge graph 306 also includes node 322 representing the famous actor "Celebrity X," and nodes 324 and 326 representing brown and green, respectively. The knowledge graph also includes edges connecting the nodes and specifies the types of relationships between them. For example, knowledge graph 306 includes edge 328 between nodes 310 and 312, indicating a relationship between a person and a chair. In this example, edge 328 has a label indicating a "sitting" relationship with an arrow pointing in the direction of node 312 representing the chair. Thus, the edge indicates that the person and the chair are related by the fact that the person is sitting in the chair, as... Figure 2 As shown. Similarly, knowledge graph 206 includes linked nodes 310 and 314 and has an edge 330 labeled "hands placed on". Therefore, edge 330 represents the fact that people place their hands on the table, as... Figure 2 As shown. Knowledge graph 306 also includes edge 332 between nodes 301 and 322, which indicates that the person represented by node 310 is celebrity X, who is an actor. Edge 334 is between node 312 and node 326, which indicates that the chair is green, and edge 336 is between node 314 and node 324, which indicates that the table is brown.

[0059] Like knowledge graph 306, knowledge graph 308 also includes node 338 representing brown. However, knowledge graph 308 does not include nodes representing celebrity X or green. Knowledge graph 308 also includes edges that link the nodes of knowledge graph 308 and represent the relationships between the nodes. For example, edge 340 linking nodes 316 and 318 represents the fact that a person is sitting in a chair, while edge 342 linking nodes 316 and 320 indicates that the person has their feet on the table. Furthermore, edges 344 and 346 indicate that both the table and the chair are brown.

[0060] Service device 110 processes sets of knowledge graphs (e.g., V1KG 302 and V2KG 304) to determine conceptual similarity between videos, and clusters the videos, for example, based on the similarity of the knowledge graphs in each set.

[0061] As part of the processing, service device 110 generates concept index 1 (348) for video 1 and concept index 2 (350) for video 2. The concept index of each video represents the importance of each concept depicted by each video. The concept index of a given video can be generated based on the concepts conveyed by the objects depicted in the given video and the salience level of each concept over the video length. In some implementations, the concept index of a given video can be generated, for example, using a set of knowledge graphs obtained for that given video. For example, see reference... Figure 3 Service device 110 can use the set 304 of the knowledge graph of video 2 to generate concept index 2 (350).

[0062] More specifically, service device 110 can examine all knowledge graphs in set 304 to identify the total presence share of each knowledge graph over the length of video 2. Since each knowledge graph represents a concept conveyed by a given video, the total presence share of a particular knowledge graph can represent the total presence share of the corresponding concept conveyed over the video length. In some implementations, the total presence share of a particular knowledge graph over the video length can be determined by aggregating the presence shares of each knowledge graph at each timestamp of the video being evaluated.

[0063] For example, suppose five timestamps are selected, as shown in the reference. Figure 2 The total presence share of a particular knowledge graph discussed in this example can be the sum of the presence shares of that particular knowledge graph at each of the five timestamps (e.g., total presence share = Σts5 ts1 presence share, where ts represents a timestamp). The output obtained by summing the presence shares over the video length (e.g., at each selected timestamp) will typically be a numeric value that can be combined with other information to create a concept index, as discussed further below. References Figure 3 Service device 110 can determine the total existence share of knowledge graph 308 by summing the existence shares of knowledge graph 308 for each timestamp evaluated in video 2.

[0064] The presence share of a particular knowledge graph can be based on several factors, such as the portion of the frame occupied by the object represented by the knowledge graph, the position of the object in the frame, or other factors corresponding to the salience of the object in the frame. Generally, the presence share of the knowledge graph representing an object will increase as the object's salience in the frame increases. For example, when an object is large or occupies a large portion of the frame, the presence share of the knowledge graph representing that object will increase. The presence share assigned to each knowledge graph at each timestamp can be a number between 0 (least salience) and 1 (most salience), although other appropriate scales (e.g., 0-10 or 0-100) can also be used.

[0065] As part of concept index generation, service device 110 can also determine the total number of timestamps representing each concept detected at a given video location. In some implementations, service device 110 can determine whether a specific knowledge graph has been collected for that timestamp by searching the knowledge graph obtained at each timestamp, and incrementing a counter for each instance of the specific knowledge graph detected at the timestamp, thereby determining the total number of timestamps of the video where the specific knowledge graph was detected. For example, if five timestamps are selected, as in reference... Figure 2 The service device discussed here can search the knowledge graph collected at each timestamp to determine whether any of the collected knowledge graphs matches a specific knowledge graph, and increment a counter for each timestamp of a knowledge graph that matches a specific knowledge graph. The value of the counter after all timestamps of a given video have been searched represents the total number of timestamps of the given video described by the specific knowledge graph (also known as the number of parts).

[0066] For a given video, service device 110 can combine the total presence share of a specific knowledge graph over the length of the given video with the total number of timestamps of the given video where the specific knowledge graph was detected to derive a concept value corresponding to the salience level of the concept represented by the specific knowledge graph over the length of the given video. In some implementations, the concept value can be calculated by taking the ratio of the summed presence share of the specific knowledge graph representing the concept to the total number of timestamps of the given video where the specific knowledge graph was detected. The generalized ratio can be expressed in the form of relation (1).

[0067] (1)

[0068] in:

[0069] The concept value KGi is the concept value of the concept represented by the i-th knowledge graph;

[0070] The summed existence share KGi is the summed existence share of the i-th knowledge graph on a given video;

[0071] The total instances KGi is the total number of timestamps (or portions thereof) of a given video where the i-th knowledge graph is detected.

[0072] In some implementations, each concept value generated for a given video can be considered a concept index. In some implementations, the concept index for a given video includes concept values ​​for each of several distinct concepts conveyed by the video and represented by different knowledge graphs. For example, when the concept index of a video includes several distinct concept values ​​for different concepts, each value in the concept index may correspond to a different concept and a knowledge graph representing that concept.

[0073] Service device 110 generates a knowledge graph inverse document frequency (KGIDF) metric (352) based on the occurrence of different knowledge graphs across all (or subsets) of the videos to be clustered. KGIDF can be used to adjust the inverse document frequency of individual knowledge graphs in each video. For example, relation (2) can be used to generate KGIDF for all videos.

[0074] (2)

[0075] Where |{v∈Number of Videos: kGi∈v}| is the number of videos where KGi is the i-th knowledge graph. Service device 110 applies KGIDF to the set of knowledge graphs for video 1 (302, 354) and also to the set of knowledge graphs for video 2 (304, 356). Applying KGIDF to the set of knowledge graphs yields an adjusted inverse document frequency for each knowledge graph. The adjusted KGIDF prevents errors that occur when two videos have very few shared knowledge graphs. For example, suppose two videos share only one common attribute (e.g., purple), but the common attribute is very similar (e.g., with a small similarity distance). In this case, some similarity techniques will yield very high similarity, even though the videos may actually be very dissimilar. In other words, using some similarity techniques for the similarity of a common attribute (e.g., similarity = 1 / sqrt(distance)) may cause the similarity of one attribute to exceed its dissimilarity, leading to an incorrect similarity determination.

[0076] In some implementations, KGIDF is applied to a set of knowledge graphs based on relation (3).

[0077] (3)

[0078] in:

[0079] It is the adjustment value of the i-th knowledge graph in the j-th video;

[0080] It is the summation of the existence share of the i-th knowledge graph over the length of the j-th video; and

[0081] It is the KGIDF of the i-th knowledge graph.

[0082] For example, service device 110 can generate an adjusted value for knowledge graph 308, which can also be referred to as the adjusted existence share, by multiplying the total existence share of knowledge graph 308 in video 2 by the KGIDF of knowledge graph 308 across all videos being analyzed by service device 110.

[0083] Service device 110 derives (e.g., determines or calculates) a similarity score (358) of the shared knowledge graph between a pair of videos. (See reference) Figure 3 The service device 110 can use the set of knowledge graphs 302 and 304 to determine the similarity score of the video pair including video 1 and video 2. For example, the service device 110 can derive a shared similarity score and a probable similarity score for the video pair. In some implementations, a separate shared similarity score is derived for each knowledge graph detected in both videos of the pair (e.g., video 1 and video 2). Similarly, the service device can derive a separate probable similarity score for each knowledge graph detected in the two videos of the pair.

[0084] A shared similarity score refers to a similarity score based on the minimum amount of conceptual salience in the pair of videos. For example, a shared similarity score could represent the minimum total presence share of a particular knowledge graph in either video of the pair. In other words, the service device 110 can determine which video in the pair (e.g., video 1 or video 2) has a lower adjustment value for a particular knowledge graph, for example using the adjustment value determined by relation (3), and select the lower adjustment value of that particular knowledge graph as the shared similarity score of the pair of videos. This determination can be performed, for example, using relation (4).

[0085]

[0086] in:

[0087] It represents the shared similarity of the i-th knowledge graph between video 1 and video 2;

[0088] It is the adjustment value of the i-th knowledge graph in video 1 (see relation 3); and

[0089] It is the adjustment value of the i-th knowledge graph in video 2 (see relation 3).

[0090] A potential similarity score refers to the maximum number of similarity scores based on the conceptual salience in the pair of videos. For example, a potential similarity score could represent the maximum total presence share of a particular knowledge graph in either video of the pair. In other words, the service device 110 can determine which video in the pair (e.g., video 1 or video 2) has a higher adjustment value for a particular knowledge graph, for example using the adjustment value determined by relation (3), and select the higher adjustment value of that particular knowledge graph as the potential similarity score for the pair of videos. This determination can be performed, for example, using relation (5).

[0091]

[0092] in:

[0093] It represents the possible similarity of the i-th knowledge graph between video 1 and video 2;

[0094] It is the adjustment value of the i-th knowledge graph in video 1 (see relation 3); and

[0095] It is the adjustment value of the i-th knowledge graph in video 2 (see relation 3).

[0096] Service device 110 obtains (e.g., determines or calculates) a dissimilarity score (358) for the dissimilarity knowledge graph in each video of the pair of videos. (See reference) Figure 3 The service device 110 can use sets 302 and 304 of knowledge graphs to determine the dissimilarity score of the video pair including video 1 and video 2. In some implementations, the service device 110 can generate the dissimilarity score of the video pair by summing the concept values ​​of the dissimilarity knowledge graphs in each video of the pair. For example, the service device 110 can identify the knowledge graph of video 1 that is not included in video 2, and the knowledge graph of video 2 that is not included in video 1. In this example, the service device can sum the concept indices of the dissimilarity knowledge graphs of video 1 and video 2, and output the sum as the dissimilarity score of the video pair including video 1 and video 2.

[0097] Another measure of dissimilarity that the service device 110 can determine is the total number of dissimilarity knowledge graphs in each video of the pair, referred to as the dissimilarity count. This can be determined by incrementing the counter whenever a knowledge graph identified by the service device 110 for one video in the pair does not find a match in the other video in the pair.

[0098] The service device clusters the videos based on similarity and dissimilarity scores (362). This clustering creates multiple video groups, each containing videos with at least a specified similarity level. For example, a group created by clustering would include multiple videos, each having a specified similarity level with one or more other videos in that group. (See reference) Figure 4 This section discusses in more detail how to use similarity and dissimilarity measures to create groupings.

[0099] Figure 4 This is a flowchart of an example process 400 that uses video similarity to provide insights into video grouping and modify the way videos are distributed. The operation of process 400 can be, for example, by... Figure 1The operation of process 400 may also be implemented using instructions encoded on one or more computer-readable media, which may be non-transitory. When executed by one or more data processing devices (e.g., one or more computing devices), the instructions cause the one or more data processing devices to perform the operation of process 400.

[0100] The video is obtained by the service device (402). In some implementations, the obtained video is acquired when a video publisher uploads the video. The video publisher can be a user who uploads the video to a video-sharing website, or an entity that creates the video to be presented along with other content. For example, see references... Figure 1 The video being discussed can be in the form of a digital component provided for distribution along with other content.

[0101] One or more concept indices (404) are generated for each given video obtained. The one or more concept indices may include separate concept indices generated for each given knowledge graph obtained for the given video. In some implementations, the concept index for a given video is generated based at least on (i) the concept conveyed by one or more objects depicted in the given video and (ii) the salience level of that concept in the given video. (See above reference...) Figure 3 The concepts conveyed by one or more objects depicted in a given video can be represented by one or more knowledge graphs obtained from multiple portions of the given video (e.g., at multiple different timestamps). These knowledge graphs can be obtained, for example, through evaluation of the video and / or from a database storing previously generated knowledge graphs, as referenced. Figure 2 The subject of discussion.

[0102] In some implementations, concept indexes can be generated by evaluating the presence of knowledge graphs in a video. For example, a specific concept index can be generated for concepts represented by a particular knowledge graph based on the number of times a knowledge graph is detected in the video and the total presence share of the knowledge graph over the video length, as shown in the reference. Figure 3 This will be discussed in detail. More specifically, the concept index of a particular knowledge graph can be determined as (i) the total presence share of the particular knowledge graph over the length of the video and (ii) the ratio of the total number of instances of the particular knowledge graph in a given video.

[0103] As described above, the presence share at a given timestamp within a video indicates the salience level of one or more concepts represented by a given knowledge graph at that timestamp in the video, and it also indicates the salience level of one or more concepts conveyed by the video. In some implementations, the total presence share of a given knowledge graph (and its corresponding concepts) is determined by summing the presence shares of the given knowledge graph over the video length. Meanwhile, the number of portions of the video described by a given knowledge graph can be determined based on the number of timestamps (or portions) of the video where a particular knowledge graph is detected.

[0104] For each given knowledge graph, generate the inverse document frequency and apply (406). For example, the inverse document frequency of a given knowledge graph is generated by determining the frequency of the given knowledge graph among all knowledge graphs obtained from the evaluated videos. In some implementations, the inverse document frequency of a given knowledge graph can be generated based on the total number of multiple videos represented by the given knowledge graph. In other words, the inverse document frequency of a given knowledge graph can be based on how many videos have a set of knowledge graphs that include the given knowledge graph. In some implementations, the inverse document frequency of a given knowledge graph can be determined using relation (2), as referenced. Figure 3 The subject of discussion.

[0105] For example, the inverse document frequency of a given knowledge graph can be applied by multiplying the inverse document frequency by the total presence share of the given knowledge graph in a specific video. For example, refer to... Figure 3 The relationship discussed (3) can be used to apply inverse document frequencies. In some implementations, separate inverse document frequencies are generated for each distinct knowledge graph, and the total presence share of each given knowledge graph in each video can be adjusted using the inverse document frequencies generated for a given knowledge graph. In other words, for each of multiple videos with timestamps represented by a given knowledge graph, the generated inverse document frequencies are applied to the total presence share of the given knowledge graph.

[0106] Select a pair of videos from the multiple obtained videos (408). In some implementations, the pair of videos can be selected randomly (or pseudo-randomly).

[0107] A similarity metric is derived for the pair of videos (410). In some implementations, deriving the similarity metric includes: deriving a shared similarity score for the shared knowledge graph between the pair of videos; and deriving a probable similarity score for the shared knowledge graph. A similarity score can be generated for each specific shared knowledge graph between the pair of videos. In some implementations, the shared knowledge graph is the knowledge graph found in each video in the pair. For example, a specific knowledge graph identified for video 1 could be considered a shared knowledge graph if a matching knowledge graph is also identified for video 2. Note that a matching knowledge graph can be an exact match, but is not required to be an exact match. In other words, a matching knowledge graph can be a knowledge graph with one or more identical nodes representing one or more objects. The matching level required to consider two knowledge graphs as a match can vary depending on the application.

[0108] The shared similarity score can be derived based on the minimum presence share of the shared knowledge graph in either video of the pair. For example, as referenced above. Figure 3 The shared similarity score of a particular shared knowledge graph discussed can be the minimum adjusted share of existence (e.g., adjustment value) of the particular shared knowledge graph in any of the videos in the pair of videos, as shown in relation (4).

[0109] A potential similarity score can be derived based on the maximum presence share of the shared knowledge graph in either video within the pair. For example, as referenced above... Figure 3 The possible similarity score of a particular shared knowledge graph discussed can be the largest share of the existence of the particular shared knowledge graph in either video of the pair of videos (e.g., the adjustment value), as shown in relation (5).

[0110] Deriving a similarity measure may also include generating a count of shared knowledge graphs between the video pairs. Each shared knowledge graph is a given knowledge graph in the knowledge graph representing each video in the pair. As mentioned above, shared knowledge graphs can be identified by finding a matching knowledge graph in one of the videos that has already been identified for the given knowledge graph in the other video in the pair. The count of shared knowledge graphs can be generated, for example, by incrementing a counter (or otherwise counting) whenever a given knowledge graph in one video in the pair is considered to match a knowledge graph in the other video. In some cases, the counter may be incremented only once for all matches of a given knowledge graph. In other cases, the counter may be incremented for each instance of a match between a given knowledge graph in one video and a matching knowledge graph in the other video.

[0111] One or more dissimilarity measures are derived for the pair of videos (412). In some implementations, deriving a dissimilarity measure includes generating a dissimilarity count and calculating a dissimilarity score. For example, a dissimilarity measure can be derived using a dissimilarity knowledge graph from each video in the pair. For example, while the similarity measure discussed above is generated using a matching knowledge graph from the pair of videos, the dissimilarity measure is generated using those knowledge graphs from each video in the other video in the pair for which no matching knowledge graph was found. It should be understood that regardless of the conditions used to determine a match, there will be a set of matching knowledge graphs used to derive the similarity measure and a set of dissimilarity knowledge graphs (or non-matching knowledge graphs) used to determine the dissimilarity measure.

[0112] The dissimilarity count can be generated based on the number of dissimilarity knowledge graphs identified for the pair of videos. The number of dissimilarity knowledge graphs can be, for example, the total number of knowledge graphs for both videos for which no matching knowledge graph was identified in the other video of the pair. The number of dissimilarity knowledge graphs can be generated, for example, by incrementing a counter whenever a match for a knowledge graph in one video is not found in the other video, and the number of dissimilarity knowledge graphs can be the value of the counter after all knowledge graphs for both videos have been analyzed to find a match. In other words, the dissimilarity count can simply represent the number of dissimilarity knowledge graphs for only one video in the pair.

[0113] The dissimilarity score of the video pair can be calculated based on the concept indexes of the dissimilarity knowledge graph for each video in the pair. For example, a concept index can be identified for each dissimilarity knowledge graph for each video in the pair. These concept indices for each of the dissimilarity knowledge graphs can then be summed to obtain the dissimilarity score for the two videos in the pair.

[0114] Multiple video groups are created based on similarity and dissimilarity measures (414). In some implementations, each given video group will be created to include only those videos that have at least a specified level of similarity to the other videos in that given group. For example, to be included in a given group, a given video may be required to satisfy a set of similarity conditions relative to each other video in that group. This set of similarity conditions may be related to the similarity and dissimilarity measures discussed above and may be used to ensure that all videos are conceptually related to similar concepts.

[0115] Grouping can be created based on, for example, a concept index of a video (directly or indirectly). For instance, dissimilarity measures are derived directly from the concept index of a dissimilarity knowledge graph, while similarity measures are derived from existence shares, which are also used to determine the concept index.

[0116] In some implementations, grouping can depend on one or more of the following factors:

[0117] 1) The number of shared knowledge graphs, denoted by n_similar, refers to the number of matching knowledge graphs between a pair of graphs.

[0118] 2) The number of dissimilar knowledge graphs, denoted by m_dissimilar, refers to the number of knowledge graphs in each video of the pair that do not have a matching knowledge graph in the other video of the pair.

[0119] 3) The sum of possible similarities, denoted by total_possible_similarity, refers to the mathematical sum of the possible similarity measures of all shared knowledge graphs of the pair of videos.

[0120] 4) The sum of shared similarities, denoted by shared_similarity_mass, refers to the mathematical sum of the shared similarity measures of all shared knowledge graphs of the pair.

[0121] 5) The sum of dissimilarity scores, denoted by dissimilarity_mass, is the sum of the dissimilarity scores of all dissimilarity knowledge graphs for the pair of videos.

[0122] 6) Dissimilarity share, denoted by dissimilarity_share, where dissimilarity_share = dissimilarity_mass / shared_similarity_mass.

[0123] 7) Total amount, represented by total_mass, where total_mass = dissimilarity_mass + total_possible_similarity.

[0124] 8) The share of dissimilarity is represented by share_of_dissimilarity, where share_of_dissimilarity = dissimilarity_share / total_mass.

[0125] 9) The share of realized similarity, denoted by share_of_realized_similiarity, where share_of_realized_similiarity = shared_similarity_mass / total_mass.

[0126] 10) Share of possible similarity, denoted by share_of_possible_similarity, where share_of_possible_similarity = total_possible_similarity / total_mass.

[0127] 11) Comparison score, denoted by comparison_score, where comparison_score = shared_similarity_mass / dissimilarity_mass.

[0128] 12) Creative approach similarity score, represented by creative_approach_similarity_score, where creative_approach_similarity_score = total_possible_similarity / dissimilarity_mass.

[0129] In some implementations, one or more of these factors are used to describe the relationship (e.g., similarity) between each pair of videos being evaluated, such that each pair of videos in the resulting dataset is characterized by these factors. The factors used to describe the relationship between video pairs can be fed into a clustering algorithm, which groups the videos together based on these factors.

[0130] In some implementations, video groups can be visually presented in a graphical user interface. To reduce clutter in the graphical user interface and improve processing time, video groups can be deduplicated to remove identical videos from the representation. The deduplication process can identify, for example, those video pairs within a given group that have a shared similarity exceeding a threshold and / or a dissimilarity level less than a specified level (which would be an indication that the videos are substantially identical).

[0131] Generate insights about multiple video groups (416). In some implementations, insights for each given video group are determined based on data obtained through feedback loops. For example, user reactions to the presented videos can be obtained through feedback loops and recorded with reference to the presented videos and / or the groups containing the videos. Reactions can be aggregated on a group-by-group basis, which can be used to provide insights about the video groups, the concepts conveyed by the video groups, and other properties of the video groups.

[0132] Feedback loops can be implemented, for example, as a script that is triggered when a user interacts with the video (e.g., clicks on the video), as pings that are automatically generated during video playback to report viewing time, or using other mechanisms. Using these feedback loops, users do not need to report their reactions to the video presented to them individually. Instead, reactions (e.g., positive or negative) can be inferred based on data collected using various feedback loops.

[0133] Insights generated from aggregated data from multiple distinct video groups can include, for example, the identification of video groups that received higher levels of positive feedback data compared to other video groups that received lower levels of positive feedback data. Once groups with higher levels of positive feedback data are identified, video publishers can be provided with information about the types of videos receiving higher levels of positive feedback data, allowing them to incorporate similar features into new videos. Similarly, information about video groups that received lower levels of positive feedback data can be provided to video publishers, allowing them to omit similar features from new videos.

[0134] In some implementations, insights can be created on a per-audience-type basis. For example, feedback data can be segmented based on any type of audience characteristic (e.g., device type, interest group, time of day, geographic region, etc.), and insights can be generated for each audience type or audience segment. When data is segmented in this way, insights can be provided to video publishers (e.g., as described above) based on the audience types for which they create their videos.

[0135] The generated insights can be extended beyond those used by video publishers to create videos that will receive a higher level of positive feedback. In some implementations, these insights can be used to significantly reduce the amount of time required to train a computer to efficiently and effectively distribute newly uploaded videos. For example, when a new video is uploaded, it can be evaluated and clustered, as discussed throughout this document. Once a new video is assigned to a cluster, insights into the videos in that cluster can be used to generate initial distribution criteria for the new video based on its known similarity to other videos in that cluster. By generating initial distribution criteria in this way and distributing new videos using initial distribution criteria generated based on insights from other videos in the cluster, the training period required by existing systems (which can exceed a week) is significantly reduced or eliminated, allowing computer systems to adjust the distribution criteria for new videos more quickly to achieve an optimal set of distribution criteria.

[0136] Modifying the distribution of at least one video based on insights (418). In some implementations, modifications to the distribution of at least one video may include adjusting the distribution criteria for the video. For example, suppose the initial intended audience for a given video in a cluster is football fans in Atlanta, Georgia, but the video group containing the given video has low feedback data for that group but high feedback data for hockey fans in Atlanta, Georgia. In this example, the distribution criteria for the given video could be adjusted to increase the likelihood of presenting the given video to the hockey fan audience in Atlanta, Georgia. In another example, the timing of video distribution or the content of videos distributed with each group could be adjusted based on insights generated for each group. Furthermore, attributes of the video itself could be modified to increase the likelihood of receiving more positive feedback data.

[0137] Figure 5 This is a block diagram of an example computer system 500 that can be used to perform the operations described above. System 500 includes a processor 510, memory 520, storage device 530, and input / output device 540. Each of components 510, 520, 530, and 540 may be interconnected, for example, using a system bus 550. Processor 510 is capable of processing instructions that execute within system 500. In one implementation, processor 510 is a single-threaded processor. In another implementation, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 or on storage device 530.

[0138] Memory 520 stores information within system 500. In one implementation, memory 520 is a computer-readable medium. In one implementation, memory 520 is a volatile memory cell. In another implementation, memory 520 is a non-volatile memory cell.

[0139] Storage device 530 provides high-capacity storage for system 500. In one implementation, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other high-capacity storage device.

[0140] Input / output device 540 provides input / output operations for system 400. In one implementation, input / output device 540 may include one or more network interface devices, such as Ethernet cards, serial communication devices (e.g., RS-232 ports), and / or wireless interface devices, such as 802.11 cards. In another implementation, input / output device may include driver devices configured to receive input data and send output data to peripheral device 560, such as keyboards, printers, and display devices. However, other implementations may also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0141] Although already Figure 5 An example processing system is described herein, but the subject matter and functional operations described herein can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their equivalents, or in a combination of one or more of them.

[0142] This document relates to service device 110. As used herein, service device 110 is one or more data processing devices that perform operations to distribute content over a network. Service device 110 is depicted as a single block in the block diagram. However, while service device 110 may be a single device or a single group of devices, this disclosure contemplates that service device 110 may also be a group of devices, or even multiple different systems communicating to provide various content to client device 106. For example, service device 110 may include one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, or any other service.

[0143] Electronic documents (for brevity, we will simply call them documents) can, but do not need to, correspond to files. Documents can be stored as a part of a file that contains other documents, in a single file dedicated to the document in question, or in multiple collaborative files.

[0144] The embodiments of the subject matter and operation described in this specification can be implemented as digital electronic circuits, or as computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be or is included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Furthermore, although the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, discs, or other storage devices).

[0145] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0146] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0147] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on one computer or on multiple computers located in one location or distributed across multiple locations and interconnected through a communication network.

[0148] The processing and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processing and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0149] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), and so on. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as: semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated into them by dedicated logic circuitry.

[0150] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and pointing device, such as a mouse or trackball, for the user to use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0151] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or web browser through which a user can interact with the implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0152] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from that user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0153] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification in the context of independent embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0154] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0155] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and the desired result can still be obtained. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by a data processing device, the method comprising: The service device obtains videos uploaded by video publishers; The service device generates a concept index for each given video, wherein the concept index is generated based at least on (i) the concept conveyed by one or more objects depicted in the video and (ii) the salience level of the concept in the given video; The service device creates multiple video groups based on a concept index of the videos, wherein each given video group is created to include two or more distinct videos, each of the two or more distinct videos having a specified similarity level with other videos in the given video group; The service device generates insights about the multiple video groups based on data obtained through a feedback loop; and The service device modifies the way the at least one video is distributed over the network based on insights into a given video packet that includes at least one video. Generating a concept index for each given video includes: For each given video in the video: Obtain one or more knowledge graphs of multiple portions of a given video, wherein each of the knowledge graphs represents one or more concepts conveyed by the given video; and For each given knowledge graph, determine the existence share indicating the salience level of the concepts represented by that given knowledge graph, wherein the concept index is generated based at least in part on the number of instances of the given knowledge graph in a given video and the total existence share of the given knowledge graph over the video length.

2. The method according to claim 1, wherein, Generate a concept index for each given video, including: For each given knowledge graph obtained from a given video: Sum the existence shares of a given knowledge graph over the length of a given video; Determine the number of parts of a given video described by a given knowledge graph; and A concept index for a given video is generated based on the summed share of existence relative to the number of said parts.

3. The method according to claim 2, further comprising: For each given knowledge graph obtained from multiple videos uploaded by the video publisher: A reverse document frequency metric that generates a given knowledge graph based on the total number of videos represented by the given knowledge graph. as well as For each of the plurality of videos, the generated inverse document frequency is applied to the total presence share of the given knowledge graph.

4. The method according to claim 3, further comprising: Select a pair of videos from the plurality of videos; Generate a count of the shared knowledge graphs between the pair of videos, where each shared knowledge graph is a given knowledge graph representing the knowledge graph of each video in the pair of videos; For each specific shared knowledge graph between the pairs of videos: Based on the minimum presence share of the knowledge graph for any video in the pair, a shared similarity score for a specific shared knowledge graph is derived; and Based on the maximum presence share of the knowledge graph for any video in the pair, a possible similarity score for a specific shared knowledge graph is derived.

5. The method according to claim 4, further comprising: Dissimilarity counts are generated based on the number of dissimilarity knowledge graphs that represent only one of the videos in the pair. The dissimilarity score of the video pair is calculated based on the concept index of the dissimilarity knowledge graph for each video in the pair. as well as Based on dissimilarity score, possible similarity score, and shared similarity score, generate one or more clustering factors for the pair of videos.

6. The method according to claim 5, wherein, Creating multiple video groups includes: For each pair of videos, compare the clustering factor of the pair of videos with the specified similarity level; Include the first pair of videos in the same group whose clustering factors satisfy the specified level; and Exclude third videos from the same group whose clustering factors do not meet the specified level relative to the first pair of videos.

7. A video processing system, comprising: Memory devices; as well as One or more processors are configured to interact with the memory device and execute instructions that cause the one or more processors to perform operations, including: Get videos uploaded by video publishers; Generate a concept index for each given video, wherein the concept index is generated based at least on (i) the concept conveyed by one or more objects depicted in the video and (ii) the salience level of the concept in the given video; Multiple video groups are created based on a concept index of the video, wherein each given video group is created to include two or more distinct videos, each of the two or more distinct videos having a specified similarity level with the other videos in the given video group; Insights about the multiple video groups are generated based on data obtained through feedback loops; and Modify the method of distributing said at least one video over the network based on insights into a given video group that includes at least one video. Generating a concept index for each given video includes: For each given video in the video: Obtain one or more knowledge graphs of multiple portions of a given video, wherein each of the knowledge graphs represents one or more concepts conveyed by the given video; and For each given knowledge graph, determine the existence share indicating the salience level of the concepts represented by that given knowledge graph, wherein the concept index is generated based at least in part on the number of instances of the given knowledge graph in a given video and the total existence share of the given knowledge graph over the video length.

8. The system according to claim 7, wherein, Generate a concept index for each given video, including: For each given knowledge graph obtained from a given video: Sum the existence shares of a given knowledge graph over the length of a given video; Determine the number of parts of a given video described by a given knowledge graph; and A concept index for a given video is generated based on the summed share of existence relative to the number of said parts.

9. The system according to claim 8, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: For each given knowledge graph obtained from multiple videos uploaded by the video publisher: A reverse document frequency metric is generated based on the total number of videos represented by a given knowledge graph; and For each of the plurality of videos, the generated inverse document frequency is applied to the total presence share of the given knowledge graph.

10. The system according to claim 9, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: Select a pair of videos from the plurality of videos; Generate a count of the shared knowledge graphs between the pair of videos, where each shared knowledge graph is a given knowledge graph representing the knowledge graph of each video in the pair of videos; For each specific shared knowledge graph between the pairs of videos: Based on the minimum presence share of the knowledge graph for any video in the pair, a shared similarity score for a specific shared knowledge graph is derived; and Based on the maximum presence share of the knowledge graph for any video in the pair, a possible similarity score for a specific shared knowledge graph is derived.

11. The system according to claim 10, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: Dissimilarity counts are generated based on the number of dissimilarity knowledge graphs that represent only one of the videos in the pair. The dissimilarity score of the video pair is calculated based on the concept index of the dissimilarity knowledge graph for each video in the pair; and Based on dissimilarity score, possible similarity score, and shared similarity score, generate one or more clustering factors for the pair of videos.

12. The system according to claim 11, wherein, Creating multiple video groups includes: For each pair of videos, compare the clustering factor of the pair of videos with the specified similarity level; Include the first pair of videos in the same group whose clustering factors satisfy the specified level; and Exclude third videos from the same group whose clustering factors do not meet the specified level relative to the first pair of videos.

13. A non-transitory computer-readable medium storing one or more instructions, said instructions causing said one or more data processing means to perform operations when executed by said one or more data processing means, said operations including: Get videos uploaded by video publishers; Generate a concept index for each given video, wherein the concept index is generated based at least on (i) the concept conveyed by one or more objects depicted in the video and (ii) the salience level of the concept in the given video; Multiple video groups are created based on a concept index of the video, wherein each given video group is created to include two or more distinct videos, each of the two or more distinct videos having a specified similarity level with the other videos in the given video group; Insights about the multiple video groups are generated based on data obtained through feedback loops; and Modify the method of distributing said at least one video over the network based on insights into a given video group that includes at least one video. Generating a concept index for each given video includes: For each given video: Obtain one or more knowledge graphs of multiple portions of a given video, wherein each of the knowledge graphs represents one or more concepts conveyed by the given video; and For each given knowledge graph, determine the existence share indicating the salience level of the concepts represented by that given knowledge graph, wherein the concept index is generated based at least in part on the number of instances of the given knowledge graph in a given video and the total existence share of the given knowledge graph over the video length.

14. The non-transitory computer-readable medium according to claim 13, wherein, Generate a concept index for each given video, including: For each given knowledge graph obtained from a given video: Sum the existence shares of a given knowledge graph over the length of a given video; Determine the number of parts of a given video described by a given knowledge graph; and A concept index for a given video is generated based on the summed share of existence relative to the number of said parts.

15. The non-transitory computer-readable medium according to claim 14, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: For each given knowledge graph obtained from multiple videos uploaded by the video publisher: A reverse document frequency metric is generated based on the total number of videos represented by a given knowledge graph; and For each of the plurality of videos, the generated inverse document frequency is applied to the total presence share of the given knowledge graph.

16. The non-transitory computer-readable medium according to claim 15, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: Select a pair of videos from the plurality of videos; Generate a count of the shared knowledge graphs between the pair of videos, where each shared knowledge graph is a given knowledge graph representing the knowledge graph of each video in the pair of videos; For each specific shared knowledge graph between the pairs of videos: Based on the minimum presence share of the knowledge graph for any video in the pair, a shared similarity score for a specific shared knowledge graph is derived; and Based on the maximum presence share of the knowledge graph for any video in the pair, a possible similarity score for a specific shared knowledge graph is derived.

17. The non-transitory computer-readable medium according to claim 16, wherein, The instructions cause the one or more processors to perform an operation, the operation further including: Dissimilarity counts are generated based on the number of dissimilarity knowledge graphs that represent only one of the videos in the pair. The dissimilarity score of the video pair is calculated based on the concept index of the dissimilarity knowledge graph for each video in the pair; and Based on dissimilarity score, possible similarity score, and shared similarity score, generate one or more clustering factors for the pair of videos.

Images