Video Segmentation Based on Weighted Knowledge Graphs

Abstract

A system, method, and computer program product for segmenting video includes at least one processing component, at least one memory component, a video extraction component, and a graphing component. The extraction component is configured to extract image data and text data from the video, identify entities in the image data, assign at least one entity relationship to the entities in the image data, identify entities in the text data, and assign at least one entity relationship to the entities in the text data. The graphing component is configured to generate an image knowledge graph for the entity relationships assigned to the entities in the image data, generate a text knowledge graph for the entity relationships assigned to at least two entities in the text data, and generate a weighted knowledge graph based on the image knowledge graph and the text knowledge graph.

Description

[Technical field]

[0001] The present disclosure relates to video segmentation, and more particularly, to segmenting videos based on relationships between entities identified within the videos. [Background technology]

[0002] Videos typically contain time-dependent image data (e.g., graphic images in video frames) accompanied by audio data (e.g., voice, music, and other noises). This data may be accompanied by text in the form of captions, subtitles, text extracted from audio data via a voice-to-text engine, scripts, transcripts, etc. A video may be segmented into groups of frames based on content such as entities (e.g., specific people, animals, inanimate objects, etc.) and other features extracted from the image data, audio data, or text data or a combination thereof (e.g., topics of discussion, places / environment, language, music, etc.). For example, locations in a movie may be detected based on content characteristic of that location (e.g., language, scenery, etc.). Video frames representing this environment may be grouped together to form segments. This is typically performed by video shot detection segmentation. However, this does not consider scenarios in the video occurring in more than one shot, or shots containing more than one scenario. Summary of the Invention

[0003] The present invention provides a method as claimed in claim 1, and a corresponding system and computer program product as claimed in claims 8 and 9. [Brief description of the drawings]

[0004] [Figure 1] A block diagram illustrating a video segmentation environment in accordance with some embodiments of the present disclosure. [Diagram 2] 1 is a flow diagram illustrating a process of segmenting a video according to some embodiments of the present disclosure. [Figure 3A] 1 is a schematic diagram illustrating a process for identifying entity relations in a video according to some embodiments of the present disclosure. [Figure 3B] FIG. 2 is a schematic diagram illustrating a process for generating a textual knowledge graph and an image knowledge graph according to some embodiments of the present disclosure. [Figure 3C] FIG. 2 is a schematic diagram illustrating a process for generating a weighted knowledge graph according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating a computer system according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating a cloud computing environment in accordance with some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating a set of functional abstraction model layers provided by a cloud computing environment in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Video segmentation is the process of grouping video frames into related segments. This makes it possible to localize specific parts of the video (e.g., in response to a user query). Videos may be segmented temporally based on shot transition detection. A video shot is a series of consecutive frames that, in relation to each other, represent a continuing action. Shot transitions may be detected based on visual or musical discontinuities, camera movement, etc. However, video segmentation based on shot transitions is frequently inaccurate when it comes to localizing specific scenarios / stories within a video.

[0006] For example, shot transition techniques often do not work when a scenario contains multiple shots or when a shot contains multiple scenarios. For example, a training video for an expense system may contain a scenario that describes how to register a user, how to attach a ticket, and how to enter an expense. A user may want to locate the explanation on how to enter an expense by searching for a segment that contains this scenario. However, with current automatic segmentation techniques, if a segment contains only consecutive frames or only a single shot, or both, the section of the video related to this scenario may be excluded. For example, there may be two or more non-consecutive sets of frames in a video that are related to one of the scenarios. Thus, when a user employs existing techniques to search for a scenario related to, for example, attaching a ticket in an expense system, the frames of the video related to this scenario may be missing from the segment generated for that scenario.

[0007] Disclosed herein are techniques for segmenting a video into scenarios. The disclosed techniques include using a weighted knowledge graph to identify related pairs of entities in a set of consecutive frames separated by regular intervals (e.g., 10 frames per set). These sets of frames are referred to herein as "pictures." The frames that contain related entities are then grouped into segments. For example, there may be 100 frames of a video, and frames 51-53 and 61-70 may each contain a pair of related entities, person A and person B. The pair of related entities is referred to herein as an "entity relation." In this example, the entity relations have a relation weight on the weighted knowledge graph that is greater than a threshold relation weight. The entity relations that have a weight greater than a threshold relation weight are referred to herein as "top relations." Because frames 51-53 and 61-70 contain top relations, these frames may be grouped into one video segment. Frames that contain other top relations may be grouped into additional segments. Frames that contain entity relationships that are not superordinate relationships can be linked to the segment that has the closest linked frame. For example, if frames 51-60 contain entity relationships that are not superordinate relationships, these frames can be linked to the aforementioned segment to form a segment that includes frames 51-70.

[0008] 1 is a block diagram illustrating a video segmentation environment 100 in accordance with some embodiments of the present disclosure. The video segmentation environment 100 includes a video 110 and a segmentation module 120 for segmenting the video 110. The segmentation module 120 includes an extraction component 140, a graphing component 150, and a grouping component 160.

[0009] The video 110 may be encoded according to any suitable file format standard, such as Audio Video Interleave (AVI), Waveform Audio (WAV), Animation (ANI), Audio Interchange File Format (AIFF), Graphics Interchange Format (GIF), Motion Picture Experts Group (MPEG, e.g., MPEG-4), 8-Bit Sampled Voice (8SVX), etc. The video 110 includes frames, one or more of which depict entities such as people, animals, or inanimate objects, or combinations thereof. The video 110 also includes textual data in the form of captions, subtitles, transcripts, computer-readable audio data, etc. The video 110 includes at least one set of consecutive frames, each set referred to herein as a "picture," as described in more detail below.

[0010] The extraction component 140 extracts entities and entity relationships from the image data of the video 110. Entities can include specific people, animals, inanimate objects, etc. An entity relationship is assigned to each pair of entities extracted from the same picture. The extraction component 140 can identify entities using various image recognition techniques such as region-based convolutional neural networks (R-CNN), scale-invariant feature transform (SIFT), real-time object detection techniques, pattern detection, edge / contour / ridge detection, histogram analysis, etc. There may be one or more pictures that do not contain entities. For example, entities can be identified based on face recognition. In such cases, in pictures that contain only features such as landscapes, no entities are identified.

[0011] The extraction component 140 maps entities to the pictures from which they were extracted. For example, the extraction component 140 may determine that a first picture ("Picture 1") depicts a first person and a bird, a second picture ("Picture 2") depicts a second person and a bird, and a third picture ("Picture 3") includes a first person, a second person, and a bird. The following entity relationships may then be established based on the image data: first person / bird (Picture 1 and Picture 3), second person / bird (Picture 2 and Picture 3), and first person / second person (Picture 3).

[0012] The extraction component 140 also identifies entities and entity relationships in the text and / or audio (e.g., voice) data from the video 110. The extracted text may include captions or other text associated with the video (e.g., subtitles, transcripts, etc.). The voice data from the extracted audio content may be converted to machine-coded text by at least one voice-to-text engine (e.g., using techniques such as artificial neural networks, convolutional neural networks, statistical modeling, hidden Markov models (HMMs), lattice-based analysis, entropy-based voice segmentation algorithms, CMUSphinx, etc.). The extraction component 140 may also extract features from the text obtained by converting characters (e.g., letters, numbers, punctuation, etc.) detected in the video 110 frames to machine-coded text using techniques such as optical character recognition (OCR) or intelligent character recognition (ICR). For example, the extraction component 140 can identify text printed on an object, such as a building sign, a book cover, or the like.

[0013] Entities may be identified in text in a variety of ways. For example, the extraction component 140 may identify keywords, such as character names. Additionally, text such as captions, scripts, etc. may identify which character (entity) is speaking each piece of text. In some embodiments, entities may also be identified in speech audio data using text-independent and / or text-dependent speaker recognition techniques (e.g., frequency estimation, linear predictive coding (LPC), HMM, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, decision trees, etc.).

[0014] Additionally, extraction component 140 maps entities to video frames associated with the text from which the entities were extracted. The frames may be associated with text spoken to, by, or about a particular entity. Returning to the previous example including pictures 1-3 of video 110, video 110 may be divided into pictures at 10 frame intervals, and picture 3 may include frames 21-30. In this picture, frames 21-24 may include text spoken by a first person to a second person (e.g., "hello"), and frames 27-29 may include a sentence about a bird spoken by the second person to the first person (e.g., "look at the bird"). Based on the entities and associated frames, the extraction component 140 can identify the following entity relationships: first person / second person (frames 21-24 and 27-29), first person / bird (frames 27-29), and second person / bird (frames 27-29).

[0015] The graphing component 150 generates an image knowledge graph and a text knowledge graph based on the entities and entity relationships identified in the extracted image data and text data. The graphing component 150 also generates a weighted knowledge graph based on the image knowledge graph and the text knowledge graph. The weighted knowledge graph includes each entity relationship of the entities in the text data and image data. The entity relationships are weighted based on the frequency with which each relationship occurs. Referring back to the previous example, the weight of the first person / bird entity relationship may be higher than the other two relationships because that entity relationship occurs in two pictures (picture 1 and picture 3). The weight of the first person / bird entity relationship may be even higher because that entity relationship occurs twice in picture 3 (frames 21-24 and 27-29). Examples of knowledge graphs are shown in Figures 3A-3C.

[0016] The grouping component 160 groups the video frames into segments based on the weighted entity relationships in the weighted knowledge graph. To do so, the grouping component 160 identifies a top entity relationship. A top entity relationship is an entity relationship in the weighted knowledge graph that has a weight that exceeds a threshold relationship weight value. The threshold may be a pre-set value or a user-entered value. In some embodiments, the threshold may be adjusted based on a desired segment size and / or selectivity. Frames corresponding to each entity relationship that has a relationship weight that exceeds the threshold are grouped together into a segment. Frames corresponding only to the remaining entity relationships that have a relationship weight that is below the threshold are grouped with the closest temporally linked frame. If there are two or more segments at the same distance from a frame with a remaining entity relationship, the frame may be grouped with the segment with the top entity relationship that has the highest weight value. Examples of relationship weight values ​​and grouped segments are described in more detail in connection with FIG. 3C.

[0017] Figure 2 is a flow diagram illustrating a process 200 for segmenting a video according to some embodiments of the present disclosure. For purposes of illustrating the process 200, but not by way of limitation, Figure 2 is described within the context of the video segmentation environment 100 of Figure 1. Elements shown in Figure 2 are identical to elements shown in Figure 1, and the same reference numbers are used in both figures.

[0018] Process 200 begins when video 110 is received. This is shown at step 210. Video formats are described in more detail in connection with FIG. 1. At least one frame of video 110 depicts two or more entities (e.g., humans, animals, animated characters, etc.). Additionally, at least one of the frames is associated with text data that refers to at least two of the entities. In some embodiments, the text data is from captions, subtitles, scripts, etc. The text can also be from audio data such as recorded dialogue or narration. Video 110 is divided at regular intervals to form a set of frames, referred to herein as "pictures." The number of frames in each picture may be pre-set and / or selected by a user. In some embodiments, there are 10 frames per picture, but a picture interval may include any number of consecutive frames (e.g., 24 frames, 30 frames, 100 frames, 120 frames, 240 frames, 300 frames, 500 frames, 1,000 frames, etc.).

[0019] The text knowledge graph and image knowledge graph are generated based on the entities and entity relationships identified in the video 110. This is shown in step 220. Entities and entity relationships are identified in the text data and image data extracted from the video 110 by the extraction component 140. The extraction of data and the identification of entities and entity relationships are described in more detail in connection with FIG. 1. The graphing component 150 generates an image knowledge graph that links entities that appear in the same picture based on the entity relationships. Each picture can be assigned a picture number n, which can be an integer greater than zero (e.g., picture 1, picture 2, picture 3, etc.). Each pair of linked entities is referred to herein as an "entity relationship." The image knowledge graph can also indicate the number of pictures in which each entity relationship occurs. The generation of the image knowledge graph is described in more detail in connection with FIG. 1 and FIG. 3A.

[0020] The textual knowledge graph includes entities and entity relationships identified in the textual data extracted by the extraction component 140. For example, when text containing a conversation between two entities is extracted from the video 110, entity relationships may be established in the textual knowledge graph. The textual knowledge graph may also indicate the number of occurrences of each entity relationship identified in the textual data and which frames are associated with these occurrences. Generation of the textual knowledge graph is described in more detail in connection with Figures 1 and 3B.

[0021] A weighted knowledge graph is then generated, as shown at step 230. The weighted knowledge graph is generated by the graphing component 150 and includes entity relationships from both the image knowledge graph and the text knowledge graph. The graphing component 150 weights each entity relationship based on the number of pictures in which the entity relationship appears or the number of occurrences of the entity relationship in the text data. Weighted knowledge graphs are described in more detail in connection with Figures 1 and 3C.

[0022] Frames containing a superordinate relationship from the weighted knowledge graph are grouped into video segments. This is shown in step 240. The grouping component 160 identifies superordinate relationships in the weighted knowledge graph. A superordinate relationship is an entity relationship that has a relationship weight greater than a threshold relationship weight. Each video segment formed in step 240 includes frames for which at least one superordinate relationship was identified. In the example described in connection with FIG. 1, video 110 has pictures 1, 2, and 3, and the superordinate relationship may be first person / bird. Thus, segments may be generated for frames 21-24 and 27-29 because these frames contain the first person / bird entity relationship. Examples of grouping based on relationship weights are described in more detail in connection with FIG. 1 and FIG. 3C.

[0023] Once a video segment is generated for a superordinate relationship, it is determined whether there are any frames remaining that have not been grouped into a segment. This is shown in step 250. If it is determined that there are no remaining frames, the process 200 may end. However, if there are frames remaining, the remaining frames are added to the segment formed in step 240 with the closest frame. This is shown in step 260. The remaining frames are frames that include an entity relationship with a relationship weight below a relationship weight threshold. The remaining frames do not include a superordinate relationship. In the example of video 110 including frames 1-90, in step 240, a segment including frames 20-35 and frames 45-90 may be generated for the superordinate relationship. In step 250, it may be determined that frames 36-44 are the remaining frames. Then, in step 260, these frames may be added to the segment with frames 20-35 and 45-90 as they include the closest frames. If there are segments with frames the same distance from one or more remaining frames, the remaining frames may be grouped with the segment that has the entity relationship with a larger relationship weight. Once each remaining frame has been added to the video segment, the process 200 ends.

[0024] 3A-3C are schematic diagrams illustrating processes 300, 301, and 302 for generating a knowledge graph for video segmentation according to some embodiments of the present disclosure. To explain the processes 300-302, FIGS. 3A-3C are described within the context of the video segmentation environment 100 of FIG. 1 and the process 200 of FIG. 2, without this being a limitation of the embodiments. Elements shown in FIGS. 3A-3C are identical to elements shown in FIGS. 1 and 2, and the same reference numbers are used in each figure.

[0025] FIG. 3A is a schematic diagram illustrating a process 300 for identifying entity relationships in a video 110, according to some embodiments of the present disclosure. Extraction component 140 (FIG. 1) extracts text data 310 from video 110 and identifies entities and entity relationships in data 310. The entities extracted from text data 310 are shown in table 320, which includes columns 322, 324, and 326. Column 322 includes numbers (1-8) each indicating an occurrence of at least one entity relationship in text data 310. Each portion of text 310 that includes an occurrence is associated with one or more video frames. The video frames corresponding to the occurrence in column 322 are identified by the frame number in column 324. Column 326 includes the names of the entities included in the entity relationship at each occurrence. Based on table 320, the entity relationships extracted from the text data 310 include Sarah / Geoff (occurrence 1), Thuy / Geoff (occurrence 2, occurrence 4, and occurrence 5), Thuy / Sarah (occurrence 3), Thuy / Cheryl (occurrence 4 and occurrence 5), Cheryl / Geoff (occurrence 4 and occurrence 5), Barbara / Hannah (occurrence 6), and Danielle / William (occurrence 7 and occurrence 8).

[0026] Extraction component 140 also extracts image data 330 from video 110. The entities and entity relationships extracted from image data 330 are shown in table 340, which includes columns 342, 344, and 346. Column 342 includes picture numbers 1 through 10, each of which corresponds to a set of ten consecutive frames in video 110. Column 344 includes the number of frames contained in each picture. Column 346 includes the names of the entities identified in each picture. Based on table 340, the entity relationships identified within image data 330 include Sarah / Geoff (Picture 1 and Picture 3), Sarah / Kim (Picture 1), Geoff / Kim (Picture 1), Thuy / Kim (Picture 2), Thuy / Cheryl (Picture 4 and Picture 5), Barbara / Ned (Picture 6), Barbara / Hannah (Picture 7), Barbara / Rita (Picture 7), Rita / Hannah (Picture 7), Danielle / William (Picture 8), Danielle / Ellen (Picture 8 and Picture 9), and William / Ellen (Picture 8 and Picture 10).

[0027] 3B is a schematic diagram illustrating a process 301 for generating a text knowledge graph 350 and an image knowledge graph 360 according to some embodiments of the present disclosure. Techniques that can be used to perform the process 301 are described in more detail in relation to step 220 of process 200. The text knowledge graph 350 and the image knowledge graph 360 are generated by the graphing component 150 (FIG. 1) based on data from tables 320 and 340 (FIG. 3A), respectively. The nodes in each knowledge graph 350 and 360 represent entities, and the letters in the nodes are the first initials of the entities listed in tables 320 (column 326) and 340 (column 346), respectively.

[0028] The lines in each knowledge graph 350 and 360 represent entity relationships. Information from table 320, such as the number of occurrences and corresponding frames, may be associated with the lines in textual knowledge graph 350. For example, a line representing an entity relationship of Sarah / Geoff (S / G) may be linked to information indicating that there is one occurrence of the entity relationship of S / G, and that occurrence is in frames 1-3. Additionally, the picture number (column 342) and corresponding frame (column 344) in table 340 may be associated with the lines in image knowledge graph 360. For example, the entity relationship of S / G may be linked to information indicating that both Sarah and Geoff appear in picture 1 and picture 3, and that the occurrences correspond to frames 1-10 and 21-30. Techniques that may be used to perform process 300 and process 301 are described in more detail in connection with steps 210 and 220 of process 200.

[0029] 3C is a schematic diagram illustrating a process 302 for generating a weighted knowledge graph 370 according to some embodiments of the present disclosure. The weighted knowledge graph 370 includes solid lines representing superior relations and dashed lines representing remaining entity relations. Techniques that can be used to perform the process 302 are described in more detail in relation to steps 230-260 of the process 200. The weighted knowledge graph 370 is generated by the graphing component 150, which computes the union of the entities and entity relations in the text knowledge graph 350 and the image knowledge graph 360 (FIG. 3B). This is expressed as: Entities Weighted KG}={Entities Text KG}∪{Entities Image KG} {Relations Weighted KG}={Relations Text KG}∪{Relations Image KG} where "KG" stands for "knowledge graph."

[0030] The graphing component 150 determines a relationship weight for each entity relationship based on the number of times the entity relationship occurs in the text knowledge graph 350 and the image knowledge graph 360. For example, the relationship weight for an entity relationship may be calculated using the following formula:

number

[0031] Frames (column 386) corresponding to the superordinate relationships in table 380 are grouped into video segments by grouping component 160 (FIG. 1). In the example shown in FIG. 3C, video 110 (FIGS. 1 and 3A) may be divided into three segments. The first segment may include frames 1-10 and 21-50, which include superordinate relationship entities Thuy, Cheryl, Geoff, and Sarah (nodes T, C, S, and G). The second segment may include frames 51-53 and 61-70, which include superordinate relationship entities Barbara and Hannah (nodes B and H). The third video segment may include frames 71-100, which include superordinate relationship entities Danielle, William, and Ellen (nodes D, W, and E).

[0032] Once frames with a superior relationship are grouped together, grouping component 160 may determine that frames 11-20 and 51-60 remain and are not linked to a segment. The remaining frames are then linked to the segment with the closest frame. Frames 11-20 are closest to frames 1-10 and 21-50 and are therefore linked to the first segment. Additionally, frames 51-60 are closest to frames 51-53 and 61-70 and are therefore linked to the second segment. Once the frames are grouped, the first segment includes frames 1-50, the second segment includes frames 51-70, and the third segment includes frames 71-100.

[0033] 4 is a block diagram illustrating an exemplary computer system 400 that can be used to implement one or more of the methods, tools, components, and any associated functionality described herein (e.g., using one or more processor circuits or computer processors of a computer). In some embodiments, the major components of computer system 400 include one or more processors 402, a memory subsystem 404, a terminal interface 412, a storage interface 416, an input / output device interface 414, and a network interface 418, all of which may be communicatively coupled, directly or indirectly, for inter-component communication via a memory bus 403, an input / output bus 408, a bus interface unit 407, and an input / output bus interface unit 410.

[0034] Computer system 400 includes one or more general purpose programmable central processing units (CPUs) 402-1, 402-2, and 402-N, collectively referred to herein as CPUs 402. In some embodiments, computer system 400 includes multiple processors typical of relatively large systems. However, in other embodiments, computer system 400 may alternatively be a single CPU system. Each CPU 402 may execute instructions stored in memory subsystem 404 and may include one or more levels of on-board cache.

[0035] Memory 404 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing or encoding data and programs. In some embodiments, memory 404 represents the entire virtual memory of computer system 400 and may include virtual memory of other computer systems coupled to computer system 400 or connected via a network. While memory 404 is conceptually a single monolithic entity, in other embodiments, memory 404 is a more complex construct, such as a hierarchy of caches and other memory devices. For example, memory may exist in multiple levels of caches, and these caches may be further divided by function, such that some caches hold instructions and other caches hold non-instruction data used by one or more processors. Memory may be further distributed and associated with different CPUs or sets of CPUs, as is known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures.

[0036] These components are shown as being contained within memory 404 in computer system 400. However, in other embodiments, some or all of these components may be on different computer systems and may be accessed remotely, for example, over a network. Computer system 400 may use a virtual addressing mechanism that allows programs in computer system 400 to behave as if they can only access a single larger storage entity, rather than accessing multiple smaller storage entities. Thus, although segmentation module 120 is shown as being contained within memory 404, the components of memory 404 are not necessarily all contained entirely in the same storage device at the same time. Additionally, while these components are shown as separate entities, in other embodiments, some of these components, some portions of these components, or all of these components may be packaged together.

[0037] In one embodiment, segmentation module 120 includes instructions that execute on, or are interpreted by, processor 402 to perform functions described further in this disclosure. In another embodiment, segmentation module 120 is implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, or other physical hardware devices or combinations thereof in place of or in addition to a processor-based system. In another embodiment, segmentation module 120 includes data in addition to instructions.

[0038] 4, memory bus 403 is shown as a single bus structure providing direct communication paths between CPU 402, memory subsystem 404, display system 406, bus interface 407, and input / output bus interface 410, in some embodiments memory bus 403 may include multiple different buses or communication paths, which may be configured in any of a variety of forms, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel redundant paths, or any other suitable type of configuration. Additionally, while input / output bus interface 410 and input / output bus 408 are shown as single respective units, in some embodiments computer system 400 may include multiple input / output bus interface units 410, multiple input / output buses 408, or both. Additionally, while multiple input / output interface units are shown isolating input / output bus 408 from the various communication paths extending to the various input / output devices, in other embodiments some or all of the input / output devices may be directly connected to one or more system input / output buses.

[0039] The computer system 400 may include a bus interface unit 407 for handling communication between the processor 402, memory 404, display system 406, and an input / output bus interface unit 410. The input / output bus interface unit 410 may be coupled to an input / output bus 408 for transferring data to and from various input / output units. The input / output bus interface unit 410 communicates with a number of input / output interface units 412, 414, 416, and 418, also known as input / output processors (IOPs) or input / output adapters (IOAs), via the input / output bus 408. The display system 406 may include a display controller. The display controller may provide visual type, audio type, or both types of data to the display device 405. The display system 406 may be coupled to a display device 405, such as a standalone display screen, a computer monitor, a television, or a tablet or handheld device display. In alternative embodiments, one or more of the functions provided by the display system 406 may be incorporated into the processor 402 integrated circuit. Additionally, one or more of the functions provided by the bus interface unit 407 may be incorporated into the processor 402 integrated circuit.

[0040] In some embodiments, computer system 400 is a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface but receives requests from other computer systems (clients). Further, in some embodiments, computer system 400 is implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smart phone, a network switch or router, or any other suitable type of electronic device.

[0041] It should be noted that Figure 4 is intended to depict representative major components of an exemplary computer system 400. However, in some embodiments, the individual components may have more or less complexity than depicted in Figure 4. There may be components other than or in addition to those shown in Figure 4, and the number, type, and configuration of such components may vary.

[0042] In some embodiments, the data storage and data retrieval processes described herein may be implemented within a cloud computing environment, as described below in connection with Figures 4 and 5. Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0043] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0044] Its features are as follows:

[0045] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0046] Pervasive Network Access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0047] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated on demand. Location independence is achieved in that consumers generally have no control or knowledge over the exact location of the resources provided, although at a higher level of abstraction they may specify a location (e.g., country, state, or data center).

[0048] Rapid Elasticity: Capabilities can be rapidly and flexibly, sometimes automatically, provisioned to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time.

[0049] Service Metering: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a certain level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services they utilize.

[0050] The service model is as follows:

[0051] Software as a Service (SaaS): The capability offered to a consumer to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). With the expected exception of limited user-specific application configuration settings, the consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application features.

[0052] Platform as a Service (PaaS): The capability offered to a consumer to deploy consumer-generated or acquired applications generated using programming languages ​​and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure such as networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0053] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0054] The deployment model is as follows:

[0055] Private Cloud: The cloud infrastructure is operated exclusively for an organization. This cloud infrastructure may be managed by the organization or a third party and may exist on-premise or off-premise.

[0056] Community Cloud: Cloud infrastructure is shared across multiple organizations to support a particular community of shared concerns (e.g., mission, security requirements, policies, and compliance considerations). This cloud infrastructure may be managed by the organization or a third party and may exist on-premise or off-premise.

[0057] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0058] Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0059] A cloud computing environment is a service-driven approach that focuses on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0060] FIG. 5 is a block diagram illustrating a cloud computing environment 500 according to some embodiments of the present disclosure. As shown, the cloud computing environment 500 includes one or more cloud computing nodes 510 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 520-1, a desktop computer 520-2, a laptop computer 520-3, or an automobile computer system 520-4, or combinations thereof, can communicate. The nodes 510 may communicate with each other. The nodes 510 may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described herein above (not shown). This enables the cloud computing environment 500 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on a local computing device. It should be understood that the types of computing devices 520-1 through 520-4 shown in FIG. 5 are intended as examples only, and that computing node 510 and cloud computing environment 500 can communicate with any type of computerized device over any type of network and / or network-addressable connections (e.g., using a web browser).

[0061] 6 is a block diagram illustrating a set of functional abstraction model layers 600 provided by cloud computing environment 500 in accordance with some embodiments of the present disclosure. It should be understood in advance that the components, layers, and functions illustrated in FIG. 6 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As illustrated, the following layers and corresponding functions are provided:

[0062] Hardware and software layer 610 includes hardware and software components. Examples of hardware components include mainframes 611, RISC (reduced instruction set computing) architecture based servers 612, servers 613, blade servers 614, storage devices 615, and networks and networking components 616. In some embodiments, software components include network application server software 617 and database software 618.

[0063] The virtualization layer 620 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 621, virtual storage 622, virtual networks including virtual private networks 623, virtual applications and operating systems 624, and virtual clients 625.

[0064] In one example, the management layer 630 provides the functionality described below. Resource provisioning 631 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 632 provides cost tracking as resources are utilized within the cloud computing environment and charging or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 633 provides consumers and system administrators with access to the cloud computing environment. Service level management 634 provides allocation and management of cloud computing resources such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 635 provides pre-positioning and procurement of cloud computing resources anticipated to be required in the future according to SLAs.

[0065] The workload tier 640 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 641, software development and lifecycle management 652, virtual classroom instructional delivery 643, data analytics processing 644, transaction processing 645, and segmenting videos based on weighted knowledge graphs 646.

[0066] The present disclosure may be a system, method, and / or computer program product, which may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

[0067] A computer-readable storage medium is a tangible device that can hold and store instructions for use by an instruction execution device. Examples of computer-readable storage media can include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded on them, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0068] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0069] The computer readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to perform aspects of the disclosure, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0070] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.

[0071] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable medium such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0072] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

Claims

1. 1. A processor-implemented method comprising: receiving a video; extracting image data and text data from the video; identifying at least two entities within the image data; assigning an entity relationship to each pair of identified entities within the image data; identifying at least two entities within the text data; assigning an entity relationship to each pair of entities identified in the text data; generating an image knowledge graph for entity relationships assigned to each pair of entities in the image data; generating a text knowledge graph for entity relationships assigned to each pair of entities in the text data; generating a weighted knowledge graph in which a relationship weight is assigned to each entity relationship calculated based on the number or frequency of occurrence of each entity relationship in the image knowledge graph and the text knowledge graph; grouping frames of the video into video segments based on the weighted knowledge graph; A method comprising:

2. 2. The method of claim 1 , wherein the weighted knowledge graph includes a relationship weight for each entity relationship calculated based on a number of times an entity relationship assigned to each pair of entities in the image data occurs in the image knowledge graph and a number of times an entity relationship assigned to each pair of entities in the text data occurs in the text knowledge graph.

3. identifying a superior relationship among the entity relationships assigned to each pair of entities in the image data and each pair of entities in the text data, the superior relationship being an entity relationship having a relationship weight greater than a threshold relationship weight; selecting a frame of the video that corresponds to the high ranking relationship; grouping the frames into video segments; The method of claim 2 , further comprising:

4. determining that there are remaining frames of the video that do not include the high ranking relation; determining that the frame in the video segment is closest to the remaining frame; grouping the remaining frame with a video segment of at least two of the video segments in which the frame is closest to the remaining frame; The method of claim 3 further comprising:

5. The method of claim 1 , wherein the video is divided into pictures, each picture comprising a set of frames.

6. The method of claim 1 , wherein the text data is a caption.

7. The method of claim 1 , wherein the at least two entities in the image data are identified based on facial recognition.

8. A system comprising means for carrying out all the steps of the method according to claims 1 to 7.

9. A computer program product for causing a computer to carry out the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having the computer program of claim 9 recorded thereon.

Images