Graph convolutional network for video grounding

Abstract

A method and apparatus including receiving a query describing an aspect in a video, the video including a plurality of frames; identifying a plurality of proposals that potentially correspond to the query, wherein each of the proposals includes a subset of the plurality of frames; ranking the proposals using a graph convolutional network that identifies relationships between the proposals; and based on the ranking, selecting one of the proposals as a video segment that is relevant to the query.

Description

BACKGROUND

[0001] The present invention relates to video grounding, in which a query is processed to identify respective segments in a video, and more particularly, to taking into account relationships between different segments of a video.

[0002] A video grounding searches a video to identify a segment (e.g., a plurality of consecutive video frames in a video) that corresponds to a natural language query. For example, a user can want to find a particular segment in a video in which a child is pushed on a swing. The user can define a query stating “a child on a swing.” The video grounding can use a machine learning (ML) algorithm to parse the video and identify different segments in the video that can show the information described in the query (referred to herein as suggestions) (e.g., a frame segment in which a child is swinging on a play set). The video grounding ranks the suggestions and selects the suggestion with the highest rank as an answer to the query. That is, current video grounding techniques individually consider the suggestions in ranking the suggestions to determine which suggestion best matches the natural language query. SUMMARY

[0003] One embodiment of the present invention is a method comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying a plurality of suggestions that potentially correspond to the query, wherein each of the suggestions comprises a subset of the plurality of frames; ranking the suggestions using a graph convolution network that identifies relationships between the suggestions; and based on the ranking, selecting one of the suggestions as a video segment relevant to the query.

[0004] Another embodiment of the present invention is a system comprising a processor and a memory. The memory comprises a program that, when executed by the processor, performs operations comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying a plurality of suggestions that potentially correspond to the query, wherein each of the suggestions comprises a subset of the plurality of frames; ranking the suggestions using a graph convolution network that identifies relationships between the suggestions; and based on the ranking, selecting one of the suggestions as a video segment relevant to the query.

[0005] Another embodiment of the invention is a computer program product for identifying a video segment relevant to a query. The computer program product includes a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by one or more computer processors to perform operations. The operations include receiving a query, the query describing an aspect in a video comprising a plurality of frames; identifying a plurality of suggestions that potentially correspond to the query, wherein each of the suggestions comprises a subset of the plurality of frames; ranking the suggestions using a graph convolutional network that identifies relationships between the suggestions; and based on the ranking, selecting one of the suggestions as a video segment relevant to the query.

[0006] BRIEF DESCRIPTION OF DRAWINGS

[0007] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:

[0008] Figure 1 A video grounding system using a convolutional graph network is shown in accordance with one embodiment.

[0009] Figure 2 A flowchart for performing video grounding is shown in accordance with one embodiment.

[0010] Figure 3 Identified suggestions in response to a natural language query are shown in accordance with one embodiment.

[0011] Figure 4 A flowchart for ranking suggestions using a graph convolutional network is shown in accordance with one embodiment.

[0012] Figure 5 A machine learning system for ranking suggestions is shown in accordance with one embodiment. DETAILED DESCRIPTION

[0013] Embodiments herein perform video grounding, where different proposals (e.g., video segments) identified in response to a natural language query are ranked using a graph convolutional network (GCN) that identifies relationships between the proposals. That is, in contrast to previous video grounding systems where proposals are independently (or separately) ranked, embodiments herein construct a graph and implement a GCN that identifies temporal relationships between the proposals. In one embodiment, the GCN is designed where each node in the network represents a fusion of visual features (derived from the proposals) and query features (derived from the natural language query). Moreover, edges in the graph can be constructed according to relationships between the proposals as measured by a similarity network. By performing graph convolutions, the video grounding system can capture interactions of two temporal segments as well as relationships between the proposals. Advantageously, unlike previous work that processes proposals separately and locally, the techniques described herein perform video grounding from a global and holistic perspective by explicitly modeling relationships between the proposals, which significantly improves its accuracy.

[0014] Figure 1 A video grounding system 100 using a GCN 125 is shown in accordance with one embodiment. Generally, the video grounding system 100 allows a user to submit a query 110 to identify a particular aspect of a video 105, such as a scene, action, or object in the video 105. The video 105 can include a plurality of frames that contain a plurality of different scenes, actions, or objects. The user can be looking for one of the scenes, actions, objects, or any other aspect in the video 105. For example, a first segment of the video 105 (e.g., a subset of consecutive frames in the video 105) can show a child’s playset, a second segment of the video 105 can show only a particular feature of the playset (e.g., an in-built sandbox or a swing), and a third segment of the video 105 can show a child interacting with the playset (e.g., a child being pushed up on the swing of a playset or sliding down a slide). The user can use the video grounding system 100 to search the video 105 and identify segments that are relevant (or best match) to the query 110. For example, the user can decide whether to purchase a playset (where the video 105 is a promotional video for the playset) and specifically wants a playset with an in-built sandbox. The user can submit a query 110 that includes the word “sandbox.” Using the techniques discussed in detail below, the video grounding system 100 can search the video and identify segments of the video that have a sandbox. Thereby, the user can view the identified segments to determine whether the playset meets their criteria (i.e., has an in-built sandbox) rather than having to view the entire video 105.

[0015] In another example, a user can want to see a child interacting with a toy set, so she can get a better idea of the scale (or size) of the toy set. The user can generate a query 110 that says "child using slide" or "being pushed on a swing," and the video grounding system 100 can use the query to identify a segment that shows an aspect (e.g., a scene, an action, or an object) of the video described by the query 110.

[0016] The video grounding system 100 includes a suggestion generator 115 that receives the video 105 and the query 110 as input. The suggestion generator 115 can use one or more machine learning (ML) algorithms and / or video parsing techniques to identify suggestions that represent candidate segments of the video 105 that can depict a scene, an action, or an object described by the query 110. That is, if the query 110 states "child swinging," the suggestion generator 115 identifies a number of candidate suggestions (e.g., different video segments) that the generator 115 determines can contain a child swinging.

[0017] To identify suggestions, the suggestion generator 115 can use any number of image processing techniques, natural language processing techniques, or text processing techniques (e.g., ML or otherwise). In one embodiment, the suggestion generator 115 evaluates individual frames to identify a start time (or frame) and an end time for each suggestion. Suggestions can be overlapping (e.g., have at least some frames in common) or can be non-overlapping in cases where each suggestion has unique frames. Embodiments herein are not limited to any particular technique for generating suggestions.

[0018] The video grounding system 100 includes an ML system 120 to select which of the suggestions generated by the suggestion generator 115 are most likely to be the best match (or most highly relevant) to the query 110. As shown, the ML system 120 receives the suggestions as input and outputs a segment 130 that is relevant to a scene, an action, or an object described by the query 110. In other words, the ML system 120 selects one of the suggestions as the segment 130 that best matches the query 110. For example, the ML system 120 can output a segment 130 defined by a start time and an end time in the video 105. The segment 130 can be defined by consecutive frames between the start time and the end time identified by the ML system 120. In one embodiment, the segment 130 is output to the user so that the user can watch the segment 130 to desirably watch what she is most interested in without having to start the video 105 from the beginning or manually search the video 105 by selecting a random location in the video 105. Figure 1

[0019] ​The ML system 120 includes a GCN 125. As discussed in more detail below, the GCN 125 allows the video grounding system 100 to identify relationships between the suggestions output by the suggestion generator 115. That is, rather than processing the suggestions independently, the graph convolutional network 125 can identify similarities or relationships between the suggestions that can advantageously improve the accuracy of the video grounding system 100 - i.e., increase the likelihood that the grounding system 100 selects a segment that is relevant (or matches) the description of the scene, action, or object defined in the query 110. In one embodiment, the graph convolutional network 125 ranks the suggestions based on their relationships, which can be more accurate than ranking the suggestions independently or separately as has been done previously.

[0020] Further, the embodiments herein can be used with multiple videos rather than a single video 105 as shown. For example, the suggestion generator 115 can search multiple videos (whether in the same file or different files) to identify suggestions formed from segments of those videos. The suggestions can then be forwarded to the ML system 120, which identifies relationships between the suggestions and orders the suggestions based on those relationships.

[0021] In one embodiment, the query 110 is a natural language query generated by a human user, but can be any query that describes an aspect of the video 105. Generally, the video grounding system 100 attempts to find a segment 130 of the video 105 that best matches the aspect described in the query 110. The query 110 can be text or speech converted to text.

[0022] Figure 2 A flowchart of a method 200 for performing video grounding according to one embodiment is illustrated. At block 205, a video grounding system receives a natural language query that describes a scene, action, object, or any other aspect in a video (or series of videos). In one example, a user submits a query to instruct the video grounding system to find a segment of a video that includes the aspect defined by the query. The video grounding system can enable the user to identify the relevant segment (or segments) or search the video without having to rely on trial and error or simply playing the video from the beginning.

[0023] At block 210, a suggestion generator in the video grounding system identifies a plurality of suggestions that potentially correspond to the query. In other words, the suggestion generator can identify different segments (i.e., subsets of frames in a video) that it predicts correspond to the query. For example, if the query is “barking dog,” the suggestion generator attempts to identify one or more segments in a video that show a dog barking. These segments are output as suggestions. As discussed above, embodiments herein are not limited to any particular technique for generating suggestions. The suggestion generator can use image processing techniques and natural language techniques (which can include multiple ML algorithms) to understand the query and identify relevant segments in a video.

[0024] Figure 3 Identified suggestions in response to a natural language query are shown in accordance with one embodiment. Figure 3 Video frames 300 in a video (or series of videos) are shown. In this example, the suggestion generator receives a query and identifies three suggestions 305A-C (or video segments) that can contain aspects of the video described by the query. As shown, the suggestions 305A-C overlap, with at least one of the frames in suggestion 305A also included in suggestion 305B, and at least one of the frames in suggestion 305B also included in suggestion 305C. These overlapping frames can establish relationships between the suggestions 305 (e.g., they have frames 300 in common). As described above, these relationships can be leveraged to improve the accuracy of identifying which of the three suggestions 305 can best match the query.

[0025] However, embodiments herein can also be used when the suggestions 305 do not have overlapping frames. That is, the GCN can identify temporal relationships between the suggestions, even when those suggestions do not have overlapping frames but have frames that are close to each other (e.g., contiguous or separated by a number of frames). By performing graph convolutions, the video grounding system can capture the interaction of two temporal segments and the relationships between the suggestions.

[0026] Returning to the method 200, assuming the suggestion generator identifies a plurality of suggestions, at block 215, the ML system ranks the suggestions using a graph convolutional network that identifies relationships between the suggestions. That is, in this embodiment, the ML system considers the relationships between the suggestions, which can result in significantly improved accuracy, rather than ranking the suggestions (or segments) independently of each other. Details of ranking the suggestions are described in Figure 4 and 5 .

[0027] At block 220, the ML system selects the suggestion with the highest ranking as the segment relevant to the query. That is, each suggestion is assigned a ranking based at least in part on the relationships between the frames. As such, the ranking can be more accurate than a ranking formed by evaluating the suggestions in isolation. The video grounding system can output the suggestion (or segment) with the highest ranking to the user.

[0028] Figure 4 A flowchart of a method 400 for ranking suggestions using a GCN is shown in accordance with one embodiment. For clarity, the method 400 is discussed in conjunction with Figure 5 The method 400 is discussed in conjunction with Figure 5 An ML system 120 for ranking suggestions 205 is shown. The ML system 120 includes a visual feature encoder 505 that evaluates the suggestions 205 and generates a feature vector for each suggestion. The feature vectors are provided to a graph constructor 515 for generating a graph that identifies relationships between the suggestions.

[0029] The ML system 120 also receives the query 110 (the query 110 is the same query 110 used by the suggestion generator (not shown) to identify the suggestions 205). That is, the query 110 is used twice in the video grounding system: once by the suggestion generator to identify the suggestions 205 and again to perform speech recognition on the query 110 by a bidirectional long short-term memory (Bi-LTSM) model 510. However, embodiments are not limited to the Bi-LTSM model 510, but can be used by other types of recurrent neural networks (RNNs) or deep learning networks that can perform speech recognition on the query 110.

[0030] The output of the Bi-LTSM model 510 is provided to the graph constructor 515, which generates the graph in conjunction with the output of the visual feature encoder 505. Advantageously, by receiving both visual features from the encoder 505 and query features from the Bi-LTSM model 510, the nodes in the graph can be a fusion of the visual features and the query features. Further, the edges in the graph are constructed based on the relationships between the suggestions as measured by a similarity network. In one embodiment, the similarity network measures L2 distance when constructing the edges in the graph.

[0031] The graph generated by the graph constructor 515, including nodes and edges, is provided to the GCN 125 for execution. The GCN is a powerful neural network architecture for performing machine learning on graphs. That is, the input to the GCN 125 is a graph that can include a plurality of nodes interconnected by edges. The output of the GCN 125 is provided to the vision-text fusion module 520 that fuses the results of the GCN 125 with the query features generated by the Bi-LTSM 510. In one embodiment, the vision-text fusion module 520 performs feature concatenation to fuse the features identified by the GCN 125 and the Bi-LSTM - i.e., the image features and the text / query features. The fused results are then provided to the fully connected (FC) layer 525. The FC layer 525 receives an input body from the vision text fusion module 520 and outputs an N-dimensional vector, where N is the number of suggestions. Further, the output can include a ranking of the suggestions.

[0032] Further, while not shown, the ML system 120 can include any number of computing devices, which can each include any number of computer processors (which can have any number of cores) and memory for executing the software components and modules shown in FIG. 5 (e.g., the visual feature encoder 505, the Bi-LSTM 510, the graph constructor 515, the GCN 125, etc.). Figure 5

[0033] Turning to the method 400, at block 405, the graph constructor 515 updates the node features and computes the edge weights of the graph. That is, the graph constructor 515 uses the output of the visual feature encoder 505 (i.e., the visual features of the suggestions 205) and the Bi-LTSM 510 (i.e., the query features) to generate the nodes in the graph. The nodes in the graph can be a fusion of these visual features and query features.

[0034] At block 410, the graph constructor 515 updates the edge features of the graph. In one embodiment, the graph constructor 515 computes the edge weights of the graph. That is, the graph constructor 515 uses the output of the visual feature encoder 505 (i.e., the visual features of the suggestions 205) and the Bi-LTSM 510 (i.e., the query features) to generate the edges in the graph. The edges (and their respective weights) are assigned based on the relationships between the suggestions.

[0035] At block 415, the GCN 125 performs node aggregation. That is, the GCN 125 is capable of aggregating the nodes of the graph received as input from the graph constructor 515. The embodiments herein are not limited to any particular technique for performing node aggregation.

[0036] ​At block 420, the ML system 120 ranks the suggestions 205. That is, the GCN 125, the vision-text fusion module 520, the FC 525, or a combination thereof can generate weights for the suggestions 205 that can be used to rank the suggestions 205. These weights are generated based on the relationships between the suggestions.

[0037] The description of various implementations of the present disclosure have been presented for purposes of illustration but is not intended to be exhaustive or limited to the implementations disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0038] In the foregoing, reference has been made to embodiments presented in the disclosure. However, the scope of the disclosure is not limited to the embodiments specifically described. Rather, any combination of features and elements is contemplated, whether relating to different embodiments or not, to enable and realize the embodiments contemplated. Further, while the embodiments disclosed herein can achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the disclosure. Thus, aspects, features, embodiments, and advantages discussed herein are merely illustrative and are not considered elements or limitations of the appended claims, unless explicitly recited in a claim. Likewise, reference to “the invention” should not be construed as being a reference to any one of the particular embodiments disclosed herein and should not be considered as an element or limitation of the appended claims, unless explicitly recited in a claim.

[0039] Various aspects of the present disclosure can be implemented as follows: entirely hardware embodiments, entirely software embodiments (including firmware, resident software, micro-code, etc.), or embodiments combining software and hardware aspects, which can all generally be referred to herein as a “circuit,” “module” or “system.”

[0040] The present disclosure can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0041] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0042] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0043] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0044] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0045] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0046] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0047] The flow and block diagrams in the drawings show the architectural, functional, and operational aspects of possible implementations of systems, methods and computer program products according to different embodiments of the present application. In this regard, each block in the flow or block diagrams can represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending on the functionality involved. Also, it is noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by dedicated hardware-based systems which perform the specified functions or acts, or combinations of dedicated hardware and computer instructions.

[0048] While the foregoing is directed to embodiments of the present application, other and further embodiments of the application can be devised without departing from the basic scope thereof, and the scope of the application is determined by the claims that follow.

Claims

1. A method for identifying a video segment relevant to a query, comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying a plurality of suggestions that potentially correspond to the query, wherein each of the suggestions comprises a subset of the plurality of frames; generating a graph based on the query and the plurality of suggestions; ranking the suggestions using a graph convolutional network (GCN) that identifies relationships between the suggestions, wherein the graph is input to the GCN; and selecting one of the suggestions as a video segment relevant to the query based on the ranking. Generating the graph comprises:

2. The method of claim 1, wherein, identifying visual features in the suggestions using a visual feature encoder; and generating query features from the query using a recurrent neural network (RNN). The graph comprises nodes and edges based on the visual features and the query features.

3. The method of claim 2, wherein, Ranking the suggestions comprises:

4. The method of claim 3, wherein, updating node features of the nodes in the graph; and computing edge weights of the edges in the graph. Ranking the suggestions further comprises:

5. The method of claim 3, wherein, performing node aggregation; and ranking the suggestions based on the node aggregation and results from processing the graph using the GCN. At least two of the plurality of suggestions comprise overlapping frames of the plurality of frames in the video.

6. The method of claim 1, wherein, At least two of the plurality of suggestions comprise non-overlapping subsets of the plurality of frames in the video.

7. The method of claim 6, wherein, 8. A system for identifying a video segment relevant to a query, comprising: a processor; and a memory comprising a program that, when executed by the processor, performs operations comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying a plurality of suggestions that potentially correspond to the query, wherein each of the suggestions comprises a subset of the plurality of frames; generating a graph based on the query and the plurality of suggestions; ranking the suggestions using a graph convolutional network (GCN) that identifies relationships between the suggestions, wherein the graph is input to the GCN; and selecting one of the suggestions as a video segment relevant to the query based on the ranking. Generating the graph comprises: identifying visual features in the suggestions using a visual feature encoder; and 9. The system of claim 8, wherein, generating query features from the query using a recurrent neural network (RNN). The graph comprises nodes and edges based on the visual features and the query features. Ranking the suggestions comprises:

10. The system of claim 9, wherein, updating node features of the nodes in the graph; and 11. The system of claim 10, wherein, computing edge weights of the edges in the graph. Ranking the suggestions further comprises: performing node aggregation; and 12. The system of claim 10, wherein, ranking the suggestions based on the node aggregation and results from processing the graph using the GCN. At least two of the plurality of suggestions comprise overlapping frames of the plurality of frames in the video. At least two of the plurality of suggestions comprise non-overlapping subsets of the plurality of frames in the video.

13. The system of claim 8, wherein, 15. A computer program product for identifying a video segment relevant to a query, the computer program product comprising:

14. The system of claim 13, wherein, ​ ​ A computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform operations comprising: receiving the query, the query describing an aspect in a video comprising a plurality of frames; identifying a plurality of proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames; generating a graph based on the query and the plurality of proposals; ranking the proposals using a graph convolutional network (GCN) that identifies relationships between the proposals, wherein the graph is input to the GCN; and selecting one of the proposals as the video segment relevant to the query based on the ranking.

16. The computer program product of claim 15, wherein, Generating the graph comprises: identifying visual features in the proposals using a visual feature encoder; and generating query features from the query using a recurrent neural network (RNN).

17. The computer program product of claim 16, wherein, The graph comprises nodes and edges based on the visual features and the query features.

18. The computer program product of claim 17, wherein, Ranking the proposals comprises: updating node features of the nodes in the graph; and computing edge weights of the edges in the graph.

19. The computer program product of claim 17, wherein, Ranking the proposals further comprises: performing node aggregation; and ranking the proposals based on the node aggregation and results of processing the graph using the GCN.

20. The computer program product of claim 15, wherein, At least two of the plurality of proposals comprise overlapping frames of the plurality of frames in the video.

Images