Summarization apparatus, summarization method, and program

Abstract

To generate a summary that holds causal relationship for a plurality of images.SOLUTION: A summarization apparatus includes: a rhetorical structure tree generation unit configured to generate a rhetorical structure tree for a plurality of images regarding a certain event related in time series; an event relation generation unit configured to generate a tree structure that represents a relationship between events related to the images, using the images as nodes, based on the rhetorical structure tree; and a summary generation unit configured to extract subtrees from the tree structure so as to satisfy the constraints for the summary of the images.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a summarization device, a summarization method, and a program. [Background technology]

[0002] Conventionally, multimodal summarization has been realized by extracting several frames (still images corresponding to a certain time in the video) from a given video and generating captions for each frame, thereby presenting a summary of the video in two modalities: frames (still images) and text (Non-Patent Document 1). Hereinafter, the frames extracted as a summary are referred to as "key frames." [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kudo et al., "A Challenging Multimodal Video Sum-mary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video", Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7380-7402. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of Non-Patent Document 1 does not have a mechanism for analyzing and considering the relationships between events (events or scenes) in a video, and therefore the relationships between events (e.g., causal relationships) in the video do not necessarily hold between the extracted frames. Therefore, when viewed as a summary, both the frames and the text may be misleading to the user. For example, if a causal relationship exists between scene A and scene B, and a causal relationship also exists between scene C and scene D, deleting scenes B and C (i.e., extracting only scenes A and D as a summary) may lead to the user misunderstanding that a causal relationship exists between scene A and scene D.

[0005] The above problem is not limited to video, but also applies to multiple images (still images) relating to events that are related in time series.

[0006] The present invention has been made in view of the above points, and has as its object to make it possible to generate a summary that maintains the causal relationships between multiple images. [Means for solving the problem]

[0007] In order to solve the above problem, the summarization device has a rhetorical structure tree generation unit configured to generate a rhetorical structure tree for multiple images related to events that are related in chronological order, an event relationship generation unit configured to generate a tree structure based on the rhetorical structure tree, with the images as nodes, that represents the relationships between events related to the images, and a summary generation unit configured to extract subtrees from the tree structure so as to satisfy constraints on the summarization of the multiple images. [Effects of the Invention]

[0008] It is possible to generate summaries that preserve causal relationships for multiple images. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a summarizing device 10 according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a functional configuration of a summarizing device 10 according to an embodiment of the present invention. [Figure 3] 10 is a flowchart illustrating an example of a processing procedure executed by the summarizing device 10. [Figure 4] FIG. 10 is a diagram showing an example of a rhetorical structure tree in which events are leaf nodes. [Figure 5] This is the first diagram to explain the process of converting a rhetorical structure tree into a discourse dependency structure tree. [Figure 6] This is the first diagram to explain the process of converting a rhetorical structure tree into a discourse dependency structure tree. [Figure 7] FIG. 10 is a diagram showing an example of a discourse dependency structure tree converted from a rhetorical structure tree. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure for extracting a rooted subtree that satisfies a constraint as a summary from a discourse dependency structure tree. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the hardware configuration of a summarizing device 10 according to an embodiment of the present invention. The summarizing device 10 in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0011] The program that realizes the processing in the summarization device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0012] When an instruction to start the program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the summarization device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0013] In this embodiment, summarization device 10 performs multimodal summarization, which represents a summary of an input video in two modalities: image and text.

[0014] FIG. 2 illustrates an example of the functional configuration of a summarization device 10 according to an embodiment of the present invention. To address the issue of possible inaccurate summaries of relationships between events (events or scenes) in a video, the summarization device 10 divides the video into events (hereinafter referred to as "events") and then represents the rhetorical relationships between them as a tree. Hereinafter, this is referred to as a rhetorical structure tree. The summarization device 10 converts the rhetorical structure tree into a tree structure representing the relationships between events (parent-child relationships or causal relationships). In this embodiment, a discourse dependency structure tree is used as the tree structure; however, other tree structures that achieve the same purpose may be used. The summarization device 10 extracts frames and generates a text summary by pruning the branches of the tree structure. To generate such a summary, the summarization device 10 includes a division unit 11, a rhetorical structure tree generation unit 12, an event relationship generation unit 13, and a summary generation unit 14. These units are implemented by a processor 104 executing one or more programs installed in the summarization device 10.

[0015] The following describes the processing procedure executed by the summarizing device 10. Fig. 3 is a flowchart for explaining an example of the processing procedure executed by the summarizing device 10.

[0016] In step S101, the division unit 11 divides a moving image input as a target for summarization (hereinafter referred to as a "target moving image") into events. The target moving image is composed of a sequence of frames. One event (scene) is composed of a sequence of consecutive frames from the frame sequence.

[0017] To divide a video into events, publicly available tools such as (https: / / pypi.org / project / eventdetector-ts / ) can be used. Using such tools, a video is divided into segments for each event. Each event has a start time and an end time. Specifically, the start time of an event is the time corresponding to the first frame in a sequence of frames related to that event, and the end time of that event is the time corresponding to the last frame in that sequence.

[0018] Next, the rhetorical structure tree generating unit 12 generates a rhetorical structure tree in which each event that divides the target moving image is a leaf node (S102).

[0019] FIG. 4 is a diagram showing an example of a rhetorical structure tree in which leaf nodes are events. In FIG. 4, an example is shown in which a target video is divided into five events E1 to E5. Therefore, each leaf node corresponds to one of the events E1 to E5. The intermediate nodes of the rhetorical structure tree correspond to event spans (hereinafter referred to as "spans"), which are a series of one or more consecutive events corresponding to the leaf nodes that are descendants of the intermediate nodes, and represent the attributes (core (N) or satellite (S)) of the spans corresponding to the intermediate nodes. A satellite modifies its paired core with the rhetorical relationship shown on the branch (i.e., the direction of modification is from the satellite to the core). For example, the span from events E1 to E3 modifies the span from events E4 to E5 with the rhetorical relationship called "Background." Examples of rhetorical relationships include Preparation, Cause, Result, Supplement, Comparison, List, Background, Summary, Restatement, Attribution, Elaboration, Enablement, Manner-means, and Same-unit.

[0020] The generation of a rhetorical structure tree with events as leaf nodes can be achieved by modifying the input section of an existing text rhetorical structure analyzer so that vectors corresponding to spans (frame sequences) can be input, and then using a rhetorical structure analyzer obtained by retraining. To obtain a vector representation of a span, for example, TSP (Reference 1) can be used.

[0021] Hereinafter, an intermediate node representing a satellite will be referred to as a "satellite node," and an intermediate node representing a nucleus will be referred to as a "nucleus node."

[0022] Here, because the rhetorical structure tree in Figure 4 represents the relationships between spans, the relationships between individual events cannot be understood as they are. That is, when two adjacent spans (scenes) of length 1 are adjacent to each other, their relationship is clear, but when one of the adjacent spans has a length of 2 or more, the relationship between the individual events is not clearly visible. For example, the relationship between events E3 and E5 is difficult to see as it is in Figure 4. Therefore, the event relation generation unit 13 executes processing to convert the rhetorical structure tree into a discourse dependency structure tree, which is a tree structure that represents the parent-child relationships between events.

[0023] First, the event relation generation unit 13 executes a loop process L1 for each leaf node of the rhetorical structure tree. The leaf node that is the processing target in the loop process L1 will be referred to as the "target leaf node" hereinafter.

[0024] In step S103, the event relation generation unit 13 traces the nodes connected by branches from the target leaf node toward the root until it finds the first satellite node.

[0025] If a satellite node is found up to the root (Yes in S104), the event relation generation unit 13 traces only the core nodes from the core node that is the sibling of the found satellite node toward the leaves (S105). If there are multiple core nodes, the rightmost core node is traced. The event relation generation unit 13 sets the event related to the leaf node reached as the parent node of the event related to the target node in the discourse dependency structure tree (S106). Next, the event relation generation unit 13 sets a rhetorical relationship between the parent node and the target node (S107). Specifically, the rhetorical relationship between the siblings mentioned in step S105 (the rhetorical relationship between the satellite and the core) is set as the parent-child relationship.

[0026] For example, in Figure 5, arrows indicate the nodes that are traversed when event E3 is the target node in the rhetorical structure tree shown in Figure 4. As a result, event E5 is identified as the parent node. This parent-child relationship is set to Background, which is the rhetorical relationship between the sibling nodes and the core node (between the dashed arrows in Figure 5).

[0027] On the other hand, if no satellite node is found up to the root (No in S104), the event relation generation unit 13 traces only the core nodes from the root toward the leaves (S108). At this time, if there are multiple core nodes, the event relation generation unit 13 traces the rightmost core node. If the leaf node reached as a result is the target node (Yes in S109), the event relation generation unit 13 sets the event related to the target node as the root node of the discourse dependency structure tree (S110). If the reached leaf node is different from the target node (No in S109), the event relation generation unit 13 executes step S106 and onwards. Therefore, the reached leaf node becomes the parent of the target node. However, in step S107, the event relationship generation unit 13 identifies the closest common ancestor node between the reached leaf node and the target node in the rhetorical structure tree, and sets a rhetorical relationship from the ancestor node of the target node among the child nodes of the ancestor node to the ancestor node of the reached leaf node for the parent-child relationship between the reached leaf node and the target node.

[0028] Figure 6 shows an example of setting parent-child relationships in this case. For convenience, a different rhetorical structure tree is shown in Figure 6 than in Figures 4 and 5. When Event 4 is the target node, no child node is found by tracing back toward the root, so Event 6 is reached by tracing the core node on the right end from the root toward the leaves. Therefore, Event 6 becomes the parent of Event 4. In this case, the rhetorical relationship between the ancestor node of Event 6 and the ancestor node of Event 4 (indicated by the dashed arrow in Figure 6), which is the child node of the core node that is the closest common ancestor node to Event 6 and Event 4, is List, so List is set for this parent-child relationship.

[0029] When the loop process L1 is completed, the conversion to the discourse dependency structure tree is completed. When the loop process L1 is applied to the rhetorical structure tree shown in Figure 4, the discourse dependency structure tree shown in Figure 7 is obtained.

[0030] In step S111 and thereafter, the summary generation unit 14 generates a summary by pruning the branches of the discourse dependency structure tree (FIG. 7). That is, the summary in this embodiment is obtained in the form of a discourse dependency structure tree from which one or more branches have been pruned.

[0031] In step S111, the summary generation unit 14 extracts a frame corresponding to the median time from the frame sequence of each event corresponding to each node in the discourse dependency structure tree (FIG. 7). As described above, each event has a start time and an end time, so the frame corresponding to the median time can be identified. Note that it does not necessarily have to be the median frame; it is sufficient to extract any one frame.

[0032] Next, the summary generator 14 generates and assigns captions to each extracted frame using CLIP (Reference 2) or similar (S112). As a result, each node in the discourse dependency tree (Fig. 7) corresponds to a pair of an extracted frame (image) and caption (text). A caption is text that indicates the content of a still image, which is a frame. Note that captions may be generated not for a single frame (still image) but for an event (i.e., a sequence of frames). In this case, for example, VideoCLIP (Reference 3) or similar may be used.

[0033] Next, the summary generation unit 14 executes a process of extracting rooted subtrees that satisfy the constraints for a summary from the discourse dependency structure tree (S113). A rooted subtree is a subtree that includes the root node. As described above, each node in the discourse dependency structure tree corresponds to a pair of a frame (image) and a caption (text), and therefore the subtree corresponds to a multimodal summary that includes key frames (images) and their captions. The constraints for a summary are, for example, length constraints (number of frames, number of text characters, number of words), which are given in advance. The extraction of rooted subtrees can be likened to pruning the branches of a discourse dependency structure tree.

[0034] For example, the summary generation unit 14 may traverse the nodes of the discourse dependency structure tree in an appropriate order (for example, depth-first or breadth-first search order) and extract a rooted subtree including the nodes visited before the constraint is exceeded. Whether the constraint is satisfied can be determined by comparing the number of nodes visited with the constraint if the constraint is the number of frames, or by comparing the total number of characters or words in the caption corresponding to each node visited with the constraint if the constraint is the number of characters or words.

[0035] Alternatively, the summary generator 14 may prune to maximize a function such as:

[0036]

number

[0037] where T is a discourse dependency tree, rs is all possible (i.e., all possible) rooted subtrees of T, and α is a hyperparameter that adjusts the importance of the two terms. f(x i ) is the score that represents the importance of the node, and sim(x i ,x j) is a score that represents the similarity between two nodes. f() can be determined only from the image (frame) (for example, by applying object detection to the image = frame and evaluating the importance based on whether any important objects are included), or it can be determined only from the text (caption) (for example, by determining the importance of words in the text using tfidf or the like and evaluating the importance of the text based on that), or it can be a weighted average of both. sim(x i ,x j ) can be determined using either or both of the two modalities as a weighted average. Image similarity can be evaluated using, for example, LPIPS (https: / / arxiv.org / abs / 1801.03924). Text similarity can be evaluated by converting sentences (text) into vectors using sentence-BERT and calculating the cosine similarity between the vectors.

[0038] Length() represents either the number of frames, characters, or words contained in the summary, i.e., the rooted subtree, depending on the measure of L. L represents the length of the summary (number of frames, characters, or words).

[0039] The above function means finding the node set that maximizes the value obtained by subtracting the sum of the similarities between nodes x α from the sum of the importance of each node. By subtracting the sum of the similarities x α (using the sum of the similarities x α as a penalty), it is possible to avoid extracting only events with high similarity. As a result, it is possible to reduce the redundancy of the summary.

[0040] The summary generation unit 14 outputs the extracted rooted subtree as a multimodal summary of the target video.

[0041] The above function maximization problem can be solved by, for example, a greedy algorithm. The processing procedure in this case will be explained below.

[0042] FIG. 8 is a flowchart illustrating an example of a processing procedure for extracting a rooted subtree that satisfies the constraints as a summary from a discourse dependency structure tree.

[0043] In step S201, the summary generation unit 14 stores the root node of the discourse dependency structure tree (FIG. 7) in a variable S. S is a variable for storing a group of nodes that constitute a rooted subtree that will ultimately become the summary, and has, for example, a list-type structure.

[0044] Next, the summary generation unit 14 stores all child nodes of the root node in a variable C (S202). The variable C is a variable for storing nodes that are candidates for addition to S, and has, for example, a list structure. One element of C corresponds to one node.

[0045] Next, the summary generator 14 selects c elements of C to be added to S based on the following formula: * is determined (S203).

[0046]

number

[0047] The meanings of f() and sim() are as described above.

[0048] Next, the summary generation unit 14 * When added to S, it is determined whether the set of elements stored in S satisfies the constraints of the summary (S204).

[0049] c * If the set of elements stored in S satisfies the constraints of the summary when added to S (Yes in S204), the summary generation unit 14 * Add to S and c * is deleted from C (S205).

[0050] Next, the summary generation unit 14 *It is determined whether or not there is a child node of c in the discourse dependency structure tree (FIG. 7) (S206). If there is a child node of c (Yes in S206), the summary generation unit 14 * All child nodes of are added to C (S207), and step S203 and subsequent steps are repeated.

[0051] On the other hand, c * If adding c to S, the set of elements stored in S will no longer satisfy the summary constraint (No in S204), or * If there are no child nodes in S (No in S206), the summary generation unit 14 outputs a subtree (rooted subtree) consisting of each node stored in S at that time as a multimodal summary of the target video (S208).

[0052] As described above, according to this embodiment, a video is represented as a discourse dependency structure tree that represents the dependency relationships between events, and a multimodal summary is generated by extracting frames and captions corresponding to events as subtrees obtained by pruning the tree. This makes it possible to generate a summary that can express the causal relationships between events. Therefore, it is possible to generate a summary that maintains the causal relationships between multiple images. As a result, compared to summarization methods that generate summaries without considering the relationships between events, it is possible to reduce the possibility of users misunderstanding the content of the source video.

[0053] Although the above description has been given of an example of summarizing a moving image, this embodiment can also be applied to cases where a plurality of images (still images) relating to events related in time series are to be summarized.

[0054] [Reference 1]Humam Alwassel, Silvio Giancola, Bernard Ghanem, "TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks", In Proc. of ICCV workshop [Reference 2] Alec Radford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, "Learning Transferable Visual Models From Natural Language Supervision", In Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021 [Reference 3] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer, "VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Un-derstanding", In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing The following additional notes are provided regarding the above-described embodiments.

[0055] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: generating a rhetorical structure tree for a plurality of images relating to a time-series related event; generating a tree structure based on the rhetorical structure tree, the tree structure representing relationships between events related to the images, with the images as nodes; extracting a subtree from the tree structure so as to satisfy a constraint on summarizing the plurality of images; A summarization device characterized by:

[0056] (Additional note 2) generating a rhetorical structure tree for a plurality of images relating to a time-series related event; generating a tree structure based on the rhetorical structure tree, the tree structure representing relationships between events related to the images, with the images as nodes; extracting a subtree from the tree structure so as to satisfy a constraint on summarizing the plurality of images; A recording medium that records a program that causes a computer to execute a process.

[0057] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0058] 10 Summary Device 11 Division 12 Rhetorical structure tree generator 13 Event Relation Generation Unit 14 Summary generator 100 Drive device 101 Recording media 102 Auxiliary storage 103 Memory Device 104 processors 105 Interface Device B Bus

Claims

1. a rhetorical structure tree generator configured to generate a rhetorical structure tree for a plurality of images relating to a time-series related event; an event relation generation unit configured to generate a tree structure based on the rhetorical structure tree, the tree structure representing relations between events related to the images, with the images as nodes; a summary generator configured to extract subtrees from the tree structure so as to satisfy constraints on summaries of the plurality of images; A summarization device comprising:

2. the summary generation unit is configured to assign text indicating the content of the image related to each node of the tree structure to the node, and to extract the subtree such that the image or the text corresponding to each node satisfies the constraint.

2. The summarization device according to claim 1.

3. the rhetorical structure tree generation unit is configured to generate the rhetorical structure tree in which a frame sequence for each event included in a video having the images as frames corresponds to a leaf node; the event relation generation unit is configured to generate a tree structure representing relations between events related to the images, with the frame sequence as a node; the summary generation unit is configured to extract a subtree from the tree structure by associating one frame from the frame sequence corresponding to each node of the tree structure with the node, so as to satisfy a constraint on the summary of the video.

3. The summarization device according to claim 1 or 2.

4. the tree structure is a discourse dependency structure tree; 2. The summarization device according to claim 1.

5. a rhetorical structure tree generation procedure for generating a rhetorical structure tree for a plurality of images relating to events that are related in time series; an event relation generation step of generating a tree structure based on the rhetorical structure tree, the tree structure representing relations between events related to the images, with the images as nodes; a summary generation step of extracting subtrees from the tree structure so as to satisfy constraints on summaries of the plurality of images; A summarization method characterized by being executed by a computer.

6. A program for causing a computer to function as the summarizing device according to claim 1.

Images