Apparatus and method for constructing a transformer model for video story question answering

Abstract

To provide a device, a method and a program for constructing a transformer model for video story question answering, which learn a video story by considering the context of prior and subsequent video clips included in video data.SOLUTION: The device comprises: an input-output unit for receiving video data, which includes a plurality of consecutive video clips, and question data for video question answering, and outputting the result from video story question answering; a storage unit in which a program and data for performing the video story question answering are stored; and a control unit that includes at least one processor and constructs a transformer model for the video story question answering by executing the program. The control unit learns a video story from the video data including the plurality of consecutive video clips by considering the context of neighboring prior and subsequent video clips according to chronological order.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The embodiments disclosed in this specification relate to an apparatus and method for constructing a Transformer model for video story question answering, and more particularly, to an apparatus and method for constructing a Transformer model for video story question answering that learns video stories by taking into account the context of video clips included in video data.

[0002] This research was carried out as a result of research into the "Development of question answering technology based on video story understanding that can pass the video Turing test (IITP-2017-0-01772-005)" project of the Innovative Growth Engine Project of the Ministry of Science and ICT and the Institute of Information and Communications Technology Planning (IITP) and the "Development of cognitive agent SW based on everyday life learning (SW Starlab) (IITP-2015-0-00310-007)" project of the Software Computing Industry Source Technology Development Project. [Background technology]

[0003] Recently, interest in multimodal data has grown thanks to the development of deep learning techniques for vision and natural language processing, and many forms of tasks that measure understanding of video data have come into focus.

[0004] Among the various types of tasks, video question answering measures video comprehension ability based on the accuracy of multiple-choice objective questions in natural language format. In particular, to solve video question answering, it is necessary to learn the complex correlations of various data appearing in multimodal videos and find the core information about the given question and answer.

[0005] Recently, to solve this problem, a model that performs large-scale learning based on a transformer has been introduced. Large-scale dictionary learning for videos has demonstrated considerable performance in various tasks for evaluating video comprehension, and is built on a model that has demonstrated good performance in natural language processing. However, in the case of a transformer that performs video story question answering according to the conventional technology, there was a problem that it could only be processed for short videos, since the computational cost increases exponentially as the length of the video increases.

[0006] On the other hand, the above-mentioned background art is technical information possessed by the inventor for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be publicly known art that was disclosed to the general public prior to the filing of the present invention. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent Publication No. 10-2020-0144417 Summary of the Invention [Problem to be solved by the invention]

[0008] The embodiments disclosed in this specification aim to provide an apparatus and method for constructing a Transformer model for video story question answering that learns video stories taking into account the context of video clips contained in video data.

[0009] Other objects and advantages of the present invention can be understood from the following description and will become more apparent from the following embodiments. Also, it will be easily understood that the objects and advantages of the present invention can be realized by the means and combinations thereof as claimed in the claims. [Means for solving the problem]

[0010] As a technical means for achieving the above-mentioned technical objectives, an apparatus for constructing a Transformer model for video story question answering includes an input / output unit for receiving video data including a plurality of consecutive video clips and question data for video question answering, and outputting results of video story question answering, a storage unit for storing a program and data for performing video story question answering, and a control unit including at least one processor and constructing a Transformer model for video story question answering by executing the program, wherein the control unit learns a video story from the video data including the plurality of consecutive video clips by taking into account the context of adjacent video clips in chronological order.

[0011] According to another embodiment, a method for constructing a Transformer model for video story question answering performed by an apparatus for constructing a Transformer model for video story question answering includes a step of receiving video data including a plurality of consecutive video clips and question data for video question answering, and a step of learning a video story from the video data including the plurality of consecutive video clips by taking into account the context of adjacent previous and next video clips in chronological order.

[0012] According to yet another embodiment, the recording medium is a computer-readable recording medium having a program recorded thereon for executing a method for constructing a Transformer model for video story question answering. The method for constructing a Transformer model for video story question answering performed by an apparatus for constructing a Transformer model for video story question answering includes receiving video data including a plurality of continuous video clips and question data for video question answering, and learning a video story from the video data including the plurality of continuous video clips by taking into account the context of adjacent previous and next video clips in a time order.

[0013] According to yet another embodiment, the computer program is a computer program recorded on a recording medium for being executed by an apparatus for constructing a Transformer model for video story question answering and for performing a method for constructing a Transformer model for video story question answering. The method for constructing a Transformer model for video story question answering performed by the apparatus for constructing a Transformer model for video story question answering includes receiving video data including a plurality of continuous video clips and question data for video question answering, and learning a video story from the video data including the plurality of continuous video clips by taking into account the context of adjacent previous and next video clips according to a time order. Effect of the Invention

[0014] According to any one of the above-mentioned problem solving means, it is possible to construct a Transformer that takes into account the context before and after a video clip contained in video data, thereby performing video story question answering, and to effectively process long videos without incurring large computational costs.

[0015] According to another of the above-mentioned problem-solving means, by constructing a Transformer that takes into account the context before and after a video clip contained in video data, it is possible to utilize the Transformer in a variety of fields, such as not only video story question answering but also prediction of subsequent scenes in a video and inference of causal relationships.

[0016] The effects obtained by the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the disclosed embodiments pertain in the following description.

[0017] The following attached drawings illustrate preferred embodiments disclosed in this specification and serve to further understand the technical ideas disclosed in this specification, together with specific content for implementing the invention, and therefore the content disclosed in this specification should not be interpreted as being limited to only the matters shown in the drawings. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram for explaining a transformer model according to the prior art. [Diagram 2] FIG. 1 is a diagram for explaining a transformer model according to an embodiment. [Diagram 3] FIG. 2 is a functional block diagram of an apparatus for constructing a Transformer model for video story question answering according to one embodiment. [Figure 4] 1 is a flow chart illustrating a method for building a Transformer model for video story question answering according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various embodiments will be described in detail below with reference to the accompanying drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters that are well known to those skilled in the art to which the following embodiments belong will be omitted. Also, parts of the drawings that are not related to the description of the embodiments will be omitted, and similar parts are designated by similar reference numerals throughout the specification.

[0020] Throughout the specification, when a certain component is said to be connected to another component, this includes not only the case where the component is directly connected to another component, but also the case where the component is connected via an intermediate component. Furthermore, when a certain component includes another component, unless otherwise specified, it does not mean that the other component is excluded, but that the other component can also be included.

[0021] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0022] FIG. 1 is a diagram for explaining a transformer model according to the prior art.

[0023] FIG. 1 shows a conventional Transformer model, which illustrates the structure of a Transformer model used in video representation learning. Here, the Transformer model may be a Vanilla Transformer. Meanwhile, the Transformer shown in FIG. 1 may be configured such that the encoder 100 separates every video frame into layers. Here, in the Transformer shown in FIG. 1, each section S 1 , S 2 , S 3 The encoder 100 separated for each interval S may be a temporal transformer. The transformer model shown in FIG. 1 can be used to perform video story question answering. The transformer shown in FIG. 1 , S 2 , S 3 In contrast, the separated encoder 100 does not consider the context before and after the video clips included in the input video data, and therefore, as the length of the video increases, a problem occurs in which the computation cost increases exponentially, and therefore, it has only been used for video story question answering having a short length. Therefore, a transformer that can process long videos more effectively is needed, and therefore a transformer that considers the context before and after the video clips included in the video data has been constructed. The transformer that considers the context before and after the video clips included in the video data according to an embodiment will be described in more detail below with reference to FIGs. 2 and 3.

[0024] FIG. 2 is a diagram for explaining a transformer model according to an embodiment.

[0025] FIG. 2 illustrates a Transformer model according to an embodiment, showing the structure of a Transformer model used for video representation learning. Here, the Transformer model may be a Contextual Transformer. Meanwhile, the Transformer shown in FIG. 2 may be configured such that the encoder 200 separates every video frame for each layer. The Transformer shown in FIG. 2 divides each interval S 1 , S 2 , S 3 The separated encoder 200 learns the video story by considering the context before and after the video clips included in the input video data, so that the number of video clips in the preceding and following sections that can be considered may change as the hierarchy increases. Here, a video clip may mean a short recorded video. For example, S 1 and S 3 In the interval, two intervals can be considered, and the second hierarchical S 2 Three intervals can be considered. Here, the video data can include multiple consecutive video clips, and the video clips can include multiple visual tokens and text tokens. Meanwhile, in the contextual transformer shown in FIG. 2, each interval S 1 , S 2 , S 3 The encoder 200 separated for each section S may be a cross-modal transformer. 1 , S 2 , S 3 As input, a transformer such as the one shown in Figure 2 can take visual and text tokens corresponding to each interval S 1 , S 2 , S 3Since the separated encoder 100 learns a video story by considering the context before and after the video clips included in the input video data, it is possible to effectively process a long video without incurring a large computational cost. In addition, by constructing a transformer that considers the context before and after the video clips included in the video data, it is possible to utilize the transformer in various fields such as prediction of subsequent scenes of a video, inference of causal relationships, etc., in addition to video story question answering.

[0026] On the other hand, the Transformer of FIG. 2 described above can be constructed by an apparatus for constructing a Transformer model for video story question answering shown in FIG.

[0027] FIG. 3 is a functional block diagram of an apparatus for constructing a Transformer model for video story question answering according to one embodiment.

[0028] Referring to FIG. 3, an apparatus 300 for constructing a Transformer model for video story question answering according to an embodiment includes an input / output unit 310, a storage unit 320, and a control unit 330.

[0029] The input / output unit 310 may include an input unit for receiving an input from a user, and an output unit for displaying information such as a result of task execution or a state of the device 300 for constructing a Transformer model for video story question answering. That is, the input / output unit 310 is configured to receive video data including a plurality of continuous video clips and question data for video question answering, and output a result of the video story question answering. Here, the video clip may include a plurality of visual tokens and text tokens.

[0030] The storage unit 320 is configured to store files and programs and may be configured with various types of memory. In particular, the storage unit 320 may store data and programs that enable the control unit 330 described below to construct a Transformer model for video story question answering according to an algorithm presented below.

[0031] The control unit 330 includes at least one processor such as a CPU, a GPU, an Arduino, etc., and can control the overall operation of the apparatus 300 for constructing a Transformer model for video story question answering. That is, the control unit 330 can control other components included in the apparatus 300 for constructing a Transformer model for video story question answering to perform video story question answering. The control unit 330 can perform a calculation for constructing a Transformer model for video story question answering according to an algorithm presented below by executing a program stored in the storage unit 320. A method in which the control unit 330 performs a calculation for constructing a Transformer model for video story question answering will be described later.

[0032] Hereinafter, a process in which the control unit 330 executes a program stored in the storage unit 320 to perform a method of constructing a Transformer model for video story question answering according to an embodiment will be described in detail.

[0033] JPEG0007669335000001.jpg80166

[0034]

number

[0035] JPEG0007669335000003.jpg55168

[0036]

number

[0037] According to one embodiment, the controller 330 can construct a Transformer model for video story question answering using Equation 2 above.

[0038] The control unit 330 receives visual tokens and text tokens included in each video clip for each video clip corresponding to a predetermined section through each separated encoder, and outputs lower layers of adjacent video clips. The hidden representation of each layer is calculated, and the hidden representation is used to calculate a representation of the video data that takes into account the context. story Here, the control unit 330 may learn a temporal order by using a masked modality model (hereinafter, referred to as MMM) for each video clip. The masked modality model (MMM) may be an extension of a token-based masking technique proposed in an existing model, the masked language model, to masking of all tokens in a predetermined section. The masked modality model (MMM) allows one modality to be generated from another modality, and may prevent an encoder from generating a masked token too easily from surrounding tokens, and may learn alignment between modalities. Here, the modality may be video, text, etc. Therefore, by performing the above-mentioned learning using a contextual transformer according to one embodiment, it is possible to predict the content of a segment (e.g., video data separated into sections) based on the surrounding context, thereby learning a natural flow of speech.

[0039] Meanwhile, the Masked Modality Model can be trained by Negative Contrastive Learning. Here, the Masked Modality Model can be expressed as the following Equation 3.

[0040]

number

[0041] JPEG0007669335000006.jpg31165

[0042] FIG. 4 is a flow chart illustrating a method for building a Transformer model for video story question answering according to one embodiment.

[0043] The method for constructing a Transformer model for video story question answering according to the embodiment shown in Figure 4 includes a step of time-sequential processing by the device 100 for constructing a Transformer model for video story question answering shown in Figures 2 and 3. Therefore, although the contents are omitted below, the contents described above regarding the device 100 for constructing a Transformer model for video story question answering shown in Figures 2 and 3 can also be applied to the method for constructing a Transformer model for video story question answering according to the embodiment shown in Figure 4.

[0044] 4, the apparatus 100 for constructing a Transformer model for video story question answering may receive video data including a plurality of continuous video clips and question data for video question answering (S410), where the video clip may include a plurality of visual tokens and text tokens.

[0045] The apparatus 100 for constructing a Transformer model for video story question answering can learn a video story from video data including a plurality of consecutive video clips received in step S410, taking into consideration the context of adjacent video clips in a time order (S420). The apparatus 100 for constructing a Transformer model for video story question answering receives as input visual tokens and text tokens included in each video clip for each of the video clips corresponding to a preset section through each separate encoder, calculates hidden representations of lower layers of the adjacent video clips, and calculates a representation of the video data taking into consideration the context of the video clips using the calculated hidden representations, thereby learning a video story. storyHere, the apparatus 100 for constructing a Transformer model for video story question answering can learn a temporal order by using a Masked Modality Model for each video clip. The Masked Modality Model (MMM) may be an extension of a token-based masking technique proposed in the Masked Language Model, which is an existing model, to masking of all tokens in a predetermined section. The Masked Modality Model (MMM) allows one modality to be generated from another modality, and can prevent an encoder from generating a masked token too easily from surrounding tokens, and can learn alignment between modalities. Meanwhile, the Masked Modality Model can be learned by negative contrastive learning. Here, the masked modality model can be expressed as in Equation 3 above.

[0046] The term 'module' used in the above embodiments means software or hardware components such as FPGA (field programmable gate array) or ASIC, and 'module' performs a certain function. However, 'module' is not limited to software or hardware. 'module' may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, 'module' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0047] The functionality provided within the components and 'units' may be combined into fewer components and 'units' or separated out into additional components and 'units'.

[0048] Furthermore, the components and 'parts' may be implemented to implement one or more CPUs within a device or a secure multimedia card.

[0049] Meanwhile, the method of constructing a Transformer model for video story question answering according to an embodiment described in this specification may also be embodied in the form of a computer-readable medium storing instructions and data executable by a computer. Here, the instructions and data may be stored in the form of a program code, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. In addition, the computer-readable medium may be any available medium accessible by a computer, including both volatile and non-volatile media, and both separate and non-separate media. In addition, the computer-readable medium may be a computer recording medium. The computer recording medium may include both volatile and non-volatile, separate and non-separate media embodied by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as a HDD or SSD, an optical storage medium such as a CD, a DVD, or a Blu-ray disc, or a memory included in a server accessible via a network.

[0050] In addition, the method for constructing a Transformer model for video story question answering according to an embodiment described in this specification may be embodied in a computer program (or a computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be embodied in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like. In addition, the computer program may be recorded in any type of computer-readable recording medium (e.g., a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD), etc.).

[0051] Thus, the method for constructing a Transformer model for video story question answering according to one embodiment described herein may be implemented by a computing device running a computer program as described above. The computing device may include at least a processor, a memory, a storage device, a high-speed interface connecting to the memory and a high-speed expansion port, and a low-speed interface connecting to a low-speed bus and the storage device. Each of these components is connected to each other using various buses and may be mounted on a common motherboard or in any other suitable manner.

[0052] Here, the processor may process instructions within a computing device. Such instructions may include instructions stored in a memory or storage device for displaying graphic information for providing a Graphic User Interface (GUI) on an external input and output device, such as a display connected to a high speed interface. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. The processor may also be implemented as a chipset, which is made up of chips including multiple independent analog and / or digital processors.

[0053] Also, memory stores information within a computing device. As an example, memory may be comprised of a volatile memory unit or a collection thereof. As another example, memory may be comprised of a non-volatile memory unit or a collection thereof. Memory may also be in other forms of computer readable media, such as, for example, a magnetic or optical disk.

[0054] A storage device can provide a large amount of storage space to a computing device. A storage device can be a computer-readable medium or a configuration that includes such a medium, such as a device in a Storage Area Network (SAN) or other configuration, and can be a floppy disk drive, hard disk drive, optical disk drive, or tape drive, flash memory, or other similar semiconductor memory device or device array.

[0055] The above-described embodiments are merely illustrative, and those skilled in the art will understand that the above-described embodiments can be easily modified into other specific forms without changing the technical ideas or essential features of the above-described embodiments. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described as a single type may be implemented in a distributed form, and similarly, each component described as a distributed type may be implemented in a combined form.

[0056] The scope of protection sought by this specification is determined by the claims set forth below rather than the above detailed description, and should be interpreted to include all modifications and variations derived from the meaning and scope of the claims and their equivalent concepts. [Explanation of symbols]

[0057] 300 A device for building transformer models for video story question answering 310 Input / output section 320 Preservation Department 330 Control Unit

Claims

1. an input / output unit for receiving video data including a plurality of continuous video clips and question data for video question answering, and outputting a result of the video story question answering; a storage unit for storing a program and data for performing video story question answering; A control unit including at least one processor and executing the program to construct a Transformer model for video story question answering; The control unit is A device for constructing a Transformer model for video story question answering, comprising: a device for learning a video story from video data including the plurality of consecutive video clips, taking into account the context of adjacent video clips in a chronological order; The video clip is a plurality of visual tokens and a plurality of text tokens; The control unit is A video story is learned by separating each of the layers of the Transformer model, setting each encoder to separate each of the layers of the Transformer model, and setting an encoder of a second layer section of the Transformer model to be connected to an encoder of an adjacent previous and next sections of a first layer of a lower layer, receiving visual tokens and text tokens included in each video clip for each of the video clips corresponding to a predetermined section through each encoder of the first layer section, calculating hidden representations of the adjacent previous and next video clips, and calculating a representation of video data taking into account a context of the previous and next scenes using the hidden representations of the adjacent previous and next lower layer sections calculated through each encoder of the second layer section, An apparatus for constructing a Transformer model for video story question answering, comprising: constructing the Transformer model; the Transformer model receiving question data for question answering; and outputting results of video story question answering.

2. The control unit is 2. The apparatus of claim 1, further comprising: learning a temporal order for each of the video clips using a masked modality model.

3. The apparatus for constructing a Transformer model for video story question answering according to claim 2, wherein the Masked Modality Model is trained by Negative Contrastive Learning.

4. A method for constructing a Transformer model for video story question answering, which is performed by an apparatus for constructing a Transformer model for video story question answering, comprising: receiving video data including a plurality of consecutive video clips and question data for video question answering; and learning a video story from video data including the plurality of consecutive video clips, taking into account the context of adjacent video clips in a time order, the video story being learned from the video data including the plurality of consecutive video clips, the method comprising: The video clip is a plurality of visual tokens and a plurality of text tokens; The step of learning the video story includes: A video story is learned by separating each of the layers of the Transformer model, setting each encoder to separate each of the layers of the Transformer model, and setting an encoder of a second layer section of the Transformer model to be connected to an encoder of an adjacent previous and next sections of a first layer of a lower layer, receiving visual tokens and text tokens included in each video clip for each of the video clips corresponding to a preset section through each encoder of the first layer section, calculating hidden representations of the adjacent previous and next video clips, and calculating a representation of video data taking into account a context of the previous and next scenes using the hidden representations of the adjacent previous and next lower layer sections calculated through each encoder of the second layer section, A method for constructing a Transformer model for video story question answering, comprising: constructing the Transformer model, the Transformer model receiving question data for question answering, and outputting a result of video story question answering.

5. The step of learning the video story includes:

5. The method of claim 4, further comprising: learning a temporal order for each of the video clips using a masked modality model.

6. The method of constructing a Transformer model for video story question answering according to claim 5, wherein the Masked Modality Model is trained by Negative Contrastive Learning.

7. A computer-readable recording medium having recorded thereon a program for causing said apparatus to execute the method according to claim 4.

8. A computer program recorded on a recording medium for performing the method of claim 4 when executed by an apparatus for constructing a Transformer model for video story question answering.

Images