Electronic device for multi-modal time axis fusion artificial intelligence model and method for operating same

Abstract

Disclosed is an electronic device for a multi-modal time axis fusion artificial intelligence model. The electronic device comprises: a storage unit; and a processor, wherein the processor may obtain a plurality of first visual features respectively corresponding to a plurality of different time points or time intervals from a video, obtain a plurality of text features respectively corresponding to the plurality of time points or time intervals from a text, obtain a plurality of first local fusion features respectively corresponding to the plurality of time points or time intervals from the plurality of first visual features and the plurality of text features by fusing together the first visual features and the text features corresponding to the same time point or time interval, and obtain at least one global fusion feature from the plurality of first local fusion features.

Description

Electronic device and operating method for a multi-modal time axis fusion artificial intelligence model

[0001] The present invention relates to an electronic device and a method of operating the same. More specifically, the present invention relates to an electronic device and a method of operating the same for a multi-modal time-axis fusion artificial intelligence model.

[0002] Research on multimodal AI models capable of comprehensively utilizing diverse inputs is actively underway. Recently, research on video-text multimodal AI models has garnered significant attention, extending beyond still image-text multimodal AI models. Existing video-text multimodal AI models typically extract summarized visual vectors from videos, extract text vectors from text, and fuse these summarized visual and text vectors. This approach limits the performance of existing video-text multimodal AI models, as visual and text features cannot be precisely fused across the time axis.

[0003] The problem to be solved by the present invention is to provide an electronic device and an operating method thereof for a video-text multi-modal artificial intelligence model with improved performance.

[0004] According to one embodiment of the present invention, an electronic device for a multi-modal time axis fusion artificial intelligence model includes a storage unit and a processor, wherein the processor obtains a plurality of first visual features respectively corresponding to a plurality of different time points or time periods from a video, obtains a plurality of text features respectively corresponding to the plurality of time points or time periods from a text, and obtains a plurality of first local fusion features respectively corresponding to the plurality of time points or time periods from the plurality of first visual features and the plurality of text features by fusion of the first visual features and the text features corresponding to the same time points or time periods, and obtains at least one global fusion feature from the plurality of first local fusion features.

[0005] According to one embodiment of the present invention, an operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model may include the steps of: obtaining a plurality of first visual features each corresponding to a plurality of different time points or time periods from a video; obtaining a plurality of text features each corresponding to the plurality of time points or time periods from a text; obtaining a plurality of first local fusion features each corresponding to the plurality of time points or time periods from the plurality of first visual features and the plurality of text features by fusing the first visual features and text features corresponding to the same time points or time periods; and obtaining at least one global fusion feature from the plurality of first local fusion features.

[0006] According to one embodiment of the present invention, the present invention can improve the performance of a video-text multi-modal artificial intelligence model by enabling more fine-grained and complete fusion of visual features and text features by extending text features to the time axis, locally fusion of visual features and text features at each time point or time interval, and globally fusion of local fusion features to the time axis across multiple time points or time intervals, i.e., performing multi-level time axis fusion.

[0007] FIG. 1 is a block diagram of an electronic device for a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention.

[0008] FIG. 2 is a conceptual diagram of a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention.

[0009] FIG. 3 is a conceptual diagram of a text feature extraction module used in the electronic device according to one embodiment of the present invention.

[0010] FIG. 4 is a conceptual diagram of a first local fusion module used in the electronic device according to one embodiment of the present invention.

[0011] FIG. 5 is a conceptual diagram of a first visual feature extraction module used in the electronic device according to one embodiment of the present invention.

[0012] FIG. 6 is a conceptual diagram of an object graph extraction module used in the electronic device according to one embodiment of the present invention.

[0013] Figure 7 is a conceptual diagram of a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention.

[0014] FIGS. 8 and 9 are conceptual diagrams illustrating a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention.

[0015] Figures 10 to 20 are flowcharts of an operating method of the electronic device according to one embodiment of the present invention.

[0016] [Explanation of symbols]

[0017] 100: Electronic devices

[0018] 110: Storage

[0019] 120: Processor

[0020] MD, MD': Artificial Intelligence Model

[0021] MD1a, MD1b: Visual Feature Extraction Module

[0022] MD2: Text Feature Extraction Module

[0023] MD3, MD6: Local fusion modules

[0024] MD4, MD4': Global fusion module

[0025] MD7: Summary Vector Extraction Module

[0026] MD8: Choice Feature Extraction Module

[0027] MD9: Classification Module

[0028]

[0029] Hereinafter, the operating principles of preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Furthermore, when describing embodiments of the invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, the terms used below are defined based on their functions in the present invention and may vary depending on the intent or custom of the user or operator. Therefore, the definitions of the terms used should be interpreted based on the contents and corresponding functions throughout this specification.

[0030] FIG. 1 is a block diagram of an electronic device for a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention. Referring to FIG. 1, the electronic device (100) may include a storage unit (110) and a processor (120).

[0031] The storage unit (110) can store various data and programs. For example, the storage unit (110) can store a multi-modal time axis fusion artificial intelligence model (MD, MD', see FIGS. 2 and 7, respectively). In one embodiment, the storage unit (110) can store video data and text data input to the artificial intelligence model. The storage unit (110) can include at least one of volatile memory and non-volatile memory. For example, the volatile memory can include DRAM, SRAM, SDRAM, DDR SDRAM, FeRAM, MRAM, PRAM, PoRAM, or ReRAM. For example, the non-volatile memory can include flash memory, mask ROM, PROM, OTPROM, EPROM, EEPROM, a hard disk, or an optical disk.

[0032] The processor (120) can control the overall operation of the electronic device (100). The processor (120) can control the storage unit (110). For example, the processor (120) can include at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0033] In one embodiment, the processor (120) may obtain a plurality of first visual features corresponding to a plurality of different time points or time periods from a video, obtain a plurality of text features corresponding to a plurality of time points or time periods from a text, and fuse the first visual features and text features corresponding to the same time points or time periods to obtain a plurality of first local fusion features corresponding to a plurality of time points or time periods from the plurality of first visual features and the plurality of text features, and obtain at least one global fusion feature from the plurality of first local fusion features.

[0034] In one embodiment, the processor (120) may obtain a plurality of text features, each corresponding to a plurality of time points or time periods, from the text using a plurality of mapping layers, each corresponding to a plurality of time points or time periods, to obtain a plurality of text features.

[0035] In one embodiment, the processor (120) may obtain a plurality of first intermediate text features from a text, obtain a plurality of second intermediate text features corresponding to a plurality of time points or time periods from the plurality of first intermediate text features using a plurality of mapping layers each corresponding to a plurality of time points or time periods, and obtain a plurality of text features from the plurality of second intermediate text features by changing the dimension of the second intermediate text features.

[0036] In one embodiment, the number of the plurality of first intermediate text features may be different from the number of the plurality of second intermediate text features.

[0037] In one embodiment, the dimension of each of the plurality of text features may be equal to the dimension of each of the plurality of first visual features.

[0038] In one embodiment, the processor (120) may obtain first local fusion features from the first visual features and the text features using convolution to obtain a plurality of first local fusion features.

[0039] In one embodiment, the processor (120) may obtain a first intermediate local fusion feature from the first visual feature and the text feature, obtain a second intermediate local fusion feature from the first intermediate local fusion feature using convolution, and obtain the first local fusion feature from the first visual feature and the second intermediate local fusion feature to obtain a plurality of first local fusion features.

[0040] In one embodiment, the processor (120) may obtain a plurality of object features from a video, obtain a plurality of object graphs from the plurality of object features, and obtain a plurality of first visual features from the plurality of object graphs to obtain a plurality of first visual features.

[0041] In one embodiment, the processor (120) may obtain a plurality of second visual features corresponding to a plurality of time points or time periods from a video, and obtain a plurality of second local fusion features corresponding to a plurality of time points or time periods from the plurality of second visual features and the plurality of text features by fusing the second visual features and text features corresponding to the same time points or time periods, obtain a plurality of third local fusion features corresponding to a plurality of time points or time periods from the plurality of first local fusion features and the plurality of second local fusion features, and obtain at least one global fusion feature from the plurality of third local fusion features.

[0042] In one embodiment, the processor (120) may use self-attention to obtain at least one global fusion feature from a plurality of first local fusion features to obtain at least one global fusion feature.

[0043] In one embodiment, the text includes a question, and the processor (120) can obtain an answer to the question using at least one global fusion feature.

[0044] In one embodiment, at least one global fusion feature comprises a plurality of global fusion features, and the processor (120) can obtain a summary vector from the plurality of global fusion features, obtain a plurality of choice features from the plurality of choices, and obtain an answer to the question from the summary vector and each of the choice features, in order to obtain an answer to the question.

[0045] In one embodiment, the processor (120) can classify, search, predict, or generate a video using at least one global fusion feature.

[0046] Fig. 2 is a conceptual diagram of a multi-modal time axis fusion artificial intelligence model according to an embodiment of the present invention. Referring to Fig. 2, a multi-modal time axis fusion artificial intelligence model (MD) extracts a plurality of first visual features (V) corresponding to different multiple points in time or time intervals from a video. a1 Inland V an ) may include a first visual feature extraction module (MD1a) for obtaining a first visual feature extraction module (MD1a). In one embodiment, the first visual feature extraction module (MD1a) may include a first visual feature extraction module (MD1a) as illustrated in FIG. 5. The processor (120, see FIG. 1) may use the first visual feature extraction module (MD1a) to extract a plurality of first visual features (V) corresponding to different points in time or time periods from a video. a1 Inland V an ) can be obtained. Hereinafter, unless otherwise stated, the processor (120) means the processor (120) of FIG. 1.

[0047] In one embodiment, the first visual feature extraction module (MD1a) may include at least one of AlexNet (Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012), VGGNet (Simonyan et.al, Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015), GoogleNet (Szegedy et al., Going Deeper with Convolutions, CVPR 2015), ResNet (He et al., Deep Residual Learning for Image Recognition, CVPR 2016), and ViT (Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers or Image Recognition at Scale, ICLR 2021).

[0048] In one embodiment, the video may be an advertising video. In one embodiment, the video may include multiple frames, each corresponding to a different point in time, and each frame may be a still image.

[0049] In one embodiment, each first visual feature (V a1 Inland V an ) may be a vector, a matrix, or a tensor. In one embodiment, a plurality of first visual features (V a1 Inland V an ) may each correspond to a different plurality of points in time or time intervals. In one embodiment, each first visual feature (V a1 Inland V an ) may correspond to each frame (viewpoint). In another embodiment, each first visual feature (V a1 Inland V an) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of first visual features (V a1 Inland V an ) can correspond to each frame (viewpoint), and a plurality of first visual features (V a1 Inland V an ) each of which can correspond to a plurality of consecutive frames (time periods).

[0050] A multi-modal time axis fusion artificial intelligence model (MD) is a model that extracts multiple text features (L1 to L1) corresponding to different time points or time intervals from a text. n ) may further include a text feature extraction module (MD2) for obtaining the text feature. In one embodiment, the text feature extraction module (MD2) may include a text feature extraction module (MD2) as illustrated in FIG. 3. The processor (120) uses the text feature extraction module (MD2) to obtain a plurality of text features (L1 to L) corresponding to different times or time periods from the text. n ) can be obtained.

[0051] In one embodiment, the text may include at least one of a subtitle for the video, a comment for the video, a title for the video, a description for the video, and a video classification criterion.

[0052] In one embodiment, each text feature (L1 to L n ) can be a vector, a matrix, or a tensor. In one embodiment, each text feature (L1 to L n ) is the dimension of each first visual feature (V a1 Inland V an ) may be the same as the dimension of each text feature (L1 to L n ) is the number of real numbers that constitute each first visual feature (V a1 Inland V an ) may be equal to the number of real numbers that make up the whole number.

[0053] In one embodiment, multiple text features (L1 to L n ) may correspond to different multiple points in time or time intervals. In one embodiment, each text feature (L1 to L n ) can correspond to each frame (viewpoint). In another embodiment, each text feature (L1 to L n ) may correspond to each of a plurality of consecutive frames (time intervals). In another embodiment, a plurality of text features (L1 to L n ) can correspond to each frame (viewpoint), and multiple text features (L1 to L n ) each of which can correspond to a plurality of consecutive frames (time periods).

[0054] The multi-modal time axis fusion artificial intelligence model (MD) fuses multiple first visual features (V) by fusing the first visual features and text features corresponding to the same time point or time interval. a1 Inland V an ) and multiple text features (L1 to L n ) from a plurality of first local fusion features (T ) each corresponding to a plurality of different time points or time intervals. a1 Inland T an ) may further include a first local fusion module (MD3) for obtaining the first visual features (V). In one embodiment, the first local fusion module (MD3) may include a first local fusion module (MD3) as illustrated in FIG. 4. The processor (120) may fuse first visual features and text features corresponding to the same point in time or time period using the first local fusion module (MD3) to obtain a plurality of first visual features (V a1 Inland V an ) and multiple text features (L1 to L n ) from a plurality of first local fusion features (T ) each corresponding to a plurality of different time points or time intervals. a1 Inland T an) can be obtained. For example, the processor (120) may obtain a first visual feature (V) corresponding to a first time point. a1 ) and the text feature (L1) corresponding to the first time point are fused to form the first local fusion feature (T) corresponding to the first time point. a1 ) can be obtained.

[0055] In one embodiment, each first local fusion feature (T a1 Inland T an ) can be a vector, a matrix, or a tensor. In one embodiment, multiple local fusion features (T a1 Inland T an ) may correspond to different multiple points in time or time intervals, respectively. In one embodiment, each of the first local fusion features (T a1 Inland T an ) may correspond to each frame (viewpoint). In another embodiment, each first local fusion feature (T a1 Inland T an ) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of first local fusion features (T a1 Inland T an ) can correspond to each frame (viewpoint), and a plurality of first local fusion features (T a1 Inland T an ) each of which can correspond to a plurality of consecutive frames (time periods).

[0056] The multi-modal time axis fusion artificial intelligence model (MD) is a multi-first local fusion feature (T a1 Inland T an ) from at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) may further include a global fusion module (MD4) for obtaining a plurality of first local fusion features (T). The processor (120) may use the global fusion module (MD4) to obtain a plurality of first local fusion features (T a1 Inland T an) from at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) can be obtained.

[0057] In one embodiment, the global fusion module (MD4) may include a self-attention layer. The processor (120) may use self-attention to generate a plurality of first local fusion features (T a1 Inland T an ) from at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n) can be obtained. The self-attention can be at least one of narrow-sense self-attention (Vaswani et al., Attention is All you Need, NIPS 2017) and modified self-attention. The narrow-sense self-attention can include multi-head self-attention. For example, the global fusion module (MD4) can include at least one of a narrow-sense transformer (Vaswani et al., Attention is All you Need, NIPS 2017 or An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021) and a modified transformer. For example, the modified transformer may include at least one of the Swin transformer (Liu et al., Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021), Reformer (Kitaev et al., Reformer: The efficient transformer, ICLR 2020), Linformer (Wang et al., Linformer: Self-attention with linear complexity, arXiv 2020), Performer (Choromanski et al., Rethinking attention with performers, ICLR 2020), and Longformer (Beltagy et al., Longformer: The long-document transformer, arXiv 2020).

[0058] In one embodiment, each global fusion feature (G1 to G n ) can be a vector, a matrix, or a tensor. In one embodiment, a plurality of global fusion features (G1 to G n) can correspond to different multiple points in time or time intervals, respectively. In one embodiment, each global fusion feature (G1 to G n ) can correspond to each frame (viewpoint). In another embodiment, each global fusion feature (G1 to G n ) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of global fusion features (G1 to G n ) can correspond to each frame (viewpoint), and a plurality of global fusion features (G1 to G n ) each of which can correspond to a plurality of consecutive frames (time periods).

[0059] FIG. 3 is a conceptual diagram of a text feature extraction module used in the electronic device according to one embodiment of the present invention. Referring to FIG. 3, the text feature extraction module (MD2) includes a plurality of mapping layers (W1 to W) each corresponding to a plurality of different points in time or time periods. n ) may be included. The processor (120) may include a plurality of mapping layers (W1 to W) each corresponding to a plurality of different points in time or time periods. n ) to obtain multiple text features (L1 to L1) corresponding to different points in time or time intervals from the text. n ) can be obtained. In one embodiment, the plurality of mapping layers may include different plurality of matrices. The processor (120) may obtain a different mapping layer (W1 to W) for each point in time or time interval. n ) by using different text features (L1 to L) at each point in time or time interval. n ) can be obtained.

[0060] In one embodiment, the text feature extraction module (MD2) extracts a plurality of first intermediate text features (L'1 to L') from the text. m) may include a language model (MD2a) for obtaining the plurality of first intermediate text features (L'1 to L') from the text using the language model (MD2a). The processor (120) may m ) can be obtained. In one embodiment, a plurality of first intermediate text features (L'1 to L' m ) may each correspond to a token of the text. In one embodiment, each of the first intermediate text features (L'1 to L' m ) can be a vector, matrix, or tensor.

[0061] In one embodiment, the language model (MD2a) may include at least one of a Recurrent Neural Network (RNN)-based language model and a Transformer-based language model. The Recurrent Neural Network-based language model may include at least one of a narrow RNN, a Long Short Term Memory (LSTM), a Gated Recurrent Unit (GRU), and a Bidirectional-RNN (Bi-RNN). The Transformer-based language model may include, for example, at least one of BERT (Devlin et al., BERT: Pre-training of Deep Bidirectional Transformer for Language Understanding, ACL 2019) and RoBERTa (Liu et al., RoBERTa: A Robustly Optimized BERT Pretraining Approach, ICLR 2020).

[0062] In one embodiment, the text feature extraction module (MD2) extracts a plurality of first intermediate text features (L'1 to L' m ) and a plurality of second intermediate text features (L''1 to L'') corresponding to different points in time or time intervals, respectively. n ) to obtain multiple mapping layers (W1 to W n) can be included. Each mapping layer (W1 to W n ) can correspond to each point in time or time interval. The processor (120) may have a plurality of mapping layers (W1 to W) each corresponding to a plurality of different points in time or time intervals. n ) using multiple first intermediate text features (L'1 to L' m ) and a plurality of second intermediate text features (L''1 to L'') corresponding to different points in time or time intervals, respectively. n ) can be obtained. For example, the processor (120) may use a mapping layer (W1) corresponding to a first time point to obtain a plurality of first intermediate text features (L'1 to L') corresponding to a plurality of tokens of the text, respectively. m ) can obtain the second intermediate text feature (L''1) corresponding to the first time point. In one embodiment, the first intermediate text feature (L'1 to L' m ) is the number (m) of the second intermediate text features (L''1 to L'' n ) may be different from the number (n).

[0063] In one embodiment, each of the second intermediate text features (L''1 to L'' n ) can be a vector, a matrix, or a tensor. In one embodiment, a plurality of second intermediate text features (L''1 to L'' n ) may each correspond to a different plurality of points in time or time intervals. In one embodiment, each of the second intermediate text features (L''1 to L'' n ) may correspond to each frame (viewpoint). In another embodiment, each of the second intermediate text features (L''1 to L'' n ) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of second intermediate text features (L''1 to L'' n ) can correspond to each frame (viewpoint), and a plurality of second intermediate text features (L''1 to L'' n) each of which can correspond to a plurality of consecutive frames (time periods).

[0064] In one embodiment, the text feature extraction module (MD2) extracts second intermediate text features (L''1 to L'' n ) by changing the dimension of multiple second intermediate text features (L''1 to L'' n ) from multiple text features (L1 to L n ) may include a dimension change module (MD2b) to obtain the second intermediate text features (L''1 to L''). The processor (120) may use the dimension change module (MD2b) to obtain the second intermediate text features (L''1 to L'' n ) by changing the dimension of multiple second intermediate text features (L''1 to L'' n ) from multiple text features (L1 to L n ) can be obtained. For example, the processor (120) can obtain the text feature (L1) corresponding to the first time point from the second intermediate text feature (L''1) corresponding to the first time point using the dimension change module (MD2b). Each of the second intermediate text features (L''1 to L'' n ) and the number of real numbers that constitute each text feature (L1 to L n ) may be different. In one embodiment, each text feature (L1 to L n ) is the dimension of each first visual feature (V a1 Inland V an , see Fig. 2) may be the same as the dimension of each text feature (L1 to L n ) and the number of real numbers that constitute each first visual feature (V a1 Inland V an , see Fig. 2) may be the same number of real numbers.

[0065] FIG. 4 is a conceptual diagram of a first local fusion module used in the electronic device according to one embodiment of the present invention. Referring to FIG. 4, the first local fusion module (MD3) may include a convolutional network (MD3a). In one embodiment, the convolutional network (MD3a) may include a pooling layer, a 1x1 convolutional layer, and an activation layer. For example, the pooling layer may include at least one of a channel-wise average layer, a channel maximum (max) layer, and a channel minimum (min) layer. For example, the activation layer may include a sigmoid layer, a ReLU layer, or a tanh layer. The processor (120) uses convolution to generate a first visual feature (V). a ) and the first local fusion feature (T) from the text features (L) a ) can be obtained.

[0066] In one embodiment, the processor (120) comprises a first visual feature (V a ) and the first intermediate local fusion feature (T') from the text features (L). a ) can be obtained. In one embodiment, the first intermediate local fusion feature (T' a ) may be a vector, a matrix, or a tensor. For example, the processor (120) may generate a first visual feature (V a ) and the text features (L) to obtain the first intermediate local fusion feature (T' a ) can be obtained.

[0067] In one embodiment, the processor (120) uses a convolutional network (MD3a) to generate a first intermediate local fusion feature (T' a ) from the second intermediate local fusion feature (T'' a ) can be obtained. In one embodiment, the second intermediate local fusion feature (T'' a ) can be a vector, matrix, or tensor.

[0068] In one embodiment, the processor (120) comprises a first visual feature (V a ) and the second intermediate regional fusion feature (T'' a ) from the first local fusion feature (T a ) can be obtained. For example, the processor (120) may obtain a first visual feature (V a ) and the second intermediate regional fusion feature (T'' a ) by obtaining the dot product of the first local fusion feature (T a ) can be obtained.

[0069] FIG. 5 is a conceptual diagram of a first visual feature extraction module used in the electronic device according to an embodiment of the present invention. Referring to FIG. 5, the first visual feature extraction module (MD1a) extracts a plurality of object features (O1 to O) from a video. n ) may include an object feature extraction module (MD1a-1) for obtaining the object features. The processor (120) may extract a plurality of object features (O1 to O) from a video using the object feature extraction module (MD1a-1). n ) can be obtained. For example, the object feature extraction module (MD1a-1) may include at least one of Faster-RCNN (Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015) and YOLO (Redmon et al., You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016).

[0070] In one embodiment, a plurality of object features (O1 to O n ) can be a vector, a matrix, or a tensor. In one embodiment, a plurality of object features (O1 to O n ) can correspond to different multiple points in time or time intervals, respectively. In one embodiment, each object feature (O1 to On ) can correspond to each frame (viewpoint). In another embodiment, each object feature (O1 to O n ) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of object features (O1 to O n ) can correspond to each frame (viewpoint), and a plurality of object features (O1 to O n ) each of which can correspond to a plurality of consecutive frames (time periods).

[0071] In one embodiment, the first visual feature extraction module (MD1a) extracts a plurality of object features (O1 to O n ) may include an object graph extraction module (MD1a-2) for obtaining a plurality of object graphs (E1 to En). For example, the object graph extraction module (MD1a-2) may include the object graph extraction module (MD1a-2) illustrated in FIG. 6. The processor (120) may use the object graph extraction module (MD1a-2) to obtain a plurality of object features (O1 to O n ) from multiple object graphs (E1 to E n ) can be obtained. In one embodiment, the processor (120) uses the object graph extraction module (MD1a-2) to obtain object features (O1 to O) corresponding to each point in time or time interval. n ) from multiple object graphs (E1 to E) corresponding to each point in time or time interval. n ) can be obtained. For example, the processor (120) can obtain an object graph (E1) corresponding to the first time point from an object feature (O1) corresponding to the first time point using the object graph extraction module (MD1a-2).

[0072] In one embodiment, a plurality of object graphs (E1 to E n ) can correspond to different multiple points in time or time intervals, respectively. In one embodiment, each object graph (E1 to E n) can correspond to each frame (viewpoint). In another embodiment, each object graph (E1 to E n ) may correspond to each of a plurality of consecutive frames (time periods). In another embodiment, a plurality of object graphs (E1 to E n ) can correspond to each frame (viewpoint), and a plurality of object graphs (E1 to E n ) each of which can correspond to a plurality of consecutive frames (time periods).

[0073] In one embodiment, the first visual feature extraction module (MD1a) extracts a plurality of object graphs (E1 to E n ) from a plurality of first visual features (V a1 Inland V an ) may include an object graph feature extraction module (MD1a-3) for obtaining the object graph feature. The processor (120) may use the object graph feature extraction module (MD1a-3) to extract a plurality of object graphs (E1 to E n ) from a plurality of first visual features (V a1 Inland V an ) can be obtained. For example, the object graph feature extraction module (MD1a-3) may include a graph convolution layer. In one embodiment, the processor (120) uses the object graph feature extraction module (MD1a-3) to extract an object graph (E1 to E) corresponding to each point in time or time interval. n ) from which a plurality of first visual features (V) corresponding to each point in time or time interval a1 Inland V an ) can be obtained. For example, the processor (120) may use the object graph feature extraction module (MD1a-3) to extract the first visual feature (V) corresponding to the first time point from the object graph (E1) corresponding to the first time point. a1 ) can be obtained.

[0074] Fig. 6 is a conceptual diagram of an object graph extraction module used in the electronic device according to one embodiment of the present invention. Referring to Fig. 6, the object graph extraction module (MD1a-2) extracts a plurality of object features (O1 to O n ) from a set of multiple object graph nodes (Z1 to Z n ) and a node extraction module (MD1a-2N) for obtaining a plurality of object graph node sets (Z1 to Z n ) from each object graph node set (Z1 to Z n ) by obtaining the relationships between nodes in multiple object graphs (E1 to E n ) may include a relationship extraction module (MD1a-2R) for obtaining the object features (O1 to O). The processor (120) may use the node extraction module (MD1a-2N) to extract a plurality of object features (O1 to O n ) from a set of multiple object graph nodes (Z1 to Z n ) can be obtained, and a set of multiple object graph nodes (Z1 to Z) can be obtained using the relationship extraction module (MD1a-2R). n ) from each object graph node set (Z1 to Z n ) by obtaining the relationships between nodes in multiple object graphs (E1 to E n ) can be obtained.

[0075] In one embodiment, the node extraction module (MD1a-2N) may include at least one of a transformer-based model and a convolutional network-based model. In one embodiment, the relationship extraction module (MD1a-2R) may include at least one of a transformer-based model and a convolutional network-based model.

[0076] In one embodiment, the processor (120) uses the node extraction module (MD1a-2N) to extract object features (O1 to O) corresponding to each point in time or time interval. n ) from a set of object graph nodes (Z1 to Z) corresponding to each point in time or time interval. n) can be obtained. For example, the processor (120) can obtain an object graph node set (Z1) corresponding to the first time point from an object feature (O1) corresponding to the first time point using a node extraction module (MD1a-2N).

[0077] In one embodiment, the processor (120) uses the relationship extraction module (MD1a-2R) to extract a set of object graph nodes (Z1 to Z) corresponding to each point in time or time interval. n ) from a set of object graph nodes (Z1 to Z) corresponding to each point in time or time interval. n ) by obtaining the relationship between nodes in the object graph (E1 to E) corresponding to each point in time or time interval. n ) can be obtained. For example, the processor (120) can obtain an object graph (E1) corresponding to the first time point by obtaining a relationship between nodes within the object graph node set (Z1) corresponding to the first time point from the object graph node set (Z1) corresponding to the first time point using the relationship extraction module (MD1a-2R).

[0078] FIG. 7 is a conceptual diagram of a multi-modal time axis fusion artificial intelligence model according to one embodiment of the present invention. Hereinafter, the multi-modal time axis fusion artificial intelligence model (MD) illustrated in FIG. 2 and the multi-modal time axis fusion artificial intelligence model (MD') illustrated in FIG. 7 will be described, focusing on the differences between them.

[0079] Referring to Figure 7, the multi-modal time axis fusion artificial intelligence model (MD') generates multiple second visual features (V) corresponding to different multiple time points or time intervals from the video. b1 Inland V bn ) to obtain a second visual feature extraction module (MD1b) and a plurality of second visual features (V) by fusing the second visual features and text features corresponding to the same time point or time interval. b1Inland V bn ) and multiple text features (L1 to L n ) from multiple second local fusion features (T ) corresponding to multiple points in time or time intervals, respectively. b1 Inland T bn ) may further include a second local fusion module (MD6) for obtaining the second visual features (V) corresponding to each of a plurality of time points or time periods from the video using the second visual feature extraction module (MD1b). b1 Inland V bn ) is obtained, and multiple second visual features (V) are obtained by fusing the second visual features and text features corresponding to the same time or time interval using the second local fusion module (MD6). b1 Inland V bn ) and multiple text features (L1 to L n ) from multiple second local fusion features (T ) corresponding to multiple points in time or time intervals, respectively. b1 Inland T bn ) can be obtained.

[0080] Multiple first visual features (V a1 Inland V an , see Fig. 2) and a plurality of second visual features (V b1 Inland V bn ) may be different. For example, multiple first visual features (V a1 Inland V an ) may be a visual feature based on an object graph as described with reference to FIG. 5, and a plurality of second visual features (V b1 Inland V bn) may be visual features based on appearance. For example, the first visual feature extraction module (MD1a) may include Faster-RCNN, and the second visual feature extraction module (MD1b) may include ResNet. In one embodiment, the second local fusion feature module (MD6) may have the same structure as the first local fusion module (MD3) but may have different parameters. In another embodiment, the second local fusion feature module (MD6) may have a different structure than the first local fusion module (MD3).

[0081] The processor (120) comprises a plurality of first local fusion features (T a1 Inland T an ) and multiple second regional fusion features (T b1 Inland T bn ) from multiple third regional fusion features (T c1 Inland T cn ) can be obtained. Each third regional fusion feature (T c1 Inland T cn ) may correspond to each point in time or time interval. For example, the processor (120) may obtain a third local fusion feature corresponding to the same point in time or time interval by concatenating, adding, or multiplying the first local fusion feature and the second local fusion feature corresponding to the same point in time or time interval. For example, the processor (120) may obtain a first local fusion feature (T) corresponding to the first point in time. a1 ) and the second regional convergence feature (T b1 ) by concatenating, adding, or multiplying the third local fusion feature (T) corresponding to the first time point. c1 ) can be obtained.

[0082] The global fusion module (MD4') has multiple third local fusion features (T c1 Inland T cn) from at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) can be obtained. The processor (120) uses a global fusion module (MD4') to obtain multiple third local fusion features (T c1 Inland T cn ) from at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) can be obtained. As described with reference to FIG. 2, the global fusion module (MD4') can include a self-attention layer.

[0083] Fig. 8 is a conceptual diagram for explaining a multi-modal time axis fusion artificial intelligence model according to an embodiment of the present invention. Referring to Fig. 8, the multi-modal time axis fusion artificial intelligence model (MD, MD') includes at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) can be used to obtain answers to questions. The text can contain questions.

[0084] A multi-modal time axis fusion AI model (MD, MD') comprises at least one global fusion feature, for example, multiple global fusion features (G1 to G n ) may include a summary vector extraction module (MD7) that obtains a summary vector (G) from the summary vector extraction module (MD7). The processor (120) may use the summary vector extraction module (MD7) to obtain at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) can be used to obtain a summary vector (G).

[0085] The multi-modal time axis fusion artificial intelligence model (MD, MD') may include a choice feature extraction module (MD8) that obtains a plurality of choice features from each of a plurality of choices. The processor (120) may use the choice feature extraction module (MD8) to obtain a plurality of choice features from each of the plurality of choices. In one embodiment, the choice feature extraction module (MD8) may include at least one of BERT and RoBERTa. In one embodiment, each choice feature may be a vector, a matrix, or a tensor.

[0086] The processor (120) can obtain an answer to a question expressed in text from the summary vector (G) and each choice feature. For example, the processor (120) can obtain the similarity between the summary vector (G) and each choice feature, and select the choice with the highest similarity to the summary vector (G) as the answer. For example, the processor (120) can obtain the cosine similarity between the summary vector (G) and each choice feature or the distance between the summary vector (G) and each choice feature to obtain the similarity between the summary vector (G) and each choice feature.

[0087] Fig. 9 is a conceptual diagram for explaining a multi-modal time axis fusion artificial intelligence model according to an embodiment of the present invention. Referring to Fig. 9, the multi-modal time axis fusion artificial intelligence model (MD, and MD') includes at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n ) may further include a classification module (MD9) that classifies the video using the processor (120). The processor (120) may further include at least one global fusion feature, for example, a plurality of global fusion features (G1 to G n) and the classification module (MD9) can be used to classify videos. In one embodiment, the processor (120) can classify videos by religion or political orientation. In one embodiment, the processor (120) can classify videos by whether they are adult content, hate content, related to a specific issue, related to a specific brand, etc.

[0088] Fig. 10 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to Fig. 10, the operating method of the electronic device may include a step (S1010) of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video, a step (S1020) of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text, a step (S1030) of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time section, and a step (S1040) of obtaining at least one global fusion feature from the plurality of first local fusion features.

[0089] Fig. 11 is a flowchart of an operating method of the electronic device according to one embodiment of the present invention. Referring to Fig. 11, the step of obtaining multiple text features (S1020, see Fig. 10) may include the step of obtaining multiple text features corresponding to multiple time points or time periods from a text using multiple mapping layers corresponding to multiple time points or time periods, respectively (S1120). Specifically, the operating method of the electronic device may include a step (S1110) of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video, a step (S1120) of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text using a plurality of mapping layers corresponding to a plurality of viewpoints or time sections, a step (S1130) of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time section, and a step (S1140) of obtaining at least one global fusion feature from the plurality of first local fusion features.

[0090] FIG. 12 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to FIG. 12, the step of obtaining a plurality of text features (S1020, see FIG. 10) may include the step of obtaining a plurality of first intermediate text features from a text (S1221), the step of obtaining a plurality of second intermediate text features corresponding to a plurality of time points or time periods from the plurality of first intermediate text features using a plurality of mapping layers corresponding to a plurality of time points or time periods (S1222), and the step of obtaining a plurality of text features from the plurality of second intermediate text features by changing the dimensions of the second intermediate text features (S1223). In one embodiment, the number of the plurality of first intermediate text features may be different from the number of the plurality of second intermediate text features. In one embodiment, the dimension of each of the plurality of text features may be the same as the dimension of each of the plurality of first visual features.

[0091] Specifically, the operating method of the present electronic device may include a step of obtaining a plurality of first visual features each corresponding to a plurality of different viewpoints or time sections from a video (S1210), a step of obtaining a plurality of first intermediate text features from a text (S1221), a step of obtaining a plurality of second intermediate text features each corresponding to a plurality of viewpoints or time sections from the plurality of first intermediate text features using a plurality of mapping layers each corresponding to a plurality of viewpoints or time sections (S1222), a step of obtaining a plurality of text features from the plurality of second intermediate text features by changing a dimension of the second intermediate text features (S1223), a step of obtaining a plurality of first local fusion features each corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time section (S1230), and a step of obtaining at least one global fusion feature from the plurality of first local fusion features (S1240).

[0092] FIG. 13 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to FIG. 13, the step of obtaining a plurality of first local fusion features (S1030, see FIG. 10) may include a step of obtaining a first local fusion feature from a first visual feature and a text feature using convolution (S1330). Specifically, the operating method of the electronic device may include a step of obtaining a plurality of first visual features each corresponding to a plurality of different viewpoints or time sections from a video (S1310), a step of obtaining a plurality of text features each corresponding to a plurality of viewpoints or time sections from a text (S1320), a step of obtaining a first local fusion feature from the first visual feature and the text feature using convolution (S1330), and a step of obtaining at least one global fusion feature from the plurality of first local fusion features (S1340).

[0093] FIG. 14 is a flowchart of an operating method of the electronic device according to one embodiment of the present invention. Referring to FIG. 14, the step of obtaining a plurality of first local fusion features (S1030, see FIG. 10) may include a step of obtaining a first intermediate local fusion feature from a first visual feature and a text feature (S1431), a step of obtaining a second intermediate local fusion feature from the first intermediate local fusion feature using convolution (S1432), and a step of obtaining the first local fusion feature from the first visual feature and the second intermediate local fusion feature (S1433). Specifically, the operating method of the electronic device may include a step (S1410) of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video, a step (S1420) of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text, a step (S1431) of obtaining a first intermediate local fusion feature from the first visual feature and the text feature, a step (S1432) of obtaining a second intermediate local fusion feature from the first intermediate local fusion feature using convolution, a step (S1433) of obtaining a first local fusion feature from the first visual feature and the second intermediate local fusion feature, and a step (S1440) of obtaining at least one global fusion feature from the plurality of first local fusion features.

[0094] FIG. 15 is a flowchart of an operating method of the electronic device according to one embodiment of the present invention. Referring to FIG. 15, the step of obtaining a plurality of first visual features (S1010, see FIG. 10) may include a step of obtaining a plurality of object features from a video (S1511), a step of obtaining a plurality of object graphs from the plurality of object features (S1512), and a step of obtaining a plurality of first visual features from the plurality of object graphs (S1513). Specifically, the operating method of the electronic device may include a step of obtaining a plurality of object features from a video (S1511), a step of obtaining a plurality of object graphs from the plurality of object features (S1512), a step of obtaining a plurality of first visual features from the plurality of object graphs (S1513), a step of obtaining a plurality of text features corresponding to a plurality of viewpoints or time periods from a text (S1520), a step of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time periods from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time period (S1530), and a step of obtaining at least one global fusion feature from the plurality of first local fusion features (S1540).

[0095] FIG. 16 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to FIG. 16, the operating method of the electronic device further includes a step (S1610B) of obtaining a plurality of second visual features corresponding to a plurality of time points or time sections from a video, a step (S1630B) of obtaining a plurality of second local fusion features corresponding to a plurality of time points or time sections from the plurality of second visual features and the plurality of text features by fusing the second visual features and text features corresponding to the same time points or time sections, and a step (S1650) of obtaining a plurality of third local fusion features corresponding to a plurality of time points or time sections from the plurality of first local fusion features and the plurality of second local fusion features, and the step of obtaining at least one global fusion feature (S1040, see FIG. 10) may include a step (S1640) of obtaining at least one global fusion feature from the plurality of third local fusion features.

[0096] Specifically, the operating method of the present electronic device comprises the steps of: obtaining a plurality of first visual features corresponding to a plurality of different time points or time sections from a video (S1610A); obtaining a plurality of text features corresponding to a plurality of time points or time sections from a text (S1620); obtaining a plurality of first local fusion features corresponding to a plurality of time points or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same time points or time sections (S1630A); obtaining a plurality of second visual features corresponding to a plurality of time points or time sections from the video (S1610B); obtaining a plurality of second local fusion features corresponding to a plurality of time points or time sections from the plurality of second visual features and the plurality of text features by fusing the second visual features and the text features corresponding to the same time points or time sections (S1630B); obtaining a plurality of third local fusion features corresponding to a plurality of time points or time sections from the plurality of first local fusion features and the plurality of second local fusion features (S1650); and obtaining at least one of the third local fusion features from the plurality of third local fusion features. It may include a step (S1640) of acquiring global fusion characteristics.

[0097] FIG. 17 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to FIG. 17, the step of obtaining at least one global fusion feature (S1040, see FIG. 10) may include a step of obtaining at least one global fusion feature from a plurality of first local fusion features using self-attention (S1740). Specifically, the operating method of the electronic device may include a step of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video (S1710), a step of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text (S1720), a step of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time section (S1730), and a step of obtaining at least one global fusion feature from the plurality of first local fusion features using self-attention (S1740).

[0098] Fig. 18 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. The text on Fig. 18 may include a question. Referring to Fig. 18, the operating method of the electronic device may further include a step (S1850) of obtaining an answer to the question using at least one global fusion feature. Specifically, the operating method of the electronic device may include a step (S1810) of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time periods from a video, a step (S1820) of obtaining a plurality of text features corresponding to a plurality of viewpoints or time periods from a text, a step (S1830) of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time periods from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time period, a step (S1840) of obtaining at least one global fusion feature from the plurality of first local fusion features, and a step (S1850) of obtaining an answer to the question using the at least one global fusion feature.

[0099] Fig. 19 is a flowchart of an operating method of the electronic device according to one embodiment of the present invention. At least one global fusion feature on Fig. 19 may include multiple global fusion features. Referring to Fig. 19, the step of obtaining an answer to a question (S1850, see Fig. 18) may include the step of obtaining a summary vector from at least one global fusion feature (S1951), the step of obtaining a plurality of choice features from each of a plurality of choices (S1952), and the step of obtaining an answer to the question from the summary vector and each of the choice features (S1953). Specifically, the operating method of the present electronic device may include a step of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video (S1910), a step of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text (S1920), a step of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same viewpoint or time section (S1930), a step of obtaining at least one global fusion feature from the plurality of first local fusion features (S1940), a step of obtaining a summary vector from the at least one global fusion feature (S1951), a step of obtaining a plurality of option features from a plurality of options (S1952), and a step of obtaining an answer to a question from the summary vector and each option feature (S1953).

[0100] Fig. 20 is a flowchart of an operating method of the electronic device according to an embodiment of the present invention. Referring to Fig. 20, the operating method of the electronic device may further include a step (S2050) of classifying, searching, predicting, or generating a video using at least one global fusion feature. Specifically, the operating method of the electronic device may include a step (S2010) of obtaining a plurality of first visual features corresponding to a plurality of different viewpoints or time sections from a video, a step (S2020) of obtaining a plurality of text features corresponding to a plurality of viewpoints or time sections from a text, a step (S2030) of obtaining a plurality of first local fusion features corresponding to a plurality of viewpoints or time sections from the plurality of first visual features and the plurality of text features by fusing the first visual features and text features corresponding to the same viewpoint or time section, a step (S2040) of obtaining at least one global fusion feature from the plurality of first local fusion features, and a step (S2050) of classifying, searching, predicting, or generating a video using the at least one global fusion feature.

[0101] [Examples and Comparative Examples]

[0102] First embodiment: Using object graph-based first visual features and appearance-based second visual features as shown in Fig. 7.

[0103] Second embodiment: As shown in Fig. 2, object graph-based visual features are used as the first visual feature, and appearance-based visual features are not used.

[0104] Third embodiment: Using appearance-based visual features as the first visual feature as shown in Fig. 2 and not using object graph-based visual features.

[0105] Example 4: In addition to the object graph-based first visual feature and the appearance-based second visual feature, a motion-based third visual feature is further used.

[0106] Example 5: Using the first visual feature obtained using the method of VGT (Xiao et al., Video graph transformer for video question answering, ECCV 2022), and not using the second visual feature. Performing both local and global fusion.

[0107] First comparative example: HME (Fan et al., Heterogeneous memory enhanced multimodal attention model for video question answering, CVPR 2019)

[0108] Second comparative example: CoMem (Gao et al., Motion-appearance co-memory networks for video question answering, CVPR 2018)

[0109] Third comparative example: HCRN (Le et al., Hierarchical conditional relation networks for video question answering, CVPR 2020)

[0110] Fourth comparative example: HGA (Jiang et al., Reasoning with heterogeneous graph alignment for video question answering, AAAI 2020)

[0111] Fifth comparative example: B2A (Park et al., Bridge to answer: Structure-aware graph interaction network for video question answering, CVPR 2021)

[0112] Comparative Example 6: IGV (Li et al., Invariant grounding for video question answering, CVPR 2022)

[0113] Comparative example 7: MHN (Peng et al., Multilevel hierarchical network with multi-scale sampling for video question answering, arXiv 2022)

[0114] Comparative Example 8: VGT

[0115] Comparative Example 9: HQGA (Xiao et al., Video as conditional graph hierarchy for multi-granular question answering, AAAI 2022)

[0116] Comparative Example 10: ML (Select the answer with the highest frequency of occurrence)

[0117] Comparative example 11: PSAC (Li et al., Beyond rnns: Positional self-attention with co-attention for video question answering, AAAI 2019)

[0118] Comparative Example 12: Performing global fusion using visual feature vectors obtained using VGT and attention modules.

[0119] Comparative Example 13: Performing both local and global fusion using visual feature vectors obtained using VGT and attention modules.

[0120] Comparative Example 14: Using the first visual feature obtained using the VGT method, and not using the second visual feature. Performing global fusion.

[0121] Comparative Example 15: Using appearance-based second visual features and not using object graph-based first visual features. Multiple text features (L1 to Ln ) are identical to each other.

[0122] [Evaluation dataset]

[0123] Causal-VidQA: Wang et al., Causal attention for unbiased visual recognition, ICCV 2021

[0124] NExT-QA: Xiao et al., Next-qa: Next phase of question-answering to explaining temporal actions, CVPR 2021

[0125] MSVD-QA: Xu et al., Video question answering via gradually refined attention over appearance and motion, ACM international conference on Multimedia 2017

[0126] AGQA-2.0: Grunde-McLaughlin et al., Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning, arXiv 2022

[0127] [Comparison with prior art]

[0128] Referring to Table 1 below, the first embodiment showed higher performance than the first to fifth comparative examples on the Causal-VidQA dataset. In Table 1 below, ACC D ACC is the answer accuracy for descriptive problem types. E ACC is the answer accuracy for explanatory problem types. P ACC is the answer accuracy for predictive problem types C Accuracy of answers to counterfactual problem types, ACC Arepresents the accuracy across the dataset. In particular, the first embodiment showed significant performance improvements for predictive and counterfactual problem types.

[0129]

[0130]

[0131] Referring to Table 2 below, Test-C represents the answer accuracy for causal question types in the NExT-QA dataset, Test-T represents the answer accuracy for temporal question types in the NExT-QA dataset, Test-D represents the answer accuracy for descriptive question types in the NExT-QA dataset, and Test-A represents the overall accuracy of the NExT-QA dataset. The overall accuracy of the first embodiment was higher than that of the first comparative example and the sixth to ninth comparative examples. In addition, the first embodiment showed higher performance than the first comparative example and the sixth to ninth comparative examples on the MSVD-QA dataset.

[0132]

[0133]

[0134] Referring to Table 3 below, the accuracy performance of Comparative Examples 10, 11, 1, 3, and Embodiment 1 by problem type of the AGQA v2.0 dataset was evaluated (obj: object, rel: relationship, act: action, comp: comparison, recog: recognition in Table 3). The performance of Embodiment 1 for all problems was higher than the performance of Comparative Examples 10, 11, 1, and 3. In particular, Embodiment 1 showed significant performance improvement for the action recognition question type compared to Comparative Examples 10, 11, 1, and 3.

[0135]

[0136]

[0137] [Effects of Multi-Level Fusion]

[0138] Referring to Table 4 below, the performance of Comparative Examples 12 to 14 and Example 5 were compared to verify the effectiveness of multi-level fusion, including local fusion and global fusion. In Table 4 below, ACC D Accuracy of answers to descriptive problem types, ACC E Accuracy of answers to descriptive problem types, ACC P is the answer accuracy for predictive problem types, ACC C Accuracy of answers to counterfactual problem types, ACC A represents the overall accuracy for the dataset. Comparative examples 12 and 13 use the same visual feature vector and the same attention module, but Comparative example 12 performs only global fusion, while Comparative example 13 performs both local fusion and global fusion. Meanwhile, Comparative example 14 and embodiment 5 use the same visual feature vector and the same artificial intelligence model (MD, see Fig. 2), but Comparative example 14 performs only global fusion, while Embodiment 5 performs both local fusion and global fusion. Given that the overall performance of Comparative example 13 is higher than that of Comparative example 12, and that the overall performance of Embodiment 5 is higher than that of Comparative example 14, it can be seen that multi-level fusion that combines local fusion and global fusion contributes to performance improvement compared to performing only global fusion. In particular, multi-level fusion improves the performance for predictive question types (ACC). P ) can play a significant role in improving the performance.

[0139]

[0140]

[0141] [Influence of visual features]

[0142] Referring to Table 5 below, the second embodiment uses object-graph-based visual features, the third embodiment uses appearance-based visual features, the first embodiment uses object-graph-based first visual features and appearance-based second visual features, and the fourth embodiment uses object-graph-based first visual features, appearance-based second visual features, and motion-based third visual features. Test-C represents the answer accuracy for causal question types in the NExT-QA dataset, Test-T of the NExT-QA dataset represents the answer accuracy for temporal question types, Test-D of the NExT-QA dataset represents the answer accuracy for descriptive question types, and Test-A represents the overall accuracy of the NExT-QA dataset. From Table 5, it can be confirmed that the first to fourth embodiments exhibit satisfactory performance for various types of visual features. Therefore, it can be seen that the present invention is not limited by the visual feature extraction modules (MD1a and MD1b) or the types of visual features.

[0143]

[0144]

[0145] [Extending the time axis of text features]

[0146] Referring to Table 6 below, Comparative Example 15 uses the same text feature for all time points or time intervals by using a method of copying a single text feature multiple times, whereas Embodiment 3 uses multiple different text features corresponding to each time point or time interval. In Table 6 below, ACC D Accuracy of answers to descriptive problem types, ACC E Accuracy of answers to descriptive problem types, ACC P is the answer accuracy for predictive problem types, ACC C Accuracy of answers to counterfactual problem types, ACC Arepresents the accuracy for the entire Causal-VidQA dataset.

[0147] Overall Performance (ACC) A ) in terms of performance, the third embodiment showed better performance than the fifteenth comparative example. Therefore, it can be seen that the time axis extension method of the text feature of the present invention, which extracts different text features for each time, contributes to the performance improvement. In particular, the third embodiment showed higher performance (ACC) for the counterfactual problem type than the fifteenth comparative example. C ) showed improvement.

[0148]

[0149]

[0150] [Temporal similarity with the correct answer]

[0151] To examine whether embodiments of the present invention are effective in understanding temporal context, the similarity between the correct answer vector and the global fusion vector corresponding to each time point or time interval was calculated.

[0152] For example, for the question "Which object were they behind before standing up but after lying on a bed?", the correct answer vector showed the highest similarity to the global fusion feature corresponding to the time point corresponding to "just before standing" in the first embodiment. This indicates that the first embodiment can learn a global fusion feature related to the correct answer.

[0153] Additionally, for the question "While putting on a shoe, which object did the person close?", the correct answer vector showed the highest similarity to the global fusion feature corresponding to the moment in Example 1 when the person interacted with the shoe and closed the laptop. This indicates that Example 1 was able to learn a global fusion feature related to the correct answer.

[0154] Furthermore, for the question "Was the person tidying up the object they were behind first before or after tidying something on the thing they stood on?", the correct answer vector showed the highest similarity to the global fusion feature corresponding to the point at which the person changed his or her behavior between the point corresponding to "tidying the floor" in the first embodiment and the point corresponding to "tidying the object they were behind." In other words, it can be seen that the first embodiment can learn the global fusion feature related to the correct answer.

[0155] While the embodiments of the present invention have been illustrated and described above, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims and their equivalents.

Claims

1. Storage; and including a processor; The above processor, Obtaining a plurality of first visual features corresponding to different multiple points in time or time intervals from a video, Obtaining multiple text features corresponding to each of the multiple points in time or time intervals from the text, By fusing first visual features and text features corresponding to the same time point or time interval, a plurality of first local fusion features corresponding to the plurality of time points or time intervals are obtained from the plurality of first visual features and the plurality of text features, Obtaining at least one global fusion feature from the plurality of first local fusion features, The above processor, in order to obtain the plurality of text features, Obtaining a plurality of first intermediate text features from the above text, Obtaining a plurality of second intermediate text features corresponding to the plurality of time points or time periods from the plurality of first intermediate text features using a plurality of mapping layers corresponding to the plurality of time points or time periods, An electronic device for a multi-modal time axis fusion artificial intelligence model, which obtains a plurality of text features from a plurality of second intermediate text features by changing the dimension of the second intermediate text features.

2. In paragraph 1, An electronic device for a multi-modal time axis fusion artificial intelligence model, wherein the number of the plurality of first intermediate text features is different from the number of the plurality of second intermediate text features.

3. In paragraph 1, An electronic device for a multi-modal time axis fusion artificial intelligence model, wherein the dimension of each of the plurality of text features is the same as the dimension of each of the plurality of first visual features.

4. In paragraph 1, The above processor, In order to obtain the above plurality of first local fusion features, An electronic device for a multi-modal temporal axis fusion artificial intelligence model that obtains the first local fusion feature from the first visual feature and the text feature using convolution.

5. In paragraph 1, The above processor, In order to obtain the above plurality of first local fusion features, Obtaining a first intermediate local fusion feature from the first visual feature and the text feature, Obtaining the second intermediate local fusion feature from the first intermediate local fusion feature using convolution, An electronic device for a multi-modal time axis fusion artificial intelligence model, which obtains the first local fusion feature from the first visual feature and the second intermediate local fusion feature.

6. In paragraph 1, The above processor, In order to obtain the above plurality of first visual features, Obtain multiple object features from the above video, Obtaining multiple object graphs from the above multiple object features, An electronic device for a multi-modal time axis fusion artificial intelligence model, which obtains the plurality of first visual features from the plurality of object graphs.

7. In paragraph 1, The above processor, Obtaining a plurality of second visual features corresponding to each of the plurality of time points or time periods from the above video, By fusing second visual features and text features corresponding to the same point in time or time interval, a plurality of second local fusion features corresponding to the plurality of points in time or time interval are obtained from the plurality of second visual features and the plurality of text features, Obtaining a plurality of third regional fusion features corresponding to the plurality of time points or time intervals from the plurality of first regional fusion features and the plurality of second regional fusion features, An electronic device for a multi-modal time axis fusion artificial intelligence model, which obtains at least one global fusion feature from the plurality of third local fusion features.

8. In paragraph 1, The above processor, To obtain at least one global fusion feature, An electronic device for a multi-modal time axis fusion artificial intelligence model, which obtains at least one global fusion feature from the plurality of first local fusion features using self-attention.

9. In paragraph 1, The above text contains a question, The above processor, An electronic device for a multi-modal time axis fusion artificial intelligence model that obtains an answer to the above question by using at least one global fusion feature.

10. In paragraph 9, wherein said at least one global fusion feature comprises a plurality of global fusion features, The above processor, To get answers to the above questions, Obtain a summary vector from the above multiple global fusion features, Obtain multiple choice characteristics from multiple choices, respectively, An electronic device for a multi-modal time axis fusion artificial intelligence model that obtains an answer to the question from the above summary vector and each of the above selection features.

11. In paragraph 1, The above processor, An electronic device for a multi-modal temporal axis fusion artificial intelligence model that classifies, searches, predicts, or generates the video using at least one global fusion feature.

12. A step of obtaining a plurality of first visual features each corresponding to a plurality of different time points or time periods from a video; A step of obtaining a plurality of text features each corresponding to a plurality of time points or time periods from a text; A step of obtaining a plurality of first local fusion features corresponding to the plurality of points in time or time intervals from the plurality of first visual features and the plurality of text features by fusing the first visual features and the text features corresponding to the same point in time or time interval; and A step of obtaining at least one global fusion feature from the plurality of first local fusion features; The step of obtaining the above multiple text features is: A step of obtaining a plurality of first intermediate text features from the above text; A step of obtaining a plurality of second intermediate text features corresponding to the plurality of time points or time periods from the plurality of first intermediate text features using a plurality of mapping layers corresponding to the plurality of time points or time periods respectively; and An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising: a step of obtaining a plurality of text features from the plurality of second intermediate text features by changing the dimension of the second intermediate text features.

13. In paragraph 12, A method of operating an electronic device for a multi-modal time axis fusion artificial intelligence model, wherein the number of the plurality of first intermediate text features is different from the number of the plurality of second intermediate text features.

14. In paragraph 12, A method of operating an electronic device for a multi-modal time axis fusion artificial intelligence model, wherein the dimension of each of the plurality of text features is the same as the dimension of each of the plurality of first visual features.

15. In paragraph 12, The step of obtaining the plurality of first local fusion features is: An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising a step of obtaining the first local fusion feature from the first visual feature and the text feature using convolution.

16. In paragraph 12, The step of obtaining the plurality of first local fusion features is: A step of obtaining a first intermediate local fusion feature from the first visual feature and the text feature; A step of obtaining a second intermediate local fusion feature from the first intermediate local fusion feature using convolution; and An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising: a step of obtaining the first local fusion feature from the first visual feature and the second intermediate local fusion feature.

17. In paragraph 12, The step of acquiring the plurality of first visual features is: A step of obtaining multiple object features from the above video; A step of obtaining a plurality of object graphs from the plurality of object features; and An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising a step of obtaining the plurality of first visual features from the plurality of object graphs.

18. In paragraph 12, A step of obtaining a plurality of second visual features each corresponding to the plurality of time points or time periods from the video; A step of obtaining a plurality of second local fusion features corresponding to the plurality of points in time or time intervals from the plurality of second visual features and the plurality of text features by fusing the second visual features and the text features corresponding to the same point in time or time interval; and Further comprising a step of obtaining a plurality of third regional fusion features corresponding to the plurality of time points or time periods from the plurality of first regional fusion features and the plurality of second regional fusion features; A method of operating an electronic device for a multi-modal time axis fusion artificial intelligence model, wherein the step of obtaining at least one global fusion feature comprises the step of obtaining the at least one global fusion feature from the plurality of third local fusion features.

19. In paragraph 12, The step of obtaining at least one global fusion feature comprises: An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising a step of obtaining at least one global fusion feature from the plurality of first local fusion features using self-attention.

20. In paragraph 12, The above text contains a question, A method of operating an electronic device for a multi-modal time axis fusion artificial intelligence model, further comprising the step of obtaining an answer to the question using at least one global fusion feature.

21. In paragraph 20, wherein said at least one global fusion feature comprises a plurality of global fusion features, The steps to obtain the answer to the above question are: A step of obtaining a summary vector from the above plurality of global fusion features; A step of obtaining multiple choice features from each of multiple choices; and An operating method of an electronic device for a multi-modal time axis fusion artificial intelligence model, comprising: obtaining an answer to the question from the summary vector and each of the selection features.

22. In paragraph 12, A method of operating an electronic device for a multi-modal time axis fusion artificial intelligence model, further comprising the step of classifying, searching, predicting, or generating the video using at least one global fusion feature.

Images