Electronic device for at least one of video moment retrieval and highlight detection and operating method thereof

Abstract

Disclosed is an electronic device for at least one of video moment retrieval and highlight detection. This electronic device may comprise a storage unit and a processor, wherein the processor: acquires a plurality of first video features from a video; acquires a text query feature from a text query; acquires a plurality of weights from the plurality of first video features and the text query feature; acquires a plurality of second video features from the plurality of weights and the plurality of first video features; acquires a plurality of third video features from the plurality of second video features by using an encoder; acquires a plurality of fourth video features from the plurality of third video features and a time query by using a decoder; and selects at least one of a time point and a time interval by using the plurality of fourth video features.

Description

Electronic device and method of operation thereof for at least one of video scene search and highlight detection

[0001] The present invention relates to an electronic device and a method of operating the same. More specifically, the present invention relates to an electronic device and a method of operating the same for at least one of video scene retrieval and highlight detection. Additionally, the present invention relates to an electronic device and a method of operating the same for learning at least one of video scene retrieval and highlight detection. Here, video moment retrieval refers to the task of finding moments related to a given text query in a video. Meanwhile, highlight detection aims to measure the importance of each point or time period in a given video in relation to a given text query.

[0002]

[0003] With the advancement of digital devices and platforms, the volume of video data is exploding. However, with this growth in video data, more and more viewers want to search for specific scenes or watch only key moments. While research on video scene search and highlight detection technologies is ongoing, performance improvements are still needed. For example, existing studies have not adequately modeled the impact of text queries. For example, the relevance of text queries has not significantly impacted video scene search and highlight detection results. Therefore, research is needed to develop video scene search and highlight detection technologies that can improve performance by properly modeling the impact of text queries.

[0004]

[0005] The problem to be solved by the present invention is to provide an electronic device and an operating method thereof for at least one of improved performance video scene search and highlight detection.

[0006] The problem to be solved by the present invention is to provide an electronic device and an operating method thereof for learning at least one of improved performance video scene search and highlight detection.

[0007]

[0008] According to one embodiment of the present invention, an electronic device for at least one of video scene search and highlight detection includes a storage unit and a processor, wherein the processor is configured to obtain a plurality of first video features each corresponding to a plurality of different time points or time periods from a video, obtain a text query feature from a text query, obtain a plurality of weights from the plurality of first video features and the text query feature, obtain a plurality of second video features from the plurality of weights and the plurality of first video features, obtain a plurality of third video features from the plurality of second video features using an encoder, obtain a plurality of fourth video features from the plurality of third video features and a time query using a decoder, and select at least one of the plurality of different time points or time periods of the video using the plurality of fourth video features.

[0009] According to one embodiment of the present invention, there is provided an operating method of an electronic device for at least one of video scene search and highlight detection, the method comprising: obtaining a plurality of first video features each corresponding to a plurality of different viewpoints or time periods from a video; obtaining a text query feature from a text query; obtaining a plurality of weights from the plurality of first video features and the text query features; obtaining a plurality of second video features from the plurality of weights and the plurality of first video features; obtaining a plurality of third video features from the plurality of second video features using an encoder; obtaining a plurality of fourth video features from the plurality of third video features and a time query using a decoder; and selecting at least one of a plurality of different viewpoints or time periods from the video using the plurality of fourth video features.

[0010] According to one embodiment of the present invention, an electronic device for learning at least one of video scene retrieval and highlight detection includes a storage unit and a processor, wherein the processor obtains, from a video, a first text query that is a positive pair with the video, and an importance token using a cross-attention model and an encoder, a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query, and obtains, from the video, a second text query that is a negative pair with the video, and the importance token using the cross-attention model and the encoder, a plurality of second video features and a third importance of the video for the second text query, wherein the first importance is greater than the second importance, and the cross-attention model, the encoder, and the importance token can be modified such that the first importance increases and the second importance and the third importance decrease.

[0011] According to one embodiment of the present invention, an operating method of an electronic device for learning at least one of video scene retrieval and highlight detection may include: obtaining, by using a cross-attention model and an encoder, a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query from a video, a first text query that is a positive pair with the video, and an importance token; obtaining, by using the cross-attention model and the encoder, a plurality of second video features and a third importance of the video for the second text query from the video, a second text query that is a negative pair with the video, and the importance token; and modifying the cross-attention model, the encoder, and the importance token such that the first importance is greater than the second importance, the first importance increases, and the second importance and the third importance decrease.

[0012]

[0013] According to one embodiment of the present invention, the influence of text queries on video features can be modeled through cross-attention between text queries and video clips. Therefore, text queries can influence the output values ​​of video scene retrieval and highlight detection, thereby improving the performance of video scene retrieval and highlight detection.

[0014] According to one embodiment of the present invention, the influence of text queries can be effectively learned by training models to reduce the importance of text queries unrelated to video. Therefore, the influence of text queries on the output values ​​of video scene search and highlight detection can be more accurately modeled, thereby improving the performance of video scene search and highlight detection.

[0015] According to one embodiment of the present invention, input-adaptive (query-dependent) importance prediction is possible by utilizing importance tokens whose values ​​change depending on input video and text queries. Therefore, video highlight detection performance can be improved.

[0016]

[0017] FIG. 1 is a schematic block diagram of an electronic device for at least one of video scene retrieval and highlight detection according to one embodiment of the present invention.

[0018] Figure 2 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 1.

[0019] Figure 3 is a conceptual diagram of a cross-attention model used in the electronic device of Figure 1.

[0020] Figure 4 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 1.

[0021] FIG. 5 is a schematic block diagram of an electronic device for learning at least one of video scene search and highlight detection according to one embodiment of the present invention.

[0022] Figure 6 is a conceptual diagram for explaining the electronic device of Figure 5.

[0023] Figure 7 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 5.

[0024] Figure 8 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 5.

[0025] Figures 9 to 12 are flowcharts of the operating method of the electronic device of Figure 1.

[0026] Figures 13 to 16 are flowcharts of the operating method of the electronic device of Figure 5.

[0027] Figure 17 illustrates the results of scene search and highlight detection of a video according to the relevance of text queries (positive pair: related, negative pair: not related) of one embodiment and one comparative example of the present invention.

[0028] Figure 18 shows the results of an experiment to confirm the influence of the configurations of one embodiment of the present invention on the saliency score.

[0029] Figure 19 illustrates the results of scene search and highlight detection of a video according to the relevance of the text query of the present invention.

[0030] [Explanation of symbols]

[0031] 100, 200: Electronic devices

[0032] 110, 210: Storage

[0033] 120, 220: Processor

[0034] MD, MD2: Artificial Intelligence Model

[0035]

[0036] Hereinafter, the operating principles of preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Furthermore, when describing embodiments of the invention, detailed descriptions of related known functions or configurations will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, the terms used below are defined based on their functions in the present invention and may vary depending on the intent or custom of the user or operator. Therefore, the definitions of the terms used should be interpreted based on the contents and corresponding functions throughout this specification.

[0037] FIG. 1 is a schematic block diagram of an electronic device for at least one of video scene retrieval and highlight detection according to one embodiment of the present invention. Referring to FIG. 1, the electronic device (100) may include a storage unit (110) and a processor (120).

[0038] The storage unit (110) can store various data and programs. For example, the storage unit (110) can store an artificial intelligence model (MD, MD-4, see FIGS. 2 and 4, respectively). In one embodiment, the storage unit (110) can store video data and text query data input to the artificial intelligence model. The storage unit (110) can include at least one of volatile memory and non-volatile memory. For example, the volatile memory can include DRAM, SRAM, SDRAM, DDR SDRAM, FeRAM, MRAM, PRAM, PoRAM, or ReRAM. For example, the non-volatile memory can include flash memory, mask ROM, PROM, OTPROM, EPROM, EEPROM, a hard disk, or an optical disk.

[0039] The processor (120) can control the overall operation of the electronic device (100). The processor (120) can control the storage unit (110). For example, the processor (120) can include at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0040] In one embodiment, the processor (120) may obtain a plurality of first video features, each corresponding to a plurality of different points in time or time periods from a video, obtain a text query feature from a text query, obtain a plurality of weights from the plurality of first video features and the text query features, obtain a plurality of second video features from the plurality of weights and the plurality of first video features, obtain a plurality of third video features from the plurality of second video features using an encoder, obtain a plurality of fourth video features from the plurality of third video features and the time query using a decoder, and select at least one of the plurality of different points in time or time periods from the video using the plurality of fourth video features. In one embodiment, the encoder may be an encoder of a transformer, and the decoder may be a decoder of a transformer. In one embodiment, the time query may include a center point and a length defining the time period.

[0041] In one embodiment, the processor (120) may obtain a plurality of video queries from a plurality of first video features to obtain a plurality of weights, obtain a text key from a text query feature, obtain a text value from a text query feature, obtain a plurality of attention scores using the plurality of video queries, text keys, and text values, and obtain a plurality of weights from the plurality of attention scores.

[0042] In one embodiment, the processor (120) may use an encoder to obtain a plurality of third video features and a second importance token from a plurality of second video features and a first importance token to obtain a plurality of third video features.

[0043] In one embodiment, the processor (120) can obtain multiple importance scores for multiple different points in time or time intervals from multiple third video features and second importance tokens.

[0044] Figure 2 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 1. Referring to Figure 2, the artificial intelligence model (MD) is a video encoder (E V ), text encoder (E T ), may include a cross-attention model (MD-1), an encoder (MD-2), and a decoder (MD-3).

[0045] Video Encoder (E V ) are a plurality of first video features (V) each corresponding to a plurality of different time points or time intervals from the video. 1A Inland V tA ) can be obtained. That is, the processor (120) is a video encoder (E V ) to obtain multiple first video features (V ) corresponding to different points in time or time intervals from the video. 1A Inland V tA ) can be obtained. For example, the processor (120) may be a video encoder (E V ) to generate multiple video clips (V1 to V) corresponding to different points in time or time intervals. t ) from multiple first video features (V 1A Inland V tA ) can be obtained. In one embodiment, a video encoder (E V ) may include at least one of a convolutional neural network-based model and a transformer-based model. Each of the first video features (V 1A Inland V tA ) can be a vector.

[0046] Text Encoder (E T ) is a text query feature (T) derived from a text query (T). q ) can be obtained. That is, the processor (120) can obtain a text encoder (E T ) from a text query (T) to a text query feature (T q ) can be obtained. In one embodiment, a text encoder (E T) may include at least one of a recurrent neural network-based model and a transformer-based model. If a text query includes n tokens, the text query features (T q ) can contain n vectors.

[0047] The cross-attention model (MD-1) is a text query feature (T q ) using multiple first video features (V 1A Inland V tA ) by correcting multiple first video features (V 1A Inland V tA ) and text query features (T q ) from multiple second video features (V 1B Inland V tB ) can be obtained. That is, the processor (120) uses a cross-attention model (MD-1) and a text query feature (T q ) using multiple first video features (V 1A Inland V tA ) by correcting multiple first video features (V 1A Inland V tA ) and text query features (T q ) from multiple second video features (V 1B Inland V tB ) can be obtained. The cross-attention model (MD-1) is a text query feature (T q ) using multiple first video features (V 1A Inland V tA ) by correcting the text query (T) and each video clip (V1 to V t ) can model the relationship between them. Therefore, the cross-attention model (MD-1) can contribute to improving the performance of scene search and highlight detection.

[0048] Fig. 3 is a conceptual diagram of a cross-attention model used in the electronic device of Fig. 1. Referring to Fig. 3, the cross-attention model (MD-1) is a plurality of first video features (V 1A Inland VtA ) and text query features (T q ) from multiple weights (W1 to W t ) and obtain multiple weights (W1 to W t ) and multiple first video features (V 1A Inland V tA ) from multiple second video features (V 1B Inland V tB ) can be obtained. That is, the processor (120) uses a cross-attention model (MD-1) to obtain a plurality of first video features (V 1A Inland V tA ) and text query features (T q ) from multiple weights (W1 to W t ) and obtain multiple weights (W1 to W t ) and multiple first video features (V 1A Inland V tA ) from multiple second video features (V 1B Inland V tB ) can be obtained.

[0049] In one embodiment, the processor (120) has a plurality of weights (W1 to W t ), to obtain multiple first video features (V 1A Inland V tA ) from multiple video queries (Q1 to Q t ) and obtain text query features (T q ) from the text key (K) q ) and obtain text query features (T q ) from the text value (V q ) and obtain multiple video queries (Q1 to Q t ), text key (K q ), and text value (V q ) using multiple attention scores (A T-V ) and obtain multiple attention scores (A T-V ) from multiple weights (W1 to W t) can be obtained.

[0050] In one embodiment, the processor (120) uses an artificial neural network, such as a multi-layer perceptron (MLP), to generate a plurality of first video features (V 1A Inland V tA ) from multiple video queries (Q1 to Q t ) can be obtained. In addition, the processor (120) may obtain text query features (T) using an artificial neural network, such as a multilayer perceptron, for example. q ) from the text key (K q ) and text value (V q ) can be obtained. If the text query (T) contains n tokens, the text query features (T q ) can contain n vectors, and the text key (K q ) can contain n vectors and text values ​​(V q ) may include n vectors. The processor (120) uses [Mathematical Formula 1] to generate multiple video queries (Q1 to Q t ), text key (K q ) and text value (V q ) from multiple attention scores (A T-V ) can be obtained.

[0051]

[0052] Here, Q is a video query (Q1 to Q t ) and K q is the text key, and V q is a text value. d is a video query (Q1 to Q t ), text key (K q ), and text value (V q ) is the dimension of the text key (K q ) and video queries (Q1 to Q t ) are dependent on the text key (K q) can be expressed based on the similarity with.

[0053] The processor (120) uses an artificial neural network, such as a multilayer perceptron, to generate multiple attention scores (A). T-V ) from multiple first video features (V 1A Inland V tA ) multiple weights for each (W1 to W t ) can be obtained.

[0054] The processor (120) may, for example, generate a plurality of first video features (V 1A Inland V tA ) each with multiple weights (W1 to W t ) by multiplying each of the second video features (V 1B Inland V tB ) can be obtained. Refer back to Figure 2 below.

[0055] The encoder (MD-2) generates multiple second video features (V 1B Inland V tB ) from multiple third video features (V 1C Inland V tC ) can be obtained. That is, the processor (120) uses the encoder (MD-2) to obtain a plurality of second video features (V 1B Inland V tB ) from multiple third video features (V 1C Inland V tC ) can be obtained. In one embodiment, the encoder (MD-2) may be, for example, an encoder of a transformer.

[0056] The decoder (MD-3) decodes multiple third video features (V 1C Inland V tC ) and time queries (M q ) from multiple fourth video features (V 1D Inland V tD ) can be obtained. That is, the processor (120) uses a decoder (MD-3) to obtain a plurality of third video features (V 1C Inland V tC) and time queries (M q ) from multiple fourth video features (V 1D Inland V tD ) can be obtained. In one embodiment, the decoder (MD-3) may be, for example, a decoder of a transformer. In one embodiment, a plurality of third video features (V 1C Inland V tC ) contains information about the text query (T), so the text query may not be input separately to the decoder (MD-3).

[0057] In one embodiment, a temporal query (M q ) may include a central point defining a time interval and a length of the time interval. For example, if a time query (Mq) includes a central point 100 and a time interval length 20, the time query (Mq) may mean a time interval of 80 to 120. The time query (M q ) may have learned values. In one embodiment, the processor (120, see FIG. 1) may extract features around a central viewpoint and modulate a cross-attention map with a time interval length. The central viewpoint and time interval length may be learned for each layer of the decoder (MD-3).

[0058] In one embodiment, the processor (120) comprises a plurality of fourth video features (V 1D Inland V tD ) can be used to select at least one of multiple different points in time or time intervals (1 to t) of the video. That is, the processor (120) can perform a scene search of the video for a given text query.

[0059] Fig. 4 is a conceptual diagram of an artificial intelligence model used in the electronic device of Fig. 1. Referring to Fig. 4, an encoder (MD-42) of an artificial intelligence model (MD-4) generates a plurality of second video features (V 1B Inland V tB ) and a plurality of third video features (V) from the first importance token (S1). 1C Inland VtC ) and a second importance token (S2). That is, the processor (120) may obtain a plurality of second video features (V) using an encoder (MD-42). 1B Inland V tB ) and a plurality of third video features (V) from the first importance token (S1). 1C Inland V tC ) and a second importance token (S2). For example, the processor (120) may obtain a second video feature (V 1B Inland V tB ) and the first importance token (S1) are concatenated to obtain one input tensor, and multiple third video features (V) are extracted from the input tensor using an encoder (MD-42). 1C Inland V tC ) and a second importance token (S2) can be obtained. The second importance token (S2) can be called an input-adaptive importance predictor and is reconstructed into an input-dependent context by the encoder (MD-42).

[0060] In one embodiment, the processor (120) comprises a plurality of third video features (V 1C Inland V tC ) and a second importance token (S2) for each of multiple different points in time or time intervals. 1C Inland S tC ) can be obtained respectively. In one embodiment, each of the first importance token (S1) and the second importance token (S2) can be a learnable vector and can be randomly set at the start of learning. For example, the processor (120) can obtain a plurality of importance tokens (S) for each of a plurality of different time points or time periods according to [Mathematical Formula 2]. 1C Inland S tC ) can be obtained respectively.

[0061]

[0062] S here iis the importance corresponding to the ith time point or time interval, and FC S () and FCc() are fully connected layers, respectively, S2 is the second importance token, and V i is a multiple third video feature (V 1C Inland V tC ) is the third video feature corresponding to the ith time point or time interval, and d is FC S (S2) and FC C (V i ) is the dimension of. · is the inner product. The encoder (MD-42) uses a text query (T) and a video clip (V1 to V t ) to obtain the importance of each time point or time interval using a different second importance token (S2), so that the importance can be calculated more accurately, and the change in importance according to the text query (T) can be modeled more accurately.

[0063] FIG. 5 is a schematic block diagram of an electronic device for learning at least one of video scene retrieval and highlight detection according to one embodiment of the present invention. Referring to FIG. 5 , the electronic device (200) may include a storage unit (210) and a processor (220).

[0064] The storage unit (210) can store various data and programs. For example, the storage unit (210) can store artificial intelligence models (MD2, MD2-4, see FIGS. 7 and 8, respectively). In one embodiment, the storage unit (210) can store video data and text query data input to the artificial intelligence model. The storage unit (210) can include at least one of volatile memory and non-volatile memory. For example, the volatile memory can include DRAM, SRAM, SDRAM, DDR SDRAM, FeRAM, MRAM, PRAM, PoRAM, or ReRAM. For example, the non-volatile memory can include flash memory, mask ROM, PROM, OTPROM, EPROM, EEPROM, a hard disk, or an optical disk.

[0065] The processor (220) can control the overall operation of the electronic device (200). The processor (220) can control the storage unit (210). For example, the processor (220) can include at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0066] In one embodiment, the processor (220) may obtain, using a cross-attention model and an encoder, a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query from a video, a first text query that is a positive pair with the video, and an importance token, and may obtain, using the cross-attention model and the encoder, a third importance of the video for the second text query from the plurality of second video features and the video, and the first importance may be greater than the second importance, and the cross-attention model, the encoder, and the importance token may be modified such that the first importance increases and the second importance and the third importance decrease.

[0067] In one embodiment, the processor (220) may use a decoder to obtain a plurality of third video features from a plurality of first video features and a time query, and may use the plurality of third video features to select at least one of a plurality of different points in time or time intervals of the video.

[0068] In one embodiment, the processor (220) can modify the cross-attention model, encoder, and decoder to reduce the difference between the selected time point or time interval and the correct time point or time interval.

[0069] In one embodiment, the processor (220) can classify whether at least one selected point in time or time interval is foreground or background, and modify the cross-attention model, encoder, and decoder so that the prediction of whether it is foreground or background is the same as the actual correct answer.

[0070] Fig. 6 is a conceptual diagram for explaining the electronic device of Fig. 5. Referring to Fig. 6, the processor (220) uses an artificial intelligence model (MD2) to generate a first text query (T) that is a pair of a video (V) and a video (V). q1 ), and a plurality of first video features (V) from importance tokens (S). 1C1 Inland V tC1 ) and the first text query (T q1 ) for multiple importance (S) of multiple different points in time or time intervals (1 to t) of a video (V) 1C1 Inland S tC1 ) can be obtained. In addition, the processor (220) uses an artificial intelligence model (MD2) to obtain a second text query (T) which is a pair of video (V) and video (V) and sound. q2 ), and a plurality of second video features (V) from importance tokens (S). 1C2 Inland V tC2 ) and a second text query (T q2 ) for multiple importance (S) of multiple different points in time or time intervals (1 to t) of a video (V)1C2 Inland S tC2 ) can be obtained. In this specification, a positive pair of a video and a text query means that the text query is related to the video, and a negative pair of a video and a text query means that the text query is unrelated to the video.

[0071] Figure 7 is a conceptual diagram of an artificial intelligence model used in the electronic device of Figure 5. First, during the training of the artificial intelligence model (MD2-4), a video encoder (E V ) and text encoder (E T ) may not be fixed and learned. Referring to Fig. 7, the artificial intelligence model (MD2) may include a cross-attention model (MD2-1) and an encoder (MD2-2). The artificial intelligence model (MD2) may be a first text query (T q1 ) for multiple importance (S) of multiple different points in time or time intervals (1 to t) of a video (V) 1C1 Inland S tC1 ) can consider the first importance and the second importance, where the first importance is greater than the second importance. In addition, the artificial intelligence model (MD2) can consider the second text query (T q2 ) can be considered as the third importance of the video (V). That is, the first importance is the first text query (T) among different points in time or time intervals within the video (V). q1 ) is the importance of a point in time or time interval that is highly related to the first text query (T) among different points in time or time intervals within the video (V). q1 ) is the importance of the point in time or time interval with low correlation with the video (V) and the first text query (T q1 ) can be a positive pair. On the other hand, the third importance is the video (V) and the second text query (T) which is a negative pair. q2 ) may be of importance.

[0072] In one embodiment, the processor (220) can modify the artificial intelligence model (MD2) so that the first importance increases and the second importance and the third importance decrease. For example, the processor (220) can modify the cross-attention model (MD2-1), the encoder (MD2-2), and the importance token (S1) so that the first importance increases and the second importance and the third importance decrease. For example, the processor (220) can obtain the margin ranking loss of [Mathematical Formula 3] using the first importance and the second importance, and modify the artificial intelligence model (MD2) so that the margin ranking loss decreases, i.e., the first importance increases and the second importance decreases. In [Mathematical Formula 3], L margin is the margin ranking loss, Δ is the margin, and S high is of first importance, and S low is of second importance.

[0073]

[0074] In one embodiment, the processor (220) can obtain the rank-based comparison loss of [Mathematical Formula 4] and modify the artificial intelligence model (MD2) to reduce the rank-based comparison loss, thereby increasing the first importance and decreasing the second and third importance. By utilizing the rank-based comparison loss, the processor (220) can achieve precise importance prediction.

[0075]

[0076] In [Equation 4], L cont is the rank-based comparison loss, and X r pos means a positive set (with a higher importance than r), and X r negdenotes the negative set (with lower importance than r), τ is the temperature scaling parameter, and S(x) is the importance. R is the maximum rank value, and each video clip mini-batch has a saliency score lower than R. This mini-batch is cycled through R times, and in each cycle, a positive set (X) is selected from samples with higher importance than the cycle index r (r∈{0, 1, ..., R-1}). r pos ) is generated. A negative set (X) is generated from samples of rank lower than the circular index r. r neg ) generates a negative set (X r neg ) may include a negative pair used in the negative loss calculation of [Mathematical Formula 5].

[0077]

[0078] In one embodiment, the processor (220) can reduce the third importance by obtaining the negative loss of [Equation 5] and modifying the artificial intelligence model (MD2) so that the negative loss is reduced. In [Equation 5], L neg is a negative loss, and S neg is the third importance, that is, the importance of video-text queries that are negative pairs. By training the AI ​​model (MD2) using this loss function, we can prevent the AI ​​model (MD2) from obtaining scenes or importance regardless of whether the text query is a positive or negative pair with the video.

[0079] Fig. 8 is a conceptual diagram of an artificial intelligence model used in the electronic device of Fig. 5. Referring to Fig. 8, the processor (220) uses a decoder (MD2-3) to generate a plurality of third video features (V 1C Inland V tC ) and time queries (M q ) from multiple fourth video features (V 1D Inland V tD ) and obtain multiple fourth video features (V 1DInland V tD ) can be used to select at least one of a plurality of different points in time or time intervals of the video. That is, the processor (220) can select a plurality of fourth video features (V 1D Inland V tD ) can be used to perform scene search.

[0080] In one embodiment, the processor (220) can modify the cross-attention model (MD2-1), the encoder (MD2-2), and the decoder (MD2-3) to reduce the difference between the selected time point or time interval and the correct time point or time interval. For example, the processor (220) can obtain the L1 loss and the gIoU loss of [Equation 6], and reduce the difference between the selected time point or time interval and the correct time point or time interval by reducing the L1 loss and the gIoU loss. In [Equation 6], L1 is the L1 loss, m is the correct time point or time interval, is a point in time or time interval selected by the processor (220). The gIoU loss can be calculated according to the definition disclosed in Hamid Rezatofighi et al., "generalized intersection over union: A metric and a loss for bounding box regression", In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.

[0081]

[0082] In one embodiment, the processor (220) can classify whether a selected time point or time interval is a foreground or a background, and modify the cross-attention model (MD2-1), the encoder (MD2-2), and the decoder (MD2-3) so that the prediction of whether it is a foreground or a background is the same as the actual correct answer. For example, the processor (220) can obtain the cross-entropy loss of [Mathematical Formula 7], and modify the cross-attention model (MD2-1), the encoder (MD2-2), and the decoder (MD2-3) so as to reduce the cross-entropy. In [Mathematical Formula 7], L CE is the cross-entropy loss, y is whether it is the actual foreground or background, is whether the processor (220) predicts the foreground or background.

[0083]

[0084] In one embodiment, comprehensively, the processor (220) can obtain the total loss of [Mathematical Formula 8] and train the artificial intelligence model (MD2-4) to reduce the loss.

[0085]

[0086] In [Equation 8], L tot is the total loss, and L hl is the highlight detection loss, and L mr is the scene search loss, and L neg is the negative loss (see [Equation 5]), and λ neg is the balance constant for the negative loss. The highlight detection loss (L hl ) can be used to estimate the importance. Scene search loss (L mr ) can measure the difference between the actual correct time or time interval and the predicted time or time interval. For example, highlight detection loss (L hl ) and scene detection loss (L mr ) can be defined as in [Mathematical Formula 9] and [Mathematical Formula 10], respectively.

[0087]

[0088] In [Equation 9], L hl is the highlight detection loss, and L margin is the margin ranking loss of [Equation 3], and λ margin is the equilibrium constant for margin ranking loss, and L cont is the rank-based comparison loss of [Equation 4], and λ cont is the balance constant for the rank-based comparison loss.

[0089]

[0090] In [Equation 10], L mr is the scene search loss, L1 is the L1 loss of [Equation 6], and λ L1 is the balance constant for L1 loss, and L gIoU is the gIoU loss, and λ gIoU is the balance constant for gIoU loss. L CE is the cross-entropy loss of [Equation 7], and λ CE is the balance constant for cross-entropy loss.

[0091] FIG. 9 is a flowchart of an operating method of the electronic device of FIG. 1. Referring to FIG. 9, the operating method of the electronic device may include a step of obtaining a plurality of first video features corresponding to a plurality of different points in time or time sections from a video (S910), a step of obtaining a text query feature from a text query (S920), a step of obtaining a plurality of weights from the plurality of first video features and the text query features (S930), a step of obtaining a plurality of second video features from the plurality of weights and the plurality of first video features (S940), a step of obtaining a plurality of third video features from the plurality of second video features using an encoder (S950), a step of obtaining a plurality of fourth video features from the plurality of third video features and the time query using a decoder (S960), and a step of selecting at least one of a plurality of different points in time or time sections of the video using the plurality of fourth video features (S970). In one embodiment, the encoder may be an encoder of a transformer, and the decoder may be a decoder of a transformer. In one embodiment, a time query may include a center point and a length defining a time interval.

[0092] FIG. 10 is a flowchart of an operating method of the electronic device of FIG. 1. Referring to FIG. 10, the step of obtaining a plurality of weights (S930, see FIG. 9) may include a step of obtaining a plurality of video queries from a plurality of first video features (S1031), a step of obtaining a text key from a text query feature (S1032), a step of obtaining a text value from a text query feature (S1033), a step of obtaining a plurality of attention scores using a plurality of video queries, text keys, and text values ​​(S1034), and a step of obtaining a plurality of weights from a plurality of attention scores (S1035).

[0093] That is, the operating method of the electronic device of FIG. 10 comprises the steps of: obtaining a plurality of first video features each corresponding to a plurality of different time points or time periods from a video (S1010); obtaining a text query feature from a text query (S1020); obtaining a plurality of video queries from a plurality of first video features (S1031); obtaining a text key from a text query feature (S1032); obtaining a text value from a text query feature (S1033); obtaining a plurality of attention scores using a plurality of video queries, text keys, and text values ​​(S1034); obtaining a plurality of weights from a plurality of attention scores (S1035); obtaining a plurality of second video features from a plurality of weights and a plurality of first video features (S1040); obtaining a plurality of third video features from a plurality of second video features using an encoder (S1050); obtaining a plurality of fourth video features from a plurality of third video features and a time query using a decoder (S1060); and obtaining a plurality of fourth video features using a plurality of fourth video features. It may include a step (S1070) of selecting at least one of a plurality of different points in time or time intervals of the video.

[0094] FIG. 11 is a flowchart of an operating method of the electronic device of FIG. 1. Referring to FIG. 11, the step of obtaining a plurality of third video features (S950, see FIG. 9) may include a step of obtaining a plurality of third video features and a second importance token from a plurality of second video features and a first importance token using an encoder (S1150). That is, the operating method of the electronic device of FIG. 11 may include a step of obtaining a plurality of first video features each corresponding to a plurality of different points in time or time sections from a video (S1110), a step of obtaining a text query feature from a text query (S1120), a step of obtaining a plurality of weights from the plurality of first video features and the text query features (S1130), a step of obtaining a plurality of second video features from the plurality of weights and the plurality of first video features (S1140), a step of obtaining a plurality of third video features and a second importance token from the plurality of second video features and the first importance token using an encoder (S1150), a step of obtaining a plurality of fourth video features from the plurality of third video features and the time query using a decoder (S1160), and a step of selecting at least one of a plurality of different points in time or time sections of the video using the plurality of fourth video features (S1170).

[0095] Fig. 12 is a flowchart of an operating method of the electronic device of Fig. 1. Referring to Fig. 12, the operating method of the electronic device may further include a step (S1280) of obtaining multiple importance levels for multiple different points in time or time periods from multiple third video features and second importance tokens. That is, the operating method of the electronic device of FIG. 12 may include a step of obtaining a plurality of first video features each corresponding to a plurality of different time points or time sections from a video (S1210), a step of obtaining a text query feature from a text query (S1220), a step of obtaining a plurality of weights from the plurality of first video features and the text query features (S1230), a step of obtaining a plurality of second video features from the plurality of weights and the plurality of first video features (S1240), a step of obtaining a plurality of third video features and a second importance token from the plurality of second video features and the first importance token using an encoder (S1250), a step of obtaining a plurality of fourth video features from the plurality of third video features and the time query using a decoder (S1260), a step of selecting at least one of a plurality of different time points or time sections of the video using the plurality of fourth video features (S1270), and a step of obtaining a plurality of importances of a plurality of different time points or time sections from the plurality of third video features and the second importance token (S1280).

[0096] FIG. 13 is a flowchart of an operating method of the electronic device of FIG. 5. Referring to FIG. 13, the operating method of the electronic device may include a step (S1310) of obtaining a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query from a video, a first text query that is a pair of video and positive, and an importance token using a cross-attention model and an encoder, a step (S1320) of obtaining a third importance of the video for the second text query from a plurality of second video features and a second video query using a cross-attention model and an encoder, and a step (S1330) of modifying the cross-attention model, the encoder, and the importance token such that the first importance is greater than the second importance, the first importance increases, and the second importance and the third importance decrease.

[0097] Fig. 14 is a flowchart of an operating method of the electronic device of Fig. 5. Referring to Fig. 14, the operating method of the electronic device may further include a step (S1440) of obtaining a plurality of third video features from a plurality of first video features and a time query using a decoder, and a step (S1450) of selecting at least one of a plurality of different time points or time sections of the video using the plurality of third video features. That is, the operating method of the electronic device of FIG. 14 comprises: a step of obtaining a plurality of first video features, a first importance of a first time point or a first time section of the video for the first text query, and a second importance of a second time point or a second time section of the video for the first text query from a video, a first text query that is a pair of video and positive, and an importance token using a cross-attention model and an encoder (S1410); a step of obtaining a third importance of the video for the second text query from a plurality of second video features and a video using a cross-attention model and an encoder (S1420); a step of modifying the cross-attention model, the encoder, and the importance token so that the first importance is greater than the second importance, and the first importance increases and the second importance and the third importance decreases (S1430); and a step of obtaining a plurality of third video features from the plurality of first video features and the time query using a decoder. Step (S1440), and step (S1450) of selecting at least one of a plurality of different points in time or time intervals of the video using a plurality of third video features.

[0098] Fig. 15 is a flowchart of an operating method of the electronic device of Fig. 5. Referring to Fig. 15, the operating method of the electronic device may further include a step (S1560) of modifying a cross-attention model, an encoder, and a decoder to reduce the difference between a selected time point or time period and a correct time point or time period. That is, the operating method of the electronic device of FIG. 15 comprises: a step of obtaining a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query from a video, a first text query that is a pair of video and positive, and an importance token using a cross-attention model and an encoder (S1510); a step of obtaining a plurality of second video features and a third importance of the video for the second text query from a video, a second text query that is a pair of video and negative, and an importance token using a cross-attention model and an encoder (S1520); a step of modifying the cross-attention model, the encoder, and the importance token so that the first importance is greater than the second importance, and the first importance increases and the second importance and the third importance decreases (S1530); and obtaining a plurality of third video features from the plurality of first video features and the time query using a decoder. The step (S1540) may include a step (S1550) of selecting at least one of a plurality of different time points or time periods of the video using a plurality of third video features, and a step (S1560) of modifying the cross-attention model, the encoder, and the decoder to reduce the difference between the selected time point or time period and the correct time point or time period.

[0099] Fig. 16 is a flowchart of an operating method of the electronic device of Fig. 5. Referring to Fig. 16, the operating method of the electronic device may further include a step (S1660) of classifying whether at least one selected point in time or time period is a foreground or a background, and a step (S1670) of modifying a cross-attention model, an encoder, and a decoder so that the prediction of whether it is a foreground or a background is the same as the actual correct answer. That is, the operating method of the electronic device of FIG. 16 comprises: a step of obtaining a plurality of first video features, a first importance of a first time point or a first time period of the video for the first text query, and a second importance of a second time point or a second time period of the video for the first text query from a video, a first text query that is a pair of video and positive, and an importance token using a cross-attention model and an encoder (S1610); a step of obtaining a third importance of the video for the second text query from a plurality of second video features and a video using a cross-attention model and an encoder (S1620); a step of modifying the cross-attention model, the encoder, and the importance token so that the first importance is greater than the second importance, and the first importance increases and the second importance and the third importance decreases (S1630); and obtaining a plurality of third video features from the plurality of first video features and the time query using a decoder. The step (S1640) may include a step of selecting at least one of a plurality of different viewpoints or time sections of the video using a plurality of third video features (S1650), a step of classifying whether the at least one selected viewpoint or time section is a foreground or a background (S1660), and a step of modifying the cross-attention model, the encoder, and the decoder so that the prediction of whether it is a foreground or a background is the same as the actual correct answer (S1670).

[0100] [Examples and Comparative Examples]

[0101] Example 1: Using video features

[0102] Example 2: Using video features and audio features

[0103] Example 3: Using video features extracted using VGG

[0104] Example 4: Using video features extracted using VGG + audio features

[0105] Example 5: Using video features extracted using C3D

[0106] Example 6: Using video features extracted using Slowfast and CLIP

[0107] Example 7: Including four self-attention layers instead of a cross-attention model

[0108] Example 8: A cross-attention model containing four cross-attention layers.

[0109] First comparative example: BeautyThumb (Yale Song et al., "To click or not to click: Automatic selection of beautiful thumbnails from videos", In Proceedings of the 25 th ACM international on conference on information and knowledge management, 659-668, 2016)

[0110] Second comparative example: DVSE (Wu Liu et al., "Multi-task deep visual-semantic embedding for video thumbnail selection", In Proceedings of the IEEE conference on computer vision and pattern recognition, 3707-3715 2015)

[0111] 제3 비교예: MCN (Lisa Anne Hendricks et al., "localizing moments in video with natural language", In Proceedings of the IEEE international conference on computer vision, 5803-5812, 2017)

[0112] 제4 비교예: CAL (Victor Escorcia et al., "Temporal localization of moments in video collections with natural language", arXiv 2019)

[0113] 제5 비교예: XML (Jie Lei et al., "Tvr: A large-scale dataset for video-subtitle moment retrieval", In European Conference on Computer Vision, 447-463, 2020)

[0114] 제6 비교예: XML+ (Jie Lei et al., "Tvr: A large-scale dataset for video-subtitle moment retrieval", In European Conference on Computer Vision, 447-463, 2020)

[0115] 제7 비교예: Moment-DETR (Jie Lei et al., "Detecting moments and highlights in videos via natural language queries", Advances in Neural Information Processing Systems, 34:11846-11858, 2021)

[0116] 제8 비교예: UMT (Ye Liu et al., "Umt:Unified multi-modal transformers for joint video moment retrieval and highlight detection", In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 3042-3051, 2022)

[0117] 제9 비교예: sLSTM (Ke Zhang et al., "Video summarization with long short-term memory", In European conference on computer vision, 766-782, 2016)

[0118] 제10 비교예: SG (Behrooz Mahasseni et al., "Unsupervised video summarization with adversarial lstm networks", In Proceedings of the IEEE conference on computer Vision and Pattern Recognition, 202-211, 2017)

[0119] 제11 비교예: LIM-S (Bo Xiong et al., "Less is more: Learning highlight detection from video duration", In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 1258-1267, 2019)

[0120] 제12 비교예: Trailer (Lezi Wang et al., "Learning trailer moments in full-length movies with co-contrastive attention", In European Conference on Computer Vision, 300-316, 2020)

[0121] 제13 비교예: SL-Module (Minghao Xu et al., "Cross-category video highlight detection via set-based learning", In Proceedings of the IEEE / CVF International Conference on Computer Vision, 7970-7979, 2021)

[0122] 제14 비교예: MINI-NET (Fa-Ting Hong et al., "Mini-net: Multiple instance ranking network for video highlight detection", In European Conference on Computer Vision, 345-360, 2020)

[0123] 제15 비교예: TCG (Qinghao Ye et al., "Temporal cue guided video highlight detection with low-rank audio-visual fusion", In Proceedings of the IEEE / CVF International Conference on Computer Vision, 7950-7959, 2021)

[0124] Comparative example 16: Joint-VA (Taivanbat Badamdorj et al., "Joint visual and audio learning for video highlight detection", In Proceedings of the IEEE / CVF International Conference on Computer Vision, 8127-8137, 2021)

[0125] Comparative Example 17: SAP

[0126] Comparative Example 18: TripNet

[0127] Comparative Example 19: SM-RL

[0128] Comparative Example 20: MAN

[0129] Comparative Example 21: 2D_TAN

[0130] Comparative Example 22: FVMR

[0131] Comparative Example 23: CTRL

[0132] Comparative Example 24: ACL

[0133] Comparative Example 25: RWM-RL

[0134] Comparison Example 26: DEBUG

[0135] Comparative Example 27: VSLNet

[0136] Figure 17 illustrates the results of scene retrieval and highlight detection of a video according to the relevance of text queries (positive pair: relevant, negative pair: not relevant) of one embodiment and one comparative example of the present invention. Referring to Figure 17, in the case of the seventh comparative example, the importance prediction result was the same regardless of whether the text query was a positive pair or a negative pair. That is, in the case of the seventh comparative example, the influence of the text query on the video scene retrieval and highlight detection was minimal, and a part that was completely different from the actual correct answer was selected. On the other hand, in one embodiment according to the present invention, the importance greatly differed depending on whether the text query was a positive pair or a negative pair, and the selected scene was close to the actual correct scene. This is because, according to the present invention, video features can be made dependent on the text query through learning using cross-attention and negative pairs between the text query and the video.

[0137]

[0138]

[0139] Table 1 compares the performance of comparative examples and embodiments for scene search and highlight detection tasks using the QVHighlights dataset (Jie Lei et al., "Detecting moments and highlights in videos via natural language queries", Advances in Neural Information Processing Systems, 34:11846-11858, 2021). Referring to Table 1, the embodiments of the present invention showed higher performance than the comparative examples. In addition, it can be seen that the performance of the second embodiment of the embodiments of the present invention can be improved by inputting audio as well as video, given that the performance of the second embodiment is higher than that of the first embodiment.

[0140]

[0141]

[0142] Table 2 compares the performance of comparative examples and embodiments on a highlight detection task using the TVsum (Yale Song et al., "TVsum: Summarizing web videos using titles", In Proceedings of the IEEE conference on computer vision and pattern recognition, 5179-5187, 2015) dataset. Referring to Table 2, the embodiments of the present invention generally showed higher performance than the comparative examples.

[0143]

[0144]

[0145] Table 3 compares the performance of comparative examples and embodiments for a scene search task using the Charades-STA dataset (Jiyang Gao et al., "Tall: Temporal activity localization via language query", In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017). Referring to Table 3, the embodiments of the present invention showed higher performance than the comparative examples.

[0146]

[0147]

[0148] Table 4 shows the experimental results for examining the effectiveness of each configuration of the present invention. MR refers to scene search performance, and HD refers to highlight detection performance. Referring to Table 4, embodiment (e) only improved scene search performance compared to (a), because temporal queries only affect the decoder. In contrast, embodiments (b), (c), and (d) improved both scene search performance and highlight detection performance compared to (a). This is because the cross-attention model, negative pair loss, and importance tokens can commonly obtain text query-dependent video representations.

[0149] Figure 18 illustrates the experimental results for verifying the influence of the configurations of one embodiment of the present invention on the saliency score. Referring to Figure 18, in Comparative Example 7, the graphs of positive and negative pairs overlap significantly. That is, in Comparative Example 7, the influence of text queries on the output is small. On the other hand, in the embodiment, the overlap between the graphs of positive and negative pairs is significantly reduced, indicating that text queries have a significant influence on the output.

[0150]

[0151]

[0152] Table 5 presents experimental results to address concerns that the increased number of model parameters resulting from the addition of cross-attention may contribute to the superior performance of the embodiments of the present invention over the comparative examples. * indicates that only text query features are used as encoder conditions and only video features are processed by the decoder. Referring to Table 5, cross-attention outperforms self-attention with the same number of features in the comparative examples. Furthermore, comparing the seventh and eighth embodiments, cross-attention also outperforms self-attention with the same number of features. Therefore, it can be seen that the superior performance of the embodiments of the present invention is due to the nature of cross-attention, not the increased number of parameters.

[0153] Figure 19 illustrates the results of scene retrieval and highlight detection in a video according to the relevance of a text query according to the present invention. Referring to Figure 19, it can be seen that the higher the actual relevance between a text query and a video scene, the higher the importance. For example, negative pairs with low relevance showed the lowest importance, while positive pairs with high relevance showed the highest importance.

[0154] While the embodiments of the present invention have been illustrated and described above, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims and their equivalents.

Claims

1. Storage; and including a processor; The above processor, Using a video encoder, obtain multiple first video features from multiple video clips, Using a text encoder, we obtain text query features from a text query, The above processor uses a cross-attention model, Obtaining a plurality of video queries from the plurality of first video features, Obtain a text key from the above text query features, Obtain text values ​​from the above text query features, Obtaining multiple attention scores using the multiple video queries, the text keys, and the text values, Obtain multiple weights from the multiple attention scores, An electronic device for at least one of video scene search and highlight detection, wherein a plurality of second video features are obtained by multiplying the plurality of weights and the plurality of first video features, respectively.

2. In paragraph 1, The above processor, An electronic device for at least one of video scene search and highlight detection, wherein a plurality of third video features and a second saliency token are obtained from the plurality of second video features and the first saliency token using an encoder of a transformer.

3. In paragraph 2, The above processor, An electronic device for at least one of video scene search and highlight detection, wherein the saliency score of each of the plurality of video clips is obtained by taking the inner product of each of the plurality of output values ​​of the first fully connected layer that has received the plurality of third video features and the output value of the second fully connected layer that has received the second saliency token. 4.(a) A step of obtaining a plurality of first video features from a plurality of video clips using a video encoder; (b) a step of obtaining text query features from a text query using a text encoder; (c) Using the cross-attention model, (c1) a step of obtaining a plurality of video queries from the plurality of first video features; (c2) a step of obtaining a text key from the above text query features; (c3) a step of obtaining a text value from the above text query features; (c4) a step of obtaining a plurality of attention scores using the plurality of video queries, the text keys, and the text values; (c5) a step of obtaining multiple weights from the multiple attention scores; and (c6) A method of operating an electronic device for at least one of video scene search and highlight detection, comprising: obtaining a plurality of second video features by multiplying the plurality of weights and the plurality of first video features, respectively.

5. In paragraph 4, (d) A method of operating an electronic device for at least one of video scene search and highlight detection, further comprising the step of obtaining a plurality of third video features and a second saliency token from the plurality of second video features and the first saliency token using an encoder of a transformer.

6. In paragraph 5, Step (d) above, An operating method of an electronic device for at least one of video scene search and highlight detection, comprising a step of obtaining a saliency score of each of the plurality of video clips by taking the inner product of each of the plurality of output values ​​of the first fully connected layer that has received the plurality of third video features and the output value of the second fully connected layer that has received the second saliency token.

Images