Method and apparatus for weakly-supervised learning video corpus moment retrieval

Abstract

A method of operation of a video segment search device operated by at least one processor comprises the steps of receiving a video including a sentence and a frame described by said sentence, extracting a sentence feature vector and a video feature vector, and generating a video candidate region composed of at least one frame associated with said sentence based on the similarity between said sentence feature vector and said video feature vector.

Description

Technology Field

[0001] The present invention relates to a video segment search method and apparatus based on weak supervised learning, and more specifically, to a technology for searching video segments related to a sentence. Background Technology

[0002] The sentence-related video segment search technology understands the sentence, that is, the given input sentence (Query), and based on the understanding, searches for moments related to the sentence in a large video corpus to provide that time range, that is, the video segment.

[0003] However, artificial intelligence techniques for such video segment search require numerous datasets and corresponding ground truth samples for training. Consequently, this creates a labor-intensive problem where correct answers—specifically, identifying the segment within a video where the scene matching a given sentence is located—must be generated for these vast datasets. The problem to be solved

[0004] The present disclosure provides a method and apparatus for searching for video segments related to sentences using a weakly-supervised learning technique. means of solving the problem

[0005] According to one feature, a method of operation of a video segment search device operated by at least one processor comprises the steps of receiving a video including a sentence and a frame described by said sentence, extracting a sentence feature vector and a video feature vector, and generating a video candidate region composed of at least one frame associated with said sentence based on the similarity between said sentence feature vector and said video feature vector.

[0006] The step of generating the above video candidate region may include generating a similarity curve representing the cosine similarity between a sentence and a video frame based on the similarity between the sentence feature vector and the video feature vector, generating a similarity integral curve by integrating the similarity curve, and generating a plurality of video candidate regions related to the sentence using the similarity integral curve.

[0007] After the step of generating a plurality of video candidate regions related to the above sentence, the method may further include the step of generating a multi-scale video candidate region of various sizes by combining at least two video candidate regions located in mutually adjacent time regions.

[0008] The above similarity curve can be generated using the frame-level cosine similarity between the sentence feature vector and the video feature vector.

[0009] The above frame-level cosine similarity can be calculated for the average pooling of the sentence feature vectors.

[0010] After the above-mentioned search step, the method may further include a step of determining one video candidate region with the highest cosine similarity with a sentence feature vector associated with the sentence in the positive video segment of the video candidate region through a video search model that has undergone control group learning trained to determine hard negative frames in positive video segments.

[0011] The above video search model can learn a first loss representing the probability of selecting a candidate region having the lowest similarity among positive frames.

[0012] The above video search model can additionally learn a second loss representing the probability of selecting negative frames from other videos per batch using negative frames and sentence feature vectors.

[0013] The above video search model can learn a total loss by summing the first loss and the second loss.

[0014] According to another feature, the video segment search device includes a sentence feature extraction unit that extracts a sentence feature vector of an input sentence, a video feature extraction unit that extracts a video feature vector of an input video including a scene represented by the input sentence, and a candidate video region generation unit that generates a video candidate region composed of at least one frame related to the sentence among a plurality of frames constituting the input video, based on the similarity between the sentence feature vector and the video feature vector.

[0015] The above candidate video region generation unit can generate a similarity integral curve by integrating a similarity curve representing the cosine similarity between a sentence and a video frame based on the similarity between the sentence feature vector and the video feature vector, and generate multiple video candidate regions related to the sentence using the similarity integral curve.

[0016] The above candidate video region generation unit can generate a multi-scale video candidate region of various sizes by combining at least two video candidate regions located in mutually adjacent time regions.

[0017] The above video feature extraction unit may include a text encoder that receives a plurality of subtitles describing scenes included in the input video and extracts a plurality of subtitle feature vectors, a video decoder that extracts a plurality of frame feature vectors of a plurality of frames constituting the input video, and a hierarchical encoder that extracts a video feature vector by combining the plurality of subtitle feature vectors and the plurality of frame feature vectors.

[0018] The above video segment search device further includes a hard negative sampling unit that performs control group learning learned to determine hard negative frames in positive video segments, and the control group learning can learn a total loss by summing a first loss representing the probability of selecting a candidate region having the lowest similarity in positive frames, and a second loss representing the probability of selecting negative frames in other videos per batch using negative frames and sentence feature vectors.

[0019] The above candidate video region generation unit can provide as a search result one video candidate region that has the highest cosine similarity with the sentence feature vector related to the input sentence in the positive video segment of the above video candidate region. Effects of the invention

[0020] According to the present disclosure, by introducing a weakly-supervised learning technique, the problem of generating correct answers for a dataset can be mitigated, and at the same time, a video instant search function can be performed.

[0021] In addition, while conventional methods selected a control group from different videos, the present disclosure selects a control group within the same video and thereby ultimately selects the video segment related to the text query, thereby increasing the efficiency of learning. Brief explanation of the drawing

[0022] FIG. 1 is a block diagram showing the configuration of a video segment search device according to one embodiment. Figure 2 illustrates the learning configuration of the video segment search device of Figure 1. FIG. 3 shows a similarity curve according to one embodiment. Figure 4 shows an integral curve derived by integrating the similarity curve of Figure 3. Figure 5 shows the generation of video candidate regions through the integral curve of Figure 4. FIG. 6 shows a multi-scale candidate region according to one embodiment. Figure 7 illustrates the search configuration of the video segment search device of Figure 1. FIG. 8 illustrates a learning procedure of a video segment search device according to one embodiment. FIG. 9 illustrates the search procedure of a video segment search device according to one embodiment. FIG. 10 shows the video segment search results related to a sentence according to one embodiment. Specific details for implementing the invention

[0023] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0024] In the description, drawing symbols and names are provided for convenience of explanation and are not limited to drawing symbols or names.

[0025] In descriptions, when a part is said to "include" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0026] Additionally, terms such as “…part,” “…unit,” and “…module” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0027] The device of the present disclosure is a computing device configured and connected such that at least one processor can perform the operation of the present disclosure by executing instructions. A computer program includes instructions described to enable a processor to perform the operation of the present disclosure and may be stored on a non-transitory computer-readable storage medium. The computer program may be downloaded over a network or sold as a product.

[0028] The artificial intelligence model (AI model) of the present disclosure is a machine learning model that learns at least one task and may be implemented as a computer program executed by a processor. The task learned by the artificial intelligence model may refer to a problem to be solved through machine learning or a task to be performed through machine learning. The artificial intelligence model may be implemented as a computer program executed on a computing device, downloaded via a network, or sold in the form of a product. Alternatively, the artificial intelligence model may be linked with various devices via a network.

[0030] FIG. 1 is a block diagram showing the configuration of a video segment search device according to one embodiment.

[0031] Referring to FIG. 1, the video segment search device (100) is a computing device operated by at least one processor.

[0032] The video segment search device (100) includes an input data processing unit (110), a model learning unit (120), and a search unit (130).

[0033] The input data processing unit (110) receives a scene description sentence and extracts a sentence feature vector, and receives a video and extracts a video feature vector.

[0034] The model training unit (120) receives sentence feature vectors and video feature vectors from the input data processing unit (110) and can train a video search model using the input data as training data.

[0035] The video segment search device (100) uses a video search model trained by the model learning unit (120) to search for video segments related to the input sentence from the video corpus and provides them as search results.

[0036] The video segment search device (100) of FIG. 1 may include all the configurations of FIG. 2 and FIG. 3 to be described later, and the configuration for learning and the configuration for searching will be explained separately.

[0038] FIG. 2 illustrates the learning configuration of the video segment search device of FIG. 1, FIG. 3 shows a similarity curve according to one embodiment, FIG. 4 shows an integral curve derived by integrating the similarity curve of FIG. 3, FIG. 5 shows the generation of a video candidate region through the integral curve of FIG. 4, and FIG. 6 shows a multi-scale candidate region according to one embodiment.

[0039] At this time, in FIG. 1, the configuration related to the video segment search operation of the search unit (130) is briefly shown in FIG. 2.

[0040] Referring to FIG. 2, the video segment search device (100) operates based on a weakly-supervised learning technique and does not have correct information (i.e., time domain information) about where the video scene corresponding to a given sentence is located during the learning process.

[0041] Weakly supervised learning techniques utilize query-video pairing information, which consists of bundles of scene description sentences and long video clips containing those scenes, even though no temporal domain information is provided. This bundle information is more efficient in terms of data generation labor and has lower production costs compared to obtaining temporal domain information for training conventional video segment search systems. Furthermore, if the high accessibility of this bundle information allows for its effective utilization for video segment search systems, significant economic benefits can be obtained.

[0042] The input data processing unit (110) may include a sentence feature extraction unit (111), a video feature extraction unit (112), a candidate video region generation unit (104), and a sentence feature extraction unit (111) and a video feature extraction unit (112).

[0043] The sentence feature extraction unit (111) and the video feature extraction unit (112) receive a pair of scene description sentences and a source video as input. Here, the source video includes a scene (frame) that describes the sentence given as input.

[0044] The sentence feature extraction unit (111) includes a text encoder (101) that extracts a sentence feature vector corresponding to an input sentence.

[0045] The video feature extraction unit (112) includes a text encoder (101), a video encoder (102), and a hierarchical encoder (103). Here, the text encoder (101) may be optionally included.

[0046] The text encoder (101) extracts multiple subtitle feature vectors corresponding to multiple subtitles in frame units received as input.

[0047] The video encoder (102) can receive a source video and extract video feature vectors. The video encoder (102) can receive multiple frames constituting the source video and extract frame feature vectors for the multiple frames.

[0048] The hierarchical encoder (103) receives a video-subtitle pair, such as a plurality of subtitle feature vectors and frame feature vectors (or video feature vectors), and extracts a video feature vector.

[0049] A hierarchical encoder (103) can extract video feature vectors by applying Multi-Head Attention to video-subtitle pairs.

[0050] The model training unit (120) may include a candidate video region generation unit (104) and a hard negative sampling unit (106).

[0051] The candidate video region generation unit (104) generates a text-guided proposal based on a language map, searches for a region related to a given input sentence in the video, and generates multiple candidate video regions that can be selected from the surrounding area of ​​the searched region. By doing so, it becomes possible to perform a more detailed and extensive search of video segments in the video region related to the input sentence.

[0052] The candidate video region generation unit (104) searches for a multi-scale candidate region composed of at least one frame related to a sentence among a plurality of frames based on the similarity between the sentence feature vector and the video feature vector.

[0053] The candidate video region generation unit (104) can search for multi-scale candidate regions with high similarity using a Similarity Integral Curve (SIC).

[0054] The candidate video region generation unit (104) can generate multiple candidate regions related to the scene to be searched and generate fewer candidate regions for other regions. That is, it can generate a group of candidates that can increase the probability of searching for the region to be searched. To this end, the candidate video region generation unit (104) performs a 'Similarity Integral Curve' and 'Multi-scale Proposals Generation'.

[0055] Referring to Figure 3, the graph shows the cosine similarity of the expression with the given sentence according to the video frame, where the area with high similarity is said to be highly related to the given sentence, and the area with low similarity is said to be less related to the given sentence.

[0056] The candidate video region generation unit (104) can generate a similarity integral curve as shown in FIG. 4 by integrating the similarity curve of FIG. 3. The similarity integral curve is defined as in Equation 1.

[0057] [Mathematical Formula 1]

[0058]

[0059] is a function of the integral curve and n is the number of frames. is as a similarity value, is video-query's It means. i represents frame position information. Therefore, is the video-query similarity value corresponding to the i-th video sentence.

[0060] The candidate video region generation unit (104) can generate a video candidate region as shown in Fig. 5 through the integral curve of Fig. 4.

[0061] Referring to Fig. 5, the y-axis is uniformly divided to generate video candidate regions, and the x-axis values ​​corresponding to the divided y-axis represent the positions of the video frames.

[0062] The candidate video region generation unit (104) can generate video candidate regions based on video frame positions. represents video candidate regions. N p represents the total number of video candidate regions.

[0063] In other words, video candidate region It has the values ​​of x1 and y1, and is a video candidate region It has values ​​of x2, y2 and is a video candidate region silver , It can have a value of.

[0064] The dense sections on the y-axis are due to the steep slope of the curve. The reason the slope of the integral curve becomes steeper is that it represents a section with high similarity in terms of the similarity curve. Therefore, it becomes possible to generate a denser video candidate region for such high-similarity sections.

[0065] As the video candidate regions generated as shown in Fig. 5 only contain dense segments, they cannot handle cases where the region to be found is a long segment. Therefore, these segments can be combined to generate video candidate regions of longer segments, i.e., multi-scale candidate regions.

[0066] That is, as shown in FIG. 6, the candidate video region generation unit (104) and By combining Create, and By combining It generates, and By combining It can generate. Also, , , and By combining It can generate.

[0067] In this way, the candidate video region generation unit (104) can generate candidate regions for longer sections by combining each candidate region with adjacent candidate regions. Through this, candidate regions of various sizes can be generated, and more candidate regions can be created at key locations to be found, thereby increasing the probability of search.

[0068] The Hard Negative Sampling unit (105) performs Hard Negative Proposal Sampling and provides the selected Hard Negative Proposal as a negative sample for contrast learning.

[0069] The hard negative sampling unit (105) performs contrastive learning on the video segment search model, and the learning efficiency of the contrastive learning increases as it becomes very similar to the scene describing the sentence. For example, when an input sentence is given as a scene that is ‘a man is riding a bicycle,’ and a video containing a scene that matches it (a scene of riding a bicycle) is provided, if a scene other than the scene of riding a bicycle (a scene of getting off a bicycle) is set as a contrastive, it can be a ‘good’ candidate for contrastive learning because it is a scene with a different meaning, even though it is a video related to the same bicycle.

[0070] The hard negative sampling unit (105) selects a control group from a video containing a scene to be searched, unlike the conventional method of selecting a control group from a video of a ‘different’ video to find such a good candidate, and defines this method as hard negative proposal sampling.

[0071] The hard negative sampling unit (105) calculates the cosine similarity between a pre-generated candidate video representation and a given sentence representation and selects the candidate region with the lowest similarity as a control group. Through this, video language alignment can be achieved even in weakly supervised learning, and video search can ultimately be performed.

[0072] The hard negative sampling unit (105) calculates two contrast learning losses, one is the Hard Negative Proposal Sampling (HNPS) loss, and the other is the original hinge loss.

[0073] The hard negative sampling unit (105) sets the candidate with the highest similarity among the candidate groups in one video as a positive sample and the candidate with the lowest similarity among the candidate groups in the same video as a negative sample, and calculates the hard negative candidate region sampling loss using the positive sample and the negative sample, which is represented by Equation 16.

[0074] The hard negative sampling unit (105) sets the candidate with the highest similarity among the candidate groups in one video as a positive sample and the candidate with the highest similarity among the candidate groups in another video as a negative sample, and can calculate the original hinge loss using the positive sample and the negative sample. This is represented by Equation 15.

[0076] Figure 7 illustrates the search configuration of the video segment search device of Figure 1.

[0077] At this time, the configuration related to the video segment search operation of the search unit (130) in FIG. 1 is briefly shown.

[0078] Referring to FIG. 7, the video segment search device (100) may include an input data processing unit (110) and a search unit (130).

[0079] The sentence feature extraction unit (111) of the input data processing unit (110) includes a text encoder (101) and receives a scene description sentence as input to extract a sentence feature vector.

[0080] The video feature extraction unit (112) of the input data processing unit (110) includes a video encoder (102) and receives a video as input to extract a video feature vector.

[0081] The search unit (130) includes a candidate video region generation unit (104) of the video segment search model that has undergone training as described in FIGS. 1 to 6.

[0082] The candidate video region generation unit (104) receives a sentence feature vector and a video feature vector as input, generates a multi-scale video candidate region based on the similarity between the sentence feature vector and the video feature vector, and outputs the video candidate region with the highest cosine similarity as the final video candidate region, that is, the video segment related to the input sentence.

[0084] FIG. 8 illustrates a learning procedure of a video segment search device according to one embodiment.

[0085] Referring to FIG. 8, the sentence feature extraction unit (111) inputs a scene description sentence into a text encoder (101) to obtain a sentence feature vector corresponding to the sentence (S101).

[0086] The video feature extraction unit (112) inputs multiple subtitles in frame units into the text encoder (101) to obtain multiple subtitle feature vectors (S102).

[0087] The video feature extraction unit (112) inputs a plurality of frames constituting the source video into a video encoder (102) to obtain frame feature vectors of the plurality of frames (S103), and inputs a video-subtitle pair composed of a plurality of subtitle feature vectors and frame feature vectors into a hierarchical encoder (103) to obtain video feature vectors (S104).

[0088] Here, the video (V) is defined as in Equation 2.

[0089] [Mathematical Formula 2]

[0090]

[0091] represents the i-th frame that makes up the video (V). is the number of frames that make up a single video.

[0092] Subtitles (S) are defined as in mathematical formula 3.

[0093] [Mathematical Formula 3]

[0094]

[0095] Is It means the i-th subtitle that matches. is the number of subtitles that make up a single video.

[0096] The text encoder (101) is a pre-trained language model that receives subtitles (S) and text queries (q) as input and extracts context token features of the subtitles and context token features of the text queries.

[0097] Subtitle Context Token Features ( ) is defined as in mathematical formula 4.

[0098] [Mathematical Formula 4]

[0099]

[0100] L is the i-th subtitle This is the number of tokens extracted from.

[0101] Context token features of text queries ( ) is defined as in mathematical formula 5.

[0102] [Mathematical Formula 5]

[0103]

[0104] is the number of tokens extracted from the text query (q).

[0105] The video encoder (102) connects 2-dimensional features from a known video recognition model, ResNet (Residual neural network), and 3-dimensional features from a known video recognition model, SlowFast, through L2 normalization, and this It is defined as, This is a list of d-dimensional video features arranged in frame order. N v means frame length.

[0106] The video characteristics that appear with subtitles are defined as in mathematical formula 6.

[0107] [Mathematical Formula 6]

[0108]

[0109] K is It is the number of frames that match.

[0110] Video features normalized to d-dimensional dimensions through a text encoder (101) and a video encoder (102) ), Subtitle Features( ), Text query features( ) can be defined as in mathematical formula 7, respectively.

[0111] [Mathematical Formula 7]

[0112]

[0113] In mathematical equation 7, PE stands for positional encoding. LN stands for layer normalization.

[0114] is subtitles( It refers to the query token features in ). These are video token features, which are subtitle token features using a hierarchical encoder (103). It is derived by. The hierarchical encoder consists of a Cross-modal Transformer and a Temporal Transformer, and performs local context matching and global context matching.

[0115] Cross-modal Transformer is a video-subtitle pair ( Crossed video features and subtitle features using ) as input( Prints ).

[0116] Temporal Transformer is attended visual features ( After collecting all ) cross-referenced video features ( ) and original features( Attended video features from ) It temporarily outputs ), which is defined as in mathematical formula 8.

[0117] [Mathematical Formula 8]

[0118]

[0119] Here, MultiHead refers to Multi-head Attention.

[0120] The candidate video region search unit (104) generates video candidate regions based on integral curves based on the similarity between sentence feature vectors and video feature vectors, and trains a video search model to generate multi-scale video candidate regions through linear combination of video candidate regions (S105).

[0121] The candidate video region search unit (104) can generate a multi-scale candidate region related to the sentence through the similarity between the sentence and the video subtitle.

[0122] The candidate video region search unit (104) generates multi-scale candidate regions (Multi-scale Proposals, MP) through a Similarity Integral Curve (SIC). Here, multi-scale candidate regions refer to a group of video candidates of various sizes and become candidate regions for final video search.

[0123] The candidate video area search unit (104) Average pooling for, i.e., and and Between them, frame-level cosine similarity, i.e., Calculate. is defined as in mathematical formula 9.

[0124] [Mathematical Formula 9]

[0125]

[0126] [Mathematical Formula 10]

[0127]

[0128] Here, is a learnable parameter. In this case, negative values ​​are truncated to 0.

[0129] Final is similarity along the frame axis as in the previously described mathematical formula 1 It is defined by integrating.

[0130] Based on, multi-scale text guide dog candidate regions second and It can be created by defining. It is equal to mathematical formula 11.

[0131] [Mathematical Formula 11]

[0132]

[0133] It is equal to mathematical formula 12.

[0134] [Mathematical Formula 12]

[0135]

[0136] In Fig. 2, by Divided by length is the k-th position on the y-axis. Is It is the output of the inverse function based on the input.

[0137] Candidate area Is Average along frames in the interval It is generated using and is equal to mathematical formula 13.

[0138] [Mathematical Formula 13]

[0139]

[0140] Multiscale candidate regions derived through Equation 3 It is the same as Figure 3.

[0141] Referring to Fig. 3, and Multiple candidate regions defined by It can generate.

[0142] Higher similarity It generates steep curves in SIC and generates refined candidate regions.

[0143] Combined candidate regions is a progressively combined candidate region It is generated by and is equal to mathematical formula 14.

[0144] [Mathematical Formula 14]

[0145]

[0146] M is the size of the combined candidate regions, and represents a larger area of ​​the video.

[0147] Finally, the candidate video region generation unit (104) generates a multi-scale candidate region matched with a sentence, i.e., It can generate a number of diverse candidate regions that are close to high similarity between sentences and videos.

[0148] The hard negative sampling unit (105) performs contrast learning by applying hard negative sampling, which calculates contrast learning loss using hard negative candidate regions extracted from positive video segments among multi-scale video candidate regions, to a video search model (S106).

[0149] The hard negative sampling unit (105) can determine the positive candidate area, excluding the hard negative candidate area extracted from the positive video section among the multi-scale candidate areas generated by the candidate video area generation unit (104), as the video section related to the sentence.

[0150] The hard negative sampling section (105) is a margin A video search model is contrastively trained to extract hard negative candidate regions from positive videos, which contributes to improving contrast learning.

[0151] The hard negative sampling section (105) is the original hinge loss, i.e., and hard negative candidate region sampling loss, i.e., can calculate.

[0152] Original hinge loss, i.e., In this, negative candidate regions are different videos per batch ( It is selected from ). It can represent the probability of selecting negative frames from different videos per batch using negative frames and sentence feature vectors.

[0153] However, hard negative candidate region sampling loss, i.e., is a positive video Select the candidate region with the lowest similarity from. represents the hinge loss value using the candidate region with the lowest similarity in the positive frames as a negative sample.

[0154] is defined as in mathematical formula 15.

[0155] [Mathematical Formula 15]

[0156]

[0157] is defined as in mathematical formula 16.

[0158] [Mathematical Formula 16]

[0159]

[0160] Total loss is defined as in mathematical formula 17.

[0161] [Mathematical Formula 17]

[0162]

[0163] It contributes to video-level search, It contributes to instantaneous level search, and the total loss uses the hyperparameters α and β. and It is a linear combination of

[0164] The hard negative sampling unit (105) can learn the video search model by comparison using the total loss of Equation 17.

[0166] FIG. 9 illustrates a search procedure of a video segment search device according to one embodiment, and FIG. 10 shows a video segment search result related to a sentence according to one embodiment.

[0167] Referring to FIG. 9, the sentence feature extraction unit (111) inputs a scene description sentence into a text encoder (101) to obtain a sentence feature vector corresponding to the sentence (S201).

[0168] The video feature extraction unit (112) inputs the video to the video encoder (102) to obtain a video feature vector (S202).

[0169] The candidate video region generation unit (104) of the search unit (130) inputs a sentence feature vector and a video feature vector into a video search model to determine the video candidate region with the highest cosine similarity among the multi-scale video candidate regions as the video segment related to the sentence (S203), and outputs the determined video segment as a search result (S204).

[0170] Referring to Figure 10, the results of video segment search are shown.

[0171] The sentence "Phebe covers most of her face with her hand" is presented as input, and R represents the instantaneous search location of the video search network.

[0172] C at the bottom shows candidate video regions generated through a language map-based candidate video region generation technique, demonstrating that detection accuracy can be improved by generating numerous and detailed video regions, particularly near the area to be searched.

[0173] Meanwhile, H represents the video region used for difficult control group selection. While conventionally, a control group was selected from a different video, in the embodiment of the present invention, a control group is selected from a region different from the region to be found within the same video to improve learning efficiency.

[0175] Meanwhile, the quantitative and qualitative results regarding the video search performance capability applied to the embodiments of the present invention described above are as shown in Table 1 below.

[0176] [Table 1]

[0177]

[0178] In Table 1, the existing video segment search techniques used for comparison with the embodiments of the present invention are as follows. 'MEE' is a method for finding a single video that matches a given description among a large number of videos. 'MCN' and 'CAL' are methods for finding segments that match a given description within a set of videos; by combining the two methods, a video that matches a given description is estimated from a large number of videos, and an accurate video segment is estimated from the estimated video. 'XML' and 'HERO' are models that integrate the series of processes (video search, segment search) described above into a single model. 'MEE', 'MCN', 'CAL', 'XML', and 'HERO' are all systems that require labels for the training of the search system, and 'Fully' in 'Fully-supervised learning' is indicated on the left side of Table 1.

[0179] 'Weakly' presents a model based on weakly supervised learning; previously, there were no methods based on weakly supervised learning to search for videos within a large volume of footage and estimate segments from the retrieved videos. Therefore, it utilizes the 'MEE' model for video search, while 'TGA' and 'VLANet' are methods for searching for video segments within a given video using weakly supervised learning.

[0180] Table 1 is a table evaluating video search performance, showing the results of calculating the Intersection of Union (IoU), which indicates how much the predicted search interval and the actual correct interval overlap.

[0181] Table 1 shows the percentage of subjects in the evaluation dataset with an IoU value of 0.7 or higher. Therefore, a higher number indicates that the search was performed well.

[0182] In addition, R@1, R@10, and R@100 are results of checking whether there is a candidate among 1, 10, and 100 search candidates that has an IoU of 0.7 or higher with the correct answer area.

[0183] R@1 means that the desired area was searched in a single attempt, while R@10 means the search was performed successfully after ten attempts, indicating that R@1 is the highest difficulty rating.

[0184] The meanings of Fully and Weakly are as follows: Fully means that the video segment search device has been trained by receiving all time-domain information, while Weakly means that it has been trained using only bundled information without time-domain information. Therefore, it can be seen that the search performed as Weakly is solving a more difficult problem.

[0185] WMRN (Weakly-supervised Moment Retrieval Network), a video search technique according to an embodiment of the present invention, exhibits a higher search performance capability than conventional methods.

[0187] The embodiments of the present invention described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which such program is recorded.

[0188] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 A method of operation of a video segment search device operated by at least one processor, comprising: receiving a video including a sentence and a frame described by said sentence as input and extracting a sentence feature vector and a video feature vector; generating a video candidate region composed of at least one frame associated with said sentence based on the similarity between said sentence feature vector and said video feature vector; and using a video search model that has undergone control group learning to determine a hard negative frame in a positive video segment, determining and outputting one video candidate region with the highest cosine similarity with said sentence feature vector within the positive video segment of said video candidate region as the video segment associated with said sentence. Claim 2 A method of operation according to claim 1, wherein the step of generating the video candidate region comprises: generating a similarity curve representing the cosine similarity between a sentence and a video frame based on the similarity between the sentence feature vector and the video feature vector; generating a similarity integral curve by integrating the similarity curve; and generating a plurality of video candidate regions associated with the sentence using the similarity integral curve. Claim 3 A method of operation according to claim 2, further comprising, after the step of generating a plurality of video candidate regions related to the above sentence, the step of combining at least two video candidate regions located in mutually adjacent time regions to generate a multi-scale video candidate region having various sizes. Claim 4 In paragraph 2, the method of operation wherein the similarity curve is generated using the frame-level cosine similarity between the sentence feature vector and the video feature vector. Claim 5 In paragraph 4, the method of operation wherein the frame-level cosine similarity is calculated for the average pooling of the sentence feature vector. Claim 6 delete Claim 7 In claim 1, the video search model learns a first loss representing the probability of selecting a candidate region having the lowest similarity in positive frames, a method of operation. Claim 8 In claim 7, the above video search model further learns a second loss representing the probability of selecting negative frames from other videos per batch using negative frames and sentence feature vectors, a method of operation. Claim 9 In claim 8, the above video search model learns a total loss obtained by summing the first loss and the second loss, a method of operation. Claim 10 A video segment search device comprising: a sentence feature extraction unit for extracting a sentence feature vector of an input sentence; a video feature extraction unit for extracting a video feature vector of an input video including a scene expressed by the input sentence; and a candidate video segment generation unit that generates a video candidate region composed of at least one frame related to the sentence among a plurality of frames constituting the input video based on the similarity between the sentence feature vector and the video feature vector, and uses a video search model that has undergone control group learning to determine a hard negative frame in a positive video segment, thereby determining one video candidate region with the highest cosine similarity to the sentence feature vector within the positive video segment of the video candidate region as the video segment related to the sentence and providing it as a search result. Claim 11 In claim 10, the candidate video region generation unit generates a similarity integral curve by integrating a similarity curve representing the cosine similarity between a sentence and a video frame based on the similarity between the sentence feature vector and the video feature vector, and generates a plurality of video candidate regions related to the sentence using the similarity integral curve, a video segment search device. Claim 12 In claim 11, the candidate video region generation unit is a video segment search device that generates a multi-scale video candidate region having various sizes by combining at least two video candidate regions located in mutually adjacent time regions. Claim 13 In claim 10, the video feature extraction unit comprises a text encoder that receives a plurality of subtitles describing a scene included in the input video and extracts a plurality of subtitle feature vectors, a video decoder that extracts a plurality of frame feature vectors of a plurality of frames constituting the input video, and a hierarchical encoder that extracts a video feature vector by combining the plurality of subtitle feature vectors and the plurality of frame feature vectors, a video segment search device. Claim 14 A video segment search device according to claim 13, further comprising a hard negative sampling unit that performs control group learning learned to determine hard negative frames in positive video segments, wherein the control group learning learns a total loss by summing a first loss representing the probability of selecting a candidate region having the lowest similarity in positive frames, and a second loss representing the probability of selecting negative frames in other videos per batch using negative frames and sentence feature vectors. Claim 15 delete

Images