Apparatus and method for learning deep learning model, and apparatus and method for searching video based on the deep learning model

Abstract

A device for training a deep learning model according to one embodiment may include an input unit that receives a training video and text for said training video, and a training unit that trains said deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text.

Description

Technology Field

[0001] The present invention relates to a device for searching for videos based on a deep learning model and a method for training the deep learning model, and more specifically, to a device and method for training the deep learning model and searching for videos using structural features between videos and text. Background Technology

[0002] Video search is the task of finding videos related to a given query among multiple videos. Existing deep learning-based video search methods utilize the similarity between video embeddings and text embeddings, and deep learning models are also trained to make the embeddings of paired videos and texts similar to each other. Recently, models utilizing the hierarchical structure of video frames, clips, and videos, as well as text words, sentences, and paragraphs, have been proposed.

[0003] A model utilizing the hierarchical structure of video and text includes, as illustrated in FIG. 1, a clip encoder (110) that generates a clip embedding sequence containing multiple clip embeddings based on multiple frame embeddings, a video encoder (120) that generates a video embedding based on the clip embedding sequence, a sentence encoder (130) that generates a sentence embedding sequence containing multiple sentence embeddings based on multiple word embeddings, and a paragraph encoder (140) that generates a paragraph embedding based on the sentence embedding sequence. However, models utilizing such a hierarchical structure, like existing deep learning models, are also trained simply to increase the vector similarity between clip embeddings and sentence embeddings and the vector similarity between video embeddings and paragraph embeddings, respectively, and thus do not show significant performance improvement. Therefore, a method is required to improve the video search performance and efficiency of deep learning models. Prior art literature

[0004] Republic of Korea Published Patent Application No. 10-2019-0106865, Date of publication September 18, 2019. The problem to be solved

[0005] The technical problem of the present invention is to provide an apparatus and method that can improve both the performance and efficiency of video search of a deep learning model by utilizing structural features between video and text, and an apparatus and method for searching for video based on said deep learning model.

[0006] Another technical objective of the present invention is to provide an apparatus and method capable of reducing the learning parameters of a deep learning model for video search, and an apparatus and method for searching for videos based on said deep learning model.

[0007] Another technical objective of the present invention is to provide an apparatus and method capable of improving the video search performance of a deep learning model for video search through learning common information between video clips and text paragraphs, and an apparatus and method for searching for videos based on said deep learning model.

[0008] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0009] According to one embodiment of the present invention, a device for training a deep learning model may include an input unit that receives a training video and text for said training video, and a learning unit that trains the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text.

[0010] Here, the deep learning model may include a first encoder that generates the clip embedding sequence based on frame embeddings for the training video, a second encoder that generates the sentence embedding sequence based on word embeddings for the text, and a third encoder that generates the video embedding and the paragraph embedding based on the clip embedding sequence and the sentence embedding sequence.

[0011] In addition, the learning unit can mix the clip embedding sequence and the sentence embedding sequence and input them to the third encoder.

[0012] In addition, the learning unit can input the sentence embeddings that are paired with the clip embeddings in the clip embedding sequence among the sentence embeddings in the sentence embedding sequence into the third encoder by replacing them with the clip embeddings.

[0013] In addition, the learning unit can train the deep learning model based on the loss function so that the clip embedding sequence and the sentence embedding sequence have the same amount of information.

[0014] In addition, the device for training the deep learning model further includes an evaluation unit configured to evaluate the performance of the deep learning model, wherein the evaluation unit can evaluate the performance of the deep learning model by inputting the clip embedding sequence and the sentence embedding sequence, respectively, into the third encoder.

[0015] Additionally, the learning unit may input to the third encoder some clip-sentence embedding pairs having identical information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence by removing some of them.

[0016] According to another embodiment of the present invention, a video search device includes an input unit that receives a query, a search unit that searches for a video related to the query among a plurality of videos using a deep learning model based on the query, and an output unit that outputs information about the video, wherein the deep learning model may be trained to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from text about the training video.

[0017] Here, the search unit can search for videos related to the query based on the similarity between at least one of the video embedding and the paragraph embedding and the query.

[0018] According to another embodiment of the present invention, a video search method using a video search device comprises the steps of receiving a query, searching for a video related to the query among a plurality of videos using a deep learning model based on the query, and outputting information about the video, wherein the deep learning model may be trained to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from text about the training video.

[0019] According to another embodiment of the present invention, a method for training a deep learning model using a deep learning model training device may include the steps of receiving a training video and text for said training video, and training the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text.

[0020] According to another embodiment of the present invention, a computer program stored in a computer-readable recording medium may include instructions for the processor to perform a deep learning model training method, wherein the deep learning model training device, when executed by a processor, includes the steps of receiving a training video and text for said training video, and training the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text.

[0021] According to another embodiment of the present invention, a computer-readable recording medium storing a computer program may include instructions for the processor to perform a deep learning model training method, wherein the deep learning model training device, when executed by a processor, includes the steps of receiving a training video and text for said training video, and training the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text. Effects of the invention

[0022] According to one embodiment of the present invention, since video encoding and paragraph encoding are performed with a single encoder by utilizing the feature that a clip sequence of a video and a sentence sequence of text have a one-to-one correspondence, the number of training parameters of a deep learning model is reduced, and thus the memory usage and training time of the deep learning model can be reduced.

[0023] According to one embodiment of the present invention, video search performance can be improved because a deep learning model learns common information between clips and paragraphs through mixing of clip embeddings for videos and sentence embeddings for text and clip-paragraph pair-based data augmentation. Brief explanation of the drawing

[0024] Figure 1 is a diagram showing a deep learning model utilizing the hierarchical structure of existing video and text. FIG. 2 is a drawing showing a deep learning model training device according to one embodiment of the present invention. FIG. 3 is a diagram showing a deep learning model according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a clip-sentence pair mixed embedding learning method according to one embodiment of the present invention. FIG. 5 is a diagram illustrating a clip-sentence pair-based data augmentation method according to one embodiment of the present invention. FIG. 6 is a diagram illustrating a method for training a deep learning model according to an embodiment of the present invention. FIG. 7 is a drawing showing a video search device according to an embodiment of the present invention. FIG. 8 is a diagram illustrating a video search method according to an embodiment of the present invention. Specific details for implementing the invention

[0025] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0026] We will briefly explain the terms used in this document and then describe the present invention in detail.

[0027] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0028] When any part of this document is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0029] Additionally, as used in this document, the term "part" refers to software or hardware components, such as FPGAs or ASICs, and "parts" perform certain roles. However, "parts" are not limited to software or hardware. "Parts" may be configured to reside in addressable storage media or configured to run on one or more processors. Thus, by example, "parts" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts."

[0030] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.

[0031] FIG. 2 is a drawing showing a deep learning model training device according to one embodiment of the present invention.

[0032] Referring to FIG. 2, a device (200) for training a deep learning model according to one embodiment of the present invention may include an input unit (210), a learning unit (220), and an evaluation unit (230).

[0033] The input unit (210) can receive a learning video and text about the learning video from the user through an input interface.

[0034] The learning unit (220) can train the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from the training video and a sentence embedding sequence generated from the text. Here, the deep learning model may include a clip encoder that generates the clip embedding sequence based on frame embeddings for the training video, a sentence encoder that generates the sentence embedding sequence based on word embeddings for the text, and a video-paragraph encoder that generates the video embeddings and paragraph embeddings using the loss function based on the clip embedding sequence and the sentence embedding sequence.

[0035] A deep learning model according to one embodiment of the present invention can generate video embeddings and paragraph embeddings with a single encoder (video-paragraph encoder), unlike a structure in which video embeddings and sentence embeddings are generated separately by a video encoder and a sentence encoder in a conventional hierarchical structure model. In addition, the effect of making the clip embedding sequence and the sentence embedding sequence, which are input values ​​for the video-paragraph encoder, similar or identical to each other can be obtained by using a loss function set so that the video embeddings and the paragraph embeddings become similar or identical to each other. That is, the learning unit (220) can train the deep learning model based on the loss function so that the clip embedding sequence and the sentence embedding sequence have the same amount of information.

[0036] Meanwhile, the learning unit (220) can input a mixture of video embeddings and paragraph embeddings into the video-paragraph encoder. As an example, the learning unit (220) can input a sentence embedding that is paired with a clip embedding in a clip embedding sequence among the sentence embeddings in a sentence embedding sequence into the video-paragraph encoder by replacing the clip embeddings with each other. That is, during the training phase of the deep learning model, the clip embeddings and sentence embeddings can be input into the video-paragraph encoder in a mixed state.

[0037] Additionally, the learning unit (220) may input to the video-paragraph encoder some clip-sentence embedding pairs that have identical information among the clip embeddings in the clip embedding sequence and sentence embeddings in the sentence embedding sequence. In this case, the video embedding and paragraph embedding can be generated as a clip embedding sequence and a sentence embedding sequence from which some clip-sentence pairs have been removed. Through this, it is expected that the video embedding and paragraph embedding will be generated in which the information of the removed clip-sentence pairs is lost. Since each clip-sentence pair has a one-to-one correspondence with the same semantic information, the video embedding and paragraph embedding generated from the clip embedding sequence and sentence embedding sequence from which the clip-sentence pairs have been removed also form a one-to-one correspondence with the same information. Therefore, by removing clip-sentence pairs that have the same information within the clip sequence and sentence sequence, data expansion can be induced without data corruption.

[0038] The evaluation unit (230) can evaluate the performance of the deep learning model learned by the learning unit (220). At this time, the evaluation unit (230) can evaluate the performance of the deep learning model by inputting the clip embedding sequence and the sentence embedding sequence into the video-paragraph encoder, respectively. That is, in the evaluation stage of the deep learning model, the clip embedding sequence and the sentence embedding sequence can be input into the video-paragraph encoder without being mixed.

[0039] FIG. 3 is a diagram showing a deep learning model according to an embodiment of the present invention.

[0040] Referring to FIG. 3, a deep learning model according to one embodiment of the present invention includes a clip encoder (310), a sentence encoder (320), and a video-paragraph encoder (330).

[0041] The clip encoder (310) generates clip embeddings by encoding multiple frame embeddings, and the sentence encoder (320) generates sentence embeddings by encoding multiple word embeddings. The clip embedding sequence and the paragraph embedding sequence used as inputs to the video-paragraph encoder (330) can be generated by performing an attention operation on the clip embeddings generated by the clip encoder (310) and the sentence embeddings generated by the sentence encoder (320), respectively.

[0042] The video-paragraph encoder (330) generates video embeddings by encoding a clip embedding sequence and generates paragraph embeddings by encoding a sentence embedding sequence. That is, the video-paragraph encoder (330) generates video embeddings by encoding a clip embedding sequence when input and generates paragraph embeddings by encoding a sentence embedding sequence when input. When a deep learning model according to one embodiment of the present invention is implemented as a transformer model, it can be expressed as shown in the following mathematical formula 1.

[0043]

[0044] Here, and represents the clip embedding sequence and sentence embedding sequence, respectively, and v and p represent the video embedding and paragraph embedding, respectively.

[0045] A clip embedding sequence includes multiple clip embeddings and context information therefor. A sentence embedding sequence includes multiple sentence embeddings and context information therefor. Clip embeddings and sentence embeddings can be composed of tensor sequences of the same form.

[0046] In the existing Transformer model, the video encoder and the paragraph encoder have completely identical structures and are distinguished based on whether the input is a clip embedding or a sentence embedding. Therefore, according to the present invention, based on the fact that the clip sequence and the sentence sequence have a one-to-one correspondence, the video encoder and the paragraph encoder can be merged to generate video embeddings and paragraph embeddings as a single video-paragraph encoder (300).

[0047] In FIG. 3, the video embedding and paragraph embedding generated by the video-paragraph encoder (330) can be similar or identical to each other by a loss function such as the following Equation 2, and as a result, the clip embedding sequence and sentence embedding sequence, which are input values ​​of the video-paragraph encoder, can also be similar or identical to each other.

[0048]

[0049] Here, , , The loss function for contrast learning may be a loss function that makes the similarity between paired embeddings greater than the similarity between unpaired embeddings. For example, when the first video embedding-first paragraph embedding and the second video embedding-second paragraph embedding are each paired, the loss function may be set so that the similarity between the paired first video embedding-first paragraph embedding is greater than the similarity between the unpaired first video embedding-second paragraph embedding or second video embedding-first paragraph embedding.

[0050] also, is a loss function for contrast learning, and may be a loss function set such that the similarity between the first video embedding-second video embedding pair and the first paragraph embedding-second paragraph embedding pair is reduced.

[0051] also, It may be a loss function configured to generate clip embeddings similar to the sentence embedding (e.g., using the soft nearest neighbor method), find the locations (soft indices) of sentence embeddings similar to the generated clip embeddings, and reduce the difference between the locations of the original sentence embedding and the calculated similar sentence embeddings. That is, It may be a loss function that finds (generates) clip embeddings similar to a specific sentence embedding and, when similar sentence embeddings are found again from similar clip embeddings, produces the original sentence embedding.

[0052] λ c can be a constant term. For example, λ c It can be set to 0.001.

[0053] one side, , , In this, L, H, and g may indicate from which embeddings the loss function was calculated. For example, L may represent a loss function calculated from pairs of clip embeddings and sentence embeddings generated by a clip encoder and a sentence encoder. Also, H may represent a loss function calculated from pairs of paragraph embeddings and video embeddings generated by a paragraph-video encoder, and g may represent a loss function calculated from pairs of global clip embeddings and global sentence embeddings generated by inputting a frame sequence corresponding to the entire video and a word sequence corresponding to the entire paragraph into a clip encoder and a sentence encoder.

[0054] Therefore, paired clip embeddings and sentence embeddings, and paired video embeddings and paragraph embeddings can have the same amount of information and can be trained to have the same amount of information during the training process. Here, the same amount of information may refer to the same embedding vector.

[0055] FIG. 4 is a diagram illustrating a clip-sentence pair mixed embedding learning method according to one embodiment of the present invention.

[0056] Referring to FIG. 4, a deep learning model according to an embodiment of the present invention can generate video embeddings and paragraph embeddings by substituting each pair of embeddings in a clip embedding sequence and a sentence embedding sequence. FIG. 4 illustrates, as an example, a case in which a second clip embedding among the embeddings included in a clip embedding sequence is substituted with a second sentence embedding that is paired with it. In this case, the clip embedding sequence includes the second paragraph embedding, and the sentence embedding sequence includes the second clip embedding. A clip embedding sequence mixed with a sentence embedding or a sentence embedding sequence mixed with a clip embedding may be input to the video-paragraph encoder (330).

[0057] To this end, the deep learning model training device can mix paired embeddings in the clip embedding sequence and the sentence embedding sequence based on the following mathematical formula 3.

[0058]

[0059] Here, is a mixed clip embedding sequence Represents my i-th clip embedding, and is a mixed sentence embedding sequence Represents my i-th sentence embedding. is a random number that has a value of 0 or 1. represents contextual information for a clip embedding sequence, and represents contextual information for a sentence embedding sequence.

[0060] The deep learning model training device can determine whether to use existing embeddings or to substitute embeddings for each embedding sequence based on the random number.

[0061] In the training phase, the final video embeddings and paragraph embeddings use mixed clip embedding sequences and mixed sentence embedding sequences. It becomes, and in the evaluation phase, using unmixed clip embedding sequences and sentence embedding sequences You can evaluate the performance of a deep learning model using .

[0062] FIG. 5 is a diagram illustrating a clip-sentence pair-based data augmentation method according to one embodiment of the present invention.

[0063] Referring to FIG. 5, a deep learning model training device according to one embodiment of the present invention can generate video embeddings and paragraph embeddings by removing some embedding pairs from a clip embedding sequence and a sentence embedding sequence.

[0064] FIG. 5 illustrates, as an example, a case in which a second clip embedding is removed from among the embeddings included in a clip embedding sequence, and a second sentence embedding paired with the second clip embedding is removed from a sentence embedding sequence. In this case, the clip embedding sequence from which the second clip embedding has been removed, or the sentence embedding sequence from which the second sentence embedding has been removed, may be input to the video-paragraph encoder (330). In this case, it can be expected that video embeddings and paragraph embeddings are generated in which the information possessed by the removed clip-sentence embedding pair is lost. Since each clip-sentence embedding pair has a one-to-one correspondence with semantically identical information, the video embeddings and paragraph embeddings generated from the clip embedding sequence and sentence embedding sequence from which the clip-sentence embedding pair has been removed also possess identical information. Therefore, by removing clip-sentence embedding pairs containing identical information within the clip embedding sequence and sentence embedding sequence, data augmentation without unintended data corruption can be induced.

[0065] A deep learning model training device can remove some embedding pairs from clip embedding sequences and sentence embedding sequences using a clip-sentence pair-based data augmentation algorithm such as the one shown in Table 1 below.

[0066] for number of training iterations do if do end if Calculate loss and Update modelend for

[0067] FIG. 6 is a diagram illustrating a method for training a deep learning model according to an embodiment of the present invention.

[0068] Referring to FIG. 6, a deep learning model training device according to one embodiment of the present invention can receive a training video and text about the training video from a user (S600).

[0069] The deep learning model training device can train the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from the texts (S610).

[0070] For example, a deep learning model training device may acquire frame embeddings for the training video and word embeddings for the text, input the frame embeddings into a clip encoder to generate a clip embedding sequence, and input the word embeddings into a sentence encoder to generate a sentence embedding sequence. Then, the clip embedding sequence and the sentence embedding sequence may be input into a video-paragraph encoder to generate video embeddings and paragraph embeddings. Here, an attention operation may be performed on the clip embedding sequence and the sentence embedding sequence, and the final clip embedding sequence and sentence embedding sequence to which the attention operation has been applied may be input into the video-paragraph encoder, respectively.

[0071] At this time, the deep learning model training device may input the clip embedding sequence and the sentence embedding sequence mixed together into the video-paragraph encoder when generating the video embedding and paragraph embedding. For example, the deep learning model training device may substitute the sentence embedding that is paired with the clip embedding in the clip embedding sequence among the sentence embeddings in the sentence embedding sequence with the clip embedding in the clip embedding sequence and input it into the video-paragraph encoder, thereby allowing a sequence in which the clip embedding and paragraph embedding are mixed to be encoded in the video-paragraph encoder. The deep learning model training device may train the deep learning model based on the loss function so that the clip embedding sequence and the sentence embedding sequence have the same amount of information.

[0072] Meanwhile, the deep learning model training device can input the video embedding and paragraph embedding as a sequence in which the clip embedding and paragraph embedding are mixed, by removing some of the clip-sentence embedding pairs having identical information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence.

[0073] When the training of the deep learning model is completed through the process described above, the deep learning model training device can evaluate the performance of the deep learning model by inputting the clip embedding sequence and the sentence embedding sequence, respectively, into the video-paragraph encoder (S620). That is, in the training stage, the video-paragraph encoder is trained using a sequence in which the clip embedding and paragraph embedding are mixed, and in the evaluation stage, the performance of the video-paragraph encoder and / or the deep learning model can be evaluated using the respective clip embedding sequence and paragraph embedding sequence in which the clip embedding and paragraph embedding are not mixed.

[0074] Tables 2 and 3 below show the evaluation results when a deep learning model is trained according to one embodiment of the present invention.

[0075] Sentence -> Clip R@1 R@5 R@10 Mean COOT 16.7±0.4 40.2±0.3 52.3±0.5 - Baseline 16.80±0.22 40.29±0.54 52.80±0.50 73.36±3.89 + ① 16.93±0.14 40.47±0.68 53.03±0.64 71.41±4.29 + ①, ② 16.93±0.35 40.79±0.56 53.33±0.43 71.44±3.87 + ①, ③ 16.91±0.29 40.62±0.61 53.40±0.36 70.81±2.29 + ①, ②, ③ 17.09±0.24 40.34±0.61 52.75±0.58 73.40±3.37

[0076] Paragraph -> Video R@1 R@5 R@10 Mean COOT 77.2±1.0 95.8±0.8 97.5±0.3 - Baseline 76.31±0.86 95.27±0.74 97.71±0.52 2.35±0.17 + ① 77.74±2.19 94.95±0.81 97.04±0.50 2.41±0.23 + ①, ② 79.50±1.96 95.19±0.63 96.90±0.38 2.66±0.14 + ①, ③ 80.15±1.39 95.81±0.75 97.84±0.32 2.32±0.21 + ①, ②, ③ 81.74±1.47 96.19±0.40 97.86±0.37 2.27±0.34

[0077] In Tables 2 and 3, ① represents the merging of upper-layer modules (video encoder, paragraph encoder), ② represents the clip-sentence mixed embedding learning method, and ③ represents the clip-sentence pair-based data augmentation method, respectively. R@K indicates whether related objects exist among the top K predicted to be related, and Mean represents the rank of the related objects. COOT (Cooperative Hierarchical Transformer) represents the existing deep learning model.

[0078] Referring to Tables 2 and 3, it can be seen that the deep learning model training method according to the present invention achieves significant performance improvement. It can be seen that the video search model trained according to the deep learning model training method according to the present invention shows a performance improvement of 0.29%p in sentence-clip search R@1 and 5.43%p in paragraph-video search R@1 compared to the baseline.

[0079] Table 4 below shows the results of comparing the number of training parameters and training time of a deep learning model trained according to the baseline model and the deep learning model training method according to the present invention.

[0080] model Number of learning parameters Learning time (s / epoch) Baseline 7.01M 10.22 Ours 5.23M 9.03

[0081] Referring to FIG. 4, it can be seen that the deep learning model training method according to the present invention has fewer training parameters and improved training speed because it generates video embeddings and paragraph embeddings with a single encoder.

[0082] FIG. 7 is a drawing showing a video search device according to an embodiment of the present invention.

[0083] Referring to FIG. 7, the video search device (700) includes an input unit (710), a search unit (720), and an output unit (730).

[0084] The input section (710) can receive a query from the user.

[0085] The search unit (720) can search for a video related to the query among a plurality of videos based on a query input through the input unit (710). At this time, the search unit (720) can search for a video related to the query using a deep learning model that has been trained by the deep learning model training method described above. Here, the deep learning model can be trained to generate video embeddings and paragraph embeddings using a pre-set loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from text about the training video. At this time, the clip embedding sequence and the sentence embedding sequence can be input to a single upper-level encoder (video-paragraph encoder), and the paired embeddings in each embedding sequence can be mixed with each other or partially removed according to their similarity.

[0086] The search unit (720) can search for videos related to the embedding for the query based on the similarity between at least one of the video embedding and the paragraph embedding and the query.

[0087] The output unit (730) can output information about the video searched by the search unit (720).

[0088] FIG. 8 is a diagram illustrating a video search method according to an embodiment of the present invention.

[0089] Referring to FIG. 8, when a query is input by a user (S800), the video search device can generate an embedding for the query and, based on this, search for a video related to the query among a plurality of videos using the deep learning model described above (S810). The deep learning model may be a deep learning model trained by the deep learning model training method described above, and can search for video embeddings and / or paragraph embeddings similar to the embedding for the query, and can output information about the video related to the searched embedding (S820).

[0090] Meanwhile, each step included in the deep learning model training method and video search method according to the above-described embodiment may be implemented as a computer program recorded on a recording medium, including instructions for a processor to perform these steps.

[0091] In addition, each step included in the deep learning model training method and the video search method according to the above-described embodiment may be implemented on a computer-readable recording medium on which a computer program including instructions for a processor to perform such steps is recorded.

[0092] Combinations of each step of each flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in each step of the flowchart. Since these computer program instructions may also be stored in a computer-available or computer-readable recording medium that can be oriented toward the computer or other programmable data processing equipment to implement the function in a specific manner, the instructions stored in the computer-available or computer-readable recording medium may also produce a manufactured item containing instruction means for performing the function described in each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each step of the flowchart.

[0093] Additionally, each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps described in succession may actually be performed substantially simultaneously, or the steps may sometimes be performed in reverse order according to the corresponding function.

[0094] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0095] 200: Deep learning model training device 210: Input section 220: Learning Department 230: Evaluation Department

Claims

Claim 1 An apparatus for training a deep learning model comprises: an input unit for receiving a training video and text regarding the training video; and a training unit for training the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from the training video and a sentence embedding sequence generated from the text; wherein the deep learning model comprises: a first encoder that generates the clip embedding sequence based on frame embeddings regarding the training video; and a second encoder that generates the sentence embedding sequence based on word embeddings regarding the text. A device comprising: a third encoder that generates the video embedding and the paragraph embedding based on the clip embedding sequence and the sentence embedding sequence; wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship, and the learning unit trains the deep learning model so that the clip embedding sequence and the sentence embedding sequence have the same embedding vector based on the loss function, and trains the deep learning model to generate the video embedding and paragraph embedding with information lost by removing some of the clip-sentence embedding pairs having the same information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence and inputting them into the third encoder. Claim 2 delete Claim 3 A device according to claim 1, wherein the learning unit mixes the clip embedding sequence and the sentence embedding sequence and inputs them into the third encoder. Claim 4 A device according to paragraph 3, wherein the learning unit inputs to the third encoder a sentence embedding paired with a clip embedding in the clip embedding sequence among the sentence embeddings in the sentence embedding sequence by replacing the clip embedding with the sentence embedding. Claim 5 delete Claim 6 An apparatus according to claim 1, further comprising an evaluation unit configured to evaluate the performance of the deep learning model, wherein the evaluation unit evaluates the performance of the deep learning model by inputting the clip embedding sequence and the sentence embedding sequence, respectively, into the third encoder. Claim 7 delete Claim 8 A video search device comprising: an input unit for receiving a query; a search unit for searching for a video related to the query among a plurality of videos using a deep learning model based on the query; and an output unit for outputting information about the video, wherein the deep learning model is trained to generate video embeddings and paragraph embeddings using a pre-set loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from text regarding the training video, wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship, and the clip embedding sequence and the sentence embedding sequence are trained to have the same embedding vector based on the loss function, and wherein the deep learning model is trained to generate video embeddings and paragraph embeddings with lost information by inputting information in which some of the clip-sentence embedding pairs having the same information among the clip embeddings within the clip embedding sequence and the sentence embeddings within the sentence embedding sequence have been removed. Claim 9 In claim 8, the device is characterized in that the search unit searches for videos related to the query based on the similarity between at least one of the video embedding and the paragraph embedding and the query. Claim 10 A method for searching for a video using a video search device, comprising: a step of receiving a query; a step of searching for a video related to the query among a plurality of videos using a deep learning model based on the query; and a step of outputting information about the video, wherein the deep learning model is trained to generate video embeddings and paragraph embeddings using a pre-set loss function based on a clip embedding sequence generated from a training video and a sentence embedding sequence generated from text about the training video, wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship, and the clip embedding sequence and the sentence embedding sequence are trained to have the same embedding vector based on the loss function, and wherein the deep learning model is trained to generate video embeddings and paragraph embeddings with lost information by inputting information in which some of the clip-sentence embedding pairs having the same information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence have some of the clip-sentence embedding pairs having the same information are removed. Claim 11 A method according to claim 10, wherein the searching step comprises the step of searching for a video related to the query based on the similarity between at least one of the video embedding and the paragraph embedding and the query. Claim 12 A method for a deep learning model training device to train a deep learning model comprises: a step of receiving a training video and text regarding the training video; and a step of training the deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from the training video and a sentence embedding sequence generated from the text; wherein the training step comprises: a step of obtaining frame embeddings for the training video and word embeddings for the text; a step of inputting the frame embeddings into a first encoder to generate a clip embedding sequence, and inputting the word embeddings into a second encoder to generate a sentence embedding sequence; A method for training a deep learning model to generate video embeddings and paragraph embeddings with information loss, comprising: a step of inputting the clip embedding sequence and the sentence embedding sequence into a third encoder to generate video embeddings and paragraph embeddings; wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship; the training step further comprises a step of training the deep learning model based on the loss function such that the clip embedding sequence and the sentence embedding sequence have the same embedding vectors; and the step of generating the video embeddings and paragraph embeddings comprises a step of removing some of the clip-sentence embedding pairs having the same information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence and inputting them into the third encoder. Claim 13 delete Claim 14 A method according to claim 12, wherein the step of generating the video embedding and paragraph embedding comprises the step of mixing the clip embedding sequence and the sentence embedding sequence and inputting them into the third encoder. Claim 15 A method according to claim 14, wherein the step of generating the video embeddings and paragraph embeddings comprises the step of inputting the sentence embeddings paired with the clip embeddings in the clip embedding sequence among the sentence embeddings in the sentence embedding sequence into the third encoder by replacing the sentence embeddings with the clip embeddings. Claim 16 delete Claim 17 A method according to claim 12, further comprising, after the step of generating the video embeddings and paragraph embeddings, the step of inputting the clip embedding sequence and the sentence embedding sequence, respectively, into the third encoder to evaluate the performance of the deep learning model. Claim 18 delete Claim 19 A computer program stored on a computer-readable recording medium, wherein the computer program, when executed by a processor, comprises the steps of: receiving a training video and text for said training video as input; and training a deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text; wherein the training step comprises the steps of: acquiring frame embeddings for said training video and word embeddings for said text; inputting said frame embeddings into a first encoder to generate a clip embedding sequence, and inputting said word embeddings into a second encoder to generate a sentence embedding sequence; A computer program comprising instructions for a processor to perform a deep learning model training method to generate video embeddings and paragraph embeddings with information lost, by including the step of inputting the clip embedding sequence and the sentence embedding sequence into a third encoder to generate video embeddings and paragraph embeddings; wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship, and the training step further includes the step of training the deep learning model so that the clip embedding sequence and the sentence embedding sequence have the same embedding vector based on the loss function, and wherein the step of generating the video embeddings and paragraph embeddings includes the step of removing some of the clip-sentence embedding pairs having the same information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence and inputting them into the third encoder. Claim 20 A computer-readable recording medium storing a computer program, wherein the computer program, when executed by a processor, comprises the steps of: receiving a training video and text for said training video as input; and training a deep learning model to generate video embeddings and paragraph embeddings using a preset loss function based on a clip embedding sequence generated from said training video and a sentence embedding sequence generated from said text; wherein the training step comprises the steps of: acquiring frame embeddings for said training video and word embeddings for said text; inputting said frame embeddings into a first encoder to generate a clip embedding sequence, and inputting said word embeddings into a second encoder to generate a sentence embedding sequence; A computer-readable recording medium comprising instructions for a processor to perform a deep learning model training method to generate video embeddings and paragraph embeddings with information lost, by including the step of inputting the clip embedding sequence and the sentence embedding sequence into a third encoder to generate video embeddings and paragraph embeddings; wherein the clip embedding sequence and the sentence embedding sequence have a one-to-one correspondence relationship, and the training step further includes the step of training the deep learning model so that the clip embedding sequence and the sentence embedding sequence have the same embedding vector based on the loss function, and the step of generating the video embeddings and paragraph embeddings includes the step of removing some of the clip-sentence embedding pairs having the same information among the clip embeddings in the clip embedding sequence and the sentence embeddings in the sentence embedding sequence and inputting them into the third encoder.

Images