Video summary system capable of detecting specific object with text

Abstract

A video summary system, according to one embodiment of the present invention, can find an object by converting a text describing an image into a text embedding unit vector through a text encoder, converting an image into an image embedding unit vector through an image encoder, and calculating the similarity between the text embedding unit vector and the image embedding unit vector. Therefore, when the video summary system, according to one embodiment of the present invention, is used, a user can input a description of an object or an action performed by the object as text, and detect an object corresponding to the description or an object performing the action. Accordingly, when the video summary system, according to one embodiment of the present invention, is used, it is possible to solve the problem of the number of objects appearing in a summary image being larger than the numbers of objects appearing in the original image making monitoring difficult.

Description

A video summary system capable of detecting specific objects through text.

[0001] The present invention relates to a video summary system capable of detecting a specific object through text.

[0002] In modern society, various video capture devices, such as CCTV, are used to monitor spaces. However, playing back the massive amount of video data generated by these devices in real time, analyzing it, and identifying necessary events requires significant manpower and time. Conversely, playing back the video at high speeds makes object movement too rapid, making event detection difficult. This is a trade-off. To address this issue, video summary technology has been proposed.

[0003] Video summary technology is a method of creating a compressed version of an original video. The primary focus of compression is on the objects (people, animals, vehicles, etc.) in the original video. This technology selects frames containing dynamic objects from the original video and merges them to create a summary video. The (X, Y) coordinates of the dynamic objects are maintained, ensuring that the object's movement in the summary video mirrors its actual location. Objects that appear at different times in the original video may appear at the same time in the summary video.

[0004] This video summary technique poses another challenge for event detection. Event detection requires identifying objects or their behavior. However, video summary technology often presents a significantly larger number of objects at any given point in time than the original video. This means identifying objects or their movements becomes more difficult.

[0005] To solve this problem, a feature for object detection is needed. For example, if a worker enters the text "person carrying a child," the summary video should display only "person carrying a child," excluding other objects and displaying only the objects related to the desired event.

[0006] While there have been techniques for detecting objects from images using text, their limitations have been clear. For example, let's consider a prior art technique that searches for images containing "kicking people" by inputting the text "kicking people."

[0007] Image search on platforms like Naver and Daum relies on captions provided by content creators rather than object detection. This method, like video summary technology, doesn't work when objects in a video don't have captions attached.

[0008] Another approach is to use artificial intelligence. Collect various images of "people kicking" and train the model to build one. Using the trained model, you can detect "people kicking" within the video. The problem is that training is required for each specific behavior. Finding "people cheering" and "people turning their heads" requires separate trained models. Given the sheer variety of situations, this approach is also impractical.

[0009] The present invention proposes a new solution to solve these problems.

[0010] One object of the present invention is to provide an image detection system capable of finding an object by converting text describing an image into a text embedding unit vector through a text encoder, converting an image into an image embedding unit vector through an image encoder, and calculating the similarity between the two, and a video summary system using the same.

[0011] Another object of the present invention is to provide an image detection system having high performance for Korean text and a video summary system using the same.

[0012] Meanwhile, other unspecified purposes of the present invention will be additionally considered within the scope that can be easily inferred from the detailed description and effects thereof below.

[0013] To achieve the purpose proposed above, the following solutions are proposed.

[0014] A video summary system according to one embodiment of the present invention includes an object extraction module that detects an object in an original video and generates object information including the position and size of the object within the frame and the object ID; a compression module that compresses the original video so that at least some of the objects at different times in the original video are located at the same time to generate a summary video; and an image detection module that detects an object corresponding to text input by a user; wherein the image detection module converts text describing an image into a text embedding unit vector through a text encoder, converts an image into an image embedding unit vector through the image encoder, and calculates a similarity between the text embedding unit vector and the image embedding unit vector, thereby detecting an object to be found.

[0015] In one embodiment, the image detection module may be characterized as an artificial intelligence model trained with learning data in which captions are written on images.

[0016] At this time, the English caption of the above learning data may be characterized by being translated into a Korean caption using a large-scale natural language generation model and added to the learning data.

[0017] At this time, the caption of the above learning data may be characterized by generating Korean or English captions with various expressions having the same or similar meaning using a large-scale natural language generation model (NLP) and adding them to the learning data.

[0018] At this time, it can be characterized by adding pseudo-labeled images with inappropriate captions to the learning data.

[0019] In one embodiment, when the image detection module detects an object to be found, the summary image may be characterized in that an object located at the same time as the object detected in the original image is displayed together.

[0020] An image detection system capable of detecting a specific object using text according to another embodiment of the present invention can detect an object to be found by converting text describing an image into a text embedding unit vector through a text encoder and converting the text into an image embedding unit vector through an image encoder and calculating the similarity between the text embedding unit vector and the image embedding unit vector.

[0021] In another embodiment, the image detection module may be characterized as an artificial intelligence model trained with learning data in which captions are written on images.

[0022] In another embodiment, the English captions of the learning data may be translated into Korean captions using a large-scale natural language generation model and added to the learning data.

[0023] In another embodiment, the caption of the learning data may be characterized by generating Korean or English captions with various expressions having the same or similar meaning using a large-scale natural language generation model and adding them to the learning data.

[0024] In another embodiment, the method may feature adding pseudo-labeled images with inappropriate captions to the training data.

[0025] A video summary system according to an embodiment of the present invention can find an object by converting text describing an image into a text embedding unit vector through a text encoder, converting an image into an image embedding unit vector through an image encoder, and calculating the similarity between the two. Therefore, when using the video summary system according to an embodiment of the present invention, a user can input a description of an object or an action being performed by the object as text, and detect an object corresponding to the description or an object performing the action. Therefore, when using the video summary system according to an embodiment of the present invention, the problem that the number of objects appearing in a summary video increases compared to the original video, making monitoring difficult, can be solved.

[0026] In addition, the image detection module of the video summary system according to one embodiment of the present invention uses an artificial intelligence model trained with training data containing captions for images. Currently, the captions in the training data are almost entirely in English. Therefore, training with only English captions has the problem that it does not work properly for Korean. To address this issue, attempts have been made to generate Korean captions using a translator in the past, but the problem remains that performance is low for Korean text. The video summary system according to one embodiment of the present invention uses a large-scale natural language model to translate English captions into Korean captions and adds images with Korean captions as training data. While a translator differs from the language used by actual people, a large-scale natural language model is similar to the language used by actual people. Therefore, adding images with Korean captions generated using a large-scale natural language model as training data improves the performance of image detection using Korean text compared to using a translator.

[0027] In addition, the image detection module of the video summary system according to one embodiment of the present invention uses an artificial intelligence model trained with learning data in which captions are written on images, and by generating Korean or English captions with various expressions having the same or similar meaning using a large-scale natural language generation model for the captions of the learning data and adding them to the learning data, the performance of image detection using text is significantly improved.

[0028] Meanwhile, even if the effect is not explicitly mentioned herein, it is added that the effect and its provisional effect described in the following specification expected by the technical features of the present invention are treated as described in the specification of the present invention.

[0029] FIG. 1 is a schematic diagram of a video summary system according to one embodiment of the present invention.

[0030] Figure 2 is a screenshot of the original 3-hour video.

[0031] Figure 3 is a screenshot of a two-minute summary video compressed by the compression module.

[0032] Figure 4 is a screenshot after filtering using the conventional method, with “male” and “female” as tags.

[0033] FIG. 5 is a schematic diagram of an image detection module (system) of a video summary system according to one embodiment of the present invention.

[0034] Figures 6 and 7 are screenshots when text is entered into the image detection module.

[0035] Figure 8 is an example of CoCo Captioning Data, which is open data used as learning data.

[0036] Figure 9 is an example of Flickr30k, a public data used as learning data.

[0037] It is to be understood that the attached drawings are provided for reference only to help understand the technical concept of the present invention, and the scope of the present invention is not limited thereby.

[0038] Hereinafter, with reference to the drawings, the configuration of the present invention, guided by various embodiments thereof, and the effects resulting from such configurations will be examined. In describing the present invention, detailed descriptions of related, well-known functions that are obvious to those skilled in the art and that may unnecessarily obscure the gist of the present invention will be omitted.

[0039] The term "module" as used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions.

[0040] In this document, "modules" or "nodes" use computing devices, such as CPUs and APs, to perform tasks such as moving, storing, and converting data. For example, "modules" or "nodes" may be implemented as devices such as servers, PCs, tablet PCs, and smartphones.

[0041] In this document, the term "deep learning model" refers to a neural network that is trained by an algorithm modeled on the way the human brain works, and is broadly interpreted as a meaning widely used in the industry.

[0042] FIG. 1 is a schematic diagram of a video summary system according to one embodiment of the present invention.

[0043] Referring to FIG. 1, a video summary system according to one embodiment of the present invention includes an input module (10), an object extraction module (20), an image detection module (40), and an output module (50).

[0044] Through the input module (10), the user inputs commands to control the system, or inputs text describing the object to be found or describing the behavior of the object.

[0045] The object extraction module (20) detects an object in the original image and generates object information including the object ID and the position and size of the object within the frame.

[0046] The object extraction module (20) detects and separates dynamic objects from each frame of the original image upon receiving the original image, and then detects object information for the separated objects. The object information includes coordinates (x, y) within the frame, width and height, and classification of the object, such as people, animals, and vehicles. The object information is preset and may include color, direction of movement (from left to right, etc.). The above-described operation is performed on all frames in which dynamic objects appear. Once segmentation and detection of objects are complete, the identity of objects appearing across multiple frames is determined, and if they are determined to be the same object, the same ID is assigned. Since objects have continuity of movement, the identity of objects appearing across multiple frames can be determined using the frames in which the dynamic object appears or by using object information together. As described above, the process of separating dynamic objects (segmentation), the process of detecting objects and generating object information (detecting), and the process of determining identity and assigning IDs to each object (tracking) are widely used technologies in image processing fields such as intelligent CCTV, and therefore a detailed description thereof will be omitted.

[0047] The compression module (30) compresses the original image so that at least some of the objects at different times in the original image are located at the same time to generate a summary image. Objects at different times in the original image include cases where the objects are never located in the same frame at the same time, and cases where only some of the objects are located in the same frame at the same time. Furthermore, when there is an object A with a long appearance time and an object B with a short appearance time that appears and then disappears during the appearance of object A, the appearance time of object B is different from the original image based on the appearance time of object A.

[0048] Figure 2 is a screenshot of the original three-hour video, and Figure 3 is a screenshot of the two-minute summary video compressed by the compression module. Comparing Figures 2 and 3, we can see that various objects (people, cars, etc.) that were located at different times in the original video are now located within a single frame. However, the problem is that there are too many objects appearing at a single point in the summary video, making it difficult to properly monitor them.

[0049] Figure 4 is a screenshot after filtering using the conventional method, with "male" and "female" as tags. As shown in Figure 4, when "male" and "female" are tagged, too many objects are still output, and, most importantly, objects with certain behaviors or appearances cannot be found.

[0050] To solve this problem, the present invention uses an image detection module (system) (40).

[0051] FIG. 5 is a schematic diagram of an image detection module (system) of a video summary system according to one embodiment of the present invention.

[0052] The image detection module (system, “system” is omitted below) (40) detects an object corresponding to the text entered by the user when the user inputs text. To this end, the image detection module (40) converts the text describing the image into a text embedding unit vector through a text encoder, converts the image into an image embedding unit vector through an image encoder, and calculates the similarity between the text embedding unit vector and the image embedding unit vector, thereby detecting the object to be found.

[0053] The image detection module performs the function of finding objects related to text input by the user in the image.

[0054] For example, when a user inputs the text “a girl wearing white short sleeves and shorts,” as shown in FIG. 6, the image detection module (40) can detect “a girl wearing white short sleeves and shorts” in the summary image.

[0055] Likewise, when a user inputs the text “woman wearing a black top and carrying a yellow bag,” the image detection module (40) detects “woman wearing a black top and carrying a yellow bag” in the summary image, as shown in FIG. 7.

[0056] To accomplish this, the image detection module (40) goes through two main steps. The first step is embedding of text and images, and the second step is calculating the similarity between the embedding vectors.

[0057] First, the input text is converted into a form called a text embedding unit vector through a text encoder. The text encoder converts text data into a low-dimensional (e.g., 512-dimensional) real-valued vector. Similarly, images are converted into image embedding unit vectors through an image encoder. The image encoder converts an image into a low-dimensional (e.g., 512-dimensional) vector. These converted vectors represent the key features of the text or image, enabling comparison and analysis of text and images.

[0058] The similarity between the transformed text embedding unit vector and the image embedding unit vector can be calculated using similarity calculation methods such as cosine similarity. Similarity calculation methods other than cosine similarity can also be used. Similarity measures how similar two vectors are pointing in different directions, which allows us to determine how closely a specific text is related to a specific image. More specifically, the similarity between an image and text is calculated, and images with a similarity higher than a threshold are output as detection results.

[0059] For this operation, the image detection module (40) utilizes an artificial intelligence model trained with learning data containing captions for images. Using pairs of captions and images, the model learns caption information that describes the content of the image, thereby identifying the relationship between the image and the text.

[0060] More specifically, the present invention constructs a text encoder and / or an image encoder through learning. The text encoder learns how to convert captions (text) into embedding vectors, and the image encoder learns how to convert images into embedding vectors. These learned encoders then perform the function of converting new text or images into embedding vectors when given new text or images.

[0061] Figure 8 is an example of CoCo Captioning Data, which is public data used as learning data, and Figure 9 is an example of Flickr30k, which is public data used as learning data.

[0062] The CoCo Captioning Data consists of approximately 400,000 training data and 200,000 validation data, and is image-caption pair data. Flickr30k consists of approximately 30,000 training data and 1,000 validation data, and is image-caption pair data.

[0063] Pseudo-labeled data can be used as training data. Pseudo-labeled data refers to data that replaces the model's predictions with labels for unlabeled data. If some pseudo-labeled data is included in the training data, the object detection performance of the image detection module (40) is improved. To generate pseudo-labeled data, a captioning model that generates captions from images without captions was utilized to generate captions from approximately 5 million images, and the generated pseudo-labeled data was added to the training data.

[0064] Meanwhile, the image detection module of the video summary system according to one embodiment of the present invention uses an artificial intelligence model trained with training data containing captions for images. Currently, the captions in the training data are almost entirely in English. Training solely with English captions has the problem that it does not function properly for Korean. To address this issue, attempts have been made to generate Korean captions using a translator in the past, but the performance for Korean text remains low. The video summary system according to one embodiment of the present invention uses a large-scale natural language model to translate English captions into Korean captions and adds images with Korean captions as training data. While translators differ from the language used by actual humans, large-scale natural language models are similar to the language used by actual humans. Therefore, adding images with Korean captions generated using a large-scale natural language model as training data improves the performance of image detection using Korean text compared to using a translator. For reference, large-scale natural language models include GPT (Generative Pretrained Transformer), BERT (Bidirectional Encoder Representations from Transformers), Google's BARD, and T5 (Text-to-Text Transfer Transformer). GPT and T5 were used here. However, new large-scale natural language models are constantly emerging, and the present invention is not limited to any specific large-scale natural language model.

[0065] In addition, the image detection module of the video summary system according to one embodiment of the present invention uses an artificial intelligence model trained with learning data in which captions are written on images, and by generating Korean or English captions with various expressions having the same or similar meaning using a large-scale natural language generation model for the captions of the learning data and adding them to the learning data, the performance of image detection using text is significantly improved.

[0066] As mentioned above, training requires training both the text encoder and the image encoder. The encoder was trained using the contrastive loss between the image and the caption. The contrastive loss refers to a method of arranging text embedding unit vectors in rows or columns and image embedding unit vectors in columns or rows, and then training so that the pair data (the central diagonal part of the matrix created) of the text embedding unit vector and the image embedding unit vector becomes 1. Therefore, when detecting an image, the cosine similarity between the text embedding unit vector and the image embedding unit vector is output as the result for the input text.

[0067] Additionally, the present invention utilizes both contrast loss and image-text matching loss to improve performance. Image-text matching loss calculates the loss by performing binary classification on an image and text using a cross-attention model. The cross-attention model is a technique that helps analyze and understand the complex relationships between images and text, measuring the degree to which specific parts of the image and text are related. Through this process, the AI ​​ultimately classifies the relationship between the image and text into a binary classification. That is, if a given image and text are related, it is classified as "1", otherwise "0." Finally, the loss is calculated. The loss is a numerical value indicating how much the AI's predictions differ from the actual results. A lower loss indicates that the AI ​​is better at matching images and texts, while a higher loss indicates that the AI's predictions are often incorrect. Therefore, training is performed to minimize the loss.

[0068] The present invention proceeds with learning from a text encoder by freezing a pretrained image encoder.

[0069] Meanwhile, as a result of research conducted with funding from the government (Ministry of Science and ICT), an image detection model called VL-KE-T5 is being released along with its code.

[0070] VL-KE-T5 used public data as learning data, as shown in Table 1 below, and Google's translation API was used for translation.

[0071]

[0072] EnglishCC 3MCOCOSBUVisualGenomeWIT2,862,265414,113772,4384,322,3583,265,279Korean (Translated)CC 3MCOCOSBUVisualGenomeWIT2,862,264414,113772,4384,322,3583,265,273KoreanWIT54,956

[0073]

[0074] Additionally, VL-KE-T5 used google / vit-base-patch16-384 as an image encoder and KETI-AIR / ke-t5-base as a text encoder. A larger model was used compared to the present invention.

[0075] The performance of the image detection modules of the VL-KE-T5 and the present invention was compared.

[0076] The results of testing with 1,000 Coco validation data sheets are as follows. Table 2 below compares the performance of KETI's VL-KE-T5 and the image detection module of the present invention.

[0077]

[0078] Text EncoderImage EncoderTranslatorTraining data typesPerformanceSpeedExample 1 (contrastive loss)Transformer EncoderViT-B / 32, 224GPT 3.5, Finetuned nllb modelCoCo, Flickr / Approximately 5 million imagesR@1: 0.356R@5: 0.715FPS: 1600Example 2 (contrastive loss)Transformer EncoderViT-B / 32, 224GPT 3.5, Finetuned nllb modelCoCo, Flickr / Approximately 5 million imagesTraining data generation by NLPR@1: 0.402R@5: 0.792FPS: 1600Example 3 (contrastive loss)Transformer EncoderViT-B / 32, 224GPT 3.5, Finetuned nllb modelCoCo, Flickr / About 5 million training data generated by NLP pseudo-labeling image R@1: 0.497 R@5: 0.805 FPS: 1600 Example 4 (contrastive loss + image text matching loss) Transformer Encoder ViT-B / 32, 224 GPT 3.5, Finetuned nllb model CoCo, Flickr / About 5 million training data generated by NLPR@1: 0.678 R@5: 0.922 FPS: 1600 Example 5 (contrastive loss + image text matching loss) Transformer Encoder ViT-B / 32, 224 GPT 3.5, Finetuned nllb model CoCo, Flickr / About 5 million training data generated by NLP pseudo-labeling image R@1: 0.731 R@5: 0.939 FPS: 1600 Comparison Example Pretrained T5 ModelViT-B / 16, 384Google APICC3M, CoCo, SBU, Visual Genome, WIT / Approximately 9 million shotsR@1: 0.335R@5: 0.699FPS: 100

[0079]

[0080] Here, Recall@5 means that there is a correct answer among 5 images, and when monitoring an image, if about 5 objects are output, a person can easily find the desired object, so it means that the performance is evaluated based on 5.

[0081] When the image detection module (40) detects the object to be found, the result is output to the user through the output module (50). When the image detection module detects the object to be found, the summary image displays an object located at the same time as the object detected in the original image, so that the correlation between the objects can be confirmed.

[0082] The video summary system and / or object detection system described above can be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program can be stored and provided on a non-transitory computer-readable medium. Here, the non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above can be stored and provided on a non-transitory computer-readable medium, such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0083] The scope of protection of the present invention is not limited to the description and expression of the embodiments explicitly described above. Furthermore, it should be noted that the scope of protection of the present invention may not be limited by obvious modifications or substitutions within the technical field to which the present invention pertains.

Claims

1. An object extraction module that detects an object in an original image and generates object information including the object ID and the position and size of the object within the frame; A compression module for generating a summary image by compressing the original image so that at least some of the objects at different times in the original image are located at the same time; and An image detection module that detects an object corresponding to the entered text when a user inputs text; The above image detection module is a video summary system that detects an object to be found by converting text describing an image into a text embedding unit vector through a text encoder, converting an image into an image embedding unit vector through an image encoder, and calculating the similarity between the text embedding unit vector and the image embedding unit vector.

2. In paragraph 1, A video summary system characterized in that the image detection module is an artificial intelligence model trained with learning data in which captions are written on images.

3. In paragraph 2, A video summary system characterized in that the English captions of the above learning data are translated into Korean captions using a large-scale natural language generation model and added to the learning data.

4. In paragraph 2, A video summary system characterized in that it generates Korean or English captions with various expressions having the same or similar meaning using a large-scale natural language generation model and adds them to the learning data.

5. In paragraph 2, A video summary system characterized by adding pseudo-labeled images with inappropriate captions to training data.

6. In paragraph 1, A video summary system characterized in that, when the image detection module detects an object to be found, an object located at the same time as the object detected in the original image is displayed together in the summary image.

Images