A method, computer device, and computer program for generating video snapshots using image hashing.

Abstract

This invention provides a method, computer device, and computer program for generating video snapshots using image hashing. [Solution] The video snapshot generation method includes the step of generating a snapshot by removing redundant frames from the frames included in the video and using the remaining frames, while maintaining frames that contain user-defined target information.

Description

Technical Field

[0001] The following description relates to a technique for generating video snapshots.

Background Art

[0002] When analyzing a video, if all frames are processed in a method that requires a large amount of computation like a deep learning model, a considerable amount of time and resources are consumed for the analysis.

[0003] As one of the techniques to solve this, there is a method of removing some frames among the frames included in a video, generating a snapshot composed of the remaining frames, and utilizing it for video analysis.

[0004] In addition to video analysis, snapshot generation technology is sometimes utilized to provide highlight images of a video or the like.

[0005] For example, Patent Document 1 (registration date: August 8, 2012) discloses a technique for generating a thumbnail that summarizes and provides the overall content of a video file by sampling and selecting snapshots at multiple points from the entire running time of the video file.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] By removing redundant frames, it is possible to generate snapshots that do not lose the main frames containing target information when generating snapshots.

[0008] By using hashing, it is possible to determine whether a user-defined class is included in a frame using the zero-shot technique. [Means for solving the problem]

[0009] A video snapshot generation method is provided, which is performed on a computer device, wherein the computer device includes at least one processor configured to execute computer-readable instructions contained in memory, and the video snapshot generation method includes the step of using at least one processor to remove redundant frames from the frames contained in the video and generate a snapshot of the remaining frames, but maintaining frames that contain user-defined target information when generating the snapshot.

[0010] One aspect of this approach is that the generation step may involve generating a video snapshot using a hash generated through a machine learning model.

[0011] From another perspective, the generation step may involve generating video snapshots through a multimodal model that has been trained by representing embeddings of different modalities in a single vector space.

[0012] From another perspective, the generation step may include a step of calculating a first similarity that indicates the image similarity between frames included in the video, a step of calculating a second similarity that indicates the similarity relationship between each frame included in the video and a prompt registered as target information, and a step of selecting some frames included in the video as duplicate frames based on the first and second similarities.

[0013] Another aspect of this approach is that the first similarity calculation step involves calculating the first similarity using a multimodal model trained by representing embeddings of different modalities in a single vector space, and the first similarity may be calculated through the vector distance between image embeddings in which the features of the corresponding images for each frame are embedded.

[0014] In other words, the step of calculating the second similarity may include the step of registering the prompt as content in one of the following forms: text, image, or audio.

[0015] Another aspect of this approach is that the second similarity calculation step involves using a multimodal model trained by representing embeddings of different modalities in a single vector space to calculate the second similarity, and the second similarity may be calculated through the vector distance between a prompt embedding that embeds the features of the prompt registered as target information and an image embedding that embeds the features of the image corresponding to each frame.

[0016] From another perspective, the selection step may include a step of clustering frames that have a similarity of a certain level or higher based on the first similarity criterion; a step of selecting the remaining frames, after excluding a certain number of frames from each cluster, as candidate frames for duplicate frames; and a step of excluding frames from the candidate frames whose second similarity is a certain level or higher from the selection of duplicate frames.

[0017] According to another aspect, the multimodal model may be a model that learns the similarity of embeddings for data pairs using contrastive learning.

[0018] According to yet another aspect, the multimodal model may be a VLP (vision-language pre-trained) model learned through contrastive learning.

[0019] Provided is a computer program recorded on a non-transitory computer-readable recording medium for causing a computer device to execute a video snapshot generation method.

[0020] Provided is a computer device including at least one processor configured to execute computer-readable instructions included in a memory, the at least one processor processing a process of removing duplicate frames from the frames included in a video and generating a snapshot with the remaining frames, while maintaining frames including target information defined by a user and generating a snapshot.

Brief Description of the Drawings

[0021] [Figure 1] A diagram showing an example of a network environment according to an embodiment of the present invention. [Figure 2] A block diagram showing an example of a computer device according to an embodiment of the present invention. [Figure 3] A diagram showing an example of frames constituting a video. [Figure 4] A diagram showing an example of generation of a video snapshot. [Figure 5] A flowchart showing an example of a method that can be executed by a computer device according to an embodiment of the present invention. [Figure 6] An exemplary diagram for explaining a searchable multimodal model in an embodiment of the present invention. [Figure 7] This is an exemplary diagram for explaining a searchable multi-modal model in an embodiment of the present invention. [Figure 8] This is a flowchart showing an example of the duplicate frame filtering process in an embodiment of the present invention. [Figure 9] This is a diagram showing an example of generating a snapshot using a prompt in an embodiment of the present invention. [Figure 10] This is a diagram showing an example of generating a snapshot using a prompt in an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0023] Embodiments of the present invention relate to a technique for generating video snapshots.

[0024] Embodiments including matters specifically disclosed in this specification can generate a snapshot in a form that maintains a main frame including target information when generating a video snapshot.

[0025] A video snapshot generation system according to an embodiment of the present invention may be implemented by at least one computer device, and a video snapshot generation method according to an embodiment of the present invention may be executed through at least one computer device included in the video snapshot generation system. In this case, a computer program according to one embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the video snapshot generation method according to an embodiment of the present invention in accordance with the control of the executed computer program. The above-described computer program may be combined with the computer device and recorded on a computer-readable recording medium to cause the computer to execute the video snapshot generation method.

[0026] Figure 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Figure 1 shows an example that includes a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Figure 1 is merely an example for the purpose of explaining the invention, and the number of electronic devices and servers is not limited to that shown in Figure 1. Furthermore, the network environment in Figure 1 is merely an example of an environment applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment in Figure 1.

[0027] The multiple electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented by computer devices. Examples of the multiple electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), and tablets. As an example, Figure 1 shows a smartphone as an example of electronic device 110, but in embodiments of the present invention, electronic device 110 may mean one of a variety of physical computer devices that can communicate with other electronic devices 120, 130, 140, and / or servers 150, 160 via the network 170 using substantially wireless or wired communication methods.

[0028] The communication method is not limited, and may include not only communication methods that utilize communication networks that can be included in network 170 (for example, mobile communication networks, wired internet, wireless internet, broadcasting networks), but also short-range wireless communication between devices. For example, network 170 may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Furthermore, network 170 may include, but is not limited to, one or more network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks.

[0029] Servers 150 and 160 may each be implemented by one or more computer devices that communicate with multiple electronic devices 110, 120, 130, and 140 via a network 170 to provide commands, code, files, content, services, etc. For example, server 150 may be a system that provides services (for example, an image editing service) to multiple electronic devices 110, 120, 130, and 140 connected via a network 170.

[0030] Figure 2 is a block diagram showing an example of a computer device in one embodiment of the present invention. Each of the aforementioned electronic devices 110, 120, 130, and 140, as well as each of the servers 150 and 160, may be implemented by the computer device 200 shown in Figure 2.

[0031] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in Figure 2. The memory 210 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and a permanent mass storage device such as a disk drive. Here, the permanent mass storage device such as ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such a separate computer-readable recording medium may include a floppy disk drive, disk, tape, DVD / CD-ROM drive, memory card, and other computer-readable recording media. In other embodiments, the software components may be loaded into the memory 210 through a communication interface 230, which is not a computer-readable recording medium. For example, software components may be loaded into the memory 210 of the computer device 200 based on computer programs installed by files received via the network 170.

[0032] The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 220 by memory 210 or a communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a recording device such as memory 210.

[0033] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the recording device described above) via the network 170. For example, requests, instructions, data, files, etc., generated by the processor 220 of the computer device 200 according to program code recorded in a recording device such as memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc., from other devices may be received by the computer device 200 via the network 170 through the communication interface 230 of the computer device 200. Signals, instructions, data, etc., received via the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc., may be recorded on a recording medium (the persistent recording device described above) that the computer device 200 may further include.

[0034] The input / output interface 240 may be a means for interface with the input / output device 250. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. In another example, the input / output interface 240 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 250 may consist of the computer device 200 and one other device.

[0035] In other embodiments, the computer device 200 may include fewer or more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the computer device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as transceivers and databases.

[0036] The following describes specific embodiments of a method and apparatus for generating video snapshots using image hashing.

[0037] Video snapshots are sometimes used when analyzing video files such as MP4 and AVI, or animated images such as GIF and WEBP.

[0038] As an example, uniform sampling can be applied to generate video snapshots by detecting one frame at specific time intervals.

[0039] As another example, you could calculate the sum of absolute difference (SAD), which shows the difference in pixel values ​​between adjacent frames, and then detect frames where the SAD is greater than a user-defined value to generate a snapshot.

[0040] In addition to the methods described above, an algorithm can also be applied to remove duplicate frames through hash-based image processing techniques to generate video snapshots.

[0041] In this specification, duplicate frames may refer not only to frames that are exactly the same, but also to frames that have a certain degree of similarity. For example, frames with slight deformations such as translation or rotation may be recognized as duplicate frames. In fact, most similar frames can be said to be near duplicates.

[0042] This embodiment allows for the deletion of some of the frames contained in a given video, and the generation of a snapshot using only the remaining frames.

[0043] In the process of generating video snapshots by removing duplicate or other unwanted frames from a video, a problem can occur where important frames are missing. For example, when generating video snapshots for content monitoring purposes, frames containing target information to be monitored may be removed and missing from the snapshot.

[0044] Figure 3 shows an example of video frames for snapshot generation. As shown in Figure 3, we assume that the given video 30 consists of 10 frames.

[0045] Duplicate frames may be removed based on the image similarity between frames included in video 30, and a snapshot may be generated from the remaining frames. Of the 10 frames included in video 30, frames (1) and (2) have a high probability of being removed due to image similarity. For example, for video 30 in Figure 3, as shown in Figure 4, frames (1), (4), (5), (7), and (10) may be selected as duplicate frames and deleted, and then a snapshot 41 may be generated from frames (2), (3), (6), (8), and (9).

[0046] For example, quick response codes (QR codes) used on messengers serve various purposes, such as QR codes for accessing official accounts (OAs) or QR codes shared for personal encounters. During content monitoring of videos, some of these QR codes may require censorship as they are prohibited from being shared.

[0047] However, algorithms that remove duplicate frames based on image similarity have difficulty distinguishing between frame (1) and frame (2), so even if a frame requires censorship, it may be detected as a duplicate and deleted.

[0048] Therefore, when generating snapshots for video analysis or highlight creation, techniques are needed to preserve frames that contain target information, such as key monitoring content.

[0049] The computer device 200 according to this embodiment may provide video analysis services to a client by connecting to a dedicated application installed on the client or to a web / mobile site associated with the computer device 200. The computer device 200 may be configured with a computer-implemented video snapshot generation system. For example, the video snapshot generation system may be implemented as an independently operating program, or it may be implemented in an in-app form for a specific application so that it can operate on the specific application.

[0050] The processor 220 of the computer device 200 may be implemented by components for performing the following video snapshot generation method. Depending on the embodiment, the components of the processor 220 may be selectively included in or excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or merged for the representation of the functionality of the processor 220.

[0051] Such a processor 220 and its components may control the computer device 200 to perform steps included in the following video snapshot generation method. For example, the processor 220 and its components may be implemented to execute instructions from the operating system code contained in the memory 210 and the code of at least one program.

[0052] Here, the components of the processor 220 may be representations of different functions that are executed by the processor 220 according to instructions provided by the program code recorded in the computer device 200.

[0053] The processor 220 may read necessary instructions from the memory 210, which is loaded with instructions related to the control of the computer device 200. In this case, the instructions read may include instructions for controlling the processor 220 to perform the steps described below.

[0054] The steps included in the video snapshot generation method described below may be performed in a different order than shown in the diagram, some steps may be omitted, or additional processes may be included.

[0055] The steps included in the video snapshot generation method may be performed on server 150, and in some embodiments, some of the steps may be performed on the client.

[0056] Figure 5 is a flowchart showing an example of a method that a computer device can perform according to one embodiment of the present invention.

[0057] Referring to Figure 5, in step 510, the processor 220 may calculate the image similarity between frames contained in the video (hereinafter referred to as the "first similarity"). For example, the processor 220 may use a machine learning-based hash to calculate the degree of similarity between frames. The processor 220 may embed the features of each image corresponding to the frame and calculate the first similarity, which indicates the image similarity through the distance between the embedding vectors.

[0058] In step 520, if a user-defined prompt is provided, the processor 220 may calculate a similarity (hereinafter referred to as "second similarity") between each frame in the video and the prompt. Similarly, the processor 220 may use a hash on a machine learning platform to calculate the degree of similarity between the frame and the prompt. In this embodiment, the prompt may be used to define target information to be included in the snapshot, in which case the prompt may be input by the user as content in various forms such as text, images, and audio. The processor 220 may embed features of the target information defined through the prompt and calculate the second similarity through the vector distance between the prompt embedding and the image embedding of each frame. In this embodiment, the hash may be used and a zero-shot technique may be used to determine the similarity relationship between the image frame and a user-defined class.

[0059] In step 530, the processor 220 may select duplicate frames to be removed from the frames included in the video based on the first and second similarity scores. After clustering frames with a similarity level above a certain level based on the first similarity score, the processor 220 may select the remaining frames as candidate frames, excluding a predetermined number of frames (e.g., 1) from each cluster. At this point, the processor 220 may finally select frames with a second similarity level below a certain level as duplicate frames. In other words, the processor 220 may exclude frames with a second similarity level above a certain level from the selection of duplicate frames.

[0060] In step 540, the processor 220 may remove the frames that have been final selected as duplicate frames from the frames included in the video and generate a snapshot with the remaining frames. After clustering the frames included in the video based on a first similarity criterion, a snapshot may be generated while retaining the maximum total frames of a particular cluster corresponding to a second similarity criterion. In this embodiment, duplicate frames are removed based on image similarity and a snapshot is generated with the remaining frames, but the snapshot can be generated to include as many frames as possible that are similar to a user-defined class.

[0061] Figures 6-7 are illustrative diagrams illustrating a searchable multimodal model in one embodiment of the present invention.

[0062] This invention uses a multimodal model that enables retrieval using clustering and various forms of prompts.

[0063] Referring to Figure 6, the multimodal model according to the present invention extracts embeddings for various modalities such as text 601, video (or images) 602, and audio 603, but includes a method for learning such that embeddings of different modalities are shared in a single vector space 600.

[0064] As mentioned above, contrastive learning is a common model training method, and a representative example is a large-scale VLP (vision-language pre-trained) model such as CLIP (contrastive language-image pre-training model).

[0065] Referring to Figure 7, CLIP learns in such a way that the similarity of the embeddings extracted from the text encoder 701 and the image encoder 702 increases for image-text pairs that belong to contexts with the same meaning, and decreases the similarity of the embeddings extracted from the text encoder 701 and the image encoder 702 for image-text pairs that belong to content with different meanings.

[0066] In the case of an image encoder 702 trained together with a text encoder 701, it can appropriately represent the visual characteristics of an image while simultaneously representing the semantic features of text, thus enabling dual-purpose use.

[0067] Image-text embedding is a method for embedding and combining the features of images and text, allowing for the proper representation of relationships between embedded classes. It can be primarily used in image-text search, image captioning, and visual question answering (VQA).

[0068] Figure 8 is a flowchart showing an example of a duplicate frame filtering process in one embodiment of the present invention.

[0069] In this invention, by utilizing embeddings extracted through a multimodal model, filtering based on user-defined prompts and filtering based on image similarity can be performed simultaneously.

[0070] Referring to Figure 8, the processor 220 may extract image embeddings for each frame contained in the video through the image encoder 702, which has been trained together with the text encoder 701. In this case, the image similarity between frames may be calculated using the embeddings for each frame (S801).

[0071] When a text-based prompt 80 is input by the user, the processor 220 may extract text embeddings for the prompt 80 through a text encoder 701 that has been learned together with the image encoder 702.

[0072] For example, when censoring images containing QR codes during the video analysis process, the user may enter prompts such as "qr," "qr image," "photo of qr code," or "qr code image" as text that represents the target information.

[0073] The processor 220 may compare the image embedding extracted through the image encoder 702 for each frame contained in the video with the text embedding extracted through the text encoder 701 to calculate the similarity to the prompt (S802).

[0074] In other words, the processor 220 can use hashes generated by the machine learning infrastructure and employ zero-shot techniques to determine whether each frame contains a user-defined class based on its similarity to the prompt.

[0075] The processor 220 may select frames from the video that have an image similarity level of a certain level or higher with other frames as duplicate frames. However, among the frames selected as duplicate frames, frames that have an image similarity level of a certain level or higher with a prompt may be excluded from the selection of duplicate frames (S803).

[0076] Therefore, the processor 220 can remove the frames that were ultimately selected as duplicate frames from the frames included in the video and generate a snapshot from the remaining frames, in which case the snapshot can include frames whose similarity to the prompt is above a certain level.

[0077] In an algorithm that removes duplicate frames based on image similarity, one of the frames (1) and (2) containing the QR code may be selected as a duplicate frame and deleted. However, in this embodiment, the user can specify desired target information via prompt 80, and the frame containing that target information can be maintained to generate a video snapshot.

[0078] Figures 9-10 show examples of snapshot generation using prompts in one embodiment of the present invention.

[0079] Figures 9 and 10 show examples of generating snapshots of video 30, which consists of 10 frames numbered (1) through (10).

[0080] As shown in Figure 9, if the user enters the text "QR" as a prompt to define the desired target information, the processor 220 may exclude frames (1) and (2) containing the QR code from the duplicate frame selection among the 10 frames included in the video 30.

[0081] For example, a snapshot 91 of video 30 may consist of frames (1), (2), (3), (6), (8), and (9) remaining after frames (4), (5), (7), and (10) are selected as duplicate frames and deleted. That is, the processor 220 can generate a snapshot 91 in a form that retains frames (1) and (2) in which a QR code exists as target information defined through a prompt.

[0082] Although the embodiments described above use text prompts as an example, the method is not limited to this, and filtering can also be applied using audio or image format prompts.

[0083] By representing embeddings of different modalities in a single vector space and training a model, it is possible to generate snapshots using zero-shot techniques, utilizing not only text prompts but also other forms of modality prompts, ensuring that frames containing target information are not lost.

[0084] For example, as shown in Figure 10, if the user inputs the audio "empty" as a prompt to define the desired target information, the features of the audio "empty" may be embedded, and frames with a similarity to the prompt embedding above a certain level may be maintained. The processor 220 may create a snapshot 101 in a form that maintains frames (7) and (8) corresponding to the "empty" information, which is the target information defined through the audio prompt. In other words, the snapshot 101 for video 30 will consist of the remaining frames (2), (3), (6), (7), (8), and (9), with frames (1), (4), (5), and (10) being deleted as duplicate frames.

[0085] By applying a hashing methodology based on machine learning, it becomes possible to search not only for frames associated with specific keywords input in the form of text, audio, or images, but also for frames containing specific sounds (e.g., cheers, car crashes, waves, alarms, etc.).

[0086] This embodiment includes zero-shot features and allows all pre-trained modality information to be applied as a target. In addition to registering specific keywords as target information in the form of text, audio, or images, users can also register various types of content, such as sounds, adult content, and violent content, through prompts, enabling them to search for frames containing the information they desire.

[0087] Thus, according to the embodiments of the present invention, when generating a video snapshot by removing duplicate frames, it is possible to generate a snapshot in which the main frames containing target information are not missing. Furthermore, according to the embodiments of the present invention, by utilizing hashes generated by a machine learning platform and using a zero-shot technique to determine whether a user-defined class is included in the frame, it is possible to generate a video snapshot that includes as many frames similar to the user-defined class as possible.

[0088] The above-described apparatus may be implemented by hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microprocessors, or various devices capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. The processing unit may also respond to software execution, access data, record, manipulate, process, and generate data. For convenience of understanding, it may be described as if a single processing unit is used, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

[0089] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired, or to instruct the processing unit independently or collectively. Software and / or data may be embodied in any kind of machine, component, physical device, computer recording medium, or device for interpretation based on the processing unit or for providing instructions or data to the processing unit. Software may be distributed across a network of computer systems, and may be recorded and executed in a distributed manner. Software and data may be recorded on one or more computer-readable recording media.

[0090] The method according to the embodiment may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record computer-executable programs or may temporarily record them for execution or download. Furthermore, the medium may be various recording or storage means in the form of a combination of one or more hardware components, and may be a medium directly connected to a computer system or distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROM, RAM, and flash memory. Other examples of media include recording media and storage media managed by app stores that distribute applications, and sites and servers that supply and distribute various other software.

[0091] As described above, embodiments have been explained based on limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or the described systems, structures, devices, circuits, and other components may be combined or assembled in a different manner than described, or opposed or replaced by other components or equivalents, and still achieve suitable results.

[0092] Therefore, even if the embodiment is different, it falls within the scope of the attached claims if it is equivalent to the claims.

Claims

1. A method for generating video snapshots performed on a computer device, The computer device includes at least one processor configured to execute computer-readable instructions contained in memory, The aforementioned video snapshot generation method is: The process involves at least one processor removing redundant frames from the video and generating a snapshot from the remaining frames, while retaining frames containing user-defined target information when generating the snapshot. A method for generating video snapshots, including the method described above.

2. The above generation step is, To generate a snapshot of the video using a hash generated through a machine learning model. A video snapshot generation method according to claim 1, characterized by the above.

3. The above generation step is, To generate snapshots of the video through a multimodal model trained by representing embeddings of different modalities in a single vector space. A video snapshot generation method according to claim 1, characterized by the above.

4. The above generation step is, A step of calculating a first similarity that indicates the image similarity between frames included in the video, A step of calculating a second similarity score for each frame included in the video, which shows the similarity relationship with the prompt registered as the target information, and Steps to select some frames included in the video as duplicate frames based on the first similarity and the second similarity. A video snapshot generation method according to claim 1, including the method described in claim 1.

5. The step of calculating the first similarity is: The first similarity is calculated using a multimodal model that has been trained by representing embeddings of different modalities in a single vector space, The first similarity is calculated by taking the vector distance between image embeddings into which the image features corresponding to each frame are embedded. A video snapshot generation method according to claim 4, characterized by the above.

6. The step of calculating the second similarity is: Steps to register the aforementioned prompt as content in one of the following forms: text, image, or audio. A video snapshot generation method according to claim 4, including the method described in claim 4.

7. The step of calculating the second similarity is: The second similarity is calculated using a multimodal model that has been trained by representing embeddings of different modalities in a single vector space. The second similarity is calculated by taking the vector distance between a prompt embedding that incorporates the features of the prompt registered as the target information and an image embedding that incorporates the features of the image corresponding to each frame. A video snapshot generation method according to claim 4, characterized by the above.

8. The aforementioned selection step is, A step of clustering frames that have a similarity of a certain level or higher based on the first similarity score, For each cluster, the steps include selecting the remaining frames, after removing a certain number of frames, as candidate frames for the duplicate frames, and Step 1: Exclude from the selection of duplicate frames frames any frame among the candidate frames whose second similarity is above a certain level. A video snapshot generation method according to claim 4, including the method described in claim 4.

9. The aforementioned multimodal model is a model in which the similarity of embeddings for data pairs has been learned using contrastive learning. A video snapshot generation method according to claim 3, characterized by the above.

10. The aforementioned multimodal model is a VLP (vision-language pre-trained) model learned through contrast learning. A video snapshot generation method according to claim 3, characterized by the above.

11. A computer program recorded on a non-temporary computer-readable recording medium for causing the computer device to execute the video snapshot generation method according to any one of claims 1 to 10.

12. A computer device, At least one processor configured to execute computer-readable instructions contained in memory Includes, The aforementioned at least one processor is The process involves removing duplicate frames from the video and generating a snapshot from the remaining frames, while retaining frames that contain user-defined target information when generating the snapshot. A computer device that processes data.

13. The aforementioned at least one processor is Using hashes generated through a machine learning model, a snapshot of the video is generated. The computer device according to claim 12, characterized by the above.

14. The aforementioned at least one processor is To generate a snapshot of the video through a multimodal model that has been trained by representing embeddings of different modalities in a single vector space. The computer device according to claim 12, characterized by the above.

15. The aforementioned at least one processor is A first similarity score is calculated, which indicates the image similarity between frames included in the aforementioned video. For each frame included in the video, a second similarity score is calculated that indicates the similarity relationship with the prompt registered as the target information. Based on the first similarity and the second similarity, some frames included in the video are selected as duplicate frames. The computer device according to claim 12, characterized in that it is a computer device.

16. The aforementioned at least one processor is The first similarity is calculated using a multimodal model that has been trained by representing embeddings of different modalities in a single vector space, The first similarity is calculated by taking the vector distance between image embeddings into which the image features corresponding to each frame are embedded. The computer device according to claim 15, characterized by the above.

17. The aforementioned at least one processor is Register the aforementioned prompt as content in one of the following forms: text, image, or audio. The computer device according to claim 15, characterized by the above.

18. The aforementioned at least one processor is The second similarity is calculated using a multimodal model that has been trained by representing embeddings of different modalities in a single vector space. The second similarity is calculated by taking the vector distance between a prompt embedding that incorporates the features of the prompt registered as the target information and an image embedding that incorporates the features of the image corresponding to each frame. The computer device according to claim 15, characterized by the above.

19. The aforementioned at least one processor is Based on the aforementioned first similarity criterion, frames having a similarity level above a certain level are clustered. For each cluster, the remaining frames after removing a certain number of frames are selected as candidate frames for the duplicate frames. Frames among the candidate frames whose second similarity is above a certain level are excluded from the selection of duplicate frames. The computer device according to claim 15, characterized by the above.

20. The aforementioned multimodal model is a VLP model in which the similarity of embeddings for data pairs is learned using contrast learning. The computer device according to claim 14, characterized by the above.

Images