Method, computer device and computer program for generating video snapshots by using image hashing

Abstract

A method, a computer device and a computer program for generating a video snapshot by using image hashing are disclosed. The method for generating video snapshots comprises a step of removing redundant frames from among frames included in a video and generating snapshots by using remaining frames, the snapshots being generated by maintaining frames that include target information defined by a user.

Description

Method for generating a video snapshot using an image hash, computer device, and computer program

[0001] The description below describes the technique for creating a video snapshot.

[0002] When analyzing a video, if you use a method that requires a lot of computation, such as a deep learning model, for every frame, video analysis consumes a significant amount of time and resources.

[0003] One technique to solve this problem is to remove some of the frames contained in the video and create a snapshot consisting of only the remaining frames, which can then be used for video analysis.

[0004] In addition to video analysis, snapshot generation technology can be used to provide highlight reels for videos.

[0005] For example, Korean Patent No. 10-1174176 (registration date: August 8, 2012) discloses a technology for generating a thumbnail that summarizes the entire content of a video file by sampling and selecting snapshots at multiple points in the entire running time of the video file.

[0006] By removing redundant frames, a snapshot can be created in which key frames containing target information are not missed when creating a snapshot.

[0007] Using hashing, we can determine whether a frame contains a user-defined class using a zero-shot technique.

[0008] A method for generating a video snapshot executed on a computer device, the computer device including at least one processor configured to execute computer-readable instructions contained in a memory, the method for generating a video snapshot including the step of removing, by the at least one processor, redundant frames from frames included in a video and generating a snapshot with the remaining frames, while maintaining frames containing target information defined by a user to generate the snapshot.

[0009] According to one aspect, the generating step may generate a snapshot of the video using a hash generated through a machine learning model.

[0010] According to another aspect, the generating step can generate a snapshot of the video through a multi-modal model learned by expressing embeddings of different modalities in a single vector space.

[0011] According to another aspect, the generating step may include: calculating a first similarity indicating image similarity between frames included in the video; calculating a second similarity indicating a similarity relationship with a prompt registered as the target information for each frame included in the video; and selecting some frames included in the video as the duplicate frames based on the first similarity and the second similarity.

[0012] According to another aspect, the step of calculating the first similarity calculates the first similarity by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space, and the first similarity can be calculated through the vector distance between image embeddings that embed features of images corresponding to each frame.

[0013] According to another aspect, the step of calculating the second similarity may include a step of registering the prompt as content in any one of text, image, and audio forms.

[0014] According to another aspect, the step of calculating the second similarity calculates the second similarity by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space, and the second similarity can be calculated through the vector distance between the prompt embedding that embeds the features of the prompt registered as the target information and the image embedding that embeds the features of the image corresponding to each frame.

[0015] According to another aspect, the selecting step may include: a step of clustering frames having a similarity of a certain level or higher based on the first similarity; a step of selecting frames remaining after excluding a certain number of frames per cluster as candidate frames for the duplicate frames; and a step of excluding frames having the second similarity of a certain level or higher among the candidate frames from the selection target for the duplicate frames.

[0016] According to another aspect, the multi-modal model may be a model in which the similarity of embeddings for data pairs is learned using contrastive learning.

[0017] According to another aspect, the multi-modal model may be a vision-language pre-trained (VLP) model learned through contrastive learning.

[0018] A computer program stored in a non-transitory computer-readable recording medium is provided for executing the above video snapshot generation method on the computer device.

[0019] In a computer device, at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor processes a process of generating a snapshot by removing duplicate frames from frames contained in a video and using the remaining frames, while maintaining frames containing target information defined by a user.

[0020] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment of the present invention.

[0021] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0022] Figure 3 illustrates an example of frames that make up a video.

[0023] Figure 4 illustrates an example of creating a video snapshot.

[0024] FIG. 5 is a flowchart illustrating an example of a method that a computer device according to an embodiment of the present invention can perform.

[0025] FIGS. 6 and 7 are exemplary drawings for explaining a searchable multi-modal model in one embodiment of the present invention.

[0026] FIG. 8 is a flowchart illustrating an example of a duplicate frame filtering process in one embodiment of the present invention.

[0027] Figures 9 and 10 illustrate examples of snapshot creation using a prompt in one embodiment of the present invention.

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0029] Embodiments of the present invention relate to a technique for generating a video snapshot.

[0030] Embodiments including those specifically disclosed herein can generate a snapshot in a form that retains key frames containing target information when generating a video snapshot.

[0031] A video snapshot generation system according to embodiments of the present invention can be implemented by at least one computer device, and a video snapshot generation method according to embodiments of the present invention can be performed by at least one computer device included in the video snapshot generation system. At this time, a computer program according to an embodiment of the present invention can be installed and run on the computer device, and the computer device can perform the video snapshot generation method according to embodiments of the present invention under the control of the run computer program. The above-described computer program can be stored on a computer-readable recording medium so as to be combined with the computer device and cause the computer to execute the video snapshot generation method.

[0032] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 represents an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is merely an example for explaining the invention, and the number of electronic devices or servers is not limited to that of FIG. 1. In addition, the network environment of FIG. 1 merely illustrates one example of environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0033] The plurality of electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices (110, 120, 130, 140) include smart phones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, in FIG. 1, the shape of a smart phone is shown as an example of the electronic device (110), but in embodiments of the present invention, the electronic device (110) may actually mean one of various physical computer devices that can communicate with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0034] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (170) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0035] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) via a network (170) to provide commands, codes, files, contents, services, etc. For example, the server (150) may be a system that provides a service (e.g., a video analysis service, etc.) to multiple electronic devices (110, 120, 130, 140) connected via a network (170).

[0036] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the multiple electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0037] Such a computer device (200) may include a memory (210), a processor (220), a communication interface (230), and an input / output interface (240), as illustrated in FIG. 2. The memory (210) may be a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a non-permanent mass storage device such as a disk drive. Here, the non-permanent mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). In addition, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) via a communication interface (230) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received over a network (170).

[0038] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0039] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices described above) via the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program codes stored in a recording device such as the memory (210) may be transmitted to other devices via the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) via the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0040] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (250) may also be configured as a single device with the computer device (200).

[0041] Additionally, in other embodiments, the computer device (200) may include fewer or more components than those illustrated in FIG. 2. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a database, etc.

[0042] Below, specific embodiments of a method and device for generating a video snapshot using an image hash are described.

[0043] Video snapshots can be used to analyze video files such as MP4 or AVI, or animated images such as GIF or WEBP.

[0044] For example, a uniform sampler can be applied to detect one frame at a specific time to generate a video snapshot.

[0045] As another example, a snapshot can be generated by calculating the sum of absolute difference (SAD), which represents the difference in pixel values ​​between neighboring frames, and then detecting frames where the SAD is greater than a user-defined value.

[0046] In addition to the above-described methods, an algorithm for removing duplicate frames through image processing based hash technology can be applied to generate video snapshots.

[0047] In this specification, a "duplicate frame" can refer to a perfectly identical frame, as well as a frame with a certain degree of similarity. For example, a frame with slight variations, such as translation or rotation, can be recognized as a duplicate frame. In practice, most similar frames can be considered "near duplicates."

[0048] The present embodiments can remove some of the frames contained in a given video and create a snapshot using only the remaining frames.

[0049] When creating a video snapshot, certain frames, such as duplicate frames, may be removed from the video, potentially resulting in the loss of key frames. For example, when creating a video snapshot for content monitoring, frames containing target information for monitoring may be removed, resulting in the snapshot being missing.

[0050] Figure 3 illustrates an example of a video frame for snapshot generation. As illustrated in Figure 3, it is assumed that the given video (30) consists of 10 frames.

[0051] Based on the image similarity between frames included in the video (30), duplicate frames can be removed and a snapshot can be generated with the remaining frames. Among the 10 frames included in the video (30), there is a high probability that one of frames (1) and (2) will be removed due to image similarity. For example, with respect to the video (30) of FIG. 3, as shown in FIG. 4, frames (1), (4), (5), (7), and (10) are selected as duplicate frames and deleted, and then a snapshot (41) can be generated with frames (2), (3), (6), (8), and (9).

[0052] For example, QR (quick response code) codes used on messengers exist for various purposes, such as linking to official accounts (OAs) or sharing for personal meetings. When monitoring video content, some QR codes may be prohibited from sharing and require censorship.

[0053] However, in an algorithm that removes duplicate frames based on image similarity, it is difficult to distinguish between frames (1) and (2), so frames that require censorship may be detected as duplicate frames and deleted.

[0054] Therefore, a technology is needed that can retain frames containing target information, such as key monitoring content, when creating snapshots for video analysis, highlight production, etc.

[0055] The computer device (200) according to the present embodiment can provide a video analysis service to a client through a dedicated application installed on the client or access to a web / mobile site related to the computer device (200). The computer device (200) can be configured with a computer-implemented video snapshot generation system. For example, the video snapshot generation system can be implemented in the form of an independently operating program, or can be configured in the form of an in-app of a specific application so as to be operable on the specific application.

[0056] The processor (220) of the computer device (200) may be implemented as a component for performing the following video snapshot generation method. Depending on the embodiment, the components of the processor (220) may be selectively included or excluded from the processor (220). Furthermore, depending on the embodiment, the components of the processor (220) may be separated or combined to express the functions of the processor (220).

[0057] These processors (220) and components of the processor (220) may control the computer device (200) to perform the steps included in the following video snapshot generation method. For example, the processor (220) and components of the processor (220) may be implemented to execute instructions according to the code of the operating system included in the memory (210) and the code of at least one program.

[0058] Here, the components of the processor (220) may be representations of different functions performed by the processor (220) according to instructions provided by the program code stored in the computer device (200).

[0059] The processor (220) can read necessary commands from the memory (210) loaded with commands related to the control of the computer device (200). In this case, the read commands may include commands for controlling the processor (220) to execute steps to be described later.

[0060] The steps included in the method for generating a video snapshot, which will be described hereinafter, may be performed in a different order than the order shown, and some of the steps may be omitted or additional processes may be included.

[0061] The steps included in the method for generating a video snapshot may be performed on the server (150), and in some embodiments, at least some of the steps may also be performed on the client.

[0062] FIG. 5 is a flowchart illustrating an example of a method that a computer device according to one embodiment of the present invention can perform.

[0063] Referring to FIG. 5, in step S510, the processor (220) can calculate the image similarity (hereinafter referred to as “first similarity”) between frames included in the video. For example, the processor (220) can calculate the degree of similarity between frames using a hash based on machine learning. The processor (220) can calculate the first similarity, which represents the image similarity through the distance between the embedding vectors by embedding the features of each image corresponding to the frame.

[0064] In step (S520), the processor (220) may calculate the similarity (hereinafter referred to as “second similarity”) between each frame included in the video and a prompt defined by the user. Similarly, the processor (220) may calculate the degree of similarity between the frame and the prompt using a machine learning-based hash. In the present embodiment, the prompt may be used to define target information to be included in the snapshot, and the prompt may be input by the user in various forms of content such as text, image, and audio. The processor (220) may embed the features of the target information defined through the prompt and calculate the second similarity through the vector distance between the prompt embedding and the image embedding of each frame. In the present embodiment, the hash may be used to determine the similarity between the class defined by the user for the image frame using a zero-shot technique.

[0065] In step (S530), the processor (220) may select duplicate frames to be removed based on the first similarity and the second similarity among the frames included in the video. The processor (220) may cluster frames having a similarity level higher than a certain level based on the first similarity, and then select the remaining frames, excluding a predetermined number of frames (for example, 1) for each cluster, as candidate frames. At this time, the processor (220) may finally select a frame among the candidate frames having a second similarity level lower than a certain level as a duplicate frame. In other words, the processor (220) may exclude a frame among the candidate frames having a second similarity level higher than a certain level from the selection target of the duplicate frame.

[0066] In step (S540), the processor (220) can remove frames selected as duplicate frames from among the frames included in the video and create a snapshot with the remaining frames. After clustering the frames included in the video based on the first similarity, a snapshot can be created while maintaining the maximum total frames of a specific cluster corresponding to the second similarity. In the present embodiment, duplicate frames are removed based on image similarity and a snapshot can be created with the remaining frames, while including as many frames as possible that have a similar relationship with a class defined by the user.

[0067] FIGS. 6 and 7 are exemplary drawings for explaining a searchable multi-modal model in one embodiment of the present invention.

[0068] The present invention can use a multi-modal model that enables retrieval using clustering and various types of prompts.

[0069] Referring to FIG. 6, the multi-modal model according to the present invention includes a method of extracting embeddings for various modalities such as text (601), video (or image) (602), and audio (603), and learning so that embeddings of different modalities can be shared in one vector space (600).

[0070] The above-mentioned model learning method can utilize contrastive learning, etc., and a representative method is a large-scale VLP (vision-language pre-trained) model such as CLIP (contrastive language-image pre-training model).

[0071] Referring to FIG. 7, in CLIP, learning is performed so that the similarity of the embeddings extracted from the text encoder (701) and the image encoder (702) for image-text pairs corresponding to contexts with the same meaning increases, and the similarity of the embeddings extracted from the text encoder (701) and the image encoder (702) for image-text pairs corresponding to contexts with different meanings decreases.

[0072] In the case of an image encoder (702) trained together with a text encoder (701), it can be used for dual purposes as it can represent the visual characteristics of an image well while simultaneously representing the semantic characteristics of a text.

[0073] Image-text embedding is a method for embedding and combining features of images and texts. It can well express the relationship between embedded classes and can be mainly used for image-text retrieval, image captioning, and visual question answering (VQA).

[0074] FIG. 8 is a flowchart illustrating an example of a duplicate frame filtering process in one embodiment of the present invention.

[0075] In the present invention, filtering using a prompt defined by a user and filtering using image similarity can be performed simultaneously using an embedding extracted through a multi-modal model.

[0076] Referring to FIG. 8, the processor (220) can extract image embedding for each frame included in the video through an image encoder (702) trained together with a text encoder (701), and at this time, image similarity between frames can be calculated using the frame-by-frame embedding (S801).

[0077] The processor (220) can input a text-type prompt (80) by a user, and at this time, a text embedding for the prompt (80) can be extracted through a text encoder (701) trained together with an image encoder (702).

[0078] For example, if you want to censor videos that contain QR codes during video analysis, you can input a prompt such as "qr", "qr image", "photo of qr code", or "qr code image" as text representing the target information.

[0079] The processor (220) can compare the image embedding extracted through the image encoder (702) for each frame included in the video with the text embedding extracted through the text encoder (701) to calculate the similarity with the prompt (S802).

[0080] In other words, the processor (220) can determine whether each frame contains a user-defined class based on the similarity with the prompt through a zero-shot technique using a hash generated based on machine learning.

[0081] The processor (220) can select a frame among the frames included in the video that has an image similarity level higher than a certain level with other frames as a duplicate frame, and at this time, among the frames selected as duplicate frames, a frame that has a similarity level higher than a certain level with a prompt can be excluded from the selection of duplicate frames (S803).

[0082] Accordingly, the processor (220) can remove frames selected as duplicate frames from among the frames included in the video and create a snapshot with the remaining frames, and at this time, frames having a similarity level with the prompt above a certain level can be included in the snapshot.

[0083] In the case of an algorithm that removes duplicate frames based on image similarity, one of frame (1) and frame (2) containing a QR code may be selected as a duplicate frame and deleted, but in this embodiment, a user may specify desired target information through a prompt (80) to maintain the frame containing the target information and generate a video snapshot.

[0084] Figures 9 and 10 illustrate examples of snapshot creation using a prompt in one embodiment of the present invention.

[0085] Figures 9 and 10 show examples of snapshot generation of a video (30) consisting of 10 frames from (1) to (10).

[0086] As illustrated in FIG. 9, when a user inputs the text 'QR' as a prompt to define desired target information, the processor (220) can exclude frames (1) and (2) containing QR codes among the 10 frames included in the video (30) from the selection of duplicate frames.

[0087] For example, a snapshot (91) for a video (30) may be composed of frames (1), (2), (3), (6), (8), and (9) in which frames (4), (5), (7), and (10) are selected as duplicate frames and deleted, respectively. That is, the processor (220) may create a snapshot (91) in a form in which frames (1) and (2) in which a QR code exists as target information defined through a prompt are maintained.

[0088] In the above embodiment, a text prompt is described as an example, but it is not limited thereto, and filtering using a prompt in the form of audio or image can also be applied.

[0089] Embeddings of different modalities are represented in a single vector space, and the learned model can be used with zero-shot techniques to generate snapshots that do not miss frames containing target information using not only text prompts but also other forms of modality prompts.

[0090] For example, as illustrated in FIG. 10, if a user inputs the audio 'sky' as a prompt for defining desired target information, the features of the audio 'sky' can be embedded to maintain frames having a similarity level higher than a certain level with the prompt embedding. The processor (220) can create a snapshot (101) in a form of maintaining frames (7) and (8) corresponding to the 'sky' information as target information defined through the audio prompt. In other words, the snapshot (101) for the video (30) can be composed of frames (2), (3), (6), (7), (8), and (9) with frames (1), (4), (5), and (10) deleted as duplicate frames.

[0091] By applying a method using machine learning-based hashing, it is possible to search for frames related to specific keywords entered in the form of text, audio, images, etc., as well as frames containing specific sounds (e.g., cheering sounds, car crash sounds, wave sounds, alarm sounds, etc.).

[0092] These embodiments incorporate zero-shot features, allowing them to target and apply all information from pre-trained modalities. Specific keywords, which can be entered in text, audio, or image formats, can be registered as target information. Additionally, various sounds, adult content, or violent content can be registered via prompts, allowing users to search for frames containing the desired information.

[0093] According to embodiments of the present invention, by removing duplicate frames, a video snapshot can be created in which key frames containing target information are not omitted. Furthermore, according to embodiments of the present invention, by determining whether a frame contains a user-defined class through a zero-shot technique using a hash generated based on machine learning, a video snapshot can be created by maximizing the inclusion of frames with a similar relationship to the user-defined class.

[0094] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0095] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0096] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0097] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0098] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for creating a video snapshot running on a computer device, The computer device comprises at least one processor configured to execute computer-readable instructions contained in a memory, The above video snapshot creation method is: A step of generating a snapshot by removing redundant frames from frames included in a video and using the remaining frames, by at least one processor, while maintaining frames containing target information defined by a user. A method for creating a video snapshot including:

2. In paragraph 1, The above generating steps are: Creating a snapshot of the video using a hash generated through a machine learning model A method for creating a video snapshot, characterized by:

3. In paragraph 1, The above generating steps are: Generating snapshots of the video using a multi-modal model learned by representing embeddings of different modalities in a single vector space. A method for creating a video snapshot, characterized by:

4. In paragraph 1, The above generating steps are: A step of calculating a first similarity indicating image similarity between frames included in the above video; A step of calculating a second similarity indicating a similarity relationship with a prompt registered as the target information for each frame included in the video; and A step of selecting some frames included in the video as the duplicate frames based on the first similarity and the second similarity. A method for creating a video snapshot including:

5. In paragraph 4, The step of calculating the first similarity is as follows: The first similarity is calculated by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space. Calculating the first similarity through the vector distance between image embeddings that embed the features of the image corresponding to each frame. A method for creating a video snapshot, characterized by:

6. In paragraph 4, The step of calculating the second similarity is as follows: Step of registering the above prompt as content in any form of text, image, or audio. A method for creating a video snapshot including:

7. In paragraph 4, The step of calculating the second similarity is as follows: The second similarity is calculated by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space. Calculating the second similarity through the vector distance between the prompt embedding that embeds the features of the prompt registered as the target information and the image embedding that embeds the features of the image corresponding to each frame. A method for creating a video snapshot, characterized by:

8. In paragraph 4, The above selection steps are: A step of clustering frames having a similarity level higher than a certain level based on the first similarity; A step of selecting the remaining frames, excluding a certain number of frames per cluster, as candidate frames for the duplicate frames; and A step of excluding frames among the above candidate frames whose second similarity is above a certain level from the selection target of the duplicate frames. A method for creating a video snapshot including:

9. In paragraph 3, The above multi-modal model is a model that learns the similarity of embeddings for data pairs using contrastive learning. A method for creating a video snapshot, characterized by:

10. In paragraph 3, The above multi-modal model is a VLP (vision-language pre-trained) model learned through contrastive learning. A method for creating a video snapshot, characterized by:

11. A computer program stored on a non-transitory computer-readable recording medium for executing the video snapshot generation method of any one of claims 1 to 10 on the computer device.

12. In computer devices, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor, A process of removing duplicate frames from frames included in a video and creating a snapshot with the remaining frames, while maintaining frames containing target information defined by the user. A computer device that processes.

13. In paragraph 12, At least one processor, Generating a snapshot of the above video using a hash generated by a machine learning model. A computer device characterized by:

14. In paragraph 12, At least one processor, Generating snapshots of the above video using a multi-modal model learned by representing embeddings of different modalities in a single vector space. A computer device characterized by:

15. In paragraph 12, At least one processor, Compute a first similarity indicating image similarity between frames included in the above video, For each frame included in the above video, a second similarity is calculated indicating a similarity relationship with the prompt registered as the target information, Selecting some frames included in the video as the duplicate frames based on the first similarity and the second similarity. A computer device characterized by:

16. In paragraph 15, At least one processor, The first similarity is calculated by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space. Calculating the first similarity through the vector distance between image embeddings that embed the features of the image corresponding to each frame. A computer device characterized by:

17. In paragraph 15, At least one processor, Registering the above prompt as content in any form of text, image, or audio A computer device characterized by:

18. In paragraph 15, At least one processor, The second similarity is calculated by using a multi-modal model learned by expressing embeddings of different modalities in a single vector space. Calculating the second similarity through the vector distance between the prompt embedding that embeds the features of the prompt registered as the target information and the image embedding that embeds the features of the image corresponding to each frame. A computer device characterized by:

19. In paragraph 15, At least one processor, Clustering frames with a similarity level higher than a certain level based on the first similarity above, Select the remaining frames, excluding a certain number of frames per cluster, as candidate frames for the above duplicate frames, Among the above candidate frames, frames having a second similarity level above a certain level are excluded from the selection target of the duplicate frame. A computer device characterized by:

20. In paragraph 14, The above multi-modal model is a VLP model that learns the similarity of embeddings for data pairs using contrastive learning. A computer device characterized by:

Images