Method and apparatus for detecting similar image

Abstract

A method and an apparatus for detecting a similar image are disclosed. The present embodiment provides a method and an apparatus for detecting a similar image, which extract a feature value vector constituting an axis of a vector space on the basis of a combined image obtained by combining time-axis images or space-axis images, retrieve a similar image located closest to the feature value vector in the vector space, and determine an attribute of an input image by using at least one of the similar image and metadata so as to enable advertisement detection, category classification, and similarity analysis using detection of the similar image.

Description

Similar image detection method and device

[0001] One embodiment of the present invention relates to a method and device for detecting similar images.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] When entering keywords to search for an image, users sometimes fail to find the specific image they're looking for. To improve image search failures, users should enter specific keywords for the image they're looking for, and the images should be categorized according to each specific keyword.

[0004] Image search systems build and use image databases based on a variety of technologies. For example, image search systems utilize image tagging information for search, and utilize methods to recognize image content and store it in a database for retrieval, enabling content-based searches.

[0005] An image search system using the semantic web, which is an advanced version of the general keyword-based search, has been built and is in service, but it is technically insufficient to ensure efficient search.

[0006] Since general image search technologies are based on human knowledge and are utilized for keyword mapping, relationship mapping, etc., there is a problem in that concepts that humans do not recognize or concepts and attributes that humans have not explicitly developed as knowledge cannot be utilized as search keys.

[0007] The purpose of the present embodiment is to provide a method and device for detecting similar images, which extract a feature value vector constituting an axis of a vector space based on a combined image that combines a time-axis image or a space-axis image, search for a similar image that is closest to the feature value vector in the vector space, and determine an attribute of an input image using at least one of the similar image and metadata, thereby enabling advertisement detection, category classification, and similarity analysis using similar image detection.

[0008] According to one aspect of the present embodiment, a similar image detection device is provided, characterized by including: a combined image generator that generates a combined image by sampling a plurality of input images on a time axis or a space axis and then combining them into one image; an image retrieval engine that extracts a feature value vector constituting an axis of a vector space based on the combined image, searches for a similar image that is closest to the feature value vector in the vector space, and outputs the similar image and metadata corresponding to the similar image.

[0009] As described above, according to the present embodiment, there is an effect of extracting a feature value vector constituting an axis of a vector space based on a combined image that combines a time-axis image or a space-axis image, searching for a similar image that is closest to the feature value vector in the vector space, and determining the properties of the input image using at least one of the similar image and metadata, thereby performing advertisement detection, category classification, and similarity analysis using similar image detection.

[0010] FIG. 1 is a diagram illustrating a similar image attribute information detector according to the present embodiment.

[0011] Fig. 2 is a diagram showing a learning device according to the present embodiment.

[0012] Figure 3 is a diagram illustrating an advertisement detection process using similar image detection according to the present embodiment.

[0013] Figure 4 is a diagram showing the category classification and application process according to the present embodiment.

[0014] Fig. 5 is a drawing showing an example of content attribute information according to the present embodiment.

[0015] Figure 6 is a diagram showing the entire process according to the present embodiment.

[0016] Fig. 7 is a diagram showing an example of similarity analysis according to the present embodiment.

[0017] Hereinafter, the present embodiment will be described in detail with reference to the attached drawings.

[0018] FIG. 1 is a diagram illustrating a similar image attribute information detector according to the present embodiment.

[0019] A similar image detection device according to the present embodiment includes a similar image attribute information detector (100) and a learner (200).

[0020] The similar image attribute information detector (100) according to the present embodiment includes a combined image generator (110), an image retrieval engine (120), and a content attribute determination unit (130). The components included in the similar image attribute information detector (100) are not necessarily limited thereto.

[0021] The combined image generator (110) generates a single combined image by composing one or more images. The combined image generator (110) generates a combined image by combining multiple images extracted along a spatial axis from one or more input images. The combined image generator (110) extracts a certain number of input images from a temporally continuous image sequence from a video at one or more of a certain time interval, a variable time interval, and a specific feature, and then generates a combined image by combining the extracted input images.

[0022] The combined image generator (110) samples multiple input images on the time axis or the space axis and then creates a combined image by combining them into one image.

[0023] The Image Retrieval Engine (120) includes a search engine having a vector space database (128) that uses a single image as a search unit. The Image Retrieval Engine (120) includes an image search engine having a vector space database (128) that uses a combined image, which is a combination of multiple images, as a search unit.

[0024] The image retrieval engine (120) extracts feature value vectors that constitute an axis of a vector space based on a combined image, searches for a similar image that is closest to the feature value vector in the vector space, and outputs the similar image and metadata corresponding to the similar image.

[0025] The image retrieval engine (120) includes a feature value vector extraction unit (122), a similar image extraction unit (124), a similar image and meta information output unit (126), and a vector space database (128).

[0026] The feature vector extraction unit (122) configures a vector dimension and extracts a feature vector. The feature vector extraction unit (122) receives a combined image as input from the combined image generator (110), extracts feature elements that configure the axes of the vector space, and generates and outputs a feature vector.

[0027] In other words, the feature value vector extraction unit (122) extracts feature elements constituting the axes of the vector space from the combined image to generate a feature value vector.

[0028] The feature value vector extraction unit (122) extracts a feature value vector defined by reflecting the feature elements according to the image composition method of the combined image.

[0029] The feature value vector extraction unit (122) extracts elements such as the direction of movement and size according to the order of the images in the combined image, or the direction of movement and speed of an object, as feature values ​​when the image composition of the input combined image has temporal properties according to the image order, and uses them as axis information of the vector space.

[0030] The feature value vector extraction unit (122) extracts color, complexity, state, etc. within the composition image within the combined image as feature values ​​when the image composition of the input combined image has spatial feature elements according to the image space location, and uses them as axis information of the vector space.

[0031] The similar image extraction unit (124) extracts a similar image that is closest to the feature value vector in the vector space from the vector space database (128). In other words, the similar image extraction unit (124) searches the vector space database (128) and extracts a similar image that is closest to the feature value vector extracted from the combined image in the vector space. Here, the similar image includes one or more images.

[0032] The vector space database (128) constitutes a vector space with feature elements as axes. The vector space database (128) includes data constituting a vector space with feature elements as axes.

[0033] The similar image and meta information output unit (126) outputs the similar image received from the similar image extraction unit (124) and the meta information (label, similarity) corresponding to the similar image.

[0034] The content property determination unit (130) determines the properties of the input image based on similar images and metadata. In other words, the content property determination unit (130) determines the properties of the input image using similar images and metadata corresponding to the similar images included in the output information from the image retrieval engine (120) and outputs the determined properties.

[0035] The content property determination unit (130) has a determination method and determination criteria according to the content property to be determined, determines the determination method based on the meta information input from the image retrieval engine (120), and determines the determination target property with reference to the determination criteria.

[0036] The content property determination unit (130) determines whether the input image is content having a specific property (e.g., advertisement) by using at least one or more of the meta information (label, similarity) corresponding to the similar image based on the similar image and content property matching function and outputs the result. In other words, the content property determination unit (130) determines whether the input content is content having a specific property (e.g., advertisement) by using at least one or more of the similar images for the input image output from the image retrieval engine (120) and the meta information (label, similarity) corresponding to the similar image based on the content property matching function.

[0037] The content property determination unit (130) determines whether the input image is content classified into a specific field (e.g., drama, sports, entertainment, etc.) by using at least one or more of similar images and meta information (label, similarity) corresponding to the similar images based on the content category classification function and outputs the determined content. In other words, the content property determination unit (130) determines which field (e.g., drama, sports, entertainment, etc.) the input content is content of by using at least one or more of similar images and meta information (label, similarity) corresponding to the similar images for the input image output from the image retrieval engine (120) based on the content category classification function.

[0038] The content property determination unit (130) determines whether the input image is content classified as a specific event (e.g., strike / ball, happy / angry, etc.) by using at least one or more of similar images and meta information (label, similarity) corresponding to the similar images based on the content property context detection function and outputs the determined content. In other words, the content property determination unit (130) determines whether the input content is content including a specific event (e.g., strike / ball, happy / angry, etc.) by using at least one or more of similar images and meta information (label, similarity) corresponding to the similar images for the input image output from the image retrieval engine (120) based on the content property context detection function.

[0039] Fig. 2 is a diagram showing a learning device according to the present embodiment.

[0040] The learning device (200) according to the present embodiment includes a learning data set configuration unit (210) and a database construction unit (220). The components included in the learning device (200) are not necessarily limited thereto.

[0041] The learning data set configuration unit (210) generates a learning data set by specifying the combined image received from the combined image generator (110) and the target feature information and corresponding label for the combined image.

[0042] The database construction unit (220) constructs a vector space database (128) by configuring the dimension of a vector space with feature elements and assigning a label to a specific area in a vector space constructed with the dimension of the vector space.

[0043] In other words, the database construction unit (220) constructs a vector space database (128) by constructing a dimension of a vector space with feature elements constituting the axes of a vector space extracted from a combined image and assigning labels to specific areas in the vector space.

[0044] The database construction unit (220) includes a vector space axis feature element extractor learning unit (222), a data label designation unit (224), a vector space location determination and label-specific similarity generation unit (226), and a vector space database construction unit (228).

[0045] The vector space axis feature element extractor learning unit (222) trains an artificial intelligence network that extracts feature element values ​​for an input image using the learning data set input from the learning data set composition unit (210).

[0046] In other words, the vector space axis feature element extractor learning unit (222) trains an artificial intelligence network that extracts feature elements for an input image using a learning data set.

[0047] The vector space axis feature extractor learning unit (222) specifies temporal image feature elements as feature elements and applies them to learning. The vector space axis feature extractor learning unit (222) specifies spatial image feature elements as feature elements and applies them to learning.

[0048] The data labeling unit (224) maps data onto a vector space and labels the mapped area. In other words, the data labeling unit (224) maps feature elements onto a vector space and labels the mapped area.

[0049] The vector space location determination and label-based similarity generation unit (226) uses a vector composed of feature elements to designate a location and label in the vector space. The vector space database construction unit (228) uses location information, meta information, and label information in the vector space to build a vector space database (128). The vector space database construction unit (228) receives a combined image as input, extracts feature elements, and labels the positions corresponding to the feature elements in the vector space to build the vector space database (128).

[0050] The vector space database construction unit (228) uses the target feature information input from the learning data set construction unit (210) and the feature elements extracted in correspondence with the temporal order of the images configured in the combined image when the vector space axis feature element extractor learning unit (222) learns. The vector space database construction unit (228) uses the target feature information input from the learning data set construction unit (210) and the feature elements extracted in correspondence with the spatial position of the images configured in the combined image when the vector space axis feature element extractor learning unit (228) learns.

[0051] Figure 3 is a diagram illustrating an advertisement detection process using similar image detection according to the present embodiment.

[0052] A similar image detection device receives advertisement images (e.g., advertisement 1, advertisement 2, non-advertisement) as input (S310). In step S310, the similar image detection device samples advertisement images (e.g., advertisement 1, advertisement 2, non-advertisement) along a time axis or a space axis and then combines them into a single combined image.

[0053] The similar image detection device extracts (1…n) vector elements from the combined image (S320). In step S320, the similar image detection device extracts feature elements constituting the axes of the vector space based on the combined image.

[0054] The similar image detection device generates a vector based on vector elements for the combined image (S330). In step S330, the similar image detection device generates and outputs a feature value vector based on feature elements constituting the axes of the vector space for the combined image.

[0055] The similar image detection device generates a vector space location based on vector space mapping (S340).

[0056] The similar image detection device extracts a similar image (advertisement section) by referencing a vector space database (128) (S350). In step S350, the similar image detection device searches the vector space database (128) for a similar image (advertisement section) that is closest to the feature value vector extracted from the combined image in the vector space and extracts it.

[0057] The similar image detection device outputs the extracted similar image (advertisement section) (S360). In step S360, the similar image detection device outputs the extracted similar image (advertisement section) and meta information (label, similarity) corresponding to the similar image (advertisement section).

[0058] Although FIG. 3 describes steps S310 to S360 as being executed sequentially, this is not necessarily the case. In other words, the steps described in FIG. 3 can be modified and executed, or one or more steps can be executed in parallel, so FIG. 3 is not limited to a chronological order.

[0059] As described above, the advertisement detection process using similar image detection according to the present embodiment described in FIG. 3 can be implemented as a program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the advertisement detection process using similar image detection according to the present embodiment is recorded includes all types of recording devices that store data that can be read by a computer system.

[0060] Figure 4 is a diagram showing the category classification and application process according to the present embodiment.

[0061] A similar image detection device receives images of various categories (e.g., dramas, sports, advertisements) (S410). In step S410, the similar image detection device samples images of various categories (e.g., dramas, sports, advertisements) along the time or space axis and then combines them into a single combined image.

[0062] The similar image detection device extracts vector elements (1…n) from the combined image (S420). In step S420, the similar image detection device extracts feature elements constituting the axes of the vector space based on the combined image.

[0063] The similar image detection device generates a vector based on vector elements for the combined image (S430). In step S430, the similar image detection device generates and outputs a feature value vector based on feature elements constituting the axes of the vector space for the combined image.

[0064] The similar image detection device generates a vector space location based on vector space mapping (S440).

[0065] The similar image detection device extracts a similar image (an image with a feature label) by referencing a vector space database (128) (S450). In step S450, the similar image detection device searches the vector space database (128) for a similar image (an image with a feature label) that is closest to the feature value vector extracted from the combined image in the vector space and extracts it.

[0066] The similar image detection device outputs the extracted similar image (image with feature labels) or the feature labels (S460). In step S460, the similar image detection device outputs the extracted similar image (image with feature labels), meta information (label, similarity) corresponding to the similar image (image with feature labels), and the feature labels.

[0067] The similar image detection device controls application functions based on the extracted similar image (image with feature labels), meta information (labels, similarity) corresponding to the similar image (image with feature labels), and feature labels (S470). In step S470, the similar image detection device controls the classification of the input image category based on the feature labels, and performs application functions such as encoding parameter control using the classified category information.

[0068] Although FIG. 4 describes steps S410 to S470 as being executed sequentially, this is not necessarily the case. In other words, it is possible to change the steps described in FIG. 4 and execute them, or to execute one or more steps in parallel, and thus FIG. 4 is not limited to a chronological order.

[0069] As described above, the category classification and application process according to the present embodiment described in FIG. 4 can be implemented as a program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the category classification and application process according to the present embodiment is recorded includes all types of recording devices that store data that can be read by a computer system.

[0070] Fig. 5 is a drawing showing an example of content attribute information according to the present embodiment.

[0071] The similar image detection device extracts a feature value vector constituting an axis of a vector space based on a combined image that combines input images (time axis images or space axis images), searches for a similar image that is closest to the feature value vector in the vector space, and determines the properties of the input image using the similar image and metadata corresponding to the similar image.

[0072] The similar image detection device can output one or more of action recognition information (AR), scene recognition information (IMAGE Recognition), facial expression recognition information (FER, Facial-Expression Recognition), optical character recognition information (OCR, Optical-Character Recognition), and video captioning information based on the properties of a previously constructed image for a new input image.

[0073] Figure 6 is a diagram showing the entire process according to the present embodiment.

[0074] A similar image detection device extracts a feature value vector constituting an axis of a vector space based on a combined image that combines input images (temporal axis images or spatial axis images), and when constructing a vector space database (128) constructed with data constituting a vector space with feature elements as axes, a Vision Transformer (ViT) can be used to extract the feature value vector. Here, Vision Transformer (ViT) refers to a deep learning model that performs computer vision tasks using a Transformer architecture instead of a conventional convolutional neural network (CNN) architecture.

[0075] The similar image detection device is based on the Vision Transformer (ViT), and applies the transformer architecture, which was mainly used for natural language processing tasks, to vision tasks to process image information and perform tasks such as classification, object detection, and segmentation.

[0076] The similar image detection device performs input representation of the input image based on the Vision Transformer (ViT). The similar image detection device divides the input image into patches and processes them. Each patch has a constant size, and each patch is expressed in vector form rather than pixel units.

[0077] The similar image detection device performs tokenization based on the Vision Transformer (ViT). The similar image detection device embeds each patch using an embedding layer and then processes it into tokens using the Transformer.

[0078] The image-like detection device performs positional encoding based on the Vision Transformer (ViT). Similar to the Transformer, the image-like detection device performs positional encoding on the location information of the input patch and provides it as a model.

[0079] The pseudo-image detection device implements a multi-head self-attention mechanism based on the Vision Transformer (ViT). The pseudo-image detection device models the interactions between tokens using the multi-head self-attention mechanism, a core component of the Transformer. Based on this modeling of interactions between tokens, the pseudo-image detection device can identify relationships between parts of an image.

[0080] The similar image detection device creates a feed-forward neural network (FNN) based on the Vision Transformer (ViT). The FNN follows a multihead attention layer with a feed-forward neural network layer, adding nonlinearity and extracting features.

[0081] The similar image detection device performs layer normalization based on the Vision Transformer (ViT). The similar image detection device stabilizes each layer by performing layer normalization on the output of each layer.

[0082] The similar image detection device uses a Transformer Encoder Stack based on the Vision Transformer (ViT). The similar image detection device uses multiple Transformer Encoder Stacks to extract various features and hierarchical structures from an image.

[0083] The similar image detection device creates a classification head based on the Vision Transformer (ViT). The similar image detection device transmits the output of the ViT model to the head for classification.

[0084] The similar image detection device uses the Euclidean or Cosine method when searching the vector space database (128) constructed to find similar data at the closest location in the vector space to the feature value vector extracted from the combined image.

[0085] The similar image detection device can measure and compare the similarity between the feature value vector and the data located closest to it in the vector space based on the Euclidean distance and cosine similarity.

[0086] When identifying the data closest to a feature vector in vector space, the similar image detection device measures the linear distance (Euclidean distance) between data points based on the Euclidean distance. The similar image detection device considers two data points more similar when the Euclidean distance is smaller. Since the similar image detection device considers the distance between data points when measuring similarity based on the Euclidean distance, it can measure spatial distances.

[0087] A similar image detection device can assess similarity by measuring the angle between data points based on cosine similarity when identifying the data closest to a feature vector in vector space. When representing data points as vectors, the similar image detection device can calculate the cosine similarity between two vectors. When measuring similarity based on cosine similarity, the similar image detection device considers the direction and angle of the data points.

[0088] Fig. 7 is a diagram showing an example of similarity analysis according to the present embodiment.

[0089] The similar image detection device can perform similarity analysis in pixel space when searching a vector space database (128) constructed with similar data at the closest location in vector space to the feature value vector extracted from the combined image.

[0090] A similar image detection device generates a combined image by combining input images (e.g., a baseball game image) into a single pixel space. Based on the combined image combined in pixel space, the similar image detection device extracts feature elements (e.g., pitching, spectator infield) that constitute the axes of the vector space. Based on the feature elements (e.g., pitching, spectator infield) that constitute the axes of the vector space for the combined image, the similar image detection device generates and outputs a feature value vector.

[0091] The similar image detection device can check the similarity of each feature element (e.g., pitching, spectator infield) in pixel space when searching for similar data at the closest location in the vector space from the feature value vector extracted from the combined image and the vector space database (128), and extract similar data from the vector space database (128).

[0092] The similar image detection device can perform a similarity analysis in the embedding space when searching a vector space database (128) constructed with similar data at the closest location in the vector space to the feature value vector extracted from the combined image.

[0093] A similar image detection device combines input images (e.g., images of a baseball game) into a single combined image using an embedding space. Based on the combined image combined into the embedding space, the similar image detection device extracts feature elements (e.g., pitching, spectator infield) that constitute the axes of the vector space. Based on the feature elements (e.g., pitching, spectator infield) that constitute the axes of the vector space for the combined image, the similar image detection device generates and outputs a feature value vector.

[0094] The similar image detection device can check the similarity of each feature element (e.g., pitching, spectator infield) in the embedding space when searching for similar data at the closest location in the vector space from the feature value vector extracted from the combined image and the vector space database (128), and extract similar data from the vector space database (128).

[0095] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

Claims

1. A combined image generator that samples multiple input images on the time axis or the space axis and then creates a combined image by combining them into one image; An image retrieval engine that extracts a feature value vector constituting an axis of a vector space based on the combined image, searches for a similar image located closest to the feature value vector in the vector space, and outputs the similar image and metadata corresponding to the similar image; A similar image detection device characterized by including a .

2. In paragraph 1, A content property determination unit for determining properties of the input image based on at least one of the similar images and the metadata; A similar image detection device characterized by additionally including:

3. In paragraph 1, The above combined image generator A similar image detection device characterized in that it generates the combined image by combining a plurality of images extracted from the input image along a spatial axis, or generates the combined image by combining a plurality of input images extracted from a temporally continuous image series from a video in a manner of a predetermined time interval, a variable time interval, or a specific feature.

4. In paragraph 1, The above image retrieval engine, A feature value vector extraction unit for generating the feature value vector by extracting feature elements constituting the axes of the vector space from the above combined image; A vector space database that constitutes a vector space with the above feature elements as axes; A similar image extraction unit for extracting a similar image located closest to the feature value vector in the vector space from the vector space database; A similar image and meta information output unit that outputs the similar image and meta information corresponding to the similar image; A similar image detection device characterized by including a .

5. In paragraph 2, The above content property determination part A similar image detection device characterized in that it determines and outputs whether the input image is content having a specific attribute by using at least one or more of the meta information corresponding to the similar image based on the similar image and content attribute matching function.

6. In paragraph 2, The above content property determination part A similar image detection device characterized in that it determines and outputs whether the input image is content classified into a specific field by using at least one or more of the meta information corresponding to the similar image based on the similar image and content category classification function.

7. In paragraph 2, The above content property determination part A similar image detection device characterized in that it determines and outputs whether the input image is content classified as a specific event by using at least one or more of the meta information corresponding to the similar image based on the similar image and content attribute context detection function.

8. In paragraph 1, A learning data set configuring unit for generating a learning data set by specifying target feature information and a label for the above combined image; A database construction unit that learns to extract target feature elements constituting an axis of a vector space from the combined image using target feature information and labels input from the learning data set construction unit, constructs a dimension of the vector space, and constructs the vector space database by assigning the label to a specific area in the vector space; A similar image detection device characterized by additionally including:

9. In paragraph 8, The above database construction department A vector space axis feature extractor learning unit that learns an artificial intelligence network that extracts the target feature elements for the input image using the above learning data set; A data labeling unit for mapping the target feature elements onto the vector space and assigning the label to the mapped area; A vector space location determination and label-specific similarity generation unit that specifies a location in the vector space and the label using a vector composed of the above target feature elements; A vector space database construction unit that constructs a vector space database using location information, meta information, and label information on the above vector space; A similar image detection device characterized by including a .

Images