Content-based searching apparatus and method for image using scene graph and knowledge graph

Abstract

The present invention relates to a content-based image search device using scene graph and knowledge graph matching, and more specifically, to an image search device comprising: an SGG DB in which images are converted into scene graphs and stored using a scene graph generation (SGG) algorithm; a knowledge graph generation unit that divides a query received for image search into morphemes and generates a knowledge graph using the divided morphemes; a graph matching unit that inputs the knowledge graph generated by the knowledge graph generation unit and the scene graph stored in the SGG DB into a graph neural network (GNN), respectively, converts the knowledge graph and the scene graph into feature vectors, and measures the similarity between each feature vector converted by the GNN; and a result presentation unit that presents an image corresponding to the scene graph with the highest similarity as a search result image based on the similarity measurement result of the graph matching unit. The present invention relates to a content-based image search method using scene graph and knowledge graph matching, and more specifically, to an image search method in which each step is performed on a computer, comprising: (1) a step of converting images into scene graphs using a scene graph generation (SGG) algorithm and storing them in an SGG DB; (2) a step of dividing a query received for image search into morphemes and generating a knowledge graph using the divided morphemes; (3) a step of inputting the knowledge graph generated in step (2) and the scene graph stored in the SGG DB into a graph neural network (GNN), respectively, converting the knowledge graph and the scene graph into feature vectors, and measuring the similarity between each feature vector converted in the GNN; and (4) a step of presenting an image corresponding to the scene graph with the highest similarity as a search result image based on the similarity measurement result of step (3). According to the content-based image search device and method using scene graph and knowledge graph matching proposed in the present invention, images are converted into scene graphs using a scene graph generation (SGG) algorithm and stored in an SGG DB, and a knowledge graph generated by analyzing a query received for image search and a scene graph stored in the SGG DB are each input into a GNN, and the image corresponding to the scene graph with the highest similarity between the converted feature vectors is presented as the search result image. Since the scene graph generated from the image and the knowledge graph generated from the user's query are compared with each other rather than using the image as is, detailed objects and relationships of the image are understood, and search results that match the intent of the query can be provided.

Description

Technology Field

[0001] The present invention relates to a content-based image search device and method, and more specifically, to a content-based image search device and method using scene graph and knowledge graph matching. Background Technology

[0002] The content described in this section merely provides background information regarding an embodiment of the present invention and does not constitute prior art.

[0004] While there are various existing video search methods, each has its own advantages and disadvantages. ① Keyword-based search (query-by-text) is the simplest and most intuitive method; however, it may lack accuracy as it does not reflect the content of the video itself, and requires a human to manually assign keywords. ② Image-based search (query-by-image) searches by extracting visual features such as color, texture, shape, and pattern; while it offers higher accuracy than keyword-based search, it can be affected by the video recording environment, such as lighting and viewpoint. ③ Object-based search (query-by-object) identifies objects within a video and searches for videos in which those objects appear; while effective for finding specific objects, the accuracy of object recognition may be lower. ④ Deep learning-based search utilizes deep learning models to understand video content more accurately and improve the accuracy of search results; however, it has limitations as it measures the overall similarity of the video rather than understanding the relationships between detailed objects.

[0006] Meanwhile, recent advancements in Deep Neural Networks (DNNs) have achieved unprecedented results in visual recognition tasks. Consequently, research has continuously flowed to explain how DNNs aggregate and utilize visual information. One active research area is Scene Graph Generation (SGG), which uses DNNs to automatically map images to a semantically structured scene graph. To comprehensively capture the relationships and interactions between objects within an image, SGG focuses primarily on generating structured representations in the form of graphs. Therefore, SGG requires proper labeling of detected objects and their relationships.

[0008] The SGG approach generally involves two main steps: object detection and relationship modeling. Object detection serves to identify and localize individual objects within an image to generate a set of bounding boxes. The next step, relationship modeling, aims to understand the interactions between these objects and capture contextual dependencies. Several methods have been proposed to address the challenges of SGG, including both rule-based and learning-based approaches (Tang et al. 2019; Zellers et al. 2018). Rule-based methods typically use predefined templates or heuristics to establish relationships between objects. Conversely, learning-based methods utilize Deep Neural Networks (DNNs) to learn relationships from extensively annotated datasets. Graph neural networks (GNNs) often model structured representations and leverage graph-based operations for inference and prediction within the SGG framework.

[0010] Meanwhile, prior art related to the present invention has been disclosed, including Published Patent No. 10-2022-0103407 (Title of invention: Method for generating a scene graph for an object in an image and an electronic device using the same, Date of publication: July 22, 2022).

[0012] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as publicly known technology disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0013] The present invention is proposed to solve the aforementioned problems of previously proposed methods, and aims to provide a content-based image search device and method using scene graph and knowledge graph matching, wherein images are converted into scene graphs using a scene graph generation (SGG) algorithm and stored in an SGG DB, and a knowledge graph generated by analyzing a query received for image search and the scene graph stored in the SGG DB are each input into a GNN, and the image corresponding to the scene graph with the highest similarity between the transformed feature vectors is presented as the search result image, thereby allowing the image to be used as is, but rather the scene graph generated from the image and the knowledge graph generated from the user's query are compared with each other, so that detailed objects and relationships of the image are understood and search results that match the intent of the query are provided.

[0015] However, the technical problem that the present invention aims to solve is not limited to the technical problem described above, and other technical problems may exist. It goes without saying that objectives or effects that can be understood from the means of solving the problem or the embodiments, even if not explicitly mentioned, are also included. means of solving the problem

[0016] A content-based image search device using scene graph and knowledge graph matching according to the features of the present invention for achieving the above-mentioned purpose is,

[0017] As an image search device,

[0018] An SGG DB that stores images converted into scene graphs using the scene graph generation (SGG) algorithm;

[0019] A knowledge graph generation unit that divides an input query for image search into morphemes and generates a knowledge graph using the divided morphemes;

[0020] A graph matching unit that inputs the knowledge graph generated by the knowledge graph generation unit and the scene graph stored in the SGG DB into a graph neural network (GNN), respectively, converts the knowledge graph and the scene graph into feature vectors, respectively, and measures the similarity between each feature vector converted by the GNN; and

[0021] The composition is characterized by including a result presentation unit that presents an image corresponding to the scene graph with the highest similarity as a search result image, based on the similarity measurement result of the graph matching unit.

[0023] Preferably, the knowledge graph generating unit is,

[0024] You can use natural language input queries for the video to be searched.

[0026] More preferably,

[0027] It may further include a voice processing unit that receives a detailed description of a target video in natural language voice, converts the received voice into text using a speech-to-text (STT) analyzer, and transmits it to the knowledge graph generation unit.

[0029] Preferably, the knowledge graph generating unit is,

[0030] In the above-described morphemes, objects can be represented as nodes and relationships between objects as edges to generate the knowledge graph.

[0032] Preferably, in the graph matching unit,

[0033] The GNN to which the above knowledge graph is input and the GNN to which the above scene graph is input can share the same weights.

[0035] Preferably, the graph matching unit is,

[0036] The similarity between the first feature vector transformed from the above knowledge graph and the second feature vector transformed from the above scene graph can be measured as a similarity metric.

[0038] More preferably, the graph matching unit is,

[0039] The Euclidean distance between the first feature vector and the second feature vector transformed from all scene graphs stored in the SGG DB can be measured.

[0041] Preferably, the above result presenting unit is,

[0042] Images corresponding to a predetermined number of scene graphs in order of high similarity can be presented as search result images.

[0044] A content-based image search method using scene graph and knowledge graph matching according to the features of the present invention for achieving the above-mentioned purpose is,

[0045] As an image search method in which each step is performed on a computer,

[0046] (1) A step of converting images into a scene graph using a scene graph generation (SGG) algorithm and storing them in an SGG DB;

[0047] (2) A step of dividing a query received for image search into morphemes and generating a knowledge graph using the divided morphemes;

[0048] (3) inputting the knowledge graph generated in step (2) and the scene graph stored in the SGG DB into a graph neural network (GNN), respectively, converting the knowledge graph and the scene graph into feature vectors, and measuring the similarity between each converted feature vector in the GNN; and

[0049] (4) The compositional feature includes a step of presenting the image corresponding to the scene graph with the highest similarity as the search result image based on the similarity measurement result of the above step (3).

[0051] Preferably, in step (2),

[0052] You can use natural language input queries for the video to be searched.

[0054] More preferably, between step (1) and step (2),

[0055] (1-5) A step of receiving a detailed description of the target image in natural language speech; and

[0056] (1-6) The step of converting the input speech into text using an STT (speech-to-text) analyzer may be further included.

[0058] Preferably, in step (2),

[0059] In the above-described morphemes, objects can be represented as nodes and relationships between objects as edges to generate the knowledge graph.

[0061] Preferably, in step (3),

[0062] The GNN to which the above knowledge graph is input and the GNN to which the above scene graph is input can share the same weights.

[0064] Preferably, in step (3),

[0065] The similarity between the first feature vector transformed from the above knowledge graph and the second feature vector transformed from the above scene graph can be measured as a similarity metric.

[0067] More preferably, in step (3),

[0068] The Euclidean distance between the first feature vector and the second feature vector transformed from all scene graphs stored in the SGG DB can be measured.

[0070] Preferably, in step (4),

[0071] Images corresponding to a predetermined number of scene graphs in order of high similarity can be presented as search result images. Effects of the invention

[0072] According to the content-based image search device and method using scene graph and knowledge graph matching proposed in the present invention, images are converted into scene graphs using a scene graph generation (SGG) algorithm and stored in an SGG DB, and a knowledge graph generated by analyzing a query received for image search and a scene graph stored in the SGG DB are each input into a GNN, and the image corresponding to the scene graph with the highest similarity between the converted feature vectors is presented as the search result image. Since the scene graph generated from the image and the knowledge graph generated from the user's query are compared with each other rather than using the image as is, detailed objects and relationships of the image are understood, and search results that match the intent of the query can be provided.

[0074] Furthermore, the various and beneficial advantages and effects of the present invention are not limited to those described above and may be more easily understood in the process of explaining specific embodiments of the present invention. Brief explanation of the drawing

[0075] FIG. 1 is a diagram illustrating the configuration of a content-based image search device using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a scene graph generation device and method using co-occurring knowledge and learnable term frequency. FIG. 3 is a diagram illustrating a scene graph generation device and method using an edge dual scene graph and a message passing neural network. FIG. 4 is a diagram showing the overall operation process of a content-based image search device using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 5 is a diagram showing the overall structure of a content-based image search device using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 6 is a diagram illustrating the detailed configuration of a graph matching unit in a content-based image search device using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 7 is a diagram illustrating graph matching processed in a graph matching unit in a content-based image search device using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 8 is a diagram illustrating the flow of a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 9 is a diagram illustrating the flow of a method for processing a query when a query is input as natural language speech in a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. FIG. 10 is a diagram illustrating the detailed flow of step S130 in a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. Specific details for implementing the invention

[0076] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0078] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "indirectly connected" with other elements interposed between them. Furthermore, terms such as "include," "have," or "have" described below should be interpreted as indicating the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Additionally, singular expressions used in the present invention include plural expressions unless the context clearly indicates otherwise.

[0080] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0082] Additionally, terms such as “…part,” “…unit,” and “module” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0084] In addition, some of the operations or functions described as being performed by a terminal, device, or device in the present invention may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0086] In particular, the means for executing the system according to each embodiment of the present invention may be an application or a web server, and the terminal, which is the means for reading the recording medium on which the application or web server is recorded, may include not only general PCs such as general desktops or laptops, but also mobile terminals such as smartphones and tablet PCs.

[0088] The following examples are detailed descriptions to aid in understanding the present invention and are not intended to limit the scope of the present invention. Accordingly, inventions within the same scope that perform the same function as the present invention will also fall within the scope of the present invention.

[0090] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0092] FIG. 1 is a diagram illustrating the configuration of a content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention. As shown in FIG. 1, the content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention may be configured to include an SGG DB (110), a knowledge graph generation unit (130), a graph matching unit (140), and a result presentation unit (150), and may further include a voice processing unit (120).

[0094] The SGG DB (110) can convert images into scene graphs and store them using a scene graph generation (SGG) algorithm. A scene graph is generated by understanding the scenes of the images. A content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention can construct the SGG DB (110) by converting all images into scene graphs in advance, and can use only scene graphs for image search without using actual images. Below, a scene graph generation algorithm is introduced with reference to FIGS. 2 and FIGS. 3.

[0096] FIG. 2 illustrates an apparatus and method for generating a scene graph using co-occurrence knowledge and learnable term frequencies. As illustrated in FIG. 2, the apparatus and method for generating a scene graph using co-occurrence knowledge and learnable term frequencies relates to an SGG apparatus and method that calculates the co-occurrence between objects in training data and applies it to a Message Passing Neural Network (MPNN) to learn object relationships based on Co-occurrence Knowledge (CooK). This method is a technique that provides the model with knowledge regarding the co-occurrence between objects, which has not been carefully considered previously, thereby enabling more accurate inference regarding object relationships and mitigating the long-tail problem present in scene graph datasets. Detailed information regarding this is as described in Application No. 10-2024-0060804 (Title of Invention: Apparatus and Method for Generating a Scene Graph Using Co-occurrence Knowledge and Learnable Term Frequency).

[0098] FIG. 3 illustrates an apparatus and method for generating a scene graph using an edge dual scene graph and a message passing neural network. As shown in FIG. 3, the apparatus and method for generating a scene graph using an edge dual scene graph and a message passing neural network generate a raw graph and an edge dual scene graph from the input, and predict a scene graph by learning a DualMPNN module comprising two message passing neural networks (MPNNs) that update object-centric features and relationship-centric features using the two graphs. By learning both object-centric and relationship-centric features, the technique can accurately predict the relationship recognition context and predict a more detailed scene graph by allowing for the updating of finer relationships between objects. Detailed information regarding this is as described in Application No. 10-2023-0150380 (Title of Invention: Apparatus and Method for Generating a Scene Graph Using an Edge Dual Scene Graph and a Message Passing Neural Network).

[0100] FIG. 4 is a diagram showing the overall operation process of a content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention, and FIG. 5 is a diagram showing the overall structure of a content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention. Hereinafter, with reference to FIG. 4 and FIG. 5, a specific configuration for performing image search using an SGG DB (110) that stores a scene graph generated by the SGG algorithm as shown in FIG. 2 and FIG. 3 will be described in detail.

[0102] The voice processing unit (120) can receive a detailed description of the target image in natural language voice, convert the received voice into text using a speech-to-text (STT) analyzer, and transmit it to the knowledge graph generation unit (130), which will be explained in detail below. First, a user performing a search can input a detailed description of the target image in natural language voice. As illustrated in the upper left of FIG. 4, if the user describes the description of the image they intend to search for in voice, such as “A boy wearing a hat is looing at a mountain, and there are clouds on the mountain and behind the boy,” the image search device (100) can receive this voice through a microphone. Next, the image search device (100) can convert the received voice into text using a speech-to-text (STT) analyzer.

[0104] The knowledge graph generation unit (130) can divide the input query for image search into morphemes and generate a knowledge graph using the divided morphemes. More specifically, the knowledge graph generation unit (130) can use a query received in natural language regarding the image to be searched. At this time, the knowledge graph generation unit (130) can use a query in which the user's voice description is converted into text by the voice processing unit (120), but it can also receive a description of the image to be searched from the user as text and use it as a query.

[0106] Additionally, the knowledge graph generation unit (130) generates a knowledge graph by dividing a text-based query into morphemes and tokenizing them. In the divided morphemes, objects (concepts) are nodes and relationships (relations) between objects are edges to generate the knowledge graph. In the middle of FIG. 4, the objects (concepts) are boy, hat, mountain, and cloud, and the relationships are wearing, looking at, above, and behind. The knowledge graph generated using these objects and relationships can be verified. This process can be implemented using the Python library NetworkX.

[0108] The graph matching unit (140) inputs the knowledge graph generated by the knowledge graph generation unit (130) and the scene graph stored in the SGG DB (110) into the GNN (graph neural network), respectively, converts the knowledge graph and the scene graph into feature vectors, and measures the similarity between each converted feature vector in the GNN.

[0110] FIG. 6 is a diagram illustrating the detailed configuration of a graph matching unit (140) in a content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention. As shown in FIG. 6, the graph matching unit (140) of the content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention may be configured to include a vector conversion unit (141) and a similarity measurement unit (142).

[0112] The vector transformation unit (141) of the graph matching unit (140) can generate two feature vectors (a first feature vector and a second feature vector) by inputting the knowledge graph generated by the knowledge graph generation unit (130) and the scene graph, which is previously created from an image and stored in the SGG DB (110), into a GNN that shares the same weight. FIG. 7 is a diagram illustrating graph matching processed in the graph matching unit (140) in a content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention. As shown in FIG. 7, in the graph matching unit (140) of the content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention, the GNN into which the knowledge graph is input and the GNN into which the scene graph is input can share the same weight.

[0114] The similarity measuring unit (142) of the graph matching unit (140) can measure the similarity between a first feature vector converted from a knowledge graph and a second feature vector converted from a scene graph using a similarity metric. According to an embodiment, the similarity between the first feature vector and the second feature vector can be measured using Euclidean distance. Here, the similarity measuring unit (142) of the graph matching unit (140) can measure the distance between the first feature vector and the second feature vector converted from all scene graphs stored in the SGG DB (110).

[0116] The result presentation unit (150) may present an image corresponding to the scene graph with the highest similarity as a search result image based on the similarity measurement result of the graph matching unit (140). According to an embodiment, the result presentation unit (150) may present a predetermined number of images corresponding to scene graphs in order of highest similarity as search result images. For example, 10 images may be presented as search result images in order of highest similarity, and the user may be allowed to select them.

[0118] FIG. 8 is a diagram illustrating the flow of a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. As illustrated in FIG. 8, the content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention is an image search method in which each step is performed on a computer, and can be implemented by including the step of converting images into scene graphs using an SGG algorithm and storing them in an SGG DB (110) (S110); the step of dividing a query received for image search into morphemes and generating a knowledge graph using the divided morphemes (S120); the step of inputting the knowledge graph and the scene graph stored in the SGG DB (110) into a GNN, converting the knowledge graph and the scene graph into feature vectors, and measuring the similarity between each feature vector converted in the GNN (S130); and the step of presenting an image corresponding to the scene graph with the highest similarity as a search result image (S140).

[0120] FIG. 9 is a diagram illustrating the flow of a method for processing a query when a query is input as natural language voice in a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. As shown in FIG. 8, the content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention may be implemented by including, between step S110 and step S120, a step (S115) of receiving a detailed description of the image to be searched as natural language voice and a step (S116) of converting the received voice into text using an STT analyzer.

[0122] FIG. 10 is a diagram illustrating the detailed flow of step S130 in a content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention. As shown in FIG. 8, step S130 of the content-based image search method using scene graph and knowledge graph matching according to an embodiment of the present invention may be implemented by including a step (S131) ​​of converting the knowledge graph and the scene graph into feature vectors, respectively, through a GNN that shares the same weights, and a step (S132) of measuring the similarity between each feature vector converted in the GNN.

[0124] Since the details regarding each step have been sufficiently explained above in relation to the content-based image search device (100) using scene graph and knowledge graph matching according to an embodiment of the present invention, a detailed explanation will be omitted.

[0126] As described above, according to the content-based image search device (100) and method using scene graph and knowledge graph matching proposed in the present invention, images are converted into scene graphs using a scene graph generation (SGG) algorithm and stored in an SGG DB (110), and a knowledge graph generated by analyzing a query received for image search and a scene graph stored in the SGG DB (110) are each input into a GNN, and the image corresponding to the scene graph with the highest similarity between the converted feature vectors is presented as the search result image. Since the image is not used as is but the scene graph generated from the image and the knowledge graph generated from the user's query are compared with each other, the detailed objects and relationships of the image are understood, and a search result that matches the intent of the query can be provided.

[0128] Meanwhile, the present invention may include a computer-readable medium comprising program instructions for performing operations implemented by various communication terminals. For example, the computer-readable medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0130] Such a computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. In this case, the program instructions recorded on the computer-readable medium may be those specifically designed and configured to implement the present invention, or they may be those known and available to those skilled in the art of computer software. For example, they may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0132] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0134] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0135] 100: Video search device 110: SGG DB 120: Voice processing unit 130: Knowledge Graph Generation Unit 140: Graph Matching Section 141: Vector conversion section 142: Similarity Measurement Unit 150: Result presentation section S110: A step of converting images into a scene graph using the SGG algorithm and storing it in the SGG DB. S115: Step of receiving a detailed description of the target video via natural language voice. S116: Step of converting input speech into text using an STT analyzer S120: A step of dividing an input query for image search into morphemes and generating a knowledge graph using the divided morphemes. S130: A step of inputting the knowledge graph and the scene graph stored in the SGG DB into the GNN, respectively, converting the knowledge graph and the scene graph into feature vectors, and measuring the similarity between each converted feature vector in the GNN. S131: A step of converting the knowledge graph and scene graph into feature vectors, respectively, through a GNN sharing equal weights. S132: Step to measure similarity between each feature vector transformed in the GNN S140: A step of presenting the image corresponding to the scene graph with the highest similarity as the search result image.

Claims

Claim 1 As an image search device (100), an SGG DB (110) in which images are converted into scene graphs and stored using a scene graph generation (SGG) algorithm; a knowledge graph generation unit (130) that divides a query received in natural language for image search target images into morphemes and generates a knowledge graph using the divided morphemes; and a graph matching unit (140) that inputs the knowledge graph generated by the knowledge graph generation unit (130) and the scene graph stored in the SGG DB (110) into a GNN (graph neural network), converts the knowledge graph and the scene graph into feature vectors, and measures the similarity between each feature vector converted by the GNN. The content-based image search device (100) using scene graph and knowledge graph matching includes a result presentation unit (150) that presents an image corresponding to the scene graph with the highest similarity as a search result image based on the similarity measurement result of the graph matching unit (140); the graph matching unit (140) includes a vector transformation unit (141) that generates two feature vectors by inputting the knowledge graph generated by the knowledge graph generation unit (130) and the scene graph previously created from the image and stored in the SGG DB (110) into the GNN that shares the same weight; and a similarity measurement unit (142) that measures the similarity between the first feature vector transformed from the knowledge graph and the second feature vector transformed from the scene graph using a similarity metric; and is characterized by converting all images into scene graphs and using the pre-configured SGG DB (110) to use only scene graphs for image search without using actual images. Claim 2 delete Claim 3 A content-based video search device (100) using scene graph and knowledge graph matching, characterized in that, in claim 1, it further includes a voice processing unit (120) that receives a detailed description of a video to be searched in natural language voice, converts the received voice into text using a speech-to-text (STT) analyzer, and transmits it to the knowledge graph generation unit (130). Claim 4 A content-based image search device (100) using scene graph and knowledge graph matching, wherein, in claim 1, the knowledge graph generation unit (130) generates the knowledge graph by making objects in the divided morphemes nodes and relations between objects edges. Claim 5 delete Claim 6 delete Claim 7 In claim 1, the graph matching unit (140) is characterized by measuring the Euclidean distance between the first feature vector and the second feature vector transformed from all scene graphs stored in the SGG DB (110), in a content-based image search device (100) using scene graph and knowledge graph matching. Claim 8 In claim 1, the result presentation unit (150) is characterized by presenting images corresponding to a predetermined number of scene graphs in order of high similarity as search result images, in a content-based image search device (100) using scene graph and knowledge graph matching. Claim 9 A method for image search in which each step is performed on a computer, comprising: (1) a step of converting images into scene graphs using a scene graph generation (SGG) algorithm and storing them in an SGG DB (110); (2) a step of dividing a query received in natural language for image search into morphemes for image search and generating a knowledge graph using the divided morphemes; (3) a step of inputting the knowledge graph generated in step (2) and the scene graph stored in the SGG DB (110) into a graph neural network (GNN), converting the knowledge graph and the scene graph into feature vectors, and measuring the similarity between each feature vector converted in the GNN. and (4) a step of presenting an image corresponding to the scene graph with the highest similarity as a search result image based on the similarity measurement result of the above step (3), wherein the above step (3) comprises: a step of generating two feature vectors by inputting the knowledge graph generated in the above step (2) and the scene graph previously created from the image and stored in the SGG DB (110) into the GNN that shares the same weight; and a step of measuring the similarity between the first feature vector converted from the knowledge graph and the second feature vector converted from the scene graph using a similarity metric, wherein all images are converted into scene graphs and the SGG DB (110) configured in advance is used, and only scene graphs are used for image search without using actual images. Claim 10 delete Claim 11 A content-based image search method using scene graph and knowledge graph matching, characterized in that, between step (1) and step (2), between the above steps, (1-5) a step of receiving a detailed description of the image to be searched in natural language speech; and (1-6) a step of converting the received speech into text using a speech-to-text (STT) analyzer. Claim 12 A content-based image search method using scene graph and knowledge graph matching, characterized in that, in step (2) above, objects in the divided morphemes are nodes and relationships between objects are edges to generate the knowledge graph. Claim 13 delete Claim 14 delete Claim 15 A content-based image search method using scene graph and knowledge graph matching, wherein in step (3) above, the Euclidean distance between the first feature vector and the second feature vector transformed from all scene graphs stored in the SGG DB (110) is measured. Claim 16 A content-based image search method using scene graph and knowledge graph matching, characterized in that, in step (4) of claim 9, images corresponding to a predetermined number of scene graphs in order of high similarity are presented as search result images.

Images