A scene graph-based image content retrieval method
Through a scene graph-based image content retrieval method, the association between images and text is extracted using graph convolutional neural networks and layout prediction models, which solves the problem of fine-grained association in cross-modal image and text retrieval and achieves high-precision image retrieval on long-tail datasets.
Patent Information
- Application Number
- CN202211550485.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-05
- Publication Date
- 2025-09-09
- Estimated Expiration
- 2042-12-05
AI Technical Summary
Traditional unimodal retrieval cannot meet the needs of cross-modal data. It is difficult to extract fine-grained correlations between images and texts in cross-modal image-text retrieval, and there are interference and annotation errors, resulting in insufficient retrieval accuracy.
A scene graph-based image content retrieval method is adopted. The scene graph embedding is generated through a graph convolutional neural network. The layout prediction model and triple embedding vector are used for target matching, and structured queries are combined for image retrieval.
It improves the accuracy of image content retrieval, can effectively retrieve the visual relationship between multiple targets in long-tail data sets, meets the requirements of fine-grained retrieval, and improves the accuracy of retrieval.
Smart Images
Figure CN115952306B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of target detection and natural language processing, and in particular relates to an image content retrieval method based on scene graph. Background Art
[0002] With the continuous growth of information, a vast amount of data in various modalities, such as text, images, audio, and video, has emerged on the internet. These data may describe the same topic, such as images and text on a website or subtitles, audio, and video in a movie. This is called multimodal data. Traditional single-modal search can no longer meet this growing demand, and cross-modal search has become a hot research topic.
[0003] The key to cross-modal image-text retrieval lies in bridging the "semantic gap" of heterogeneous data and measuring the similarity between different modalities. A basic solution is to first map the input data of different modalities into a common subspace in some way, then measure the similarity between them in this common space, rank them, and output the final search results. Early image-text retrieval mostly used traditional machine learning methods, including subspace learning and topic models. However, these methods suffer from shortcomings such as non-end-to-end processing and weak feature representation. Deep learning has powerful feature representation capabilities and can learn deep semantic features. Furthermore, its "end-to-end" learning approach allows machines to perform feature extraction and filtering independently. Consequently, a growing number of studies are integrating deep learning techniques into the field of image-text retrieval. However, these methods fail to consider the fine-grained relationships between images and text. This is particularly true in image-text matching tasks, which require retrieving unique, correct matching samples. However, the large number of similar samples in real-world retrieval databases makes accurately identifying the correct sample from among these similar samples a challenging task. Thanks to advances in object detection and natural language processing, images can be represented as a series of region features, and text as a series of word features. This allows for the association between images and text to be captured at the region-word level. The key to fine-grained image and text retrieval lies in extracting appropriate image regions and phrases or words, and measuring the correlation between them. However, in real-world datasets, issues such as image background interference, partial object occlusion, errors in text annotation, and incomplete annotation information all complicate fine-grained image and text retrieval. Summary of the Invention
[0004] The purpose of the present invention is to provide an image content retrieval method based on scene graph.
[0005] The purpose of the present invention is achieved through the following technical solutions:
[0006] A scene graph-based image content retrieval method, the specific steps are:
[0007] Step 1: Scene graph embedding;
[0008] The graph convolutional neural network processes the input scene graph and generates an embedded graph corresponding to the target node in the graph. The feature extraction network is used to extract the deep visual features of the image to obtain the overall feature map of the image. The triple region is selected according to the <subject, relation, object> required for detection and reasoning. The regional features of the target are processed to obtain the predicted target and relation.
[0009] Step 2: Layout prediction model;
[0010] The singleton object embedding is used as input to the next stage of the network model. The output of the second stage of the prediction model is used as a scene layout with object positioning. This object embedding is used to form a series of triplet (<subject, predicate, object>) embedding vectors. The triplet embedding vectors are passed through a ternary mask prediction network to mark the target object as the subject or object. The purpose is to calibrate the subject-predicate relationship between multiple objects. The triplet embedding vectors are passed through a triplet regression network. In this regression network, the network is trained to perform connection positioning on the bounding boxes of the subject and object; the box is defined as the bounding box of the subject and object.
[0011] Step 3: Target matching;
[0012] The query database is formed by forming structured queries from object embeddings of a learned scene graph embedding; through various forms of structured queries, including visual semantics; a database of 3100 visual relationships is extracted from the annotated test scene graph on the COCO-Stuff dataset; the retrieved images are sorted using a similarity metric S according to their respective embedding space representations; and the top N images that meet the search requirements are displayed for screening.
[0013] Furthermore, the input scene graph described in step 1 is specifically:
[0014] The input scene graph describes the multiple relationships between objects in the graph. Given a set of object categories C and a set of relationships R, a scene graph is a (O, E) tuple, O = {o1, ..., o n} is a set of categories, o i ∈C, It is a set of directed edges. The relationship between a scene graph can be expressed as a set of (o i ,r,o j ), o i ,o j ∈O, r∈R.
[0015] Furthermore, the scene graph embedding described in step 1 is specifically as follows:
[0016] For all objects o i ∈O, all edges (o i ,r,o j )∈E given input vector Using g s ,g p and g o Three graph convolution functions to generate output vectors for all computational nodes and edges The triplet vector (v i ,v r ,v j ) as input, and output groups of new vector subjects o respectively i , predicate r, object o j ;
[0017] Let v' r =g p (v i ,v r ,v j ), an object o i The output vector v' i Should rely on i All vectors v of objects connected to them by graph edges j , and the vectors v of these edges r ; For each from o i Starting edge, use g s To calculate a candidate vector, in the set Collect all such candidate vectors, also using g o To calculate the termination and o i A set of candidate vectors for all edges of
[0018]
[0019]
[0020] Object o i V' i The output vector is calculated as h is a symmetric function that pools an input collection of vectors into a single output vector.
[0021] Furthermore, the layout prediction model described in step 2 is specifically:
[0022] The object layout network accepts an embedding vector v i Shape D object o i , and pass it to a mask regression network to predict a soft binary classification mask of shape M×M A bounding box regression network to predict the location of the bounding box Embed the vector v i With mask Multiplying together, we get a mask embedding of shape D×M×M, which is then warped to the position of the bounding box using bilinear interpolation, giving the object layout, which is the sum of all the above object layouts.
[0023] Furthermore, the mask regression network is composed of several transposed convolutions, and the elements in the mask are made to be in the range of (0, 1) through the nonlinear sigmoid activation function. The boundary regression network is a multi-layer perceptron MLP.
[0024] Furthermore, the retrieved images are sorted using the similarity metric S according to their respective embedding space representations as described in step 3:
[0025]
[0026] Where d is the query q and the retrieval result r at position k k The L2 distance between them.
[0027] The beneficial effects of the present invention are:
[0028] To overcome the problems of fine-grained image and text retrieval, the present invention provides a content-based image retrieval method that retrieves a set of images that meet the description in a structured manner. The scene graph-based image retrieval method starts from the overall content of the image and focuses more on the visual relationship between objects. This visual relationship is composed in the form of a graph (e.g., <man, kicking, football>), presenting the image in the form of a structured description. This complex structured form is used to complete the retrieval task. This more detailed description improves the accuracy of image content-based retrieval and meets people's requirements for fine-grained image retrieval tasks.
[0029] The present invention uses a content-based image retrieval method to retrieve a set of images that match the description through structured questions. This method is not limited to retrieving the names and characteristics of a single or multiple objects, but rather searches for the interactions between multiple objects. Therefore, image retrieval based on this image description approach is more practical, but also more challenging. Scene graph-based image retrieval methods start with the overall content of the image, presenting the image in the form of a structured description, and using this complex structured form to complete the retrieval task. This more detailed description improves the accuracy of image content-based retrieval and meets the requirements for fine-grained image retrieval tasks. The present invention proposes a scene graph embedding-based method for image retrieval. Visual relationships are extracted from the scene graph to form a structured description. Visual relationships originate from the directed subgraph of the scene graph, with subjects and objects being nodes connected by predicate relationships. This retrieval method also achieves good performance in long-tail datasets. On the long-tail distributed COCO-Stuff dataset, the retrieval performance is good even for medium-frequency or low-frequency objects. Importantly, exact matches are still obtained despite omitting the predicate, and exact matches are obtained for all retrieved results despite omitting the predicate. Even incorrect results have the correct predicate and are often semantically similar. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is the sub-scene graph query graph of the present invention;
[0031] Figure 2 It is a network structure diagram of the present invention;
[0032] Figure 3 is the tuple mask of the present invention;
[0033] Figure 4 It is a single-graph convolution calculation graph of the present invention;
[0034] Figure 5 It is a scene layout flow chart of the present invention;
[0035] Figure 6 It is the image retrieval result diagram of the present invention. DETAILED DESCRIPTION
[0036] The present invention will be further described below with reference to the accompanying drawings.
[0037] The present invention uses scene graph embedding learned from a scene layout prediction model to solve the image retrieval problem. A scene graph is a structured data format used to encode semantic relationships between objects. In a scene graph, nodes in the graph are represented by targets in the image, and the edges between nodes in the graph are learned as a subgraph from the scene graph. The subgraph is used to represent the visual relationship in the image, thereby forming a structured query database for matching with the massive image scene graphs in the database, as shown in the attached figure. Figure 1 Each subgraph consists of a triple, including a subject, an object, and the visual relationship between the subject and the object.
[0038] The specific implementation process of the present invention is:
[0039] 1. Scene graph input
[0040] The model input is a scene graph, which describes the multiple relationships between objects in the graph. Given a set of object categories C and a set of relationships R, a scene graph is a (O, E) tuple, O = {o1, ..., o n} is a set of categories,
[0041] o i ∈C, It is a set of directed edges. The relationship between a scene graph can be expressed as a set of (o i ,r,o j ), o i ,o j ∈O, r∈R.
[0042] 2. Scene Graph Embedding
[0043] The scene graph can describe the image content concisely and structuredly. It can not only encode the semantic and spatial information of a single object in the scene, but also represent the relationship between each pair of objects. A scene graph, where nodes and edges are represented by D in Given a vector of D, this scene graph is used as the input of the graph convolutional neural network. The graph convolutional network uniformly associates and compresses the information in the scene graph to generate D out dimensional vectors. The output vectors are functions of their corresponding input neighborhoods, so each graph convolutional layer propagates information along the edges of the graph. A graph convolutional layer applies the same function to all edges of the graph, allowing a single layer to operate on graphs of arbitrary shapes. The graph convolutional neural network is a five-layer perceptron model with 128-dimensional input and output layers and 512-dimensional intermediate layers. The graph convolutional network uniformly correlates and compresses information in the scene graph, converting it into a vector, enabling correlation between multiple scene graphs at the same scale.
[0044] Specifically, for all objects oi ∈O, all edges (o i ,r,o j )∈E Given an input vector v i , Using g s ,g p and g o Three graph convolution functions are used to generate the output vector v' for all computing nodes and edges i , It converts the triple vector (v i ,v r ,v j ) as input, and output groups of new vector subjects o respectively i , predicate r, object o j .
[0045] To calculate the output vector v' of the edge r , assuming v' r =g p (v i ,v r ,v j ). Since an object may participate in multiple relationships, it is difficult to update the target vector. i The output vector v' i Should rely on i All vectors v of objects connected to them by graph edges j , and the vectors v of these edges r To this end, for each i Starting edge, use g s To calculate a candidate vector, in the set Collect all such candidate vectors, also using g o To calculate the termination and o i A set of candidate vectors for all edges of
[0046]
[0047]
[0048] Object o i V' i The output vector is calculated as h is a symmetric function that pools a set of vector inputs into a single output vector. The computational diagram of a single graph convolutional layer is shown in the following example. Figure 4 shown.
[0049] In the implementation, the function g s , g p and g oIt is implemented using a single network that concatenates its three input vectors, feeds them into a multilayer perceptron (MLP), and computes three output vectors using a fully connected output head. The pooling function h takes the average of its input vectors and feeds the result into the MLP.
[0050] 3. Layout prediction
[0051] The singleton target embedding is used as the input to the next stage of the network model. The output of the second stage of the prediction model is used as the scene layout with target positioning. This target embedding is then used to form a series of triplet (<subject, predicate, object>) embedding vectors. The triplet embedding vector is passed through a ternary mask prediction network to mark whether the target object belongs to the subject or the object. The purpose of this is to calibrate the subject-predicate relationship between multiple targets. The triplet embedding vector is passed through a triplet regression network, in which the network is trained to connect and locate the bounding box of the subject and the object. The box is defined as the bounding box of the subject and the object. As shown in the attached figure, the target object belongs to the subject or the object. Figure 2 shown.
[0052] The present invention processes the input scene graph using a series of graph convolutions, providing each object with an embedding vector that aggregates information about all objects and relationships in the graph.
[0053] In order to generate an image layout, we must move from the image domain to the image domain. To this end, we use the object embedding vector to calculate a scene layout, which gives the rough two-dimensional structure of the image to be generated; the scene layout is calculated by using the object layout network to predict the segmentation mask and bounding box of each object. Figure 5 shown.
[0054] The object layout network accepts an embedding vector v i Shape D object o i , and pass it to a mask regression network to predict a soft binary classification mask of shape M×M A bounding box regression network to predict the location of the bounding box The mask regression network consists of several transposed convolutions, which are activated by the nonlinear sigmoid function to make the elements in the mask in the range of (0,1). The boundary regression network is an MLP. Figure 3 shown.
[0055] Embed the vector v i With mask Multiplying them together, we get a mask embedding of shape D×M×M, which is then warped to the position of the bounding box using bilinear interpolation to give the object layout. Finally, the object layout is the sum of all the above object layouts.
[0056] 4. Result matching
[0057] Through various forms of structured queries, including visual semantics, the query database extracts 3100 visual relationships from the annotated test scene graphs on the COCO-Stuff dataset. The retrieved images are ranked according to their respective embedding space representations using a similarity metric S:
[0058]
[0059] Where d is the L2 distance between the query q and the retrieval result rk at position k. The results are shown in the following figure. Figure 6 shown.
[0060] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.
Claims
1. A scene graph-based image content retrieval method, characterized by: The specific steps are: Step 1: Scene graph embedding; The graph convolutional neural network processes the input scene graph and generates an embedded graph corresponding to the target node in the graph. The feature extraction network is used to extract the deep visual features of the image to obtain the overall feature map of the image. The triple region is selected according to the <subject, relation, object> required for detection and reasoning. The regional features of the target are processed to obtain the predicted target and relation. The scene graph embedding is specifically as follows: for all objects o i ∈O, all edges (o i ,r,o j )∈E given input vector Using g s ,g p and g o Three graph convolution functions to generate output vectors for all computational nodes and edges The triplet vector (v i ,v r ,v j ) as input, and output groups of new vector subjects o respectively i , predicate r, object o j ; Let v' r =g p (v i ,v r ,v j ), an object o i The output vector v' i Should rely on i All vectors v of objects connected to them by graph edges j , and the vectors v of these edges r ; For each from o i Starting edge, use g s To calculate a candidate vector, in the set Collect all such candidate vectors, also using g o To calculate the termination and o i A set of candidate vectors for all edges of Object o i V' i The output vector is calculated as h is a symmetric function that pools an input collection of vectors into a single output vector; Step 2: Layout prediction model; The singleton object embedding is used as input to the next stage of the network model. The output of the second stage of the prediction model is used as a scene layout with object positioning. This object embedding is used to form a series of triplet (<subject, predicate, object>) embedding vectors. The triplet embedding vectors are passed through a ternary mask prediction network to mark the target object as the subject or object. The purpose is to calibrate the subject-predicate relationship between multiple objects. The triplet embedding vectors are passed through a triplet regression network. In this regression network, the network is trained to perform connection positioning on the bounding boxes of the subject and object; the box is defined as the bounding box of the subject and object. The layout prediction model is specifically as follows: the object layout network accepts an embedding vector v i Shape D object o i , and pass it to a mask regression network to predict a soft binary classification mask of shape M×M A bounding box regression network to predict the location of the bounding box Embed the vector v i With mask Multiply them together to get a mask embedding of shape D×M×M, and then use bilinear interpolation to warp it to the position of the bounding box to give the object layout, which is the sum of all the above object layouts; Step 3: Target matching; We query the database by forming structured queries from object embeddings learned from a scene graph. We use various forms of structured queries, including visual semantics. We extract a database of 3,100 visual relations from annotated test scene graphs on the COCO-Stuff dataset. Sort the retrieved images using a similarity metric S according to their respective embedding space representations; display the top N images that meet the retrieval requirements for screening.
2. The scene graph-based image content retrieval method according to claim 1, characterized in that: The input scene graph described in step 1 is specifically: The input scene graph describes the multiple relationships between objects in the graph. Given a set of object categories C and a set of relationships R, a scene graph is a (O, E) tuple, O = {o1, ..., o n } is a set of categories, o i ∈C, It is a set of directed edges. The relationship between a scene graph can be expressed as a set of (o i ,r,o j ), o i ,o j ∈O, r∈R.
3. The scene graph-based image content retrieval method according to claim 1, characterized in that: The mask regression network is composed of several transposed convolutions. The nonlinear sigmoid activation function is used to make the elements in the mask within the range of (0, 1). The boundary regression network is a multi-layer perceptron MLP.
4. The scene graph-based image content retrieval method according to claim 1, characterized in that: Sort the retrieved images using the similarity metric S according to their respective embedding space representations as described in step 3: Where d is the query q and the retrieval result r at position k k The L2 distance between them.