An image retrieval method based on graph structure matching

Abstract

The application discloses an image retrieval method based on graph structure matching, and inputs the graph structure of a query image into a trained graph neural network to obtain the node features of the query image and the relative and absolute positions between the nodes, wherein the graph neural network comprises a feature extraction module, a relative position prediction head and an absolute position prediction head; the feature extraction module is used for extracting the node features of the graph structure; the relative position prediction head and the absolute position prediction head are respectively used for predicting the relative and absolute positions between the nodes; and the training of the network is completed by calculating a classification loss and an auxiliary loss. The application improves the training effect of the network and greatly improves the performance of image retrieval.

Description

Technical Field

[0001] This application belongs to the field of image retrieval technology, and in particular relates to an image retrieval method based on graph structure matching. Background Technology

[0002] Image retrieval refers to the process where a user provides query information (images, text, graph structures, etc.), calculates the similarity of images in an image database based on the query information, and returns the top n images sorted by similarity to the user.

[0003] Image retrieval can be categorized into several types based on the type of query information provided by the user during the image retrieval process:

[0004] Based on text, users provide query information in text form, such as "There is a cat on the table" or "There is a dog on the left side of the car and a pedestrian behind it," which is usually unstructured.

[0005] Based on images, the query information provided by the user is also an image.

[0006] Based on the graph structure, users provide structured query information, including the categories of objects that may appear in the image and the relative positions between objects.

[0007] Early graph-based image retrieval methods were limited by the development of image segmentation algorithms and the constraints of discrete spatial relationships, resulting in significant room for improvement in retrieval performance and usability. Current technical solutions focus on extracting semantic information from user-provided text or images while neglecting how to assist users in directly providing semantic information to the retrieval system. Summary of the Invention

[0008] The purpose of this application is to provide an image retrieval method based on graph structure matching, which directly uses structured information provided by the user to more accurately describe the user's query intent and avoids the semantic extraction process that may result in information loss, thus achieving satisfactory results.

[0009] To achieve the above objectives, the technical solution of this application is as follows:

[0010] An image retrieval method based on graph structure matching includes:

[0011] Obtain the query image, take each object in the query image as a node, the connection between the nodes as an edge, take the object's attributes as the initial node features, and take the difference between the center coordinates of the starting node and the ending node of the edge as the edge features to construct a graph structure.

[0012] The graph structure of the query image is input into the trained graph neural network to obtain the node features of the query image and the relative and absolute positions between nodes. The graph neural network includes a feature extraction module, a relative position prediction head and an absolute position prediction head. The feature extraction module is a three-layer Transformer graph convolution module used to extract the node features of the graph structure. The relative position prediction head and the absolute position prediction head are multilayer perceptrons used to predict the relative and absolute positions between nodes, respectively.

[0013] Based on the node features output by the graph neural network, the graph structure similarity between the query image and the images in the image database is calculated, and the retrieved image is fed back to the user.

[0014] Furthermore, the initial node feature is a one-hot encoding of object attributes.

[0015] Furthermore, the edge is characterized by the difference between the center coordinates of the starting node and the ending node of the edge after normalization.

[0016] Furthermore, calculating the similarity between the query image and images in the image database includes:

[0017] For any two images, calculate the node similarity between each node feature of one image and each node feature of the other image, where the node similarity is the dot product of the two node features;

[0018] The maximum value among the node similarities between each node feature of one image and each node feature of another image is selected as the graph structure similarity between the two images.

[0019] Furthermore, the loss function of the graph neural network includes classification loss and auxiliary loss, wherein the classification loss is used to calculate the loss of node similarity, and the auxiliary loss includes regression loss of relative position between nodes and regression loss of absolute position.

[0020] This application proposes an image retrieval method based on graph structure matching. It uses a relative position prediction head and an absolute position prediction head to predict the relative and absolute positions between nodes, respectively. By calculating classification loss and auxiliary loss, the network is trained, which improves the training effect of the network and significantly enhances the performance of image retrieval. Attached Figure Description

[0021] Figure 1 This is a flowchart of the image retrieval method based on graph structure matching in this application;

[0022] Figure 2 This is a schematic diagram of the neural network structure of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In one embodiment, such as Figure 1 As shown, this application proposes an image retrieval method based on graph structure matching, including:

[0025] Step S1: Obtain the query image, take each object in the query image as a node, the connection between the nodes as an edge, take the object's attributes as the initial node features, and take the difference between the center coordinates of the starting node and the ending node of the edge as the edge features to construct a graph structure.

[0026] This application constructs a graph neural network. For the image queried by the user, a corresponding graph structure is first built. Then, the graph structure is processed by the graph neural network to obtain the node features corresponding to the queried graph structure. Then, the similarity with all graph structures in the image database is calculated, and the top k images based on similarity are returned to the user as the query results.

[0027] For any query image, i.e., the image input by the user, segmentation information can be obtained using existing segmentation models to construct a graph structure. How to segment and detect objects within the query image is a well-established technique in this field and will not be elaborated upon here.

[0028] In the graph structure, each node represents a specific entity object in the image. The initial node features are the one-hot encoding of the object's category, color, and shape. The edge features are the difference between the center coordinates of the starting and ending nodes of the normalized edge (any two entity objects have two mutually pointing directed edges). The constructed graph structure includes two sets: the initial node feature set H = {h1, h2, ..., h...} i ,…h n}, where n is the number of nodes, and the edge feature set E = {e 12 ,e 13 ,…,e 1n ,…e nn-1}, e ij The characteristics of the directed edge from node i to node j.

[0029] Step S2: Input the graph structure of the query image into the trained graph neural network to obtain the node features of the query image and the relative and absolute positions between nodes. The graph neural network includes a feature extraction module, a relative position prediction head and an absolute position prediction head. The feature extraction module is a three-layer Transformer graph convolution module used to extract the node features of the graph structure. The relative position prediction head and the absolute position prediction head are multilayer perceptrons used to predict the relative and absolute positions between nodes, respectively.

[0030] The graph neural network constructed in this application is as follows: Figure 2 As shown, it includes a feature extraction module, a relative position prediction head, and an absolute position prediction head.

[0031] The feature extraction module is a 3-layer Transformer graph convolution module. The input is a graph structure, namely the initial node features and edge features, and the output is the node features of the graph structure. The processing procedure is as follows:

[0032] H l This represents the set of input node features for the l-th layer Transformer graph convolutional module. Let E represent the node features of the i-th node. E represents the combination of edge features. Since edge node features are not updated, a superscript is not used to distinguish E. ij H represents the edge characteristics from node i to node j. l E is used as the input to the Transformer graph convolution module. When l equals 1, H 1 middle It refers to the initial node characteristics.

[0033] For the c-th head attention of the Transformer graph convolutional module, the multi-head attention from the start node i to the end node j is calculated as follows:

[0034]

[0035]

[0036]

[0037]

[0038] in d represents the feature length of each head. q, k, and e are the intermediate vectors required to calculate attention, W and b are the learnable matrix and vector, respectively, and α is the calculated attention, which will be used to calculate node features later.

[0039] in, e c,ij This represents the intermediate result of the c-th head in the l-th layer, used only for calculation. This represents the learnable query matrix of the c-th head in the l-th layer. This represents the learnable query vector of the c-th head in the l-th layer. This represents the learnable key matrix of the c-th head in the l-th layer. W represents the learnable key vector of the c-th head in the l-th layer. c,e Let e ​​represent the learnable matrix used for edge feature computation of the c-th head. ij Let b be the directed edge feature from node i to node j. c,e Let represent the learnable vector of the c-th head used for edge feature computation. This represents the attention of the c-th head in the l-th layer from node i to node j. This represents the number of nodes connected to node i, that is, the total number of all terminating nodes starting from node i.

[0040] First, use the corresponding trainable transformation matrix. and Start node features Convert to query vector Termination node characteristics Convert to key vector The provided edge feature e i,j This will be encoded into the key vector as additional information.

[0041] After obtaining multi-head attention, information will be aggregated to the node starting point i using the following formula:

[0042]

[0043]

[0044] || represents a join operation, connecting C head attentions together. This represents the node features of node i extracted by the l-th layer Transformer graph convolution module. This represents the learnable matrix used by the c-th head in the l-th layer to compute the intermediate result v. Let C represent the learnable vector of the c-th head in layer l used to compute the intermediate result v, where C represents the number of heads. This represents the number of nodes connected to node i.

[0045] The node features output by the final Transformer graph convolution module are the output of the entire feature extraction module. Then, relative and absolute position prediction heads are used to calculate the relative and absolute positions between nodes. The node features extracted by the feature extraction module will be used as input to the relative / absolute position prediction heads, respectively.

[0046] Among them, the relative position prediction head:

[0047] For any two node features, the relative position prediction head predicts the relative positional relationship of their corresponding objects, i.e., the difference in the object center coordinates. This part can be composed of a multilayer perceptron, and the calculation process is based on node features h. i and h j For example, we can concatenate these two tensors as input. The specific process is as follows:

[0048]

[0049] Here, mlps represents a multilayer perceptron, concat means connecting multiple vectors or matrices in a certain dimension, and the output is a 1x2 tensor. sigmoid(..)*2-1 is intended to map the output to [-1,1], and sigmoid is the activation function.

[0050] Absolute position prediction head:

[0051] For a given node feature, the absolute position prediction head predicts the absolute position of its corresponding object, i.e., its center coordinates on the corresponding image. This part can be composed of a multilayer perceptron, and the calculation process is based on the node feature h. i For example, the specific process is as follows:

[0052] sigmoid(mlps(h i ))*2-1

[0053] The output is a 1x2 tensor, and sigmoid(..)*2-1 is intended to map the output to [-1,1].

[0054] The relative position prediction head and absolute position prediction head of this application take node features as input and output the normalized coordinates of the object corresponding to the node in the image, as well as the difference between the coordinates of any two nodes.

[0055] After constructing the graph neural network, this application requires training the constructed graph neural network. Training the graph neural network first requires obtaining training samples. This embodiment uses the COCO stuff dataset as an example to illustrate the entire training and testing process of the graph neural network model. Compared to COCO, COCO stuff adds pixel-level annotation information for 81 types of stuff (such as sky, grass, etc.). When randomly sampling training samples from the dataset, the graph structure of the image can be extracted in advance, or it can be extracted during random sampling.

[0056] For each image, the center coordinates of each object (cars, people, animals, etc. in the image) are calculated. This application uses the average value of the object segmentation mask coordinates as the center coordinates of the object. Since COCO's segmentation annotations are polygon vertex coordinates, the centroid of the polygon is required as its center coordinates.

[0057] Suppose the polygon whose centroid is to be calculated consists of k points (x0, y0), (x1, y1), (x2, y2)...(x... k-1 ,y k-1 It is composed of ) and its centroid (C x C y The calculation is as follows:

[0058]

[0059]

[0060] A is the area of ​​this polygon, calculated as follows:

[0061]

[0062] For each graph with given labels, a graph structure is constructed based on these labels. The initial features of the nodes are object category, color, shape, etc. (which can be represented by one-hot encoding), and the edge features are encoded as the difference between the center coordinates of the starting node and the ending node after normalization.

[0063] This application introduces wildcard classes based on the existing categories in COCO stuff, taking into account possible application scenarios. That is, users do not need to specify that the object must be a known category in COCO stuff. Wildcard classes are used to indicate that the object is unknown to the user or unknown to the existing dataset.

[0064] Then, to improve the robustness of the model, this application adopts a random sampling method and generates positive and negative samples for the sampled graph structure. Finally, the matching degree between the positive and negative samples and the original graph structure is used as the supervision signal for model training. Specifically:

[0065] Assuming a batch size of 4, four graph structures {(N1,E1),(N2,E2),(N3,E3),(N4,E4)} are randomly selected from the dataset as parent graph structures. With a certain probability, some nodes and corresponding edges in the original graph structures are discarded, and corresponding subgraph structures {(N′1,E′1),(N′2,E′2),(N′3,E′3),(N′4,E′4)} are constructed for each parent graph structure. That is, the node and edge sets of the subgraph structures are subsets of the original graph structures, while ensuring the connectivity of the subgraph structures. Here, N represents the set of node features, E represents the set of edge features, the subscript indicates the image from which the graph structure is constructed, and the superscript distinguishes between the parent and subgraphs.

[0066] Then, the node categories of the subgraph structure are encoded into wildcard classes according to a certain proportion, that is, their original types are removed and set as wildcard classes. This makes the training samples more diverse and improves the robustness of the final model.

[0067] A subgraph structure paired with its corresponding parent graph structure constitutes a positive sample, while a subgraph structure paired with the parent graph structure of other subgraph structures constitutes a negative sample. There are currently eight graph structures in total. After feature extraction using a graph convolutional neural network, the node features of the corresponding graph structures can be obtained.

[0068] Because there is a one-to-one correspondence between nodes in the parent graph structure and the child graph structure, this application models the problems of node matching and graph structure matching based on this. Let the feature set of all parent graph structure nodes be H = H1∪H2∪…∪H n ={h 1,1 ,h 1,2 ,h 1,3 ,…,h n,k-1 ,h n,k The feature set of all subgraph structure nodes is H′=H′1∪H′2∪…∪H′ n ={h′ 1,1 ,h′ 1,2 ,h′ 1,3 ,…,h′ n,k-1 ,h′ n,k}, where h i,j With h′ i,j To represent a node feature, the first subscript is the sequence number of its corresponding image, and the second subscript indicates the corresponding entity object in that image; H i With H′ i It represents the set of all node features of a graph. The subscript indicates the index of the corresponding graph, and the superscript is used to distinguish between the parent graph and the child graph.

[0069] Node similarity is calculated as follows:

[0070] similarity (ij,pq) =dotproduct(h′)i,j ,h p,q );

[0071] Where dotproduct represents the dot product, similarity is the node similarity defined in this application, and h i,j h represents the feature of the j-th node in subgraph i. p,q This represents the feature of the q-th node in the parent graph p.

[0072] Graph structure similarity is calculated as follows:

[0073] similarity (i,p) =max({dotproduct(h′,h)|h′∈H′ i ,h∈H p});

[0074] Where max is used to find the maximum value in the set, H′ i H represents the set of features of all nodes in a subgraph. p Let h' represent the set of features of all nodes in the attached graph p, and let h' represent the subgraph H'. i The feature set of a certain image node in the set, where h represents the parent image H. p The feature set of a specific image node in the set.

[0075] Based on the above description, in this application, for any two images, the node similarity between each node feature of one image and each node feature of the other image is the dot product of the two node features; the maximum value selected from the node similarity between each node feature of one image and each node feature of the other image is the graph structure similarity between the two images.

[0076] For subgraph structure (N′) i ,E′ i A node h′ in ) i,j With parent graph structure (N) p E p A node h in ) q,p A match is made if and only if the node similarity of the two nodes satisfies the following equation:

[0077] similarity (ij,pq) =max({dotproduct(h′) i,j ,h)|h∈N p});

[0078] Where max is used to find the maximum value in the set, and the subscripts i and p represent the image numbers; h′ i,j with h q,p This represents the node features numbered j and q in the corresponding graph structure.

[0079] For subgraph structure (N′) i ,E′ i ) and parent graph structure (N p E p A graph structure is matched if and only if the graph structure similarity of the two graphs satisfies the following equation:

[0080] similarity (i,p) =max({dotproduct(h′,h)|h′∈N′ i ,h∈N p});

[0081] Where max is used to find the maximum value in the set; subscripts i and p represent image numbers; N and E represent the node feature set and edge feature set, respectively, with their subscripts being their corresponding image numbers and superscripts used to distinguish between parent and child graph structures; h′ and h represent a node feature in the child graph structure and the parent graph structure, respectively.

[0082] This embodiment designs the following loss functions for training the network, including: classification loss and auxiliary loss.

[0083] Among them, the classification loss is also known as the cross-entropy loss. In this application, the cross-entropy loss is used as a supervision signal during training to calculate the loss of node similarity.

[0084] Let the batch size be B; the feature set of all parent graph structure nodes in this batch is H = H1∪H2∪…∪H B ={h 1,1 ,h 1,2 ,h 1,3 ,…,h B,k-1 ,h n,k The feature set of all subgraph structure nodes is H′=H′1∪H′2∪…∪H′ B ={h′ 1,1 ,h′ 1,2 ,h′ 1,3 ,…,h′ B,k-1 ,h′ B,k The total number of nodes in the parent graph is of type |H| = C;

[0085] The classification loss formula is as follows:

[0086]

[0087] in:

[0088] |*| represents the number of elements in the set; y ij,pq For observation symbols:

[0089]

[0090] The observation symbol takes only 1 or 0, indicating whether node j in subgraph i and node q in parentgraph p constitute a positive example, with 1 indicating a positive example and 0 indicating a negative example;

[0091] p ij,pq The predicted probability that the observed sample is a positive example where node j in subgraph i and node q in parentgraph p are both positive examples:

[0092] p ij,pq =similarity (ij,pq)

[0093] That is, the node similarity between the two;

[0094] h represents a node feature. The first subscript is the index of its corresponding image, the second subscript is the index of the corresponding entity object in the graph, and the superscript is used to distinguish between the parent and child graphs; H i With H′ i It represents the set of all node features of a graph. The subscript indicates the index of the corresponding graph, and the superscript is used to distinguish between the parent graph and the child graph.

[0095] For the auxiliary loss, this application applies the regression loss of MSE to the relative and absolute positions predicted by the features of the subgraph nodes, including the regression loss of the relative positions between nodes and the regression loss of the absolute positions.

[0096] Let the batch size be B; the set of features of all subgraph structure nodes in this batch is H′=H′1∪H′2∪…∪H′ B ={h′ 1,1 ,h′ 1,2 ,h′ 1,3 ,…,h′ B,k-1 ,h′ B,k}

[0097] The formula is as follows:

[0098]

[0099] Where |*| represents the number of elements in the set; l1′ ij This indicates that node j in graph i is determined by h′. i,j When used as input, the relative position predicted by the relative position prediction head; l1 ij This represents the relative position of node j in graph i, calculated directly from the dataset labels; l2′ ij This indicates that node j in graph i is determined by h′. i,j When used as input, the absolute position predicted by the absolute position prediction head; l1′ ijThe absolute position of node j in graph i is directly calculated from the dataset label; h represents a node feature, with its first subscript being the sequence number of its corresponding image, the second subscript being the sequence number of the corresponding entity object in its corresponding graph direction, and the superscript being used to distinguish between the parent graph and the child graph.

[0100] Training the network model given the training data and loss function is a well-established technique in this field and will not be elaborated upon here. Using the trained graph neural network, the node features of the query image, as well as the relative and absolute positions between nodes, can be obtained.

[0101] Step S3: Based on the node features output by the graph neural network, calculate the graph structure similarity between the query image and the images in the image database, and provide the user with the retrieved image.

[0102] After training the graph neural network, the graph structure of the user input query image is {(N′1,E′1)}, and the graph structure set of the image database after preprocessing is {(N1,E1),(N2,E2),(N3,E3),…,(N…}. k E k The graph structure is defined as follows: N represents the set of node features, E represents the set of edge features, the subscript indicates the image from which the graph structure is constructed, and the superscript distinguishes the user-input query graph structure from the graph structures of images in the image database. First, a well-trained graph neural network (feature extraction module) is used to calculate the node features of the query image graph structure. Then, using the graph structure similarity method defined above, the similarity between the query image graph structure and all graph structures in the image database is calculated. The top k images based on similarity are returned to the user as the query results, thus achieving image retrieval.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for image retrieval based on graph structure matching, characterized by, The image retrieval method based on graph structure matching comprises: An image is acquired as a query image, each object in the query image is taken as a node, a connection between the nodes is taken as an edge, an attribute of the object is taken as an initial node feature, and a difference between the center coordinates of the start node and the end node of the edge is taken as a feature of the edge to construct a graph structure; The graph structure of the query image is input into a trained graph neural network to acquire node features of the query image and relative and absolute positions between the nodes, the graph neural network comprises a feature extraction module, a relative position prediction head and an absolute position prediction head, the feature extraction module is a three-layer Transformer graph convolution module and is used to extract the node features of the graph structure, the relative position prediction head and the absolute position prediction head are multilayer perceptrons and are respectively used to predict the relative and absolute positions between the nodes, a loss function of the graph neural network comprises a classification loss and an auxiliary loss, wherein the classification loss is used to calculate a loss of node similarity, and the auxiliary loss comprises a regression loss of the relative positions between the nodes and a regression loss of the absolute positions; According to the node features output by the graph neural network, graph structure similarity of the query image and images in an image database is calculated, and the images obtained by the query are fed back to a user.

2. The image retrieval method based on graph structure matching according to claim 1, characterized in that, The initial node feature is a one-hot encoding of the object attribute.

3. The image retrieval method based on graph structure matching according to claim 1, characterized in that, The feature of the edge is a difference between the center coordinates of the start node and the end node of the normalized edge.

4. The image retrieval method based on graph structure matching according to claim 1, characterized in that, The calculation of the similarity of the query image and the images in the image database comprises: For any two images, the node similarity of each node feature of one image and each node feature of the other image is calculated, and the node similarity is a dot product of the two node features; The maximum value is selected from the node similarity of each node feature of one image and each node feature of the other image as the graph structure similarity of the two images.

Images