Scene sketch retrieval system and method based on conditional graph autoencoder
Through the conditional graph autoencoder system, scene-level sketches are processed using graph neural network and gating attention mechanism, and the problem of matching scene sketches and natural images is solved, and the effectiveness and accuracy of cross-modal retrieval is achieved.
Patent Information
- Application Number
- CN202210578052.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-25
- Publication Date
- 2024-12-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to realize image retrieval based on scene-level sketches, especially due to the abstraction, stylization and complexity of scene sketches, as well as the difficulty in discriminating the interaction relationships of objects, which leads to difficulties in cross-modal alignment and matching.
Using a system based on conditional graph autoencoder, the region of interest and feature maps are extracted through object detectors and feature extractors, the condition map is constructed and weighted encoding is used, and the graph embedding vector is generated in combination with the gating attention mechanism, and the semantic resonance is weakly supervised. The structure module realizes the matching of scene sketches and natural images.
It effectively realizes the retrieval of scene sketches and natural images, combining instance semantics, global vision and spatial layout representation, improving the accuracy and practicality of the search.
Smart Images
Figure CN114821043B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a scene sketch retrieval system and method based on a conditional graph autoencoder, and belongs to the technical field of computer vision, in particular to the technical field of scene sketch retrieval. Background Art
[0002] Sketch-based image retrieval allows users to find the natural images they need with intuitive hand-drawn sketches. In today's information age led by multimedia data, it has strong commercial prospects and application value. As a cross-modal learning task that connects hand-drawn sketches and natural images, sketch-based image retrieval also helps explore the visual cognitive ability of the human brain.
[0003] Sketch-based image retrieval research has received widespread attention and has made great progress in academia. The main challenge lies in how to associate sparse line representations with dense pixel representations and establish correspondences between different visual modalities. Most existing solutions transform sketch retrieval into a cross-modal embedding learning problem. The conventional method is to extract feature vectors from the query sketch and the photo to be retrieved, embed them into the same vector space, and then compare the similarities based on the nearest neighbor search. Some recent methods have achieved excellent performance, but there is a key problem: these works only study "instance-level" sketches that depict a single object, but rarely explore "scene-level" sketches that contain multiple objects at the same time.
[0004] The instance-level sketch retrieval task itself is highly constrained, assuming that the user has only drawn a single object, which limits its practicality. Furthermore, due to the high abstractness of hand-drawn sketches, isolated objects are often difficult to identify in the absence of contextual information and object interactions. In real life, images containing only a single object are rare (primarily close-ups of products, etc.). More common are scenes with complex backgrounds and depicting multiple interacting objects. Therefore, instance-level sketch retrieval can only meet a small number of needs and is highly specialized. Scene-level sketch retrieval overcomes these shortcomings and has greater application potential.
[0005] Scene-level sketch retrieval provides a more intuitive way for users to directly hand-draw their mental images. This unique combination of sketch and scene could have a profound impact on the commercial application of image retrieval. However, scene-level sketch retrieval is extremely challenging, not only because it faces the same difficulties as instance-level sketch retrieval, but also because of its unique scene characteristics. The main challenges of scene-level sketch retrieval are as follows: (i) As a cross-modal learning task, there is a domain gap between the two input modalities. Scene sketches are more abstract and stylized than natural images (e.g., cartoons or anthropomorphic depictions of objects), making learning cross-modal visual correspondences challenging. (ii) When hand-drawing a scene, people tend to only depict salient and representative objects (e.g., main foreground objects such as people, cars, and buildings), rather than detailing every detail (some plants and less important objects may be omitted). The missing content in scene sketches makes cross-modal alignment even more difficult. (iii) A scene is determined not only by the objects it contains, but also by their spatial positions and interactions. Identifying scene matches requires considering multiple similarities simultaneously. From a theoretical perspective, instance-level sketches can be viewed as a relatively narrow domain where the category and appearance of a single object have relatively limited and predictable variability, while scene-level sketches are in a wider domain, further encoding spatial locations and interaction relationships, and have more complex variability.
[0006] In summary, how to realize image retrieval based on scene-level sketches has become a technical problem that urgently needs to be solved in the field of sketch retrieval technology. Summary of the Invention
[0007] In view of this, the purpose of the present invention is to invent a system and method based on structured feature embedding technologies such as graph neural networks to achieve the goal of image retrieval based on scene-level sketches.
[0008] To achieve the above objectives, the present invention proposes a scene sketch retrieval system based on a conditional graph autoencoder, which includes the following modules:
[0009] Conditional Graph Construction Module: This module uses an object detector and feature extractor to extract regions of interest (ROIs) and feature maps from scene sketches or natural images. Based on the graph structure, each graph node corresponds to the regional features of the extracted ROIs. Global context features are embedded as conditional constraints into each graph node to construct a conditional graph and generate conditional graph node features.
[0010] Weighted Graph Construction Module: This module uses a graph neural network to perform weighted encoding on the graph nodes of the conditional graph generated by the conditional graph construction module, based on the spatial layout of the region of interest in the original scene sketch or natural image, to construct a weighted graph and generate weighted graph node features.
[0011] Graph embedding vector generation module: This module uses a gated attention mechanism to learn the importance of each graph node in the weighted graph, performs weighted aggregation on all graph nodes, and generates a graph embedding vector.
[0012] Weakly supervised semantic reconstruction module: The function of this module is to reconstruct the semantic information of the region of interest in the scene sketch or natural image based on the conditional graph or weighted graph, that is, to generate a multi-label classification prediction vector for the region of interest;
[0013] Retrieval module: The function of this module is to calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them from small to large according to the distance, and return the natural images as retrieval results in sequence.
[0014] The specific contents of the conditional graph construction module for constructing the conditional graph and generating conditional graph node features are:
[0015] For any scene sketch or natural image, extract the feature map F of size h×w×d, where h, w, and d are the length, width, and number of channels of the feature map, respectively, and extract the set of regions of interest Where n is the number of regions of interest in the scene sketch or natural image; the regions of interest are set Mapped onto the feature map F and pooled into a set of regional features of the region of interest The regions of interest are used as nodes, and the connection between two nodes is used as an edge to abstract the scene sketch or natural image into a graph structure;
[0016] Apply a global average pooling operation on the feature map F to obtain the global context features of the scene sketch or natural image
[0017]
[0018] The global context features Add to the regional features of all regions of interest, that is, embed the global context features as conditional constraints into each graph node to generate conditional graph node features of scene sketches or natural images Thus constructing a conditional graph; The binary operator ‖ represents vector concatenation.
[0019] The specific contents of the weighted graph construction module for constructing a weighted graph and generating weighted graph node features are:
[0020] Based on the conditional graph node features and the set of regions of interest of the scene sketch or natural image, a graph convolution operation is used to generate weighted graph node features. in:
[0021]
[0022] Where W' is the learnable weight of the linear transformation and σ is the nonlinear activation function; Represents all neighbor nodes of node i;
[0023] In the above formula, α ij is the attention coefficient, calculated according to the following formula:
[0024]
[0025] In the above formula, e ij It represents the attention coefficient of node i's neighbor node j to node i, that is, the importance of node j to node i, and is calculated according to the following formula:
[0026]
[0027] In the above formula, W and Represents a learnable weight; LeakyReLU() represents a nonlinear activation function;
[0028] In the previous formula, β ij is the weighting coefficient, calculated according to the following formula:
[0029]
[0030] In the above formula, dist(b i ,b j ) represents two regions of interest b i and b j The absolute distance in the original scene sketch or natural image, H and W are the length and width of the original scene sketch or natural image respectively.
[0031] The specific content of the graph embedding vector generation module generating the graph embedding vector is:
[0032] According to the following formula, the gated attention mechanism is used to learn the importance of each graph node in the weighted graph, and all graph nodes are weightedly aggregated to generate a graph embedding vector:
[0033]
[0034] In the above formula, W a and W b They correspond to the learnable weights of the two fully connected layers respectively; ⊙ represents the Hadamard product, is an n-dimensional vector of all 1s.
[0035] The specific content of the weakly supervised semantic reconstruction module generating the multi-label classification prediction vector of the region of interest is:
[0036] Two fully connected layers are extended from the graph nodes of the conditional graph or weighted graph to classify the object instances in the region of interest corresponding to each graph node, and obtain the confidence score for each predicted category. The softmax function is applied to the output branch of one of the fully connected layer branches to normalize the scores in the category dimension, and the probability of each region of interest belonging to each category is calculated to obtain the classification probability matrix:
[0037]
[0038] The sigmoid function is applied to another fully connected layer branch. The fully connected layer calculates the contribution of all regions of interest to each category through a gating mechanism to obtain the positioning probability matrix:
[0039]
[0040] Where C is the number of predicted categories, n is the number of regions of interest, and W cls and W det are the learnable weights of the two fully connected layers; the outputs U and V of the two fully connected layer branches are bitwise multiplied by the Hadamard product to obtain a comprehensive cross-attention matrix, indicating the association between each region of interest and each category; the scores of all regions of interest in the attention matrix are added together to obtain the C-dimensional multi-label classification prediction vector
[0041]
[0042] In the above formula, ⊙ represents the Hadamard product, is an n-dimensional vector of all 1s.
[0043] The system is trained and optimized according to the following objective function:
[0044]
[0045] In the above formula, is a set of triplets sampled from 2N input images, where N represents N pairs of matching scene sketches and natural images randomly selected from the training data as an input batch each time, and Θ is the optimization parameter of the system;
[0046]
[0047] Where Δ is the margin hyperparameter, t=(S,P + ,P - ), S is the anchor scene sketch, P + is a positive sample natural image paired with S, P - is a negative sample image that is not paired with S; Graph embedding vectors of anchor scene sketches generated by the system; is the graph embedding vector of the positive sample natural image paired with S generated by the system; is the graph embedding vector of the negative sample image generated by the system that is not paired with S;
[0048]
[0049] in, The multi-label classification prediction vector of the region of interest generated by the weakly supervised semantic reconstruction module according to the conditional graph, The multi-label classification prediction vector of the region of interest generated by the weakly supervised semantic reconstruction module based on the weighted graph.
[0050] The present invention also proposes a scene sketch retrieval system method based on a conditional graph autoencoder, which includes the following steps:
[0051] (1) Object detectors and feature extractors are used to extract regions of interest and feature maps respectively. Based on the graph structure, each graph node corresponds to the regional features of the extracted region of interest, and the global context features are embedded into each graph node as conditional constraints to construct a conditional graph and generate conditional graph node features.
[0052] (2) Based on the generated conditional graph and the spatial layout of the region of interest in the original scene sketch or natural image, a graph neural network is used to perform weighted encoding on the graph nodes of the conditional graph, construct a weighted graph and generate weighted graph node features;
[0053] (3) Using a gated attention mechanism to learn the importance of each graph node in the weighted graph, performing weighted aggregation on all graph nodes to generate a graph embedding vector;
[0054] (4) Calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them from small to large according to the distance, and return the natural images as retrieval results in sequence.
[0055] The beneficial effects of the present invention are as follows: the system and method of the present invention use scene sketches and object instance features in natural images as nodes, and the global context features of the image as constraints, and construct the query scene sketches and natural images into a conditional graph; with a graph neural network as the core, the nodes are encoded by combining the graph attention mechanism and graph structure weighting to construct a weighted graph, and weakly supervised semantic decoding is used to maintain the consistency of semantic distribution. The instance semantic representation, global visual representation, and spatial layout representation are combined to jointly learn scene embedding features, thereby realizing the retrieval of natural images based on scene sketches and obtaining good experimental results. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 This is a module diagram of the scene sketch retrieval system based on conditional graph autoencoder proposed in the present invention.
[0057] Figure 2 This is a flowchart of the scene sketch retrieval method based on conditional graph autoencoder proposed in the present invention. DETAILED DESCRIPTION
[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.
[0059] See also Figure 1 , introduces the scene sketch retrieval system based on conditional graph autoencoder proposed in this invention, which includes the following modules:
[0060] Conditional Graph Construction Module: This module uses an object detector and feature extractor to extract regions of interest (ROIs) and feature maps from scene sketches or natural images. Based on the graph structure, each graph node corresponds to the regional features of the extracted ROIs. Global context features are embedded as conditional constraints into each graph node to construct a conditional graph and generate conditional graph node features.
[0061] Weighted Graph Construction Module: This module uses a graph neural network to perform weighted encoding on the graph nodes of the conditional graph generated by the conditional graph construction module, based on the spatial layout of the region of interest in the original scene sketch or natural image, to construct a weighted graph and generate weighted graph node features.
[0062] Graph embedding vector generation module: This module uses a gated attention mechanism to learn the importance of each graph node in the weighted graph, performs weighted aggregation on all graph nodes, and generates a graph embedding vector.
[0063] Weakly supervised semantic reconstruction module: The function of this module is to reconstruct the semantic information of the region of interest in the scene sketch or natural image based on the conditional graph or weighted graph, that is, to generate a multi-label classification prediction vector for the region of interest;
[0064] Retrieval module: The function of this module is to calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them from small to large according to the distance, and return the natural images as retrieval results in sequence.
[0065] The specific contents of the conditional graph construction module for constructing the conditional graph and generating conditional graph node features are:
[0066] For any scene sketch or natural image, extract a feature map F of size h×w×d, where h, w, and d are the length, width, and number of channels of the feature map, respectively. In the embodiment, two pre-trained ResNet-50 networks (see: He, K., Zhang, X., Ren, S., et al. Deep residual learning for image recognition[A]. / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C], Las Vegas, NV, USA: IEEE, 2016: 770-778.) are used, and the last pooling layer and classification output layer are removed as feature extractors to extract feature maps F for scene sketches and natural images, respectively.
[0067] In the embodiment, Mask R-CNN (see: He, K., Gkioxari, G., Dollár, P., et al. Mask R-CNN [A]. / / Proceedings of the IEEE International Conference on Computer Vision [C], Venice, Italy: IEEE, 2017: 2980-2988.) is used as an object detector to extract a set of regions of interest. Where n is the number of regions of interest in the scene sketch or natural image; in the embodiment, the region of interest set is aligned using RoIAlign (see: He, K., Gkioxari, G., Dollár, P., et al. Mask R-CNN[A]. / / Proceedings of the IEEE International Conference on Computer Vision[C], Venice, Italy: IEEE, 2017: 2980-2988.) technology. Mapped onto the feature map F and pooled into a set of regional features of the region of interest The regions of interest are used as nodes, and the connection between two nodes is used as an edge to abstract the scene sketch or natural image into a graph structure;
[0068] Apply a global average pooling operation on the feature map F to obtain the global context features of the scene sketch or natural image
[0069]
[0070] The global context features Add to the regional features of all regions of interest, that is, embed the global context features as conditional constraints into each graph node to generate conditional graph node features of scene sketches or natural images Thus constructing a conditional graph; The binary operator ‖ represents vector concatenation.
[0071] In the embodiment, the RoIAlign technique is used to pool the features in each region of interest into a 2048-dimensional feature vector. At the same time, global average pooling is applied to the entire feature map F to obtain a 2048-dimensional global context feature.
[0072] The specific contents of the weighted graph construction module for constructing a weighted graph and generating weighted graph node features are:
[0073] Based on the conditional graph node features and the set of regions of interest of the scene sketch or natural image, a graph convolution operation is used to generate weighted graph node features. in:
[0074]
[0075] Where W' is the learnable weight of the linear transformation and σ is the nonlinear activation function; Represents all neighbor nodes of node i;
[0076] In the above formula, α ij is the attention coefficient, calculated according to the following formula:
[0077]
[0078] In the above formula, e ij Indicates the attention coefficient of node i's neighbor node j to node i, that is, the importance of node j to node i; in the embodiment, e ij It is learned through the self-attention mechanism of the graph attention network (graph attention network is a special graph neural network, see: Velickovic, P., Cucurull, G., Casanova, A., et al. Graph attention networks[A]. / / Proceedings of the International Conference on Learning Representations[C], Vancouver, BC, Canada: OpenReview.net, 2018.), that is, calculated according to the following formula:
[0079]
[0080] In the above formula, W and LeakyReLU() represents a nonlinear activation function (see: Maas, AL, Hannun, AY, Ng, AYRectifier nonlinearities improve neural network acoustic models[A]. / / Proceedings of the International Conference on Machine Learning[C], Atlanta, Georgia, USA: PMLR, 2013: 3.).
[0081] In the previous formula, β ij is the weighting coefficient, calculated according to the following formula:
[0082]
[0083] In the above formula, dist(b i ,b j ) represents two regions of interest b i and b j The absolute distance in the original scene sketch or natural image, H and W are the length and width of the original scene sketch or natural image respectively. In the embodiment, the Euclidean distance is used. i ,b j )The closer the distance, the larger the weighting coefficient, and the value is guaranteed to fall within the interval [0,1].
[0084] The specific content of the graph embedding vector generation module generating the graph embedding vector is:
[0085] According to the following formula, the gated attention mechanism (see: Li, Y., Tarlow, D., Brockschmidt, M., et al. Gated graph sequence neural networks[A]. / / Proceedings of the International Conference on Learning Representations[C], San Juan, Puerto Rico: ICLR, 2016.) is used to learn the importance of each graph node of the weighted graph, perform weighted aggregation on all graph nodes, and generate a graph embedding vector:
[0086]
[0087] In the above formula, W a and W b They correspond to the learnable weights of the two fully connected layers respectively; ⊙ represents the Hadamard product, is an n-dimensional vector of all 1s.
[0088] The specific content of the weakly supervised semantic reconstruction module generating the multi-label classification prediction vector of the region of interest is:
[0089] Two fully connected layers are extended from the graph nodes of the conditional graph or weighted graph to classify the object instances in the region of interest corresponding to each graph node, and obtain the confidence score for each predicted category. The softmax function is applied to the output branch of one of the fully connected layer branches to normalize the scores in the category dimension, and the probability of each region of interest belonging to each category is calculated to obtain the classification probability matrix:
[0090]
[0091] The sigmoid function is applied to another fully connected layer branch. The fully connected layer calculates the contribution of all regions of interest to each category through a gating mechanism to obtain the positioning probability matrix:
[0092]
[0093] Where C is the number of predicted categories, n is the number of regions of interest, and W cls and W det are the learnable weights of the two fully connected layers; the outputs U and V of the two fully connected layer branches are bitwise multiplied by the Hadamard product to obtain a comprehensive cross-attention matrix, indicating the association between each region of interest and each category; the scores of all regions of interest in the attention matrix are added together to obtain the C-dimensional multi-label classification prediction vector
[0094]
[0095] In the above formula, ⊙ represents the Hadamard product, is an n-dimensional vector of all 1s.
[0096] The system is trained and optimized according to the following objective function:
[0097]
[0098] In the above formula, is a set of triplets sampled from 2N input images, where N represents N pairs of matching scene sketches and natural images randomly selected from the training data as an input batch each time, and Θ is the optimization parameter of the system;
[0099]
[0100] Where Δ is the margin hyperparameter, t=(S,P + ,P - ), S is the anchor scene sketch, P + is a positive sample natural image paired with S, [ - is a negative sample image that is not paired with S; Graph embedding vectors of anchor scene sketches generated by the system; is the graph embedding vector of the positive sample natural image paired with S generated by the system; is the graph embedding vector of the negative sample image generated by the system that is not paired with S;
[0101]
[0102] in, The multi-label classification prediction vector of the region of interest generated by the weakly supervised semantic reconstruction module according to the conditional graph, The weakly supervised semantic reconstruction module generates a multi-label classification prediction vector for the region of interest based on the weighted graph. The weakly supervised semantic reconstruction module uses the Kullback-Leibler (see: Kullback, S., Leibler, RA On information and sufficiency [J]. The Annals of Mathematical Statistics, 1951, 22 (1): 79-86.) divergence constraint to ensure the consistency of the semantic distribution.
[0103] See also Figure 2 , introduces a scene sketch retrieval system method based on conditional graph autoencoder proposed in the present invention, the method comprising the following steps:
[0104] (1) Object detectors and feature extractors are used to extract regions of interest and feature maps respectively. Based on the graph structure, each graph node corresponds to the regional features of the extracted region of interest, and the global context features are embedded into each graph node as conditional constraints to construct a conditional graph and generate conditional graph node features.
[0105] (2) Based on the generated conditional graph and the spatial layout of the region of interest in the original scene sketch or natural image, a graph neural network is used to perform weighted encoding on the graph nodes of the conditional graph, construct a weighted graph and generate weighted graph node features;
[0106] (3) Using a gated attention mechanism to learn the importance of each graph node in the weighted graph, perform weighted aggregation on all graph nodes to generate a graph embedding vector;
[0107] (4) Calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them from small to large according to the distance, and return the natural images as retrieval results in sequence.
[0108] The inventors conducted extensive experiments on the large-scale scene sketch dataset SketchyScene proposed by Zou et al. (see: Zou, C., Yu, Q., Du, R., et al. SketchyScene: Richly-annotated scene sketches[A]. / / Proceedings of the European Conference on Computer Vision[C], Munich, Germany: Springer, 2018: 421-436.). During the experiment, the dimension of the node features in the conditional graph and the final aggregated graph embedding vector was set to 512. The Adam optimizer (see: Kingma, DP, Ba, J. Adam: A method for stochastic optimization [A]. / / Proceedings of the International Conference on Learning Representations [C], San Diego, CA, USA: ICLR, 2015.) was used for 800 iterations on the training set, with a fixed learning rate of 1e-4 and a batch size of 12 for the input data. The resolution of all input images (including scene sketches and natural images) was scaled to 448*448 and randomly flipped horizontally during the training phase. Compared with some existing methods, the system and method of the present invention achieved the best experimental results in both the top K accuracy (acc.@K) and the mean accuracy (MeanAcc) technical indicators, demonstrating the effectiveness and feasibility of the system and method of the present invention.
Claims
1. A scene sketch retrieval system based on conditional graph autoencoder, characterized by: The system includes the following modules: Conditional graph construction module: This module uses object detectors and feature extractors to extract regions of interest and feature graphs from scene sketches or natural images. Based on the graph structure, each graph node corresponds to the regional features of the extracted region of interest, and the global context features are embedded as conditional constraints into each graph node to construct a conditional graph and generate conditional graph node features. Weighted graph construction module: The function of this module is to use a graph neural network to weight the graph nodes of the conditional graph generated by the conditional graph construction module and based on the spatial layout of the region of interest in the original scene sketch or natural image, construct a weighted graph and generate weighted graph node features; Graph embedding vector generation module: The function of this module is to use the gated attention mechanism to learn the importance of each graph node of the weighted graph, perform weighted aggregation on all graph nodes, and generate a graph embedding vector; Weakly supervised semantic reconstruction module: The function of this module is to reconstruct the semantic information of the region of interest in the scene sketch or natural image according to the conditional graph or weighted graph, that is, to generate a multi-label classification prediction vector for the region of interest; Retrieval module: The function of this module is to calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them from small to large according to the distance, and return the natural images as retrieval results in turn.
2. The scene sketch retrieval system based on conditional graph autoencoder according to claim 1, characterized in that: The specific contents of the conditional graph construction module constructing the conditional graph and generating the conditional graph node features are: For any scene sketch or natural image, extract a feature map F of size h×w×d, where h, w, and d are the length, width, and number of channels of the feature map, respectively, and extract the set of regions of interest Where n is the number of regions of interest in the scene sketch or natural image; the region of interest set Mapped onto the feature map F and pooled into a set of regional features of the region of interest The regions of interest are taken as nodes, and the connection between two nodes is taken as an edge, so that the scene sketch or natural image is abstracted into a graph structure; Apply a global average pooling operation on the feature map F to obtain the global context feature of the scene sketch or natural image The global context feature Add to the regional features of all regions of interest, that is, embed the global context features as conditional constraints into each graph node to generate conditional graph node features of scene sketches or natural images Thus constructing a conditional graph; The binary operator || represents vector concatenation.
3. The scene sketch retrieval system based on conditional graph autoencoder according to claim 2, characterized in that: The specific contents of the weighted graph construction module for constructing a weighted graph and generating weighted graph node features are: Based on the conditional graph node features and the set of regions of interest of the scene sketch or natural image, the weighted graph node features are generated by graph convolution operation. in: Where W' is the learnable weight of the linear transformation, and σ is the nonlinear activation function; Represents all neighbor nodes of node i; In the above formula, α ij is the attention coefficient, calculated according to the following formula: In the above formula, e ij It represents the attention coefficient of node i's neighbor node j to node i, that is, the importance of node j to node i, which is calculated according to the following formula: In the above formula, W and Represents a learnable weight; LeakyReLU() represents a nonlinear activation function; In the above formula, β ij is the weighting coefficient, calculated according to the following formula: In the above formula, dist(b i ,b j ) represents two regions of interest b i and b j The absolute distance in the original scene sketch or natural image, H and W are the length and width of the original scene sketch or natural image, respectively.
4. The scene sketch retrieval system based on conditional graph autoencoder according to claim 3, characterized in that: The specific content of the graph embedding vector generation module generating the graph embedding vector is: According to the following formula, the gated attention mechanism is used to learn the importance of each graph node of the weighted graph, and all graph nodes are weightedly aggregated to generate a graph embedding vector: In the above formula, W a and W b Corresponding to the learnable weights of the two fully connected layers; ⊙ represents Hadamard. is an n-dimensional vector of all 1s.
5. The scene sketch retrieval system based on conditional graph autoencoder according to claim 4, characterized in that: The specific content of the weakly supervised semantic reconstruction module generating the multi-label classification prediction vector of the region of interest is: Extending two fully connected layers on the graph nodes of the conditional graph or the weighted graph, classifying the object instances in the region of interest corresponding to each graph node, and obtaining a confidence score for each predicted category; The output branch of one of the fully connected layer branches applies the softmax function, normalizes the scores in the category dimension, calculates the probability that each region of interest belongs to each category, and obtains the classification probability matrix: The sigmoid function is applied to another fully connected layer branch. The fully connected layer calculates the contribution of all regions of interest to each category through a gating mechanism to obtain a positioning probability matrix: Where C is the number of predicted categories, n is the number of regions of interest, and W cls and W det are the learnable weights of the two fully connected layers; The outputs U and V of the two fully connected layer branches are bitwise multiplied by the Hadamard product to obtain a comprehensive cross-attention matrix, indicating the relevance of each region of interest to each category; the scores of all regions of interest in the attention matrix are added to obtain a C-dimensional multi-label classification prediction vector In the above formula, ⊙ represents the Hadamard product, is an n-dimensional vector of all 1s.
6. The scene sketch retrieval system based on conditional graph autoencoder according to claim 5, characterized in that: The system is trained and optimized according to the following objective function: In the above formula, is a set of triplets sampled from 2N input images, where N represents N pairs of matching scene sketches and natural images randomly selected from the training data as an input batch each time, and Θ is the optimization parameter of the system; Where Δ is the margin hyperparameter, t=(S,P + ,P - ), S is the anchor scene sketch, P + is a positive sample natural image paired with S, P - is a negative sample image that is not paired with S; The graph embedding vector of the anchor scene sketch generated by the system; is the graph embedding vector of the positive sample natural image paired with S generated by the system; is the graph embedding vector of the negative sample image generated by the system that is not paired with S; in, The multi-label classification prediction vector of the region of interest generated by the weakly supervised semantic reconstruction module according to the conditional graph, The multi-label classification prediction vector of the region of interest generated by the weakly supervised semantic reconstruction module based on the weighted graph.
7. A scene sketch retrieval system method based on conditional graph autoencoder, characterized by: The method comprises the following steps: (1) Object detectors and feature extractors are used to extract regions of interest and feature maps respectively. Based on the graph structure, each graph node corresponds to the regional features of the extracted region of interest, and the global context features are embedded into each graph node as conditional constraints to construct a conditional graph and generate conditional graph node features. (2) Based on the generated conditional graph and the spatial layout of the region of interest in the original scene sketch or natural image, a graph neural network is used to weight the graph nodes of the conditional graph, construct a weighted graph and generate weighted graph node features; (3) Using a gated attention mechanism to learn the importance of each graph node in the weighted graph, performing weighted aggregation on all graph nodes, and generating a graph embedding vector; (4) Calculate the distance between the graph embedding vector of the scene sketch to be retrieved and the graph embedding vectors of all natural images in the database, sort them in ascending order of distance, and return the natural images as retrieval results in sequence.