Attribute-based neighborhood relation guided combined image retrieval method and system

Abstract

The invention relates to a combined image retrieval method and system guided by a neighborhood relation based on attributes, and the method comprises the steps: reading training set data in batches, and carrying out the extraction of global features and local features of the training set data; splicing the local features and the global features to form original attribute features of the vision and the text; extracting attribute prototype features; constructing coherent prototype semantics; combining the attribute prototype characteristics of each element in the multi-modal query; for the combined attribute prototype features, modeling is carried out in combination with double relationships, and modeling is carried out on a pairwise relationship and a neighborhood relationship through attribute similarity, so that the measurement learning process is optimized; the COMBINER model generates a corresponding combination feature; similarity scores are solved, and descending order arrangement is carried out; and according to the actual demand, selecting the first K target images as a formal result set so as to complete the combined image retrieval. According to the method, the precision and robustness of combined image retrieval are improved.

Description

Technical Field

[0001] This invention relates to a method and system for querying and understanding e-commerce image data and open-world data and for similarity retrieval, and particularly to a method and system for combined image retrieval guided by attribute-based neighborhood relationships, belonging to the technical field of multimodal information retrieval. Background Technology

[0002] With the exponential growth in the number of images, image retrieval that matches user interests has become a focus in the field of information retrieval. Traditional methods, limited to pure image or text queries, struggle to meet diverse needs, thus giving rise to combined image retrieval tasks. These tasks consist of a reference image, a target image, and modified text: the reference image is an approximation of the target image, while the modified text clarifies the differences between the two. This retrieval method effectively expresses search intent and is highly practical in intelligent retrieval tasks such as product retrieval and multimodal recommendation.

[0003] Specifically, the combined image retrieval task involves retrieving the target image required by the user from a database based on a reference image and a modified description input by the user. The key to this task lies in the integration of multimodal query features and metric learning, requiring precise handling of visually similar but attribute-independent images to improve retrieval accuracy. For ease of description, this invention refers to the attribute-related elements involved in the reference image and modified description as "attribute prototypes," and the associations established based on these elements as "neighborhood relationships."

[0004] While existing research has attempted to improve the performance of combined image retrieval based on neighborhood relationship modeling, it has failed to fully consider the differences in attribute relevance among visually similar images. That is, some visually similar images may be unrelated to the attributes required for modification, leading existing methods to incorrectly narrow the gap between such images and the query, leaving room for performance improvement. Therefore, to construct an effective combined image retrieval method, it is necessary not only to consider cross-modal feature fusion of visual and textual data but also to optimize neighborhood relationship modeling based on attribute similarity. Currently, guiding neighborhood relationship modeling through attribute prototypes to construct effective combined image retrieval methods faces the following three main challenges:

[0005] (1) Entanglement of attribute-level semantics. Whether it is an image or text, the semantic information of different attributes is often entangled with each other and does not exist in an independent form. It is difficult to separate the various attribute prototypes from the original features, which affects the judgment of attribute correlation.

[0006] (2) Inconsistency between modalities. Images and text belong to different modalities, and their attribute prototypes may differ, which makes it difficult to conduct cross-modal information interaction and extract cross-modal unified prototypes, and is not conducive to the evaluation of attribute similarity.

[0007] (3) Lack of supervision signals. When modeling pairwise and neighborhood relationships based on attribute similarity, there is a lack of explicit supervision signals for neighborhood relationships, making it difficult to accurately identify visually similar but attribute-independent images, which affects the optimization of metric learning. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a combined image retrieval method guided by attribute-based neighborhood relationships, in order to achieve accurate and effective retrieval of user target images.

[0009] The present invention also proposes a combined image retrieval system guided by attribute-based neighborhood relationships.

[0010] To address this, this invention first proposes an adaptive semantic separation module to separate attribute features and solve the problem of attribute-level semantic entanglement. Secondly, it proposes a unified prototype-based combination module to construct a cross-modal unified prototype through cross-modal semantic interaction and derive attribute-level combination features to address inconsistencies between modalities. Finally, it proposes a dual relation modeling module to model pairwise and neighborhood relations based on attribute similarity, optimizing the metric learning process and improving retrieval performance in the absence of explicit supervision signals.

[0011] Terminology Explanation: CLIP is a deep learning model built on a contrastive learning approach, with the core objective of achieving cross-modal fusion understanding of text and image information. This model, pre-trained on large-scale image-text pair datasets, can flexibly adapt to various task scenarios, including image retrieval, text generation, and image classification. With its superior cross-modal feature learning capabilities, CLIP demonstrates excellent performance in handling visual and linguistic information interaction tasks.

[0012] 2. Attention mechanisms, as an important computational method in deep learning models, are widely used in various fields such as natural language processing and computer vision. Their core function is to dynamically focus on key parts of the input data, highlighting important information by assigning different weights, thereby effectively improving the model's processing efficiency and effectiveness.

[0013] 3. The Multilayer Perceptron (MLP) is a feedforward neural network with at least three layers: an input layer, hidden layers, and an output layer. Each layer consists of several neurons connected by weights. This model is widely used in tasks such as classification, regression, and feature extraction. With its powerful nonlinear fitting capabilities, it demonstrates excellent performance in pattern recognition, image processing, natural language processing, and many other fields.

[0014] 4. The Softmax function, as a commonly used activation function, is widely used in the output layer of multi-class classification problems. Its core function is to transform the input real-valued vector into a probability distribution, which not only limits each output value to the range of 0 to 1, but also ensures that the sum of all output values ​​is 1.

[0015] 5. The Tanh function, as a common activation function, is widely used in the hidden layers of neural networks. Its core function is to map the real values ​​of the input to the interval between -1 and 1. This not only enables non-linear transformation of the data but also gives the output the characteristic of zero mean, which helps to accelerate the convergence of the model.

[0016] 6. Average Pooling, a commonly used downsampling technique in convolutional neural networks, effectively reduces the spatial dimension of data by averaging the input feature map across regions, while simultaneously extracting important features.

[0017] 7. KL divergence (Kullback-Leibler Divergence) is an asymmetric metric that measures the difference between two probability distributions. It is mainly used to assess the information loss of one probability distribution relative to another, and is also known as relative entropy.

[0018] The technical solution of the present invention is as follows: Attribute-based neighborhood relation-guided combined image retrieval methods include: The training set data is read in batches, and global and local features are extracted from the training set data. The local and global features are concatenated to form the original attribute features of visual and textual data. Semantic attribute attention is used to further extract the attribute prototype features from the original attribute features. We use a semantic attribute attention mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP. For visual and textual attribute prototype features, the attribute prototype features of each element in the multimodal query are combined by combining them based on a unified prototype and forcing alignment through a unified prototype standard. For the combined attribute prototype features, a dual-relationship model is used to model pairwise relationships and neighborhood relationships through attribute similarity, thereby optimizing the metric learning process. Specifically, this includes: supervised pairwise relationship modeling, which calculates the classification loss for the combined features and target features after average pooling; and unsupervised neighborhood relationship modeling, which uses attribute similarity evaluation to measure the attribute correlation between samples. The COMBINER model extracts and combines attribute prototype features, performing multimodal queries on the input reference image and modified text to generate corresponding combined features. Then, it calculates the dot product of these combined features with each image in the image library to obtain a similarity score. All similarity scores are then sorted in descending order. Based on actual needs, the top-ranked features are selected. K The target image is used as the formal result set to complete the combined image retrieval.

[0019] As a further preferred approach, the training set data is read in batches, with each batch containing data sets in the form of <reference image, modified text, target image> triples. Then, global and local features are extracted from the training set data. The reference image is extracted using CLIP. global features and local features , represented as: ; ; in, for S OK A real matrix of columns, and These represent the global and local feature dimensions of CLIP, respectively. S The number of channels for each sample, and These represent the last and penultimate layers of the CLIP image encoder, respectively. Employing a linear transformation layer Local feature dimensions Align to global features and dimensionality To form a unified dimension of features The formula is as follows: , in, Align local feature dimensions to D Obtain local features aligned with the global feature dimensions. .

[0020] As a further preferred approach, semantic attribute attention is used to further extract attribute prototype features from the original attribute features; including: Based on local or global features, semantic attribute attention is used to capture contextual semantics and adaptive separation is performed at the prototype level. The prototype weights obtained from the multilayer perceptron are used to weight and aggregate local or global features to obtain local or global attribute prototype features. The global attribute prototype features are fused with the local attribute prototype features.

[0021] Further preferred, for the reference image Local features of unified feature dimension By capturing contextual semantics, adaptive separation is performed at the prototype level, as shown below: ; in, Represents the weights of the linear layer. Indicates the embedding dimension. , Indicates the total number of prototypes for local attributes. This represents the output of an unbiased multilayer perceptron; Tanh is the Tanh activation function. To unify the local features of the feature dimension, The transpose of the output of an unbiased multilayer perceptron, where Softmax is the Softmax activation function. For prototype weights, and use Weighted aggregation is performed to obtain the local attribute prototype features of the reference image. The formula is as follows: ; Extract the global attribute prototype features of the reference image from the global features of the reference image. , where X is the number of global property prototypes; Will and Fusion constructs multi-granularity attribute prototype features of the reference image, denoted as... , ; Separate the multi-granularity attribute prototype features of the modified text and the target image, denoted as follows: .

[0022] As a further preferred approach, for multi-granularity attribute prototype features, a semantic attribute attention mechanism is used to separate the cross-modal unified prototype (CUP) from simply concatenated multimodal query features, and a coherent prototype semantics is constructed based on the CUP; including: Perform a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to extract a cross-modal unified prototype; The extracted cross-modal unified prototype is used as the basis for combination constraints. With the help of linear learning capabilities, the prototype feature weights of the reference image and modified text are used for learning.

[0023] A further preferred approach involves performing a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to generate concatenated features. ,in, U This represents the number of attribute prototypes. D The feature dimension is used; subsequently, semantic attribute attention is invoked to separate the cross-modal unified prototype from the concatenated features, denoted as . The mathematical expression is: .

[0024] Further optimized, the extracted cross-modal unified prototype is used as the basis for combination constraints. Leveraging linear learning capabilities, the weights of the prototype features in the reference image and the modified prototype features are learned; this is achieved through the Sigmoid function. Weight calculation with multilayer perceptron: ; in, , These correspond to the weights of the reference image features and the modified text features relative to the CUP, respectively; MLP stands for Multilayer Perceptron. based on and For the original attribute prototype features and Aggregation yields attribute-level combined features for multimodal queries. The formula is: ; This is an element-wise multiplication operation.

[0025] As a further preferred approach, supervised pairwise relation modeling is used, which calculates a classification loss for the combined features and target features after average pooling; including: Combined features after average pooling The calculation is as follows: ; Calculate the target features after average pooling ; Based on the combined features after average pooling and target features The batch-based classification loss is defined as follows. : ; in, and , represent the average pooling results of the combined features and the target features of the i-th triplet, respectively. This represents the average pooling result of the combined features and target features of the j-th triplet, where B represents the batch size. Used to calculate cosine similarity This is the temperature scaling factor.

[0026] As a further preferred approach, an unsupervised neighborhood relationship modeling method is used to measure the attribute correlation between samples using attribute similarity evaluation; including: Using the K-means clustering algorithm, H semantic cluster centers are set for the prototype features of the target image after average pooling, where H is a manually set parameter. Unsupervised clustering is then performed to obtain... H Let there be semantic cluster centers, and denote the set of semantic centers as . ,in, H Indicates the number of semantic cluster centers; For each triple, the semantic clustering center is used as a common visual cluster and attribute-related neighborhood. The target features of each training triple and its triples are identified as neighborhood relation triples; the set of neighborhood relation triples is represented as follows: Where N is the number of training triples, It is the first i The semantic cluster centers corresponding to each triple .

[0027] A further preferred approach is to use a batch-based classification loss function. The definition is as follows: ; in, B Let s represent the number of triples in each batch, and s be the similarity function. Here, is the temperature coefficient, exp() is the exponential function, and log() is the logarithmic function.

[0028] A further optimized approach utilizes a mutual learning strategy to ensure that the similarity distribution between each multimodal query and the semantic clustering center remains consistent; specifically, assuming that each batch contains... B The third triplet will be the first i The batch-level similarity distribution of a combination of features is defined as follows: ,in Indicates the first i The first combination feature and the first j The similarity of the cluster centers of a triplet is calculated using the following formula: ; Derive the similarity distribution of target features .

[0029] A further preferred approach is to define the loss term for the similarity distribution regularization term oriented towards clustering using the Kullback-Leibler divergence. ,Promote and The consistency between them is shown in the following formula: ; in, Let KL divergence be the KL divergence. Let represent the similarity distribution between the i-th sample and the j-th target feature. Let represent the similarity distribution between the combined features of the i-th sample and the target feature of the j-th sample.

[0030] Further optimized, a pooling-oriented distribution consistency regularization term is designed; specifically including: First, define and ;in, Indicates the first i The first combination feature and the first j The similarity of the average pooling target features; Indicates the first i The target feature and the first j The similarity of the average pooling target features; Then, a pooling-oriented distribution consistency regularization term is defined using KL divergence, as shown in the following formula: ; Finally, the metric space optimization function is as follows: ; in, This represents the set of learnable parameters in the COMBINER model. As a trade-off hyperparameter.

[0031] A computer device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, enables the implementation of the steps of the aforementioned attribute-based neighborhood relation-guided combined image retrieval method.

[0032] A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of an attribute-based neighborhood relation-guided combined image retrieval method.

[0033] Attribute-based neighborhood relation-guided combined image retrieval systems include: The adaptive semantic separation module is configured to: read training set data in batches and extract global and local features from the training set data; concatenate the local and global features to form the original attribute features of visual and textual data; and further extract the prototype features of visual and textual attributes from the original features using semantic attribute attention. The unified prototype-based combination module is configured to: use a semantic attribute attention mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP; for attribute prototype features, the unified prototype-based combination module is combined to force alignment through a unified prototype standard to combine the attribute prototypes of each element in the multimodal query. The dual relation modeling module is configured to: for the combined attribute prototype features, combine the dual relation modeling module to model the pairwise relations and neighborhood relations through attribute similarity, thereby optimizing the metric learning process; The image retrieval module is configured as follows: The COMBINER model performs multimodal query combination on the input reference image and modified text through attribute prototype feature extraction and combination operations, generating corresponding combined features; subsequently, the dot product of this combined feature with each image in the image library is calculated to obtain a similarity score, and all similarity scores are then sorted in descending order; based on actual needs, the top-ranked images are selected. K The target image is used as the formal result set to complete the combined image retrieval.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: 0. This invention proposes an adaptive semantic separation module, which constructs a base of original attribute features for visual and textual data by splicing local and global features; then, it uses a semantic attribute attention mechanism to deeply separate the prototype features of visual and textual attributes from the original features, thereby achieving accurate anchoring of the core semantics and providing high-purity basic feature input for subsequent modules.

[0035] 2. This invention proposes a combination module based on a unified prototype. For attribute prototype features, a forced alignment strategy is implemented based on the unified prototype standard to perform structured combination of attribute prototypes of each element in multimodal queries. By standardizing the fusion logic of cross-modal features, the interference of modal heterogeneity is eliminated, and a combination feature with a consistent semantic benchmark is constructed, laying a solid foundation for accurate matching in retrieval tasks.

[0036] 3. This invention proposes a dual relation modeling module, which, based on attribute prototype features after combination, jointly models pairwise relations (supervised semantic association) and neighborhood relations (unsupervised semantic clustering) by leveraging attribute similarity. By mining implicit semantic connections between features, the metric space learning process is optimized, enabling the retrieval system to more sensitively capture semantic matching patterns, ultimately improving the accuracy and robustness of combined image retrieval. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the combined image retrieval method based on attribute-based neighborhood relationships, as described in this invention. Detailed Implementation

[0038] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0039] Example 1 COMBINER, an attribute-based neighborhood relation-guided composite image retrieval method, such as Figure 1 As shown, a composite image retrieval method COMBINER based on attribute-based neighborhood relationships is constructed to achieve composite image retrieval for multimodal queries. This method first uses an adaptive semantic separation module to construct an original attribute feature base by splicing local and global features, and then uses a semantic attribute attention mechanism to separate visual and textual attribute prototype features, achieving accurate anchoring of core semantics and providing high-purity basic features for subsequent modules. Then, a composite module based on a unified prototype performs forced alignment on the attribute prototype features, using a unified prototype standard to structurally combine the attribute prototypes of each element in the multimodal query, eliminating modal heterogeneity interference and constructing composite features with a consistent semantic benchmark, laying a solid foundation for accurate matching in retrieval. Finally, a dual relationship modeling module, targeting the combined attribute prototype features, uses attribute similarity to jointly model pairwise relationships (supervised semantic association) and neighborhood relationships (unsupervised semantic clustering), mining implicit semantic connections between features, optimizing the metric space learning process, enabling the retrieval system to more sensitively capture semantic matching patterns, and ultimately improving the accuracy and robustness of composite image retrieval. This includes:

[0040] The training set data is read in batches, and global and local features are extracted from the training set data. The local and global features are concatenated to form the original attribute features of visual and textual data. Semantic attribute attention is used to further extract the attribute prototype features from the original attribute features. We use the Semantic Attribute Attention (SAA) mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP. For visual and textual attribute prototype features, the attribute prototype features of each element in the multimodal query are combined by combining them based on a unified prototype and forcing alignment through a unified prototype standard. For the combined attribute prototype features, a dual-relationship model is used to model pairwise relationships and neighborhood relationships through attribute similarity, thereby optimizing the metric learning process. Specifically, this includes: supervised pairwise relationship modeling, which calculates the classification loss for the combined features and target features after average pooling; and unsupervised neighborhood relationship modeling, which uses attribute similarity evaluation to measure the attribute correlation between samples. The COMBINER model extracts and combines attribute prototype features to perform multimodal query combination on the input reference image and modified text, generating corresponding combined features. Then, it calculates the dot product of these combined features with each image in the image library to obtain a similarity score, and then sorts all similarity scores in descending order. Based on actual needs (e.g., retrieving the image that best matches the query requirements), the model then... K (Images), select the top ones K The target image is used as the formal result set to complete the combined image retrieval.

[0041] Example 2 The difference between the attribute-based neighborhood relation-guided combined image retrieval method described in Example 1 and the method described in Example 1 is as follows: The training set data was read in batches, with each batch containing data sets in the form of triplets of <reference image, modified text, target image>. Global and local features were then extracted from the training set data. The reference image was extracted using CLIP. global features and local features , represented as: ; ; in, for S OK A real matrix of columns, and These represent the global and local feature dimensions of CLIP, respectively. S The number of channels for each sample, and These represent the last and penultimate layers of the CLIP image encoder, respectively. To facilitate the concatenation of global and local attribute prototype features, a linear transformation layer is employed. Local feature dimensions Align to global features and dimensionality To form a unified dimension of features The formula is as follows: , in, Align local feature dimensions to D Obtain local features aligned with the global feature dimensions. .

[0042] The Semantic Attribute Attention (SAA) mechanism aims to extract prototypes from the original features to adaptively separate attribute prototype features; it further extracts attribute prototype features from the original attribute features using semantic attribute attention; including: Based on local or global features, semantic attribute attention is used to capture contextual semantics and adaptive separation is performed at the prototype level. The prototype weights obtained from the multilayer perceptron are used to weight and aggregate local or global features to obtain local or global attribute prototype features. The global attribute prototype features are fused with the local attribute prototype features.

[0043] For local features, semantic attribute attention is first used to capture contextual semantics, and adaptive separation is performed at the prototype level. Then, prototype weights generated by a multilayer perceptron are used to perform weighted aggregation of local features to obtain local attribute prototype features. Afterwards, global attribute prototype features are extracted through the same process and fused with the aforementioned local attribute prototype features; including:

[0044] For reference image Local features of unified feature dimension By capturing contextual semantics, adaptive separation is performed at the prototype level, as shown below: ; in, Represents the weights of the linear layer. Indicates the embedding dimension. , This represents the total number of local attribute prototypes (can be set separately for local features and global features). This represents the output of an unbiased multilayer perceptron (MLP); Tanh is the Tanh activation function. To unify the local features of the feature dimension, The transpose of the output of an unbiased multilayer perceptron (MLP), where Softmax is the Softmax activation function. For prototype weights, and use Weighted aggregation is performed to obtain the local attribute prototype features of the reference image. The formula is as follows: ; Similar to the separation process described in SAA, global attribute prototype features of the reference image are separated from the global features of the reference image. , where X is the number of global property prototypes; Will and Fusion constructs multi-granularity attribute prototype features of the reference image, denoted as... , ; Similarly, the present invention utilizes the same and Adaptively separate the multi-granularity attribute prototype features of the modified text and the target image, denoted as follows: .

[0045] For multi-granularity attribute prototype features, a cross-modal unified prototype (CUP) is extracted from simply concatenated multimodal query features using the Semantic Attribute Attention (SAA) mechanism, and coherent prototype semantics are constructed based on the CUP to unify cross-modal features, including: Perform a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to extract a cross-modal unified prototype; The extracted cross-modal unified prototype is used as the basis for combination constraints. With the help of linear learning capabilities, the prototype feature weights of the reference image and modified text are used for learning.

[0046] Perform a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to generate concatenated features. ,in, U This represents the number of attribute prototypes. D The feature dimension is used; subsequently, semantic attribute attention is invoked to separate the cross-modal unified prototype from the concatenated features, denoted as . The mathematical expression is: .

[0047] The extracted cross-modal unified prototype is used as the basis for combination constraints. Leveraging linear learning capabilities, the weights of the prototype features in the reference image and the modified prototype features are learned; the Sigmoid function is then used. Weight calculation with multilayer perceptron: ; in, , These correspond to the weights of the reference image features and the modified text features relative to the CUP, respectively; MLP stands for Multilayer Perceptron. based on and For the original attribute prototype features and Aggregation yields attribute-level combined features for multimodal queries. The formula is: ; This is an element-wise multiplication operation. This module enhances cross-modal semantic consistency through unified prototype construction and feature aggregation, laying a precise semantic foundation for subsequent retrieval tasks.

[0048] Supervised pairwise relation modeling involves calculating a classification loss for the combined features after average pooling and the target features; including: To achieve supervised pairwise relationship modeling, this invention employs batch collaborative loss, commonly used in image retrieval tasks. This loss prompts each combined feature to approximate the matching target feature in the learned metric space. The process of modeling neighborhood relationships also enables the cross-modal unified prototype to implicitly approximate the target feature.

[0049] Combined features after average pooling The calculation is as follows: ; Similarly, the target features after average pooling are calculated. ; Specifically, based on the combined features after average pooling and target features The batch-based classification loss is defined as follows. : ; in, and , represent the average pooling results of the combined features and the target features of the i-th triplet, respectively. This represents the average pooling result of the combined features and target features of the j-th triplet, where B represents the batch size. Used to calculate cosine similarity This is the temperature scaling factor.

[0050] This study employs an unsupervised neighborhood relationship modeling method, utilizing attribute similarity evaluation to measure the attribute correlation between samples. This method can simultaneously push images with unrelated attributes away from each other while bringing images with related attributes closer together, thus optimizing metric learning. The method includes: Using the K-means clustering algorithm, H semantic cluster centers are set for the prototype features of the target image after average pooling, where H is a manually set parameter. Unsupervised clustering is then performed to obtain... H Let there be semantic cluster centers, and denote the set of semantic centers as . ,in, H Indicates the number of semantic cluster centers; To evaluate attribute correlation between samples based on attribute prototypes, attribute prototype features are extracted from each sample. Then, the similarity between these attribute prototype features is calculated. Based on these similarities, K-means is used to cluster samples with similar attributes together, forming semantic clusters. The semantic cluster center corresponding to each triple serves as a common visual cluster and attribute-related neighborhood, because the multimodal query and target image in the triple are relatively close within the cluster. Through the semantic clustering process, visually similar and attribute-related images are clustered together in the metric space to facilitate learning, while attribute-irrelevant images are pushed apart. Thus, the target feature and its triples for each training triple are identified as neighborhood relation triples; the set of neighborhood relation triples is represented as... Where N is the number of training triples, It is the first i The semantic cluster centers corresponding to each triple .

[0051] To explicitly push attribute-related elements and their corresponding target images from multimodal queries to their associated semantic clustering centers, this invention employs a batch-based classification loss function. The definition is as follows: ; in, B Let s represent the number of triples in each batch, and s be the similarity function. Here, is the temperature coefficient, exp() is the exponential function, and log() is the logarithmic function.

[0052] To ensure that attribute-related multimodal queries and target images are as close as possible in the metric space, it is assumed that their similarity distributions at the cluster center level and the corresponding cluster center features should also be consistent. Inspired by this, this invention utilizes a mutual learning strategy to ensure that the similarity distributions of each multimodal query and semantic cluster center remain consistent; specifically, it is assumed that each batch contains... B The third triplet will be the first i The batch-level similarity distribution of a combination of features is defined as follows: ,in Indicates the first i The first combination feature and the first j The similarity of the cluster centers of a triplet is calculated using the following formula:

[0053] ; Similarly, the similarity distribution of target features is derived. .

[0054] The loss term for the regularization of the similarity distribution for clustering is defined using the Kullback-Leibler (KL) divergence. ,Promote and The consistency between them is shown in the following formula: ; in, Let KL divergence be the KL divergence. Let represent the similarity distribution between the i-th sample and the j-th target feature. Let represent the similarity distribution between the combined features of the i-th sample and the target feature of the j-th sample.

[0055] To alleviate the attribute semantic entanglement problem caused by average pooling of combined features and target features, this invention designs a pooling-oriented distribution consistency regularization term; specifically including: First, define and ;in, Indicates the first i The first combination feature and the first j The similarity of the average pooling target features; Indicates the first i The target feature and the first j The similarity of the average pooling target features; Then, a pooling-oriented distribution consistency regularization term is defined using KL divergence, as shown in the following formula: ; Finally, the metric space optimization function is as follows: ; in, This represents the set of learnable parameters in the COMBINER model. As a trade-off hyperparameter.

[0056] The COMBINER model uses attribute prototype feature extraction and combination operations to perform multimodal query combination on the input reference image and modified text, generating corresponding combined features; including: First, CLIP is used to extract global (last layer) and local (second to last layer) features from the reference image and modified text, and the local feature dimensions are unified through linear transformation; Then, through semantic attribute attention, the global and local attribute prototype features of the reference image and the modified text are decoupled respectively, and concatenated into multi-granular attribute prototype features with consistent dimensions. Subsequently, multi-granularity attribute prototype features from the reference image and modified text are concatenated, and cross-modal unified prototypes are decoupled using specific semantic attribute attention. Based on this, feature weights of both are calculated using a multilayer perceptron and activation functions. Finally, the combined features are generated through element-wise weighted aggregation.

[0057] like Figure 1 As shown, the COMBINER model optimizes the combined image retrieval task through three key modules. First, the adaptive semantic decoupling module uses a pre-trained model based on contrastive learning to decouple the attribute features of the reference image, modified text, and target image, helping the model clearly identify and separate different semantic information. Next, the unified prototype combination module generates a unified cross-modal representation by fusing the attribute prototypes of the reference image and modified text, overcoming the modal heterogeneity problem. Finally, the dual-relation modeling module further optimizes the similarity matching between multimodal queries and target images through supervised pairwise relations and unsupervised neighborhood relations modeling, and optimizes the aggregation of attribute-related images through clustering.

[0058] During the training of the COMBINER model, CLIP was used as the feature extraction backbone network. The number of global and local attribute prototypes was set to 4 each, resulting in 8 attribute prototypes. COMBINER was trained using the AdamW optimizer (which improves the stability and effectiveness of model training by applying weight decay and gradient updates separately) with an initial learning rate of 0.0001, while the learning rate of CLIP was set to 0.000001. All experiments were conducted on an NVIDIA A40 GPU with 48 GB of memory, with training lasting 10 epochs and a batch size of 32.

[0059] Table 1 shows the comparison of the retrieval accuracy of the present invention on the FashionIQ dataset; Table 1

[0060] In Table 1, Dresses, Shirts, and Tops & Tees are the three clothing categories from the FashionIQ dataset, and Avg is the average of the rating metrics R@10 and R@50. R@10 and R@50 represent the proportion of the target image appearing in the top 10 or 50 results returned in the search results.

[0061] Table 2 shows a comparison of the retrieval accuracy of the present invention on the Shoes dataset; Table 3 shows a comparison of the retrieval accuracy of the present invention on the CIRR dataset. Table 2

[0062] Table 3

[0063] In Tables 2 and 3, R@k (k=1, 5, 10, 50) represents the proportion of the target image appearing in the first 1, 5, 10, and 50 results, respectively. In Table 3, Rsubset@k (k=1, 2, 3), Rsubset@1, Rsubset@2, and Rsubset@3 indicate whether a relevant subset of the target image was retrieved in the first 1, 2, or 3 results.

[0064] The comparison methods involved in Tables 1, 2, and 3 include: Published at ACM MM 2023, TG-CIR models the conflict between objectives and queries using a "hold and replace" strategy and captures complex user intent by leveraging teacher-student combined branches, achieving strong performance across multiple benchmarks.

[0065] SADN, published in ACMMM in 2024, addresses the issues of false negatives and subtle semantic differences in CIR by leveraging neighborhood-based semantic relevance and discriminative modeling.

[0066] CoVR-2 was published at TPAMI in 2024 and has also been evaluated for CIR performance on fashion and open-domain datasets. It also shows great potential for performance in CVR tasks.

[0067] Candidate, published in TMLR in 2024, proposed a two-stage CIR pipeline that efficiently filters out easy negative samples based on visual similarity and explicitly reorders the remaining difficult negative samples using fine-grained text-image interaction modeling.

[0068] SPRC, published at ICLR 2024, proposed a novel CIR method based on sentence-level hints. It utilizes a lightweight query Transformer (Q-Former) to generate expressive hints by jointly modeling reference images and relative descriptions.

[0069] LIMN and LIMN+ were published at TPAMI in 2024. LIMN is based on CLIP-Transformer and solves the problem of existing CIR methods ignoring multiple matching factors through latent factor token mining, dual aggregation matching token learning, and dual query-target matching modeling. LIMN+ further designs an iterative dual self-training paradigm and uses unlabeled reference-target images to improve the model's generalization ability.

[0070] DQU-CIR, published at SIGIR in 2024, transforms multimodal fusion from the feature level to the raw data level. With the help of image and text encoders of the VLP model, it generates unified text queries and unified visual queries through two untrained query unification components, and then linearly combines the encoded features of the two to adapt to diverse search intentions, thereby achieving target image retrieval.

[0071] Published at TOIS 2025 by IUDC, this paper proposes an enhanced dual-channel matching framework based on a large-scale language model (LLM) that effectively performs semantic reasoning through multi-level attribute extraction and performs fine-grained matching of multimodal embeddings.

[0072] DIPNEC, published at AAAI in 2025, introduces optimal transmission with semantic preservation to redistribute hard negative triples, as well as differential quantization alignment (DQA) and combinatorial word alignment (CWA). DQA focuses on the correlation between text features and visual differences from source to target, while CWA further emphasizes differential semantics.

[0073] ENCODER, published at AAAI 2025, explores modal-shared learnable relation queries between visual entities and modified actions through entity-action binding, and learns implicit modification relations of entity-action binding to improve combined image retrieval.

[0074] Published in TMLR in 2024, Prog. Lrn. adapts the model to the CTI-IR task in three stages: leveraging the open-domain semantic embedding space, transferring knowledge through fashion pre-training, and enhancing the model to mixed-modal queries; and proposing a self-supervised adaptive weighting strategy to dynamically determine modal importance.

[0075] CRN, published in TIP in 2024, introduced a cross-relation network for CIR, which significantly reduced computational complexity by dynamically balancing retrieval strategies between modified and auxiliary query texts through relevance modeling and shared feature extractors.

[0076] AFCE, published in TIP in 2024, improves the learning objective from the perspective of multimodal fusion and constructs a method for mining difficult negative examples: it combines reference images with unpaired sentences to generate component-level negative examples, and also proposes a sentence enhancement method to generate element-level difficult-to-distinguish negative examples.

[0077] SSN, published at AAAI 2024, reconstructs CIR by using a text input frame as instructions to describe semantic changes from a reference image to a target image, and explicitly decomposes it into degradation and upgrade components.

[0078] As shown in Tables 1-3, the comparison results of the query efficiency and retrieval accuracy of the COMBINER model of this invention with internationally leading methods show that its retrieval accuracy is superior to similar combined image retrieval methods on three mainstream benchmark datasets, demonstrating significant performance advantages.

[0079] Example 3 A computer device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, enables the implementation of the steps of the attribute-based neighborhood relation-guided combined image retrieval method described in Embodiment 1 or 2.

[0080] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the attribute-based neighborhood relation-guided combined image retrieval method described in Embodiment 1 or 2.

[0081] Example 5 Attribute-based neighborhood relation-guided combined image retrieval systems include: The adaptive semantic separation module is configured to: read training set data in batches and extract global and local features from the training set data; concatenate the local and global features to form the original attribute features of visual and textual data; and further extract the prototype features of visual and textual attributes from the original features using semantic attribute attention. The unified prototype-based combination module is configured to: use the Semantic Attribute Attention (SAA) mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP; for attribute prototype features, the unified prototype-based combination module is combined to force alignment through a unified prototype standard to combine the attribute prototypes of each element in the multimodal query. The dual relation modeling module is configured to: for the combined attribute prototype features, combine the dual relation modeling module to model the pairwise relations and neighborhood relations through attribute similarity, thereby optimizing the metric learning process; The image retrieval module is configured as follows: The COMBINER model performs multimodal query combination on the input reference image and modified text through attribute prototype feature extraction and combination operations, generating corresponding combined features; subsequently, the dot product of this combined feature with each image in the image library is calculated to obtain a similarity score, and all similarity scores are then sorted in descending order; based on actual needs (e.g., retrieving the image that best matches the query requirements), the module retrieves the image. K (Images), select the top ones K The target image is used as the formal result set to complete the combined image retrieval.

Claims

1. A combined image retrieval method guided by attribute-based neighborhood relationships, characterized in that, include: The training set data is read in batches, and global and local features are extracted from the training set data. The local and global features are concatenated to form the original attribute features of visual and textual data. Semantic attribute attention is used to further extract the attribute prototype features from the original attribute features. We use a semantic attribute attention mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP. For visual and textual attribute prototype features, the attribute prototype features of each element in the multimodal query are combined by combining them based on a unified prototype and forcing alignment through a unified prototype standard. For the combined attribute prototype features, a dual-relationship model is used to model pairwise relationships and neighborhood relationships through attribute similarity, thereby optimizing the metric learning process. Specifically, this includes: supervised pairwise relationship modeling, which calculates the classification loss for the combined features and target features after average pooling; and unsupervised neighborhood relationship modeling, which uses attribute similarity evaluation to measure the attribute correlation between samples. The COMBINER model extracts and combines attribute prototype features, performing multimodal queries on the input reference image and modified text to generate corresponding combined features. Then, it calculates the dot product of these combined features with each image in the image library to obtain a similarity score. All similarity scores are then sorted in descending order. Based on actual needs, the top-ranked features are selected. K The target image is used as the formal result set to complete the combined image retrieval.

2. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, The training set data was read in batches, with each batch containing data sets in the form of <reference image, modified text, target image> triples. Global and local features were then extracted from the training set data. The reference image was extracted using CLIP. global features and local features , represented as: ； ； in, for S OK A real matrix of columns, and These represent the global and local feature dimensions of CLIP, respectively. S The number of channels for each sample, and These represent the last and penultimate layers of the CLIP image encoder, respectively. Employing a linear transformation layer Local feature dimensions Align to global features and dimensionality To form a unified dimension of features The formula is as follows: ， in, Align local feature dimensions to D Obtain local features aligned with the global feature dimensions. .

3. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, Semantic attribute attention is used to further extract attribute prototype features from the original attribute features; including: Based on local or global features, semantic attribute attention is used to capture contextual semantics and adaptive separation is performed at the prototype level. The prototype weights obtained from the multilayer perceptron are used to weight and aggregate local or global features to obtain local or global attribute prototype features. Fuse global attribute prototype features with local attribute prototype features; Further preferred, for the reference image Local features of unified feature dimension By capturing contextual semantics, adaptive separation is performed at the prototype level, as shown below: ； in, Represents the weights of the linear layer. Indicates the embedding dimension. , Indicates the total number of prototypes for local attributes. This represents the output of an unbiased multilayer perceptron; Tanh is the Tanh activation function. To unify the local features of the feature dimension, The transpose of the output of an unbiased multilayer perceptron, where Softmax is the Softmax activation function. For prototype weights, and use Weighted aggregation is performed to obtain the local attribute prototype features of the reference image. The formula is as follows: ； Extract the global attribute prototype features of the reference image from the global features of the reference image. , where X is the number of global property prototypes; Will and Fusion constructs multi-granularity attribute prototype features of the reference image, denoted as... , ; Separate the multi-granularity attribute prototype features of the modified text and the target image, denoted as follows: .

4. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, For multi-granularity attribute prototype features, a semantic attribute attention mechanism is used to separate the cross-modal unified prototype (CUP) from simply concatenated multimodal query features, and a coherent prototype semantics is constructed based on the CUP; including: Perform a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to extract a cross-modal unified prototype; The extracted cross-modal unified prototype is used as the basis for combination constraints. With the help of linear learning capabilities, the prototype feature weights of the image and modified text are used for learning. A further preferred approach involves performing a concatenation operation on the multi-granularity attribute prototype features of the reference image and the modified text to generate concatenated features. ,in, U This represents the number of attribute prototypes. D The feature dimension is used; subsequently, semantic attribute attention is invoked to separate the cross-modal unified prototype from the concatenated features, denoted as . The mathematical expression is: ; Further optimized, the extracted cross-modal unified prototype is used as the basis for combination constraints. Leveraging linear learning capabilities, the weights of the prototype features in the reference image and the modified prototype features are learned; this is achieved through the Sigmoid function. Weight calculation with multilayer perceptron: ； in, , These correspond to the weights of the reference image features and the modified text features relative to the CUP, respectively; MLP stands for Multilayer Perceptron. based on and For the original attribute prototype features and Aggregation yields attribute-level combined features for multimodal queries. The formula is: ; This is an element-wise multiplication operation.

5. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, Supervised pairwise relation modeling involves calculating a classification loss for the combined features after average pooling and the target features; including: Combined features after average pooling The calculation is as follows: ； Calculate the target features after average pooling ; Based on the combined features after average pooling and target features The batch-based classification loss is defined as follows. : ； in, and , represent the average pooling results of the combined features and the target features of the i-th triplet, respectively. This represents the average pooling result of the combined features and target features of the j-th triplet, where B represents the batch size. Used to calculate cosine similarity This is the temperature scaling factor.

6. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, This study utilizes unsupervised neighborhood relationship modeling methods and attribute similarity assessment to measure the attribute correlation between samples; including: Using the K-means clustering algorithm, H semantic cluster centers are set for the prototype features of the target image after average pooling, where H is a manually set parameter. Unsupervised clustering is then performed to obtain... H Let there be semantic cluster centers, and denote the set of semantic centers as . ,in, H Indicates the number of semantic cluster centers; For each triple, the semantic clustering center is used as a common visual cluster and attribute-related neighborhood. The target features of each training triple and its triples are identified as neighborhood relation triples; the set of neighborhood relation triples is represented as follows: Where N is the number of training triples, It is the first i The semantic cluster centers corresponding to each triple .

7. The attribute-based neighborhood relation-guided combined image retrieval method according to claim 1, characterized in that, Using a batch-based classification loss function The definition is as follows: ； in, B Let s represent the number of triples in each batch, and s be the similarity function. Here, is the temperature coefficient, exp() is the exponential function, and log() is the logarithmic function; A further optimized approach utilizes a mutual learning strategy to ensure that the similarity distribution between each multimodal query and the semantic clustering center remains consistent; specifically, assuming that each batch contains... B The third triplet will be the first i The batch-level similarity distribution of a combination of features is defined as follows: ,in Indicates the first i The first combination feature and the first j The similarity of the cluster centers of a triplet is calculated using the following formula: ； Derive the similarity distribution of target features ; A further preferred approach is to define the loss term for the similarity distribution regularization term oriented towards clustering using the Kullback-Leibler divergence. ,Promote and The consistency between them is shown in the following formula: ； in, Let KL divergence be the KL divergence. Let represent the similarity distribution between the i-th sample and the j-th target feature. Let represent the similarity distribution between the combined features of the i-th sample and the target feature of the j-th sample; Further optimized, a pooling-oriented distribution consistency regularization term is designed; specifically including: First, define and ;in, Indicates the first i The first combination feature and the first j The similarity of the average pooling target features; Indicates the first i The target feature and the first j The similarity of the average pooling target features; Then, a pooling-oriented distribution consistency regularization term is defined using KL divergence, as shown in the following formula: ； Finally, the metric space optimization function is as follows: ； in, This represents the set of learnable parameters in the COMBINER model. As a trade-off hyperparameter.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it is able to implement the steps of the attribute-based neighborhood relation-guided combined image retrieval method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the attribute-based neighborhood relation-guided combined image retrieval method according to any one of claims 1-7.

10. A composite image retrieval system guided by attribute-based neighborhood relationships, characterized in that, include: The adaptive semantic separation module is configured to: read training set data in batches and extract global and local features from the training set data; concatenate the local and global features to form the original attribute features of visual and textual data; and further extract the prototype features of visual and textual attributes from the original features using semantic attribute attention. The unified prototype-based combination module is configured to: use a semantic attribute attention mechanism to separate the cross-modal unified prototype CUP from the simple concatenation of multimodal query features, and construct coherent prototype semantics based on the CUP; for attribute prototype features, the unified prototype-based combination module is combined to force alignment through a unified prototype standard to combine the attribute prototypes of each element in the multimodal query. The dual relation modeling module is configured to: for the combined attribute prototype features, combine the dual relation modeling module to model the pairwise relations and neighborhood relations through attribute similarity, thereby optimizing the metric learning process; The image retrieval module is configured as follows: The COMBINER model performs multimodal query combination on the input reference image and modified text through attribute prototype feature extraction and combination operations, generating corresponding combined features; subsequently, the dot product of this combined feature with each image in the image library is calculated to obtain a similarity score, and all similarity scores are then sorted in descending order; based on actual needs, the top-ranked images are selected. K The target image is used as the formal result set to complete the combined image retrieval.

Images