A combined image retrieval method and system based on entity mining and modified relationship binding

Abstract

The application relates to a combined image retrieval method and system based on entity mining and modification relation binding, which comprises the following steps: reading training set data in batches, and extracting global features and local features of the training set data; filtering the local features through latent factors, and further extracting visual and text latent factor features related to modification semantics; for the latent factor features, combining entity-action binding, deeply mining semantic relations in reference images and modification texts, detecting semantic boundaries, and respectively aggregating the latent factors into visual entities and modification actions; for the obtained features of different scales, obtaining final combined features through multi-scale combination; calculating dot products of the combined features and different images in a gallery respectively as similarity scores, arranging the similarity scores in descending order, selecting target images with top several similarity scores, and completing combined image retrieval. The application realizes effective retrieval of a user target image.

Description

Technical Field

[0001] This invention relates to a method and system for querying, understanding, and retrieving similarity data of e-commerce images, and more particularly to a combined image retrieval method and system based on entity mining and modification of relationship binding, belonging to the technical field of multimodal information retrieval. Background Technology

[0002] In the context of the rapid development of e-commerce, the effective combination of product images and text descriptions has become increasingly important. Users often want to modify existing reference images, such as adjusting colors, adding or deleting elements, or combining different products. In such cases, traditional image retrieval cannot meet users' personalized needs. Therefore, combined image retrieval tasks have emerged, aiming to meet users' complex personalized needs in a flexible way by integrating multimodal information (images and text), and have attracted widespread attention from researchers.

[0003] Specifically, the combined image retrieval task retrieves the target image required by the user from a database based on a reference image and modified text input by the user. The key to this task lies in accurately identifying the modification requirement and locating the corresponding entity in the reference image that needs modification. For ease of description, the visual entity in the reference image and the modification action in the modified text will be referred to as "entity" and "action," respectively. Although some researchers have attempted to effectively retrieve target images based on cross-modal semantic alignment, they have failed to fully consider the inherent semantic asymmetry between the reference image and the modified text; that is, the entity contained in the action may not have a direct semantic correspondence in the reference image, resulting in room for improvement in retrieval performance. Therefore, to construct an effective combined image retrieval method, it is necessary to consider not only the semantic alignment between visual data and text data but also the binding of modification relationships. Currently, achieving the binding of modification relationships between entities and actions to construct an effective combined image retrieval method faces the following three main challenges:

[0004] (1) Irrelevant Factor Perturbation. Before binding modification relationships, it is necessary to first mine entities and actions. However, not all words in the text and not all regions in the image are directly related to the modification requirements. Irrelevant words and visual regions can affect the mining of entities and actions. Therefore, it is very challenging to identify the visual and textual semantics related to the modification behavior and eliminate the interference of irrelevant factors.

[0005] (2) Fuzzy semantic boundaries. Actions are often hidden in different word combinations within the modified text, and there are no clear supervisory signals to achieve accurate segmentation. Furthermore, due to the irregular shapes of entities, their boundaries are also difficult to identify. Therefore, identifying semantic boundaries to mine entities and actions is equally challenging.

[0006] (3) Implicit Modification Relationships. Since there may be no semantic match between entities and actions, it is challenging to measure the correspondence of modification relationships solely based on feature similarity, and direct supervision signals are lacking. Therefore, identifying modification relationships and binding entities to corresponding operations is very challenging. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a combined image retrieval method based on entity mining and modified relationship binding, enabling effective retrieval of user target images.

[0008] This invention also proposes a combined image retrieval system based on entity mining and modified relationship binding.

[0009] To this end, this invention first proposes a latent factor filtering module to calculate cross-modal semantic relevance and filter out visual and textual latent factors related to modification semantics. Second, this invention proposes an entity-action binding module to study the semantic boundaries of visual and textual latent factors, aggregating them into visual entities and modification actions respectively, while learning the implicit modification relationships between them to achieve entity-action binding. Finally, this invention proposes a multi-scale combination module to construct multi-modal query features at multiple scales, guided by entity-action binding relationships, to achieve effective retrieval of user target images.

[0010] Terminology Explanation:

[0011] 1. CLIP is a deep learning model designed to combine textual and image information through contrastive learning to achieve cross-modal understanding. CLIP is pre-trained on large-scale image-text datasets and can be used for various tasks such as image retrieval, text generation, and image classification. Its ability to learn cross-modal features makes it perform well in processing visual and linguistic information.

[0012] 2. Attention mechanism is a computational method used in deep learning models, which aims to dynamically focus on specific parts of the input data to improve the efficiency and effectiveness of information processing.

[0013] 3. Multilayer Perceptron (MLP) is a type of feedforward neural network consisting of at least three layers: an input layer, a hidden layer, and an output layer. Each layer consists of multiple neurons connected by weights. It is widely used in tasks such as classification, regression, and feature extraction, and has shown superior performance in multiple fields such as pattern recognition, image processing, and natural language processing.

[0014] 4. The Softmax function is an activation function widely used in the output layer of multi-class classification problems. Its main function is to transform the input real-valued vector into a probability distribution, such that each output value is between 0 and 1, and the sum of all output values ​​is 1.

[0015] 5. Average Pooling is a downsampling technique widely used in convolutional neural networks. It aims to reduce the spatial dimension of data and extract important features by averaging the input feature maps across regions.

[0016] 6. An identity matrix is ​​a special type of square matrix whose main characteristic is that all elements on the diagonal are 1, while all elements on the off-diagonal are 0. It is usually represented as I. n , where n represents the dimension of the matrix.

[0017] 7. The Frobenius norm is a mathematical metric used to measure the size or "length" of a matrix. It is defined as the square root of the sum of the squares of all the elements in the matrix.

[0018] 8. KL divergence is an asymmetric metric that measures the difference between two probability distributions. Specifically, KL divergence is used to assess the information loss or relative entropy of one probability distribution relative to another.

[0019] The technical solution of the present invention is as follows:

[0020] Image retrieval methods based on a combination of entity mining and modified relation binding include:

[0021] The training set data is read in batches, and global and local features are extracted from the training set data. The local features are filtered by latent factors to further extract visual and textual latent factor features related to the semantic modification.

[0022] For latent factor features, combined with entity-action binding, we delve into the semantic relationships in the reference image and modified text, detect semantic boundaries, and aggregate latent factors into visual entities and modification actions respectively.

[0023] The features obtained at different scales are combined through multi-scale analysis to obtain the final combined features.

[0024] The combined features are used to calculate the dot product with different images in the image library, which is then used as the similarity score. The similarity scores are sorted in descending order, and the top-ranked target images are selected to complete the combined image retrieval.

[0025] As a further preferred approach, local features are filtered through latent factors to further extract visual and textual latent factor features related to the semantic modification; including:

[0026] Based on local features, the cross-relationship score between each visual and textual factor is calculated through cross-modal cross-attention and intra-modal self-attention; intra-modal self-attention score is also calculated.

[0027] By combining self-attention scores and cross-relationship scores, potential and shelved factors in visual and textual contexts are distinguished.

[0028] Adaptive weights for latent factors are obtained using a multilayer perceptron to enhance and modify the weights of factors that are more semantically relevant.

[0029] Further preferably, based on local features, the cross-relationship score between each visual factor and text factor is calculated through cross-modal cross-attention and intra-modal self-attention; simultaneously, the intra-modal self-attention score is also calculated; including:

[0030] Based on local features of the reference image, the intermodal crossover score w is calculated. c and intramodal self-attention score w s , means as follows:

[0031]

[0032] Among them, w c w s ∈R C Inter-CA stands for intermodal cross-attention, and Intra-SA stands for intramodal self-attention. Q and K are feature vectors used to calculate attention weights, and V is a vector representing the input features. This refers to modifying local features of the text; Refers to the reference image x r Local features.

[0033] Further preferred methods combine self-attention scores and cross-relationship scores to distinguish between latent and shelved factors in visual and textual contexts; including:

[0034] Combining cross-relationship score w c and self-attention score w s Factors exceeding a specified threshold σ are retained as potential factors for subsequent entity-action bindings, while factors below the specified threshold σ are designated as shelved factors.

[0035] A further preferred embodiment utilizes a multilayer perceptron to obtain adaptive weights for latent factors; including:

[0036] Adaptive weights for latent factors are obtained using a multilayer perceptron and the Softmax function. These weights are then used to enhance the latent factors in the reference image. The final representation is as follows:

[0037]

[0038] Among them, w # ∈R 0×2 , where represents the factor weight, and P represents the number of potential factors to be learned. This refers to the potential factors that could modify the text. This refers to the shelving factors in modifying the text. Softmax(·) is the activation function, and MLP(·) is a multilayer perceptron. In addition, average pooling is applied to the shelving factors. Similarly, the final latent factor features of the modified text are obtained. and the final latent factor features of the target image

[0039] As a further preferred approach, for latent factor features, combined with entity-action binding, the semantic relationships in the reference image and modified text are deeply mined to detect semantic boundaries, and latent factors are aggregated into visual entities and modification actions respectively; including:

[0040] Learnable relation queries with shared modalities are trained to mine semantic relationships in reference images and modified text, detect semantic boundaries, and aggregate their potential factors into visual entities and modification actions, respectively.

[0041] By using learnable relation queries, the implicit modification relationship between visual entities and modification actions is learned, serving as a medium to assist entity-action binding.

[0042] Further preferredly, for latent factor features, combined with entity-action binding, the semantic relationships in the reference image and modified text are deeply mined to detect semantic boundaries, and latent factors are aggregated into visual entities and modification actions respectively; including:

[0043] Initialize a learnable relation query R∈R 8×3 Where E is the number of queries and D is the global embedding dimension of CLIP;

[0044] First, perform interactions between learnable relation queries and calculate the self-attention weight e. s It also calculates the cross-attention weights between learnable relation queries and latent factors of different modalities. and It is expressed as follows:

[0045]

[0046] Among them, e s∈R 8×8 , Cross-attention weights are calculated for latent factors in the learnable relation query and the reference image. Cross-attention weights are calculated for potential factors in learnable relational queries and text modifications;

[0047] Based on e s , The adaptive attention weights for latent factors in the reference image and latent factors in the modified text in the learnable relation query are respectively... and

[0048] e r and e m Each latent factor is assigned a weight, and a multilayer perceptron is used to adaptively learn the weights, thereby mapping the latent factors to corresponding tokens. Each token matches a corresponding query of a learnable relation, as shown below:

[0049]

[0050] in, and These correspond to physical tokens and action tokens, respectively.

[0051] For learnable relational queries, an orthogonalized loss function is bound to the design. It is expressed as follows:

[0052]

[0053] Where, I∈R 8×8 , Let Frobenius norm be the matrix;

[0054] Calculate the similarity distribution of visual entities and modification actions using learnable relation queries respectively; let Let represent the similarity distribution between the i-th token in the entity token set and the learnable relation query, where It is expressed as follows:

[0055]

[0056] Where s(·) represents the cosine similarity, E rL R Q Represents the i-th entity token and the j-th query in the learnable relation query, τ is the temperature coefficient, E represents the number of triples, and R A This represents the e-th query in the learnable relation query; similarly, we obtain the similarity distribution between each token in the action tokens and each query in the learnable relation query, denoted as . in, This represents the similarity between the i-th action token and the j-th learnable relation query;

[0057] Calculate the KL divergence loss function between two similarity distributions. By binding the corresponding entity and action, it can be represented as follows:

[0058]

[0059] in, Let D be the KL divergence loss function. 2` (·) represents the KL divergence. This represents the similarity between the i-th target image token and the j-th learnable relation query. This represents the similarity between the i-th action token and each learnable relation query. This represents the similarity between the i-th entity token and each learnable relation query.

[0060] As a further preferred approach, the obtained features at different scales are combined through multi-scale analysis to obtain the final combined features; including:

[0061] Multi-scale features from the reference image and modified text are integrated separately;

[0062] Based on the obtained multi-scale features, multi-scale interaction is performed via a multilayer perceptron to learn the modification weights of the reference image and the modified text.

[0063] Based on the learned modified weights, the final combined features are obtained.

[0064] As a further preferred approach, the obtained features at different scales are combined through multi-scale analysis to obtain the final combined features; including:

[0065] The corresponding global features, local features, latent factors, and shelved factors are concatenated; for the reference image, the multi-scale features are represented as follows: Q = 1 + P + E + 1; similarly, the multi-scale features of the modified text are obtained. and multi-scale features of the target image Q m =1 + P + 1;

[0066] Using multilayer perceptrons in E r and E m Multi-scale interactions are performed to learn the modification weights of the reference image and the modified text, as shown below:

[0067] W = MLP([E r E m ])

[0068] Where W∈Ri×J3 The block operation is used to divide W into W0. r ∈R i×3 and W m ∈R i×3 , as the reference image and the weights of the modified text;

[0069] The modified weights are respectively compared with the multi-scale features E of the corresponding reference image and modified text. r E m The aggregation is performed to obtain the aggregated reference image features W. r E r and modifying text features W m E m Then for W r E r and W m E m Summing yields the final multimodal combined feature E. c , means as follows:

[0070] E c =W r E r +W m E m ,

[0071] Among them, E c ∈Q×D,E r E represents the multi-scale features of the reference image. m To modify the multi-scale weights of the text;

[0072] Using a batch-based classification loss, the combined features are approximated to the target image features. The batch-based classification loss is represented as follows:

[0073]

[0074] in, E represents the value corresponding to the i-th triplet. c The result after E6 average pooling, where B represents the batch size and s(·) represents the cosine similarity;

[0075] make Let be the similarity distribution of the i-th combined feature, where is the similarity with the j-th target image. The calculation is as follows:

[0076]

[0077] Similarly, the similarity distribution between the features of the i-th target image and other target images in the batch is obtained, denoted as . Then, KL divergence is used to converge these two similar distributions, and the combined feature space is optimized as follows:

[0078]

[0079] The final objective function is as follows:

[0080]

[0081] Where, Θ * Let κ, τ, and μ be the parameters to be optimized in the model ENCODER, and let μ be the trade-off hyperparameters.

[0082] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a combined image retrieval method based on entity mining and modified relationship binding.

[0083] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a combined image retrieval method based on entity mining and modification of relation binding.

[0084] A combined image retrieval system based on entity mining and modified relationship binding includes:

[0085] The latent factor filtering module is configured to: read training set data in batches and extract global and local features from the training set data; and further extract visual and textual latent factor features related to the modified semantics by filtering the local features through latent factor filtering.

[0086] The entity-action binding module is configured to: for latent factor features, combine entity-action binding to deeply mine the semantic relationships in the reference image and modified text, detect semantic boundaries, and aggregate latent factors into visual entities and modification actions respectively;

[0087] The multi-scale combination module is configured to combine features of different scales to obtain the final combined features.

[0088] The image retrieval module is configured to: calculate the dot product between the combined features and different images in the image library, use the product as a similarity score, sort the similarity scores in descending order, select the top-ranked target images, and complete the combined image retrieval.

[0089] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0090] 1. This invention proposes a latent factor filtering module, which helps to filter latent visual and textual factors, and eliminates interference from irrelevant factors as much as possible in the process of identifying and modifying visual and textual semantics related to the behavior.

[0091] 2. This invention proposes an entity-action binding module, aiming to deeply explore the relationship between entities and actions and detect semantic boundaries. Facing the challenge of implicit modification relationships, this module achieves effective binding between entities and actions, further improving the accuracy of combined image retrieval.

[0092] 3. This invention proposes a multi-scale combination module that integrates multi-scale features from reference images and modified text to generate final combined features. Guided by entity-action binding, this module successfully enhances the multi-scale semantic perception of multi-modal combined features, thereby further improving the accuracy of combined image retrieval. Attached Figure Description

[0093] Figure 1 This is a flowchart illustrating the combined image retrieval method based on entity mining and modified relationship binding of the present invention; Detailed Implementation

[0094] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0095] Example 1

[0096] Image retrieval methods based on a combination of entity mining and modified relationship binding, such as Figure 1 As shown, a combined image retrieval method, ENCODER, based on entity mining and modification relationship binding, is constructed to achieve combined image retrieval for multimodal queries. This method first filters latent factors based on the semantics of the multimodal query. Then, based on these latent factors, it studies the semantic boundaries of visual and textual latent factors, aggregating them into visual entities and modification actions respectively. Simultaneously, it learns the implicit modification relationships between them. Finally, based on the entity-action binding relationship, it constructs multimodal combined features at multiple scales, and uses these combined features to collaboratively filter the candidate target image set, obtaining the final result set and achieving combined image retrieval. Specifically, it includes:

[0097] S1: Latent Factor Filtering: Read the training set data in batches and extract global and local features from the training set data; filter the local features through latent factors to further extract visual and textual latent factor features related to the semantic modification.

[0098] S2: Entity-Action Relationship Binding: For latent factor features, combine entity-action binding to deeply explore the semantic relationships in reference images and modified text, detect semantic boundaries, and aggregate latent factors into visual entities and modification actions respectively;

[0099] S3: Multi-scale feature combination: For features obtained at different scales, the final combined features are obtained through multi-scale combination.

[0100] Finally, in the inference stage, the ENCODER model of this invention (i.e., the combined image retrieval method of this invention) performs multimodal query combination on the input reference image and modified text through latent factor filtering, entity-action relationship binding, and multi-scale feature combination to obtain combined features. The combined features are then used to calculate the dot product with different images in the image library to obtain similarity scores. The similarity scores are then sorted in descending order. Based on the needs (e.g., if it is necessary to retrieve the K images that best meet the query requirements), the top (K) target images with the highest similarity scores are selected (i.e., the formal result set), thus completing the combined image retrieval.

[0101] Example 2

[0102] The difference between the combined image retrieval method based on entity mining and modified relationship binding described in Example 1 is as follows:

[0103] In this method, the training set data is read in batches, and global and local features are extracted from the training set data; including:

[0104] The training set data is read in batches; each data set in each batch is in the form of a triplet: <reference image, modified text, target image>; the reference image x is extracted using CLIP. r global features and local features Represented as:

[0105]

[0106] in, and These represent the last and penultimate layers of the CLIP image encoder, respectively, where D is the CLIP global embedding dimension, and FC is the final layer. I Align the local embedding dimension to D; similarly, obtain the global features of the target image. and local features With modifying global features of text and local features Where C and S represent the number of channels for the image and text, respectively.

[0107] In this method, local features are filtered through latent factors to further extract visual and textual latent factor features related to the semantic modification; including:

[0108] Based on local features, the cross-relationship score between each visual and textual factor is calculated through cross-modal cross-attention and intra-modal self-attention; intra-modal self-attention score is also calculated.

[0109] By combining self-attention scores and cross-relationship scores, potential and shelved factors in visual and textual contexts are distinguished.

[0110] Adaptive weights for latent factors are obtained using a multilayer perceptron to enhance and modify the weights of factors that are more semantically relevant.

[0111] To fully leverage cross-modal semantic association, based on local features, a cross-relationship score between each visual and textual factor is calculated through cross-modal cross-attention and intra-modal self-attention; simultaneously, an intra-modal self-attention score is also calculated; including:

[0112] Taking a reference image as an example, the intermodal crossover score w is calculated based on the local features of the reference image as the primary feature and the local features of the text as a secondary feature. c and intramodal self-attention score w s , means as follows:

[0113]

[0114] Among them, w c w s ∈R C Inter-CA stands for intermodal cross-attention, and Intra-SA stands for intramodal self-attention. Q and K are feature vectors used to calculate attention weights, and V is a vector representing the input features. Both are derived from the input features. This refers to modifying local features of the text; Refers to the reference image x r Local features.

[0115] When modifying text, the primary focus should be on modifying the local features of the text, with the local features of the reference image serving as a secondary consideration. For the target image, however, all calculations should be performed using its local features.

[0116] By combining self-attention scores and cross-relationship scores, latent and shelved factors in visual and textual data are distinguished; including:

[0117] Combining cross-relationship score w c and self-attention score w sThis invention implements a threshold gating mechanism that retains factors exceeding a specified threshold σ (based on experimental experience, σ = 0.6) as potential factors for subsequent entity-action binding. Simultaneously, factors below the specified threshold σ are retained as shelved factors because they may appear as reserved regions in the target image. Taking a reference image as an example, its potential factors... and shelving factors The formalized expression is as follows:

[0118]

[0119] Similarly, we can identify the potential factors for modifying the text. and shelving factors Potential factors related to the target image and shelving factors

[0120] Adaptive weights for latent factors are obtained using a multilayer perceptron; including:

[0121] By employing a multilayer perceptron and the Softmax function, adaptive weights are obtained for latent factors. This enhances the weights of factors more relevant to semantic modification, easing the strict boundaries of latent factors and preventing some shelved factors from being incorrectly classified as latent factors. The enhanced latent factors are shown in the reference image. The final representation is as follows:

[0122]

[0123] Among them, w # ∈R 0×2 , where represents the factor weight, and P represents the number of potential factors to be learned. This refers to the potential factors that could modify the text. This refers to the shelved factors in the modified text. Softmax(·) is the activation function, and MLP(·) is a multilayer perceptron. Furthermore, to preserve the semantic information of the shelved factors without distracting from the latent factors, average pooling is applied to them. Similarly, the final latent factor features of the modified text are obtained. and the final latent factor features of the target image

[0124] For latent factor features, combined with entity-action binding, we delve deeper into the semantic relationships in the reference image and modified text, probe semantic boundaries, and aggregate latent factors into visual entities and modification actions, respectively; including:

[0125] Learnable relation queries with shared modalities are trained to mine semantic relationships in reference images and modified text, detect semantic boundaries, and aggregate their potential factors into visual entities and modification actions, respectively.

[0126] By using learnable relation queries, the implicit modification relationship between visual entities and modification actions is learned, serving as a medium to assist entity-action binding.

[0127] For latent factor features, combined with entity-action binding, we delve deeper into the semantic relationships in the reference image and modified text, probe semantic boundaries, and aggregate latent factors into visual entities and modification actions, respectively; including:

[0128] Initialize a learnable relation query R∈R 8×3 Where E is the number of queries and D is the global embedding dimension of CLIP;

[0129] First, perform interactions between learnable relation queries and calculate the self-attention weight e. s It also calculates the cross-attention weights between learnable relation queries and latent factors of different modalities. and It is expressed as follows:

[0130]

[0131] Among them, e s ∈R 8×8 , Cross-attention weights are calculated for latent factors in the learnable relation query and the reference image. Cross-attention weights are calculated for potential factors in learnable relational queries and text modifications;

[0132] Based on e s , The adaptive attention weights for latent factors in the reference image and latent factors in the modified text in the learnable relation query are respectively... and

[0133] e r and e m Each latent factor is assigned a weight, and a multilayer perceptron is used to adaptively learn the weights, thereby mapping the latent factors to corresponding tokens. Each token matches a corresponding query of a learnable relation, as shown below:

[0134]

[0135] in, and These correspond to physical tokens and action tokens, respectively.

[0136] To ensure semantic independence and act as a medium for implicit modification of entity-action relationship bindings, a binding orthogonalization loss function is designed for learnable relation queries. It is expressed as follows:

[0137]

[0138] Where, I∈R 8×8 , Let Frobenius norm be the matrix;

[0139] Calculate the similarity distribution of visual entities and modification actions using learnable relation queries respectively; let Let represent the similarity distribution between the i-th token in the entity token set and the learnable relation query, where It is expressed as follows:

[0140]

[0141] Where s(·) represents the cosine similarity, E rL R Q Represents the i-th entity token and the j-th query in the learnable relation query, τ is the temperature coefficient, E represents the number of triples, and R A This represents the e-th query in the learnable relation query; similarly, we obtain the similarity distribution between each token in the action tokens and each query in the learnable relation query, denoted as . in, This represents the similarity between the i-th action token and the j-th learnable relation query;

[0142] Calculate the KL divergence loss function between two similarity distributions. By binding the corresponding entity and action, it can be represented as follows:

[0143]

[0144] in, Let D be the KL divergence loss function. 2` (·) represents the KL divergence. This represents the similarity between the i-th target image token and the j-th learnable relation query. This represents the similarity between the i-th action token and each learnable relation query. This represents the similarity between the i-th entity token and each learnable relation query.

[0145] In this method, features obtained at different scales are combined through multi-scale analysis to obtain the final combined features; including:

[0146] Multi-scale features from the reference image and modified text are integrated separately;

[0147] Based on the obtained multi-scale features, multi-scale interaction is performed via a multilayer perceptron to learn the modification weights of the reference image and the modified text.

[0148] Based on the learned modified weights, the final combined features are obtained.

[0149] The features obtained at different scales are combined through multi-scale analysis to obtain the final combined features, including:

[0150] The corresponding global features, local features, latent factors, and shelved factors are concatenated; for the reference image, the multi-scale features are represented as follows: Q = 1 + P + E + 1; similarly, the multi-scale features of the modified text are obtained. and multi-scale features of the target image Q m =1 + P + 1;

[0151] Using multilayer perceptrons in E r and E m Multi-scale interactions are performed to learn the modification weights of the reference image and the modified text, as shown below:

[0152] W = MLP([E r E m ])

[0153] Where W∈R i×J3 The block operation is used to divide W into W0. r ∈R i×3 and W m ∈R i×3 , as the reference image and the weights of the modified text;

[0154] The modified weights are respectively compared with the multi-scale features E of the corresponding reference image and modified text. r E m The aggregation is performed to obtain the aggregated reference image features W. r E r and modifying text features W m E m Then for W r E r and W m E m Summing yields the final multimodal combined feature E. c , means as follows:

[0155] E c =W r E r +W m E m ,

[0156] Among them, E c ∈Q×D,E r E represents the multi-scale features of the reference image. mTo modify the multi-scale weights of the text;

[0157] Batch-based classification loss is used to approximate the combined features to the target image features, thereby improving the accuracy of the combination. The batch-based classification loss is represented as follows:

[0158]

[0159] in, E represents the value corresponding to the i-th triplet. c The result after E6 average pooling, where B represents the batch size and s(·) represents the cosine similarity;

[0160] make Let be the similarity distribution of the i-th combined feature, where is the similarity with the j-th target image. The calculation is as follows:

[0161]

[0162] Similarly, the similarity distribution between the features of the i-th target image and other target images in the batch is obtained, denoted as . Then, KL divergence is used to converge these two similar distributions, and the combined feature space is optimized as follows:

[0163]

[0164] The final objective function is as follows:

[0165]

[0166] Where, Θ * Let κ, τ, and μ be the parameters to be optimized in the model ENCODER, and let μ be the trade-off hyperparameters.

[0167] Table 1 is a comparison table of the retrieval accuracy of the present invention on the FashionIQ dataset;

[0168] Table 1

[0169]

[0170] Table 2 is a comparison table of the retrieval accuracy of the present invention on the Shoes dataset;

[0171] Table 2

[0172]

[0173] Table 3 is a comparison table of the retrieval accuracy of the present invention on the CIRR dataset;

[0174] Table 3

[0175]

[0176] Table 4 is a comparison table of the retrieval accuracy of the present invention on the Fashion200K dataset;

[0177] Table 4

[0178]

[0179] Tables 1-4 show the comparison and results of query efficiency and retrieval accuracy with leading international methods of the same type. The results demonstrate that, compared to similar combined image retrieval methods, the ENCODER model of this invention achieves higher accuracy on four widely used benchmark datasets.

[0180] Example 3

[0181] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the combined image retrieval method based on entity mining and modified relationship binding as described in Embodiment 1 or 2.

[0182] Example 4

[0183] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the combined image retrieval method based on entity mining and modified relationship binding as described in Embodiment 1 or 2.

[0184] Example 5

[0185] A combined image retrieval system based on entity mining and modified relationship binding includes:

[0186] The latent factor filtering module is configured to: read training set data in batches and extract global and local features from the training set data; and further extract visual and textual latent factor features related to the modified semantics by filtering the local features through latent factor filtering.

[0187] The entity-action binding module is configured to: for latent factor features, combine entity-action binding to deeply mine the semantic relationships in the reference image and modified text, detect semantic boundaries, and aggregate latent factors into visual entities and modification actions respectively;

[0188] The multi-scale combination module is configured to combine features of different scales to obtain the final combined features.

[0189] The image retrieval module is configured to: calculate the dot product between the combined features and different images in the image library, use the product as a similarity score, sort the similarity scores in descending order, select the top-ranked target images, and complete the combined image retrieval.

Claims

1. A combined image retrieval method based on entity mining and modifying relationship bindings, characterized in that, Comprise: Batch reading training set data, and extracting global features and local features of the training set data; filtering the local features through latent factors to further extract visual and text latent factor features related to modification semantics; For latent factor features, combined with entity-action binding, deeply mining semantic relationships in reference images and modification texts, detecting semantic boundaries, and aggregating latent factors into visual entities and modification actions, respectively; For the different scale features obtained, through multi-scale combination, the final combined features are obtained; Calculate the dot product of the combined features and different images in the gallery respectively as the similarity score, and arrange the similarity scores in descending order, select the target images ranked in the top several of the similarity scores, and complete the combined image retrieval; Filtering the local features through latent factors to further extract visual and text latent factor features related to modification semantics; comprising: Based on the local features, the cross-modal cross-attention and intra-modal self-attention are used to calculate the cross-relationship scores between each visual factor and text factor; At the same time, the intra-modal self-attention scores are also calculated; Combined with the self-attention scores and cross-relationship scores, the visual and text latent factors and the remaining factors are distinguished; Using a multi-layer perceptron to obtain adaptive weights of the latent factors to enhance the weights of the factors more related to the modification semantics; Based on the local features, the cross-modal cross-attention and intra-modal self-attention are used to calculate the cross-relationship scores between each visual factor and text factor; At the same time, the intra-modal self-attention scores are also calculated; comprising: Based on local features of the reference image, an inter-modality cross relationship score is calculated and an intra-modality self-attention score is represented as follows: wherein, , ; is inter-modal cross attention, is intra-modal self-attention, Q, K are feature vectors to compute attention weights, V is a vector representing input features, refers to local features of the modified text; refers to local features of the reference image ; Combined with the self-attention scores and cross-relationship scores, the visual and text latent factors and the remaining factors are distinguished; comprising: crossing relationship score and self-attention score factors exceeding a specified threshold are retained as potential factors for subsequent entity-action bindings, while factors below a specified threshold are represented as shelved factors; Using a multi-layer perceptron to obtain adaptive weights of the latent factors; comprising: Through the multi-layer perception and the Softmax function, the adaptive weight of the latent factor is obtained, and the enhanced latent factor in the reference image The final representation is as follows: ， wherein, represents the factor weight, P represents the number of latent factors to be learned, , refers to the latent factors of the modified text, refers to the latent factors of the modified text, is an activation function, is a multi-layer perception, in addition, average pooling is also used for the latent factors; similarly, the final latent factor feature of the modified text , and the final latent factor feature of the target image are obtained. For latent factor features, combined with entity-action binding, deeply mining semantic relationships in reference images and modification texts, detecting semantic boundaries, and aggregating latent factors into visual entities and modification actions, respectively; comprising: Training a modality-shared learnable relationship query to mine semantic relationships in reference images and modification texts, detect semantic boundaries, and aggregate latent factors into visual entities and modification actions, respectively; Through the learnable relationship query, the implicit modification relationship between the visual entity and the modification action is learned as a medium to assist the entity-action binding; For latent factor features, combined with entity-action binding, deeply mining semantic relationships in reference images and modification texts, detecting semantic boundaries, and aggregating latent factors into visual entities and modification actions, respectively; comprising: Initializing learnable relational queries where E is the number of queries and D is the global embedding dimension of CLIP. The learnable relation queries are first interacted between each other, and self-attention weights are calculated Cross-attention weights between the learnable relation queries and latent factors of different modalities are also calculated And are represented as follows: wherein, is a cross-attention weight computed for the latent factors of the learnable relation query and the reference image, is a cross-attention weight computed for the latent factors of the learnable relation query and the modified text; Based on , , , the adaptive attention weights of the learnable relationship query to the latent factors of the reference image and the latent factors of the modified text are and , respectively; Will and Each latent factor is assigned a weight, and a multilayer perceptron is used to adaptively learn the weights, thereby mapping the latent factors to corresponding tokens. Each token matches a corresponding query of a learnable relation, as shown below: wherein, and correspond to entity tokens and action tokens, respectively; For learnable relation query design binding an orthogonalization loss function is represented as follows: wherein , is the Frobenius norm of the matrix; respectively, to compute the similarity distribution of the visual entity and the modified action; let denote the similarity distribution of the token in the entity token and the learnable relation query, where is denoted as follows: in, Represents cosine similarity. Representing the The first entity token and the first learnable relation query One query, For temperature coefficient, Indicates the number of triples. This represents the e-th query in the learnable relation query; similarly, we obtain the similarity distribution between each token in the action tokens and each query in the learnable relation query, denoted as . ,in, This represents the similarity between the i-th action token and the j-th learnable relation query; computing a kl divergence loss function between two similarity distributions to bind the respective entity and action, is represented as follows: ， wherein, is a KL divergence loss function, is a KL divergence, denotes a similarity of the i-th target image token and the j-th learnable relation query, denotes a similarity of the i-th action token and each learnable relation query, denotes a similarity of the i-th entity token and each learnable relation query.

2. The combined image retrieval method based on entity mining and modifying relationship binding according to claim 1, characterized in that, For the different scale features obtained, through multi-scale combination, the final combined features are obtained; comprising: Integrating the multi-scale features of the reference image and the modification text respectively; Based on the obtained multi-scale features, multi-scale interaction is performed through a multi-layer perceptron to learn the modification weights of the reference image and the modification text respectively; Based on the learned modification weights, the final combined features are obtained.

3. The combined image retrieval method based on entity mining and modifying relationship binding according to claim 1, characterized in that, For the different scale features obtained, through multi-scale combination, the final combined features are obtained; comprising: The corresponding global features, local features, latent factors and shelving factors are spliced; for the reference image, the multi-scale feature is represented as , ; similarly, the multi-scale feature of the modified text is obtained and the multi-scale feature of the target image ; ; Using a multi-layer perceptron in and multi-scale interactions, learning reference image and modification text respective modification weights, represented as follows: wherein, , the modified text is segmented into , and , and , as respective weights of the reference image and the modified text; The modified weights are compared with the multi-scale features of the corresponding reference image and modified text, respectively. , Perform aggregation to obtain aggregated reference image features. and modify text features And then and Summing yields the final multimodal combined features. , means as follows: wherein, , is a multi-scale feature of the reference image, is a multi-scale weight of the modified text; Using a batch-based classification loss to make the combined features approach the target image features, and the batch-based classification loss is represented as follows: wherein, , represents the th triplet corresponds to and the average-pooled results, represents the batch size, represents the cosine similarity; Let be the similarity distribution of the th combined feature, where the similarity to the th target image is calculated as follows: Similarly, the distribution of similarities between the target image features and other target images in the batch is obtained, denoted as and the combined feature space is optimized using the KL divergence to converge the two similarity distributions, represented as follows: The final target function is obtained as follows: wherein, are the parameters to be optimized for the model ENCODER, , , is a trade-off hyperparameter.

4. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the combined image retrieval method based on entity mining and modification relation binding according to any one of claims 1-3 when executing the computer program.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The processor implements the steps of the combined image retrieval method based on entity mining and modification relation binding according to any one of claims 1-3 when executing the computer program.

6. A combined image retrieval system based on entity mining and modifying relation binding, for implementing the combined image retrieval method based on entity mining and modifying relation binding according to any one of claims 1-3, characterized in that, Comprise: A potential factor filtering module is configured to read training set data in batches, and extract global features and local features from the training set data; the local features are filtered through potential factors to further extract visual and textual potential factor features related to modification semantics; An entity-action binding module is configured to, for the potential factor features, combine entity-action binding, deeply mine semantic relationships in the reference image and the modification text, detect semantic boundaries, and aggregate the potential factors into visual entities and modification actions, respectively; A multi-scale combination module is configured to, for the obtained features of different scales, obtain final combined features through multi-scale combination; An image retrieval module is configured to use the combined features to screen images in a candidate target image set and obtain a formal result set, i.e., a final result of the combined image retrieval.

Images