Multi-modal combined image retrieval method fusing fine-grained semantic positioning and optimization generation features

Abstract

The invention relates to a multi-modal combined image retrieval method fusing fine-grained semantic positioning and optimization generation features, which comprises the following steps: acquiring a reference image, predicting a bounding box of a target object based on the reference image and a positioning keyword, and dividing a background retaining region and a foreground editing region; semantic divergence prompt words of description information are obtained, text features in the semantic divergence prompt words are extracted to serve as guide targets to be used for establishing a composite optimization target function, a gradient descent algorithm is adopted to conduct iterative updating on learnable random noise vectors, and potential visual feature vectors are generated; the foreground editing area is filled with the image data, a background-foreground mixed feature map is generated, the fine-grained visual similarity between the background-foreground mixed feature map and the dense visual feature map is calculated, and the global text feature similarity between global text feature vectors in description information and global visual feature vectors is calculated; and carrying out weighted fusion on the fine-grained visual similarity and the global text feature similarity, and outputting a retrieval result.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal information retrieval technology, and in particular to a multimodal combined image retrieval method that integrates fine-grained semantic localization and optimized generated features. Background Technology

[0002] Composed Image Retrieval (CIR) is an emerging multimodal retrieval task. Its goal is to retrieve images from a candidate image library that meet the modification requirements, given a reference image and modification text. For example, if a user provides an image of a "black dog" and inputs the text "replace the dog with a white cat," the system needs to retrieve images with "similar backgrounds but with the subject changed to a white cat."

[0003] Existing CIR methods are mainly divided into two categories: methods based on full-image feature fusion and methods based on generative models. The first category typically utilizes pre-trained models such as CLIP (Contrastive Language-Image Pretraining) to extract global features from the reference image and text, fusing them through simple concatenation or attention mechanisms. However, these methods often struggle to handle fine-grained modification needs. For example, when a user only wants to modify a specific object in an image while preserving the background, global feature fusion often leads to the loss or obfuscation of background information. The second category attempts to use Generative Adversarial Networks (GANs) or Diffusion Models to first generate a target image, and then use the generated image for retrieval. While this method is intuitive, the generation process is computationally expensive, and the generated image often contains visual artifacts, leading to inaccurate retrieval features. Furthermore, current generation methods struggle to precisely control the boundaries between the "preserved region" and the "edited region," easily destroying the structural information that should be retained in the original image.

[0004] Therefore, there is an urgent need for a multimodal image retrieval method that can accurately locate the editing area, efficiently generate high-quality target features, and effectively preserve background information. Summary of the Invention

[0005] To address the problems of existing technologies, the present invention aims to provide a multimodal image retrieval method that integrates fine-grained semantic localization and optimized feature generation. This method generates prompt words and image descriptions by parsing instructions using a large language model, generates masks to distinguish image regions using a visual language model for semantic localization, designs a visual feature construction module to integrate divergent prompt words to generate potential visual feature vectors to fill the edit region and construct a hybrid feature map, and employs a dual-stream strategy to dynamically fuse fine-grained visual similarity and global text similarity. This addresses the problems of inaccurate localization, background loss, and low-quality generated features in existing technologies when processing fine-grained editing intentions.

[0006] To achieve the above objectives, the present invention provides the following solution: A multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation includes: Acquire a reference image, use the reference image to parse the user's editing intent, and extract the location keywords and descriptive information of the target image; Based on the reference image and the location keywords, the bounding box of the target object is predicted, and a subject mask is generated to divide the background preservation area and the foreground editing area. Obtain semantic divergent prompts from the description information, extract text features from the semantic divergent prompts as guiding targets, establish a composite optimization objective function, and use gradient descent algorithm to iteratively update the learnable random noise vector to generate a potential visual feature vector. The latent visual feature vectors are filled into the foreground editing region to generate a background-foreground blended feature map. The fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map, as well as the global text feature similarity between the global text feature vector and the global visual feature vector in the description information are calculated. The fine-grained visual similarity and the global text feature similarity are weighted and fused to obtain the retrieval results.

[0007] Optionally, using the reference image to parse the user's editing intent and extract location keywords and descriptive information of the target image includes: The reference image is processed by a pre-trained image description generation model to generate a global semantic description text that covers the image subject and environment information. Construct a joint input sequence containing the global semantic description text, the modified text input by the user, and the system prompt words; The joint input sequence is parsed using a large language model to obtain structured data. The structured data is then parsed a second time to extract the editing intent, the location keywords, and the descriptive information of the target image. The operation types of the editing intent include: target addition, target deletion, target replacement, and background replacement.

[0008] Optionally, dividing the background preservation area and the foreground editing area includes: Using a large visual language model with grounding capabilities, location cue words are extracted for each location object in the reference image to locate keywords, and the normalized bounding box coordinates of the target object in the image coordinate system are obtained as the bounding box of the target object. Based on the editing intent and the bounding box, a binary subject mask matrix is ​​generated to divide the background preservation region and the foreground editing region: If the editing intent is target deletion or target replacement, the area within the bounding box is defined as the foreground editing area, the area outside the bounding box is defined as the background preservation area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area. If the editing intent is background replacement, a reverse masking operation is performed, defining the area within the bounding box as the background preservation area, the area outside the bounding box as the foreground editing area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area. If the editing intent is target addition, a region is randomly selected from the edge region of the reference image and defined as the foreground editing area, the remaining region is defined as the background preservation area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area.

[0009] Optionally, generating the latent visual feature vector includes: The descriptive information is semantically divergent using a large language model to generate a preset number of semantically divergent prompt words containing different attribute or state descriptions. A learnable noise vector with the same feature dimensions as the image encoder is constructed, and the learnable noise vector is randomly initialized using a standard normal distribution; The text feature vectors of the semantically divergent prompts are extracted using a text encoder as the guiding target. A composite optimization objective function containing semantic alignment loss and structural constraint loss is constructed. The gradient descent algorithm is used to iteratively update the initialized learnable noise vectors to generate multiple optimized potential visual feature vectors. The semantic relevance between the text features corresponding to each semantic divergence prompt word and the descriptive information of the target image is calculated to generate a fusion weight. Then, the weighted summation and normalization of multiple optimized potential visual feature vectors are performed to generate the potential visual feature vector.

[0010] Optionally, constructing the composite optimization objective function includes: ; In the formula, For learnable noise vectors The unit vector after L2 normalization This is the text feature vector of the current prompt word. To constrain the distance between the generated features and the semantics of the target text, To constrain the generated features and the corresponding region features of the original image The distance is used to prevent excessive feature drift. This is the balance coefficient.

[0011] Optionally, generating the background-foreground blending feature map includes: The latent visual feature vectors are broadcast in the spatial dimension to keep the size of the latent visual feature vectors consistent with the spatial resolution of the dense visual feature map, thereby generating a foreground filling feature map; the dense visual feature map is obtained by weighted averaging of slice label sequences from multiple network layers using preset weights or learnable weights. Using the subject mask as a gating coefficient, the dense visual feature map and the foreground filling feature map are fused element-wise with weight: the feature value of the foreground filling feature map is used in the foreground editing region where the mask value is the first target value, and the original feature value of the dense visual feature map is retained in the background preservation region where the mask value is the second target value, thereby generating the background-foreground hybrid feature map.

[0012] Optionally, obtaining the search results includes: The slices in the background-foreground blended feature map are divided into a background slice set and a foreground slice set. The maximum similarity of each slice in the background slice set is determined among all slices in the candidate image feature map. The matching degree between the foreground slice set and all slices in the candidate image feature map is calculated. By utilizing the maximum similarity and the degree of matching, and combining the foreground region proportion, a fine-grained visual similarity between the background-foreground mixed feature map and the dense visual feature map is obtained; The text encoder extracts features from the descriptive information of the target image to generate a global text feature vector. The cosine similarity between the global text feature vector and the global image feature vector is calculated to obtain the global text retrieval similarity. The category label of the last layer of the image encoder is used as the global image feature vector. Based on a preset balance weight, the fine-grained visual similarity and the global text retrieval similarity are linearly weighted and fused to generate a final retrieval score. The final retrieval score is then used to sort the candidate image set in descending order, and a preset number of images at the top of the sorted list are selected as the retrieval results.

[0013] Optionally, calculating the fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map includes: , ; ; in, This represents a fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map. For the proportion of the prospective area, For mask The number of foreground slices with a median value of 1. The total number of image slices, For adjustment coefficients, Forward weighting, This is the background weight.

[0014] Optionally, calculating the global text feature similarity between the global text feature vector and the global visual feature vector in the description information includes: ; ; in, For dimension The target global text feature vector, For the text encoder's mapping function, Descriptive information for the target image. For global text feature similarity For the first Zhang candidate image feature vectors, This is the transpose of a vector. The beneficial effects of this invention are: Achieving fine-grained, precise editing and retrieval: This invention combines the intent parsing capabilities of a large language model with the localization capabilities of a large visual language model to generate accurate subject masks, clearly defining the "background preservation region" and the "foreground editing region." This allows the retrieval process to focus on the specific object the user wants to modify, while perfectly preserving the original background information, solving the problem of easily lost or incorrectly modified background information in traditional global feature retrieval methods.

[0015] High-quality and computationally efficient feature generation: Unlike traditional methods that generate pixel-level images based on GANs or diffusion models, this invention employs a "semantic divergence" and "noise vector iterative optimization" strategy to directly generate latent visual features in the feature space. This method not only avoids common visual artifacts and structural distortions in image generation but also significantly reduces computational overhead, enabling the rapid generation of high-quality feature vectors that are highly aligned with text descriptions.

[0016] This invention employs a dual-stream retrieval strategy that dynamically weights and fuses "fine-grained visual similarity" and "global textual feature similarity." It utilizes dense feature maps to capture detailed information such as texture and shape in images, while also leveraging global textual features to capture high-level semantic information. By adjusting the balancing parameters, it can adapt to different types of editing needs (addition, deletion, replacement, etc.), significantly improving the recall and accuracy of multimodal combined image retrieval (CIR) tasks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a multimodal image retrieval method that integrates fine-grained semantic localization and optimized feature generation according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the candidate image feature library construction process according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of the semantic intent parsing and fine-grained region localization module in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of the iterative optimization generation process of the latent visual feature vector in an embodiment of the present invention; Figure 5 This is a schematic diagram of the dual-stream feature similarity calculation and weighted fusion retrieval process according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This embodiment discloses a multimodal image retrieval method that integrates fine-grained semantic localization and optimized feature generation. The method includes: acquiring a reference image; analyzing the user's editing intent using the reference image and extracting localization keywords and descriptive information of the target image; predicting the bounding box of the target object based on the reference image and the localization keywords, generating a subject mask to divide the background-preserving region and the foreground editing region; acquiring semantic divergent cue words from the descriptive information, extracting text features from the semantic divergent cue words as guiding targets to establish a composite optimization objective function, and iteratively updating a learnable random noise vector using a gradient descent algorithm to generate a latent visual feature vector; filling the latent visual feature vector into the foreground editing region to generate a background-foreground hybrid feature map; calculating the fine-grained visual similarity between the background-foreground hybrid feature map and the dense visual feature map, and the global text feature similarity between the global text feature vector and the global visual feature vector in the descriptive information; and weightedly fusing the fine-grained visual similarity and the global text feature similarity to obtain the retrieval result.

[0021] Specifically, this embodiment discloses a multimodal image retrieval method that integrates fine-grained semantic localization and optimized generated features, including: Step 1: Collect large-scale image data, use an image encoder to extract global image feature vectors and dense visual feature maps of candidate images, and construct two feature libraries containing different features of candidate images; Step 2: Obtain the reference image and modified text. Use a pre-trained image description model to generate a global semantic description of the reference image. Input the semantic description and the user's modified text into a large language model for reasoning, parse the user's editing intent, and extract the location keywords and description information of the target image. Step 3: Input the reference image and the target keywords parsed in Step 2 into the visual language big model to predict the bounding box of the target object, generate the subject mask, and divide the background preservation area and the foreground editing area. Step 4: Use a large language model to generate semantic divergent prompts describing the target content of the editing area, extract their text features as guidance targets, initialize the mask region as a learnable random noise vector, construct an optimized loss function, and use an optimization algorithm to iteratively update the noise vector of the mask region to generate a potential visual feature vector. Step 5: Based on the mask, retain the dense features of the background region of the reference image, and fill the foreground editing region with the latent visual feature vector obtained in Step 4 to construct a background-foreground hybrid feature map. Calculate the fine-grained visual similarity between the map and the dense visual feature map in the candidate image library. Step 6: Using the target image description information obtained in Step 2, extract the global text feature vector of the target image description information, calculate its similarity with the global visual feature vector in the candidate image library, and perform weighted fusion of fine-grained visual similarity and global text feature similarity according to preset weights to output the retrieval results.

[0022] Further, step 1 specifically includes: acquiring a candidate image set and using a pre-trained visual encoder based on the Transformer architecture as the image encoder; inputting the images from the candidate image set into the image encoder, extracting the class token ([CLS] Token) of the last layer of the image encoder as the global image feature vector, and constructing the first feature library; extracting the patch token sequence of the multi-layer network layers of the image encoder, and performing a weighted average calculation on the patch token sequence of the multi-layer network layers according to preset weights or learnable weights to generate the dense visual feature map, and constructing the second feature library. The pre-trained visual encoder based on the Transformer architecture is specifically a visual encoder in a CLIP model pre-trained from a public or authorized data source.

[0023] Further, step 2 specifically includes: using a pre-trained image description model to encode visual features and decode text for the reference image, generating a global semantic description text reflecting the visual content of the reference image; constructing a prompt word template, concatenating the global semantic description text with the modified text to form joint input data, and inputting the joint input data into a large language model; using the large language model to perform semantic reasoning and instruction parsing on the joint input data, outputting structured data, wherein the structured data includes editing intent, location keywords, and descriptive information of the target image; wherein the operation types of the editing intent include: target addition, target deletion, target replacement, and background replacement.

[0024] Further, step 3 specifically includes: inputting the reference image and the localization keywords into a large visual language model with region awareness capabilities, performing visual localization on the reference image, predicting and outputting the bounding box of the target object corresponding to the localization keywords; generating a binarized subject mask according to the editing intent parsed in step 2, to divide the background retention region and the foreground editing region according to the following rules: if the editing intent is target deletion or target replacement, the region within the bounding box of the target object is defined as the foreground editing region, and the region outside the bounding box is defined as the background retention region, and the generated mask takes a value of 1 in the foreground editing region and a value of 0 in the background retention region; if the editing intent is background replacement, performing a reverse mask operation, defining the region within the bounding box of the target object as the background retention region, and the region outside the bounding box as the foreground editing region; if the editing intent is target addition, randomly selecting a region on the edge of the reference image, defining it as the foreground editing region, and defining the remaining region as the background retention region.

[0025] Furthermore, step 4 specifically includes the following sub-steps: S1. Semantic divergence: The descriptive information of the target image is used as the original editing target and input into a large language model for semantic divergence processing to generate a set of semantic divergence prompts containing different attributes or state descriptions of a preset number N. S2. Vector initialization: Construct a learnable noise vector with the same feature dimensions as the image encoder, and randomly initialize it using a standard normal distribution; S3. Iterative optimization: For each semantic divergent prompt word in the set, extract its text feature vector using a text encoder, and use it as the guidance target; construct a composite optimization objective function that includes semantic alignment loss and structural constraint loss, fix the text encoder parameters, and use the gradient descent algorithm to iteratively update the learnable noise vector to generate N optimized potential visual feature vectors. S4. Weighted Fusion: Calculate the semantic relevance between each semantic divergence prompt and the original editing target, generate fusion weights accordingly, and perform weighted summation and normalization on the N optimized potential visual feature vectors to obtain the final potential visual feature vectors.

[0026] Further, the process of constructing the background-foreground hybrid feature map in step 5 specifically includes: acquiring a dense visual feature map extracted from the reference image by an image encoder; broadcasting the latent visual feature vector generated in step 4 in the spatial dimension to make its size consistent with the spatial resolution of the dense visual feature map, thereby generating a foreground filling feature map; using the subject mask generated in step 3 as a gating coefficient, performing element-wise weighted fusion of the dense visual feature map and the foreground filling feature map: in the foreground editing region with a mask value of 1, the feature value of the foreground filling feature map is used; in the background preservation region with a mask value of 0, the original feature value of the dense visual feature map is retained, thereby obtaining the background-foreground hybrid feature map.

[0027] Further, step 6 specifically includes: using a text encoder to extract features from the target image description information generated in step 2 to generate a target global text feature vector; calculating the cosine similarity between the target global text feature vector and the global image feature vector of each candidate image in the feature library of step 1 to obtain a global text retrieval similarity; obtaining the fine-grained visual similarity calculated in step 5, and linearly weighting and fusing the fine-grained visual similarity and the global text retrieval similarity according to a preset balanced weight to generate a final retrieval score; sorting the candidate image set in descending order according to the final retrieval score, and selecting a preset number of images at the top of the sort as the final retrieval result output.

[0028] The second aspect of this embodiment provides a multimodal image retrieval device that integrates fine-grained semantic localization and optimized generated features, including one or more processors and a memory, characterized in that the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multimodal image retrieval method integrating fine-grained semantic localization and optimized generated features.

[0029] The third aspect of this embodiment provides a computer-readable storage medium, characterized in that it stores a program thereon, which, when executed by a processor, is used to implement the above-described multimodal combined image retrieval method that integrates fine-grained semantic localization and optimized generated features.

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1 As shown in the figure, this embodiment discloses a multimodal image retrieval method that integrates fine-grained semantic localization and optimized generated features, specifically including the following steps: Step (1): Collect large-scale image data, use an image encoder to extract global image feature vectors and dense visual feature maps of candidate images, and construct two feature libraries containing different features of candidate images, such as... Figure 2 As stated above.

[0032] Specifically, a candidate image set is formed by collecting large-scale image data from public datasets (such as Fashion-iq, CIRR, CIRCO), denoted as . ,in This represents the total number of candidate images. For any image in the set... The model undergoes normalization preprocessing: it is scaled and center-cropped, and the pixel values ​​are normalized to obtain the model input tensor. The visual encoder configuration uses a pre-trained visual encoder based on the Transformer architecture. Feature extraction is performed on the image. The encoder contains a Transformer encoding layer. The input tensor is divided into... The input sequence consists of fixed-size image patches, with a learnable class token ([CLS] Token) added to the beginning of the sequence.

[0033] A first feature library is constructed, and a global image feature library is used to extract macroscopic semantic representations of images using an encoder. The input sequence is passed through the encoder. Forward propagation, extract the last layer (the... The hidden state of the category label output by the layer) The feature space is mapped to a common feature space through a visual projection layer, and L2 normalization is performed to obtain a dimension of [dimensional value missing]. global image feature vector The calculation formula is as follows: ; in, Represents a linear projection transformation. Represents the L2 norm. The set of global feature vectors for all images. Stored as the first feature library.

[0034] A second feature library is constructed, and the dense visual feature library extracts fine-grained local representations of the image using the encoder. To balance deep semantics and shallow details, this embodiment employs a multi-layer feature fusion strategy. A layer index set is defined. (For example, select index as) (Some key layers in the set). Each layer Extract the slice marker sequence from its output. For each layer's slice sequence, layer normalization (LayerNorm) and visual projection are performed, and then the results are calculated according to preset weights. Perform weighted fusion to generate a fused dense feature map. The calculation process is expressed as follows: ; in, Representation layer normalization, This represents the final L2 normalization. As a preferred and efficient implementation, only the last layer (i.e., ...) can be selected. ,correspond To reduce computational overhead, a dense visual feature library is ultimately constructed. It contains massive amounts of high-dimensional data.

[0035] Step (2): Obtain the reference image and modified text, use the pre-trained image description model to generate a global semantic description of the reference image, input the semantic description and the user's modified text into the large language model for reasoning, parse the user's editing intent, and extract the location keywords and the description information of the target image.

[0036] Specifically, obtaining the reference image And the modified text entered by the user First, a pre-trained image captioning model (BLIP-2 or LLaVA) is used to perform visual-to-text processing on the reference image, generating a global semantic description text that covers information about the image subject and its environment. The calculation formula is expressed as follows: ; in, This represents the mapping function for the image description generation model.

[0037] To enable Large Language Models (LLMs) to understand the user's modification intent by incorporating visual context, a model is constructed that includes system prompts and reference image descriptions. And user-edited text joint input sequence .Will Inputting data into a large language model for inference, the model outputs parsed structured data (usually in JSON format), denoted as... The reasoning process can be represented as follows: ; in, These are preset system prompt word templates used to constrain output format and reasoning logic; This describes the process of generating a large language model.

[0038] Structured data output by the model Analysis revealed three key control variables: edit operation type. Targeting keyword set and complete descriptive information of the target image The specific analytical relationships are as follows: ; Editing operation class : Belongs to the predefined set of operations , This is used in subsequent steps to determine the logic for mask generation (such as foreground / background inversion). Locating keywords. : The name of the object to be modified or located in the original image (e.g., "black dog"), serving as the input anchor point for the subsequent visual localization model. Target image description The original image's unmodified background is combined with the newly generated text after the user's modification instructions (e.g., "A whitecat sitting on the grass") for subsequent text feature extraction and semantic consistency constraints.

[0039] Step (3) compare the reference image with the target keywords of the user-modified text parsing. Input a large visual language model, predict the bounding box of the target object, generate the subject mask, and divide the background preservation area and the foreground editing area.

[0040] like Figure 3 As shown, the reference image Compared with the set of positioning keywords extracted in step (2) Input a large-scale visual-language model (VLM) with grounding capabilities, such as VPP-LLaVA. For each localized object in the set... The model constructs location prompts, such as: "Please provide the bounding box coordinates of the region this sentence describes: [obj]". The model outputs the normalized bounding box coordinates of the target object in the image coordinate system. : ; In the formula, The coordinates of the top left corner The coordinates are the bottom right corner.

[0041] Based on editing operation type (From step (2)) and the mapped grid coordinates Generate a binary subject mask matrix. The median value of the mask is The region is defined as the foreground editing region (the region where new features need to be generated and modified), and its value is... The region is defined as the background-preserving region (i.e., the unmodified region that needs to retain its original features). It should be noted that this region definition is based on the logical requirements of the editing operation, rather than a fixed image semantic category.

[0042] Scenario A: Object Replacement / Delete. This type of operation aims to modify a specified object. In this case, the area inside the bounding box is the editing region. ; Scenario B: Background / Environment Replacement. This type of operation aims to preserve a specified object while modifying its surrounding environment. Based on the aforementioned definition (a value of 1 for the editable area, and a value of 0 for the preserved area), a logical reversal is required: the area of ​​the target object within the bounding box that does not need modification is defined as the "background preserved area" (assigned a value of 0), and the area of ​​the environment outside the bounding box that needs modification is defined as the "foreground editable area" (assigned a value of 1). The formula is as follows: ; in, To normalize the bounding box coordinates of the target object A set of region indices mapped to feature map network coordinates.

[0043] Scenario C: Adding an object. If the operation is to add an object and there is no explicit reference position, a rectangular area is randomly sampled from the image edge or blank area as the target location. and set the area The final generated mask This will be used as a spatial gating signal for feature fusion in subsequent steps.

[0044] Step (4): Use a large language model to generate semantic divergent prompts describing the target content of the editing area, extract their text features as guidance targets, initialize the mask area as a learnable random noise vector, construct an optimized loss function, use an optimization algorithm to iteratively update the noise vector of the mask area, and generate a potential visual feature vector.

[0045] S1. Semantic divergence: To cover the diversity of visual expression of the target object (different angles, texture details) and avoid feature overfitting caused by a single text description, this embodiment introduces a semantic divergence mechanism. The target image description information obtained in step (2) is then used... Input a Large Language Model (LLM). Construct a System Prompt template containing divergent instructions, such as: "Generate N variant descriptions of...". Based on this instruction, the LLM generates N (preferably N=15) semantic divergent prompt words with different emphases, denoted as a set. These prompts, while maintaining consistency in core semantics, enrich the adjective modifiers and scene descriptions.

[0046] S2. Vector initialization: Construct a learnable noise vector with the same feature dimensions as the image encoder, and randomly initialize it using a standard normal distribution; specifically, initialize a latent vector to be optimized in the feature space. Construct a vector with dimension... (In this embodiment) Learnable noise vectors (with the same feature dimensions as CLIP ViT-Large) To break the symmetry and provide a stochastic starting point for optimization, the system is randomly initialized using a standard normal distribution and multiplied by a scaling factor to control the initial norm. ; in, It is an identity matrix.

[0047] S3. Iterative optimization: For each semantic divergent prompt word in the set, extract its text feature vector using a text encoder, and use it as the guidance target; construct a composite optimization objective function that includes semantic alignment loss and structural constraint loss, fix the text encoder parameters, and use the gradient descent algorithm to iteratively update the learnable noise vector to generate N optimized potential visual feature vectors. Specifically, for sets Each semantic divergence cue word in Perform an independent gradient descent optimization process to generate the corresponding latent features. First, the prompt words are extracted using a text encoder with frozen parameters. Text feature vectors Then define the composite optimization objective function. Includes semantic alignment loss and structural constraint loss .

[0048] ; In the formula: yes Unit vector after L2 normalization; It is the text feature vector of the current prompt word; ( This is used to constrain the distance between the generated features and the semantics of the target text; Used to constrain the generated features to the corresponding region features of the original image. The distance is used to prevent excessive feature drift; This is a balance factor. When the editing intent is "replace environment" or "focus object", this factor is set to preserve the main structure. For other operations (such as complete replacement), set .

[0049] S4. Weighted Fusion: To suppress the influence of low-quality variants or prompts that deviate from the core semantics, a weighted fusion strategy is adopted. This involves calculating the semantic divergence of each prompt. Corresponding text features Compared with the original editing target description semantic relevance score : ; Normalized weights are generated based on the relevance scores: ; in, The semantic relevance score of the j-th prompt word. This represents the total number of semantic divergence prompts.

[0050] Finally, all optimized feature vectors are weighted and summed, and then L2 normalized again to obtain the final latent visual feature vectors. : .

[0051] Step (5): Based on the mask, retain the dense features of the background region of the reference image, and fill the foreground editing region with the potential visual feature vector obtained in step (4) to construct a background-foreground hybrid feature map. The aim is to accurately inject the generated potential semantic features into the editing region of the image, and calculate the visual similarity while maintaining background consistency, such as... Figure 4 As shown; Specifically, the original dense visual feature map of the reference image is obtained after being extracted by the image encoder. (in The total number of slices, (For feature dimensions). Utilize the subject mask generated in step (3). (A binary matrix, with dimensions of 1 / 2) The latent visual feature vector generated in step (4) is used as a spatial gating signal. (dimension is) Broadcasting is performed in the spatial dimension to adapt it to the foreground editing region, and a hybrid feature map is constructed using a gating fusion formula. Then, re-normalization is performed to eliminate norm inconsistencies caused by splicing: ; in, This represents element-wise broadcast multiplication. In Corresponding prospects Corresponding background. At this time, The original image's texture details are preserved in the retained area, while the edited area has ideal semantic features aligned with the text description.

[0052] To more accurately measure image similarity, this embodiment employs a region-aware patch matching strategy, decomposing similarity calculation into two parts: background similarity and foreground similarity. Based on the mask... , mix feature maps The slices in the image are divided into a set of background slices. and foreground slice set .

[0053] Specifically, background similarity ,for Each slice in In the second feature library (dense visual feature library) All slices The algorithm finds the best-match among all background slices and then averages the results. This ensures consistent background texture. ; Foreground Similarity Similarly, the degree of matching between the foreground slice set and the candidate image is calculated to measure the achievement of the editing objective: ; To balance the impact of different editing area proportions, the foreground area proportion is calculated. .in, For mask The number of foreground slices with a median value of 1. This represents the total number of image slices. It is determined by the percentage of the edited region's area within the entire image. The weights of foreground and background are dynamically adjusted. Generally, the smaller the editing area, the more easily its features are obscured by the background, thus requiring an appropriate increase in their weight. The final fine-grained visual similarity is then calculated. The calculation is as follows: ; ; in, For adjustment coefficients, Forward weighting, This is the background weight.

[0054] Step (6): Using the target image description information obtained in step (2), extract the global text feature vector of the target image description information, calculate its similarity with the global visual feature vector in the candidate image library, and perform weighted fusion of fine-grained visual similarity and global text feature similarity according to preset weights to output the retrieval result. This step aims to combine the global semantic information of the target image to correct and enhance the fine-grained visual retrieval result obtained in step (5). By linearly weighting the global text feature similarity and fine-grained visual similarity, the final retrieval result is output, such as... Figure 5 As shown. The specific implementation process includes the following sub-steps: S1. Target Global Text Feature Extraction: Using a pre-trained text encoder (e.g., CLIPText Encoder, which is in the same multimodal feature space as the image encoder in step (1)), the target image description information generated in step (2) is extracted. Feature extraction is performed on the input text (including the unmodified background description and the modified main description). The input text is converted into a labeled sequence, and then forward-propagated through a text encoder to extract the final layer features, which are then L2 normalized to obtain a dimension of [dimensional value missing]. Target global text feature vector : ; in, This represents the mapping function of the text encoder.

[0055] S2. Global text retrieval similarity calculation: Traverse the first feature library (global image feature library) constructed in step (1) and obtain the global image feature vector of each candidate image. Calculate the feature vector of the target text. With the Zhang Candidate Image Feature Vector The cosine similarity between them is denoted as the global text retrieval similarity. Since the vectors have been pre-normalized, cosine similarity is equivalent to the dot product operation: ; S3, Dual-stream Feature Weighted Fusion: To balance fine-grained texture matching of the image (by... (provided) and high-level semantic consistency (by) (Provided), and a linear weighting strategy is used to fuse the two. The fine-grained visual similarity calculated in step (5) is obtained. Define the balancing weight parameters. (range of values) This is used to adjust the contribution ratio of visual feature matching and text semantic matching in the search results. Final search score. The calculation formula is as follows: ; In this embodiment, parameters It can be set according to the specific application scenario, and is usually set to Treat visual details and semantic descriptions equally; or set them as learnable parameters and adaptively adjust them using a validation set.

[0056] S4. Sorting and Result Output: Based on the calculated final search score... The system sorts all images in the candidate image library in descending order. It selects a preset number of images (e.g., Top-10 or Top-50) as the final search results and returns them to the user.

[0057] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation, characterized in that, include: Acquire a reference image, use the reference image to parse the user's editing intent, and extract the location keywords and descriptive information of the target image; Based on the reference image and the location keywords, the bounding box of the target object is predicted, and a subject mask is generated to divide the background preservation area and the foreground editing area. Obtain semantic divergent prompts from the description information, extract text features from the semantic divergent prompts as guiding targets, establish a composite optimization objective function, and use gradient descent algorithm to iteratively update the learnable random noise vector to generate a potential visual feature vector. The latent visual feature vectors are filled into the foreground editing region to generate a background-foreground blended feature map. The fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map, as well as the global text feature similarity between the global text feature vector and the global visual feature vector in the description information, are calculated. The fine-grained visual similarity and the global text feature similarity are weighted and fused to obtain the retrieval results.

2. The multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation according to claim 1, characterized in that, The process of analyzing the user's editing intent using the reference image and extracting location keywords and descriptive information of the target image includes: The reference image is processed by a pre-trained image description generation model to generate a global semantic description text that covers the image subject and environment information. Construct a joint input sequence containing the global semantic description text, the modified text input by the user, and the system prompt words; The joint input sequence is parsed using a large language model to obtain structured data. The structured data is then parsed a second time to extract the editing intent, the location keywords, and the descriptive information of the target image. The operation types of the editing intent include: target addition, target deletion, target replacement, and background replacement.

3. The multimodal image retrieval method integrating fine-grained semantic localization and optimized generated features according to claim 2, characterized in that, The division of the background preservation area and the foreground editing area includes: Using a large visual language model with grounding capabilities, location cue words are extracted for each location object in the reference image to locate keywords, and the normalized bounding box coordinates of the target object in the image coordinate system are obtained as the bounding box of the target object. Based on the editing intent and the bounding box, a binary subject mask matrix is ​​generated to divide the background preservation region and the foreground editing region: If the editing intent is target deletion or target replacement, the area within the bounding box is defined as the foreground editing area, the area outside the bounding box is defined as the background preservation area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area. If the editing intent is background replacement, a reverse masking operation is performed, defining the area within the bounding box as the background preservation area, the area outside the bounding box as the foreground editing area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area. If the editing intent is target addition, a region is randomly selected from the edge region of the reference image and defined as the foreground editing area, the remaining region is defined as the background preservation area, and the generated mask is set to a first target value in the foreground editing area and a second target value in the background preservation area.

4. The multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation according to claim 1, characterized in that, Generating the latent visual feature vector includes: The descriptive information is semantically divergent using a large language model to generate a preset number of semantically divergent prompt words containing different attribute or state descriptions. A learnable noise vector with the same feature dimensions as the image encoder is constructed, and the learnable noise vector is randomly initialized using a standard normal distribution; The text feature vectors of the semantically divergent prompts are extracted using a text encoder as the guiding target. A composite optimization objective function containing semantic alignment loss and structural constraint loss is constructed. The gradient descent algorithm is used to iteratively update the initialized learnable noise vectors to generate multiple optimized potential visual feature vectors. The semantic relevance between the text features corresponding to each semantic divergence prompt word and the descriptive information of the target image is calculated to generate a fusion weight. Then, the weighted summation and normalization of multiple optimized potential visual feature vectors are performed to generate the potential visual feature vector.

5. The multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation according to claim 1, characterized in that, Constructing the composite optimization objective function includes: ； In the formula, For learnable noise vectors The unit vector after L2 normalization This is the text feature vector of the current prompt word. To constrain the distance between the generated features and the semantics of the target text, To constrain the generated features and the corresponding region features of the original image The distance is used to prevent excessive feature drift. This is the balance coefficient.

6. The multimodal image retrieval method according to claim 1, which integrates fine-grained semantic localization and optimized feature generation, is characterized in that... Generating the background-foreground blending feature map includes: The latent visual feature vectors are broadcast in the spatial dimension to keep the size of the latent visual feature vectors consistent with the spatial resolution of the dense visual feature map, thereby generating a foreground filling feature map; the dense visual feature map is obtained by weighted averaging of slice label sequences from multiple network layers using preset weights or learnable weights. Using the subject mask as a gating coefficient, the dense visual feature map and the foreground filling feature map are fused element-wise with weight: the feature value of the foreground filling feature map is used in the foreground editing region where the mask value is the first target value, and the original feature value of the dense visual feature map is retained in the background preservation region where the mask value is the second target value, thereby generating the background-foreground hybrid feature map.

7. The multimodal image retrieval method integrating fine-grained semantic localization and optimized generated features according to claim 1, characterized in that, Obtaining the search results includes: The slices in the background-foreground blended feature map are divided into a background slice set and a foreground slice set. The maximum similarity of each slice in the background slice set is determined among all slices in the candidate image feature map. The matching degree between the foreground slice set and all slices in the candidate image feature map is calculated. By utilizing the maximum similarity and the degree of matching, and combining the foreground region proportion, a fine-grained visual similarity between the background-foreground mixed feature map and the dense visual feature map is obtained; The text encoder extracts features from the descriptive information of the target image to generate a global text feature vector. The cosine similarity between the global text feature vector and the global image feature vector is calculated to obtain the global text retrieval similarity. The category label of the last layer of the image encoder is used as the global image feature vector. Based on a preset balance weight, the fine-grained visual similarity and the global text retrieval similarity are linearly weighted and fused to generate a final retrieval score. The final retrieval score is then used to sort the candidate image set in descending order, and a preset number of images at the top of the sorted list are selected as the retrieval results.

8. The multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation according to claim 1, characterized in that, Calculating the fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map includes: ， ； ； in, This represents a fine-grained visual similarity between the background-foreground blended feature map and the dense visual feature map. For the proportion of the prospective area, For mask The number of foreground slices with a median value of 1. The total number of image slices, For adjustment coefficients, Forward weighting, This is the background weight.

9. The multimodal image retrieval method integrating fine-grained semantic localization and optimized feature generation according to claim 1, characterized in that, Calculating the global text feature similarity between the global text feature vector and the global visual feature vector in the description information includes: ； ； in, For dimension The target global text feature vector, For the text encoder's mapping function, Descriptive information for the target image. For global text feature similarity For the first Zhang candidate image feature vectors, It is the transpose of the vector.

Images