A Zero-Shot Image Retrieval Method and System Based on Bimodal Semantic Decoupling
By employing a bimodal semantic decoupling method, textual and visual semantics are decoupled to generate accurate visual proxy images. This solves the semantic entanglement problem in zero-shot image retrieval, achieves high-precision retrieval, and reduces deployment costs. It is suitable for multimodal search and digital asset management.
Patent Information
- Application Number
- CN202611593658.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from semantic entanglement problems in zero-shot image retrieval, leading to the dilution or tampering of visual details, making it impossible to accurately match the user's text modification intent, and relying on expensive triple datasets and costly model deployment.
A bimodal semantic decoupling method is adopted, which decouples text semantics into independent anchor points for retention, addition and removal through a multimodal large language model. Combined with a mask segmentation model, pixel-level masks are generated, and target visual proxy images are adaptively generated. Accurate retrieval is achieved through multidimensional retrieval and multimodal rearrangement.
It achieves high-precision zero-sample image retrieval, avoids tampering with visual details, reduces deployment costs, has zero-sample generalization capability, and is suitable for scenarios such as e-commerce multimodal search, digital asset management, and intelligent embodied agents.
Smart Images

Figure CN123115446A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and natural language processing, and in particular to a zero-shot image retrieval method and system based on bimodal semantic decoupling. Background Technology
[0002] An ideal zero-shot combined image retrieval system must accurately execute the text modification intent while faithfully preserving the fine-grained visual details in the reference image that do not need modification. However, the two major existing technical paradigms are both hampered by the underlying structural defects of semantic entanglement: (1) Non-multimodal large language model methods: These methods use text inversion or geometric interpolation to crudely compress and fuse image and text features into a single vector. This "rigid feature fusion" causes the original image details (such as texture, lighting, and background structure) to be severely diluted or even irreversibly altered during the dimensionality reduction process; at the same time, the user's text modification intent is easily masked by the powerful global visual prior, resulting in a very high error rate when the model processes complex modifications. (2) Multimodal large language model (MLLM-based) methods: This method bridges the semantic gap between multimodal approaches by generating intermediate proxies for text or vision. However, text-based proxies, limited by the bandwidth of language expression, struggle to perfectly replicate unstructured visual details. Visual proxies (such as those combined with diffusion models) suffer from the problem of "unconstrained global generation," where their global redrawing mechanism aggressively overwrites the underlying structure and context of the original image, failing to strictly limit modifications to the target region. It can be seen that whether performing feature compression or generating intermediate proxies, existing technologies ultimately suffer from the systemic problem of "semantic entanglement," leading to misleading results from spurious relevance in high-dimensional space and matching erroneous images that disrupt the original visual consistency. Therefore, how to completely decouple stable visual priors from dynamic textual instructions at the computational mechanism level, effectively prevent unconstrained alteration of original details, and achieve accurate alignment of multi-dimensional features in a completely training-free architecture constitutes a key technical challenge that urgently needs to be overcome in the current field of cross-modal retrieval. Summary of the Invention
[0003] To overcome the shortcomings of the prior art, the present invention provides a zero-shot image retrieval method and system based on bimodal semantic decoupling, aiming to solve the "semantic entanglement" problem in zero-shot image retrieval.
[0004] To achieve the above objectives, the present invention adopts the following technical solution, including: A zero-shot image retrieval method based on bimodal semantic decoupling, the method comprising: S1, Text Semantic Decoupling: Based on the reference image and the modified text, the modified text is decoupled into three independent text semantic anchors: retention, addition, and removal, using a multimodal large language model; S2, Visual-Semantic Decoupling: Based on the reference image and the modified text, a multimodal large language model is used to predict the modification intent type and the target spatial bounding box. A mask segmentation model is used to generate a pixel-level target mask, and an editing mask is adaptively generated according to the modification intent. Then, the target visual proxy image is synthesized under the mask constraint. S3, Multidimensional Retrieval: Calculate the similarity scores between the candidate image features and the semantic anchor features of each text and the visual proxy image features in the candidate image database, and calculate the initial retrieval score based on each similarity score to recall the top N candidate images; S4, Multimodal Reordering: The reference image, modified text, and the top N recalled candidate images are constructed into an interleaved sequence, which is then input into a multimodal large language model to output the final image retrieval ranking result.
[0005] Preferably, in step S1, a multimodal large language model is used as a structured intent decoupler. The multimodal large language model outputs structured information containing three independent key values, constrained by task-specific prompts, thereby generating three independent text semantic anchors. Subsequently, a text encoder is used... Extract the feature vectors of the three text semantic anchors respectively; The three independent text semantic anchors include: a retention anchor, used to describe the visual priors and background context in the reference image that are expected to be retained in the target image; an addition anchor, used to extract the semantics of newly added entities, attributes, or actions specified in the modified text; and a removal anchor, used to indicate misleading features that exist in the reference image but need to be discarded in the target image.
[0006] Preferably, step S2 is as follows: S21, Spatial Visual Grounding and Edit Mask Generation The multimodal large language model first classifies the modification intent c into: object modification, background modification, or global modification; For object modifications, the multimodal large language model predicts a normalized target spatial bounding box B and generates a local visual description of the target spatial bounding box region. ; Set the target space bounding box As input to the mask segmentation model, a pixel-level initial target mask is extracted. , ,in, , The image's height and width resolutions are defined, and the pixel value of the target object is 1. The final edit mask is adaptively generated based on the modification intention c. : When the modification intent c is to modify an object. ; When the modification intent c is to modify the background. ; When the modification intent c is a global modification ; S22, Target visual proxy image generation Get edit mask Then, using a diffusion model, local visual descriptions are obtained. Guide and edit mask Under the dual influence of pixel constraints, the diffusion model affects the reference image. Perform a denoising diffusion process to obtain the target visual proxy image. ; Using an image encoder Extracting target visual proxy image eigenvectors .
[0007] Preferably, in step S22, the latent space evolution of the constrained diffusion model only occurs when the mask is edited. Diffusion redrawing is performed within the defined pixel area; in each round of the diffusion iteration, the reference image is redrawn. In the pixel regions that do not require modification, the gradient inverse update path and the intrusion path of random noise in the latent space are cut off.
[0008] Preferably, step S3 is as follows: Using an image encoder Extract candidate images eigenvectors Calculate the similarity scores between the candidate image features and the three text semantic anchor features of retention, addition, and removal, respectively. , , And calculate the similarity score between the candidate image features and the target visual proxy image features. ; Calculate the text multidimensional retrieval score using a multidimensional score alignment strategy. ;in, , , These are weight hyperparameters; Then combine the similarity scores between the candidate image features and the target visual proxy image features. Calculate the initial search score ;in, To balance the hyperparameters; Retrieve the top N candidate images from the candidate image database, sorted in descending order of their initial retrieval scores.
[0009] Preferably, step S4 is as follows: Reference image Edit text The first N candidate images form the initial candidate set. The sequences are constructed as independent interleaved visual and text sequences and directly input into a multimodal large language model that acts as a rearranger, in order to perform the reordering task prompts. Guided by the algorithm, a global correlation analysis of all candidate images in the initial candidate set is completed through a single forward propagation. Attention feature cross-operation is performed between candidate images, and all candidate images are evaluated simultaneously. The corrected optimal candidate image ranking index is then output. This serves as the final image retrieval and ranking result.
[0010] This invention also provides a zero-shot image retrieval system based on bimodal semantic decoupling, applicable to the aforementioned zero-shot image retrieval method based on bimodal semantic decoupling. The system includes: The text semantic decoupling module is used to decouple the modified text into three independent text semantic anchors: retain, add, and remove, based on the reference image and the modified text, using a multimodal large language model. The visual semantic decoupling module is used to predict the modification intent type and target spatial bounding box based on the reference image and the modified text using a multimodal large language model, generate a pixel-level target mask using a mask segmentation model, and adaptively generate an edit mask based on the modification intent. Then, the target visual proxy image is synthesized under the mask constraint. The multidimensional retrieval module calculates similarity scores between candidate images in the candidate image database and each text semantic anchor feature and target visual proxy image feature. Based on these similarity scores, it calculates an initial retrieval score and performs pre-retrieval retrieval. 1 candidate image; The multimodal rearrangement module is used to combine the reference image, modified text, and recalled previous text. The candidate images are constructed into an interleaved sequence, input into a multimodal large language model, and the final image retrieval ranking result is output.
[0011] The present invention also provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the aforementioned zero-sample image retrieval method based on bimodal semantic decoupling.
[0012] The present invention also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned zero-sample image retrieval method based on bimodal semantic decoupling.
[0013] The present invention also provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the aforementioned zero-sample image retrieval method based on bimodal semantic decoupling.
[0014] The advantages of this invention are: (1) This invention avoids feature cancellation by independently extracting the added and deleted attention maps of the text; constrains local visual synthesis to prevent damage to the details of the original image; constructs a training-free architecture to accurately aggregate and decouple image and text features to achieve high-precision retrieval; and solves the "semantic entanglement" problem in zero-sample image retrieval.
[0015] (2) This invention aims to accurately retrieve target images in unseen open-domain databases using a combination of reference images and modified text. Unlike traditional combined image retrieval methods that rely on expensive and massive amounts of manually labeled triples, this method has zero-shot generalization capability and can be applied directly without fine-tuning. This technology has enormous commercial application value in scenarios such as next-generation e-commerce multimodal search, digital asset management, and intelligent embodied agents.
[0016] (3) This invention eliminates semantic entanglement by physically isolating the intent of “retaining, adding, and removing” and aligning the fractional levels, avoiding the annihilation of fine-grained information caused by early feature fusion, and greatly improving the recognition of complex modifications.
[0017] (4) The present invention introduces pixel-level masking to completely abandon global redrawing, strictly limit local generation, and absolutely protect the structure and details of the unrelated areas of the original image from being tampered with, perfectly preserving visual priors.
[0018] (5) The present invention directly coordinates the scheduling of pre-trained models such as MLLM, CLIP, and Diff at the bottom layer, without the need to build expensive triplet datasets or fine-tuning, which greatly reduces the deployment and migration costs of open domain.
[0019] (6) This invention achieves the isolation and reconstruction of bimodal features by introducing a cognitive decomposition mechanism, thus breaking through the bottleneck of the prior art.
[0020] (7) In the rigorous zero-sample benchmark tests such as CIRR and FashionIQ, the core indicators of this invention significantly outperform the existing state-of-the-art methods, demonstrating strong robustness and a leapfrog improvement in accuracy. Attached Figure Description
[0021] Figure 1 This diagram illustrates a comparison of the principles of the bimodal semantic decoupling method of the present invention with existing non-multimodal large language model methods and methods based on multimodal large language models. Figure 1 In the table, (a) represents an existing non-multimodal large language model method, (b) represents an existing multimodal large language model-based method, and (c) represents the bimodal semantic decoupling method of the present invention.
[0022] Figure 2 This is a system architecture flowchart of the bimodal semantic decoupling and training-free framework of the present invention.
[0023] Figure 3 This is a flowchart of a zero-sample image retrieval method based on bimodal semantic decoupling according to the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] This invention proposes a zero-shot image retrieval method based on bimodal semantic decoupling. This method overturns the traditional approach of blindly compressing combined queries into a single feature vector, formalizing it in the computational space as a process of multi-dimensional distance alignment and weighted game theory using multiple independent cross-modal semantic anchors. Given an initial reference image... and modified text containing complex intentions The combined query request aims to accurately extract the target image that satisfies all multimodal constraints from a large database D of unseen candidate images. Traditional retrieval methods rely excessively on early fusion paradigms. Directly searching for the maximum cosine similarity leads to a common feature collapse phenomenon; this invention, however, completely reconstructs the matching equation of the feature space through a decoupled architecture.
[0026] Figure 2 This is a system architecture flowchart of the bimodal semantic decoupling training-free framework of the present invention. Figure 2 It provides a detailed demonstration of the interaction and data flow of text semantic decoupling, visual semantic decoupling, multidimensional retrieval, and multimodal rearrangement. Figure 3 This is a flowchart of a zero-sample image retrieval method based on bimodal semantic decoupling according to the present invention.
[0027] Depend on Figure 2 and Figure 3 As shown, the zero-shot image retrieval method based on bimodal semantic decoupling of the present invention includes the following steps: S1 performs textual semantic decoupling to extract positive and negative orthogonal features.
[0028] The purpose of this step is to dismantle modified text containing complex intentions. The semantic self-contradictions and entanglements that often exist in the text are reduced in dimension and recombined into several absolutely orthogonal semantic indicator vectors.
[0029] S11, structured instruction decomposition guided by a multimodal large language model.
[0030] To establish unambiguous search directions, the system treats it not merely as simple text processing, but introduces a multimodal large language model (MLLM) with advanced instruction compliance capabilities as a structured intent decoupling mechanism. Task-specific high-order cue words, optimized through multiple rounds, are injected into this MLLM. Forced Large Model (MLLM) to read reference images The global semantics, and in-depth analysis of the modified text. Due to local formatting requirements, the final output is a JSON parsing result with an immutable format containing three specific dictionary key-value pairs: ; in, These are three text semantic anchors: retain, add, and remove. These are task-specific hints.
[0031] In the decoding protocol, task-specific prompts The internal instructions impose strict system-level cognitive constraints on the three output slots: (1) Preserve the semantic dimension of anchor points Used to capture and solidify stable visual priors. Large models are forced to discard all terms related to modification, describing the reference image purely and extremely objectively. The visible physical properties, large-scale topology, and macroscopic background context that are expected to be inherited one-to-one in the final target image are expected to be inherited. (2) Add semantic dimension anchors : Used to add incremental value to the retrieval system. Large models extract modified text from users that contains complex intents. In this process, newly introduced entities, color attributes, or dynamic action states are explicitly stripped away and isolated, severing their connection with other textual vocabulary. (3) Remove the anchor points in the semantic dimension Used to endow retrieval systems with filtering capabilities. Large models must think in reverse and actively identify reference images. Visual features that originally existed but have become outdated, contradictory, or must be completely discarded under the logic of new user intentions.
[0032] By imposing strong structural constraints at the end of text generation in a large model, the system physically cuts off the possibility of mutual interference and contamination between different semantics at the level of text symbols before the features enter the encoder.
[0033] S12, Vectorized extraction of independent anchor points in feature space.
[0034] To maintain the absolute purity of each decoupling intention in the high-dimensional latent semantic space of the large model, the system employs a text encoder based on a contrastive language image pre-trained model (CLIP) with strong alignment capabilities. Each of the three decoupled plain text strings (anchors) is independently encoded using tensors, resulting in three independent fixed-dimensional text semantic anchor feature vectors. : ; in, Let i be the feature vector of the text semantic anchor point i.
[0035] This engineering implementation, which physically separates semantics during the feature extraction stage, mathematically guarantees a core principle: the system adds semantic dimensions to anchor feature vectors. The activation and understanding process will never alter, overwrite, or contaminate the anchor feature vectors that preserve the semantic dimension at any computational step. At the cost.
[0036] S2 performs visual semantic decoupling, strictly synthesizing physically constrained local visual proxies.
[0037] This step abstracts and transforms abstract, difficult-to-quantify plain text editing instructions across modalities into high-dimensional, locally constrained entity visual image representations that can be directly visually aligned. The innovation of visual-semantic decoupling lies in introducing a high-precision pixel-level physical masking constraint mechanism into the image generation chain. It abandons the global image redrawing diffusion scheme commonly used in existing generative techniques, which has serious negative interference. It fundamentally avoids the problems of loss of original image texture and distortion of image details caused by the global redrawing diffusion process, and achieves complete and lossless preservation of original image texture information, greatly improving the accuracy and fidelity of image editing.
[0038] S21, Adaptive generation of dynamic spatial visual grounding and precise editing mask.
[0039] The system first deploys a multimodal large language model (MLLM) with strong cognitive logic and spatial understanding capabilities, serving as a macroscopic intent classifier and a microscopic spatial planner for the visual world, with an input reference image. Edit text and its dedicated spatial positioning prompts Large Model-Level Model (MLLM) performs rigorous classification and two-dimensional coordinate planning operations: ; The large model will modify the intent. The system is forcibly categorized into one of three categories with different subsequent processing permissions: Object, Background, and Global. Object, Background, and Global represent object modification, background modification, and global modification, respectively.
[0040] If the large model determines that the modification intention is a specific edit targeting a local area, i.e., the modification intention is an object modification, its built-in spatial planner will rely on its multimodal alignment capabilities to directly predict the target spatial bounding box, accurately normalized to the image's width and height, on the two-dimensional plane of the original image. , where x min x max The minimum and maximum x-coordinates of the rectangular bounding box are y and y, respectively. min y max These represent the minimum and maximum ordinates of the rectangular bounding box, respectively. Furthermore, the large model simultaneously outputs a dedicated local visual description of the region enclosed by bounding box B, stripped of all external environmental information. .
[0041] Furthermore, to achieve the system's goal of decoupling visual and physical features, deep analysis revealed that relying solely on the coarse rectangular bounding boxes output by the large model is extremely dangerous, as these boxes inevitably contain a large number of irrelevant background noise pixels that should not be modified. Therefore, the system uses the rectangular bounding box, i.e., the target spatial bounding box B, as an explicit spatial cue coordinate, inputting it into the basic mask segmentation model SAM. Through processing by the deep decoder of the SAM model, the system can extract a binary target physical mask matrix (initial target mask) that perfectly fits the edge contour of the target object, accurate to the single pixel level, from the coarse rectangular bounding box. ,in, , The image's height and width resolutions are represented by a pixel value of 1, indicating that the target object has been segmented and is allowed to be modified subsequently.
[0042] To ensure that the subsequent powerful diffusion generation network only uses computational power for the correct semantic regions, the final edit mask participating in the computation... An adaptive matrix transformation must be performed based on the previously modified classification results: When the intention to modify When modifying an object, ; When the intention to modify When modifying the background, ; When the intention to modify When making global modifications, .
[0043] When editing the pointed-to object, edit the mask. The system strictly covers the complex edges of the object entity; when the user command intends to modify the context, the system immediately performs a mathematical inversion by subtracting the target matrix from the matrix of all ones, thereby locking the modification range to the background; the high-dimensional matrix of the entire image canvas will be fully activated only if the system determines that this is a global modification to the overall environment style or filter.
[0044] S22, synthesize and extract the target visual proxy image under physical constraints.
[0045] An edit mask was constructed for spatial physical isolation. Subsequently, the system invokes a pre-trained large-scale high-resolution text-driven diffusion model, Diff, to synthesize a concrete visual prior proxy for the target. This is done within a specific local visual description. Guide and edit mask Under the dual influence of pixel constraints, the diffusion model Diff affects the reference image. Perform a denoising diffusion process to obtain the target visual proxy image. : ; This highly adaptive constrained synthesis process constrains the latent space evolution of the diffusion model Diff only when the edit mask is used. Diffusion redrawing is performed within a defined pixel area, thereby fusing and rendering the newly generated details to the corresponding image region. During each iteration of the diffusion process, for pixel regions in the reference image that do not require modification, the gradient back-update path and the intrusion path of latent space random noise are cut off, ensuring that this portion of the image data remains constant during model generation. Through this rigid constraint control mechanism, while making significant image modifications to the target region according to the editing intent, the microscopic details, light and shadow reflection features, and complex geometric topology of the original reference image are prevented from being damaged or distorted.
[0046] Subsequently, the system uses the image encoder of the Contrastive Language Image Pre-trained Model (CLIP). Extract the target visual proxy image Visual feature vectors .
[0047] S3, construct a multi-dimensional feature-weighted fusion retrieval system.
[0048] In the actual massive database scanning and retrieval phase, any candidate target image in database D... via consistent image encoder The extracted high-dimensional visual feature vector is .
[0049] S31, construct a delayed retrieval alignment and negative penalty mechanism based on fractional orders of magnitude.
[0050] Text semantic decoupling abandons the traditional feature concatenation distance calculation and instead calculates the features of the single candidate image independently. With three independent text semantic anchor features The cosine similarity on a high-dimensional hypersphere yields the similarity score. : ; Subsequently, the system employs a multi-dimensional score alignment strategy, dynamically aggregating these discrete scalar similarities into a text multi-dimensional retrieval score that characterizes the adaptability of plain text instructions through a multi-dimensional linear fractional alignment equation. : ; In this equation, , This represents the weight of positive incentives, while This represents the parameter indicating the severity of the negative penalty.
[0051] This mechanism completely eliminates the complexity and uncontrollability of the high-dimensional space in the vector fusion stage, directly implementing strategic control at the scalar score level. It explicitly rewards and significantly improves the ranking of candidate images that score extremely high in expected semantics. Simultaneously, through… This deduction factor imposes a hard score penalty on any candidate image detected to contain discarded visual features. This efficient negative sample mining logic achieves more sophisticated intent matching than traditional metric learning.
[0052] S32, using inner product calculation to visualize the target visual proxy image. Features Features of any candidate image in database D Visual distance is used to obtain a similarity score. : .
[0053] S33, Perform preliminary recall search.
[0054] In scenarios involving the initial screening and filtering of a massive (millions of) candidate image database D, to ensure system computational efficiency and reduce inference latency, this invention calculates an initial retrieval score for each candidate image to be scanned. Specifically, the initial retrieval score for a single candidate image is obtained using a dual-modal fusion scoring mechanism. This involves a finely tuned weighted fusion equation to deeply fuse the decoupled discrete abstract text semantic similarity with the concrete synthesized visual proxy similarity, ultimately outputting the initial retrieval score for the candidate image. : ; in, To balance the weight hyperparameters, the system is determined to either favor highly precise but easily misunderstood abstract instruction features, or to favor locally synthesized visual features that are WYSIWYG, contain specific spatial and shape information, but may be limited by the generator's own capabilities. The system's built-in massively parallel computation matrix engine is based on the initial retrieval scores of all candidate images. Perform a full descending sort to efficiently and completely extract the first element at the head with extremely low time complexity. A clean, high-density initial positive sample candidate set is constructed from potential target candidate images. .
[0055] S4 constructs the ultimate sequence rearrangement guided by a multimodal large language model, and performs lossless context-aware multimodal rearrangement.
[0056] While preliminary recall based on the superposition of multidimensional independent cosine similarity scalars is far superior to existing methods, the calculation of cosine angles in high-dimensional space still suffers from inherent geometric insensitivity when dealing with extremely challenging images containing extremely subtle differences in lighting and shadows, missing local components, intricate occlusions, and confusing spatial topological relationships. Therefore, to further improve retrieval accuracy, this invention introduces a computationally intensive, precise large-model comparison and ranking mechanism at the very end of the recall process.
[0057] Reference image used as the original starting point Modified text serving as the basis for the rules And the screened-out ones include the previous ones Candidate set of potential target candidate images A massive, interleaved, and time-consistent sequence of independent high-dimensional visual tokens (Visual Tokens) is constructed using image segmentation and serialization techniques, and used in reordering task prompts. Guided by system-level rules, the data is directly input into a multimodal large language model with ultra-deep attention head configuration in batch processing. Output the optimal candidate image sorting index. : ; Empowering a multimodal large language model The ability to perceive the entire visual sequence globally enables It can perform global correlation analysis of all candidate images in a single large-scale forward inference process, and simultaneously perform fine-grained attention feature cross-operations among multiple highly similar candidate images. By comparing subtle image features, it effectively avoids visual distortion and detail loss problems introduced during low-resolution image cropping and pooling, and latent space forced information compression, and outputs the optimal candidate image ranking index. .
[0058] Thus, this invention completes a full cross-modal retrieval loop, from parsing user semantic intent to accurately locating the target image, under a zero-sample working paradigm.
[0059] This invention provides a zero-shot image retrieval system based on bimodal semantic decoupling, applicable to the method described above. The system includes: The text semantic decoupling module is used to decouple the modified text into three independent text semantic anchors: retain, add, and remove, based on the reference image and the modified text, using a multimodal large language model. The visual semantic decoupling module is used to predict the modification intent type and target spatial bounding box based on the reference image and the modified text using a multimodal large language model, generate a pixel-level target mask using a mask segmentation model, and adaptively generate an edit mask based on the modification intent. Then, the target visual proxy image is synthesized under the mask constraint. The multidimensional retrieval module calculates similarity scores between candidate images in the candidate image database and each text semantic anchor feature and target visual proxy image feature. Based on these similarity scores, it calculates an initial retrieval score and performs pre-retrieval retrieval. 1 candidate image; The multimodal rearrangement module is used to combine the reference image, modified text, and recalled previous text. The candidate images are constructed into an interleaved sequence, input into a multimodal large language model, and the final image retrieval ranking result is output.
[0060] To further verify the effectiveness and technical advantages of the zero-sample image retrieval method and system based on bimodal semantic decoupling provided by this invention, the invention will be described below from three aspects: experimental setup, experimental data, and comparative analysis, in conjunction with specific experiments.
[0061] (1) Experimental setup.
[0062] Regarding datasets, the proposed method was evaluated using three zero-shot combined image retrieval benchmark datasets: Fashion-IQ, CIRR, and CIRCO. The Fashion-IQ dataset focuses on clothing-related retrieval, including three subcategories: Dress, Shirt, and Toptee, with a total of 30,135 combined query triples and 77,683 images. Evaluation metrics included Recall@10, Recall@50, and the average of the three subcategories. The CIRR dataset is the first combined image retrieval dataset for open-domain real-world images, with evaluation metrics including Recall@k (k values of 1, 5, 10, and 50) and RecallSubset@k (k values of 1, 2, and 3). The CIRCO dataset, built on the COCO 2017 unlabeled image set, is the first combined image retrieval dataset to provide multiple ground truth labels for each query, containing 123,403 database images and 800 test queries. Evaluation metrics included mAP@k (k values of 5, 10, 25, and 50).
[0063] In terms of implementation details, the method and system of this invention operate entirely in a training-free paradigm: the intent decoupler in the text semantic decoupling module, the planner and classifier in the visual semantic decoupling module, and the rearranger in the multimodal rearrangement module all use the same multimodal large language model as the inference engine; in the visual semantic decoupling module, a pixel-level segmentation model is used to extract the target mask, and a pre-trained text-driven diffusion model is used to synthesize local visual proxy images under mask constraints; the text semantic anchor features and candidate image visual features are extracted by a pre-trained encoder of a homologous contrastive language image model, and are evaluated on image encoder backbone networks of three specifications: ViT-B / 32, ViT-L / 14, and ViT-G / 14. The candidate pool size in the initial retrieval phase was uniformly set to N=16; the weight hyperparameters in the text multidimensional retrieval score were kept consistent across the three datasets and determined by grid search on the hold-out validation set; the visual fusion weights were set according to the image background complexity of each dataset, with Fashion-IQ having its weight appropriately increased due to the clean background and centered subject of the product image, in order to further amplify the contribution of visual proxy features.
[0064] (2) Comparative experimental results with existing technical methods.
[0065] On the CIRCO dataset, with the ViT-G / 14 backbone network, the method of this invention achieves an absolute improvement of approximately 4.6 percentage points in mAP@5 and approximately 3.1 percentage points in mAP@10 compared to the existing best training-free methods. It also leads in mAP@25 and mAP@50, which fully demonstrates that the present invention can effectively avoid the problem of different semantic features canceling each other out by explicitly isolating and retaining, adding, and removing three types of text semantic intents.
[0066] On the CIRR dataset, the method of this invention performs particularly well in the RecallSubset@k series of metrics that emphasize fine-grained visual consistency. Under the ViT-G / 14 backbone network, compared with the existing best methods, RecallSubset@1 is improved by about 1.1 percentage points, RecallSubset@2 by about 2.5 percentage points, and RecallSubset@3 by about 3.5 percentage points. This shows that the visual-semantic decoupling mechanism proposed in this invention can effectively preserve the original visual details of the unmodified areas in the reference image and avoid unnecessary damage to the texture of the original image caused by the synthesis process.
[0067] On the Fashion-IQ dataset, the method of this invention achieved the best results in terms of average index for three subclasses under the training free paradigm; under the ViT-L / 14 backbone network, the average Recall@50 was improved by about 3.7 percentage points compared with the existing best training-free method; under the ViT-G / 14 backbone network, the method of this invention even surpassed the method that relies on large-scale labeled triples for supervised training in terms of average Recall@10, indicating that the present invention can extract a purer retrieval signal than the data-driven supervised method without any fine-tuning.
[0068] (3) Ablation experiment analysis.
[0069] To verify the effectiveness of each technical module of this invention, ablation experiments were conducted on the CIRCO dataset and the ViT-G / 14 backbone network. Experimental results show that the baseline method, which directly performs early splicing and fusion of reference image features and modified text features, has the lowest retrieval accuracy. Based on this, introducing a separate text semantic decoupling module improves the mAP@5 index by approximately 9 percentage points, indicating that explicitly decoupling the three intentions of retention, addition, and removal into independent anchor points effectively prevents different semantic features from canceling each other out in the vector space. Introducing a separate visual semantic decoupling module improves the mAP@5 index by approximately 4.9 percentage points, indicating that synthesizing modification constraints within local regions avoids global generation from polluting the semantics of the original image. Introducing both modules further improves retrieval accuracy. Furthermore, adding a multimodal rearrangement module further improves the mAP@5 index by approximately 1.1 percentage points, demonstrating that performing fine-grained cross-image comparison directly in the native visual label space of candidate images effectively distinguishes visually highly similar candidate images with subtle differences.
[0070] Furthermore, an experiment was conducted to decompose the contributions of the three types of anchors within the text semantic decoupling module: using only the retained anchors and adding anchors already achieved an improvement of about 4.3 percentage points compared to the early fusion baseline. On this basis, after introducing the removal of anchors as a negative constraint, the retrieval accuracy achieved a significant improvement of about 4.7 percentage points, verifying the necessity of applying fractional penalties to discarded visual features in filtering false positive candidate images.
[0071] Furthermore, a comparative experiment was conducted to examine the impact of spatial positioning accuracy on the visual semantic decoupling module: compared to the global editing method without any spatial constraints, using only rectangular bounding boxes for coarse-grained positioning can bring an improvement of approximately 1.7 percentage points; and after further utilizing a segmentation model to extract pixel-level fine masks based on the rectangular bounding boxes, the retrieval accuracy is improved by approximately 3.2 percentage points. This indicates that pixel-level physical mask constraints, compared to coarse-grained rectangular box constraints, can more thoroughly prevent semantic leakage of the modified area to irrelevant background pixels.
[0072] (4) Effectiveness analysis of dynamic rollback mechanism.
[0073] Since the target visual proxy image synthesized by the visual-semantic decoupling module may fail to generate in rare cases due to spatial positioning errors or mask boundary overflow, this invention further sets up a dynamic backoff mechanism based on third-party confidence verification. This mechanism automatically decays the visual fusion weight to zero when the proxy image has low confidence, causing the retrieval process to revert to the pure text semantic decoupling score. Validation experiments on the CIRCO dataset and the ViT-G / 14 backbone network show that on the complete test set containing all 800 test queries, the introduction of the dynamic backoff mechanism results in a slight but stable improvement in all mAP metrics. Furthermore, on the subset of 80 hard queries with the lowest confidence levels selected based on confidence, the introduction of the dynamic backoff mechanism significantly increases the mAP@5 metric. This fully demonstrates that the mechanism can automatically revert to a more reliable pure text semantic decoupling matching path when the quality of the generated visual proxy image is unreliable, thereby ensuring the robustness of the overall retrieval performance of the system.
[0074] (5) Computational efficiency analysis.
[0075] Under identical experimental conditions, the computational overhead of the method of this invention is compared with that of existing training-free methods that use closed-source large model interfaces for inference. The results show that the method of this invention runs entirely on a locally controllable medium-sized open-source model. While maintaining low GPU memory usage, the number of calls to the multimodal large language model and generative model required is reduced by about half compared to the existing best training-free methods. The inference time of the core retrieval step for a single query is reduced by about 16% compared to the existing best training-free methods. Simultaneously, an absolute improvement of about 4.6 percentage points is achieved in the mAP@5 metric. This indicates that the present invention achieves simultaneous improvement in retrieval accuracy and inference efficiency without relying on high-cost closed-source interface calls. Furthermore, the visual proxy image synthesis step can be executed separately as an offline asynchronous preprocessing step on the query side, without being included in the real-time latency of online retrieval, thus ensuring the scalability of the method of this invention in large-scale image database scenarios.
[0076] In summary, the experimental data and comparative analysis of the above examples demonstrate that the zero-shot image retrieval method and system based on bimodal semantic decoupling provided by this invention achieves superior retrieval accuracy compared to existing training-free methods on multiple standard benchmark datasets such as CIRR, Fashion-IQ, and CIRCO. Furthermore, it exhibits significant advantages in computational efficiency. This verifies that the bimodal semantic decoupling architecture of this invention, which combines textual and visual semantic decoupling, effectively solves the semantic entanglement problem in existing technologies, demonstrating excellent technical effectiveness and practical value.
[0077] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A zero-shot image retrieval method based on bimodal semantic decoupling, characterized in that, The methods include: S1, Text Semantic Decoupling: Based on the reference image and the modified text, the modified text is decoupled into three independent text semantic anchors: retention, addition, and removal, using a multimodal large language model; S2, Visual-Semantic Decoupling: Based on the reference image and the modified text, a multimodal large language model is used to predict the modification intent type and the target spatial bounding box. A mask segmentation model is used to generate a pixel-level target mask, and an editing mask is adaptively generated according to the modification intent. Then, the target visual proxy image is synthesized under the mask constraint. S3, Multidimensional Retrieval: Calculate the similarity scores between the candidate image features and the semantic anchor features of each text and the visual proxy image features in the candidate image database, and calculate the initial retrieval score based on each similarity score to recall the top N candidate images; S4, Multimodal Reordering: The reference image, modified text, and the top N recalled candidate images are constructed into an interleaved sequence, which is then input into a multimodal large language model to output the final image retrieval ranking result.
2. The zero-shot image retrieval method based on bimodal semantic decoupling according to claim 1, characterized in that, In step S1, a multimodal large language model is used as a structured intent decoupler. Task-specific prompts constrain the multimodal large language model's output to contain structured information with three independent key values, thereby generating three independent text semantic anchors. Subsequently, a text encoder is used... Extract the feature vectors of the three text semantic anchors respectively; The three independent textual semantic anchors include: a preservation anchor, which describes the visual priors and background context in the reference image that are expected to be preserved in the target image; Add anchor points to extract newly added entities, attributes, or action semantics specified in the modified text; remove anchor points to indicate misleading features that exist in the reference image but need to be discarded in the target image.
3. The zero-shot image retrieval method based on bimodal semantic decoupling according to claim 1, characterized in that, Step S2 is detailed below: S21, Spatial Visual Grounding and Edit Mask Generation The multimodal large language model first classifies the modification intent c into: object modification, background modification, or global modification; For object modifications, the multimodal large language model predicts a normalized target spatial bounding box B and generates a local visual description of the target spatial bounding box region. ; Set the target space bounding box As input to the mask segmentation model, a pixel-level initial target mask is extracted. , ,in, , The image's height and width resolutions are defined, and the pixel value of the target object is 1. The final edit mask is adaptively generated based on the modification intention c. : When the modification intent c is to modify an object. ; When the modification intent c is to modify the background. ; When the modification intent c is a global modification. ; S22, Target visual proxy image generation Get edit mask Then, using a diffusion model, local visual descriptions are obtained. Guide and edit mask Under the dual influence of pixel constraints, the diffusion model affects the reference image. Perform a denoising diffusion process to obtain the target visual proxy image. ; Using an image encoder Extracting target visual proxy image eigenvectors .
4. The zero-shot image retrieval method based on bimodal semantic decoupling according to claim 3, characterized in that, In step S22, the latent space evolution of the constrained diffusion model only occurs when the mask is edited. Diffusion redrawing is performed within the defined pixel area; in each round of the diffusion iteration, the reference image is redrawn. In the pixel regions that do not require modification, the gradient inverse update path and the intrusion path of random noise in the latent space are cut off.
5. The zero-shot image retrieval method based on bimodal semantic decoupling according to claim 1, characterized in that, Step S3 is as follows: Using an image encoder Extract candidate images eigenvectors Calculate the similarity scores between the candidate image features and the three text semantic anchor features of retention, addition, and removal. , , And calculate the similarity score between the candidate image features and the target visual proxy image features. ; Calculate the text multidimensional retrieval score using a multidimensional score alignment strategy. ;in, , , These are weight hyperparameters; Then combine the similarity scores between the candidate image features and the target visual proxy image features. Calculate the initial search score ;in, To balance the hyperparameters; Retrieve the top N candidate images from the candidate image database, sorted in descending order of their initial retrieval scores.
6. The zero-shot image retrieval method based on bimodal semantic decoupling according to claim 1, characterized in that, Step S4 is as follows: Reference image Edit text The first N candidate images form the initial candidate set. The sequences are constructed as independent interleaved visual and text sequences and directly input into a multimodal large language model that acts as a rearranger, in order to perform the reordering task prompts. Guided by the algorithm, a global correlation analysis of all candidate images in the initial candidate set is completed through a single forward propagation. Attention feature cross-operation is performed between candidate images, and all candidate images are evaluated. The corrected optimal candidate image ranking index is then output as the final image retrieval ranking result.
7. A zero-shot image retrieval system based on bimodal semantic decoupling, characterized in that, A zero-shot image retrieval method based on bimodal semantic decoupling, applicable to any one of claims 1-6, comprises: The text semantic decoupling module is used to decouple the modified text into three independent text semantic anchors: retain, add, and remove, based on the reference image and the modified text, using a multimodal large language model. The visual semantic decoupling module is used to predict the modification intent type and target spatial bounding box based on the reference image and the modified text using a multimodal large language model, generate a pixel-level target mask using a mask segmentation model, and adaptively generate an edit mask based on the modification intent. Then, the target visual proxy image is synthesized under the mask constraint. The multidimensional retrieval module calculates similarity scores between candidate images in the candidate image database and each text semantic anchor feature and target visual proxy image feature. Based on these similarity scores, it calculates an initial retrieval score and performs pre-retrieval retrieval. 1 candidate image; The multimodal rearrangement module is used to combine the reference image, modified text, and recalled previous text. The candidate images are constructed into an interleaved sequence, input into a multimodal large language model, and the final image retrieval ranking result is output.
8. A readable storage medium, characterized in that, It stores a computer program, which, when executed, implements a zero-sample image retrieval method based on bimodal semantic decoupling as described in any one of claims 1-6.
9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the zero-sample image retrieval method based on bimodal semantic decoupling as described in any one of claims 1-6.
10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the zero-sample image retrieval method based on bimodal semantic decoupling as described in any one of claims 1-6.
