Dual-balance-based combined image retrieval method, device and equipment and storage medium

Abstract

The application relates to a dual-balance-based combined image retrieval method and device, equipment and a storage medium, wherein the method comprises the following steps: acquiring unannotated training texts, constructing text pseudo-triplets, and encoding the texts to obtain triplet features; adding modal alignment noise to reference text features and target text features; performing feature fusion on the noise-added reference text features and modified text features to obtain predicted query features; constructing a mask text, projecting the noise-added target text features to obtain pseudo-labeled text embedding, and constructing predicted features based on the pseudo-labeled text and the mask text; performing adaptive residual fusion on the predicted query features and the predicted features to obtain fused features; training a model according to the features to obtain a combined image retrieval model; and performing retrieval based on a to-be-processed reference image and a to-be-processed modified text by using the combined image retrieval model to obtain a target image. The application can improve the combined image retrieval precision and adapt to complex real scene application requirements.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dual-balanced combined image retrieval method, apparatus, device and storage medium. Background Technology

[0002] As a core application of multimodal information processing, combined image retrieval technology has wide-ranging value in practical scenarios such as fashion recommendation and interactive search. Its core objective is to accurately retrieve target images that retain the original image style while conforming to the modification intent, based on user-provided reference images and text modification instructions. To overcome the high dependence of traditional supervised methods on manually labeled triplet data, the zero-shot combined image retrieval paradigm has emerged. Among them, the CLIP-based text inversion strategy and pure text training method significantly reduce data collection and labeling costs and improve the practicality and scalability of the method by using only unlabeled text data for model training.

[0003] Existing zero-shot combined image retrieval techniques trained on pure text still face deep-seated contradictions: On the one hand, there is a significant imbalance between generalization ability and task specialization ability. Since existing methods mainly rely on text inversion strategies to map text embeddings to the word space, their training process is essentially a "text-to-text" transformation, lacking an explicit visual-semantic alignment supervision mechanism. This makes it difficult for the model to effectively model cross-modal feature interactions and fine-grained attribute modifications. Although such methods show strong generalization adaptability in general scenarios, their task specialization ability is clearly insufficient when dealing with complex combined queries, and the retrieval accuracy is difficult to meet practical needs. On the other hand, there is a trade-off between training efficiency and retrieval performance. This presents an irreconcilable trade-off. While existing methods offer high computational efficiency, their reliance on text input during training leads to a lack of detailed visual features from images. To improve performance, long text descriptions are often necessary. However, the effective tag length limit of mainstream encoders like CLIP (typically no more than 77 tags) results in significant truncation and loss of crucial semantic information in long texts during encoding. This prevents the model from fully parsing complex modification instructions, severely restricting the upper limit of retrieval performance. Introducing large language models to generate pseudo-triples to compensate for information loss would significantly increase training time and resource consumption, thus undermining the inherent efficiency advantage of pure text training. These mutually constraining technical bottlenecks prevent existing solutions from maintaining training efficiency while simultaneously achieving a dynamic balance between generalization and task specialization, and effectively utilizing the rich semantic information in long texts. This limits the depth and breadth of combined image retrieval techniques in complex real-world scenarios. Summary of the Invention

[0004] The purpose of this application is to propose a dual-balanced combined image retrieval method, apparatus, device, and storage medium that can maintain high training efficiency and low annotation cost while taking into account the dynamic balance between generalization ability and task specialization ability, effectively improving the accuracy of combined image retrieval and adapting to the application needs of complex real-world scenarios.

[0005] To address the aforementioned technical problems, embodiments of this application provide a dual-balanced combined image retrieval method, comprising: Obtain unlabeled training text, and construct text pseudo triples based on the unlabeled training text using a pseudo triple construction strategy driven by a controllable function; The text pseudo triples are encoded to obtain triple features, and modality alignment noise is added to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features. The feature fusion network is used to perform feature fusion processing on the noisy reference text features and the modified text features in the triplet features to obtain the predicted query features; Keywords are extracted from the unlabeled training text to construct masked text, and the features of the noisy target text are projected through a mapping network to obtain pseudo-labeled text embeddings. Predictive features are constructed based on the pseudo-labeled text and the masked text. The predicted query features and the predicted features are fused by nonlinear residual interaction and linear adaptive weighted fusion through the residual fusion module to obtain fused features. The loss value is calculated based on the fused features and the target text features, and the trainable parameters in the feature fusion network, the mapping network and the residual fusion module are updated based on the loss value to train the model and obtain the combined image retrieval model. The system acquires a reference image to be processed and text to be modified, and performs image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

[0006] To address the aforementioned technical problems, embodiments of this application provide a dual-balanced combined image retrieval device, comprising: The text pseudo-triple construction module is used to obtain unlabeled training text and construct text pseudo-triples based on the unlabeled training text through a pseudo-triple construction strategy driven by a controllable function. The modal noise addition module is used to encode the text pseudo triples to obtain triple features, and add modal alignment noise to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features. The feature fusion processing module is used to perform feature fusion processing on the noisy reference text features and the modified text features in the triplet features through a feature fusion network to obtain the predicted query features; The prediction feature construction module is used to extract keywords from the unlabeled training text to construct masked text, and to project the features of the noisy target text through a mapping network to obtain pseudo-labeled text embeddings. Based on the pseudo-labeled text and the masked text, prediction features are constructed. The feature weighted fusion module is used to perform nonlinear residual interaction and linear adaptive weighted fusion on the predicted query features and the predicted features through the residual fusion module to obtain fused features; The retrieval model training module is used to calculate the loss value based on the fused features and the target text features, and update the trainable parameters in the feature fusion network, the mapping network and the residual fusion module based on the loss value to train the model and obtain the combined image retrieval model. The target image retrieval module is used to acquire a reference image to be processed and text to be modified, and to perform image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

[0007] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the dual-balanced combined image retrieval method described in any one of the above-mentioned methods.

[0008] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the dual-balanced combined image retrieval method described above.

[0009] This invention provides a dual-balanced combined image retrieval method, apparatus, device, and storage medium. By constructing pseudo triples based on unlabeled text to complete model training, and combining nonlinear residual interaction with linear adaptive weighted feature fusion, it can balance generalization ability and task specialization ability without increasing training resource consumption. It solves the problem of difficulty in balancing training efficiency and retrieval performance in the prior art. It has the advantages of maintaining high training efficiency and low labeling cost, while improving the accuracy of combined image retrieval and adapting to the application needs of complex real-world scenarios. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of an implementation of the dual-balanced combined image retrieval method provided in this application embodiment; Figure 2 This is a schematic diagram of the training framework structure of the combined image retrieval model provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the implementation of the first sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 4 This is a flowchart illustrating the implementation of the second sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 5 This is a flowchart illustrating the implementation of the third sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 6 This is a flowchart illustrating the implementation of the fourth sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 7 This is a flowchart illustrating the implementation of the fifth sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 8 This is a flowchart illustrating the implementation of the sixth sub-process in the dual-balanced combined image retrieval method provided in this application embodiment; Figure 9 This is a schematic diagram of the reasoning process of the combined image retrieval model provided in the embodiments of this application; Figure 10 This is a schematic diagram of a dual-balanced combined image retrieval device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] Traditional combined image retrieval methods, especially zero-shot combined image retrieval methods trained on pure text, suffer from an imbalance between generalization ability and task specialization ability when handling multimodal queries, making it difficult to effectively model cross-modal feature interactions and fine-grained attribute modifications. Furthermore, there is a trade-off between training efficiency and retrieval performance; semantic information in long texts is easily truncated and lost due to encoder limitations, severely restricting the upper limit of retrieval performance.

[0015] To address this issue, this application proposes a dual-balanced combined image retrieval method. It constructs text pseudo-triples using a controllable function-driven pseudo-triple construction strategy and adds modality alignment noise to both reference and target text features. This method further processes the noisy reference and modified text features through a feature fusion network, extracts keywords from unlabeled training text to construct masked text, and combines this with a mapping network to project the noisy target text features to obtain pseudo-labeled text embeddings, thereby constructing predicted features. Finally, a residual fusion module performs nonlinear residual interaction and linear adaptive weighted fusion on the predicted query features and predicted features to obtain fused features. Model training is then based on these fused features to balance generalization and task specialization capabilities while maintaining the efficiency of pure text training, effectively utilizing the semantic information of long texts, and improving the performance of combined image retrieval.

[0016] For ease of understanding, the following explains some key terms in this embodiment: The combined image retrieval method aims to retrieve target images from an image library that retain the style of the reference image while meeting the user's modification intent, based on a multimodal query consisting of a reference image and modified text.

[0017] Unlabeled training text refers to raw text data that has not been manually labeled. It does not contain any explicit label information about images, text pairs, or triples and is used only as training input for the model.

[0018] The controllable function-driven pseudo-triple construction strategy is a method that automatically generates pseudo-triple data containing reference text, modified text, and target text from unlabeled training text using preset functions or rules. This strategy aims to simulate real-world combinatorial query scenarios and provide supervision signals for model training.

[0019] A text pseudo triple consists of a reference text, a modified text, and a target text. The reference text describes an initial state or object, the modified text describes the intention to modify that state or object, and the target text describes the final state or object after modification.

[0020] Modal alignment noise refers to random perturbations introduced into text features to simulate the inherent differences or uncertainties between cross-modal data (such as images and text), prompting the model to learn robustness to these differences during training, thereby improving cross-modal alignment capabilities.

[0021] Feature fusion networks are a type of neural network architecture used to receive and integrate features from different sources or different types, and generate more expressive fused features by learning their intrinsic relationships.

[0022] Masked text refers to text formed by extracting key information from the original text and replacing part of the content with placeholders (such as [MASK]). It aims to guide the model to focus on the core semantics of the text while processing long text information.

[0023] Mapping networks are a type of neural network structure used to project or transform input features from one feature space to another, in order to achieve feature alignment, dimensionality reduction, or enhancement.

[0024] The residual fusion module is a feature fusion structure that combines the idea of ​​residual connection. It can perform nonlinear interaction and linear weighted fusion of multiple input features to effectively integrate the information of different features while avoiding the gradient vanishing problem.

[0025] A combined image retrieval model refers to a deep learning model that, after training, can receive reference images and modified text as input and retrieve target images from an image library that match the user's intent.

[0026] This application's embodiments effectively address the reliance on manually labeled data in traditional methods by employing a controllable function-driven pseudo-triple construction strategy, significantly reducing data acquisition and labeling costs. By introducing modal alignment noise and residual fusion modules, a balance is achieved between model generalization and task specialization capabilities, enhancing the modeling ability for cross-modal feature interactions and fine-grained attribute modifications. Simultaneously, through keyword extraction and masked text construction, the semantic information of long texts is effectively utilized, avoiding information truncation. This significantly improves the retrieval accuracy and performance ceiling of zero-shot combined image retrieval in complex scenarios such as fashion recommendations and interactive search, while maintaining the efficiency of pure text training.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0028] It should be noted that the dual-balanced combined image retrieval method provided in this application is generally executed by a server, and correspondingly, the dual-balanced combined image retrieval device is generally configured in the server.

[0029] Please see Figure 1 and Figure 2 , Figure 1 This paper illustrates a specific implementation of a dual-balanced combined image retrieval method. Figure 2 This is a schematic diagram of the training framework structure of the combined image retrieval model provided in the embodiments of this application.

[0030] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 Limited to the order of the processes shown, this method includes the following steps: S1: Obtain unlabeled training text and construct text pseudo-triples based on the unlabeled training text using a pseudo-triple construction strategy driven by a controllable function.

[0031] like Figure 2As shown, this application introduces two prediction branches with complementary inductive biases in parallel within a unified framework, namely a dual-branch collaborative mechanism. A Feature Fusion Branch (FFB) is introduced into the zero-shot method to achieve supervised learning by training text pseudo-triples, while the Text Inversion Branch (TIB) is retained to maintain generalization ability. Finally, an adaptive collaboration between the two is achieved through a residual fusion module, thereby balancing task specialization and generalization capabilities. Simultaneously, to maintain computational efficiency while ensuring training effectiveness, a lightweight Controllable Function-Driven (CFD) pseudo-triple construction strategy is designed for text pseudo-triple generation. Unlike existing methods, CFD relies solely on natural language processing tools, providing approximate supervision signals to the model without introducing a large model through controllable semantic parsing, noise injection, and text reconstruction, significantly reducing computational costs. Furthermore, to address the information loss problem in long text encoding of pure text training methods, this invention introduces Long-CLIP technology for the first time, extending the effective word length of the text to 248. Due to compatibility issues between its native architecture and the pure text training paradigm (such as the mask tag replacement mechanism), we further propose a Long-Text Adaptation (LTA) module to solve the compatibility problem between pseudo-tag replacement and mask.

[0032] To alleviate the imbalance between generalization and task specialization capabilities in pure text training methods, this application proposes a dual-branch collaborative mechanism. For example... Figure 2 As shown, the Feature Fusion Branch (FFB) on the left aims to simulate the feature interaction process in supervised combined image retrieval methods, enhancing the model's task specialization ability by training text pseudo-triples. Specifically, it first utilizes the CFD (Controllable Function Driven) method based on unlabeled training text. Constructing text pseudo triples ,in For reference text, To modify the text, For the target text.

[0033] Please see Figure 3 , Figure 3 A specific implementation of step S1 is shown below: S11: Obtain the unlabeled training text and extract keywords from the unlabeled training text, wherein the keywords include noun phrases, proper nouns, adjectives and qualifiers.

[0034] The embodiments of this application aim to identify units in text that carry core semantics and modification intentions, providing an accurate and efficient foundation for the subsequent construction of pseudo-trigrams. One implementation is to utilize the Spacy library, for example through part-of-speech tagging and syntactic analysis, to identify and extract noun phrases, proper nouns, adjectives, and determiners from the text. These word types are typically key information describing image content, attributes, and modification instructions. Another implementation is to employ a method based on a pre-trained language model, fine-tuning the model to perform named entity recognition (NER) or keyword extraction tasks, thereby automatically identifying and extracting keywords that conform to the above categories from unlabeled text.

[0035] S12: Randomly divide the keyword set into three mutually exclusive subsets, and replace the words in the selected mutually exclusive subsets with antonyms or related words to generate a new word set after noise word replacement.

[0036] The purpose of this application is to introduce controlled semantic changes at the keyword level to simulate the modification intent in combined image retrieval tasks and ensure clear semantic differences between the parts of the generated pseudo-trigrams. One implementation is to utilize a lexical resource library, such as WordNet or a thesaurus, to find antonyms or synonyms / related words for keywords. For example, for adjectives, antonyms can be found and replaced (e.g., "big" replaced with "small"); for nouns, related words or hypernyms can be found and replaced (e.g., "car" replaced with "vehicle"). Randomly partitioning mutually exclusive subsets ensures that the reference text, modified text, and target text can be constructed based on different semantic foundations when building pseudo-trigrams, avoiding semantic confusion. Another implementation is to utilize pre-trained word embedding models or large language models to find antonyms or related words by calculating the semantic distance between words. For example, cosine similarity can be used to find semantically similar words as related words, or specific prompt engineering can guide the language model to generate antonyms.

[0037] In a specific instance, antonym substitution achieves semantic reversal, such as "female child→boy", while "red→pink" simulates non-critical attribute noise through related word substitution.

[0038] S13: Reconstruct the text of the new word set to generate the text pseudo-triples, wherein the text pseudo-triples include the reference text, the modified text, and the target text.

[0039] This application aims to combine semantically processed keywords into structured pseudo-triples, making them directly conform to the training data format of the combined image retrieval task. One implementation is to pre-set a series of text reconstruction templates. For example, based on the type and substitution relationship of the keywords, words in the original keyword set can be used to construct reference text, replaced words can be used to construct target text, and phrases describing the substitution operation can be used as modified text.

[0040] In one specific embodiment, the new word set is reconstructed using rule operations such as deletion, addition, replacement, and modification, where "\" represents deletion. "" indicates semantic concatenation. The time complexity of the CFD algorithm is linearly related to the length of the input text, denoted as O(L), where L is the length of the text. The length.

[0041] This application, without requiring manual annotation or relying on large language models, achieves efficient and controllable construction of high-quality text pseudo-triples based on unannotated training text. First, by accurately extracting core keywords such as noun phrases, proper nouns, adjectives, and determiners, it ensures that the pseudo-triples focus on key semantic modification points in combined image retrieval tasks, thus providing more targeted supervision signals for the model. Second, by randomly dividing the keyword set into mutually exclusive subsets and replacing them with antonyms or related words, a clear modification intention and semantic difference are cleverly introduced at the text level, resulting in a clear logical relationship between the generated reference text, modified text, and target text, effectively simulating user instructions to modify images during actual retrieval. Finally, through text reconstruction, these processed keywords are organized into structured pseudo-triples, directly providing compliant input for subsequent model training. This not only significantly reduces the time and resource consumption of data preparation and maintains the efficiency of pure text training, but also effectively improves the quality of training samples and task specialization capabilities through this controllable pseudo-triple construction strategy. This enables the model to better learn cross-modal feature interactions and fine-grained attribute modifications, thereby improving the retrieval accuracy of the combined image retrieval model in complex combined query scenarios.

[0042] S2: Encode the text pseudo triples to obtain triple features, and add modal alignment noise to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features.

[0043] Specifically, a text encoder is used to process text pseudo-triples. Encoding is performed to obtain triplet features, and the specific formula is expressed as follows: ,in , For batch size, This is for hidden dimensions. Furthermore, due to the input during the inference phase... Representing reference image features, this application needs to address modal gaps to align the reference image features used in the inference process. Therefore, modal alignment noise is added to the reference text features and target text features in the triplet features to obtain noisy reference text features and noisy target text features.

[0044] Please see Figure 4 , Figure 4 A specific implementation of step S2 is shown below: S21: The text pseudo triples are encoded by a text encoder to obtain the triple features.

[0045] Specifically, encoding text pseudo-triples using a text encoder to obtain triple features involves using specialized models or algorithms for processing text information to convert the reference text, modified text, and target text within the text pseudo-triples into numerical vector representations with semantic information. These vectors collectively constitute the triple features, providing the foundation for subsequent noise addition and feature fusion. S22: Generate a first random variable and a second random variable that are independent of each other, wherein the first random variable follows a uniform distribution on the interval from 0 to 1, and the second random variable follows a standard normal distribution.

[0046] Specifically, the first random variable takes a uniform value between 0 and 1, and its main function is to control the overall amplitude or intensity of the final noise. The second random variable takes a value that follows a standard normal distribution (mean 0, variance 1), and its distribution shape is closer to many random phenomena in nature, providing random disturbances with good statistical properties. These two random variables are independent of each other, ensuring the flexibility and controllability of the noise generation process.

[0047] S23: Calculate the element-wise product of the first random variable and the second random variable to obtain the modal alignment noise.

[0048] Specifically, calculating the element-wise product of the first and second random variables to obtain modal alignment noise involves multiplying the two generated random variables element-wise at corresponding positions. This element-wise multiplication operation combines the amplitude control characteristics of a uniform distribution with the randomness of a normal distribution, generating a composite modal alignment noise. The amplitude of this noise is limited by a uniform distribution, avoiding excessive disturbances, while also possessing the randomness and naturalness of a normal distribution, making it closer to real modal differences.

[0049] S24: Add the modal alignment noise to the reference text feature and the target text feature respectively to obtain the noisy reference text feature and the noisy target text feature.

[0050] Specifically, the generated modal alignment noise is added element-wise to the original reference text feature vector and the target text feature vector. This addition operation introduces random perturbations into each dimension of the feature vector without changing the original dimensions and spatial structure of the features. The purpose is to simulate the small differences or alignment errors that may exist between different modalities, thereby introducing modal perturbations during the pure text training phase and forcing the model to learn robustness to these differences.

[0051] S3: The noisy reference text features and the modified text features in the triplet features are fused using a feature fusion network to obtain the predicted query features.

[0052] Specifically, using feature fusion networks Features of the reference image after adding noise Modifying text features Perform feature fusion learning to generate predictive query features. The specific formula is expressed as follows:

[0053] S4: Extract keywords from the unlabeled training text to construct masked text, and project the features of the noisy target text through a mapping network to obtain pseudo-labeled text embeddings. Construct predicted features based on the pseudo-labeled text and the masked text.

[0054] like Figure 2 The text inversion branch (TIB) on the right inherits the design philosophy of the zero-shot pure text training method, focusing on maintaining the model's generalization ability. In this embodiment, keywords are extracted from the unlabeled training text to construct masked text, and the features of the noisy target text are projected through a mapping network to obtain pseudo-labeled text embeddings. Predictive features are then constructed based on the pseudo-labeled text and the masked text.

[0055] Please see Figure 5 , Figure 5 A specific implementation of step S4 is shown below: S41: Extract keywords from the unlabeled training text to construct the masked text, and project the features of the noisy target text through the mapping network to obtain the pseudo-labeled text embedding.

[0056] Specifically, keyword extraction can employ various natural language processing (NLP) techniques. For example, keywords can be extracted from unlabeled training text using the Spacy NLP library. These keywords are typically noun phrases, proper nouns, adjectives, or determiners, carrying the core semantic information of the text. Masked text can be constructed by retaining the extracted keywords while replacing other non-keyword parts of the text with specific mask markers (such as "[MASK]" or "[UNK]"), thus forming a sparse text structure that highlights the keywords. Alternatively, the extracted keywords can be used as components of the masked text, combined into a simplified text fragment with a specific semantic meaning using a template or rule. The mapping network typically consists of one or more fully connected layers (multilayer perceptron, MLP), or it can be a neural network layer containing non-linear activation functions (such as ReLU, GeLU). Its function is to transform input features from one feature space to another to achieve feature dimensionality matching or semantic alignment. Projection processing refers to using a mapping network to perform nonlinear or linear transformations on the features of the noisy target text, mapping them from the original feature space to a new feature space compatible with the masked text embedding space. This process aims to transform the noisy target text features into a representation that can effectively integrate with the masked text structure, namely, pseudo-labeled text embeddings, making them semantically and dimensionally compatible with the placeholders in the masked text.

[0057] S42: Convert the masked text into a masked text embedding.

[0058] This application aims to transform structured masked text into a numerical vector representation that can be processed by a model. Common methods include: based on a pre-trained text encoder, a pre-trained Transformer model such as BERT, RoBERTa, or CLIP text encoder can be used. The masked text is taken as input, and its encoding layer outputs the corresponding context embedding vector. This method can capture the deep semantics and contextual relationships of words. Alternatively, word embedding models such as Word2Vec or GloVe can be used to map each word in the masked text to a pre-trained word vector, and then these word vectors are combined into the embedding of the entire masked text through averaging, weighted averaging, or sequence models (such as LSTM or GRU).

[0059] S43: Replace the placeholder markers in the mask text embedding with the pseudo-marked text embedding to obtain the replaced embedding features.

[0060] Specifically, the replacement operation is crucial for aligning the semantics of the pseudo-tags with the structure of the masked text. Specific implementation methods can include: direct replacement, where the position corresponding to a predefined placeholder marker (e.g., an embedding of a special token representing the masked word) is found in the masked text embedding, and the embedding vector at that position is directly replaced with the pseudo-tag text embedding. This method ensures that the semantic information of the pseudo-tags is accurately injected into specific structural positions within the masked text. Another approach is weighted fusion, where the pseudo-tag text embedding is weighted and summed with the original placeholder marker embedding at the position corresponding to the placeholder marker, or fused using a small network, to introduce the semantics of the pseudo-tags while preserving some of the original contextual information.

[0061] S44: The replaced embedded features are positionally embedded by hybrid positional encoding to obtain the embedded features.

[0062] Specifically, positional encoding aims to provide each element in a sequence with its relative or absolute position within the sequence, as models like the Transformer lack the ability to handle sequence order. Hybrid positional encoding implementations can include combining sinusoidal positional encoding with learned positional encoding, i.e., superimposing or concatenating the positional encoding generated by the traditional sine function with positional embeddings learned through a neural network. Sine positional encoding has the advantage of good generalization, while learned positional encoding is better adapted to the distribution of specific tasks. Alternatively, multi-scale positional encoding can be used, for example, considering both word-level and phrase-level positional information simultaneously, or combining absolute and relative positional encoding to provide richer positional context.

[0063] S45: The embedded features are normalized and projected to the joint visual and linguistic embedding space to generate the predicted features.

[0064] Specifically, layer normalization is a commonly used normalization technique that normalizes the input of each layer in a neural network to a mean of 0 and a variance of 1. This helps stabilize the training process, accelerate model convergence, and reduce sensitivity to initialization parameters. In implementation, it is typically applied after each Transformer layer or fully connected layer. The projection layer is usually a linear transformation layer (such as a fully connected layer) that adjusts the dimension of the embedded features to match the dimension of the joint visual and linguistic embedding space. For example, if the joint space has a dimension of D, the projection layer maps the input features from their current dimension to D dimensions. The joint visual and linguistic embedding space is a high-dimensional vector space where features from the visual modality (images) and the linguistic modality (text) are mapped to the same space, allowing for direct comparison and similarity calculation. In this space, semantically similar image and text feature vectors are closer together, thus supporting cross-modal retrieval tasks.

[0065] In one specific embodiment, to fully leverage the potential of the plain text training method for processing long text information, this application introduces a Long Text Adaptation (LTA) module, which effectively integrates Long-CLIP into the plain text training framework. This module is designed to correctly replace the mapping network. The generated pseudo-markers are embedded, maintaining architectural compatibility with the masked text conditional mechanism used in text inversion. In this embodiment, the masked text is first... Convert to mask markup embedding Then embedding using pseudo-tags Replace the corresponding embedding (ID=259, representing the placeholder "$"). Finally, obtain the replaced embedding features. (L = 248). This structured embedding replacement ensures consistency with the text inversion mechanism while avoiding conflicts with the Long-CLIP improved attention mask and its extended context window. To model long-distance dependencies, this application employs hybrid positional encoding, which integrates absolute and relative positional embeddings to effectively capture local and global dependencies. The specific formula is as follows: in, , This represents the learnable positional embedding matrix. , It is a binary mask; These are post-embedding features used to control the application of the embedding at specific locations, thereby enabling customized encoding of short-span and long-span regions. The operator " "" indicates element-wise multiplication broadcast along the batch and embedding dimensions. This design allows the initial 20 positions and the subsequent 228 positions to use independent position encoding schemes while maintaining dimensionality consistency with the original input.

[0066] Then embedding features After processing by the Transformer layer The hidden states are mapped to the joint visual-language embedding space through layer normalization and projection layers, ultimately generating predictive features. .

[0067] S5: The predicted query features and the predicted features are fused by nonlinear residual interaction and linear adaptive weighted fusion through the residual fusion module to obtain fused features.

[0068] Please see Figure 6 , Figure 6 A specific implementation of step S5 is shown below: S51: Concatenate the predicted query feature with the predicted feature, and input the concatenated feature into a multilayer perceptron for high-order interaction modeling to obtain residual features. S52: Calculate the linear fusion feature by weighting the predicted query feature and the predicted feature using learnable scalar weights. S53: Add the residual feature and the linear fusion feature, and then perform layer normalization to obtain the fusion feature.

[0069] The residual fusion module aims to effectively integrate features from different sources to generate a fused feature containing richer and more discriminative information. Its core function is to balance the preservation of original feature information with the modeling of complex interactions between features, thereby improving the model's ability to understand multimodal information. This module can consist of multiple sub-layers or sub-units, such as linear layers for feature transformation, activation function layers for non-linear activation, and pooling layers for feature aggregation. Predicted query features are obtained during model training by fusing the features of the noisy reference text and the modified text through a feature fusion network. They represent a preliminary understanding of the user's query intent (reference image and modified text). They are typically represented as high-dimensional vectors, encoding the semantic information of the query. Predicted features are constructed during model training by projecting the noisy target text features through a mapping network and combining them with masked text. They represent the model's prediction of the latent features of the target image. They also exist in high-dimensional vector form, aiming to capture the visual and semantic attributes of the target image.

[0070] Concatenation refers to joining two or more feature vectors along a certain dimension to form a longer feature vector. For example, predictive query features and predictive features can be concatenated along the feature dimension, allowing the information from both features to exist side-by-side in the same vector, providing a unified input for subsequent interaction modeling. Besides direct concatenation, other forms of feature combination can be used, such as element-wise multiplication or addition, but concatenation preserves the original information more completely. A Multilayer Perceptron (MLP) is a feedforward neural network consisting of at least one input layer, one output layer, and at least one hidden layer. Neurons in each layer are fully connected to all neurons in the next layer. MLPs, through non-linear activation functions, can learn and represent complex non-linear relationships, thereby achieving high-order interaction modeling of input features. This can be achieved by stacking multiple fully connected layers and inserting non-linear activation functions such as ReLU and GeLU between layers. High-order interaction modeling refers to capturing the non-linear, complex, and deep-level relationships between input features. Unlike simple linear combinations, high-order interaction modeling can reveal finer semantic dependencies and conditional relationships between features, which is crucial for understanding the complex interactions between reference images and modified text in combined image retrieval. Residual features are feature representations obtained after high-order interaction modeling using a multilayer perceptron. They encode incremental or differential information between the predicted query features and the predicted features after nonlinear transformation and interaction, aiming to capture deeper semantic relationships between them.

[0071] Learnable scalar weights are numerical parameters that the model automatically adjusts during training through optimization algorithms. These weights are used to measure the relative importance or contribution of different input features in the fusion process. For example, a learnable scalar weight can be set for the predicted query feature and the predicted feature, and the model will automatically learn how to allocate the weights of these two features based on the training data to achieve the best fusion effect. Weighted summation is a linear fusion method that multiplies each input feature by its corresponding learnable scalar weight and then sums all the weighted features to obtain a comprehensive feature. This method can adaptively adjust the contribution of features to the fusion result based on their importance. Linear fusion features are obtained by weighted summation of the predicted query feature and the predicted feature. It preserves the linear combination information of the original input features, ensuring that the fused features contain the basic semantic content of the original features. Layer normalization is a commonly used neural network normalization technique that normalizes all features of a single sample to a mean of 0 and a variance of 1. Layer normalization helps stabilize the training process of neural networks, accelerates convergence, and improves the model's generalization ability. The implementation typically involves applying the mean and standard deviation to the feature vectors, followed by scaling and translation.

[0072] Specifically, a simple two-layer MLP nonlinear network is used to model higher-order interactions and cross-branch correlations in the nonlinear residual path. Simultaneously, two learnable scalar weights are used. and By balancing linear combinations, the original data distribution of the predicted features is avoided, resulting in linearly fused features. The results are then summed and normalized to obtain the final predicted features. Finally, feature fusion. Through mean squared error loss and target features Constraints are imposed to achieve a dynamic balance between generalization ability and task specialization ability during training. This can be formally represented as: in, For residual characteristics, For linear fusion features, As a feature of fusion, , For learnable scalar weights, For predicting features, To predict query features.

[0073] From the perspective of representation learning, predict query features and predictive features These respectively reflect different inductive biases: It focuses on task-specific cross-modal fusion, while This preserves generalizable textual semantic information. Therefore, the representations generated by these two branches exist on different feature manifolds. Simply concatenating or averaging them implies they reside in the same space, which may lead to feature scale imbalance or premature alignment. To address this issue, this application's embodiments design a residual fusion module. This module achieves dual-path aggregation through nonlinear residual interaction and a linear adaptive weighting scheme, where learnable residual connections can be viewed as a form of manifold alignment.

[0074] S6: Calculate the loss value based on the fused features and the target text features, and update the trainable parameters in the feature fusion network, the mapping network and the residual fusion module based on the loss value to train the model and obtain the combined image retrieval model.

[0075] Please see Figure 7 , Figure 7 A specific implementation of step S6 is shown below: S61: Calculate the mean squared error loss value between the fused feature and the target text feature.

[0076] Specifically, Mean Squared Error (MSE) is a commonly used regression loss function used to quantify the difference between the fused features generated by the model and the expected target text features. Its role is to provide a clear optimization objective for model training, namely, minimizing the difference between the predicted features and the target features, thereby guiding the model to learn the correct feature representation and fusion method. This loss value can be obtained by calculating the average of the sum of the squares of the differences between corresponding elements of the two feature vectors.

[0077] S62: Using the mean squared error loss value as the objective function, calculate the gradients of the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module using the backpropagation algorithm.

[0078] Specifically, backpropagation is the core algorithm for training neural networks. It calculates the gradient of the loss function with respect to all trainable parameters in the network using the chain rule. The direction of the gradient indicates the direction in which the loss function increases most rapidly; therefore, the opposite direction of the gradient is the direction in which the loss function decreases most rapidly, providing a basis for parameter updates. This process can utilize the built-in backpropagation mechanism of automatic differentiation frameworks (such as PyTorch and TensorFlow), which can automatically construct the computation graph and calculate the gradients of all parameters, greatly simplifying the development process. Alternatively, the gradient calculation formula for each network layer (such as fully connected layers, convolutional layers, activation functions, etc.) can be manually derived and implemented, and then the gradients can be calculated and accumulated layer by layer according to the backpropagation sequence.

[0079] S63: An optimizer is used to update the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module based on the calculated gradient in order to train the model.

[0080] Specifically, an optimizer updates the trainable parameters in the feature fusion network, mapping network, and residual fusion module based on the calculated gradients for model training. An optimizer is an algorithm that adjusts model parameters based on the calculated gradients. Its purpose is to gradually reduce the value of the loss function during training to optimize model performance. The Adam (Adaptive Moment Estimation) optimizer can be used; Adam combines the advantages of Adagrad and RMSprop, adaptively adjusting the learning rate of each parameter, and typically has fast convergence speed and good generalization performance. Alternatively, the SGD (Stochastic Gradient Descent) optimizer and its variants (such as SGD with momentum) can be used. SGD calculates gradients and updates parameters using mini-batch data; the momentum term helps accelerate convergence and overcome local optima.

[0081] S64: When the preset number of training steps is reached or the latest mean squared error loss value reaches the preset threshold, the combined image retrieval model is obtained.

[0082] Specifically, the combined image retrieval model is obtained when a preset number of training steps is reached or the latest mean squared error loss value reaches a preset threshold. The training termination condition is a key mechanism for controlling the model training process, aiming to balance training efficiency and model performance, and avoid underfitting or overfitting. The preset number of training steps ensures that training does not proceed indefinitely, while the loss value threshold ensures that the model reaches a certain performance level. A fixed maximum number of training iterations (epochs or steps) can be set; when this number is reached, training will stop regardless of the loss value. Alternatively, the validation set loss value can be monitored after each training epoch or a fixed number of steps. If the loss value does not decrease within N consecutive epochs, or the decrease is less than a certain minimum value, training is stopped early. This application combines the above two methods: setting a maximum number of training steps and monitoring the loss value during training; once the loss value falls below a preset threshold, training is also stopped immediately.

[0083] In one specific implementation, a training dataset (totaling 5.5 million training titles) was constructed using titles generated by CC3M and 2.47 million manually selected StableDiffusion prompts. A three-layer MLP was used as the architecture for the mapping network Φ, trained using the AdamW optimizer with a learning rate of 0.0001, weight decay of 0.01, and a mini-batch size of 64, for 20,000 training steps. Regularization was performed with a 50% dropout rate. To ensure fair evaluation of the Feature Fusion Branch (FFB), the application adopted the feature fusion network architecture configured in the typical supervised method CLIP4CIR. Throughout the training process, the parameters of all encoders were frozen, and only the trainable parameters of the feature fusion network Cϑ, the mapping network Φ, and the residual fusion module were updated. All training processes were performed using the PyTorch framework.

[0084] S7: Obtain the reference image to be processed and the text to be modified, and perform image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

[0085] Please see Figure 8 , Figure 8 A specific implementation of step S7 is shown below: S71: Obtain the reference image to be processed and the modified text to be processed.

[0086] Specifically, the system aims to receive raw query information from the user: a reference image to be processed and a text to be modified. The reference image is the visual object the user wishes to modify, while the text specifies the user's intended modification or desired attributes. These inputs form the basis for the combined image retrieval task, providing a clear starting point for subsequent feature extraction and model inference. For example, users can upload images and enter text through a graphical user interface (GUI), or pass image data and text strings to the system through an application programming interface (API).

[0087] S72: The text to be modified is encoded by an image encoder to obtain reference image features, and the text to be modified is encoded by a text encoder to obtain modified text features.

[0088] Specifically, the purpose of this application is to convert raw inputs (images and text) from different modalities into a unified, high-dimensional feature representation so that subsequent models can process and fuse them. The image encoder is responsible for extracting visual semantic information from the reference image to be processed, generating reference image features; the text encoder is responsible for extracting semantic information from the modified text to be processed, generating modified text features. The image encoder can employ a pre-trained convolutional neural network (CNN), such as ResNet or EfficientNet, or a Transformer-based visual model, such as the image encoder portion of ViT or CLIP. The text encoder can employ a pre-trained Transformer model, such as BERT or RoBERTa, or the text encoder portion of CLIP, to convert the text sequence into semantic embedding vectors.

[0089] S73: The feature fusion network is used to fuse the features of the reference image and the modified text to obtain the predictive query features to be processed.

[0090] Specifically, the embodiments of this application aim to perform preliminary cross-modal interaction and fusion of encoded reference image features and modified text features to generate a preliminary query feature representation. The role of the feature fusion network is to learn how to combine the visual content of the image and the modification instructions of the text to form a feature that can reflect the user's query intent.

[0091] S74: Project the reference image features through the mapping network to obtain pseudo-label embeddings, and construct the prediction features to be processed based on the pseudo-label embeddings.

[0092] Specifically, the purpose of this application is to extract or generate a pseudo-labeled embedding from the features of a reference image. This embedding can capture specific attributes or semantic information of the reference image, and on this basis, construct the predictive features to be processed. A mapping network projects the reference image features from their original embedding space to a new space, generating a pseudo-labeled embedding, which helps the model better understand the intrinsic properties of the reference image. The mapping network can be a simple linear layer that transforms the dimensionality of the input features; or it can be a multilayer perceptron containing multiple non-linear activation layers to learn more complex projection relationships. Constructing the predictive features based on the pseudo-labeled embedding can be done by directly using the pseudo-labeled embedding as the predictive feature, or by combining it with the reference image features or other relevant information.

[0093] S75: The residual fusion module fuses the target query feature with the predicted feature to be processed to obtain the target query feature.

[0094] Specifically, this application aims to deeply fuse preliminary and supplementary predictive features to generate the final target query features. The residual fusion module fully integrates the advantages of both types of features through non-linear interaction and adaptive weighting, resulting in a more comprehensive and accurate feature representation that reflects the user's search intent. The residual fusion module can contain multiple residually connected neural network layers, allowing information to be directly transferred between different layers and avoiding the gradient vanishing problem. S76: Calculate the similarity between the target query feature and the pre-extracted image features in the image library using cosine similarity, and determine the target image based on the similarity.

[0095] Specifically, cosine similarity is a commonly used metric to measure the cosine of the angle between two non-zero vectors, effectively reflecting the similarity along the vector direction, i.e., semantic similarity. By calculating the cosine similarity between the target query feature and all pre-extracted image features in the image library, the degree of matching between the query intent and the images in the library can be quantified. The pre-extracted image features in the image library refer to the feature vectors that have been encoded and stored by an image encoder for all images in the library before retrieval, which can significantly improve retrieval efficiency. Based on the calculated similarity, the system selects the image with the highest similarity (or the top N images) as the target image, i.e., the image that best matches the user's modification intent.

[0096] like Figure 9 As shown, Figure 9 A schematic diagram of the inference process of the combined image retrieval model is shown. In this embodiment, a reference image to be processed is given. and a piece of text to be processed and modified Features of reference images ,in It is an image encoder. The inference phase differs from the training phase; it does not require feeding reference text features. and target text features Noise is injected because the reference information input during the inference phase is image features, not text pseudo-triple data, thus ensuring the stability and repeatability of the predicted query representation. The model first generates two predicted features through two branches: in, It is a text encoder. It is a feature fusion network. It is a mapping network. It is a long text adaptation module. Depend on It consists of a fixed template "A photo of $ that T". Finally, these features are effectively integrated through a residual fusion module to obtain the final query features, which use cosine similarity to match the image features in the image library to retrieve the target image.

[0097] Please refer to Figure 10 As a response to the above Figure 1 To implement the method shown, this application provides an embodiment of a dual-balanced combined image retrieval device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0098] like Figure 10 As shown, the dual-balanced combined image retrieval device of this embodiment includes: a text pseudo-triple construction module 81, a modal noise addition module 82, a feature fusion processing module 83, a predicted feature construction module 84, a feature weighted fusion module 85, a retrieval model training module 86, and a target image retrieval module 87, wherein: The text pseudo-triple construction module 81 is used to obtain unlabeled training text and construct text pseudo-triples based on the unlabeled training text through a pseudo-triple construction strategy driven by a controllable function. Modal noise addition module 82 is used to encode the text pseudo triples to obtain triple features, and add modal alignment noise to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features. Feature fusion processing module 83 is used to perform feature fusion processing on the noisy reference text features and the modified text features in the triplet features through a feature fusion network to obtain predictive query features; The prediction feature construction module 84 is used to extract keywords from the unlabeled training text to construct masked text, and to project the features of the noisy target text through a mapping network to obtain pseudo-labeled text embeddings, and to construct prediction features based on the pseudo-labeled text and the masked text. The feature weighted fusion module 85 is used to perform nonlinear residual interaction and linear adaptive weighted fusion on the predicted query feature and the predicted feature through the residual fusion module to obtain the fused feature; The retrieval model training module 86 is used to calculate the loss value based on the fused features and the target text features, and update the trainable parameters in the feature fusion network, the mapping network and the residual fusion module based on the loss value to train the model and obtain the combined image retrieval model. The target image retrieval module 87 is used to acquire the reference image to be processed and the text to be modified, and to perform image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

[0099] Furthermore, the text pseudo-triple construction module 81 includes: The keyword extraction unit is used to acquire the unlabeled training text and extract keywords from the unlabeled training text, wherein the keywords include noun phrases, proper nouns, adjectives and qualifiers; The mutually exclusive subset partitioning unit is used to randomly divide the keyword set into three mutually exclusive subsets, and to replace the words in the selected mutually exclusive subsets with antonyms or related words to generate a new set of words after noise word replacement; The text reconstruction unit is used to reconstruct the text of the new word set and generate the text pseudo triples, wherein the text pseudo triples include reference text, modified text and target text.

[0100] Furthermore, the predictive feature construction module 84 includes: The masked text construction unit is used to extract keywords from the unlabeled training text to construct the masked text, and to project the features of the noisy target text through the mapping network to obtain the pseudo-labeled text embedding. A masked text embedding conversion unit is used to convert the masked text into a masked text embedding. An embedding position replacement unit is used to replace the embedding position corresponding to the placeholder mark in the mask text embedding with the pseudo-marked text embedding to obtain the replaced embedding feature; A location embedding unit is used to perform location embedding on the replaced embedding feature through hybrid location encoding to obtain the embedded feature; The predictive feature generation unit is used to map the embedded features to a joint visual and linguistic embedding space through layer normalization and projection layers to generate the predicted features.

[0101] Furthermore, the feature weighted fusion module 85 includes: The feature concatenation unit is used to concatenate the predicted query feature with the predicted feature, and input the concatenated feature into a multilayer perceptron for high-order interaction modeling to obtain residual features; A linear fusion feature generation unit is used to obtain linear fusion features by weighted summation of the predicted query features and the predicted features using learnable scalar weights; The fusion feature generation unit is used to add the residual feature and the linear fusion feature and then perform layer normalization to obtain the fusion feature.

[0102] Furthermore, the target image retrieval module 87 includes: A text acquisition unit is used to acquire the reference image to be processed and the modified text to be processed; The encoding unit is used to encode the text to be modified by an image encoder to obtain reference image features, and to encode the text to be modified by a text encoder to obtain modified text features. The feature fusion unit is used to fuse the features of the reference image and the modified text features through the feature fusion network to obtain the predictive query features to be processed; The projection unit is used to project the reference image features through the mapping network to obtain pseudo-label embeddings, and to construct the prediction features to be processed based on the pseudo-label embeddings; The target query feature generation unit is used to fuse the unprocessed predicted query feature with the unprocessed predicted feature through the residual fusion module to obtain the target query feature; The similarity calculation unit is used to calculate the similarity between the target query feature and the pre-extracted image features in the image library using cosine similarity, and to determine the target image based on the similarity.

[0103] Furthermore, the modal noise addition module 82 includes: A text pseudo-triple encoding unit is used to encode the text pseudo-triples by a text encoder to obtain the triple features; A random variable generation unit is used to generate a first random variable and a second random variable that are independent of each other, wherein the first random variable follows a uniform distribution on the interval from 0 to 1, and the second random variable follows a standard normal distribution; The modal alignment noise generation unit is used to calculate the element-wise product of the first random variable and the second random variable to obtain the modal alignment noise; The feature addition unit is used to add the modality alignment noise to the reference text feature and the target text feature respectively to obtain the noisy reference text feature and the noisy target text feature.

[0104] Furthermore, the retrieval model training module 86 includes: The loss calculation unit is used to calculate the mean squared error loss value between the fused feature and the target text feature; The gradient calculation unit is used to calculate the gradients of the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module using the mean squared error loss value as the objective function through the backpropagation algorithm. The parameter update unit is used to update the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module according to the calculated gradient using an optimizer, so as to train the model. The model training completion unit is used to obtain the combined image retrieval model when a preset number of training steps is reached or the latest mean squared error loss value reaches a preset threshold.

[0105] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 11 , Figure 11 This is a basic structural block diagram of the computer device in this embodiment.

[0106] Computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected via a system bus. It should be noted that... Figure 11 Only a computer device 9 with three components—memory 91, processor 92, and network interface 93—is shown. It should be understood that implementing all shown components is not required; more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0107] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0108] The memory 91 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 91 may also include both internal storage units and external storage devices of the computer device 9. In this embodiment, the memory 91 is typically used to store the operating system and various application software installed on the computer device 9, such as program code based on a dual-balanced combined image retrieval method. In addition, the memory 91 may also be used to temporarily store various types of data that have been output or will be output.

[0109] In some embodiments, processor 92 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 92 is typically used to control the overall operation of computer device 9. In this embodiment, processor 92 is used to run program code stored in memory 91 or process data, for example, to run the program code of the aforementioned dual-balanced combined image retrieval method to implement various embodiments of the dual-balanced combined image retrieval method.

[0110] The network interface 93 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0111] This application also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the dual-balanced combined image retrieval method described above.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0113] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. A combined image retrieval method based on dual-balance, characterized in that, include: Obtain unlabeled training text, and construct text pseudo triples based on the unlabeled training text using a pseudo triple construction strategy driven by a controllable function; The text pseudo triples are encoded to obtain triple features, and modality alignment noise is added to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features. The feature fusion network is used to perform feature fusion processing on the noisy reference text features and the modified text features in the triplet features to obtain the predicted query features; Keywords are extracted from the unlabeled training text to construct masked text, and the features of the noisy target text are projected through a mapping network to obtain pseudo-labeled text embeddings. Predictive features are constructed based on the pseudo-labeled text and the masked text. The predicted query features and the predicted features are fused by nonlinear residual interaction and linear adaptive weighted fusion through the residual fusion module to obtain fused features. The loss value is calculated based on the fused features and the target text features, and the trainable parameters in the feature fusion network, the mapping network and the residual fusion module are updated based on the loss value to train the model and obtain the combined image retrieval model. The system acquires a reference image to be processed and text to be modified, and performs image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

2. The dual-balanced combined image retrieval method according to claim 1, characterized in that, The process of obtaining unlabeled training text and constructing text pseudo-triples based on the unlabeled training text using a controllable function-driven pseudo-triple construction strategy includes: Obtain the unlabeled training text and extract keywords from the unlabeled training text, wherein the keywords include noun phrases, proper nouns, adjectives and qualifiers; The keyword set is randomly divided into three mutually exclusive subsets, and the words in the selected mutually exclusive subsets are replaced with antonyms or related words to generate a new word set after noise word replacement. The new word set is reconstructed to generate the text pseudo-triples, wherein the text pseudo-triples include the reference text, the modified text, and the target text.

3. The dual-balanced combined image retrieval method according to claim 1, characterized in that, The process of extracting keywords from the unlabeled training text to construct masked text, projecting the features of the noisy target text through a mapping network to obtain pseudo-labeled text embeddings, and constructing predicted features based on the pseudo-labeled text and the masked text includes: Keywords are extracted from the unlabeled training text to construct the masked text, and the noisy target text features are projected through the mapping network to obtain the pseudo-labeled text embedding. Convert the masked text into an embedded masked text; The placeholder markers in the mask text embedding are replaced with the pseudo-marked text embeddings to obtain the replaced embedding features. The replaced embedded features are positionally embedded by hybrid positional encoding to obtain the embedded features; The embedded features are normalized and projected to a joint visual and linguistic embedding space to generate the predicted features.

4. The dual-balanced combined image retrieval method according to claim 1, characterized in that, The step involves using a residual fusion module to perform nonlinear residual interaction and linear adaptive weighted fusion on the predicted query features and the predicted features to obtain fused features, including: The predicted query feature is concatenated with the predicted feature, and the concatenated feature is input into a multilayer perceptron for high-order interaction modeling to obtain the residual feature; Linear fusion features are obtained by weighting and summing the predicted query features and the predicted features using learnable scalar weights; The residual features are added to the linear fusion features and then subjected to layer normalization to obtain the fusion features.

5. The dual-balanced combined image retrieval method according to claim 1, characterized in that, The process of acquiring the reference image to be processed and the text to be modified, and then performing image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image, includes: Obtain the reference image to be processed and the text to be modified; The text to be modified is encoded by an image encoder to obtain reference image features, and the text to be modified is encoded by a text encoder to obtain modified text features. The feature fusion network is used to fuse the features of the reference image and the modified text to obtain the predictive query features to be processed. The reference image features are projected through the mapping network to obtain pseudo-label embeddings, and the predicted features to be processed are constructed based on the pseudo-label embeddings. The residual fusion module fuses the unprocessed prediction query features with the unprocessed prediction features to obtain the target query features; The similarity between the target query feature and the pre-extracted image features in the image library is calculated using cosine similarity, and the target image is determined based on the similarity.

6. The dual-balanced combined image retrieval method according to claim 1, characterized in that, The process of encoding the text pseudo triples to obtain triple features, and adding modality alignment noise to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features, includes: The text pseudo triples are encoded using a text encoder to obtain the triple features; Generate a first random variable and a second random variable that are independent of each other, wherein the first random variable follows a uniform distribution on the interval from 0 to 1, and the second random variable follows a standard normal distribution; The modal alignment noise is obtained by calculating the element-wise product of the first random variable and the second random variable. The modality alignment noise is added to the reference text feature and the target text feature respectively to obtain the noisy reference text feature and the noisy target text feature.

7. The dual-balanced combined image retrieval method according to any one of claims 1 to 6, characterized in that, The process of calculating a loss value based on the fused features and the target text features, and updating the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module based on the loss value to train the model and obtain a combined image retrieval model includes: Calculate the mean squared error loss value between the fused features and the target text features; Using the mean squared error loss value as the objective function, the gradients of the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module are calculated using the backpropagation algorithm. An optimizer is used to update the trainable parameters in the feature fusion network, the mapping network, and the residual fusion module based on the calculated gradient in order to train the model. When the preset number of training steps is reached or the latest mean squared error loss value reaches the preset threshold, the combined image retrieval model is obtained.

8. A dual-balanced combined image retrieval device, characterized in that, include: The text pseudo-triple construction module is used to obtain unlabeled training text and construct text pseudo-triples based on the unlabeled training text through a pseudo-triple construction strategy driven by a controllable function. The modal noise addition module is used to encode the text pseudo triples to obtain triple features, and add modal alignment noise to the reference text features and target text features in the triple features to obtain noisy reference text features and noisy target text features. The feature fusion processing module is used to perform feature fusion processing on the noisy reference text features and the modified text features in the triplet features through a feature fusion network to obtain the predicted query features; The prediction feature construction module is used to extract keywords from the unlabeled training text to construct masked text, and to project the features of the noisy target text through a mapping network to obtain pseudo-labeled text embeddings. Based on the pseudo-labeled text and the masked text, prediction features are constructed. The feature weighted fusion module is used to perform nonlinear residual interaction and linear adaptive weighted fusion on the predicted query features and the predicted features through the residual fusion module to obtain fused features; The retrieval model training module is used to calculate the loss value based on the fused features and the target text features, and update the trainable parameters in the feature fusion network, the mapping network and the residual fusion module based on the loss value to train the model and obtain the combined image retrieval model. The target image retrieval module is used to acquire a reference image to be processed and text to be modified, and to perform image retrieval based on the reference image to be processed and the text to be modified using the combined image retrieval model to obtain the target image.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the dual-balanced combined image retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the bibalanced combined image retrieval method as described in any one of claims 1 to 7.

Images