A homogeneous garment image retrieval method based on multi-modal information
By using multimodal information processing technology and combining image and text feature fusion, the problem of insufficient accuracy of single-modal clothing retrieval methods in the retrieval of homogeneous clothing is solved, achieving more efficient clothing retrieval and improved user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU HOPERUN SOFTWARE CO LTD
- Filing Date
- 2026-02-02
- Publication Date
- 2026-06-02
AI Technical Summary
Existing clothing retrieval methods mainly rely on single-modal information, making it difficult to accurately capture subtle differences between clothing items, especially in the case of searching for homogeneous clothing, resulting in insufficient accuracy of search results and user satisfaction.
Employing multimodal information processing technology, the system converts clothing images and accompanying descriptions into vectors using image encoders and text encoders. It then fuses image and text features using an image-text combiner and generates more robust feature representations through a two-stage training and dynamic feature modulation mechanism, which are then combined with natural language commands for retrieval.
It enables more accurate capture of differences in clothing styles, materials, colors, textures, etc., improving search accuracy and user interaction experience, and significantly improving the search effect for homogeneous clothing.
Smart Images
Figure CN122132590A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal information processing technology, specifically relating to a method for retrieving homogeneous clothing images based on multimodal information. Background Technology
[0002] With the rapid development of e-commerce and smart retail, apparel retrieval technology has gradually become a core component of apparel e-commerce platforms, fashion recommendation systems, and personalized shopping experiences. Most existing apparel retrieval methods rely on single-modal information, such as image-based similarity calculations or text-based keyword matching. Even methods that rely on both text and images typically treat text only as supplementary information, rather than using it in conjunction with images for a unified search. However, due to the complexity of clothing in terms of color, style, material, and design details, relying solely on a single modality often fails to accurately capture subtle differences between garments, leading to insufficient accuracy and user satisfaction in search results. This limitation is particularly pronounced in scenarios involving homogeneous apparel retrieval, where the search requires finding clothing highly similar to the target garment in style, design, or details.
[0003] Currently, most fashion image-based retrieval methods rely primarily on feature extraction from the images themselves. While multimodal retrieval methods consider both reference images and text retrieval needs, they typically treat these two inputs as two separate processes—for example, first extracting visual features from the image, then semantic features from the text, and finally fusing them for retrieval. However, in existing application scenarios, fashion images are usually accompanied by a text description that emphasizes key design details such as style, material, cut, and decorative elements. This crucial information is often difficult to fully capture from the visual features of the image. Summary of the Invention
[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A method for retrieving homogeneous clothing images based on multimodal information, comprising the following steps:
[0005] S1. Collect target images and combine them with their accompanying descriptions to form a set;
[0006] S2. Vectorize the target image text group, specifically including:
[0007] The image was processed by a Clip image encoder. Convert to The text will pass through a Clip text encoder. Convert to ; via image-text combiner The image and text vectors are fused, as represented by equation (1):
[0008] (1),
[0009] in It is an input image. This is the text content that corresponds to this image. It is the feature vector obtained by combining text and image information.
[0010] S3. Vectorize and integrate search commands with reference images and text, specifically including:
[0011] During the retrieval process, the system simultaneously receives reference image and text information groups as well as user-input instructions, and performs the retrieval based on these. Equation (2) is used to generate a function for retrieving nearby images. vector :
[0012] (2),
[0013] in, It is obtained through training. , and The combination is a hybrid fashion encoder. These are reference images and text input by the user, and These are user commands. Make minor adjustments based on the new task. It is a new combiner. and The architecture is the same. It is an image and text combiner. It is a graphic command combiner. Its function is to be an encoder that integrates information from images and text, making the vectors representing fashion information more robust. The received text is a supplementary description of the image; The received text is a search command, which is a reference image or text input. The algorithm adjusts or shifts the skew according to the retrieval command to retrieve the information the user wants. Although both are combined encoders, their functions differ, therefore they need to be trained separately and cannot be used interchangeably.
[0014] As an improvement of the present invention, the image-text combiner The steps for fusing image features and text features include:
[0015] S21, the image-text combiner The inputs are image feature representations. Text feature representation After passing through a fully connected layer and the ReLU activation function, the two obtain a non-linear transformation result. The transformed features are then fed into a Concat layer for concatenation, which is used to capture the interaction information between the image and the text.
[0016] S22. The concatenated features include image features and text features. After further FC, ReLU and FC operations, they are connected to the Sigmoid activation function to generate a weight vector.
[0017] S23. Finally, the features after the two weighted sums are added together to obtain the fused representation. .
[0018] As an improvement of the present invention, the image-text combiner The training process consists of two phases, the first phase and The parameters are not used in training; only the image-text combiner is used. The parameters will be adjusted, such as those of the image-text combiner. After convergence, then and The parameters were fine-tuned for the second stage of training, which involved adjusting the image-text combiner. The loss function for the training steps is shown in equation (3):
[0019] (3),
[0020] in, It is a sample The vectorized result is for the sample After modification, the modified samples are vectorized to obtain... ,and These are the samples respectively. Vectorized features of Chinese text, vectorized features of images, modified text vectorized features, modified image vectorized features. Represents cosine similarity. and It is a temperature parameter used to control the range of logits. This indicates the number of samples in a batch. These are the parameters of the balanced two-stage loss function.
[0021] S31, and The training process is divided into two stages, the first stage The parameters are not involved in training, only... The parameters will be adjusted, etc. After convergence, then The parameters were also fine-tuned for the second phase of training. and The design of the loss function is shown in formula (4):
[0022] (4),
[0023] sim(·) represents cosine similarity. It is a temperature parameter used to control the range of logits. This indicates the number of samples in a batch, where each sample contains a source image and text group. A text instruction And a target image and text group .
[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0025] 1) Integrating multimodal information: Unlike traditional single-modal retrieval methods, this invention extracts and retrieves features from images and their accompanying text as a whole, which can more accurately capture the differences in clothing in terms of style, material, color, texture, etc.
[0026] 2) Dynamic Feature Modulation Mechanism: Through the designed image-text combiner By introducing an attention mechanism, weights can be adaptively assigned to features of different modalities, avoiding information loss caused by simple addition and making the fused feature representation more robust.
[0027] 3) Two-stage contrastive loss function design. In the contrastive loss function of the first stage... It acts as a variant of the positive sample, while the batch contains samples... The samples outside the range act as negative samples. This loss function causes the model to converge, allowing it to move away from positive samples. The feature distances should be as close as possible. The second half also uses a contrastive loss function, but negative samples are no longer features of other samples; instead, they are the features of the samples themselves. Vectorized features of Chinese text, vectorized features of images, modified text vectorized features, and modified image vectorized features. This invention aims to provide feature vectors that integrate image and text features to more comprehensively express the overall characteristics of image and text.
[0028] 4) Innovative two-stage training strategy:
[0029] Phase 1: Freeze the pre-trained model parameters and train only the combiner. This ensures stable convergence of the fusion process.
[0030] Phase Two: Unfreeze the image and text encoders, and fine-tune them together with the combiner to improve the overall feature representation ability, ensuring the stability of training and the final accuracy.
[0031] 5) Enhance robustness through data augmentation and contrastive learning: Introduce text rewriting and image perturbation during training to ensure that fused features remain consistent when faced with data deformation, thereby improving the robustness of the model.
[0032] 6) Improved search experience: Users can not only input text based on reference images, but also input additional natural language commands to form a three-dimensional search mode, which significantly improves the interactive experience and search accuracy.
[0033] 7) Development of the retrieval vector generation model: Similar to the image-text combination, this invention also designs a combiner for combining instructions and samples. This allows the instructions to better perform the task of migrating source samples to target samples. Similarly, this invention includes a specially fine-tuned instruction encoder for the instructions. This allows the text encoder to shift as much as possible in the direction of instruction content extraction, thus better completing the task. Similarly, this part of the invention also employs a two-stage training method, first freezing... All parameters are initialized with random numbers during training. ,when Train again after convergence and This allows for better fine-tuning results. Attached Figure Description
[0034] Figure 1 Each fashion item is often accompanied by a textual description of its corresponding design concept or design details. When collecting relevant information, it is necessary to make connections and pair them. The image shows an example of a fashion item and its paired text.
[0035] Figure 2 This is a vectorized flowchart of text and image combinations. The images are processed by a Clip image encoder. Convert to The text will pass through a Clip text encoder. Convert to ; via image-text combiner Fusion of image and text vectors ;
[0036] Figure 3 For image and text combiner Architecture diagram, graphical instruction combiner The architecture diagrams are the same;
[0037] Figure 4To generate retrieval vectors based on user instructions and the images or text selected or referenced by the user. The flowchart. Detailed Implementation
[0038] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0039] Example:
[0040] This embodiment provides a method for retrieving homogeneous clothing images based on multimodal information, including the following steps:
[0041] 1. Collect fashion images and create a set by combining the images with their accompanying descriptions:
[0042] a) such as Figure 1 As shown, each fashion item is often accompanied by a textual description of its corresponding design concept or design details. When collecting relevant information, it's necessary to correlate and match these descriptions.
[0043] 2) Vectorize the image-text group:
[0044] a) The image will pass through a Clip image encoder. Convert to In this case, any pre-trained Clip image encoder can be used. ResNet50 can be used as the image encoder.
[0045] b) The text will be processed by a Clip text encoder. Convert to Here, any pre-trained Clip text encoder can be used. Declutr can be used as the text encoder.
[0046] c) Because the text and image content have overlapping but also different focuses, this invention cannot simply add the two vectors to obtain the final image-text vector. Therefore, this invention designs an image-text combiner. To merge image and text vectors.
[0047] (1),
[0048] in It is an input image. This is the text content that corresponds to this image. It is the feature vector obtained by combining text and image information.
[0049] d) The design is as follows Figure 3 As shown, An image feature and a text feature were fused. The inputs are the image feature representations, respectively. Text feature representation The two features are processed through a fully connected layer (FC) and a ReLU activation function to obtain a nonlinear transformation result. The transformed features are then fed into a Concat layer for concatenation, used to capture the interaction information between the image and text. The concatenated features, containing both image and text features, undergo further FC, ReLU, and FC operations before being connected to a Sigmoid activation function to generate a weight vector. This weight vector is multiplied by the original features (i.e., a gating mechanism), thereby achieving adaptive modulation of the input modality features. Finally, the two weighted features are summed to obtain the fused representation. The core idea of this design is that, compared to the traditional simple addition method, it first uses a concatenation operation to synthesize image and text features into a single vector. This vector completely and independently contains both image and text features. Then, this vector is used by a neural network to synthesize the image and text features, generating weight vectors for each feature. This allows for the importance filtering of image and text features, thus balancing global interaction and dynamic adjustment during the fusion process, resulting in a more robust representation of the fused features.
[0050] e) This embodiment proposes The training method and the design of the loss function are shown in the following formula (3), where It is a sample After vectorization, this invention will analyze the samples. Modifications can be made, such as reorganizing or rewriting the language, adding noise to images, rotating them, etc., but the principle is to not change the main information. The modified samples are then vectorized to obtain... ,and These are the samples respectively. Vectorized features of Chinese text, vectorized features of images, modified vectorized features of text, and modified vectorized features of images. Represents cosine similarity. and It is a temperature parameter used to control the range of logits, where 𝐵 represents the number of samples in a batch.
[0051] (3),
[0052] Furthermore, this embodiment designs a two-stage contrastive loss function, the contrastive loss function at the front end... It acts as a positive sample, while the batch contains samples other than the positive sample. The samples outside the range act as negative samples. This loss function causes the model to converge, allowing it to move away from positive samples. The feature distances should be as close as possible. The second half also uses a contrastive loss function, but negative samples are no longer features of other samples; instead, they are the features of the samples themselves. Vectorized features of Chinese text, vectorized features of images, modified text vectorized features, and modified image vectorized features. This invention aims to provide feature vectors that integrate image and text features to more comprehensively express the overall characteristics of image and text. These are the parameters of the balanced two-stage loss function.
[0053] Furthermore, The training process is also divided into two stages, the first stage and The parameters are not involved in training, only... The parameters will be adjusted because and It is a pre-trained model with a certain feature representation ability, etc. After convergence, then and The parameters were also fine-tuned for the second phase of training.
[0054] f) This embodiment also includes retrieval vector generation: When retrieving fashion images, this invention can simultaneously reference image information and text information. During the retrieval process, the system can simultaneously receive the referenced image and text information group as well as the user's input instructions, and perform the retrieval based on this. Because instructions and text descriptions are two-dimensional information, they cannot be encoded using the same text encoding model and the same combination model. Figure 4 Hybrid fashion encoder It is obtained by training in step e of this invention. , and The combination These are reference images and text input by the user, and These are user commands, as shown in the image below. It's not the model that was fine-tuned in step e, but they all use the same initial pre-trained model. It will be fine-tuned based on the new tasks. It is a new combiner. and The architecture is the same, and the generated It is used to retrieve nearby items. The vector.
[0055] g) and Training method: This invention designs and training methods training methods and The training methods are quite different. Used for training and The design of the loss function is shown in the following formula (4):
[0056] (4),
[0057] sim(·) represents cosine similarity. It is a temperature parameter used to control the range of logits. This indicates the number of samples in a batch. Each sample contains a source image / text group, a text instruction, and a target image / text group (…). The reason for using this loss function is that, as a batch-based contrastive loss, it does not require an additional sampling strategy: it treats all samples in a mini-batch as negative samples. Query text features are used as displacement vectors from the query image group to the target image group.
[0058] and The training process is also divided into two stages, the first stage The parameters are not involved in training, only... The parameters will be adjusted because It is a pre-trained model with a certain feature representation ability, etc. After convergence, then The parameters were also fine-tuned for the second phase of training.
[0059] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A method for retrieving homogeneous clothing images based on multimodal information, characterized in that, Includes the following steps: S1. Collect target images and combine them with their accompanying descriptions to form a set; S2. Vectorize the target image text group, specifically including: The image was processed by a Clip image encoder. Transform into feature vectors that can express the characteristics of the image and its fashion features. The text is processed by a Clip text encoder. Transform into feature vectors that can express text features and the fashion features contained within those text features. ; via image-text combiner The image and text vectors are fused, as represented by equation (1): （1）， in It is an input image. This is the text content that corresponds to this image. It is the feature vector obtained by combining text and image information. , and Combination ; S3. Vectorize and integrate search commands with reference images and text, specifically including: During the retrieval process, the system simultaneously receives reference image and text information groups as well as user-input instructions, and performs the retrieval based on these. Equation (2) is used to generate a function for retrieving nearby images. vector : （2）， in, It is obtained through training. , and The combination is a hybrid fashion encoder. These are reference images and text input by the user, and These are user commands. Make minor adjustments based on the new task. It is a new combiner. and The architecture is the same. It is an image and text combiner. It is a graphic command combiner. The received text is a supplementary description of the image; The received text is a search command, which is a reference image or text input. Corrections or offsets are made based on the search instructions.
2. The method according to claim 1, characterized in that, Image Text Combiner The steps for fusing image features and text features include: S21, Image-Text Combiner The inputs are image feature representations. Text feature representation After passing through a fully connected layer and the ReLU activation function, the two obtain a non-linear transformation result. The transformed features are then fed into a Concat layer for concatenation, which is used to capture the interaction information between the image and the text. S22. The concatenated features include image features and text features. After further FC, ReLU and FC operations, they are connected to the Sigmoid activation function to generate a weight vector. S23. Finally, the features after the two weighted sums are added together to obtain the fused representation. .
3. The method according to claim 2, characterized in that, Image Text Combiner The training process consists of two phases, the first phase and The parameters are not used in training; only the image-text combiner is used. The parameters will be adjusted, such as those of the image-text combiner. After convergence, then and The parameters were fine-tuned for the second stage of training, which involved adjusting the image-text combiner. The loss function for the training steps is shown in equation (3): （3）， in, It is a sample The vectorized result is for the sample After modification, the modified samples are vectorized to obtain... ,and These are the samples respectively. Vectorized features of Chinese text, vectorized features of images, modified text vectorized features, modified image vectorized features. Represents cosine similarity. and It is a temperature parameter used to control the range of logits. This indicates the number of samples in a batch. These are the parameters of the balanced two-stage loss function.
4. The method according to claim 3, characterized in that, and The training process is divided into two stages, the first stage The parameters are not involved in training, only... The parameters will be adjusted, etc. After convergence, then The parameters were also fine-tuned for the second phase of training. and The design of the loss function is shown in formula (4): （4）， sim(·) represents cosine similarity. It is a temperature parameter used to control the range of logits. This indicates the number of samples in a batch, where each sample contains a source image and text group. A text instruction And a target image and text group .