Fine-grained clothing image retrieval method and device based on large language model common sense knowledge injection

Abstract

The application discloses a fine-grained clothing image retrieval method and device for injecting common sense knowledge of a large language model. The method first extracts fine-grained visual features of an input image through an image encoder and optimizes them in combination with a low-rank adapter to improve the representation ability at the image patch level. Then, a pre-trained large language model generates attribute-enhanced common sense knowledge context to enrich the image attribute representation, thereby helping the model understand and reason about unknown attribute information in an open scenario. The application introduces a switchable modal prompt and interpolation mechanism to ensure that the agent embedding can be dynamically supplemented when attributes or text are missing. During the retrieval process, through an attribute-guided cross-modal attention mechanism, fine-grained image content matching is performed based on the relationship between image features and attribute-enhanced context. Through multi-modal feature alignment and optimization, the application improves the accuracy and robustness of clothing image retrieval in an open world scenario.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and information retrieval, and in particular to an open-scene fine-grained fashion image retrieval method and device based on large language model (LLM) common sense knowledge injection. BACKGROUND

[0002] Fashion image retrieval has a wide range of applications in various e-commerce platforms, including fashion product recommendation, fashion trend prediction, and copied fashion product detection. Traditional fashion image retrieval methods usually retrieve visually similar products by measuring the global similarity between images, using a shared embedding space. Although this approach performs well in overall visual matching, it fails to effectively capture fine-grained visual features that are crucial for practical applications. Especially when performing specific design element retrieval or copied product detection, traditional methods often fail to provide sufficiently fine-grained matching, resulting in inaccurate or user-unfriendly search results.

[0003] To address this issue, the Attribute-Specific Fashion Retrieval (ASFR) task has been proposed in recent years, aiming to retrieve products by specific attribute values to achieve more fine-grained fashion product search. Unlike traditional retrieval methods, ASFR focuses on retrieving products based on given attributes such as skirt length, clothing color, etc., allowing users to retrieve fashion products that perfectly match the query attributes. However, existing ASFR methods usually assume that all attribute values are predefined and known within the training dataset, which has limitations in real-world open-world scenarios. In open-world scenarios, many attributes are not present in the training set, leading to a significant drop in retrieval performance when encountering unknown or missing attributes. Therefore, ASFR methods exhibit low generalization ability when facing cross-domain and open-set application scenarios.

[0004] Therefore, there is a need for a new method to handle unknown attributes, missing attributes, and complex multi-modal information in open-world ASFR scenarios, thereby improving the robustness and accuracy of fashion image retrieval systems in practical applications. SUMMARY

[0005] To address the limitations of existing technology, the present application proposes an open-scene fine-grained fashion image retrieval method and device based on large language model (LLM) common sense knowledge injection. For the Attribute-Specific Fashion Retrieval (ASFR) task proposed in the background art, this method no longer limits itself to global visual similarity, but achieves precise retrieval through fine-grained attribute matching. Specifically, given an input image and a specific fashion attribute (such as skirt length), the system can retrieve a list of images that best match that attribute.

[0006] The purpose of the present application is achieved by the following technical solutions: in the first aspect, the present application provides a fine-grained clothing image retrieval method for large language model common sense knowledge injection, which comprises the following steps:

[0007] (1) Image feature representation: obtain clothing images, and extract fine-grained visual features based on a CLIP image encoder;

[0008] (2) Attribute and common sense knowledge representation: map the given attribute to an attribute embedding vector, and generate a common sense description using a large language model;

[0009] (3) Attribute knowledge robust fusion under modal missing: based on a modal configuration vector, judge the availability of the current modal, including complete modal, only attribute modal or only context modal; when a certain modal is missing, use a default value or a proxy embedding to supplement the missing modal information through an interpolation mechanism;

[0010] (4) Attribute-guided cross-modal reasoning: construct an attribute-guided query vector, align the fine-grained visual features with the attribute information, and generate fine-grained image retrieval results according to the similarity.

[0011] Further, in step (1), the image encoder is optimized by a low-rank adapter LoRA, and the image is split into multiple non-overlapping patches to generate embedding representation of each patch.

[0012] Further, in step (2), enhanced text embedding is obtained through a large language model to enhance the semantic representation of the attribute, and the attribute embedding and the enhanced context form a conditional query vector together to guide the retrieval process.

[0013] Further, in step (3), a prompt vector is defined to encode the current modal configuration, and when a modal is unavailable, an interpolation mechanism is introduced to replace the missing embedding with a trainable default value or proxy text.

[0014] Further, in step (4), the attribute embedding and the attribute-enhanced context embedding are combined to construct an attribute-guided query vector to guide the retrieval process and accurately capture the visual content related to the query attribute.

[0015] Further, in step (4), the attribute-guided query vector is used to align with the patch features extracted by the image encoder in cross-modal, and the similarity between the query and the image patch is calculated to weight the image, and finally the image features matching the query are generated.

[0016] Further, the model is optimized by using a triple loss function, which shortens the matching distance between the query vector and the positive sample image, and simultaneously lengthens the distance between the negative samples, so that the query vector is aligned with the target attribute image, and the image most relevant to the query attribute can be accurately selected during retrieval.

[0017] In a second aspect, the present application further provides a fine-grained clothing image retrieval device based on large language model common sense knowledge injection, comprising a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to realize the fine-grained clothing image retrieval method based on large language model common sense knowledge injection.

[0018] In a third aspect, the present application further provides a computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to realize the fine-grained clothing image retrieval method based on large language model common sense knowledge injection.

[0019] In a fourth aspect, the present application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to realize the fine-grained clothing image retrieval method based on large language model common sense knowledge injection.

[0020] The beneficial effects of the present application are:

[0021] 1. The present application innovatively combines the attribute-enhanced context knowledge generated by LLM with fine-grained visual features, effectively handles the problem of unknown or missing attributes in an open world scenario, and improves the accuracy and robustness of the retrieval system in an open scenario.

[0022] 2. The present application does not rely on traditional attribute predefinition, greatly improves the adaptability to unknown attributes, and can handle incomplete information in multi-modal input, and has wide practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The fine-grained clothing image retrieval diagram for the open scenario of the present application.

[0024] Figure 2 The fine-grained clothing image retrieval flowchart based on large language model common sense knowledge injection of the present application.

[0025] Figure 3 The performance comparison diagram of the open world ASFR task of the present application.

[0026] Figure 4 The performance comparison diagram of the cross-domain open world ASFR task of the present application.

[0027] Figure 5A structural diagram of a fine-grained clothing image retrieval device for large language model common knowledge injection is provided. DETAILED DESCRIPTION

[0028] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0029] To solve the limitations of traditional clothing image retrieval methods in the face of unknown attributes and missing attributes in open world scenarios, the present application proposes an open scene fine-grained clothing image retrieval method based on large language model (LLM) common knowledge injection. The present application improves the robustness and accuracy of the retrieval system in cross-modal, multi-modal and open world scenarios by fusing attribute-enhanced contextual knowledge generated by LLM and fine-grained visual features. As shown in Figure 1 and Figure 2 The specific steps are as follows:

[0030] (1) Image feature representation

[0031] First, a CLIP-based image encoder is used to process the input image to extract fine-grained visual features. The image encoder is optimized by a low-rank adapter (LoRA), and the image is split into multiple non-overlapping patches to generate embedding representation for each patch, enhancing the patch-level representation ability of the image. The output of the image encoding is:

[0032]

[0033] where, represents the visual encoding function, K represents the number of non-overlapping patches in the image, and d represents the embedding dimension of each patch. This patch-level representation is crucial for identifying and aligning specific visual regions related to attributes, laying the foundation for subsequent cross-modal interaction and fine-grained retrieval.

[0034] (2) Attribute and common knowledge representation

[0035] To enable the model to handle attributes in an open world, including those not seen in the training dataset, the present invention enriches the information from the original attribute representation and its common-sense knowledge from a pre-trained large language model (LLM). Specifically, the present invention uses the GPT-4o model API interface provided by OpenAI as an implementation means to enhance the common sense of attributes. The system first inputs the given attribute (such as "sleeve length", "collar design", etc.) in the form of a natural language prompt to GPT-4o, for example, the input format is: "As a fashion design and styling expert, your task is to create a detailed description of a specific fashion attribute to support fine-grained fashion retrieval. The attribute is: sleeve length. Please clearly explain how this attribute affects the appearance, functionality, and overall style of the garment. Highlight its aesthetic significance, practical impact, and interaction with other design elements or contexts. Ensure that your explanation is structured, insightful, easy to understand, and provides detailed details to enhance the semantic representation of the attribute." The natural language text returned by GPT-4o contains common-sense descriptions of the attribute, for example: "Sleeve length determines the coverage of the arm and affects the overall balance of the garment between exposure and coverage." More samples can be referred to Table 1 in the specification. Subsequently, the specified attribute and its corresponding common-sense text description are input into the attribute-aware context encoder module, which embeds these heterogeneous inputs into a shared latent space for multi-modal complementary information integration. It should be noted that the GPT-4o model used in this method only makes inference calls through the API interface and does not involve model retraining, which has good realizability and engineering deployment convenience.

[0036] (2-1) Original attribute representation

[0037] Original attribute is mapped to a d-dimensional embedding vector through a fully connected layer and a tanh activation function, represented as:

[0038]

[0039] where is the attribute embedding function. This process bridges the gap between categorical attributes and high-dimensional latent representations, mapping attributes from simple categorical labels to more expressive high-dimensional semantic embedding vectors, effectively integrating visual and textual modalities.

[0040] (2-2) Attribute-aware context knowledge representation

[0041] To enrich attribute-related information in unseen cases, the present invention embeds each attribute The extension is a comprehensive textual description. Table 1 shows the text prompts designed by the invention, which can capture the aesthetic and functional dimensions of the given attributes.

[0042] Take "sleeve length" as an example. By providing the text description "a photo of fashion with focus on sleeve length" to the text encoder of CLIP, common-sense knowledge is generated to enrich the semantic representation of the attribute. The generated attribute-enhanced context together with the attribute embedding forms a conditional query vector that guides the retrieval process. This process helps improve the adaptability to unknown or missing attributes in the open world. The generated attribute-enhanced context expression is:

[0043]

[0044] where t is the attribute-enhanced text embedding, represents the text encoding function.

[0045] Table Several representative attribute descriptions generated by LLM

[0046]

[0047] (3) Robust fusion of attribute knowledge under missing modalities

[0048] Although the construction of query q assumes that both attribute and text common-sense knowledge are available, actual world scenarios often involve flexible modalities. That is, one or more attribute information may be missing. To address these situations, the invention introduces the following mechanism to indicate the availability of the current modality with vector p, including three modes: complete, attribute only, and context only. When a modality is missing, the missing modality interpolation mechanism is called to fill in the proxy embedding a or t obtained during training, based on the indication of the p prompt, to ensure effective reasoning even when a modality is missing. The specific implementation process is as follows:

[0049] (3-1) Modality switching prompt mechanism

[0050] By defining a modality configuration vector, the availability of the current modality is made clear, such as complete modality, attribute-only modality, or context-only modality. The modality configuration is switched according to the actual situation to ensure efficient processing of queries under different modality conditions. Specifically, by defining a prompt vector , the current modality configuration is made clear.

[0051]

[0052] where is the complete modality, is the attribute-only available modality, For only context available modality.

[0053] (3-2) Missing modality imputation mechanism

[0054] When a modality is unavailable, an imputation mechanism is introduced to replace the missing embedding with a trainable default value. For example, when text information is missing, the system will use a proxy text embedding learned during training process to replace the real text description. The corresponding imputation formula is:

[0055]

[0056] where, and are the proxy embedding vectors. The introduction of this mechanism ensures that the system can still maintain high retrieval performance in the case of missing modalities.

[0057] (4) Attribute-guided cross-modal reasoning

[0058] After obtaining the representation of attributes, attribute-aware common sense context, and image input, the invention designs an attribute-guided cross-modal attention mechanism to reason the relevance between them for fine-grained, image patch-based content matching. First, an attribute-aware query is constructed to provide matching guidance, and then the query is used to retrieve relevant content in the image input.

[0059] (4-1) Attribute-guided query construction

[0060] To retrieve images under specific attribute-text conditions, an attribute-aware conditional query q is defined by fusing attribute and context embeddings together to provide fine-grained guidance, as follows:

[0061]

[0062] where a and t come from steps (2-1) and (2-2), respectively, and is a learnable transformation that merges these embeddings into a single vector . This query encodes the user's fine-grained requirements (i.e., attribute a and text description t) and can effectively guide the retrieval process to accurately capture visual content related to the query attribute.

[0063] (4-2) Cross-modal attention interaction

[0064] After obtaining fine-grained patch features from the image encoder (step 1), the attribute-guided query Attention alignment across modalities is performed. The image is weighted by computing the similarity between the query and the image patches, resulting in image features that match the query. Specifically, q serves as the query (Q), while the image patch feature matrix X is projected as the key (K) and value (V). For each attention head h, there are:

[0065]

[0066] where d is the dimension of each head, then

[0067]

[0068] where is a learnable projection parameter. The outputs of all attention heads are concatenated and linearly transformed to obtain a multi-head output O is flattened into a vector , and the following is given:

[0069]

[0070] where o contains the most relevant patch-level information aligned with the specified attribute and attribute-aware context. By precisely highlighting patches relevant to the fine-grained query, this cross-modal attention mechanism enables conditional-specific attribute retrieval with high granularity.

[0071] (5) Model training

[0072] After constructing the query q (possibly with a modal hint or an interpolation vector) and computing the conditional-specific visual output o, the model is trained to rank correct images higher than incorrect ones. To prioritize the model's matching of correct images over incorrect ones, a triplet loss is defined, balancing attribute-text alignment and retrieval accuracy.

[0073]

[0074] where d is a distance function (e.g., Euclidean distance or cosine similarity), is a margin that controls the minimum distance difference between positive and negative samples, respectively represent the i-th query vector, positive sample feature, and negative sample feature. Minimizing drives the query embedding vector closer to the matching result of the positive sample.

[0075] The final loss function balances attribute-text alignment and retrieval accuracy:

[0076] ​​​

[0077] wherein, is the alignment loss, controls its relative contribution.

[0078] (6) Model inference

[0079] In the inference process, the missing modalities (attributes or text) are first identified. If needed, the system applies the corresponding modal prompt and interpolation strategy to generate consistent and complete representations. Then, the query vector q is constructed, and the similarity with the candidate images is calculated to perform ranking. Specifically, the system calculates the similarity of the output o(q) and o(c) of the query image and the candidate image, and finally ranks the candidates by similarity to generate fine-grained retrieval results.

[0080]

[0081] wherein, is the attention mechanism function, denotes the feature of the positive sample query image, denotes the feature of the candidate set image. To verify the effectiveness of the method of the present application, the present application not only evaluates in the common in-domain ASFR task, but also evaluates in the challenging cross-domain scenario, in which the image distribution is significantly different between the training set and the test set.

[0082] In-domain comparison: The method proposed in the present application achieves the most advanced performance on three traditional ASFR benchmarks (FashionAI, DARN and DeepFashion) (Tables 2, 3 and 4). It is worth noting that unlike the two-stage methods such as ASEN++ and RPF, which are prone to error accumulation and computational cost, the present method uses a unified end-to-end single-stage pipeline, simplifying the inference process while maintaining high accuracy. In addition, using the same backbone network such as ResNet-50, the present method always performs better than its counterparts (such as ASEN, ISLN and AttnFashion).

[0083] Table Performance comparison on the FashionAI dataset

[0084]

[0085] Table Performance comparison on the DARN dataset

[0086]

[0087] Table Performance comparison on the DeepFashion dataset

[0088]

[0089] Cross-domain comparison: The traditional ASFR model was also evaluated in cross-domain settings, where the image distribution differs between datasets. Table 5 compares the proposed method with the baselines in two transfer scenarios: FashionAI → DARN and DARN → FashionAI. Although these two datasets have similar attributes (e.g., dress length vs. coat length), the domain shift challenges the generalization ability of traditional methods. By leveraging its powerful attribute-enhanced representation, the proposed method outperforms the baselines and achieves accurate retrieval in cross-domain retrieval.

[0090] Table Cross-domain evaluation on FashionAI → DARN and DARN → FashionAI

[0091]

[0092] Generalization to unknown attributes: The open-world ASFR task requires models to handle unknown attributes across different domains. To evaluate the effectiveness of the proposed model, experiments were conducted focusing on the generalization ability and unknown attribute generalization across domains.

[0093] Figure 3 The model's ability to handle unknown attributes was demonstrated, focusing on eight fashion attributes: dress length, sleeve length, coat length, pant length, collar design, neckline design, neck line design, and neck design, with experiments conducted in three progressively increasing difficulty scenarios. In scenario Figure 3 (a), sleeve length and collar design were excluded from training. In scenario Figure 3 (b), the difficulty was increased by excluding four attributes: dress length, sleeve length, collar design, and neckline design. Scenario Figure 3 (c) represents the most challenging condition, where only pant length and neck design are included in the training data.

[0094] Since traditional ASFR methods rely on fixed attribute distributions, they cannot handle unknown attributes, and their performance on these attributes is equivalent to the random baseline, which randomly sorts all candidate images. In contrast, the proposed method integrates attribute-enhanced contexts generated by LLMs, enabling the model to learn rich semantic representations. As Figure 3 shown, the proposed method consistently outperforms the random baseline, demonstrating its ability to effectively capture and align the semantics of unknown attributes, even in the most challenging scenario (c).

[0095] Cross-domain generalization to unknown attributes: Cross-domain generalization tests the model's ability to adapt to unknown attributes in a new domain, combining the challenges of domain shift and unknown attribute configuration. Figure 4 The performance of training on the DARN dataset and testing on the FashionAI dataset, where unknown attributes (e.g., skirt length, pant length, and collar design) are introduced, is evaluated.

[0096] In this setting, the random baseline performs poorly because it cannot adapt to the domain-specific attributes missing in training. In contrast, the proposed method effectively bridges the gap between domains by generating attribute-enhanced contexts with LLM, enriching the embeddings and endowing them with semantic knowledge. As Figure 4 shown, the proposed method achieves significant performance improvements, demonstrating strong adaptability to cross-domain and cross-attribute configuration. These results highlight its robustness in real-world open-world ASFR tasks.

[0097] Corresponding to the foregoing embodiment of the large language model common sense knowledge injection fine-grained clothing image retrieval method, the application also provides an embodiment of a large language model common sense knowledge injection fine-grained clothing image retrieval device.

[0098] Referring to Figure 5 , the large language model common sense knowledge injection fine-grained clothing image retrieval device provided by the embodiment of the application includes a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to implement the large language model common sense knowledge injection fine-grained clothing image retrieval method in the foregoing embodiment.

[0099] The embodiment of the large language model common sense knowledge injection fine-grained clothing image retrieval device provided by the application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability where it is located. From the hardware level, as Figure 5 shown, it is a hardware structure diagram of the device with data processing capability where the large language model common sense knowledge injection fine-grained clothing image retrieval device provided by the application is located, in addition to the processor, memory, network interface, and non-volatile memory shown in the embodiment, the device with data processing capability where the device is located in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, which will not be described again. Figure 5 ​

[0100] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0101] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0102] The embodiment of the application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the fine-grained costume image retrieval method of large language model common knowledge injection in the above embodiment.

[0103] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0104] The application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the fine-grained costume image retrieval method of large language model common knowledge injection.

[0105] The above embodiments are used to explain and illustrate the application, but not to limit the application. Any modifications and changes made to the application within the spirit and protection scope of the claims fall within the protection scope of the application.

Claims

1. A fine-grained method for retrieving clothing images by injecting common-sense knowledge from a large language model, characterized in that, The method includes the following steps: (1) Image feature representation: Acquire clothing images and extract fine-grained visual features based on CLIP image encoder; The image encoder is optimized by low-rank adapter LoRA, the image is split into multiple non-overlapping patches, and an embedding representation of each patch is generated; (2) Attribute and common sense knowledge representation: The given attribute is mapped to an attribute embedding vector, and a common sense description is generated using a large language model; specifically, it includes: obtaining enhanced text embedding through a large language model, generating common sense text descriptions related to the attribute to enhance the semantic representation of the attribute, inputting the specified attribute and its corresponding common sense text description together into the attribute-aware context encoder module, embedding these heterogeneous inputs into a shared latent space to perform multimodal complementary information integration, and realizing the generalized representation of unknown attributes not appearing in the training set. The attribute embedding and the enhanced context together form a conditional query vector to guide the retrieval process. (3) Robust fusion of attribute knowledge under modality missing: Define a cue vector to encode the current modality configuration. Based on the modality configuration vector, determine the availability of the current modality, including complete modality, attribute-only modality, or context-only modality. When a modality is missing, introduce an imputation mechanism to replace the missing embedding with a trainable default value or surrogate embedding to supplement the missing modality information. (4) Attribute-guided cross-modal reasoning: Combining attribute embedding and attribute-enhanced context embedding, an attribute-guided query vector is constructed to guide the retrieval process, accurately capture visual content related to the query attribute, and use the attribute-guided query vector to perform cross-modal alignment with the patch features extracted by the image encoder. By calculating the similarity between the query and the image patch, the image is weighted, and finally, image features matching the query are generated. Fine-grained clothing image retrieval results are generated based on the similarity. (5) Joint optimization: In order to make the model prioritize matching correct images over incorrect images, a triplet loss is defined, and the attribute-text alignment and retrieval accuracy are balanced; the distance between the conditional query vector and positive samples is minimized and the interval between negative samples is widened, while the attribute-text alignment and retrieval accuracy are balanced.

2. A fine-grained clothing image retrieval device with common-sense knowledge injection from a large language model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements the fine-grained clothing image retrieval method with common sense knowledge injection of a large language model as described in claim 1.

3. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the fine-grained clothing image retrieval method with common sense knowledge injection of a large language model as described in claim 1.

4. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the fine-grained clothing image retrieval method described in claim 1, which injects common sense knowledge into a large language model.

Images