Multi-modal clothing recommendation method

Abstract

The invention discloses a multi-modal clothing recommendation method. The method comprises the following steps: firstly, constructing a knowledge graph by taking related data as nodes and edges, and storing the knowledge graph in a Neo4j graph database; extracting unified multi-modal vector representation of image and text pairs, performing entity linking, establishing a multi-modal knowledge base, finding a Top-m result most similar to problem embedding through an RAG strategy based on the knowledge base, positioning nodes of the result in a knowledge graph, generating a sub-graph, and according to edge weights in the knowledge graph, establishing a multi-modal knowledge base; and calculating a recommendation score of each scheme in combination with the sub-graph to obtain a final Top-n recommendation result, splicing knowledge of the Top-n scheme with a user question to generate a structured text answer, and rendering a text description part into visual content to realize mixed output of image-text combination. By combining the knowledge graph and the knowledge base driven RAG with the generation of the high-quality generation model, the semantic information and visual consistency of complex clothing recommendation can be flexibly captured, so that the clothing recommendation accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent clothing recommendation, and in particular to a multimodal clothing recommendation method. Background Art

[0002] Fashion is a silent language that not only conveys a person's personality, taste, and approach to life, but also demonstrates careful coordination and creativity. Nowadays, clothing coordination plays a vital role in people's daily lives. Many people prioritize fashion, often seeking inspiration through posts or blogs from fashion bloggers on their favorite social media platforms. However, this approach is not only time-consuming and laborious, but the results can also be unsatisfactory.

[0003] With the rapid development of artificial intelligence (AI) technology, large language models (LLMs) have demonstrated remarkable capabilities in natural language processing (NLP) tasks, enabling the generation of high-quality text content and complex language understanding. Large language models are not only capable of processing massive amounts of text data but also, through pre-training, can learn rich linguistic knowledge and patterns, providing powerful support for various NLP tasks. Knowledge graphs (KGs), as an effective data representation, can integrate heterogeneous data and organize information from different sources in a structured manner, thereby providing rich background knowledge and context for solving complex problems. Knowledge graphs have unique advantages in processing complex semantic relationships and providing deep reasoning. Retrieval-augmented generation (RAG) technology combines the advantages of retrieval and generation models. By first retrieving relevant information and then generating answers, it effectively improves the accuracy and timeliness of answers. RAG technology can fully leverage large-scale knowledge bases to generate highly relevant answers to questions while maintaining coherence and logic.

[0004] Existing systems in the fashion recommendation space suffer from numerous shortcomings. Most rely on user purchase history or simple preferences, resulting in repetitive and insufficiently personalized recommendations. They focus on individual items rather than overall styling, making it difficult to capture the complex relationships between garments. They rely on static labels and manual rules, making them incapable of adapting to rapidly evolving fashion trends. Furthermore, user interaction methods are limited, lacking real-time feedback and multimodal input support, limiting the diversity and innovation of recommendations. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a multimodal clothing recommendation method, aiming to solve the problems in the prior art that the recommendation system relies on user purchase history or simple preferences, resulting in repeated and insufficient personalization of recommendations, a single user interaction method, and a lack of real-time feedback and multimodal input support.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A multimodal clothing recommendation method, characterized by comprising the following steps:

[0008] Step 1: Build a knowledge graph using a combination of Few-Shot, Prompt Template, and large language models to define the entity and relationship patterns of the knowledge graph. Then, perform knowledge extraction, including named entity recognition, entity disambiguation, attribute extraction, and relationship extraction. The extracted relationship frequencies are standardized as edge weights in the knowledge graph. Finally, the relevant data is used as nodes and edges to construct a knowledge graph and store it in the Neo4j graph database.

[0009] Step 2: Build a multimodal knowledge base. Input the preprocessed multimodal data into a large language model, fuse the visual features of the image and the semantic features of the text into high-dimensional semantic embedding vectors, and link each embedding vector with its original content and related entities. Store the vectors in the FAISS vector index database to form a structured multimodal knowledge base.

[0010] Step 3: Based on the RAG strategy of knowledge graph and knowledge base, when a user asks a question about clothing, DeepSeek-VL is first used to convert the user input into a semantic embedding, and then FAISS is used to find the top-m results most similar to the question embedding in the retrieval database;

[0011] Step 4: Input the top-m results into the ReRanker module, locate the nodes corresponding to these results in the knowledge graph, perform reasoning and verification to evaluate their accuracy and effectiveness, and at the same time, start from these nodes and traverse upward in the knowledge graph to generate subgraphs in a depth-first traversal;

[0012] Step 5: Calculate the recommendation score of each solution based on the edge weights in the knowledge graph and the subgraph, and select the top-n solutions as the final recommendation results;

[0013] Step 6: Use the LLM generator and Stable Diffusion to combine the knowledge of the Top-n solutions with the user questions and input them into a structured text answer. At the same time, render the text description part into visual content to achieve a hybrid output combining text and images.

[0014] Furthermore, in step 1, first, the pattern of the knowledge graph is defined, the entity, attribute and relationship types are clarified, and the prompt template is used to guide the large language model to define the entity and relationship pattern; then, knowledge extraction is performed, including named entity recognition, entity disambiguation, attribute extraction and relationship extraction; named entity recognition uses prompt templates and a small number of examples to identify key entities such as brands, clothing types, colors, seasons, etc.; entity disambiguation uses a large language model to solve the ambiguity problem of entities with similar names; attribute extraction extracts specific features from the text; relationship extraction identifies the connection between entities and stores them as edges in the knowledge graph; then, the knowledge graph is constructed using associated data as nodes and edges, and the accuracy of the data is ensured through manual inspection and verification; finally, the relationship weight is calculated, the frequency of occurrence of each relationship is standardized, and stored as the relationship value as the edge weight of the knowledge graph, the formula is where w raw represents the original frequency of occurrence, w norm Represents the normalized weight value, which ranges from (0, 1).

[0015] Furthermore, in step 2, a unified multimodal vector representation of the image and text pair is first extracted. The image is decoded by a visual encoder to extract visual features, and the text is decoded by a language encoder to extract semantic information. These two features are then fused according to the weight parameter α to form a high-dimensional semantic embedding vector. The fusion formula is v fused =α·v image +(1-α)·v text , where v image and v text are the embedding vectors output by the visual encoder and the language encoder respectively. α is used to balance the ratio of visual features and text features. Then, each embedding vector is linked to its original content and related entities and stored in the FAISS vector index database to form a structured multimodal knowledge base. In order to improve the retrieval performance, labels are added to classify and mark samples so that they can be filtered and grouped based on semantics.

[0016] Furthermore, in step 3, the RAG strategy based on the knowledge graph and knowledge base captures local information of clothing recommendations. The knowledge graph node Si in a Reranker module uses all possible methods to connect from position i to the relevant node, which is called a knowledge subgraph. The input layer accepts the user's text or text + image query, and generates Top-m results through retrieval of the knowledge graph and knowledge base.

[0017] Furthermore, in step 4, a depth-first traversal method is used, starting from these nodes, traversing upward until their root nodes in the knowledge graph are found, recording all the paths passed along the way, and generating a subgraph, wherein the process of generating the subgraph starts from the relevant nodes, finding only their directly adjacent nodes in turn, and traversing upward in a single direction until all the relevant nodes on the path back to the root node are included.

[0018] Furthermore, in step 5, the recommendation score of each solution is calculated based on the edge weights in the knowledge graph using a designed formula combined with the subgraph. The formula is as follows:

[0019]

[0020] Among them S i represents the i-th dressing scheme, Occur represents the frequency of the matching, G i Representative and S i The corresponding knowledge subgraph, w jk Represents the corresponding side (e j , e k ), α represents the penalty adjustment parameter (controls the impact of the number of edges, generally 0.5), and λ represents the weighting coefficient (weighs the co-occurrence frequency vs. the importance of graph structure, ranging from 0 to 1);

[0021] Finally, the Top-n solution is selected as the final recommendation result.

[0022] The technical solution adopted by the present invention has the following beneficial effects:

[0023] This invention is a multimodal clothing recommendation method based on a large language model, a knowledge graph, and a knowledge base. Leveraging the powerful language understanding and generation capabilities of the large language model, combined with the knowledge representation of the knowledge graph and knowledge base, this method flexibly captures users' personalized needs and fashion knowledge. Through retrieval-enhanced generation technology, this method effectively integrates multimodal data, supports mixed input of text and images, and calculates recommendation scores based on edge weights in the knowledge graph. This significantly improves the accuracy and personalization of clothing recommendations, enhances the user's interactive experience, and is highly adaptable, enabling it to flexibly respond to rapidly changing fashion trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram showing the process of GraphRAG generating a subgraph in an embodiment of the present invention;

[0026] Figure 31 is a schematic diagram of the structure of an answer generator in an embodiment of the present invention;

[0027] Figure 4 This is a diagram of the recommendation results implemented by the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] The following is combined with Figure 1-3 The present invention is described in further detail.

[0030] Example: A multimodal outfit recommendation question-answering framework based on a large language model and RAG, such as Figure 1 、 Figure 2 and Figure 3 As shown, the following steps are included:

[0031] 1) Construct a knowledge graph using a method that combines Few-Shot Prompting technology, Prompt Template technology, and DeepSeek-VL. First, define the schema of the knowledge graph, clarify the entity, attribute, and relationship types, and use prompt templates to guide large language models to define entity and relationship schemas. Next, perform knowledge extraction, including named entity recognition, entity disambiguation, attribute extraction, and relationship extraction. Named entity recognition uses prompt templates and a small number of examples to identify key entities such as brands, clothing types, colors, and seasons; entity disambiguation uses large language models to solve the ambiguity problem of entities with similar names; attribute extraction extracts specific features from text; and relationship extraction identifies connections between entities and stores them as edges in the knowledge graph. Then, use associated data as nodes and edges to construct a knowledge graph, and ensure the accuracy of the data through manual inspection and verification. Finally, calculate the relationship weight, standardize the frequency of occurrence of each relationship, and store it as the relationship value as the edge weight of the knowledge graph. The formula is: where w raw represents the original frequency of occurrence, w norm Represents the normalized weight value, ranging from (0, 1). This edge weight is crucial for the calculation of recommendation scores in the knowledge graph.

[0032] 2) Establish a multimodal knowledge base. First, extract a unified multimodal vector representation of the image and text pairs. Specifically, the image is encoded with a visual encoder to extract visual features, and the text is encoded with a language encoder to extract semantic information. These two features are then fused according to the weight parameter α to form a high-dimensional semantic embedding vector. The fusion formula is v fused =α·vimage +(1-α)·v test , where v image and v text are the embedding vectors output by the visual encoder and the language encoder, respectively. α is used to balance the ratio of visual features to textual features. Each embedding vector is then linked to its original content and related entities (such as nodes in the knowledge graph) and stored in the FAISS vector index database, thus forming a structured multimodal knowledge base. To improve retrieval performance, labels are also added to classify and mark samples for semantic filtering and grouping.

[0033] 3) Knowledge-based RAG strategy: When a user asks a question about clothing, the system first uses DeepSeek-VL to convert the user's input into a semantic embedding. Then, the FAISS vector index database is used to measure the similarity between the question embedding and the image and text vectors in the knowledge base to identify the top-m most relevant results.

[0034] 4) Input the top-m results into the ReRanker module, locate the nodes corresponding to these results in the knowledge graph, perform reasoning and verification to evaluate their accuracy and effectiveness, and at the same time, start from these nodes and traverse upward in the knowledge graph to generate subgraphs in a depth-first traversal;

[0035] 5) Based on the edge weights in the knowledge graph, the recommended score of each solution is calculated using the designed formula combined with the subgraph. The formula is as follows:

[0036]

[0037] Among them S i represents the i-th dressing scheme, Occur represents the frequency of the matching, G i Representative and S i The corresponding knowledge subgraph, w jk Represents the corresponding side (e j ,e k ), α represents the penalty adjustment parameter (controls the impact of the number of edges, generally 0.5), and λ represents the weighting coefficient (weighs the co-occurrence frequency vs. the importance of graph structure, ranging from 0 to 1);

[0038] Finally, the Top-n solution is selected as the final recommendation result;

[0039] 6) Using the LLM generator and the image generation model, the knowledge of the top-n solutions is spliced ​​with the user questions and then input to generate structured text answers. At the same time, the text description part is rendered as visual content to achieve a hybrid output combining text and images.

[0040] In this embodiment, DeepSeek-VL is used to represent the unified multimodal vector representation of image and text pairs. It is a high-dimensional semantic embedding vector formed by fusing the visual features of the image and the semantic features of the text, called a visual-text fusion vector. Similarly, we can also fuse multiple images with text to generate a higher-dimensional fusion vector. The FAISS vector index database is used to store and retrieve these fusion vectors to achieve efficient knowledge base construction and query. The entire RAG framework is connected by the user input layer, retrieval layer, re-ranking layer and generation layer to form a complete recommendation system, such as Figure 1 shown.

[0041] In steps 3) and 4), the RAG strategy based on knowledge graph and knowledge base is used to capture local information for clothing recommendation. Specifically, a knowledge graph node Si in a Reranker module uses all possible methods to connect from position i to the relevant node, which is called a knowledge subgraph, such as Figure 2 As shown in Figure 2, the input layer accepts a user's text or text + image query and generates the top-m results by searching the knowledge graph and knowledge base. The re-ranking layer then merges these top-m results into the top-n results by calculating a recommendation score. This score not only reflects the frequency of clothing combinations but also takes into account the edge weights between nodes in the knowledge graph, helping to further improve the relevance and accuracy of the recommendation results.

[0042] Finally, the output is fed forward to the generation layer, which uses DeepSeek-LLM and Stable Diffusion to generate recommended content that combines images and text. Multiple retrieval, reranking, and generation stages can be stacked in this way until the final recommendation result is obtained. In this way, it gradually extracts high-level local information from the previous layer, and the output is ultimately input into a fully connected network. The edge weights of the knowledge graph are first used as the activation function layer, as edge weights can reflect the frequency and importance of clothing combinations. Compared to other methods, it can help generate high-quality recommendation results faster. Finally, a scoring function is used to calculate the probability distributed over predefined recommendation solutions. It converts the raw scores into normalized probabilities and uses updates to the knowledge graph and knowledge base to adjust the weights in the recommendation system. Parameter optimization operations are also used to avoid overfitting.

[0043] The specific method for generating the knowledge subgraph in step 4) is as follows: We use a depth-first traversal method, starting from these nodes and traversing upward until we find their root node in the knowledge graph, recording all paths along the way to generate a subgraph. It should be noted that this process starts from the relevant nodes, sequentially finds only their directly adjacent nodes, and traverses upward in a single direction until all relevant nodes on the path back to the root node are included.

[0044] Specifically, refer to Figure 4 , Figure 4 This is a recommendation result diagram implemented by the method of the present invention in an embodiment of the present invention. According to the diagram, after receiving the user's question, the present invention generates multiple clothing matching result diagrams for the user's reference.

[0045] The following is a verification experiment of this embodiment:

[0046] The model is compared on 2 public datasets: DeepFashion2 and Polyvore Outfits.

[0047] The implementation details are as follows:

[0048] Given our focus on clothing recommendation, we design two evaluation protocols: (1) For a pre-matched clothing set, we use text queries to test the model's ability to retrieve correct combinations; (2) For single-piece images without predefined matches, we test the model using a mixture of text and partial visual input. If the top-n results include correct matches from the dataset, the recommendation is considered successful; otherwise, it is marked as unsuccessful. The results are shown in Table 1, where bold and underlined indicate the best and second-best results, respectively.

[0049] In this experiment, we set m and n to 5 and 3, respectively. We set α to 0.6 to give more weight to images, as visual features (such as color and style) play a more important role in understanding user preferences for clothing recommendations. These values ​​were selected based on the results of small-scale retrieval and recommendation scoring tests. However, it should be noted that users can adjust these parameters according to their specific requirements.

[0050] Table 1 Experimental comparison

[0051]

[0052] On both datasets, our framework outperforms all baselines in terms of accuracy, precision, recall, and F1 score. Notably, it achieves an F1 score of 90.56% and 90.24% on DeepFashion2 and Polyvore, respectively, surpassing the second place (UniFashion) by 1.14 and 0.81 percentage points, respectively. In image generation, Moir also leads with the best FID (8.03 and 8.12) and CLIP score (31.11 and 30.71) on DeepFashion2 and Polyvore. These results highlight Moir's advantages in combining knowledge graph and foundation-driven retrieval with high-quality generative models, achieving a strong balance between semantic relevance and visual coherence, and demonstrating strong generalization capabilities across datasets.

[0053] Through the above-mentioned verification experiments, the RAG strategy using knowledge graphs and multimodal knowledge bases can flexibly capture complex clothing matching features. In this verification experiment, two real-world clothing datasets (DeepFashion2 and PolyvoreOutfits) were tested. The recommendation results were compared with existing clothing recommendation methods. The proposed recommendation method can significantly improve the accuracy of clothing recommendations, outperforming existing technologies in all indicators. It is also highly adaptable and can flexibly respond to rapidly changing fashion trends.

[0054] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but as long as they are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A multimodal clothing recommendation method, characterized in that: The following steps are involved: Step 1: Build a knowledge graph using a combination of Few-Shot, Prompt Template, and large language models to define the entity and relationship patterns of the knowledge graph. Then, perform knowledge extraction, including named entity recognition, entity disambiguation, attribute extraction, and relationship extraction. The extracted relationship frequencies are standardized as edge weights in the knowledge graph. Finally, the relevant data is used as nodes and edges to construct a knowledge graph and store it in the Neo4j graph database. Step 2: Build a multimodal knowledge base. Input the preprocessed multimodal data into a large language model, fuse the visual features of the image and the semantic features of the text into high-dimensional semantic embedding vectors, and link each embedding vector with its original content and related entities. Store the vectors in the FAISS vector index database to form a structured multimodal knowledge base. Step 3: Based on the RAG strategy of knowledge graph and knowledge base, when a user asks a question about clothing, DeepSeek-VL is first used to convert the user input into a semantic embedding, and then FAISS is used to find the top-m results most similar to the question embedding in the retrieval database; Step 4: Input the top-m results into the ReRanker module, locate the nodes corresponding to these results in the knowledge graph, perform reasoning and verification to evaluate their accuracy and effectiveness, and at the same time, start from these nodes and traverse upward in the knowledge graph to generate subgraphs in a depth-first traversal; Step 5: Calculate the recommendation score of each solution based on the edge weights in the knowledge graph and the subgraph, and select the top-n solutions as the final recommendation results; Step 6: Use the LLM generator and Stable Diffusion to combine the knowledge of the Top-n solutions with the user questions and input them into a structured text answer. At the same time, render the text description part into visual content to achieve a hybrid output combining text and images.

2. The multimodal clothing recommendation method according to claim 1, characterized in that: In step 1, first, the knowledge graph schema is defined, entities, attributes, and relationship types are clarified, and a prompt template is used to guide a large language model to define entity and relationship schemas; then, knowledge extraction is performed, including named entity recognition, entity disambiguation, attribute extraction, and relationship extraction; named entity recognition uses prompt templates and a small number of examples to identify key entities such as brand, clothing type, color, and season; entity disambiguation uses a large language model to resolve ambiguity between entities with similar names; and attribute extraction extracts specific features from text; Relationship extraction identifies the connections between entities and stores them as edges in the knowledge graph. Then, the knowledge graph is constructed using the associated data as nodes and edges, and the accuracy of the data is ensured through manual inspection and verification. Finally, the relationship weight is calculated, and the frequency of occurrence of each relationship is normalized and stored as the relationship value as the edge weight of the knowledge graph. The formula is: where w raw represents the original frequency of occurrence, w norm Represents the normalized weight value, which ranges from (0, 1).

3. The multimodal clothing recommendation method according to claim 1, characterized in that: In step 2, a unified multimodal vector representation of the image and text pair is first extracted. The image is encoded with a visual encoder to extract visual features, and the text is encoded with a language encoder to extract semantic information. These two features are then fused according to the weight parameter α to form a high-dimensional semantic embedding vector. The fusion formula is v fused =α·v image +(1-α)·v text , where v image and v text are the embedding vectors output by the visual encoder and the language encoder respectively. α is used to balance the ratio of visual features and text features. Then, each embedding vector is linked to its original content and related entities and stored in the FAISS vector index database to form a structured multimodal knowledge base. In order to improve the retrieval performance, labels are added to classify and mark samples so that they can be filtered and grouped based on semantics.

4. The multimodal clothing recommendation method according to claim 1, wherein: In step 3, the RAG strategy based on the knowledge graph and knowledge base captures local information for clothing recommendations. The knowledge graph node Si in a Reranker module uses all possible methods to connect from position i to the relevant node, which is called a knowledge subgraph. The input layer accepts the user's text or text + image query, and generates Top-m results through retrieval of the knowledge graph and knowledge base.

5. The multimodal clothing recommendation method according to claim 1, wherein: In step 4, a depth-first traversal method is used, starting from these nodes, traversing upward until their root nodes in the knowledge graph are found, recording all the paths passed along the way, and generating a subgraph, wherein the process of generating the subgraph is to start from the relevant nodes, find only their directly adjacent nodes in turn, and traverse upward in a single direction until all the relevant nodes on the path back to the root node are included.

6. The multimodal clothing recommendation method according to claim 1, characterized in that In step 5, the recommendation score of each solution is calculated based on the edge weights in the knowledge graph using the designed formula combined with the subgraph. The formula is as follows: Among them S i represents the i-th dressing scheme, Occur represents the frequency of the matching, G i Representative and S i The corresponding knowledge subgraph, w jk Represents the corresponding side (e j , e k ), α represents the penalty adjustment parameter, which controls the influence of the number of edges and is generally set to 0.5; λ represents the weighting coefficient, which weighs the co-occurrence frequency vs. the importance of the graph structure and ranges from 0 to 1; Finally, the Top-n solution is selected as the final recommendation result.

Images