A cross-modal garment item matching system and method based on fine-grained garment recognition

Abstract

The application discloses a cross-modal clothing single product matching system and method based on fine-grained clothing recognition. The cross-modal clothing single product matching system acquires clothing images, city information and dressing scene information, and acquires weather context according to the city information. High-level semantic features of the clothing images are extracted, and clothing categories, attribute labels and recognition confidence are obtained through grouped feature processing and feature alignment. Structured prompt information is constructed based on clothing attributes, weather context and dressing scene, and suggestion text is generated by using a large language model. According to the vector modalities possessed by clothing matching single products, same-modal similarity and / or cross-modal similarity are fused and calculated to recall candidate matching single products. Comprehensive scores are determined based on multi-modal semantic similarity, clothing attribute compatibility, weather adaptability and scene consistency, and the scores are reordered after adjusting the weight of the score items according to the recognition confidence.

Description

Technical Field

[0001] This invention belongs to the field of clothing recommendation technology, specifically relating to a cross-modal clothing item matching system and method based on fine-grained clothing recognition. Background Technology

[0002] With the development of apparel e-commerce, smart retail, and personalized lifestyle services, users' demand for outfit recommendations is gradually shifting from simple product retrieval to comprehensive matching suggestions that combine clothing attributes, weather conditions, and usage scenarios. Traditional recommendation methods typically rely on user history clicks, collaborative filtering, or manual rules, making it difficult to accurately understand fine-grained attributes such as clothing category, collar type, sleeve type, fabric, fit, and length when a user uploads only a single clothing image. Furthermore, it is difficult to collaboratively model these attributes with real-time weather and user context semantics.

[0003] Existing clothing image recognition technologies can output clothing categories or partial attribute labels, but a single recognition result usually cannot be directly converted into a suitable outfit plan for a user's scenario. For example, the same top may correspond to different outfits in scenarios such as commuting, campus, business, or travel; the same garment may also require different pants, shoes, and accessories in high temperature, low temperature, rainy days, or dry weather. Therefore, relying solely on classification labels or visual similarity for recommendations easily leads to problems such as a lack of weather awareness, insufficient scenario adaptability, and weak interpretability.

[0004] Furthermore, while existing cross-modal retrieval methods can recall candidate products based on image and text similarity, without the combined constraints of fine-grained clothing attributes, weather context, and user scenario semantics, the retrieval results may only be visually or semantically similar, and may not necessarily suit real-world dressing needs. Large language models can generate natural language suggestions, but if their output is not combined with candidate item retrieval and re-ranking mechanisms, it is difficult to translate it into concrete, displayable outfit suggestions. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-modal clothing item matching system and method based on fine-grained clothing recognition, in order to solve the problems of existing outfit recommendation methods, which, when making recommendations based solely on clothing category labels, visual similarity, or user historical behavior, struggle to fully integrate fine-grained clothing attributes, weather context, and user outfit scenarios. This results in insufficient scenario adaptability of recommendation results, low relevance of outfits, insufficient retrieval adaptability when modal information is missing, and weak interpretability.

[0006] In a first aspect, the present invention provides a cross-modal clothing item matching system based on fine-grained clothing recognition, which includes a multi-source input acquisition module, a weather context acquisition module, a clothing attribute recognition module, a structured prompt information construction module, a suggestion text generation module, a candidate matching item selection module, and a candidate re-sorting module;

[0007] The multi-source input acquisition module is used to acquire clothing images, city information, and outfit scene information uploaded by users;

[0008] The weather context acquisition module is used to acquire weather context based on city information;

[0009] The clothing attribute recognition module is used to extract high-level semantic features from clothing images, split and process the high-level semantic features into multiple group features, map the high-level semantic features into alignment features aligned with the group features, and concatenate the alignment features with multiple group features to perform clothing attribute recognition, thereby obtaining clothing category, attribute label and recognition confidence.

[0010] The structured prompt information construction module is used to construct structured prompt information based on clothing category, attribute tags, recognition confidence, weather context, and dressing scene information;

[0011] The suggested text generation module is used to input structured prompt information into a large language model to obtain suggested text;

[0012] The candidate matching item selection module is used to construct cross-modal retrieval text based on clothing category, attribute tags, weather context, dressing scene information and suggestion text, perform vector encoding on clothing images and cross-modal retrieval text respectively, and retrieve multiple candidate matching items from the clothing matching item vector index based on the obtained image query vector and text query vector.

[0013] The candidate re-ranking module is used to determine a comprehensive score based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability and scene consistency of each candidate item, and adjust the weight of at least one scoring item according to the recognition confidence, and re-rank multiple candidate items based on the comprehensive score.

[0014] Preferably, the system also includes a single-item library construction module; the single-item library construction module is used to construct a clothing matching single-item library; the clothing matching single-item library includes multiple clothing matching single items, each clothing matching single item includes at least one of clothing image, clothing category, clothing text description and attribute tag; the single-item library construction module is also used to perform vector encoding and normalization on each clothing matching single item to obtain the corresponding image vector and / or text vector, and construct a clothing matching single-item vector index based on the image vector and / or text vector.

[0015] Preferably, the clothing attribute recognition module includes a main feature extraction module, an adaptive grouping interaction module, a grouping context interaction module, a spatial feature selection module, a self-attention enhancement module, and a classifier connected in sequence.

[0016] The backbone feature extraction module is used to extract high-level semantic features; the adaptive grouping interaction module is used to generate interaction features; the grouping context interaction module is used to perform inter-group context interaction on the interaction features; the spatial feature selection module is used to select features for the main body area of ​​the clothing; the self-attention enhancement module is used to model the dependency relationship between different spatial locations; and the classifier is used to output clothing category, attribute label and recognition confidence.

[0017] Preferably, the adaptive grouping interaction module includes a decomposition processing branch and a feature alignment branch arranged in parallel;

[0018] In the decomposition processing branch, the high-level semantic features are embedded and mapped and the spatial dimension is split to obtain multiple spatial location features. The similarity between each spatial location feature and multiple learnable group centers is calculated and normalized to obtain group assignment weights. Based on the group assignment weights, the multiple spatial location features are weighted and aggregated to obtain multiple group features.

[0019] The feature alignment branch is used to perform linear mapping and feature dimension alignment on high-level semantic features to obtain aligned features.

[0020] Preferably, the grouping context interaction module is used to concatenate the aligned features with multiple grouping features to obtain the interaction features, perform global average pooling on the interaction features to obtain the global context, map and nonlinearly activate the global context to generate gating weights, perform residual enhancement on each grouping feature in the interaction features according to the gating weights, and fuse the residual-enhanced grouping features to obtain the fused features.

[0021] Preferably, the spatial feature selection module is used to perform average pooling and max pooling on the fused features along the channel dimension, respectively, and then convolve and nonlinearly activate the obtained pooling results to generate a spatial weight map; the fused features are weighted according to the spatial weight map, and the weighted results are residually fused with the fused features to obtain the spatial selection features.

[0022] Preferably, the self-attention enhancement module is used to reduce and flatten the spatial selection features to obtain a spatial sequence; add position encoding to the spatial sequence to obtain position enhancement features; perform linear mapping on the position enhancement features to obtain a query matrix, a key matrix and a value matrix, and obtain self-attention enhancement features through a multi-head self-attention unit;

[0023] The classifier is used to perform global average pooling and fully connected processing on the self-attention enhanced features to obtain the predicted probabilities of each category task and attribute task, and to determine the clothing category, attribute label and corresponding recognition confidence based on the predicted probabilities of each task.

[0024] Preferably, the structured prompt information includes task description, clothing attribute field, weather field, scene field, output format constraint, and length constraint;

[0025] The suggested text generation module is also used to control the length, format, and filter invalid content of the scenario-based outfit suggestions output by the large language model to obtain suggested text.

[0026] Preferably, the candidate matching item selection module is used to determine the similarity calculation method based on the modality of the item vectors of the clothing matching items; when the clothing matching item has both image vectors and text vectors, the same-modal similarity between image query vectors and image vectors, and the same-modal similarity between text query vectors and text vectors are calculated respectively; when the clothing matching item only has text vectors, the same-modal similarity between text query vectors and text vectors, and the cross-modal similarity between image query vectors and text vectors are calculated respectively; when the clothing matching item only has image vectors, the same-modal similarity between image query vectors and image vectors, and the cross-modal similarity between text query vectors and image vectors are calculated respectively; the candidate matching item selection module is also used to weightedly fuse the obtained similarities into a multimodal semantic similarity, and select a preset number of candidate matching items based on the multimodal semantic similarity;

[0027] The comprehensive score is a weighted sum of multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency.

[0028] When the recognition confidence is greater than the preset confidence threshold, the weight corresponding to clothing attribute compatibility is increased; when the recognition confidence is less than or equal to the preset confidence threshold, the weight corresponding to clothing attribute compatibility is decreased, and the weight corresponding to multimodal semantic similarity and scene consistency is increased.

[0029] Secondly, the present invention provides a cross-modal clothing item matching method based on fine-grained clothing recognition, which employs the aforementioned cross-modal clothing item matching system; the cross-modal clothing item matching method includes:

[0030] It acquires clothing images, city information, and outfit scene information uploaded by users, and obtains weather context based on city information;

[0031] High-level semantic features of clothing images are extracted, and these high-level semantic features are split and processed into multiple grouped features. The high-level semantic features are then mapped to alignment features that are aligned with the grouped features. Finally, the alignment features are concatenated with the multiple grouped features to identify clothing attributes, thereby obtaining clothing category, attribute label, and recognition confidence.

[0032] Structured prompts are constructed based on clothing category, attribute tags, recognition confidence, weather context, and dressing scenario information. The structured prompts are then input into a large language model to obtain suggested text.

[0033] A cross-modal retrieval text is constructed based on clothing category, attribute tags, weather context, outfit scene information, and suggested text. The clothing images and cross-modal retrieval text are vector-encoded separately to obtain image query vectors and text query vectors. Based on the modality of the individual item vectors of the clothing items, intramodal similarity and / or cross-modal similarity are calculated, and the obtained similarities are fused into a multimodal semantic similarity. Multiple candidate outfit items are retrieved from the clothing item vector index based on the multimodal semantic similarity.

[0034] A comprehensive score is determined based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency of each candidate item. The weight of at least one scoring item is adjusted based on the recognition confidence, and the multiple candidate items are re-ranked based on the comprehensive score.

[0035] The beneficial effects of this invention are:

[0036] 1. This invention enhances the feature expression of local areas and fine-grained attributes of clothing by grouping high-level semantic features of clothing images and splicing the resulting grouped features with alignment features. This improves the recognition accuracy of attributes such as clothing category, collar type, sleeve type, fabric, pattern and length, and provides a reliable semantic basis for subsequent clothing item matching.

[0037] 2. This invention constructs structured prompt information from clothing category, attribute tags, recognition confidence, weather context, and dressing scenario information, and uses a large language model to generate suggested text, so that the generated dressing suggestions can simultaneously consider clothing characteristics, environmental conditions, and usage scenarios, thereby improving the scenario adaptability and readability of the dressing suggestions.

[0038] 3. This invention constructs cross-modal retrieval text based on clothing attributes, weather context, dressing scenarios, and suggested text. It utilizes the shared semantic space of a multimodal coding model to fuse same-modal similarity when clothing items have both image and text vectors, and introduces cross-modal similarity to complete the information when only a single-modal vector is available. This allows clothing items with incomplete modal information to still participate in the comprehensive similarity score, improving the completeness and relevance of candidate clothing item recall.

[0039] 4. This invention comprehensively considers multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency to reorder candidate matching items, and dynamically adjusts the weight of corresponding scoring items according to the confidence of clothing attribute recognition. This can reduce the adverse impact of low-confidence recognition results on matching results and improve the accuracy, robustness, and interpretability of clothing item matching. Attached Figure Description

[0040] Figure 1 This is a system structure diagram of Embodiment 1 of the present invention.

[0041] Figure 2 This is an overall flowchart of Embodiment 2 of the present invention.

[0042] Figure 3 This is a schematic diagram of the clothing attribute recognition module in Embodiment 2 of the present invention.

[0043] Figure 4 This is a schematic diagram of the process for constructing structured prompt information in Embodiment 2 of the present invention.

[0044] Figure 5 This is a schematic diagram of the process for selecting candidate matching items in Embodiment 2 of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings.

[0046] Example 1

[0047] like Figure 1 As shown, a cross-modal clothing item matching system based on fine-grained clothing recognition can be deployed on a server, cloud computing platform, edge computing device, or local computing device, or it can be implemented collaboratively by a user terminal and a server. The user terminal can be a mobile phone, tablet, personal computer, smart mirror, or interactive terminal with image acquisition capabilities. The server can include a central processing unit (CPU), a graphics processing unit (GPU), and / or a neural network processing unit (NPU), wherein the GPU and / or NPU are used to perform inference calculations for the clothing attribute recognition model, multimodal coding model, and large language model.

[0048] The cross-modal clothing item matching system includes an item library construction module, a multi-source input acquisition module, an image preprocessing module, a weather context acquisition module, a clothing attribute recognition module, a structured prompt information construction module, a suggestion text generation module, a candidate matching item selection module, a candidate re-ranking module, and a result output module.

[0049] The item library construction module is used to build an item library for clothing combinations. The item library includes multiple clothing combinations, each containing at least one of the following: clothing image, clothing category, clothing text description, and attribute tags. When an item has a clothing image, the module uses a multimodal coding model's image encoder to perform vector encoding on the image, resulting in an image vector. When an item has at least one of the following textual information: clothing category, clothing text description, and attribute tags, the module combines the corresponding textual information into an item description text, and then uses a multimodal coding model's text encoder to perform vector encoding, resulting in a text vector. Thus, each item has an image vector and / or a text vector. The item library can be stored on the server's local disk, in-memory database, distributed database, or dedicated vector database, and uses a nearest neighbor retrieval algorithm to return clothing combinations similar to the query vector.

[0050] In this embodiment, the multimodal coding model adopts the CLIP model. The image encoder and text encoder of the multimodal coding model map image features and text features to a unified shared semantic space, enabling direct similarity calculation between image vectors and text vectors.

[0051] The multi-source input acquisition module acquires user-uploaded clothing images, city information, and styling scenario information via the user terminal. The image preprocessing module performs format conversion, size adjustment, normalization, and tensor quantization preprocessing on the user-uploaded clothing images, and then inputs the preprocessed images into the clothing attribute recognition module and the multimodal coding model. The weather context acquisition module retrieves the weather context by calling the weather service interface based on city information. The weather service interface can call a third-party weather data interface through the server's network communication interface, or it can read the weather fields for the corresponding city from a locally cached weather database.

[0052] The clothing attribute recognition module is used to obtain clothing attribute recognition results based on the preprocessed clothing image. The clothing attribute recognition results include clothing category, attribute label, and corresponding recognition confidence score. The clothing attribute recognition module includes a backbone feature extraction module, an adaptive grouping interaction module, a grouping context interaction module, a spatial feature selection module, a self-attention enhancement module, and a classifier, connected in sequence. The backbone feature extraction module extracts high-level semantic features from the preprocessed clothing image. In this embodiment, the backbone feature extraction module uses ResNet50. The adaptive grouping interaction module divides the high-level semantic features into multiple feature subspaces with discriminative tendencies and enhances the expression of local clothing regions through group center learning and intra-group feature redistribution. The grouping context interaction module establishes contextual information interaction between different feature groups and generates dynamic gating weights based on the global context to achieve semantic complementarity between different local parts of the clothing. The spatial feature selection module generates a spatial attention weight map through average pooling and max pooling along the channel dimension to highlight the main clothing region and suppress background interference. The self-attention enhancement module models long-range dependencies between different spatial locations after spatial feature selection to improve the ability to express the overall structural relationships of the clothing. The classifier is used to classify the self-attention enhanced features, outputting the predicted probabilities for each clothing category task and attribute task. The category or attribute with the highest predicted probability in each task is taken as the recognition result, and the corresponding probability is taken as the recognition confidence score. The recognition confidence score used for candidate re-ranking can be obtained by weighted aggregation of the recognition confidence scores corresponding to the category and attribute.

[0053] The adaptive grouping interaction module includes a decomposition processing branch and a feature alignment branch set up in parallel. The decomposition processing branch embeds and maps high-level semantic features using 1×1 convolutions, then splits them spatially to obtain multiple spatial location features. It calculates the similarity between each spatial location feature and its corresponding learnable group center, and performs Softmax processing along the grouping dimension to obtain the grouped features corresponding to each spatial location feature. The feature alignment branch uses 1×1 convolutions to perform linear mapping and feature dimension alignment on high-level semantic features to obtain aligned features.

[0054] The grouped context interaction module is used to concatenate aligned features with multiple grouped features to obtain interactive features. Global average pooling is then performed on the interactive features to obtain the global context. The global context is then processed through a mapping layer and activated by a Sigmoid layer to generate gated weights. Each group of features is augmented with residuals according to the corresponding gated weights, and then a unified fusion feature is formed through concatenation, convolutional mapping, or weighted aggregation.

[0055] The spatial feature selection module performs average pooling and max pooling on the fused features along the channel dimension, concatenates the pooling results, and then processes the concatenated results using a cascaded convolutional layer and a sigmoid activation function to generate a spatial weight map. The spatial feature selection module also uses the spatial weight map to weight the fused features and performs residual fusion with the weighted results to obtain the output features of the spatial feature selection module. This highlights the main body of the garment, the neckline, sleeve type, garment outline, and fabric texture areas while preserving the original semantic information.

[0056] The self-attention enhancement module uses a 1×1 convolutional layer to reduce the dimensionality of the input features, then flattens the reduced features to obtain a spatial sequence. Corresponding positional codes are added to each sequence element to obtain positional enhancement features. A linear mapping layer is used to linearly map the positional enhancement features to obtain a query matrix Q, a key matrix K, and a value matrix V. The query matrix Q, key matrix K, and value matrix V are then input into a multi-head self-attention unit to calculate the long-range dependencies between different spatial locations, thus obtaining the self-attention enhancement features.

[0057] The structured suggestion information construction module standardizes the clothing attribute recognition results, weather context, and dressing scene information to obtain clothing attribute fields, weather fields, and scene fields. Based on these fields, it constructs structured suggestion information. The structured suggestion information includes a task description, clothing attribute fields, weather fields, scene fields, output format constraints, and length constraints. The task description instructs the large language model to generate dressing suggestions. The output format constraints limit the large language model's generation of style judgments, bottom suggestions, footwear suggestions, accessory suggestions, weather precautions, and scene precautions. The length constraints control the length of the large language model's output text.

[0058] The suggested text generation module is used to input structured prompts into the large language model and generate scenario-based outfit suggestions that include style judgment, weather adaptability, scene adaptability, and matching suggestions. It also performs length control, formatting, and invalid content filtering on these suggestions to obtain suggested text suitable for front-end display and cross-modal retrieval text construction. In this embodiment, the large language model used is the Qianwen large language model. Alternatively, the large language model can be deployed on a local server's GPU or NPU, or it can call a cloud-based large language model service via a network communication interface.

[0059] The candidate outfit selection module constructs cross-modal search text based on clothing attribute fields, weather fields, scene fields, and suggested text. It then inputs user-uploaded clothing images and the cross-modal search text into a multimodal coding model to obtain image query vectors and text query vectors. The module calculates multimodal semantic similarity based on the image and / or text vectors of the clothing outfits and retrieves Top-N candidate outfits from the clothing outfit vector index based on this multimodal semantic similarity.

[0060] The calculation method for multimodal semantic similarity is determined based on the modality of the individual item vectors of the clothing items. When a clothing item has both image vectors and text vectors, the image-modal similarity between the image query vector and the item image vector, and the text-modal similarity between the text query vector and the item text vector are calculated separately, and then weighted and fused. When a clothing item only has text vectors, the text-modal similarity between the text query vector and the item text vector, and the cross-modal similarity between the image query vector and the item text vector are calculated, and then weighted and fused. When a clothing item only has image vectors, the image-modal similarity between the image query vector and the item image vector, and the cross-modal similarity between the text query vector and the item image vector are calculated, and then weighted and fused. Each similarity can be obtained using cosine similarity or vector inner product. Since multimodal coding models map image vectors and text vectors to a shared semantic space, cross-modal similarity can complete missing individual item vector modalities, allowing clothing items with incomplete modal information to still participate in the overall similarity score.

[0061] The candidate re-ranking module is used to calculate a comprehensive score based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability and scene consistency of each candidate combination item, and adjust the weight of at least one scoring item according to the recognition confidence, and re-rank the Top-N candidate combination items according to the comprehensive score.

[0062] The results output module outputs recommendation results including an identification summary, weather summary, outfit suggestions, Top-K recommended items, and recommendation reasons, which are then displayed on the user terminal. The identification summary includes clothing category, main attribute tags, and identification confidence; the weather summary includes city, temperature, and weather condition; the outfit suggestions are natural language text generated by a large language model and then processed; the Top-K recommended items are displayed as cards, including item images, categories, names, and matching scores; the recommendation reasons are generated based on the main contributors in the fusion score and explain at least one of the following recommendation criteria: weather adaptability, scene consistency, or attribute compatibility.

[0063] In this embodiment, each functional module can be implemented by computer program instructions stored in memory, which perform corresponding functions when executed by the processor. The computational tasks involving clothing attribute recognition, multimodal coding, and large language model inference can be executed by GPUs, NPUs, tensor processors, or other AI acceleration chips. The clothing matching item library, clothing matching item vector index, model parameters, rule tables, and historical matching data can be stored in relational databases, non-relational databases, file systems, object storage systems, or vector databases. Data transmission between the user terminal, server, weather service interface, and database can be achieved via wired networks, wireless networks, mobile communication networks, or local area networks. Therefore, the system can implement a complete processing flow on hardware, including clothing image input, clothing attribute recognition, cross-modal vector coding, candidate matching item retrieval, fusion and re-ranking, and recommendation result display.

[0064] Example 2

[0065] like Figure 2 As shown, a cross-modal clothing item matching method based on fine-grained clothing recognition includes the following steps:

[0066] Step S1: Build a clothing item library

[0067] The clothing combination item library consists of multiple different clothing combination items. Each clothing combination item includes at least one of the following information: clothing image, clothing category, clothing text description, and attribute tag. Image encoders and / or text encoders using a multimodal coding model are used to vectorize and normalize the clothing combination items, obtaining corresponding image vectors and / or text vectors. The resulting clothing combination item vectors are then written into a clothing combination item vector index for online nearest neighbor retrieval.

[0068] In this embodiment, the multimodal coding model adopts the CLIP model. CLIP includes an image encoder and a text encoder; the image encoder is used to encode the clothing image into an image vector; the text encoder is used to encode the text into a text vector.

[0069] Step S2: Multi-source input acquisition and image preprocessing

[0070] Acquire user-uploaded clothing images, city information, and outfit scenario information. Outfit scenario information includes at least one of the following: commuting, campus, business, leisure, dating, sports, travel, and parties. Perform format conversion, size adjustment, normalization, and tensor preprocessing on the clothing images. Call the weather service interface based on the city information to obtain the weather context. The weather context includes at least one of the following: city, temperature, weather status, humidity, wind speed, and precipitation status.

[0071] Step S3: Obtain clothing attribute recognition results

[0072] like Figure 3 As shown, a clothing attribute recognition module is constructed. This module comprises a backbone feature extraction module, an adaptive grouping interaction module (AGIF), a grouping context interaction module (GCIA), a spatial feature selection module (SFS), a self-attention enhancement module, and a classifier, all connected sequentially. The backbone feature extraction module extracts high-level semantic features from the preprocessed clothing image. The adaptive grouping interaction module divides the high-level semantic features into multiple feature subspaces with discriminative tendencies and enhances the expression of local clothing regions through group center learning and intra-group feature redistribution. The grouping context interaction module establishes contextual information interaction between different feature groups and generates dynamic gating weights based on the global context to achieve semantic complementarity between different local parts of the clothing. The spatial feature selection module generates a spatial attention weight map through average pooling and max pooling along the channel dimension to highlight the main clothing region and suppress background interference. The self-attention enhancement module models long-range dependencies between different spatial locations after spatial feature selection to improve the ability to express the overall structural relationships of the clothing.

[0073] In this embodiment, the backbone feature extraction module uses ResNet50.

[0074] The adaptive grouping interaction module includes a parallel decomposition processing branch and a feature alignment branch. In the decomposition processing branch, high-level semantic features are embedded and mapped using 1×1 convolutions, and then split along the spatial dimension to obtain multiple spatial location features. The similarity between each spatial location feature and its corresponding set learnable group center is calculated, and softmax processing is performed along the grouping dimension to obtain the grouped features corresponding to each spatial location feature. In the feature alignment branch, 1×1 convolutions are used to linearly map and align the high-level semantic features along the feature dimension to obtain aligned features.

[0075] In the grouped context interaction module, the aligned features are concatenated with multiple grouped features to obtain the interaction features; the interaction features are subjected to global average pooling to obtain the group-level global context; the global context is activated by a mapping layer and a Sigmoid function to generate gated weights; each group feature is augmented with residuals according to the corresponding gated weights, and then a unified fusion feature is formed by concatenation, convolutional mapping or weighted aggregation.

[0076] In the spatial feature selection module, average pooling and max pooling are performed on the fused features along the channel dimension. The pooling results are then concatenated and processed using a cascaded convolutional layer and a sigmoid activation function to generate a spatial weight map. The spatial weight map is used to weight the fused features, and the weighted result is then residually fused with the fused features to obtain the output features of the spatial feature selection module. This highlights the main body of the garment, the neckline, sleeve type, garment outline, and fabric texture areas while preserving the original semantic information.

[0077] In the self-attention enhancement module, a 1×1 convolutional layer is used to reduce the dimensionality of the input features, and then the reduced features are flattened to obtain a spatial sequence. Corresponding positional codes are added to each sequence element to obtain positional enhancement features. A linear mapping layer is used to linearly map the positional enhancement features to obtain a query matrix Q, a key matrix K, and a value matrix V. The query matrix Q, key matrix K, and value matrix V are then input into a multi-head self-attention unit to calculate the long-range dependencies between different spatial locations, thus obtaining the self-attention enhancement features.

[0078] In the classifier, cascaded global average pooling and fully connected layers are used to process the self-attention enhanced features, obtaining the predicted probabilities for each clothing category task and attribute task, respectively. The category or attribute corresponding to the highest predicted probability in each task is taken as the recognition result, and the corresponding probability is taken as the corresponding recognition confidence. The recognition confidence used for candidate re-ranking can be obtained by weighted aggregation of the recognition confidence corresponding to the category and attribute.

[0079] In one optional implementation, the clothing attribute recognition module can be trained on a self-built fine-grained clothing dataset FashionNet and public datasets FashionAI and DeepFashion to learn multi-task classification capabilities such as clothing category, collar type, sleeve type, fabric, pattern, garment length, and seasonal attributes. The dataset names mentioned above are merely examples of training implementations and do not constitute a limitation on the scope of protection of this invention.

[0080] Step S4: Construct structured prompt information

[0081] The clothing image is input into the clothing attribute recognition module to obtain the clothing attribute recognition results. The clothing attribute recognition results include clothing category, attribute label, and corresponding recognition confidence score. Clothing category includes at least one of tops, pants, skirts, coats, and shoes; attribute label includes at least one of collar type, sleeve type, fabric, fit, length, and season attribute.

[0082] like Figure 4As shown, the clothing attribute recognition results, weather context, and outfit scene information are standardized to obtain clothing attribute fields, weather fields, and scene fields. Structured prompt information is then constructed based on these fields. The structured prompt information includes a task description, clothing attribute fields, weather fields, scene fields, output format constraints, and length constraints. Specifically, the task description instructs the large language model to generate outfit suggestions; the output format constraints limit the large language model to generate style judgments, bottom suggestions, footwear suggestions, accessory suggestions, weather precautions, and scene precautions; and the length constraints control the output length.

[0083] Step S5: Generate suggestion text

[0084] The structured prompts are input into a large language model to generate scenario-based outfit suggestions that include style judgment, weather adaptability, scene adaptability, and matching advice. The scenario-based outfit suggestions are then subjected to length control, formatting, and filtering of invalid content to obtain suggested text that can be used for front-end display and cross-modal retrieval text construction.

[0085] In this embodiment, the large language model adopted is the Qianwen large language model.

[0086] Step S6: Select candidate matching items

[0087] like Figure 5 As shown, cross-modal retrieval text is constructed based on clothing attribute fields, weather fields, scene fields, and suggested text. Clothing images and cross-modal retrieval text are input into a multimodal coding model to obtain image query vectors and text query vectors, respectively. Multimodal semantic similarity is calculated based on the image vectors and / or text vectors of the clothing items, and Top-N candidate clothing items are retrieved from the clothing item vector index based on the multimodal semantic similarity. The specific process for obtaining multimodal semantic similarity is as follows:

[0088] When an item in a clothing combination has both image vectors and text vectors, the same-modal similarity between the image query vector and the item image vector, and the same-modal similarity between the text query vector and the item text vector are calculated separately. When an item in a clothing combination only has text vectors, the same-modal similarity between the text query vector and the item text vector, and the cross-modal similarity between the image query vector and the item text vector are calculated separately. When an item in a clothing combination only has image vectors, the same-modal similarity between the image query vector and the item image vector, and the cross-modal similarity between the text query vector and the item image vector are calculated separately. The corresponding similarities are then weighted and fused to obtain the multimodal semantic similarity.

[0089] Step S7: Candidate Reordering

[0090] The overall score (Score) for each candidate outfit is calculated based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency of each candidate outfit item. The expression for this score is:

[0091]

[0092] in, Indicates multimodal semantic similarity; Indicates clothing attribute compatibility; Indicates weather adaptability; This indicates consistency across different scenarios; These are the weighting coefficients.

[0093] In this embodiment, clothing attribute compatibility The compatibility score is obtained based on the matching compatibility relationship between clothing attribute fields and the clothing categories and attribute tags of candidate matching items. This compatibility relationship is derived from preset category matching rules, attribute compatibility matrices, manually labeled matching samples, or statistical results of historical matching data. The system pre-establishes at least one of the following: category matching rules, color compatibility rules, material compatibility rules, fit compatibility rules, season compatibility rules, and style compatibility rules; it calculates the sub-compatibility scores of candidate matching items in the dimensions of category, color, material, fit, season, and style, and then sums them according to preset weights to obtain the clothing attribute compatibility score.

[0094] In this embodiment, weather adaptability The system obtains its compatibility score based on the matching relationship between weather fields and the clothing category and attribute tags of candidate outfit items. This matching relationship can be obtained from a preset weather compatibility rule table, manually labeled outfit samples, or statistical results of historical outfit data. The system standardizes at least one of temperature, weather conditions, humidity, wind force, and precipitation conditions as weather constraints, and establishes compatibility rules between weather constraints and the warmth, breathability, waterproofness, material, thickness, color, and seasonal tags of candidate outfit items. Sub-compatibility scores for candidate outfit items in the dimensions of temperature, precipitation, humidity, season, and color are calculated separately, and then weighted and summed according to preset weights to obtain the weather compatibility score.

[0095] In this embodiment, scenario consistency The consistency score is obtained by comparing the scenario field with the clothing category, attribute tags, and style tags of the candidate outfit items. The consistency relationship can be obtained from preset scenario matching rules, manually labeled matching samples, or statistical results of historical matching data. Sub-consistency scores are calculated for each candidate outfit item in at least one of the dimensions of category, style, formality, and activity suitability, and then weighted and summed according to preset weights to obtain the scenario consistency score.

[0096] In this embodiment, when the recognition confidence is greater than the preset confidence threshold, the weight coefficient β is increased, thereby enhancing the role of clothing attribute compatibility in the comprehensive score and making the recommendation results more dependent on fine-grained clothing attributes; when the recognition confidence is less than or equal to the preset confidence threshold, the weight coefficient β is decreased, and the weight coefficients α and δ are increased, thereby enhancing the impact of multimodal semantic similarity and scene consistency, so as to reduce the impact of the uncertainty of a single recognition result on the recommendation results.

[0097] Step S8: Output recommendation results

[0098] The system displays recommendation results to users, including an identification summary, a weather summary, outfit suggestions, Top-K recommended items, and a reason for the recommendation. The identification summary includes clothing category, main attribute tags, and identification confidence; the weather summary includes city, temperature, and weather condition; the outfit suggestions are natural language text generated by a large language model and then processed; recommended items can be displayed in card format, showing the item image, category, name, and matching score; the reason for the recommendation is generated based on the main contributors in the fusion score, explaining the main recommendation criteria such as weather adaptability, scene consistency, or attribute compatibility.

[0099] It should be understood that the above embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Various modifications or substitutions can be made to the present invention by those skilled in the art without departing from the concept of the present invention, and all such modifications or substitutions should fall within the scope of protection of the claims of the present invention.

Claims

1. A cross-modal clothing item matching system based on fine-grained clothing recognition, characterized in that: It includes a multi-source input acquisition module, a weather context acquisition module, a clothing attribute recognition module, a structured prompt information construction module, a suggestion text generation module, a candidate matching item selection module, and a candidate re-sorting module; The multi-source input acquisition module is used to acquire clothing images, city information, and outfit scene information uploaded by users; The weather context acquisition module is used to acquire weather context based on city information; The clothing attribute recognition module is used to extract high-level semantic features from clothing images, split and process the high-level semantic features into multiple group features, map the high-level semantic features into alignment features aligned with the group features, and concatenate the alignment features with multiple group features to perform clothing attribute recognition, thereby obtaining clothing category, attribute label and recognition confidence. The structured prompt information construction module is used to construct structured prompt information based on clothing category, attribute tags, recognition confidence, weather context, and dressing scene information; The suggested text generation module is used to input structured prompt information into a large language model to obtain suggested text; The candidate matching item selection module is used to construct cross-modal retrieval text based on clothing category, attribute tags, weather context, dressing scene information and suggestion text, perform vector encoding on clothing images and cross-modal retrieval text respectively, and retrieve multiple candidate matching items from the clothing matching item vector index based on the obtained image query vector and text query vector. The candidate re-ranking module is used to determine a comprehensive score based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability and scene consistency of each candidate item, and adjust the weight of at least one scoring item according to the recognition confidence, and re-rank multiple candidate items based on the comprehensive score.

2. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 1, characterized in that: It also includes a single-item library construction module; the single-item library construction module is used to construct a clothing matching single-item library; the clothing matching single-item library includes multiple clothing matching single items, each clothing matching single item includes at least one of clothing image, clothing category, clothing text description and attribute tag; the single-item library construction module is also used to perform vector encoding and normalization on each clothing matching single item to obtain the corresponding image vector and / or text vector, and construct a clothing matching single-item vector index based on the image vector and / or text vector.

3. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 1, characterized in that: The clothing attribute recognition module includes a main feature extraction module, an adaptive grouping interaction module, a grouping context interaction module, a spatial feature selection module, a self-attention enhancement module, and a classifier connected in sequence. The backbone feature extraction module is used to extract high-level semantic features; the adaptive grouping interaction module is used to generate interaction features; the grouping context interaction module is used to perform inter-group context interaction on the interaction features; the spatial feature selection module is used to select features for the main body area of ​​the clothing; the self-attention enhancement module is used to model the dependency relationship between different spatial locations; and the classifier is used to output clothing category, attribute label and recognition confidence.

4. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 3, characterized in that: The adaptive grouping interaction module includes a decomposition processing branch and a feature alignment branch set in parallel. In the decomposition processing branch, high-level semantic features are embedded and mapped and spatial dimension decomposed to obtain multiple spatial location features. The similarity between each spatial location feature and multiple learnable group centers is calculated and normalized to obtain group assignment weights; the multiple spatial location features are weighted and aggregated according to the group assignment weights to obtain multiple group features; The feature alignment branch is used to perform linear mapping and feature dimension alignment on high-level semantic features to obtain aligned features.

5. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 4, characterized in that: The grouping context interaction module is used to concatenate the alignment feature with multiple grouping features to obtain the interaction feature, and to perform global average pooling on the interaction feature to obtain the global context. The global context is mapped and nonlinearly activated to generate gated weights; Based on the gating weights, residual enhancement is performed on each group of interactive features, and the grouped features after residual enhancement are fused to obtain fused features.

6. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 5, characterized in that: The spatial feature selection module is used to perform average pooling and max pooling on the fused features along the channel dimension, respectively. The pooling results are concatenated and then subjected to convolution and nonlinear activation to generate a spatial weight map. The fused features are weighted according to the spatial weight map, and the weighted results are residually fused with the fused features to obtain the spatial selection features.

7. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 6, characterized in that: The self-attention enhancement module is used to reduce and flatten the spatial selection features to obtain a spatial sequence; position encoding is added to the spatial sequence to obtain position-enhanced features; Linear mapping is performed on the location-enhanced features to obtain the query matrix, key matrix, and value matrix, and self-attention-enhanced features are obtained through multi-head self-attention units; The classifier is used to perform global average pooling and fully connected processing on the self-attention enhanced features to obtain the predicted probabilities of each category task and attribute task, and to determine the clothing category, attribute label and corresponding recognition confidence based on the predicted probabilities of each task.

8. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 1, characterized in that: The structured prompt information includes task description, clothing attribute field, weather field, scene field, output format constraints, and length constraints; The suggested text generation module is also used to control the length, format, and filter invalid content of the scenario-based outfit suggestions output by the large language model to obtain suggested text.

9. The cross-modal clothing item matching system based on fine-grained clothing recognition according to claim 1, characterized in that: The candidate matching item selection module is used to determine the similarity calculation method based on the modality of the item vectors of the clothing matching items; when the clothing matching items have both image vectors and text vectors, the image same modality similarity between the image query vector and the image vector and the text same modality similarity between the text query vector and the text vector are calculated respectively. When the clothing item only has text vectors, the same-modal similarity between text query vectors and text vectors, and the cross-modal similarity between image query vectors and text vectors are calculated respectively; when the clothing item only has image vectors, the same-modal similarity between image query vectors and image vectors, and the cross-modal similarity between text query vectors and image vectors are calculated respectively; the candidate item selection module is also used to weight and fuse the obtained similarities into a multimodal semantic similarity, and select a preset number of candidate items based on the multimodal semantic similarity; The comprehensive score is a weighted sum of multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency; when the recognition confidence is greater than the preset confidence threshold, the weight corresponding to clothing attribute compatibility is increased. Otherwise, reduce the weight corresponding to clothing attribute compatibility and increase the weight corresponding to multimodal semantic similarity and scene consistency.

10. A cross-modal clothing item matching method based on fine-grained clothing recognition, characterized in that: The cross-modal clothing item matching system based on fine-grained clothing recognition described in claim 1 is employed. This cross-modal clothing item matching method includes: It acquires clothing images, city information, and outfit scene information uploaded by users, and obtains weather context based on city information; High-level semantic features of clothing images are extracted, and these high-level semantic features are split and processed into multiple grouped features. The high-level semantic features are then mapped to alignment features that are aligned with the grouped features. Finally, the alignment features are concatenated with the multiple grouped features to identify clothing attributes, thereby obtaining clothing category, attribute label, and recognition confidence. Structured prompts are constructed based on clothing category, attribute tags, recognition confidence, weather context, and dressing scenario information. The structured prompts are then input into a large language model to obtain suggested text. A cross-modal retrieval text is constructed based on clothing category, attribute tags, weather context, outfit scene information, and suggested text. The clothing images and cross-modal retrieval text are vector-encoded separately to obtain image query vectors and text query vectors. Based on the modality of the individual item vectors of the clothing items, intramodal similarity and / or cross-modal similarity are calculated, and the obtained similarities are fused into a multimodal semantic similarity. Multiple candidate outfit items are retrieved from the clothing item vector index based on the multimodal semantic similarity. A comprehensive score is determined based on the multimodal semantic similarity, clothing attribute compatibility, weather adaptability, and scene consistency of each candidate item. The weight of at least one scoring item is adjusted based on the recognition confidence, and the multiple candidate items are re-ranked based on the comprehensive score.

Images