Clothing attribute recognition method and system based on multi-modal information hierarchical semantic modeling

Abstract

The present application relates to the technical field of clothing attribute recognition, in particular to a clothing attribute recognition method and system based on multi-modal information hierarchical semantic modeling, which starts from the semantic structure of clothing attributes, reasonably constructs the modeling mode of different hierarchical attributes, clearly defines the master-slave relationship of image and text modalities in each attribute level, and improves the stability and reliability of model learning under weak supervision through an adaptive fusion mechanism, thereby effectively overcoming the deficiencies of the prior art in semantic consistency, multi-modal fusion efficiency and self-training noise control, and providing a more practical application scenario suitable technical solution for complex clothing attribute recognition.

Description

Technical Field

[0001] This invention relates to the field of clothing attribute recognition technology, and in particular to a method and system for clothing attribute recognition based on multimodal information hierarchical semantic modeling. Background Technology

[0002] With the rapid development of e-commerce, intelligent design, virtual try-on, and clothing recommendation systems, clothing attribute recognition, as a fundamental step in intelligent clothing understanding and analysis, has become an important research direction in the fields of computer vision and multimodal information processing. The goal of clothing attribute recognition is to automatically identify multidimensional attributes of a product, such as color, material, style, fit, and overall style, from clothing images and related text information, providing structured semantic support for subsequent retrieval, recommendation, generation, and design decisions.

[0003] However, unlike general target classification or simple attribute recognition tasks, clothing attributes possess significant complexity and unique characteristics. First, clothing attributes are not independent of each other but naturally exhibit a clear hierarchical dependency. For example, overall style usually imposes prior constraints on style structure, which in turn further restricts the range of selectable materials and colors. This hierarchical relationship between "style," "style," "material," and "color" determines that clothing attribute recognition is essentially a complex recognition problem with structural constraints, rather than a simple combination of several independent classification tasks.

[0004] Secondly, clothing attributes exhibit strong semantic coupling and exclusivity. Certain materials appear only in specific styles or designs, and similar attributes (such as different lightweight fabrics or similar color schemes) have minimal visual differences, easily leading to confusion. This places higher demands on the feature representation ability and discrimination stability of clothing attribute recognition at a fine-grained level.

[0005] Furthermore, clothing attribute annotation is costly and highly subjective, especially for high-level semantic attributes such as material and style. Manual annotation often suffers from inconsistencies and incompleteness, resulting in a limited amount of high-quality labeled data available for training. Against this backdrop, many studies have attempted to introduce semi-supervised learning methods such as self-training to utilize large amounts of unlabeled data. However, because clothing attributes themselves have strong hierarchical constraints and logical relationships, without effective structural constraint mechanisms, pseudo-labels generated during self-training are prone to logical conflicts and accumulate errors, ultimately reducing model performance.

[0006] Most existing clothing attribute recognition technologies rely on deep neural networks for feature extraction and classification prediction, typically treating different attributes as independent classification targets. While these methods can improve the recognition accuracy of single attributes to some extent, they generally neglect the hierarchical structure and dependencies between clothing attributes, simplifying the complex clothing attribute recognition problem into a flattened multi-classification problem. This leads to issues such as attribute conflicts, unstable recognition, and insufficient generalization ability in complex scenarios. Consequently, the generative model is prone to attribute conflicts or semantic inconsistencies when faced with complex product descriptions, thus limiting the model's application effectiveness in personalized recommendation scenarios.

[0007] Furthermore, while existing multimodal clothing attribute recognition methods incorporate textual information to aid visual understanding, most employ equal-weight fusion, global stitching, or uniform alignment to process image and text modalities, failing to adequately consider the varying dependencies of different attribute levels on modal information. In these multimodal recognition methods, textual information is often uniformly introduced across all attribute recognition stages. This can introduce noise interference in low-level visual attribute recognition and is insufficient to provide adequate constraints in high-level semantic attribute determination, resulting in unstable multimodal fusion performance and limited improvement in attribute recognition results.

[0008] Meanwhile, the distribution of attributes in clothing datasets is highly unbalanced, with very few samples for some fine-grained or high-level semantic attributes, resulting in insufficient discriminative ability of the model for these attributes. Traditional self-training methods often rely on a single confidence threshold in the pseudo-label selection process, making it difficult to simultaneously consider the reliability and consistency of attributes at different levels. In clothing attribute recognition, a task with a complex hierarchical structure, once a high-level attribute prediction deviates, erroneous pseudo-labels are easily amplified during subsequent training and propagated to multiple related attributes, causing model performance degradation and reducing the accuracy of attribute recognition.

[0009] Existing clothing attribute recognition technologies suffer from problems such as flattened attribute modeling, coarse-grained utilization of multimodal information, and insufficient self-training stability in complex scenarios.

[0010] Therefore, how to explicitly introduce an attribute hierarchy in the process of clothing attribute recognition, rationally divide complex attributes, and build a recognition model with structural constraints and a stable and reliable self-training mechanism on this basis has become a key scientific and technological problem restricting the further development of existing clothing attribute recognition technology. Summary of the Invention

[0011] Therefore, the technical problem to be solved by the present invention is to overcome the problems in the prior art, such as the flattening of clothing attribute modeling, the instability of multimodal information fusion and the insufficient stability of model self-training, which makes it difficult to improve the accuracy and stability of clothing attribute recognition.

[0012] To address the aforementioned technical problems, this invention provides a method for clothing attribute recognition based on multimodal information hierarchical semantic modeling, comprising:

[0013] A multimodal attribute recognition model is constructed, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor; the classifier adopts a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier;

[0014] The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features;

[0015] The visual and text features of the corresponding levels are mapped to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features.

[0016] After noise enhancement, the clothing samples are input into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features;

[0017] Intramodal contrast loss is constructed for each level based on visual features and enhanced visual features, text features and enhanced text features of the corresponding level; cross-modal contrast loss is constructed for each level based on multi-level aligned visual features and multi-level aligned text features; hierarchical consistency loss is constructed for each level based on aligned visual features and aligned text features of adjacent levels.

[0018] The total loss function is constructed based on the intra-modal contrast loss, cross-modal contrast loss, and hierarchical consistency loss at each level, as well as the classification loss.

[0019] Based on the total loss function, a multimodal attribute recognition model is trained using labeled clothing samples to obtain an initial multimodal attribute recognition model.

[0020] Using the initial multimodal attribute recognition model, pseudo-labels for unlabeled clothing samples are obtained; the clothing samples with pseudo-labels and the clothing samples with labels are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model.

[0021] Using a trained multimodal attribute recognition model, predictive classification labels for clothing samples to be identified are obtained.

[0022] Preferably, the intra-modal contrast loss for each level is constructed based on the visual features and enhanced visual features of the corresponding level, and the text features and enhanced text features of the corresponding level, including:

[0023] For all levels, the intra-modal contrast loss for each level of the visual modality is constructed based on the visual features and enhanced visual features of the corresponding level, and the formula is as follows:

[0024] ;

[0025] For the middle and high levels, the intra-modal contrast loss for each level of the text modality is constructed based on the text features and enhanced text features of the corresponding level, and the formula is as follows:

[0026] ;

[0027] in, Let be the intra-modal contrast loss at the k-th level of the visual modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be sample indices. Let be the visual feature of the k-th level of the i-th sample. and These are the enhanced visual features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. For temperature coefficient, Let be the intra-modal contrast loss at the k-th level of the text modality. Let be the text feature of the k-th level of the i-th sample. and These are the enhanced text features at the k-th level for the i-th and j-th samples, respectively.

[0028] Preferably, the cross-modal contrast loss for each level is constructed based on multi-level aligned visual features and multi-level aligned text features, and the calculation formula is as follows:

[0029] ;

[0030] in, For the cross-modal contrast loss of the k-th level, Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be sample indices. Let be the alignment visual feature of the k-th level for the i-th sample. and These are the aligned text features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. This is the temperature coefficient.

[0031] Preferably, the hierarchical consistency loss for each level is constructed based on the alignment visual features and alignment text features of adjacent levels, and the calculation formula is as follows:

[0032] ;

[0033] ;

[0034] in, Let the hierarchical consistency loss be the k-th level of the visual modality. Let be the hierarchical consistency loss for the k-th level of the text modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i be the sample index. and These are the alignment visual features of the k-th and (k+1)-th levels of the i-th sample, respectively. and These are the aligned text features of the k-th and (k+1)-th levels of the i-th sample, respectively. The cosine similarity function is used. This is the temperature coefficient.

[0035] Preferably, the predicted classification label of the clothing sample to be identified is obtained using a trained multimodal attribute recognition model, including:

[0036] The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features;

[0037] By fusing the visual and textual features of the corresponding levels, multi-level fused features are obtained.

[0038] Multi-level visual features, text features, and fusion features are input into the classifier, which then passes through the visual classifier, text classifier, and fusion classifier respectively to obtain the visual prediction class probability, text prediction class probability, and fusion prediction class probability at each level. Finally, a collaborative decision-making mechanism is used to obtain the predicted classification label at each level.

[0039] Preferably, the multi-level visual features, text features, and fused features are input into the classifier. Before passing through the visual classifier, text classifier, and fused classifier, respectively, a text information determination mechanism is introduced, including:

[0040] For the middle and high layers, the similarity between the aligned visual features and aligned text features of the corresponding layers is calculated respectively;

[0041] If the similarity of the current level is greater than or equal to the preset threshold, the visual features, text features and fusion features of the current level are processed by the visual classifier, text classifier and fusion classifier respectively to obtain the visual prediction category probability, text prediction category probability and fusion prediction category probability of the current level; then the collaborative decision-making mechanism is used to obtain the predicted classification label of each level.

[0042] If the similarity of the current level is less than a preset threshold, the visual features of the current level are processed by a visual classifier to obtain the visual prediction category probability of the current level; the predicted classification label of the current level is obtained based on the visual prediction category probability of the current level.

[0043] Preferably, a collaborative decision-making mechanism is used to obtain predicted classification labels at each level, including:

[0044] The predicted category probabilities for each level (visual, text, and fused) are weighted and summed to obtain the predicted classification label for each level. The formula is as follows:

[0045] ;

[0046] in, Let k be the predicted classification label for the k-th level. , and These represent the visual prediction category probability, text prediction category probability, and fusion prediction category probability at the k-th level, respectively. , and These are the visual classification weights, text classification weights, and fusion classification weights for the k-th level, respectively.

[0047] Preferably, the calculation method for the visual classification weight, text classification weight, and fusion classification weight includes:

[0048] For each level, the maximum class probability among the visual prediction class probability, text prediction class probability, and fusion prediction class probability of that level is extracted as the prediction confidence of each modality at that level.

[0049] The initial weights of each modality at this level are calculated based on the prediction confidence of each modality at this level.

[0050] Calculate the consistency deviation between each classifier in this layer and other classifiers, and calculate the consistency inhibition weight of each modality in this layer based on the consistency deviation;

[0051] Based on the initial weights and consistency suppression weights of each modality at this level, the classification weights of each modality are calculated using the following formula:

[0052] ;

[0053] in, Let m be the classification weight of the k-th level modality. and All are modal indexes. Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. and These are the k-th level modes. and modality The initial weights, and These are the k-th level modes. and modality Consistency suppression weights.

[0054] Preferably, the initial weights of each modality at this level are calculated based on the prediction confidence of each modality, using the following formula:

[0055] ;

[0056] in, For the k-th level mode The initial weights, For modal indexing, Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. Let m be the prediction confidence level of the k-th level mode. Let be the prediction confidence score for the k-th level visual modality. Let k be the prediction confidence level for the k-th level text modality. Let be the prediction confidence level of the k-th level fusion mode. These are the initial weight adjustment parameters;

[0057] Based on the aforementioned consistency deviation, the consistency suppression weights for each mode at this level are calculated using the following formula:

[0058] ;

[0059] in, For the k-th level mode Consistency suppression weights, For consistency adjustment parameters, For the k-th level mode Consistency bias of classifiers.

[0060] This invention also provides a clothing attribute recognition system based on multimodal information hierarchical semantic modeling, comprising:

[0061] The model building module is used to build a multimodal attribute recognition model, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor; the classifier adopts a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier;

[0062] The feature extraction module is used to input clothing samples containing original images and text descriptions into the feature extractor to obtain multi-level visual features and multi-level text features;

[0063] The feature alignment module is used to map visual features and text features of corresponding levels to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features.

[0064] The feature enhancement module is used to enhance the noise of clothing samples before inputting them into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features;

[0065] The loss function construction module is used to construct intra-modal contrast loss for each level based on visual features and enhanced visual features, text features and enhanced text features of the corresponding level; construct cross-modal contrast loss for each level based on multi-level aligned visual features and multi-level aligned text features; construct hierarchical consistency loss for each level based on aligned visual features and aligned text features of adjacent levels; and construct the total loss function based on intra-modal contrast loss, cross-modal contrast loss, hierarchical consistency loss, and classification loss for each level.

[0066] The initial training module is used to train a multimodal attribute recognition model based on the total loss function using labeled clothing samples, thus obtaining the initial multimodal attribute recognition model.

[0067] The retraining module is used to obtain pseudo-labels for unlabeled clothing samples using the initial multimodal attribute recognition model; the pseudo-labeled clothing samples and labeled clothing samples are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model.

[0068] The recognition module is used to obtain the predicted classification label of the clothing sample to be recognized by utilizing the trained multimodal attribute recognition model.

[0069] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0070] The present invention discloses a clothing attribute recognition method based on hierarchical semantic modeling of multimodal information. First, according to the inherent multi-level semantic structure in clothing attribute recognition tasks, the multimodal features are divided into multiple levels. During the feature extraction stage, fine-grained attributes (such as color and texture), medium-grained attributes (such as material and structure), and high-level semantic attributes (such as style) are modeled hierarchically. During the training stage, a hierarchical consistency loss is constructed to constrain the semantic progression relationship between adjacent levels. This allows the model to form a clear and semantically continuous attribute representation space while maintaining local discriminative ability. As a result, the stability, interpretability, and overall recognition accuracy in complex clothing attribute recognition tasks are significantly improved, thus solving the problems of flattened attribute modeling and mixed semantics at different levels in existing clothing attribute recognition technologies.

[0071] Furthermore, this invention, based on multi-level features, constructs a visual classifier, a text classifier, and a fusion classifier at each semantic level, and introduces a collaborative decision-making mechanism. This mechanism dynamically adjusts the weights of each classifier by combining prediction confidence with the consistency of each classifier, generating highly reliable collaborative decision-making prediction results. By improving the accuracy of the prediction results, this invention can significantly improve the accuracy and stability of pseudo-labels during self-training, suppressing the risk of propagating erroneous pseudo-labels. This allows the model to continuously improve performance even with limited labeled data or uneven attribute distribution, thereby achieving high-precision, continuously optimized clothing attribute recognition under weak supervision. This effectively solves the problems of traditional self-training methods, which rely on a single confidence threshold for pseudo-label selection and are susceptible to noise accumulation.

[0072] Furthermore, this invention primarily uses visual modalities, supplemented by textual modalities layer by layer, to construct a multimodal contrastive learning mechanism within each modality and across modalities. By prioritizing visual features at the low-level semantic stage and introducing textual features to participate in cross-modal alignment and discrimination at the mid-to-high-level semantic stages, and combining the joint constraints of intra-modal and cross-modal contrastive learning, this invention can fully utilize the ability of text to supplement abstract attributes such as material and style while avoiding misleading textual information regarding visually dominant attributes such as color and texture. This achieves multimodal semantic alignment that better conforms to the cognitive rules of clothing attributes, improves the robustness and generalization ability of the model in real e-commerce and weakly descriptive scenarios, and effectively overcomes the problems of indiscriminate modal status, severe textual noise interference, or coarse cross-modal alignment in existing multimodal attribute recognition methods. Attached Figure Description

[0073] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0074] Figure 1 This is a flowchart of a clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to the present invention;

[0075] Figure 2 This is a flowchart of the self-training process of the present invention. Detailed Implementation

[0076] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0077] Reference Figure 1 As shown, this invention proposes a method for clothing attribute recognition based on hierarchical semantic modeling of multimodal information, including:

[0078] A multimodal attribute recognition model is constructed, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor; the classifier adopts a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier;

[0079] The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features;

[0080] The visual and text features of the corresponding levels are mapped to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features.

[0081] After noise enhancement, the clothing samples are input into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features;

[0082] Intramodal contrast loss is constructed for each level based on visual features and enhanced visual features, text features and enhanced text features of the corresponding level; cross-modal contrast loss is constructed for each level based on multi-level aligned visual features and multi-level aligned text features; hierarchical consistency loss is constructed for each level based on aligned visual features and aligned text features of adjacent levels.

[0083] The total loss function is constructed based on the intra-modal contrast loss, cross-modal contrast loss, and hierarchical consistency loss at each level, as well as the classification loss.

[0084] Based on the total loss function, a multimodal attribute recognition model is trained using labeled clothing samples to obtain an initial multimodal attribute recognition model.

[0085] Using the initial multimodal attribute recognition model, pseudo-labels for unlabeled clothing samples are obtained; the clothing samples with pseudo-labels and the clothing samples with labels are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model.

[0086] Using a trained multimodal attribute recognition model, predictive classification labels for clothing samples to be identified are obtained.

[0087] Specifically, using a trained multimodal attribute recognition model, the predicted classification labels of the clothing samples to be identified are obtained. The steps include:

[0088] The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features;

[0089] By fusing the visual and textual features of the corresponding levels, multi-level fused features are obtained.

[0090] Multi-level visual features, text features, and fusion features are input into the classifier, which then passes through the visual classifier, text classifier, and fusion classifier respectively to obtain the visual prediction class probability, text prediction class probability, and fusion prediction class probability at each level. Finally, a collaborative decision-making mechanism is used to obtain the predicted classification label at each level.

[0091] Most existing attribute recognition methods employ a general multi-classification or multi-label learning framework, treating different attributes as independent prediction targets and jointly training them within the same feature space. These methods exhibit good versatility and efficiency in general object attribute recognition tasks.

[0092] However, clothing attributes possess structural characteristics that significantly differ from those of general attribute tasks. Clear hierarchical relationships and semantic constraints naturally exist between clothing attributes; for example, overall style typically determines style type, while style structure further limits the range of selectable materials and colors. General attribute recognition techniques ignore this hierarchical characteristic, flattening clothing attributes, which easily leads to the following problems: first, semantic conflicts arise between attributes at different levels; second, the lack of constraints between higher-level and lower-level attributes affects overall recognition consistency; and third, when distinguishing fine-grained attributes, the model is susceptible to local noise interference, resulting in unstable recognition results.

[0093] To address the issue of insufficient adaptability of general attribute recognition technology in clothing scenarios, this invention introduces a multimodal information hierarchical modeling strategy based on the semantic rules of clothing attributes themselves. This strategy enables structured modeling of clothing attributes, allowing the model to express features and determine attributes at different semantic levels, thereby significantly improving the stability, rationality, and interpretability of complex clothing attribute recognition.

[0094] The multimodal attribute recognition model constructed in this invention utilizes a feature extractor to extract multi-level visual features and multi-level text features respectively. Specifically, for each element containing the original image... and text description Clothing samples, using the original images The input is fed into a visual feature extractor to obtain multi-level visual features. ; describe the text The input is fed into a text feature extractor to obtain multi-level text features. . It mainly represents the low- and mid-level visual information of clothing, such as color distribution, texture structure, and silhouette. It mainly represents high-level semantic information such as material, style, and wearing context that is difficult to infer from image stability alone.

[0095] Original image The input is fed into a visual feature extractor to obtain multi-level visual features. The process includes:

[0096] First, examine the original image. Standardized preprocessing is performed, including size normalization, pixel normalization, and data augmentation operations, to ensure consistency of different samples in the encoder input space.

[0097] Subsequently, the preprocessed image is input into the visual encoder for forward propagation to obtain multi-level visual coding features;

[0098] Finally, to eliminate the differences in spatial and channel dimensions between features at different levels and to facilitate subsequent cross-modal alignment and contrastive learning, the multi-level visual encoding features were input into independent feature mapping modules for each level. Pooling, flattening, and linear transformation were performed on the visual encoding features corresponding to each level, uniformly mapping the visual encoding features to a feature vector space of the same dimension, thus obtaining multi-level visual features. .

[0099] The visual encoder in this embodiment employs a hierarchical deep neural network. Its internal structure contains progressively deeper feature representations, allowing for selective extraction of feature maps at different depths according to a preset attribute hierarchy. The visual encoder in this embodiment comprises three layers: low-level, mid-level, and high-level, as detailed below:

[0100] First, low-level visual features are extracted in the low-level stage of the visual encoder. This stage corresponds to the convolutional layers or self-attention blocks near the input in the network, and mainly represents low-level visual features such as color distribution, local texture, and edge information in the image. The feature map output from this stage is denoted as the low-level visual encoding feature.

[0101] Secondly, mid-level visual features are extracted in the intermediate stage of the visual encoder. This stage corresponds to the intermediate layer structure in the network with a certain receptive field, used to capture local structural information and component combination relationships of clothing, such as collar shape, sleeve structure, or overall outline. The feature map output from this stage is denoted as the mid-level visual encoding feature.

[0102] Next, high-level visual features are extracted in the high-level stage of the visual encoder. This stage corresponds to the high-level representation near the output of the network, used to express the overall style, tone of the clothing, and global semantic information. The feature map output from this stage is denoted as the high-level visual encoding feature.

[0103] In this embodiment, the multi-level visual coding features are processed by the feature mapping module to obtain multi-level visual features. These are respectively denoted as low-level visual features. Mid-level visual features and high-rise visual features .

[0104] Similarly, for text descriptions corresponding to clothing images This invention first performs preprocessing operations on the text, including word segmentation, encoding, and length alignment. Then, the preprocessed text sequence is input into a text encoder for feature extraction, resulting in multi-level text encoding features. Finally, to achieve cross-modal alignment with visual features, the multi-level text encoding features are uniformly mapped to a representation space of the same dimension as the visual features through a feature mapping module at the corresponding level, thus obtaining multi-level text features. .

[0105] In this embodiment, the text encoder employs a pre-trained language model with a multi-layered structure. The hidden states at each layer progressively represent the abstraction process from lexical information to syntactic semantics and then to overall semantics. Specifically, low-level text features are extracted from the low-level hidden states of the text encoder. These features primarily represent lexical or phrase-level information, such as color words and basic material terms, and are denoted as low-level text encoding features. Mid-level text features are extracted from the intermediate-level hidden states of the text encoder. These features primarily represent local semantic units formed by combinations of multiple words, such as material descriptions and structural feature phrases, and are denoted as mid-level text encoding features. High-level text features are extracted from the high-level hidden states of the text encoder. These features primarily represent sentence-level or document-level semantic information, used to describe the overall style of clothing, usage scenarios, or subjective evaluations, and are denoted as high-level text encoding features.

[0106] After passing through the feature mapping module, the multi-level text encoding features are used to obtain multi-level text features. These are respectively denoted as low-level text features. Mid-level text features and high-level text features .

[0107] Through the above-described hierarchical feature extraction process, this invention obtains feature representations that correspond one-to-one with the semantic levels of clothing attributes in both visual and textual modalities. This lays a clear and controllable feature foundation for subsequent execution of modal learning, collaborative decision-making mechanisms, and self-training updates at different levels, thereby avoiding the semantic misalignment problem caused by arbitrarily selecting intermediate layer features in existing technologies.

[0108] In obtaining multi-level visual features and multi-level text features (in After creating a hierarchical index, the present invention further constructs corresponding noise enhancement features and alignment features for each level to support subsequent intra-modal contrastive learning.

[0109] Specifically, this invention maps visual and textual features of corresponding levels to a unified feature space through an alignment mapping network, thereby obtaining multi-level aligned visual features. and multi-level aligned text features This alignment process ensures that features of different modalities and levels are comparable in subsequent stages.

[0110] After noise enhancement, the clothing samples are input into a feature extractor to obtain multi-level enhanced visual features. and multi-level enhanced text features For visual modalities, noise enhancement does not directly add random perturbations to the feature space. Instead, it applies enhancement operations such as random cropping, color perturbation, and geometric transformations to the original image, re-inputs it into the visual feature extractor, and re-extracts features at the same network layer, thereby obtaining multi-level enhanced visual features. This method ensures semantic consistency of features before and after enhancement, but differences exist in detail. For text modalities, noise enhancement is achieved by performing operations such as synonym replacement, random masking, or slight rearrangement on the original text. The enhanced text is then re-input into the text feature extractor to extract enhanced text features at the corresponding level. In this way, the present invention obtains pairs of original features and enhanced features at each level, providing positive sample pairs for subsequent intra-modal contrastive learning within the same level.

[0111] Most existing clothing attribute recognition technologies are based on visual information modeling, typically using convolutional neural networks or visual Transformers to extract features from clothing images and then performing attribute classification tasks. These methods are effective in recognizing intuitive visual attributes such as color, contour, and length, and are therefore widely used in practical applications. With the development of multimodal learning, some research has begun to incorporate textual modal information, such as product titles, attribute descriptions, or style tags, to enhance the model's semantic understanding capabilities. Existing multimodal clothing recognition methods usually employ methods such as overall image-text feature stitching, global alignment, or equal-weighted fusion, treating images and text as equally important information sources and jointly modeling them within a unified feature space.

[0112] However, directly applying general multimodal techniques to clothing attribute recognition still has significant shortcomings. Clothing attributes differ significantly at the semantic level, and different attributes have varying degrees of dependence on images and text. For example, low-level attributes such as color and texture mainly rely on visual information, while mid-to-high-level attributes such as material and style often require the combination of textual semantics for accurate determination. Existing multimodal methods do not distinguish the role of image and text modalities at different attribute levels; textual information is usually globally involved or equally weighted, easily introducing semantic noise into low-level attribute recognition, which weakens visual discrimination capabilities.

[0113] Therefore, this invention does not force text modality to participate in discrimination at all levels. Instead, it reflects the master-slave relationship of "image as the main and text as the auxiliary" by intervening at each level: in low-level attribute modeling, visual features are mainly relied upon, while in mid-to-high-level attributes (such as material and style), text features are introduced to participate in alignment and discrimination. The judgment mechanism is the similarity of the feature alignment space of the two modalities.

[0114] To avoid the negative impact of unstable text information quality on the discrimination results, this invention inputs multi-level visual features, text features, and fused features into the classifier. Before passing through the visual classifier, text classifier, and fused classifier, a text information determination mechanism is introduced, including:

[0115] For the middle and high layers, the similarity between the aligned visual features and aligned text features of the corresponding layers is calculated respectively. ;

[0116] If the similarity of the current level Greater than or equal to the preset threshold Then, the visual features, text features, and fusion features of the current level are processed by the visual classifier, text classifier, and fusion classifier, respectively, to obtain the visual prediction category probability, text prediction category probability, and fusion prediction category probability of the current level; then, the collaborative decision-making mechanism is used to obtain the predicted classification labels of each level.

[0117] If the similarity of the current level Less than the preset threshold Then, the visual features of the current level are processed by a visual classifier to obtain the visual prediction category probability of the current level; based on the visual prediction category probability of the current level, the predicted classification label of the current level is obtained.

[0118] This invention does not simply introduce text modality for fusion, but rather starts from the hierarchical structure of clothing attributes, designing a multimodal attribute recognition mechanism that primarily uses image modality and secondarily uses text modality. This mechanism embodies a master-slave relationship between image and text at the feature level: in the modeling of low-level attributes (such as color and texture), image features are mainly relied upon for discrimination, and text modality is not forcibly intervened when corresponding information is not contained; in the recognition stage of mid-to-high-level attributes (such as material and style), text features are introduced to participate in cross-modal alignment and discrimination, thereby ensuring the stability of visual discrimination while fully utilizing the supplementary role of text in high-level semantics. Through this hierarchical intervention strategy, this invention effectively avoids the problem of "global text interference with vision" in existing multimodal methods, improving the overall accuracy and semantic consistency of complex clothing attribute recognition.

[0119] This invention performs hierarchical semantic modeling of multimodal information based on contrastive learning, including intramodal contrastive loss, cross-modal contrastive loss, and hierarchical consistency loss.

[0120] In the intramodal feature learning process, the original features and enhanced features obtained from the same sample under different enhancement conditions constitute positive sample pairs, while features from different samples within the same batch constitute negative sample pairs. Intramodal contrastive losses for each level are constructed based on corresponding level visual features and enhanced visual features, corresponding level text features and enhanced text features, including:

[0121] For all levels, the intra-modal contrast loss for each level of the visual modality is constructed based on the visual features and enhanced visual features of the corresponding level, and the formula is as follows:

[0122] ;

[0123] For the middle and high levels, the intra-modal contrast loss for each level of the text modality is constructed based on the text features and enhanced text features of the corresponding level, and the formula is as follows:

[0124] ;

[0125] in, Let be the intra-modal contrast loss at the k-th level of the visual modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be sample indices. Let be the visual feature of the k-th level of the i-th sample. and These are the enhanced visual features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. For temperature coefficient, Let be the intra-modal contrast loss at the k-th level of the text modality. Let be the text feature of the k-th level of the i-th sample. and These are the enhanced text features at the k-th level for the i-th and j-th samples, respectively.

[0126] Intramodal contrast loss of visual modalities effectively improves the robustness of the model to visual disturbances such as illumination changes, scale changes, and local occlusion by bringing the feature representations of the same image closer together under different augmented views and widening the feature distance between different images.

[0127] The intra-modal contrastive loss for textual modalities adopts a loss form isomorphic to that of visual modalities, but constraints are only imposed on mid- and high-level semantic features. This is because text at lower levels is usually dominated by lexical or local descriptions, which are noisy and have weak correlation with clothing attributes. Forcibly imposing low-level contrastive constraints may introduce instability. Therefore, this invention selectively introduces intra-modal contrastive learning in mid- and high-level semantic layers to balance semantic expressiveness and training stability.

[0128] In the cross-modal feature learning process, this invention constructs alignment constraints between images and text for mid-level and high-level semantic features respectively, to achieve cross-modal consistent modeling of clothing attributes such as material, structure, and overall style. Cross-modal contrast loss at each level is constructed based on multi-level aligned visual features and multi-level aligned text features, and the calculation formula is as follows:

[0129] ;

[0130] in, For the cross-modal contrast loss of the k-th level, Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be sample indices. Let be the alignment visual feature of the k-th level for the i-th sample. and These are the aligned text features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. This is the temperature coefficient.

[0131] Image features and text features corresponding to the same product constitute positive sample pairs, while image-text combinations between different products constitute negative sample pairs. By applying cross-modal contrast constraints at different semantic levels, this invention can establish semantic correspondences between images and text layer by layer, thereby effectively avoiding the recognition bias problem caused by prior art that only aligns global semantics while ignoring local structure and intermediate attribute information.

[0132] To ensure that features at different semantic levels are discriminative not only within the same level but also maintain a reasonable semantic progression between levels, this invention introduces Hierarchical Consistency Loss on top of the multimodal information hierarchical semantic modeling framework. This loss not only constrains cross-modal alignment between images and text at the same level but also imposes consistency constraints on feature representations at different levels within the same modality, thereby avoiding semantic fragmentation or hierarchical conflicts that may occur during the independent optimization of features at different levels.

[0133] Specifically, in the visual modality, features extracted from clothing images at different levels correspond to semantic information of different granularities: low-level features mainly represent color distribution and texture details, mid-level features reflect clothing structure and material properties, and high-level features characterize the overall style and appearance semantics. Semantically, these levels should exhibit a progressive relationship from concrete to abstract and from local to global. To this end, this invention introduces a hierarchical consistency constraint, ensuring that the feature representations of the same samples at adjacent levels maintain directional consistency, while still allowing higher-level features to possess stronger abstraction capabilities.

[0134] The hierarchical consistency loss for each level is constructed based on the alignment visual features and alignment text features of adjacent levels. The calculation formula is as follows:

[0135] ;

[0136] ;

[0137] in, Let the hierarchical consistency loss be the k-th level of the visual modality. Let be the hierarchical consistency loss for the k-th level of the text modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i be the sample index. and These are the alignment visual features of the k-th and (k+1)-th levels of the i-th sample, respectively. and These are the aligned text features of the k-th and (k+1)-th levels of the i-th sample, respectively. The cosine similarity function is used. This is the temperature coefficient.

[0138] Hierarchical consistency loss maximizes the similarity between adjacent hierarchical features of the same sample, enabling the model to inherit and integrate key information contained in lower-level features when learning higher-level semantic representations, thereby avoiding a disconnect between high-level features and lower-level perceptual information. Preferably, in the text modality, since low-level text features often contain a lot of noise or lexical information with weak correlation to specific attributes, this invention only applies hierarchical consistency constraints between mid-level and high-level semantic features. By introducing hierarchical consistency loss in the text modality, semantic coherence can be effectively ensured across different abstraction levels for high-level semantic descriptions such as style and texture, while suppressing the adverse effects of text noise on model training.

[0139] The total loss function is constructed based on intra-modal contrast loss, cross-modal contrast loss, and hierarchical consistency loss at each level, as well as classification loss. This includes: weighting and summing the losses belonging to the same level according to the loss weights to achieve joint constraints of intra-modal stability and cross-modal consistency within the same level; and then weighting and summing the losses at each level according to the hierarchical weights to establish a clear semantic progression relationship between different levels.

[0140] By introducing hierarchical consistency loss and constructing the aforementioned unified optimization objective, this invention not only achieves fine alignment of images and text in multi-layer semantic spaces, but also effectively ensures the semantic continuity and structural rationality between features at different levels, providing a stable and interpretable feature foundation for subsequent hierarchical attribute classification and self-training.

[0141] Based on the total loss function, a multimodal attribute recognition model is trained using labeled clothing samples. After obtaining the initial multimodal attribute recognition model, this invention further introduces a collaborative decision-making mechanism.

[0142] With limited labeled data, self-training is widely used in clothing attribute recognition tasks. Existing self-training methods typically use a single classifier to predict unlabeled samples and select high-confidence samples as pseudo-labels for subsequent training based on a fixed confidence threshold. To improve prediction stability, some studies have attempted to introduce multi-classifiers or voting mechanisms, but most employ static averaging or simple majority voting.

[0143] Existing general self-training and voting strategies still have significant shortcomings in clothing attribute recognition tasks. On the one hand, a single confidence threshold is difficult to adapt to the differences in prediction reliability at different attribute levels; on the other hand, most existing voting mechanisms ignore the dynamic changes in prediction consistency among different classifiers. When a classifier makes an isolated misclassification, its prediction result may still have an unreasonable impact on the final pseudo-label, leading to the introduction of noise that accumulates continuously during the self-training process.

[0144] To address the aforementioned issues, this invention proposes a collaborative decision-making mechanism for clothing attribute recognition tasks. In this mechanism, for the same sample and the same attribute level, multiple classifiers output probability distributions, with the highest class probability used as the prediction confidence. Based on this, the voting weights of each classifier are dynamically adjusted according to their prediction confidence, ensuring that high-confidence predictions have a greater weight in the fusion result. Simultaneously, a modality consistency-based weight suppression mechanism is introduced. When the prediction result of one classifier differs significantly from other classifiers, its weight is automatically reduced to prevent isolated misjudgments from adversely affecting the quality of pseudo-labels. Through this collaborative decision-making mechanism, this invention enables fine-grained control over the reliability of pseudo-labels during self-training, allowing the pseudo-label generation process to simultaneously consider prediction confidence and multimodal consistency, effectively suppressing the risk of noise propagation, thereby significantly improving the stability of the self-training stage and the final performance of the model.

[0145] Reference Figure 2 As shown, the self-training strategy of this invention makes full use of a large number of unlabeled clothing samples to continuously improve the model's ability to recognize complex clothing attributes under weak supervision. This self-training process is based on an initial multimodal attribute recognition model, and gradually expands the effective training sample size through iterative methods of pseudo-label generation, filtering, and retraining, while simultaneously suppressing the accumulation of pseudo-label noise through a collaborative decision-making mechanism.

[0146] During the self-training phase, unlabeled clothing samples are first input into the initial multimodal attribute recognition model. For each unlabeled clothing sample, the model performs forward inference using a visual classifier, a text classifier, and a fusion classifier, respectively. Based on multi-level visual features, text features, and fusion features, the model obtains the visual prediction category probability at each level. Text prediction category probability and fusion prediction of category probability .in, This indicates attribute prediction results obtained solely based on the visual modality. This indicates the prediction results obtained solely based on the text modality. This represents the prediction result after fusing image and text features.

[0147] Specifically, the hierarchical index is These correspond to fine-grained attributes (such as color and texture), medium-grained attributes (such as material and structure), and high-semantic attributes (such as style and wearing context), respectively. For each level, the visual classifier... The input is the visual features of that level. Used for attribute discrimination based solely on image modalities; text classifier The input is the text features of this level. Used for attribute discrimination based solely on text modality; fusion modality classifier The input is the fusion feature of this level, represented as It is used to perform attribute discrimination by integrating multimodal information.

[0148] The three classifiers each output the predicted class probability for the attribute category at this level, denoted as:

[0149] ;

[0150] in, These are all normalized probability vectors, and their dimension is equal to the number of attribute categories at that level.

[0151] To avoid the problem of insufficient adaptability of fixed-weight fusion across different samples and levels, this invention does not directly use the prediction result of a single classifier as a pseudo-label for the same unlabeled sample and the same attribute. Instead, it introduces a collaborative decision-making mechanism to comprehensively evaluate the prediction results of multiple classifiers. At each level, three parallel attribute discrimination units are constructed: a visual classifier, a text classifier, and a fusion classifier. Through the collaborative decision-making mechanism, the outputs of these three classifiers are adaptively weighted and fused to obtain the final predicted classification label for that level.

[0152] Specifically, by utilizing a collaborative decision-making mechanism, predicted classification labels at each level are obtained, including:

[0153] The predicted category probabilities for each level (visual, text, and fused) are weighted and summed to obtain the predicted classification label for each level. The formula is as follows:

[0154] ;

[0155] in, Let k be the predicted classification label for the k-th level. , and These represent the visual prediction category probability, text prediction category probability, and fusion prediction category probability at the k-th level, respectively. , and These are the visual classification weights, text classification weights, and fusion classification weights for the k-th level, respectively.

[0156] For different clothing samples, this invention provides a method for adaptively calculating visual classification weights, text classification weights, and fused classification weights, including:

[0157] For each level, the maximum class probability among the visual prediction class probability, text prediction class probability, and fused prediction class probability of that level is extracted as the prediction confidence of each modality at that level, expressed by the formula: ,in , and These are the prediction confidence scores for the k-th level visual modality, text modality, and fusion modality, respectively; the prediction confidence scores are used to reflect the internal determinism of each classifier for its current prediction results.

[0158] The initial weights of each modality at this level are calculated based on the prediction confidence scores of each modality, using the following formula:

[0159] ;

[0160] in, For the k-th level mode The initial weights, For modal indexing, Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. Let m be the prediction confidence level of the k-th level mode. Let be the prediction confidence score for the k-th level visual modality. Let k be the prediction confidence level for the k-th level text modality. Let be the prediction confidence level of the k-th level fusion mode. This is the initial weight adjustment parameter, used to control the degree to which the difference in confidence level affects the weight allocation. When the prediction confidence of a certain classifier at a certain level is significantly higher than that of other classifiers, its initial weight will be amplified accordingly.

[0161] However, relying solely on prediction confidence for weight allocation can still lead to isolated misjudgments of "high confidence but incorrect," especially when clothing attributes are highly subjective or the text descriptions are ambiguous. Therefore, this invention further introduces a weight suppression mechanism based on classifier prediction consistency within the hierarchy. Specifically, it assesses whether the predictions of each classifier at this level deviate from the overall consensus by measuring the consistency deviation between the prediction distributions of each classifier and other classifiers. The formula for calculating the consistency deviation is:

[0162] ;

[0163] in, For the k-th level mode Classifier consistency bias and All are modal indexes. The function representing the difference between probability distributions can be either KL divergence or cosine distance.

[0164] Based on the aforementioned consistency deviation, the consistency suppression weights for each mode at this level are calculated using the following formula:

[0165] ;

[0166] in, For the k-th level mode Consistency suppression weights, This is a consistency adjustment parameter used to control the strength of the penalty for deviations from the prediction. For the k-th level mode Classifier consistency bias. When a classifier's prediction results at a certain level differ significantly from those of other classifiers, its consistency weight will decrease significantly.

[0167] Based on the initial weights and consistency suppression weights of each modality at this level, the classification weights of each modality are calculated using the following formula:

[0168] ;

[0169] in, Let m be the classification weight of the k-th level modality. and All are modal indexes. Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. and These are the k-th level modes. and modality The initial weights, and These are the k-th level modes. and modality Consistency suppression weights.

[0170] This invention is based on predicted classification labels at each level. Candidate pseudo-labels are generated. To further ensure the quality of pseudo-labels, this invention introduces multiple constraints in the pseudo-label selection stage. Specifically, only when the maximum value of the final predicted probability satisfies the confidence threshold... ,Right now Only when the prediction results satisfy the preset hierarchical logical constraints across different attribute levels (e.g., there is no obvious conflict between style attributes and pattern attributes) will the sample and its prediction results be considered high-confidence pseudo-label samples. Samples that do not meet the above conditions will be temporarily discarded and will not participate in the self-training of the current round.

[0171] The high-confidence pseudo-label samples obtained through the above screening mechanism constitute the pseudo-label dataset, which, together with the original manually labeled data, forms the extended training set. During the self-training phase, this invention uses a relatively small learning rate to fine-tune the model, allowing it to gradually absorb effective information from the unlabeled data while maintaining its original discriminative ability. This self-training process can be repeated through multiple iterations, with each iteration regenerating and screening pseudo-labels based on the updated model, thereby achieving a gradual improvement in model performance.

[0172] The present invention discloses a clothing attribute recognition method based on hierarchical semantic modeling of multimodal information. First, according to the inherent multi-level semantic structure in clothing attribute recognition tasks, the multimodal features are divided into multiple levels. During the feature extraction stage, fine-grained attributes (such as color and texture), medium-grained attributes (such as material and structure), and high-level semantic attributes (such as style) are modeled hierarchically. During the training stage, a hierarchical consistency loss is constructed to constrain the semantic progression relationship between adjacent levels. This allows the model to form a clear and semantically continuous attribute representation space while maintaining local discriminative ability. As a result, the stability, interpretability, and overall recognition accuracy in complex clothing attribute recognition tasks are significantly improved, thus solving the problems of flattened attribute modeling and mixed semantics at different levels in existing clothing attribute recognition technologies.

[0173] Furthermore, this invention, based on multi-level features, constructs a visual classifier, a text classifier, and a fusion classifier at each semantic level, and introduces a collaborative decision-making mechanism. This mechanism dynamically adjusts the weights of each classifier by combining prediction confidence with the consistency of each classifier, generating highly reliable collaborative decision-making prediction results. By improving the accuracy of the prediction results, this invention can significantly improve the accuracy and stability of pseudo-labels during self-training, suppressing the risk of propagating erroneous pseudo-labels. This allows the model to continuously improve performance even with limited labeled data or uneven attribute distribution, thereby achieving high-precision, continuously optimized clothing attribute recognition under weak supervision. This effectively solves the problems of traditional self-training methods, which rely on a single confidence threshold for pseudo-label selection and are susceptible to noise accumulation.

[0174] Furthermore, this invention primarily uses visual modalities, supplemented by textual modalities layer by layer, to construct a multimodal contrastive learning mechanism within each modality and across modalities. By prioritizing visual features at the low-level semantic stage and introducing textual features to participate in cross-modal alignment and discrimination at the mid-to-high-level semantic stages, and combining the joint constraints of intra-modal and cross-modal contrastive learning, this invention can fully utilize the ability of text to supplement abstract attributes such as material and style while avoiding misleading textual information regarding visually dominant attributes such as color and texture. This achieves multimodal semantic alignment that better conforms to the cognitive rules of clothing attributes, improves the robustness and generalization ability of the model in real e-commerce and weakly descriptive scenarios, and effectively overcomes the problems of indiscriminate modal status, severe textual noise interference, or coarse cross-modal alignment in existing multimodal attribute recognition methods.

[0175] Based on the above-mentioned method for clothing attribute recognition based on hierarchical semantic modeling of multimodal information, this invention also provides a clothing attribute recognition system based on hierarchical semantic modeling of multimodal information, comprising:

[0176] The model building module is used to build a multimodal attribute recognition model, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor; the classifier adopts a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier;

[0177] The feature extraction module is used to input clothing samples containing original images and text descriptions into the feature extractor to obtain multi-level visual features and multi-level text features;

[0178] The feature alignment module is used to map visual features and text features of corresponding levels to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features.

[0179] The feature enhancement module is used to enhance the noise of clothing samples before inputting them into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features;

[0180] The loss function construction module is used to construct intra-modal contrast loss for each level based on visual features and enhanced visual features, text features and enhanced text features of the corresponding level; construct cross-modal contrast loss for each level based on multi-level aligned visual features and multi-level aligned text features; construct hierarchical consistency loss for each level based on aligned visual features and aligned text features of adjacent levels; and construct the total loss function based on intra-modal contrast loss, cross-modal contrast loss, hierarchical consistency loss, and classification loss for each level.

[0181] The initial training module is used to train a multimodal attribute recognition model based on the total loss function using labeled clothing samples, thus obtaining the initial multimodal attribute recognition model.

[0182] The retraining module is used to obtain pseudo-labels for unlabeled clothing samples using the initial multimodal attribute recognition model; the pseudo-labeled clothing samples and labeled clothing samples are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model.

[0183] The recognition module is used to obtain the predicted classification label of the clothing sample to be recognized by utilizing the trained multimodal attribute recognition model.

[0184] This invention constructs a multimodal attribute recognition model, aiming to provide a feasible technical path and innovative support for advanced applications such as intelligent recommendation and trend prediction in intelligent industrial environments. By integrating multi-level feature extraction, multimodal information hierarchical modeling, collaborative decision-making mechanisms, and self-training methods, this model can achieve accurate and hierarchical attribute recognition of user natural language and image input. The following detailed description of the implementation of this invention, combined with the system operation flow and typical application scenarios, further illustrates this concept.

[0185] Model input: Image (512×512, "dress"), text ("French retro floral chiffon dress, lightweight fabric, suitable for spring and summer commuting and everyday wear").

[0186] Model output: {"Lower layer": "White background", "Floral print", "Middle layer": ["Chiffon"], "High layer": ["French retro"]}.

[0187] Taking clothing product attribute recognition as an example, when the input clothing product image "dress" and its corresponding text description "This French retro floral chiffon dress, lightweight fabric, suitable for spring and summer commuting and daily wear" are input, the system of the present invention needs to automatically identify the attribute information of the product at different semantic levels, including the low-level color and pattern attributes, the mid-level material and fit attributes, and the high-level style and wearing scenario attributes.

[0188] First, the system feeds the input clothing image into a visual encoder for feature extraction. The visual encoder employs a staged structure: the front-end network models the pixel distribution and local texture of the image, the middle layer models the outline structure and fabric shape of the clothing, and the back-end layer models the overall shape and style semantics. Through this encoder, the system obtains three sets of raw visual features, corresponding to the low-level visual features. Mid-level visual features and high-rise visual features .

[0189] To enhance the model's robustness to visual perturbations, the system applies enhancement operations such as random cropping, color perturbation, and slight blurring to the original image to generate an enhanced image. The system then extracts low-level enhanced visual features using the same visual feature extractor. Middle layer enhances visual features and enhanced visual features at higher levels Thus, at each visual level, the system obtains two sets of feature representations for the same product under different visual conditions.

[0190] Simultaneously, the system inputs the product text description into a text encoder for semantic feature extraction. Considering the limited information in low-level attributes such as color and local texture, this embodiment does not introduce text features at the low level, but only extracts text semantic representations at the mid- and high semantic levels. Through the text feature extractor, the system obtains mid-level text features respectively. (Primarily representing the semantic meaning of materials such as "chiffon" and "lightweight") and high-level text features (Primarily representing style and scene semantics such as "French retro" and "commuting"). Simultaneously, the system enhances the text through methods such as synonym replacement and random masking, resulting in mid-level enhanced text features. Enhanced text features with high-level .

[0191] Subsequently, the system maps visual and textual features into a unified feature space through linear projection and normalization operations, obtaining aligned visual feature representations. and aligned text feature representation .

[0192] After completing the construction of multimodal hierarchical features, the system enters the multimodal information hierarchical semantic modeling stage. In the fine-grained layer, the system performs intra-modal contrastive learning only on the visual modality, using the original visual features of the same product. With enhanced visual features Positive sample pairs are formed, and negative sample pairs are formed using features between different products. By using contrastive loss constraints, low-level attributes such as color and pattern are kept stable under different visual perturbation conditions, thereby outputting visual features that have robust representation capabilities for color and pattern.

[0193] In the mid-granularity layer, the system simultaneously performs intra-modal and cross-modal contrastive learning. On the one hand, through constraints... and Consistency enhances the model's adaptability to changes in fabric structure and pattern; on the other hand, it constrains the aligned visual features. Text features The similarity among products establishes a stable correspondence between image textures and text descriptions for material semantics such as "chiffon." This process outputs a mid-level feature representation that exhibits cross-modal consistency at the material level.

[0194] In the high semantic layer, the system focuses on performing cross-modal contrastive learning, through constraints. and The similarity between the two styles makes the overall design of the clothing consistent with the style semantics of "French retro" and "commuting" in the text, thus outputting stable and distinguishable high-level style semantic features.

[0195] After feature learning, the system performs attribute prediction at each semantic level. For each level, the system constructs three classifiers: a visual classifier based on visual features, a text classifier based on text features, and a fusion classifier based on a combination of visual and text features. Taking the material attributes at the medium-granularity level as an example, the visual classifier is based on... Output the predicted probability of "chiffon" material; the text classifier is based on... The output corresponds to the probability, and the fusion classifier outputs a comprehensive prediction result based on the multimodal fusion features.

[0196] The system further employs a collaborative decision-making mechanism to fuse the prediction results of the three classifiers. For the same product and the same attribute, the system first uses the maximum value in the prediction probability distribution of each classifier as its prediction confidence, and dynamically adjusts the weights based on the prediction consistency among different classifiers. When the prediction result of a certain classifier deviates significantly from that of other classifiers, its weight will be automatically reduced, thereby preventing misjudgment by a single modality from interfering with the final result. Through this collaborative decision-making mechanism, the system ultimately outputs the color attribute of the dress as "white background" and the pattern attribute as "floral" at the fine-grained layer; the material attribute as "chiffon" at the medium-grained layer; and the style attribute as "French retro" at the high semantic layer.

[0197] The system described in this invention has been pre-trained based on DeepFashion2 and a self-built multimodal clothing attribute dataset. The training scale covers over 100,000 clothing images, 90,000 high-quality text descriptions, and 12,000 hierarchical attribute annotations, demonstrating excellent capabilities such as fine-grained feature representation, hierarchical semantic modeling of attributes, and robust cross-modal alignment. In deployment tests, the model achieved an average inference latency of 12ms per sample and a real-time processing capability of 85 FPS on an NVIDIA A100 80GB GPU server. Furthermore, the system can be flexibly deployed on local servers and edge devices. The modular architecture supports independent start / stop of the visual classifier, text classifier, fusion modality classifier, multimodal information hierarchical modeling module, and collaborative decision-making pseudo-label update module, enabling scalable, maintainable, and continuously optimizable operation, meeting the comprehensive requirements of real-time performance and autonomous decision-making capabilities in e-commerce scenarios.

[0198] This invention addresses the challenges of complex attribute hierarchy, significant differences in the role of multimodal information, and high costs associated with acquiring labeled data in clothing attribute recognition tasks. It proposes a clothing attribute recognition method based on hierarchical semantic modeling of multimodal information. Starting from the semantic structure of clothing attributes, this method rationally organizes the modeling of attributes at different levels, clarifies the master-slave relationship between image and text modalities at each attribute level, and improves the stability and reliability of model learning under weak supervision through a collaborative decision-making mechanism. This effectively overcomes the shortcomings of existing technologies in semantic consistency, multimodal fusion efficiency, and self-training noise control, providing a more practical technical solution for complex clothing attribute recognition.

[0199] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for clothing attribute recognition based on hierarchical semantic modeling of multimodal information, characterized in that, include: A multimodal attribute recognition model is constructed, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor. The classifier employs a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier; The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features; The visual and text features of the corresponding levels are mapped to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features. After noise enhancement, the clothing samples are input into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features; Intra-modal contrast loss is constructed based on visual features and enhanced visual features, text features and enhanced text features at the corresponding levels, including: For all levels, the intra-modal contrast loss for each level of the visual modality is constructed based on the visual features and enhanced visual features of the corresponding level, and the formula is as follows: ； For the middle and high levels, the intra-modal contrast loss for each level of the text modality is constructed based on the text features and enhanced text features of the corresponding level, and the formula is as follows: ； in, Let be the intramodal contrast loss at the k-th level of the visual modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be the sample indices. Let be the visual feature of the k-th level of the i-th sample. and These are the enhanced visual features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. For temperature coefficient, Let be the intra-modal contrast loss at the k-th level of the text modality. Let be the text feature of the k-th level of the i-th sample. and These are the enhanced text features at the k-th level for the i-th and j-th samples, respectively; Cross-modal contrast loss for each level is constructed based on multi-level alignment visual features and multi-level alignment text features; hierarchical consistency loss for each level is constructed based on the alignment visual features and alignment text features of adjacent levels. The total loss function is constructed based on the intra-modal contrast loss, cross-modal contrast loss, and hierarchical consistency loss at each level, as well as the classification loss. Based on the total loss function, a multimodal attribute recognition model is trained using labeled clothing samples to obtain an initial multimodal attribute recognition model. Using the initial multimodal attribute recognition model, pseudo-labels for unlabeled clothing samples are obtained; the clothing samples with pseudo-labels and the clothing samples with labels are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model. Using the trained multimodal attribute recognition model, the predicted classification labels of the clothing samples to be identified are obtained, including: The clothing sample containing the original image and text description is input into the feature extractor to obtain multi-level visual features and multi-level text features; By fusing the visual and textual features of the corresponding levels, multi-level fused features are obtained. Multi-level visual features, text features, and fused features are input into the classifier, which then passes through the visual classifier, text classifier, and fused classifier respectively to obtain the visual prediction class probability, text prediction class probability, and fused prediction class probability at each level. A collaborative decision-making mechanism is then used to obtain the predicted classification labels at each level. Before inputting the multi-level visual features, text features, and fused features into the classifier, a text information determination mechanism is introduced, including: For the middle and high layers, the similarity between the aligned visual features and aligned text features of the corresponding layers is calculated respectively; If the similarity of the current level is greater than or equal to the preset threshold, the visual features, text features and fusion features of the current level are processed by the visual classifier, text classifier and fusion classifier respectively to obtain the visual prediction category probability, text prediction category probability and fusion prediction category probability of the current level; then the collaborative decision-making mechanism is used to obtain the predicted classification label of each level. If the similarity of the current level is less than a preset threshold, the visual features of the current level are processed by a visual classifier to obtain the visual prediction category probability of the current level; the predicted classification label of the current level is obtained based on the visual prediction category probability of the current level.

2. The clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to claim 1, characterized in that, The cross-modal contrast loss for each level is constructed based on multi-level aligned visual features and multi-level aligned text features. The calculation formula is as follows: ； in, For the cross-modal contrast loss of the k-th level, Let k be the total number of training samples in a batch, k be the hierarchical index, and i and j be the sample indices. Let be the alignment visual feature of the k-th level for the i-th sample. and These are the aligned text features at the k-th level for the i-th and j-th samples, respectively. The cosine similarity function is used. This is the temperature coefficient.

3. The clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to claim 1, characterized in that, The hierarchical consistency loss for each level is constructed based on the alignment visual features and alignment text features of adjacent levels. The calculation formula is as follows: ； ； in, Let the hierarchical consistency loss be the k-th level of the visual modality. Let be the hierarchical consistency loss for the k-th level of the text modality. Let k be the total number of training samples in a batch, k be the hierarchical index, and i be the sample index. and These are the alignment visual features of the k-th and (k+1)-th levels of the i-th sample, respectively. and These are the aligned text features of the k-th and (k+1)-th levels of the i-th sample, respectively. The cosine similarity function is used. This is the temperature coefficient.

4. The clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to claim 1, characterized in that, Using a collaborative decision-making mechanism, predicted classification labels at each level are obtained, including: The predicted category probabilities for each level (visual, text, and fused) are weighted and summed to obtain the predicted classification label for each level. The formula is as follows: ； in, Let k be the predicted classification label for the k-th level. , and These represent the visual prediction category probability, text prediction category probability, and fusion prediction category probability at the k-th level, respectively. , and These are the visual classification weights, text classification weights, and fusion classification weights for the k-th level, respectively.

5. The clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to claim 4, characterized in that, The calculation methods for the visual classification weight, text classification weight, and fusion classification weight include: For each level, the maximum class probability among the visual prediction class probability, text prediction class probability, and fusion prediction class probability of that level is extracted and used as the prediction confidence of each modality at that level. The initial weights of each modality at this level are calculated based on the prediction confidence of each modality at this level. Calculate the consistency deviation between each classifier in this layer and other classifiers, and calculate the consistency inhibition weight of each modality in this layer based on the consistency deviation; Based on the initial weights and consistency suppression weights of each modality at this level, the classification weights of each modality are calculated using the following formula: ； in, Let m be the classification weight of the k-th level modality. and All are modal indexes. Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. and These are the k-th level modes. and modality The initial weights, and These are the k-th level modes. and modality Consistency suppression weights.

6. The clothing attribute recognition method based on multimodal information hierarchical semantic modeling according to claim 5, characterized in that, The initial weights of each modality at this level are calculated based on the prediction confidence scores of each modality, using the following formula: ； in, For the k-th level mode The initial weights, For modal indexing, Representing visual modality, Represents text modality, This indicates a fusion modality, where k is the hierarchical index. Let m be the prediction confidence level of the k-th level mode. Let be the prediction confidence score for the k-th level visual modality. Let k be the prediction confidence level for the k-th level text modality. Let be the prediction confidence level of the k-th level fusion mode. These are the initial weight adjustment parameters; Based on the aforementioned consistency deviation, the consistency suppression weights for each mode at this level are calculated using the following formula: ； in, For the k-th level mode Consistency suppression weights, For consistency adjustment parameters, For the k-th level mode Consistency bias of classifiers.

7. A clothing attribute recognition system based on multimodal information hierarchical semantic modeling, used to implement the clothing attribute recognition method based on multimodal information hierarchical semantic modeling as described in any one of claims 1 to 6, characterized in that, include: The model building module is used to build a multimodal attribute recognition model, including a feature extractor and a classifier; the feature extractor includes a visual feature extractor and a text feature extractor. The classifier employs a collaborative decision-making mechanism, including a visual classifier, a text classifier, and a fusion classifier; The feature extraction module is used to input clothing samples containing original images and text descriptions into the feature extractor to obtain multi-level visual features and multi-level text features; The feature alignment module is used to map visual features and text features of corresponding levels to a unified feature space to obtain multi-level aligned visual features and multi-level aligned text features. The feature enhancement module is used to enhance the noise of clothing samples before inputting them into the feature extractor to obtain multi-level enhanced visual features and multi-level enhanced text features; The loss function construction module is used to construct the intra-modal contrast loss for each level based on the visual features and enhanced visual features of the corresponding level, and the text features and enhanced text features of the corresponding level. Cross-modal contrast loss is constructed for each level based on multi-level aligned visual features and multi-level aligned text features; hierarchical consistency loss is constructed for each level based on the aligned visual features and aligned text features of adjacent levels; and a total loss function is constructed based on the intra-modal contrast loss, cross-modal contrast loss, hierarchical consistency loss, and classification loss of each level. The initial training module is used to train a multimodal attribute recognition model based on the total loss function using labeled clothing samples, thus obtaining the initial multimodal attribute recognition model. The retraining module is used to obtain pseudo-labels for unlabeled clothing samples using the initial multimodal attribute recognition model; the pseudo-labeled clothing samples and labeled clothing samples are merged into a self-training set, and the initial multimodal attribute recognition model is trained again to obtain a fully trained multimodal attribute recognition model. The recognition module is used to obtain the predicted classification label of the clothing sample to be recognized by utilizing the trained multimodal attribute recognition model.

Images