Diversified interpretable compatibility modeling method and system based on multi-modal dissociation

Abstract

The invention provides a diversified interpretable compatibility modeling method and system based on multi-modal dissociation, and relates to the technical field of clothing matching schemes, and the method comprises the steps: obtaining multi-modal information of two to-be-matched single items; performing feature extraction fusion on the multi-modal information to obtain multi-modal features of each single item; performing dissociation on the multi-modal features through contrast dissociation attribute learning based on deep mutual information; on the basis of the dissociated multi-modal features, calculating diversified interpretable compatibility scores of the two single items by using two matching modes of alignment attributes and candidate attributes; according to the method, multi-modal features are dissociated through contrast dissociation attribute learning based on deep mutual information, compatibility between fashion single items is measured by using two matching modes of alignment attributes and candidate attributes, and the accuracy of single item matching is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of clothing matching solutions, and in particular to a diversified interpretable compatibility modeling method and system based on multimodal dissociation. Background Art

[0002] In recent years, the booming fashion industry has shown that people have a growing demand for fashionable clothing. Appropriate clothing matching can reflect a person's good mental outlook. However, due to differences in people's aesthetics, not everyone is good at daily clothing matching, which provides an application background for clothing compatibility modeling solutions.

[0003] The purpose of fashion item compatibility modeling is to evaluate the matching degree between items. Previous work has achieved certain results in the recommendation of complementary items. However, in terms of comprehensive evaluation, there are still the following problems, which lead to the poor effect of existing diversified interpretable compatibility modeling: (1) Some existing studies attempt to solve the problem of clothing compatibility modeling by integrating the multimodal information of fashion items. However, these methods usually model the learned representations of fashion items in a holistic manner and do not effectively divide the fine-grained attributes of the items, resulting in a lack of interpretability in the recommendation task.

[0004] (2) A single attribute matching model cannot fully describe the compatibility between fashion items, resulting in the recommended complementary items not being the optimal solution. Summary of the invention

[0005] In order to solve the above problems, the present disclosure proposes a diversified interpretable compatibility modeling method and system based on multimodal dissociation, which dissociates multimodal features through comparative dissociation attribute learning based on deep mutual information, and uses two matching modes of alignment attributes and candidate attributes to measure the compatibility between fashion items, thereby improving the accuracy of item matching.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions: Diverse interpretable compatibility modeling methods based on multimodal dissociation, including: Obtain multimodal information of two items to be matched; Perform feature extraction and fusion on multimodal information to obtain multimodal features of each item; Dissociate multimodal features through contrastive dissociative attribute learning based on deep mutual information; Based on the dissociated multimodal features, the diversified interpretable compatibility scores of two items are calculated using two matching modes of alignment attributes and candidate attributes.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: A diverse interpretable compatibility modeling system based on multimodal dissociation, including: The information acquisition module is configured to: acquire multimodal information of two items to be matched; The feature extraction module is configured to: perform feature extraction and fusion on the multimodal information to obtain the multimodal features of each single product; A feature dissociation module is configured to: dissociate the multimodal features through contrastive dissociation attribute learning based on deep mutual information; The attribute matching module is configured to calculate the diversified interpretable compatibility scores of two items based on the dissociated multimodal features and using two matching modes of alignment attributes and candidate attributes.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the diversified interpretable compatibility modeling method based on multimodal dissociation is implemented.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the diversified interpretable compatibility modeling method based on multimodal dissociation is implemented.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device implements the diversified interpretable compatibility modeling method based on multimodal dissociation.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) In order to avoid the interference of redundant information of a single product, the present invention designs a contrast dissociation attribute learning method, which fits the mutual information value between feature attributes through a deep neural network, and optimizes the mutual information value between attributes by contrast loss to minimize it, thereby increasing the independence between attributes.

[0012] (2) This paper proposes a diversified and interpretable attribute matching method, which consists of two matching modes: alignment attributes and candidate attributes. The two methods complement each other and model the matching process between product attributes from a more comprehensive perspective, thereby increasing the interpretability of the complementary product recommendation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0014] Figure 1 This is the architecture diagram of Example 1.

[0015] Figure 2 This is the visual feature extraction diagram of Example 1.

[0016] Figure 3 This is the text feature extraction diagram of Example 1.

[0017] Figure 4 1 is an example diagram of the matching mode of the alignment attribute of Example 1, wherein (a) is the top 1 to be matched, (b) and (c) are the bottoms 1-1 and 1-2 to be matched respectively.

[0018] Figure 5 2 is an example diagram of matching modes of candidate attributes of Example 1, wherein (a) and (c) are two matching modes of the same top, and (b) is a schematic diagram of the Mueller-Lyer illusion. DETAILED DESCRIPTION

[0019] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] Example 1 In one embodiment of the present disclosure, a diversified interpretable compatibility modeling method based on multimodal dissociation is provided, including: Obtain multimodal information of two items to be matched; Perform feature extraction and fusion on multimodal information to obtain multimodal features of each item; Dissociate multimodal features through contrastive dissociative attribute learning based on deep mutual information; Based on the dissociated multimodal features, the diversified interpretable compatibility scores of two items are calculated using two matching modes of alignment attributes and candidate attributes.

[0023] As an embodiment, the diversified interpretable compatibility modeling method based on multimodal dissociation disclosed in the present invention is as follows: Figure 1 As shown in the figure, the contrastive dissociation attribute learning based on deep mutual information is used to extract the feature values ​​of each attribute of the single product, and the two matching modes of alignment attribute and candidate attribute are used to measure the compatibility between fashion items. The specific implementation process is described in detail below: 1. Problem Definition Suppose there is a set of tops and bottoms, respectively and ,in, and Represents the total number of tops and bottoms respectively. For each top in the collection and bottoms , both contain information of two modalities (i.e., image and text description); respectively and Represents the visual features and text features of tops (bottoms), where: and Represent their corresponding feature dimensions respectively; The compatibility modeling method of complementary fashion items aims to study the given top item In the case of It is more compatible with it.

[0024] This embodiment proposes a diversified interpretable compatibility modeling method based on multimodal dissociation , this method can dissociate the multimodal information of a single product, thereby improving the interpretability of the recommendation task. In addition, by simultaneously considering the alignment attribute compatibility score and candidate attribute compatibility scores , and obtain the final diversified interpretable compatibility score, which improves the performance of diversified compatibility modeling.

[0025] 2. Multimodal Feature Extraction and Fusion 1. Visual feature extraction At present, pre-trained models have been widely used and have made remarkable achievements in the field of computer vision. Therefore, this embodiment uses the pre-trained network ViT to (Bottoms ) to extract visual features from the original image ,like Figure 2 As shown in Figure 2, the specific operations of the ViT model are: First, put the top (Bottoms ) is divided into multiple image patches, which are then converted into low-dimensional embeddings through linear projection. The image patch embeddings are then combined with the corresponding position embeddings as a sequence and fed into a 12-layer transformer to capture contextual information. Finally, the final visual features Generated by a Multilayer Perceptron (MLP).

[0026] 2. Text feature extraction For the top (Bottoms ) ( ), using the BERT model with strong generalization ability; this embodiment selects the BERT basic version consisting of 12 transformers, such as Figure 3 As shown, with top Take the text feature extraction of BERT model as an example. The text description is decomposed into individual words, which are then converted into word embedding vectors, and finally encoded through a set of transformers to obtain text features. .

[0027] 3. Feature Fusion Different modalities express different attribute information of fashion items. In order to fully explore the potential of multimodality in compatibility modeling, visual features are and text features ( ) to obtain multimodal features ,in, is the number of dimensions of the multimodal feature, and each dimension is taken as a feature attribute.

[0028] 3. Contrastive Dissociation Attribute Learning Based on Deep Mutual Information The compatibility between complementary items is largely affected by the attributes of the items. Existing multimodal compatibility modeling methods mainly directly calculate compatibility based on the overall features in the potential compatibility space (such as the style space); however, these methods ignore the impact of repeated and redundant information in multimodal features on compatibility modeling, which reduces the interpretability of the recommendation process; in order to enhance the independence of multimodal features and improve the interpretability of compatibility modeling, this embodiment designs a contrast dissociation attribute learning method based on deep mutual information, which aims to dissociate the multimodal features of items, that is, to dissociate the multimodal features, which is divided into two steps: (1) Fitting mutual information values: Different dimensions of multimodal features represent different feature attributes. Since the mutual information value between high-dimensional continuous representations is difficult to calculate accurately, a neural network is used to approximate the mutual information value between the feature attributes of a single product.

[0029] (2) Optimizing the mutual information value: The smaller the mutual information value between feature attributes, the smaller the degree of attribute dependence, the higher the independence, and the better the dissociation effect. Therefore, a contrast loss is designed, using different feature attributes of the same product as positive pairs and randomly selecting a feature attribute of other products as negative pairs to minimize the mutual information value between the feature attributes of the products.

[0030] Specifically, mutual information is used to measure the degree of dependence between variables. Given two variables and The stronger the independence and the smaller the dependence between them, the lower the mutual information, and vice versa. It can be expressed as follows:

[0031] in, Representation variables and variables The joint distribution of and They represent the marginal distribution of the variables respectively.

[0032] This embodiment uses mutual information to quantify the degree of dependence between feature variables in multimodal features, and dissociates multimodal features based on mutual information. Multimodal features For example, Different dimensions of multimodal features, that is, different feature attributes in multimodal features, are finally dissociated into However, it is difficult to accurately calculate multimodal features The mutual information value between medium and high dimensional continuous variables is approximated by calculating the mathematical expectation of the neural network output value.

[0033] According to the above formula, the top No. Dimensional representation (i.e. feature attributes) and Dimensional representation (i.e. feature attributes) It can be expressed as:

[0034] In order to further improve the independence between the characteristic attributes of fashion items and reduce the degree of dependence between them, the following contrastive loss is constructed to train the neural network for dissociation by minimizing the degree of dependence (i.e. maximizing independence): =:

[0035] in, Represents the negative sample feature attributes randomly extracted from other items. A similar method can be used to obtain the lower garment The dissociation characteristics of .

[0036] 4. Attribute matching based on alignment attributes and candidate attributes Based on the dissociated multimodal features of the products, diversified and interpretable attribute matching modeling is carried out, and the compatibility scores between complementary products are evaluated according to the matching scores, which is divided into three steps: (1) Matching mode of alignment attributes and modeling the compatibility of alignment attributes: The representation of the same dimension of the top and bottom (the same feature attribute) is recorded as the alignment attribute, and the compatibility score between the alignment attributes is calculated based on the weights between the attributes. This is also the most common matching method that people think of in daily matching, such as matching between colors and stripes.

[0037] (2) Matching pattern of candidate attributes and candidate attribute compatibility modeling: In order to improve the single problem of complementary item attribute matching pattern, candidate attribute compatibility modeling is additionally designed to find the attribute pairs with the highest weights between tops and bottoms except for the alignment attributes as candidate attributes, and to supplement the attribute matching method with candidate attributes.

[0038] (3) Calculate the compatibility score: Take the weighted sum of the scores of the two attribute matching methods and verify the optimal value of the weight in subsequent experiments. This score is the final compatibility score between complementary products.

[0039] Specifically, in the dissociated feature representation (i.e., the dissociated multimodal features) and Based on this, we further elaborate the compatibility rules and design the following: 1) Matching mode of alignment attributes. 2) Matching mode of candidate attributes. We learn the diversified matching rules between product feature attributes from two perspectives to improve the interpretability of complementary recommendation tasks.

[0040] 1. Matching mode of alignment attributes In actual clothing matching, complementary items are usually selected based on alignment properties, e.g. Figure 4 The bottoms shown in (b) and (c) are both plaid pleated skirts, however, Figure 4 The bow tie of the top in (a) matches the color of the bottom in (b), and both are described as "blue plaid". Therefore, considering the alignment attribute (i.e., color) between complementary items, Figure 4 The bottoms shown in (b) fit the top shown in (a) better than the bottoms shown in (c).

[0041] Therefore, design a matching pattern for the alignment attribute, given the top And target bottoms , first calculate the alignment weight , expressed as:

[0042] in, Tops And target bottoms No. characteristic attributes, , and are learnable network parameters, is the sigmoid activation function.

[0043] Based on alignment weight , calculate alignment attribute compatibility score :

[0044] 2. Matching mode of candidate attributes In fact, it is not enough to rely solely on the alignment attributes between complementary products to evaluate compatibility, and candidate attributes must be combined to improve the performance of compatibility modeling. For example, Figure 5 (a) and (c) in the figure are two matching ways of the same top. The "black and white striped vest" (top) and the two "black long skirts" (bottoms) are very well matched in terms of alignment attributes (i.e., color). However, when further considering the features of the neckline and hem, Figure 5 The more ideal match between “wide-leg long skirt” and “V-neck vest” shown in (a) of the figure may be mainly attributed to Figure 5 The Müller-Lyer illusion shown in (b) makes Figure 5 The matching of (a) in the figure visually lengthens the height of the person. Therefore, we design a matching mode for candidate attributes and calculate the candidate weights. as follows:

[0045] Unlike alignment attribute compatibility modeling, the candidate weights are selected The largest characteristic attribute is used as a candidate attribute; therefore, the top and bottoms Candidate attribute compatibility scores between , as shown below:

[0046] in, middle Indicates tops No. characteristic attributes, Indicates target bottoms No. feature attributes. According to the Max in the previous formula, the maximum weight is obtained , indicating that in addition to the alignment attribute, the top The characteristic attributes and the lower garment The weight among the feature attributes is the largest, so it is selected as the candidate attribute.

[0047] 5. Target loss: Compatibility scores can be explained by diversity and , the total score of the recommended task can be calculated , that is, the diversified explainable compatibility score, is calculated as follows:

[0048] in, It is a balancing parameter that controls the weight of the matching mode of the alignment attribute and the matching mode of the candidate attribute.

[0049] The clothes matched by fashion experts form a positive example set , some unobserved combinations are regarded as negative examples. In order to accurately model the compatibility between items, the following triples are constructed based on the public dataset using the Bayesian Personalized Ranking (BPR) framework. :

[0050] Among them, the triple Represents the set of positive examples Bottoms Than the bottom More in line with the given top . From this, the following loss can be constructed :

[0051] Jointly Optimize Contrastive Loss , and the following final target loss is obtained :

[0052] in, and is a non-negative balancing parameter.

[0053] 6. Experimental analysis: In order to comprehensively evaluate the experimental effect of this scheme (abbreviated as DICM), comparative experiments were conducted on two datasets (IQON3000 and Polyvore) with the existing baseline model, and the area under the curve (AUC), mean reverse ranking (MRR), hit rate (HR@K) and normalized discounted cumulative gain (NDCG@K) were used as evaluation indicators to measure the effect of clothing complementary recommendation. Table 1 shows the performance of different methods based on AUC and MRR evaluation indicators on the two datasets, and Table 2 shows the performance of the methods with AUC values ​​not less than 0.8 in Table 1 based on HR@K and MRR evaluation indicators on the two datasets, where K is the number of bottoms in the candidate sequence: Table 1 Performance of different methods based on AUC and MRR evaluation indicators

[0054] Table 2 Performance of different methods based on HR@K and NDCG@K evaluation indicators

[0055] These results show that our proposed DICM significantly outperforms existing baseline models on both datasets.

[0056] Example 2 In one embodiment of the present disclosure, a diversified interpretable compatibility modeling system based on multimodal dissociation is provided, comprising: The information acquisition module is configured to: acquire multimodal information of two items to be matched; The feature extraction module is configured to: perform feature extraction and fusion on the multimodal information to obtain the multimodal features of each single product; A feature dissociation module is configured to: dissociate the multimodal features through contrastive dissociation attribute learning based on deep mutual information; The attribute matching module is configured to calculate the diversified interpretable compatibility scores of two items based on the dissociated multimodal features and using two matching modes of alignment attributes and candidate attributes.

[0057] Example 3 In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the diversified interpretable compatibility modeling method based on multimodal dissociation.

[0058] Example 4 In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the diversified interpretable compatibility modeling method based on multimodal dissociation is implemented.

[0059] Example 5 In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the diversified interpretable compatibility modeling method based on multimodal dissociation.

[0060] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A diversified interpretable compatibility modeling method based on multimodal dissociation, characterized by: include: Obtain multimodal information of two items to be matched; Perform feature extraction and fusion on multimodal information to obtain multimodal features of each item; Dissociate multimodal features through contrastive dissociative attribute learning based on deep mutual information; Based on the dissociated multimodal features, the diversified interpretable compatibility scores of two items are calculated using two matching modes of alignment attributes and candidate attributes.

2. The diversified interpretable compatibility modeling method based on multimodal dissociation according to claim 1, characterized in that: The feature extraction and fusion of multimodal information is specifically as follows: The ViT network and BERT model are used to extract visual features and text features from the images and text descriptions of individual products respectively; The visual features and text features are integrated to obtain multimodal features.

3. The diversified interpretable compatibility modeling method based on multimodal dissociation according to claim 1, characterized in that: The multimodal features are dissociated by learning and training a neural network based on contrast dissociation attributes of deep mutual information, using the trained neural network to dissociate the input multimodal features, and outputting the dissociated multimodal features.

4. The diversified interpretable compatibility modeling method based on multimodal dissociation according to claim 3, characterized in that: The contrastive dissociation attribute learning based on deep mutual information utilizes the mutual information values ​​between feature variables in multimodal features to quantify the degree of dependence between feature variables, and minimizes the degree of dependence between feature variables by constructing contrast loss, thereby training the neural network.

5. The diversified interpretable compatibility modeling method based on multimodal dissociation according to claim 1, characterized in that: The matching mode of the alignment attribute is to obtain the alignment weight between the same characteristic attributes of two items through learning, and calculate the alignment attribute compatibility score of the two items to be matched based on the alignment weight.

6. The diversified interpretable compatibility modeling method based on multimodal dissociation according to claim 1, characterized in that: The matching mode of the candidate attributes is to obtain the candidate weights between different feature attributes of two items by learning, and calculate the candidate attribute compatibility scores of the two items to be matched based on the feature attribute with the largest candidate weight.

7. A diversified interpretable compatibility modeling system based on multimodal dissociation, characterized by: include: The information acquisition module is configured to: acquire multimodal information of two items to be matched; The feature extraction module is configured to: perform feature extraction and fusion on the multimodal information to obtain the multimodal features of each single product; A feature dissociation module is configured to: dissociate the multimodal features through contrastive dissociation attribute learning based on deep mutual information; The attribute matching module is configured to calculate the diversified interpretable compatibility scores of two items based on the dissociated multimodal features and using two matching modes of alignment attributes and candidate attributes.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the diversified interpretable compatibility modeling method based on multimodal dissociation described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the diversified interpretable compatibility modeling method based on multimodal dissociation as described in any one of claims 1-6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device implements the diversified interpretable compatibility modeling method based on multimodal dissociation as described in any one of claims 1-6.

Images