A fashion suit compatibility modeling method based on heterogeneous graph neural network

Abstract

The application provides a fashion suit compatibility modeling method based on a heterogeneous graph neural network. The model models a fashion suit as a heterogeneous fashion graph, performs graph convolution on the heterogeneous fashion graph by using an RGCN to obtain an embedding vector of a node, and predicts the compatibility of the suit. In addition, a difficult example sampling strategy is adopted, which effectively alleviates the problem of uneven number of clothing categories. In order to evaluate the model, extensive and in-depth experiments are conducted, and the effectiveness of the model is proved.

Description

Technical Field

[0001] This invention belongs to the field of fashion recommendation research, and in particular relates to a method for predicting the visual compatibility of fashion outfits using a deep learning model. Background Technology

[0002] With the development of artificial intelligence, processing fashion big data with AI has become an exciting challenge for computer scientists. In the past few years, research on fashion recommendations has received increasing attention in the computer vision, machine learning and multimedia communities.

[0003] In the early stages, research on the compatibility learning of paired garments followed the idea of ​​mapping garments to a common latent compatibility space and estimating the distance between the style vectors of the garments. For example, Chen, L. et al. [1] used a mixed-class metric learning model to map paired garments to a common metric space, with the distance between them in the metric space reflecting the compatibility matching degree between the pair of garments. Recently, some studies have proposed mapping garments to several latent visual spaces to model the compatibility of garments and jointly model the distances in these latent spaces to measure the compatibility of garments in different aspects. He, R., Packer, C. et al. [2] proposed mapping garments to different metric spaces and put forward a novel view: two garments that match each other must have some similar features and some dissimilar features. However, the above methods often make items of the same category close in the latent visual space of the learned garments. Therefore, some studies have proposed adding classification information to improve the embedding learning of garment images. On the other hand, most compatibility modeling relies on data-driven methods and ignores knowledge in the fashion domain. To address this shortcoming, Song et al. [3] explored a new approach to compatibility modeling by introducing the rich and valuable matching rules that have been accumulated in the fashion field. Summary of the Invention

[0004] Objective: To model the overall compatibility of fashion sets while considering the various relationships between different items within a set, this invention proposes a fashion set compatibility modeling method based on a heterogeneous graph neural network. First, a pre-trained ResNet18 neural network is used to extract vector representations of fashion set images. Then, a fashion graph is constructed for all fashion garments, using the feature vector representations from the previous step as node features. Finally, a heterogeneous graph neural network is used to model the interactions between nodes and learn the representations of nodes and subgraphs. The output layer of the neural network then yields the fashion garment compatibility prediction results.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A method for modeling the compatibility of fashion suits based on heterogeneous graph neural networks includes the following steps:

[0007] Step 1) Use similar substitution to replace one or more clothing items in the positive sample set to construct a negative sample training set for the fashion set.

[0008] Step 2) Based on this dataset, put the image of each outfit into a pre-trained ResNet18 model to obtain the feature vector representation of each fashion outfit.

[0009] Step 3) Constructs a fashion graph for all fashion garments, using the feature vectors from the previous step as the features of the nodes, where each node represents a type of clothing and each edge represents the interaction between nodes.

[0010] Step 4) Use the hard sample training strategy to train the model on the training set. Nodes that appear less frequently during the training phase are more likely to be activated and trained.

[0011] Step 5) Use a Heterogeneous Graph Convolutional Network (RGCN) to model the interactions between nodes and learn the representations of nodes and subgraphs. Finally, after passing through the output layer of the neural network, obtain the compatibility prediction results of fashion clothing.

[0012] Beneficial effects:

[0013] 1) This invention proposes a novel fashion suit compatibility modeling method based on heterogeneous graph neural networks. Compared with previous compatibility modeling methods, this method can model the interactions between different types of nodes with more fine granularity and has stronger robustness to datasets containing complex samples.

[0014] 2) This invention employs a difficult-example sampling training strategy, which effectively alleviates the problem of uneven distribution of clothing categories. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the specific implementation of the method of the present invention;

[0016] Figure 2 shows some negative sample data before and after changing the negative sample sampling strategy;

[0017] Figure 3 This is a schematic diagram of the node activation process of the method of the present invention;

[0018] Figure 4This is a schematic diagram of the overall structure of the method of the present invention;

[0019] Figure 5 This is a schematic diagram of the dataset used in this invention;

[0020] Figure 6 This is a schematic diagram illustrating the experimental comparison between the method of this invention and other algorithms; Detailed Implementation

[0021] This invention provides an implementation example of a fashion suit compatibility modeling method based on heterogeneous graph neural networks. To enable those skilled in the art to better understand the technical solutions in the embodiments of this invention, and to make the above-mentioned objectives, features, and advantages of this invention more apparent, the technical solutions of this invention will be further described in detail below with reference to the accompanying drawings:

[0022] The specific implementation flowchart of the method of this invention is as follows: Figure 1 As shown, the process is as follows:

[0023] Step 1) Use similar substitutions to replace one or more clothing items in the positive sample set, constructing a negative sample training set for the fashion set:

[0024] Figure 2(a) shows some negative samples before the negative sample sampling strategy was changed, and Figure 2(b) shows some negative samples after the negative sample sampling strategy was changed. Each row represents a set of clothing. The first number "0" in each row indicates that the set belongs to the negative samples, that is, an incompatible set. The string of numbers after "0" represents the ID of the clothing item. Each item has a unique ID. The last two characters of each ID are an underscore followed by a number, with the number range being [1,8], representing the clothing type of different parts.

[0025] Step 2) Based on this dataset, put the image of each outfit into a pre-trained ResNet18 model to obtain the feature vector representation of each fashion outfit.

[0026] Step 3) Constructs a Fashion Graph for all fashion garments, using the feature vectors from the previous step as the features of the nodes. Each node represents a type of clothing, and each edge represents the interaction between nodes.

[0027] like Figure 3 As shown in the figure, the four dark nodes are in an active state, and their corresponding node features are non-zero. The nodes in an inactive state have all zero features, but their structural information and edge type information are preserved. This can be expressed by the formula: {(a,b,w(a,b))|a,b∈O} k}, where a and b represent individual items, O k This represents the set of individual items contained in set k.

[0028] Step 4) Use the hard sample training strategy to train the model on the training set. Nodes that appear less frequently during the training phase are more likely to be activated and trained.

[0029] Step 5) Use a Heterogeneous Graph Convolutional Network (RGCN) to model the interactions between nodes and learn the representations of nodes and subgraphs. Finally, after passing through the output layer of the neural network, obtain the fashion apparel compatibility prediction results:

[0030] like Figure 4 As shown, in order to infer the compatibility of fashion sets from fashion graphs, we use a compatibility modeling algorithm based on heterogeneous graph convolutional networks to build an end-to-end deep learning model. This model can better model the interactions between nodes and learn the vector representation of clothing nodes, thereby predicting the compatibility of fashion sets.

[0031] Graph neural networks map a graph to an output space in two steps. First, there is a propagation step to compute the node representation of each node; second, the output model is generated. v =g(h v ,l v This is used to map from the node representation and its corresponding label to the output ov of each node. The process is end-to-end differentiable, so gradient-based optimization can be used to jointly learn all parameters.

[0032] In the propagation step, the initial node representation is set to an arbitrary value, and then each node representation is updated according to the following recursive formula until convergence, where t is the time step:

[0033]

[0034] Furthermore, there are several variations of this recursion, including updates for specific nodes and alternative representations of neighbors, which can make f * (·) can be decomposed into the sum of each edge:

[0035]

[0036] Where f(·) is a linear function or a neural network, and the parameters of f depend on the label settings. For example, in the following linear case, A and b are trainable parameters:

[0037]

[0038] The training process is implemented using the Almeida-Pineda algorithm, which works by propagating information between nodes until convergence, and then calculating the gradient based on the converged solution. The advantage of this approach is that it eliminates the need to store intermediate states for gradient calculation. The disadvantage is that the parameters must be constrained to make the propagation step a shrinking mapping. While this ensures convergence, it may limit the model's expressive power.

[0039] Heterogeneous graph neural networks are an extension of graph neural networks, which primarily operate in local neighborhoods.

[0040]

[0041] The feature vectors of each individual item image, along with the overall fashion heterogeneity graph and individual set images, are obtained through a pre-trained ResNet18 model. Next, a heterogeneous graph neural network is used to learn the vector representation of each node by mining valuable information from its neighboring nodes. According to the heterogeneous graph neural network, the update formula for a node at layer l can be expressed as:

[0042]

[0043] in Indicates the type of edge. Given that the edge type is r, W is the set of neighboring nodes of node i, and W is the parameter that the neural network needs to learn. c is the hidden state of node i in the l-th layer of the neural network. i,r This is a predefined or trained regularization parameter. Intuitively, this formula normalizes and accumulates the transformed feature vectors of neighboring nodes. Unlike conventional GCNs, it introduces a relation-specific transformation, which depends on the type and direction of the edges. Thus, we can obtain a vector representation of each node after incorporating information from its neighbors. Finally, after passing through the output layer of the neural network, we obtain the compatibility prediction result for this fashion apparel.

[0044] Figure 6 Our experimental results are presented, and the performance of each model in the kit compatibility prediction task is analyzed as follows.

[0045] 1. Compared with other methods, SiameseNet[4] performs poorly. This indicates that the approach of ignoring the integrity of the set and only taking the average of the compatibility of clothing pairs is insufficient to learn the overall compatibility characteristics of the set.

[0046] 2. The Bi-LSTM model outperforms SiameseNet, indicating that it can better learn the underlying knowledge about outfit compatibility. However, Bi-LSTM treats the entire outfit as a fixed sequence. While this approach can model the overall compatibility of fashion outfits, it misses many different types of relationships between different items in a fashion outfit, because the items in a fashion outfit do not have a fixed order.

[0047] 3. CSN and FHN outperform the previous two models significantly. Although they also predict overall suit compatibility by measuring the compatibility between pairs of garments, they introduce garment category information, illustrating the importance of category information in modeling fashion suit compatibility tasks. However, CSN overemphasizes the impact of garment color on suit compatibility, and FHN only considers three category labels, making it unable to model more complex compatibility relationships.

[0048] 4. The graph-based suit compatibility modeling method NGNN achieves better performance than the previous methods, indicating that compared with pairwise modeling and serialization modeling, graph structures can better model the complex interactions between clothing, making it easier to infer suit compatibility information.

[0049] 5. Our model achieves state-of-the-art performance. This is not only due to our graph structure modeling approach, but also because, compared to NGNN, we further represent the sets as heterogeneous graphs, distinguishing the types of edges between nodes of different major categories. Furthermore, we introduce a data augmentation method using hard example sampling, enabling the model to better model the compatibility of sets.

[0050] References

[0051] [1]Chen,L.,&He,Y.(2018).Dress fashionably:Learn fashion collocation with deep mixed-category metric learning.AAAI2103–2110.

[0052] [2]He,R.,Packer,C.,&McAuley,JJ(2016).Learning compatibility acrosscategories for heterogeneous item recommendation.Icdm937–942

[0053] [3]Song,X.,Feng,F.,Han,X.,Yang,X.,Liu,W.,&Nie,L.(2018).Neuralcompatibility modeling with attentive knowledge distillation.SIGIR5–14.

[0054] [4]Andreas Veit,Balazs Kovacs,Sean Bell,Julian McAuley,Kavita Bala,and Serge Belongie.2015.Learning visual clothing style with heterogeneousdyadic cooccurrences.In Proceedings of the IEEE International Conference onComputer Vision.4642–4650。

Claims

1. A method for modeling the compatibility of fashion suits based on heterogeneous graph neural networks, characterized in that... Includes the following steps: Step 1) Use similar substitution to replace one or more clothing items in the positive sample set to construct a negative sample training set for the fashion set. Step 2) Based on this dataset, the image of each outfit is fed into a pre-trained ResNet18 model to obtain the feature vector representation of each fashion garment. Step 3) constructs a fashion graph for all fashion garments, using the feature vectors from the previous step as the features of the nodes. Each node represents a type of clothing, and each edge represents the interaction between nodes. Step 4) Use a hard-example sampling training strategy to train the model on the training set. Nodes that appear less frequently during the training phase are more likely to be activated and used in training. Step 5) Use a heterogeneous graph convolutional network (RGCN) to model the interactions between nodes and learn the representations of nodes and subgraphs. Finally, after passing through the output layer of the neural network, obtain the compatibility prediction results of fashion clothing.

2. The fashion suit compatibility modeling method based on heterogeneous graph neural networks according to claim 1, characterized in that: Step 1) constructs a negative sample set in the fashion suit training set based on the principle of substitution of similar samples.

3. The fashion outfit compatibility modeling method based on heterogeneous graph neural network according to claim 1, characterized in that Step 2) uses a pre-trained ResNet18 deep neural network to extract neurons from the last layer of the network, obtaining the feature vector representation of each fashion garment.

4. The fashion outfit compatibility modeling method based on heterogeneous graph neural network according to claim 1, characterized in that Step 3) divides the clothing in the dataset into 8 major categories and constructs a fashion heterogeneous graph with 89 nodes. Each node represents a subcategory of clothing, and the edge between two nodes is determined by the co-occurrence frequency of the two nodes in the dataset.

5. The fashion outfit compatibility modeling method based on heterogeneous graph neural network according to claim 1, characterized in that Step 4) uses a hard example sampling strategy to optimize the model training process, so that nodes that appear less frequently during the training phase will get more training opportunities.

6. The fashion outfit compatibility modeling method based on heterogeneous graph neural network according to claim 1, characterized in that Step 5) uses a heterogeneous graph neural network RGCN to learn the vector representation of the subgraph corresponding to the fashion suit. After passing through the output layer of the neural network composed of the sigmoid function and MSE loss, the compatibility prediction result of the fashion suit is obtained.

Images