Fashion style learning method for clothing images based on fashion puzzle and dynamic clustering

Abstract

The present application relates to a kind of fashion puzzle and dynamic clustering-based fashion image fashion style learning method, the present application proposes dynamic clustering algorithm, by introducing "centroid" this attribute, make new data can be clustered more quickly, reduce the computing burden;In addition to facilitate calculation, it is also the representative feature of the cluster, is the representative image of the fashion style;Noise is ignored in traditional clustering its role, and in the study of fashion style, noise is regarded as potential style, when enough number of noise gathers together, new cluster will be formed, which is important for the discovery of new fashion style.

Description

Technical Field

[0001] This invention relates to the field of image classification methods, specifically a fashion style learning method for clothing images based on fashion mosaic and dynamic clustering. Background Technology

[0002] In 2014, Kiapour et al. from the University of Southern California, as some of the earliest experts in fashion style learning, designed an online game to obtain artificial labels for fashion images. They divided 1,893 images with artificial annotations into five styles and studied the influence of clothing attributes on fashion style by analyzing the differences in image attributes within and between these styles.

[0003] In 2017, Professor Jia Jia's team at Tsinghua University manually proposed a multimodal method. They created a fashion runway dataset with visual features and designed a fashion semantic space based on Kobayashi's aesthetic theory. The features of the upper and lower body clothing were extracted separately and then combined as a representation of the fashion image, which was then mapped onto the designed metric space to obtain style attributes.

[0004] In the same year, Wei-Lin Hsiao and Kristen Grauman of the University of Texas at Austin were inspired by topic models in natural language processing, analogizing fashion styles to topics in text, and various attributes in images to words in the text. They then used reinforcement learning to infer the strength of fashion styles from the basic attributes of fashion images.

[0005] In 2021, Youngseung Jeon and colleagues from Ajou University in South Korea incorporated expert knowledge, completing the annotation of a fashion runway full-body photo dataset called Fancy under expert guidance, and establishing 25 styles based on expert opinions. Expert opinions improved the accuracy and reliability of the label attributes, enabling downstream tasks to achieve better results.

[0006] In 2022, Professor Peng Dunlu of Shanghai University of Science and Technology used a pre-trained model to extract attribute features from fashion images and used topic modeling in natural language processing to calculate the style vector of each fashion element. Each dimension of the vector represents the probability that the fashion element belongs to the corresponding style.

[0007] Most existing methods use supervised training to train feature extraction models, so research focuses on feature extraction capability and label accuracy. Therefore, some methods modify the model itself, such as introducing multimodality or attempting to change the model structure; others try to reduce irrelevant information in the labels, such as using natural language processing methods for classification or improving label accuracy through expert opinions. However, these methods all suffer from the following problems:

[0008] 1. It is unavoidable that people's evaluation of fashion style will be affected, and the annotation is a highly repetitive operation with a large amount of data, which makes it very troublesome for people to annotate images and will affect the accuracy of style prediction results.

[0009] 2. At the same time, these methods all presuppose a certain number of style attributes. Although this can make subsequent work easier, the limited classification results or style tags cannot accurately indicate the clothing style, do not conform to human visual perception, and cannot discover emerging fashion styles. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide a fashion style learning method for clothing images based on fashion puzzle and dynamic clustering, which addresses the above-mentioned shortcomings.

[0011] To solve the above technical problems, the present invention adopts the following technical solution:

[0012] A fashion style learning method for clothing images based on fashion mosaic and dynamic clustering includes the following steps:

[0013] Step 1: Preprocess the clothing image dataset, which includes cutting, enhancement and stitching. After preprocessing, a fashion mosaic set is obtained.

[0014] Step 2: Input the fashion puzzle set into the backbone network for feature extraction to obtain the image feature vector of each fashion puzzle;

[0015] Step 3: Calculate the clustering loss and localization loss of the image feature vector of each fashion puzzle. By minimizing the loss function, adjust the parameters of the backbone network to obtain the pre-trained model.

[0016] Step 4: Input a portion of the preprocessed target dataset into the pre-trained model, and fine-tune the parameters of the pre-trained model based on the output results to obtain the trained model.

[0017] Step 5: Input the target dataset into the trained model from Step 4 to obtain the image feature queue of the target dataset;

[0018] Step 6: Input the image feature queue of the target dataset into the dynamic density clustering algorithm to obtain the initial clustering results, calculate the centroid of each cluster, label similar images as the same cluster, images in the same cluster represent the same fashion style, each cluster represents a fashion style type, noise represents the latent style, and the centroid is the representative image of each fashion style.

[0019] Step 7: Input the newly added image data into the trained model to obtain the image features of the newly added image data. Input the image features of the newly added image data into the dynamic density clustering algorithm to calculate the new distance from the initial cluster. Cluster with the nearest cluster and mark it as an existing cluster, form a new cluster or noise. Obtain the corresponding fashion style type based on the existing clusters and take the new cluster as the new style type.

[0020] Furthermore, step 1 includes the following steps:

[0021] Step 11: Scale the input image to the preset size;

[0022] Step 12: Cut the scaled image into four image blocks from top to bottom according to the ratio of 1:1.6:2.2:1.2. The four image blocks correspond to the head image block, upper body image block, lower body image block and foot image block respectively, and the adjacent image blocks have 20% overlap.

[0023] Step 13: Use image enhancement methods to enhance the cut image blocks to obtain enhanced image blocks;

[0024] Step 14: Shuffle the enhanced image blocks and randomly piece them together in the order of head image block, upper body image block, lower body image block, and foot image block from top to bottom to form a fashion jigsaw puzzle.

[0025] Furthermore, the image enhancement method in step 13 includes one or more of the following: randomly rotating, flipping, fading, changing image brightness, changing image contrast, changing image saturation, changing image hue, grayscale conversion, or Gaussian blurring of the cut image blocks.

[0026] Furthermore, step 3 includes the following steps:

[0027] Step 31: Cut the image feature vector of the fashion puzzle into image feature vectors of four image blocks, which respectively represent the head, upper body, lower body and shoes;

[0028] Step 32: Calculate the clustering loss of the fashion jigsaw puzzle feature vector using the clustering loss function. The clustering loss function uses an adversarial learning approach, and the formula for calculating the clustering loss function is as follows:

[0029]

[0030] Where n represents the number of fashion puzzle pieces, l i，j The cosine similarity loss between each image patch of the fashion jigsaw puzzle is given by i and j, which refer to two different image patches in the fashion jigsaw puzzle, and ci represents the set of image patches from the same original image.

[0031] Step 33: Calculate the localization loss of the fashion puzzle feature vector using the localization loss function. The localization loss function uses cross-entropy loss to determine the accuracy of the location of each image patch. The formula for calculating the localization loss function is as follows:

[0032]

[0033] Where gt is the benchmark (ground truth);

[0034] Step 34: The final loss function is:

[0035]

[0036] The Adam backbone network parameters are optimized to minimize the loss function, resulting in a pre-trained model.

[0037] Furthermore, in step 6, the image feature queue of the target dataset is clustered using the DBSCAN clustering algorithm. After the clustering is completed, the centroid of each cluster is calculated by calling the center_of_mass method of the scipy library.

[0038] Furthermore, in step 7, the method of inputting the image features of the newly added image data into the dynamic density clustering algorithm, calculating the distance to the centroid of the initial cluster, clustering with the nearest cluster, and marking it as an existing cluster, forming a new cluster, or noise includes the following steps:

[0039] Step 71: Measure the distance between each feature of the data to be processed and the centroid of each cluster, find the nearest cluster and perform clustering calculation. If the clustering is successful, mark the feature as the corresponding cluster; otherwise, proceed to step 72.

[0040] Step 72: Cluster the feature with the noise points in the distance space. If the clustering is successful, mark the feature as a new cluster; otherwise, mark it as noise.

[0041] Furthermore, the amount of data in the clothing image dataset in step 1 is greater than the amount of data in the target dataset in step 4.

[0042] Compared with the prior art, the present invention, by adopting the above technical solution, has the following advantages:

[0043] 1. For the first time, clustering algorithms were introduced into fashion style learning, achieving the results of unlimited style types and the ability to detect new styles;

[0044] 2. Traditional density clustering algorithms require a lot of computing power when adding new image data. To improve this problem, this invention proposes a dynamic clustering algorithm. By introducing the attribute of "centroid", new data can be clustered more quickly, reducing the computational burden.

[0045] 3. The dynamic clustering algorithm of this invention also plays an important role in fashion style learning. The centroid of each cluster, in addition to being convenient for calculation, also reflects the representative features of the cluster and is the representative image of the fashion style. The role of noise is ignored in traditional clustering, but in the study of fashion style, noise is regarded as a potential style. When a sufficient number of noises are clustered together, a new cluster will be formed, which is of great significance for the discovery of new fashion styles.

[0046] 4. Past research on fashion styles mainly adopted supervised methods, training feature extraction networks by assigning attributes or styles to fashion images. The fashion jigsaw puzzle pre-training model proposed in this invention does not use any manual annotation or segmentation networks. It obtains fashion jigsaw puzzles by preprocessing the input images, which facilitates the model to learn the features of each part of the image at the same time. This makes it easier for the trained model to extract the features of fashion elements, fashion images, and fashion images. By introducing a clustering process, the fashion jigsaw puzzle can not only focus on the features of each fashion element within the image and the overall feeling of fashion matching, but also distinguish the differences between different fashion images.

[0047] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0049] Figure 2 This is a flowchart of the image preprocessing process of the present invention;

[0050] Figure 3 This is a flowchart of the unsupervised training model of the present invention;

[0051] Figure 4 This is a flowchart of the dynamic density clustering algorithm of the present invention;

[0052] Figure 5 This is a schematic diagram illustrating the implementation effect of the fashion detective of the present invention;

[0053] Figure 6 This is a schematic diagram illustrating the effect of implementing the fashion consultant of the present invention. Detailed Implementation

[0054] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0055] I. For example Figure 1 As shown, the overall technical solution of this project includes the following steps:

[0056] 1. Pre-task design: Divide the dataset into groups, cut the images within each group to obtain several image patches, perform random enhancements on the image patches, and then stitch the image patches together to form a fashion jigsaw puzzle;

[0057] 2. Model pre-training: The fashion jigsaw puzzle is input into the backbone network. The backbone network extracts features and segments them. The clustering loss function is used to group feature blocks of the same image together. The localization loss function is used to arrange the feature blocks in their original order. The clustering loss and localization loss help the backbone network adjust its parameters to obtain the pre-trained model.

[0058] 3. Feature extraction and clustering: The pre-trained model is fine-tuned with a small amount of data to obtain a trained model (i.e., a trained backbone network). The trained model is used to extract features from the dataset, and the feature points are dynamically clustered by density to obtain the clustering result, i.e., the fashion style result.

[0059] II. Implementation Method:

[0060] This invention proposes a fashion style learning framework, comprising two modules: (1) an unsupervised training model; and (2) image feature extraction and dynamic clustering.

[0061] (1) Unsupervised pre-trained model

[0062] In the unsupervised training module, to train a model capable of recognizing fashion image features, we designed a handcrafted pre-task, "fashion jigsaw puzzle." Then, we used two loss functions: clustering loss and localization loss, to help tune the network parameters.

[0063] 1. Pre-construction craft task – Fashion Puzzle

[0064] With the rise of unsupervised training, a variety of carefully designed pre-training tasks have been developed to train neural network models, such as rotation, coloring, image completion, relation prediction, and context prediction. Pre-training tasks are indirect tasks designed to complete specific training tasks.

[0065] The fashion jigsaw puzzle used in Step 1 is a type of relationship prediction. To solve the jigsaw puzzle problem, the model first needs to obtain the features of each puzzle piece, and then solve the puzzle problem based on its image and location. In this process, the model can also learn to distinguish the similarity between different puzzle pieces in the same image, as well as the differences between images.

[0066] Figure 2 This is a flowchart of image preprocessing, where x nThis represents the nth image in a dataset. The superscripts h, u, l, and f represent the head, upper body, lower body, and feet, respectively. x′ represents the enhanced image. n This represents the nth fashion jigsaw puzzle image. Specifically, the input image is first scaled to 300x200 pixels. Then, the image is cut from top to bottom in a ratio of 1:1.6:2.2:1.2, resulting in four image blocks representing the head, upper body, lower body, and feet, respectively. A 20% overlap is used to ensure partial overlap between the cut image blocks. Next, image enhancement methods (which are not fixed and can include rotation, fading, or other methods) are applied to the image blocks, randomly rotating, fading, color dithering (changing image brightness, contrast, saturation, and hue), grayscale conversion, Gaussian blurring, or flipping. After all the images in the set are cut and enhanced in a 1:1.6:2.2:1.2 ratio, the order of the image blocks is shuffled, and then they are stitched together to form an image of size 348x200 pixels, i.e., the fashion jigsaw puzzle.

[0067] 2. Unsupervised training model

[0068] Since the fashion jigsaw puzzle lacks manually labeled information, there is no standard data objective. Steps 2 and 3 use an unsupervised approach to adjust the parameters in the backbone network through a loss function, continuously converging until training is complete.

[0069] Figure 3 This is a flowchart of an unsupervised training model, where x′ n f represents the nth fashion puzzle piece. n This represents the feature vector obtained from the nth image through the trained backbone network. The superscripts h, u, l, and f represent the head, upper body, lower body, and feet, respectively. Specifically, the fashion mosaic has been preprocessed in step 1 and used as input to the backbone network. In this embodiment, we choose ResNet50 as the backbone network, but it can be changed to a specific network model, such as "ResNet" or "AlexNet". The extracted image feature vector is divided into four parts, representing the head, upper body, lower body, and shoes. The clustering loss function uses adversarial learning to calculate the cosine similarity loss l between each image block of the n fashion mosaics. i，j Clustering groups image patches of the same image together and separates image patches of different images. The formula for calculating the clustering loss function is:

[0070]

[0071] Where n represents the number of fashion puzzle pieces, l i，j The cosine similarity loss between each image patch of the fashion jigsaw puzzle is given by i and j, which refer to two different image patches in the fashion jigsaw puzzle, and ci represents the set of image patches from the same original image.

[0072] The localization loss function uses cross-entropy loss to determine the accuracy of the location of each image patch. The formula for calculating the localization loss function is as follows:

[0073]

[0074] Where gt is the benchmark (ground truth);

[0075] The final loss function is:

[0076]

[0077] We use Adam to optimize the backbone network parameters, continuously reducing the loss function to obtain a pre-trained model. To achieve better results with the pre-trained model, in step 4 we use a small amount of the target dataset as a training set to fine-tune the backbone network parameters, obtaining a trained model.

[0078] (2) Image feature extraction and dynamic clustering

[0079] In the image feature extraction and dynamic clustering module, a trained model (i.e., a trained backbone network) is used to extract features from the input image, and a dynamic density clustering algorithm is used to construct the feature space. The dynamic density clustering algorithm has two cases: For the initial clustering, the traditional DBSCAN algorithm is used, and the centroid of each cluster is calculated after clustering. When the feature space already exists, the distance between newly added feature points and the cluster centroids is directly calculated, and clustering is performed with the nearest cluster. If a point cannot be clustered into the nearest cluster, it is clustered with noisy points to see if a new cluster can be formed.

[0080] 3. Dynamic density clustering algorithm

[0081] The DBSCAN density clustering algorithm does not require pre-setting the cluster types (K value); instead, it performs clustering by setting the distance radius (Eps) and the minimum density reachable point (minPts). However, the DBSCAN algorithm does not provide a good solution for adding new image data later; instead, it recalculates the clustering for all data. When the data volume is very large, the computational performance requirements also increase. To address this issue, we designed a dynamic density clustering algorithm.

[0082] In step 5, we use the fine-tuned model (i.e. the trained model) to extract features from the new image data, and input all the resulting feature queues into the dynamic density clustering algorithm.

[0083] like Figure 4The flowchart shown is for the dynamic density clustering algorithm. Specifically, in step 6, for the initial dataset, we still use the traditional DBSCAN clustering algorithm for clustering. However, after clustering, we add an additional step to calculate the centroid of each cluster by calling the `center_of_mass` method of the scipy library. Images in the same cluster represent the same fashion style, while different clusters represent different style types. Noise represents a latent style, and the centroid is a representative image of each style. When new data is added, we proceed to step 7, directly measuring the distance between the features of the new image data and each cluster, and finding the nearest cluster for clustering calculation. If clustering is successful, the image is assigned to that cluster; otherwise, it is clustered with noise in the distance space. If the clustering requirements are met, a new cluster label is assigned, indicating the generation of a new style. If clustering fails, the image is labeled as noise.

[0084] In summary, past research on fashion style has primarily employed supervised methods, training feature extraction networks by assigning attributes or styles to fashion images. The fashion jigsaw puzzle pre-trained model proposed in this paper does not use any manual annotation or segmentation networks, relying instead on carefully designed manual pre-tasks to enable the model to acquire feature extraction capabilities.

[0085] Fashion puzzles are a variation of manual puzzle tasks. The initial puzzle task simply involved randomly cropping a square image, dividing it into 3x3 blocks, and locating these nine blocks to solve the puzzle, enabling the trained model to extract features and relationships between parts of the image. Fashion puzzles, however, focus not only on the features of individual fashion elements and the overall feel of a fashion combination, but also on distinguishing the differences between different fashion images. Therefore, a clustering process is introduced. It's like having multiple puzzle pieces mixed together; first, you need to identify the different pieces, group them together, and then assemble the puzzle. This task design allows the model to simultaneously learn the features of each part of the image, which constitute the overall features of the entire image, as well as the differences between different images.

[0086] We also discovered that for pre-defined style categories, density clustering can achieve similarity results without pre-setting the number of categories. We are the first to introduce clustering algorithms into fashion style learning, achieving both unlimited style categories and the ability to detect new styles. Traditional density clustering algorithms require significant computational resources when adding new image data. To address this, we propose a dynamic density clustering algorithm that introduces the "centroid" attribute, enabling faster clustering of new data. Besides reducing computational burden, this algorithm plays a crucial role in fashion style learning. The centroid of each cluster, in addition to facilitating computation, reflects the representative features of that cluster and represents the style. Noise points are often overlooked in traditional clustering, but in fashion style research, we treat them as potential styles. When a sufficient number of noise points cluster together, new clusters are formed, which is significant for discovering new styles.

[0087] Example 1:

[0088] This example uses the aforementioned fashion puzzle unsupervised pre-trained model and dynamic density algorithm, with a focus on the application of the dynamic density algorithm.

[0089] like Figure 5 As shown, a new set of images is input, and feature vectors are obtained through a pre-trained backbone network. Dynamic density clustering is then performed in the feature space. It is found that this set of images cannot be clustered with existing styles, but it can be clustered with noise to form new clusters. We thus discover a new style, whose representative form is... Figure 5 The one on the far right is a popular winter outfit.

[0090] Example 2:

[0091] This example uses the aforementioned unsupervised pre-trained model and dynamic density algorithm for fashion jigsaw puzzles, highlighting the ingenuity of the fashion jigsaw puzzle task design.

[0092] like Figure 6 As shown, we input an image of a fashion element, and using a dynamic density clustering algorithm, we calculate the four nearest feature vectors. Since we extract the features of each element during feature extraction, we can directly compare distances in the feature space based on the features of the input fashion element, and recommend other fashion elements with the most potential similarity (same style).

[0093] The above description provides examples of the preferred embodiments of the present invention. Parts not detailed herein are common knowledge to those skilled in the art. The scope of protection of the present invention is determined by the claims. Any equivalent modifications based on the technical teachings of the present invention are also within the scope of protection of the present invention.

Claims

1. A fashion style learning method for fashion image based on fashion puzzle and dynamic clustering, characterized in that, The method comprises the following steps: Step 1, preprocessing the clothing image dataset, wherein the preprocessing comprises cutting, enhancing and splicing, and the preprocessing obtains a fashion puzzle set; Step 2, inputting the fashion puzzle set into a backbone network to extract image feature vectors of each fashion puzzle; Step 3, calculating clustering loss and positioning loss of the image feature vectors of each fashion puzzle, adjusting parameters of the backbone network by minimizing the loss function, and obtaining a pre-trained model; Step 4, inputting part of the preprocessed target dataset into the pre-trained model, fine-tuning parameters of the pre-trained model according to the output result, and obtaining a trained model; Step 5, inputting the target dataset into the trained model of step 4 to obtain an image feature queue of the target dataset; Step 6, inputting the image feature queue of the target dataset into a dynamic density clustering algorithm to obtain an initial clustering result, calculating the centroid of each clustering cluster, marking similar images as the same cluster, and marking images in the same cluster as having the same fashion style, wherein each cluster represents a fashion style type, and a noise point represents a potential style, and the centroid is a representative image of each fashion style; In step 6, the image feature queue of the target dataset is clustered by the DBSCAN clustering algorithm, and the centroid of each clustering cluster is calculated by calling the center_of_mass method of the scipy library after the clustering is completed; Step 7, inputting new image data into the trained model to obtain image features of the new image data, inputting the image features of the new image data into the dynamic density clustering algorithm, calculating the distance from the centroid of the initial clustering cluster, clustering with the nearest cluster, and marking as an existing cluster, forming a new cluster or a noise point, obtaining the corresponding fashion style type according to the existing cluster, and taking the new cluster as a new style type; In step 7, the method of inputting the image features of the new image data into the dynamic density clustering algorithm, calculating the distance from the centroid of the initial clustering cluster, clustering with the nearest cluster, and marking as an existing cluster, forming a new cluster or a noise point comprises the following steps: Step 71, measuring the distance between each feature of the to-be-processed data and the centroid of each cluster, finding the nearest cluster for clustering calculation, and if the clustering is successful, marking the feature as the corresponding cluster; otherwise, step 72 is performed; Step 72, clustering the feature with the noise points in the distance space, and if the clustering is successful, marking the feature as a new cluster; otherwise, marking the feature as a noise point.

2. The fashion puzzle and dynamic clustering based fashion style learning method for garment images according to claim 1, characterized in that, The step 1 comprises the following steps: Step 11, scaling the input image to a preset size; Step 12, cutting the scaled image into four image blocks according to the ratio of 1:1.6:2.2:1.2 from top to bottom, wherein the four image blocks correspond to a head image block, an upper body image block, a lower body image block and a foot image block respectively, and adjacent image blocks have a 20% overlap; Step 13, enhancing the cut image blocks using an image enhancement method to obtain enhanced image blocks; Step 14, shuffling the enhanced image blocks, and randomly splicing them together in the order of the head image block, the upper body image block, the lower body image block and the foot image block from top to bottom to form a fashion puzzle.

3. The fashion puzzle and dynamic clustering based fashion image fashion style learning method according to claim 2, characterized in that, The image enhancement method in step 13 includes randomly rotating, flipping, fading, changing image brightness, changing image contrast, changing image saturation, changing image hue, graying or Gaussian blurring the cut image block.

4. The fashion puzzle and dynamic clustering based fashion image fashion style learning method according to claim 1, characterized in that, The step 3 includes the following steps: Step 31, cutting the image feature vector of the fashion puzzle into four image feature vectors of image blocks, respectively representing the head, upper body, lower body and shoes; Step 32, calculating the clustering loss of the fashion puzzle feature vector through a clustering loss function, the clustering loss function uses an adversarial learning method, and the calculation formula of the clustering loss function is: ； wherein n represents the number of fashion puzzles, is the cosine similarity loss between each image block of the fashion puzzle, i and j represent two different image blocks in the fashion puzzle, and ci represents the set of image blocks of the same original image. Step 33, calculating the positioning loss of the fashion puzzle feature vector through a positioning loss function, the positioning loss function uses cross-entropy loss to judge the accuracy of the position of each image block, and the calculation formula of the positioning loss function is: ； wherein, ground truth; Step 34, the final loss function is: ； Using Adam to optimize the parameters of the backbone network to minimize the loss function and obtain a pre-trained model.

5. The fashion puzzle and dynamic clustering based fashion image fashion style learning method according to claim 1, characterized in that, The data amount in the clothing image data set in step 1 is greater than the data amount in the target data set in step 4.

Images