Fashion image personalized generation method based on multi-attribute driving

Abstract

The invention discloses a fashion image personalized generation method based on multi-attribute driving, and the method comprises the steps: firstly employing clothing fashion image data as an initial image data set, carrying out the classification according to the attribute of each image in the initial image data set, carrying out the preprocessing of an image based on a personalized demand, and carrying out the classification of the image; and taking the image with the highest similarity as a truth value. Secondly, a feature extraction module is constructed for different attributes of each picture, and attribute features of fashion elements are extracted; and finally guiding generation of a diffusion model through a color module, a style module and a clothing structure module in sequence in a diffusion process according to the attribute features and a truth value, and outputting a fashion image personalized generation result graph. According to the method, the problems of attribute confusion, image detail missing and the like possibly occurring in the image generation process can be effectively relieved, and the precision and consistency of image generation are greatly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image generation and is a personalized generation method of fashion images based on multi-attribute drive. Background Art

[0002] Image generation technology refers to a computer technology that uses deep learning to generate virtual digital images. This technology can be used to generate virtual images that meet the requirements. Image generation technology based on deep learning, from the early generative adversarial network (GAN) to the diffusion model in recent years, has promoted the development of image generation technology.

[0003] The DALL-E model released by OpenAI in 2021 is a text-generated image model that can generate complex and rich images through natural language descriptions. However, in early versions, it focused more on generating highly diverse images rather than personalized customization. The release of the diffusion model StableDiffusion in 2022 has greatly promoted the development of personalized text-generated image models, which aims to personalize the generation model through a small number of images. Its main feature is that it allows users to add new concepts to the pre-trained diffusion model so that the generated images can show specific styles, objects, people or scenes, which may not be available in the original model or expressed inaccurately.

[0004] On the other hand, the fashion industry is a field with extremely high demand for personalization. For example, designers not only need to respond to trends quickly, but also need to create a unique design style for each brand and customer. Traditional generative models are difficult to meet this demand because these image generation models can usually only generate designs that conform to the public's aesthetic taste, but cannot be customized according to the designer's specific style or brand requirements. In the field of fashion design, elements such as the shape, fabric, color, and texture of clothing have a high degree of personalization. Therefore, fashion designers hope that the generated design images can not only synchronize with the trend, but also reflect their unique creativity and style.

[0005] Although the current image generation model has the initial ability to generate personalized images, it is difficult to customize it according to the user's personal preferences or specific needs. It is not suitable for personalized customization for specific needs in the fashion field. In addition, the visual semantics in fashion design has a high degree of structural characteristics, and different types of clothing (such as dresses, coats, and pants) have unique combinations of parts and forms. The visual semantics in the fashion field has unique structural characteristics, for example, different types of clothing have different shapes and parts. Existing cross-modal clothing synthesis methods often ignore the structural correspondence between clothing images and input text prompts, resulting in inaccurate semantic alignment and poor combinatoriality. Summary of the invention

[0006] To address these challenges, a personalized generation method for fashion images based on multi-attribute driven is proposed here. This method further improves the combination generation of personalized conditions and cross-modal semantic alignment, and improves the quality and accuracy of generated images. By combining control information from multiple reference images, the generated clothing images can simultaneously contain attributes such as color and shape from different reference images. In addition, this method also solves problems such as missing clothing parts and attribute confusion, providing new ideas and methods for the development of the fashion industry.

[0007] In view of the shortcomings of the prior art, the present invention proposes a personalized generation method of fashion images based on multi-attribute drive. Considering that the existing data set is not completely suitable for the present invention, a new data set is constructed, and all experiments of the present invention are carried out on this data set. In order to improve the effect of generating images, the present invention innovates the network structure and verifies its effectiveness.

[0008] A personalized generation method of fashion images based on multi-attribute driven. The steps are as follows:

[0009] Step 1: Creation of training data set. Use clothing fashion image data as the initial image data set, classify according to the attributes of each image in the initial image data set, pre-process the images based on personalized needs, and obtain the image with the highest similarity as the true value.

[0010] Step 2: Construct a feature extraction module based on the different attributes of each image to extract the attribute features of fashion elements.

[0011] This step mainly constructs corresponding feature extraction modules for different attribute elements (such as color, style, and neckline) to extract different attribute features.

[0012] Step 3: Based on the attribute features and the true value, the diffusion model is guided through the color module, style module, and clothing structure module in turn during the diffusion process to output the personalized generation result map of the fashion image.

[0013] Preferably, step 1 is the creation of a training data set:

[0014] 1.1. Classify each picture according to its attributes, including color classification, pattern classification, and clothing part structure classification.

[0015] 1.2. Randomly select one attribute from each different attribute category to form a combination to obtain personalized needs, and all personalized needs form a personalized demand set. For example: color: red + crimson, pattern: floral, clothing structure type: short-sleeved V-neck. At the same time, in order to avoid data imbalance and confusion between similar attributes, some attribute categories will be merged or deleted. The reason for deletion is that there are too few images of certain attributes in the training set, which can easily lead to overfitting. The purpose of merging is to ensure that the model can effectively identify similar but different attribute categories during the training process to avoid confusion. For example, similar blue (light blue, dark blue, indigo) are merged into one category to ensure that the model can correctly distinguish different shades of blue.

[0016] 1.3. Through image similarity, the image with the highest similarity between each image and the personalized requirement is obtained as the true value for training. The following is an expanded introduction:

[0017] 1.3.1. For each candidate image , image encoder via multimodal large language model CLIP Generate its corresponding feature vector representation:

[0018]

[0019] in, is an image Embedding vectors generated in the CLIP model.

[0020] 1.3.2. The personalized demand is a kind of "text description", and each attribute can be expressed in natural language. In this way, the text encoder in the CLIP model can be used to convert these attribute combinations into feature vectors. Input it into the text encoder of CLIP to obtain the corresponding text embedding:

[0021]

[0022] in, is the text feature vector generated by this attribute combination, It is a personalized demand. Represents text information input into the CLIP model.

[0023] 1.3.3. Calculate the similarity between the image and the personalized attribute combination.

[0024]

[0025] in, Representing images Combined with attributes The cosine similarity is between -1 and 1. The closer it is to 1, the more similar the two vectors are, and vice versa.

[0026] 1.3.4. For all images, after calculating the similarity between each image and the personalized requirements, select the image with the highest similarity as the true value.

[0027] Preferably, step 2 is to construct a feature extraction module for different attributes to extract the attribute features of fashion elements. The feature extraction module includes three parallel branches with the same structure, which respectively extract color features, pattern features and clothing part structure features. The specific implementation process of each parallel branch is as follows:

[0028] 2.1. The input image is passed through a preliminary classifier to extract the initial attribute features corresponding to color, pattern and clothing part structure:

[0029]

[0030] in, represents the primary attribute features extracted by the primary classifier, is the input image; the input image passes through the encoder of the CLIP model to extract semantically relevant feature embedding:

[0031]

[0032] in, is the embedding vector of the image.

[0033] after, Input to the multi-layer perceptron , output a pseudo text embedding to simulate the information in the text space:

[0034]

[0035] 2.2. Add the primary features and pseudo-text embedding element by element to obtain attribute features:

[0036]

[0037] The three parallel branches obtain the corresponding attribute features respectively , corresponding to color, style and clothing structure respectively.

[0038] Preferably, step 3 designs three modules to sequentially inject different conditions into the diffusion process image generation.

[0039] The three attribute features obtained in step 2 are used as guiding conditions of the diffusion model and input into the corresponding color module, style module, and clothing structure module respectively;

[0040] The color module is specifically implemented as: initial noise Combined with the color attribute features to guide the diffusion model, generate image features that match the target color features:

[0041]

[0042] in, represents the latent variable at the current time step t in the diffusion process, and represents the image features at time t; represents the latent variable at the previous time step t-1, which represents the image features generated by the diffusion process at the previous step; The noise prediction network of the time diffusion model is used to estimate the noise in the current step;

[0043] The process of conditional injection is understood as guiding the noise through model learning. The generated image gradually possesses the basic attributes of the user input text description. The noise is learned by the model to generate an image with gradually color features:

[0044]

[0045] in, is the noise image representation corresponding to the target color feature of the true value, is the color loss function, which is used to optimize the matching degree between the image generated by the model and the target color. represents the expected value, which is the average of the overall losses at different time steps t and latent variables Z_t;

[0046] The implementation process of the style module and the clothing structure module is the same as that of the color module, and the clothing structure module finally outputs the generated personalized fashion image.

[0047] Beneficial effects of the present invention:

[0048] Through the present invention, it is possible to effectively alleviate the problems of attribute confusion and missing image details that may occur in the process of generating images, and greatly improve the accuracy and consistency of image generation. This method has broad application prospects in application scenarios such as personalized generation, virtual fitting, fashion design, etc., and can provide users with image generation services that are more in line with personalized needs. This technology has broad application prospects, especially in industries such as movies, games, and virtual reality, and can bring more business opportunities and competitive advantages to related companies. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a specific schematic diagram of the process of the present invention;

[0050] Figure 2 These are some qualitative experimental results. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and actual cases.

[0052] Suppose you want to generate a fashion image with the following description: "A model wearing a red striped long-sleeved shirt with a V-neck design stands on the left side of the picture." The personalized generation method of fashion images based on multi-attribute drive, such as Figure 1 As shown, the following steps are included:

[0053] Step 1: Creation of training dataset

[0054] Data collection: First, a large number of clothing images are selected from the image library in the fashion field as the initial data set. Detailed classification and annotation are performed based on the attributes of each image (such as color, pattern, sleeve type), for example:

[0055] Color: red, blue, white.

[0056] Patterns: stripes, solid colors, florals.

[0057] Garment structure: long-sleeved V-neck, short-sleeved round neck.

[0058] Personalized combination selection: Select a combination that matches the target description from different attributes. Filter out images that match the combination in the dataset and use them as the true value in the training set.

[0059] Data processing: To ensure that the model can learn effectively, the CLIP model is used to calculate the similarity between images and texts. The CLIP model encodes images and texts into vectors respectively, and calculates the matching degree between images and attribute descriptions based on cosine similarity, so as to select the image with the highest similarity as the true value training sample.

[0060] Step 2: construct a feature extraction module for each image’s different attributes to extract the attribute features of fashion elements.

[0061] This step mainly constructs corresponding feature extraction modules for different attribute elements (such as color, style, and neckline) to extract different attribute features.

[0062] According to different fashion attributes (such as color, pattern, and clothing structure), corresponding feature extraction modules are constructed.

[0063] The color feature extraction module extracts color features, such as using color histograms and deep learning methods to identify red.

[0064] Pattern feature extraction module: extracts pattern features such as stripes, floral patterns, etc.

[0065] Garment structure feature extraction module: extracts long-sleeve features and V-neck features through the CLIP model or pattern classification process.

[0066] The input of each module is the preliminary feature representation of the input image:

[0067]

[0068] Extract semantically relevant feature vectors through CLIP encoder:

[0069]

[0070] Then, a pseudo text embedding is generated through a multi-layer perceptron to transform the image information into a representation in the text space:

[0071]

[0072] Finally, the feature vectors of each module are added together to obtain the combined attribute features:

[0073]

[0074] Step 3: Based on the attribute features and the true value, the diffusion model is guided through the color module, style module, and clothing structure module in turn during the diffusion process to output the personalized generation result map of the fashion image.

[0075] In the generation process of the diffusion model, the color attribute condition is first injected. At the initial stage of image generation, the "red" feature is injected into the model to make the tone of the generated image consistent with the input description.

[0076] In the later stage, the pattern and clothing structure feature conditions are injected respectively, and the generated image is guided by adjusting the noise to gradually approach the description of the input text.

[0077] Step 4, iterative optimization. In each time step of the diffusion process, the details of the generated image are optimized according to the feedback of the loss function to ensure that all attributes (color, pattern, sleeve type, neckline) are accurately presented.

[0078] Step 5, the final generated image. When the diffusion process is complete, the decoder decodes the latent variables into the final fashion image. The generated image shows a model standing on the left, wearing a red striped long-sleeved, V-neck shirt, which exactly matches the input text description.

[0079] In this invention, a multi-stage diffusion image generation method based on attribute fusion is proposed to solve the problems of inaccurate attribute fusion, missing image details, and mismatch between the generated results and the input text description when the traditional image generation model processes the multi-stage image generation task. The invention achieves improvement through the following key steps:

[0080] First, a training dataset generation method suitable for the fashion field was designed, and corresponding feature classifiers and feature extractors were trained according to different fashion elements (such as color, sleeve type, neckline, etc.). By fusing image attributes with text descriptions, guiding conditions were formed to ensure accurate alignment of attributes such as color, pattern, and style during the image generation process.

[0081] Secondly, the present invention introduces the early and late condition injection mechanism. In the early stage of image generation, basic attributes (such as color) are injected to generate the image tone; in the later stage, complex attributes (such as pattern, neckline shape, etc.) are injected to refine the image details. This method organically integrates image block features with text features through a multi-layer perceptron and self-attention mechanism to ensure that the generated image not only meets the structural and content requirements, but is also highly consistent with the attributes described in the text.

[0082] Example:

[0083] In actual experiments, the images generated by this method match the input text description more accurately than traditional methods. In the dataset, this method reduces problems such as attribute confusion and missing details by injecting pre- and post-conditions. The generated images can not only correctly display details such as red, stripes, long sleeves, and V-necks, but also retain the overall spatial layout of the image and the position of the characters.

[0084] As attached Figure 2 As shown, the left column is the original image, and the right column is the generated image formed by the present invention. The first row inputs an image, and the input text prompt is: "A T-shirt with a striped pattern, similar to navy blue color, round neckline, and similar ultra-short sleeve design." Generate the effect of the right picture; the second row inputs an image, and the input text prompt is: "A T-shirt with a blended pattern, similar to black color, round neckline design, similar to ultra-long sleeve style", and generate the effect shown in the right picture; the third row inputs an image, and the input text prompt is: "A T-shirt with a solid color pattern, similar to green color, round neckline, similar to ultra-short sleeve design." Generate the effect shown in the right picture.

[0085] Through these examples, we can see how the present invention generates complex images that match the description and solves the common problems in multi-attribute generation tasks.

Claims

1. A personalized generation method of fashion images based on multi-attribute drive, characterized in that: The following steps are involved: Step 1: Use clothing fashion image data as the initial image dataset, classify the images according to the attributes of each image in the initial image dataset, pre-process the images based on personalized needs, and obtain the image with the highest similarity as the true value; Step 2: Construct a feature extraction module for each image based on different attributes to extract attribute features of fashion elements; Step 3: Based on the attribute features and the true value, the diffusion model is guided through the color module, style module, and clothing structure module in turn during the diffusion process to output the personalized generation result map of the fashion image.

2. The method for generating personalized fashion images based on multi-attribute drive according to claim 1, characterized in that: The specific implementation process of step 1 is as follows: 1.

1. Classify each picture according to its attributes, including color classification, pattern classification, and clothing part structure classification; 1.

2. Randomly select one attribute from each different attribute category to form a combination to obtain personalized needs, and all personalized needs form a personalized needs set; 1.

3. Through image similarity, the image with the highest similarity between each image and the personalized requirement is obtained as the true value for training.

3. The method for generating personalized fashion images based on multi-attribute drive according to claim 2, characterized in that: The specific implementation process of step 1.3 is as follows: 1.3.

1. For each candidate image , image encoder via CLIP Generate its corresponding feature vector representation: ； in, is an image Embedding vectors generated in the CLIP model; 1.3.

2. The personalized demand is a text description, and each attribute is expressed in natural language. The attribute combination is converted into a feature vector using the text encoder in the CLIP model. The feature vector is input into the text encoder of CLIP to obtain the corresponding text embedding: ； in, is the text feature vector generated by this attribute combination, It is a personalized demand. Represents text information input into the CLIP model; 1.3.

3. Calculate the similarity between the image and the personalized attribute combination: ； in, Representing images Combined with attributes The similarity between 1.3.

4. For all images, after calculating the similarity between each image and the personalized requirements, select the image with the highest similarity as the true value.

4. The method for generating personalized fashion images based on multi-attribute drive according to claim 3 is characterized in that: The specific implementation process of step 2 is as follows: The feature extraction module includes three parallel branches with the same structure, which respectively extract color features, pattern features and clothing part structure features. The specific implementation process of each parallel branch is as follows: The input image is passed through a preliminary classifier to extract primary attribute features corresponding to color, pattern, and clothing part structure: ； in, represents the primary attribute features extracted by the primary classifier, is the input image; The input image passes through the encoder of the CLIP model to extract semantically relevant feature embeddings: ； in, is the embedding vector of the image; then Input into a multi-layer perceptron and output a pseudo text embedding: ； Add the primary attribute features to the pseudo-text embedding element by element to obtain the attribute features : ； The three parallel branches obtain the corresponding attribute features respectively , and , corresponding to color, style and clothing structure respectively.

5. The method for generating personalized fashion images based on multi-attribute drive according to claim 4, characterized in that: The color module, style module, and clothing structure module have the same structure and are connected in series in sequence; The three attribute features obtained in step 2 are used as guiding conditions of the diffusion model and input into the corresponding color module, style module, and clothing structure module respectively; The color module is specifically implemented as follows: initial noise Combine the color attribute features to guide the diffusion model and generate image features that match the target color features; ； in, represents the latent variable at the current time step t in the diffusion process, and represents the image features at time t; represents the latent variable at the previous time step t-1, which represents the image features generated by the diffusion process at the previous step; is the noise prediction network of the diffusion model; The process of conditional injection is understood as guiding the noise through model learning. The generated image gradually possesses the basic attributes of the user input text description. The noise is learned by the model to generate an image with gradually color features: ； in, is the noise image representation corresponding to the target color feature of the true value, is the color loss function, which optimizes the matching degree between the image generated by the model and the target color. represents the expected value, which refers to the different time steps t and latent variables The overall loss is averaged; The implementation process of the style module and clothing structure module is the same as that of the color module, and the clothing structure module finally outputs a personalized fashion image.

Images