A multimodal fashion trend prediction and evaluation method

Abstract

The present invention discloses a multimodal fashion trend prediction and evaluation method, including: according to the full amount of trend feature parameters, a reference target image is marked to obtain the full amount of feature parameters; a Transformer model is used to perform dimensionality reduction processing, and a strongly correlated feature parameter is extracted to form a trend feature template; CLIP is fine-tuned according to the trend feature template to enhance the cross-modal understanding ability and realize the semantic alignment of image and text; the reference target image data set is annotated by the fine-tuned CLIP model to obtain image-text data pairs; the diffusion model is fine-tuned for text-generated graphs and image-generated graphs, and image generation is performed to obtain clothing trend images; an evaluation mechanism is introduced to evaluate the effect of the generated clothing trend images. Through the present invention, strongly correlated feature parameters can be flexibly, quickly and accurately screened, more accurate data annotations can be obtained, and clothing images that are more in line with fashion trends can be generated.

Description

Technical Field

[0001] The present invention belongs to the field of generative artificial intelligence, and in particular relates to a multimodal fashion trend prediction and evaluation method. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, multimodal data analysis methods based on deep learning have been widely used in various fields. Especially in the fashion industry, how to use multimodal data (such as images, text, etc.) for trend prediction has become an important research direction. Traditional trend prediction methods usually rely on single-modal data, such as prediction through historical sales data or single image analysis. However, these methods often show certain limitations when processing complex multimodal data, and it is difficult to fully capture the semantic associations of industry trends.

[0003] In recent years, the development of diffusion models and multimodal deep learning models (such as CLIP, Contrastive Language-Image Pre-Training) has provided new solutions for trend prediction. Diffusion models have shown excellent performance in generating high-quality images, and the diversity of their generation process is particularly suitable for personalized fashion trend creation, which will have a significant impact on the development of this field. The CLIP model has strong cross-modal understanding capabilities and can extract relevant information from images and texts.

[0004] Fashion trend forecasting predicts the development direction of clothing in a specific period of the future by analyzing and summarizing historical and current popular patterns and influencing factors. With the advent of the artificial intelligence era, the threshold for informatization in the fashion industry has been lowered, the amount of information has surged, and a huge amount of information resources have emerged, including consumer information, browsing and consumption records, fashion week data, style pictures, text descriptions, and keyword searches. How to capture real fashion trends faster, more comprehensively, and more accurately will be very beneficial for companies to seize market opportunities and improve their competitiveness.

[0005] However, in the field of multimodal fashion trend forecasting, existing technologies still have the following problems:

[0006] 1. Fashion trends change quickly. Clothing products have the characteristics of short life cycle, fast replacement, and unstable demand, which bring great difficulties to the prediction of fashion trends;

[0007] 2. With the upgrading of consumption, consumers' preferences are becoming increasingly diversified and personalized. Trend forecasting needs to keep up with the rapid changes in the market and capture users' subtle personalized trends, which faces great challenges;

[0008] 3. It is necessary to produce clothing or marketing materials that conform to fashion trends while maintaining a certain degree of originality;

[0009] 4. Data complexity: existing methods have difficulty in efficiently extracting strongly correlated parameters from large-scale multimodal datasets;

[0010] 5. Insufficient cross-modal semantic alignment: In the process of trend analysis and image generation, accurate alignment and mapping of semantic information remains a challenge;

[0011] 6. Lack of image generation quality evaluation standards: There is currently a lack of a complete set of methods to systematically evaluate the quality and industry attributes of generated images.

[0012] Therefore, there is an urgent need for a method that can effectively integrate multimodal data and have visual trend prediction and effect evaluation to meet the fashion industry's needs for trend analysis and prediction. Summary of the invention

[0013] The purpose of the present invention is to address the shortcomings of the existing technology and propose a multimodal fashion trend prediction and evaluation method; by finding strongly correlated feature variables of popular trends in a specific data set and performing multimodal semantic alignment, the diffusion model is used to generate images for visualization to predict trends, and finally an evaluation mechanism is introduced to evaluate the quality, similarity and industry attributes of the generated images.

[0014] To achieve the above object, the present invention provides a multimodal fashion trend prediction and evaluation method, comprising the following steps:

[0015] (1) Based on the established full-volume trend feature parameters, the reference target image is labeled to obtain a full-volume feature parameter description;

[0016] (2) Use the Transformer model to reduce the dimension of all feature parameters, extract strongly correlated feature parameters, and form a trend feature template;

[0017] (3) Fine-tune CLIP based on trend feature templates to enhance cross-modal understanding capabilities and achieve semantic alignment of images and texts;

[0018] (4) Use the fine-tuned CLIP model to annotate the reference target image dataset and obtain an optimized feature parameter description, i.e., an image-text data pair;

[0019] (5) Based on the image-text data pairs, the diffusion model is fine-tuned for the text-generated image type and the image-generated image type, and image generation is performed to obtain a clothing trend image;

[0020] (6) An evaluation mechanism is introduced to evaluate the effectiveness of the generated clothing trend images.

[0021] Furthermore, in step (2), the Transformer model is used to perform attention weight analysis and SHAP value analysis on all feature parameters; the attention weight of each feature is extracted through attention weight analysis, and important features are selected based on these weights; the contribution of each feature to model prediction is quantified through SHAP value, and the features are ranked according to their importance.

[0022] Furthermore, the step (3) includes the following sub-steps:

[0023] (3.1) Establish a fine-tuned CLIP model structure, including an image encoder and a text encoder. These two encoders are pre-trained to make semantically related image and text features close in the same feature space.

[0024] (3.2) Freeze encoder parameters: Freeze the parameters of the image encoder so that its parameters do not participate in training;

[0025] (3.3) Add task-specific layers: Add a new output layer for the target task and perform fine-tuning learning;

[0026] (3.4) Define the loss function: Use the cross entropy loss function to measure the gap between the predicted result and the true label;

[0027] (3.5) Update only the parameters of the task-specific layers through back-propagation; evaluate the model on the validation set and adjust the hyperparameters until the model achieves optimal performance on the task-specific layers.

[0028] Furthermore, in step (4), the reference target image data set is processed to obtain the patterns and relationships of the batch of image data sets of interest, reflecting user preferences and clothing fashion trends.

[0029] Furthermore, the step (5) includes the following sub-steps:

[0030] (5.1) By fine-tuning the prompts, freeze all parameters of the diffusion model, convert the input text prompts into vector embeddings, and then fine-tune these embeddings on the training data to adapt to the image generation requirements;

[0031] (5.2) Insert a lightweight adapter into the diffusion model, keep the parameters of the diffusion model frozen, and perform fine-tuning for specific tasks to generate clothing trend images;

[0032] (5.3) Introduce a similarity-controlled loss function; add a similarity control factor to the loss function to balance the similarity between the generated image and the input image; introduce the loss function with the similarity control factor into the fine-tuning process of the adapter, and update the adapter weights through gradients to achieve similarity control between the generated image and the input image.

[0033] Furthermore, in step (6), the generated clothing trend image is evaluated using the SSIM, FID and IS indicator dimensions; wherein:

[0034] The SSIM indicator dimension is used to compare the structural similarity between the generated image and the real clothing image to ensure that the generated image is consistent with the design requirements in visual details;

[0035] The FID indicator dimension is used to ensure that the overall quality and diversity of the generated images are consistent with the distribution of real clothing images by calculating the distance between the real clothing images and the generated images in the feature space;

[0036] The IS indicator dimension is used to evaluate the diversity of generated images and their relevance to the identified clothing categories, ensuring that the generated designs are accurate and cover a variety of styles.

[0037] To achieve the above objectives, the present invention also provides a multimodal fashion trend prediction and evaluation device, comprising one or more processors for implementing the above multimodal fashion trend prediction and evaluation method.

[0038] To achieve the above-mentioned purpose, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multimodal fashion trend prediction and evaluation method.

[0039] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the above multimodal fashion trend prediction and evaluation method when executed by a processor.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Flexible, fast and accurate screening of strongly correlated feature parameters. In multimodal trend mining, existing methods are difficult to efficiently extract strongly correlated parameters from large-scale multimodal data sets, especially in scenarios with changing demands. The present invention can dynamically adjust strongly correlated feature parameters according to demand;

[0042] 2. More accurate data annotation. Incorporate strongly related feature parameters into the visual model for multimodal alignment and fine-tuning, solve the problem of insufficient cross-modal semantic alignment, and obtain more accurate image-text annotations that meet the needs;

[0043] 3. Generate clothing images that are more in line with the trend. Fine-tune the diffusion model through image-text pairs to generate clothing images that are more in line with fashion trends;

[0044] 4. Visual evaluation of generated images. Through SSIM, FID, IS and other evaluation methods, a set of evaluation frameworks for generating results in terms of structural similarity, image quality, industry attributes and other dimensions is proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a flow chart of the multimodal fashion trend prediction and evaluation method of the present invention;

[0047] Figure 2 It is a trend word cloud diagram in the present invention;

[0048] Figure 3 It is a trend segmentation feature analysis diagram in the present invention;

[0049] Figure 4 It is a structural schematic diagram of the device of the present invention;

[0050] Figure 5 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 , the present application provides a multimodal fashion trend prediction and evaluation method, comprising the following steps:

[0053] (1) Based on the established full-volume trend feature parameters, the reference target image is labeled to obtain a full-volume feature parameter description.

[0054] Specifically, according to the established full-volume trend feature parameters (image dataset FashionImageSets), the images are labeled or recognized to obtain a full-volume feature parameter description (complete feature parameter set), which contains more than 200 subdivided features. Some feature parameters are listed as follows:

[0055] Characters: number, gender, age, face, hairstyle, hands, feet, posture, and line of sight;

[0056] Clothing: brand, fabric, color, style, silhouette, details (collar, sleeves, front, pants), elements, patterns, accessories (headwear, earrings, necklaces, rings, bracelets, hats, shoes, socks, belts, bags), function, clothing style, wearing season;

[0057] Aesthetics: objective (sharpness, exposure, vignetting, color, contrast, white balance, tone, light), subjective (composition, atmosphere, texture, emotion);

[0058] Scene: style, occasion, background, season;

[0059] Shooting techniques: angle, perspective, frame, focal length.

[0060] (2) Use the Transformer model to reduce the dimensionality of all feature parameters, extract strongly correlated feature parameters, and form a trend feature template.

[0061] Use the complete industry characteristic parameters as prompt words, take the popular trend sample set as the reference target data set, use the CLIP model to perform full reasoning, and get a text description response. At this time, the full description annotation of the reference target data set will be obtained.

[0062] The sample set of popular trends is determined according to the demand target. The pictures can come from the social networking sites of TOP fashion bloggers, hot-selling items on shopping sites, and the latest information from fashion media.

[0063] In order to filter out some strongly correlated feature parameters from the full amount of inference feature parameters, the feature screening module is used here to filter the feature parameters. There are three main purposes for determining strongly correlated feature parameters:

[0064] First, a small number of strongly correlated feature parameters are selected to fine-tune the CLIP model and strengthen the semantic alignment of images and texts;

[0065] Second, reduce the training time, computing power, and human cost of model fine-tuning;

[0066] Third, filter out the influence of other non-strongly correlated feature parameters on the output effect.

[0067] Therefore, the present invention first performs dimensionality reduction processing on all feature parameters, and extracts strongly correlated feature parameters as trend feature templates.

[0068] The present invention uses the Transformer architecture to perform dimensionality reduction processing.

[0069] The Transformer model can automatically focus on the most important features through the multi-head attention mechanism. In order to extract a small number of key features, feature importance analysis is performed as follows:

[0070] Based on Attention Weights analysis: The attention mechanism in the Transformer model can tell the model which features to focus on when predicting. Then, the attention weights of each feature can be extracted, and important features can be selected based on these weights.

[0071] The SHAP model is then used to further interpret the model’s output and identify which features have the greatest impact on the prediction of popular trends. The SHAP value can be used to quantify the contribution of each feature to the model’s prediction and rank the features according to their importance.

[0072] TabTransformer is a Transformer model suitable for structured data, focusing on the most important features for target prediction. In this way, TabTransformer can handle high-dimensional structured data, as well as classification and regression tasks of structured data.

[0073] After analyzing the Attention weight or SHAP value, a small number of the most representative key features can be identified. Finally, the following nine important features are identified:

[0074] Characters: Posture;

[0075] Clothing: brand, style, clothing style;

[0076] Aesthetics: texture, color, tone;

[0077] Scene: style, background.

[0078] These features will be used to further analyze and optimize the CLIP model, and can also be used as key training parameters and prompt words for the generative model.

[0079] In addition, the present invention also introduces a mechanism for dynamically adjusting trend feature templates, which can dynamically screen and adjust parameters of trend feature templates according to user intentions and enterprise needs.

[0080] (3) Fine-tune CLIP based on trend feature templates to enhance cross-modal understanding capabilities and achieve semantic alignment of images and text.

[0081] Fine-tuning the CLIP model based on the selected trend feature template has two purposes and benefits:

[0082] A. Iteratively execute the CLIP model to revise the CLIP model's inaccurate understanding of the trend feature template parameters, such as incorrect or incomprehensible annotations in the template. The optimization can be completed by executing step (3);

[0083] B. Complete the downstream workflow, i.e., perform the precise reasoning of step (4).

[0084] In this method, considering the large size of parameters and models, in order to reduce the computational cost, frozen pre-trained layers are used for fine-tuning, keeping the parameters of most layers of the pre-trained model unchanged, and only updating a small number of task-specific layers, making the training process more efficient and reducing the risk of overfitting.

[0085] In this process, the usual steps include: loading a pre-trained model (such as a CLIP model) and keeping its main encoding layer frozen; adding a task-specific output layer (such as a classification head or a regression head) for the new task; training the model, updating only the parameters of the task-specific layers, and freezing the pre-trained part.

[0086] Next, the process of freezing the pre-trained layers for fine-tuning is described in detail.

[0087] (3.1) Establish a fine-tuning model structure. Assume that the CLIP model is used to solve the image classification problem; the CLIP model consists of the following parts:

[0088] Image Encoder , maps the input image x to a feature vector.

[0089] Text Encoder , mapping the input text t into a feature vector.

[0090] The two encoders are pre-trained to keep semantically related image and text features close to each other in the same feature space.

[0091] For the fine-tuning process, only the image encoder is operated. CLIP can be regarded as a feature extractor, its weights are fixed, and only the output layer is fine-tuned on the new task.

[0092] (3.2) Freeze the pre-trained layers. Freezing the pre-trained part of the model means not updating the parameters of this part during back-propagation. This is achieved by setting the requires_grad attribute of these layers to False in the deep learning framework. Specifically, suppose the pre-trained image encoder parameters are , these parameters will be frozen and only the newly added task layer parameters will be To update:

[0093]

[0094]

[0095] in, represents the frozen image encoder parameters, represents the added task-specific layer, Represents a constant, i.e. a frozen part The gradient of the frozen parameters does not participate in the gradient update during the training process. , Indicates the updated part, the parameters of the task layer It is constantly updated during the training process, and the parameter update is guided by calculating the loss function.

[0096] One benefit of freezing pre-trained layers is that the computational cost is greatly reduced, since most of the gradient calculations and parameter updates are concentrated on the task-specific layers.

[0097] (3.3) Fine-tune the task-specific layer. Usually, the task-specific layer is a new fully connected layer or classification head to solve a specific task. For example, for a classification task, add a new classification layer (logistic regression layer):

[0098]

[0099] in, is the output feature vector of the pre-trained image encoder (frozen encoder layer), W is the weight matrix of the new task layer, which needs to be learned through fine-tuning; b is the bias term, which needs to be learned through fine-tuning; y is the prediction result, that is, the output of the model, which is used for classification or other tasks.

[0100] (3.4) Loss Function. For classification tasks, the commonly used loss function is cross-entropy loss, which is used to compare the predicted result y and the true label. , the cross entropy loss function is defined as:

[0101]

[0102] Where N is the number of training samples, C is the number of categories, Indicates the true label (one-hot encoding) that the i-th sample belongs to category c, is the probability that the model predicts that the i-th sample belongs to category c, calculated by softmax:

[0103]

[0104] in, Represents the original value of the model result, that is, the original value of the i-th sample belonging to category c.

[0105] At this time, only the parameters W and b of the task-specific layer need to be back-propagated and updated, while the frozen image encoder part It will not participate in gradient calculation.

[0106] (3.5) Back propagation and gradient update. After calculating the loss through forward propagation, back propagation is performed to calculate the gradient of the loss with respect to the parameters W and b of the task-specific layer:

[0107]

[0108] Then use gradient descent or other optimization algorithms such as Adam to update the parameters of the task-specific layers:

[0109]

[0110]

[0111] where η is the learning rate.

[0112] Since the pre-trained image encoder is frozen, so its parameters This method only updates the weights W and bias b of the classification layer until the model reaches optimal performance on the task-specific data.

[0113] In this step, the complete training process of fine-tuning is as follows:

[0114] Load pre-trained model: Load the pre-trained CLIP model, especially its image encoder part;

[0115] Freeze encoder parameters: Freeze the image encoder parameters Freeze so that its parameters do not participate in training;

[0116] Add task-specific layers: Add a new output layer (fully connected layer) for the target task (such as classification task);

[0117] Define the loss function: Use the cross entropy loss function to measure the gap between the predicted results and the true labels;

[0118] Train the new layer: Update only the parameters W and b of the new layer through back propagation;

[0119] Evaluation and fine-tuning: Evaluate the model on the validation set and adjust hyperparameters such as learning rate until the model achieves optimal performance on the new task.

[0120] (4) Use the fine-tuned CLIP model to annotate the images of the reference target dataset and obtain an optimized feature parameter description, i.e., the image-text data pair.

[0121] The CLIP operation is performed again on the reference target image data set to obtain further optimized image-text data pairs, so that the image-text data pairs annotated by executing step (4) are more accurate than the full amount of data annotated by executing step (1).

[0122] Traditional data mining and natural language processing (NLP) rely only on text data and often ignore other modal information such as images. Especially when it comes to data such as clothing design styles, commercial photos, social media materials, etc., images can often provide rich context, emotions and visual clues.

[0123] CLIP maps images and text into the same space, making it possible to directly compare the similarity of images and text. This means that CLIP can be used to jointly analyze image data with corresponding text data to identify complex patterns and relationships.

[0124] Data analysis: After data processing of the reference target image dataset, such as word frequency analysis, clustering and other algorithms, the patterns and relationships of this batch of image datasets of interest can be intuitively derived, which can objectively restore the authenticity and richness, and can fully express user preferences and clothing trends.

[0125] See also Figure 2 ,According to the word cloud analysis of the image dataset, it can be concluded that:,cyclical changes; black and brown are more common in winter,,white and blue are more common in summer; the top three tags are long hair, upper body, and half body shots.

[0126] Other, according to Figure 3 The trend segmentation feature decomposition shows that the features of this batch of image datasets focus on:

[0127] Clothing styles: pants, jeans, dresses;

[0128] Clothing style: minimalist style, Korean style, sports style;

[0129] Clothing colors: white, black, beige;

[0130] Body Parts: Upper Body;

[0131] Character details: visible shoulders, handbag, red nail polish;

[0132] Shooting composition: cowboy shot, close-up.

[0133] (5) Based on the image-text data pairs, the diffusion model is fine-tuned for the text-generated image type and the image-generated image type, and image generation is performed to obtain a clothing trend image.

[0134] The goal of the clothing trend image generation task is to obtain high-quality images that meet expectations. The present invention uses different fine-tuning strategies for the two application scenarios of text-generated images and image-generated images to make them perform better in specific tasks. In addition, a similarity control function is introduced in the generation task to achieve the degree of difference between the generated image and the input image in style, content or structure.

[0135] Text-to-image generation: This method is more free-flowing and uses prompt tuning. This method does not require major changes to the internal structure and parameters of large-scale models. It only needs to optimize the input and can easily adjust the generation style and details. It is especially suitable for text-to-image generation tasks.

[0136] Image generation: biased towards a specific reference, using an adapter layer. This method does not significantly modify the main parameters of the model, but optimizes for a specific task. During fine-tuning, only a small number of parameters are updated to reduce resource overhead while maintaining the stability of the main model. The adapter layer can be flexibly designed according to the needs of the task, adapting to a variety of image styles or structures in the generation task.

[0137] (5.1) Prompt Tuning

[0138] Prompt Tuning is used to efficiently fine-tune the parameters of large-scale pre-trained models. It is a lightweight fine-tuning method that can be used in text-to-image generation model scenarios to optimize the generation effect. Prompt Tuning freezes all parameters of the diffusion model. The input text prompt words are converted into vector embeddings, and then these embeddings are fine-tuned on the training data to adapt to specific image generation requirements. In this way, the fine-tuned prompt embedding allows the model to better generate images with specific styles, content, or details. It includes the following sub-steps:

[0139] (5.1.1) Dataset preparation.

[0140] The training data set is ,in is a text description (prompt), and is the corresponding target image.

[0141] (5.1.2) Model selection.

[0142] The pre-trained image generation model is M, which is usually a model based on a stable diffusion architecture and can generate images based on the input text.

[0143] (5.1.3) Input Embedding Layer.

[0144] Determine the structure of the input embedding layer E, usually the text input Transformed into vector representation through embedding layer E :

[0145]

[0146] (5.1.4) Fine-tune Prompt Embedding.

[0147] During the training process, the parameter θ of the diffusion model M is kept frozen, and the embedded representation of prompt is fine-tuned. The present invention uses a trainable prompt embedding as p, which is finally input into the embedded representation of the diffusion model. for:

[0148]

[0149] Among them, p is a trainable vector obtained through learning.

[0150] The conditional probability of the diffusion model M generating the target image y is determined by the text description t, so:

[0151]

[0152] This represents a textual description of the given input , the probability that the model generates the target image.

[0153] (5.1.5) Loss function.

[0154] In order to optimize the diffusion model so that the generated image is as close to the real image as possible, some form of loss function is usually used. Common loss functions include mean square error (MSE) or adversarial loss. Assume that mean square error is used as the loss function:

[0155]

[0156] in, represents the L2 norm (Euclidean distance), It’s an overall loss.

[0157] (5.1.6) Optimization process.

[0158] Use gradient descent to update the prompt embedding p. In each iteration, calculate the gradient of the loss function and update p:

[0159]

[0160] Where η is the learning rate.

[0161] (5.1.7) Model generation process.

[0162] After fine-tuning, the optimized prompt embedding is used to generate images. Given a new text prompt , first calculate its embedding :

[0163]

[0164] Then add it to the optimized p to get the final input :

[0165]

[0166] Finally, Input diffusion model for image generation:

[0167]

[0168] in, is the generated image.

[0169] (5.2) Use Adapter Layers to generate image references.

[0170] In the image-to-image generation task, the use of Adapter Layers fine-tuning scheme can effectively combine text and image information to complete the task. Adapter Layers inserts lightweight adapter modules between certain layers of the diffusion model, keeping most of the parameters of the diffusion model frozen, which is convenient for fine-tuning for specific tasks. At the same time, the present invention has made further improvements to the generation task, and the reference similarity can be set to control the generation of similar images. Specifically, it includes the following sub-steps:

[0171] (5.2.1) Dataset preparation.

[0172] The dataset is:

[0173]

[0174] in, is the input image; is the same as the input image The associated text description (prompt); is the target output image, which represents the similar image we want to generate.

[0175] (5.2.2) Model selection.

[0176] Select a pre-trained image generation model M based on the stable diffusion generative model that is able to handle both image and text inputs.

[0177] (5.2.3) Input representation.

[0178] The input image and text description Through the image feature extraction layer and text feature extraction layer To encode:

[0179] Image encoding:

[0180]

[0181] Text encoding:

[0182]

[0183] in, and It is the encoded image feature representation and text feature representation.

[0184] (5.2.4) Application of Adapter Layers.

[0185] In the diffusion model, insert Adapter Layers. Assume that a certain layer in the model is where the present invention chooses to insert the adapter. The adapter is usually a simple feedforward neural network, which can be expressed as:

[0186]

[0187] Among them, W1 and W2 are the weights of the adapter, and b1 and b2 are bias terms. In this way, the adapter transforms the input features and outputs more adaptable feature representations.

[0188] Combine the feature representations of the image and text through the adapter to obtain the final feature representation :

[0189]

[0190] Among them, a and Represents an adapter whose input parameters vary.

[0191] (5.2.5) The conditional probability of generating similar images.

[0192] The diffusion model M generates the conditional probability of the target image y based on the feature representation processed by the adapter , which can be expressed as:

[0193]

[0194] in, It is the final feature representation that combines image and text information.

[0195] (5.2.6) Loss function.

[0196] In order to optimize the model, the generated images As close as possible to the target output image , usually some kind of loss function is used, such as mean square error (MSE) or perceptual loss. Assuming the mean square error is used:

[0197]

[0198] (5.2.7) Optimization process.

[0199] Use gradient descent to update the weights in the Adapter Layers. In each iteration, the gradient of the loss function is calculated and the weights of the adapter are updated:

[0200]

[0201] Where η is the learning rate.

[0202] (5.2.8) Image generation.

[0203] After fine-tuning, given a new image and text description , firstly encoded through the feature extraction layer:

[0204]

[0205]

[0206] in, and Represents new image feature representation and new text feature representation.

[0207] Then they are passed through the adapter layer to get the new final feature representation:

[0208]

[0209] Finally, this final feature is input into the generative model M for image generation:

[0210]

[0211] in, is the generated clothing trend image.

[0212] (5.3) Similarity control.

[0213] In many application scenarios, it is necessary to control the difference between the generated image and the input image in style, content or structure, such as closer or more different. In this case, the similarity between the generated image and the input image can be controlled by setting the similarity. The specific method is as follows:

[0214] (5.3.1) Introduce the loss function of similarity control.

[0215] In order to control the similarity between the generated image and the input image, a similarity control term can be added to the loss function of the model. Similarity control methods include perceptual loss and structural similarity index (SSIM), which can be selected according to actual needs. Here, perceptual loss is selected because we want to control the similarity of high-level features. According to the above definition, the input image is , the generated clothing trend image is , define the perceptual loss :

[0216]

[0217] in, Represents the feature extraction operation of the pre-trained network. By adjusting the weight of this loss, you can control the high-level semantic similarity of the generated image. Larger weights produce outputs that are more similar to the input image, while smaller weights focus more on the diversity of generation.

[0218] (5.3.2) Introduce the similarity control factor α.

[0219] A similarity control factor α is added to the loss function, which is used to balance the similarity between the generated image and the input image. By adjusting the value of α, the similarity of the generated image can be flexibly adjusted.

[0220]

[0221] in, It is the basic loss function of the generation task, that is, the loss between the generated image and the target image; is the similarity control loss; α represents the weight controlling the similarity, a higher α value generates an image that is more similar to the input image, while a lower α value allows the generated image to be more different from the input image.

[0222] (5.3.3) Similarity Control in Adapter Layers.

[0223] The loss function with the similarity control factor is introduced into the fine-tuning process of Adapter Layers, and the adapter weights are updated through gradients to achieve the similarity control effect.

[0224] (5.3.4) Similarity control during the final generation process.

[0225] After the diffusion model is fine-tuned, the similarity of the generated images can be dynamically controlled by adjusting the value of α during the generation process, making the generation process more flexible.

[0226] (6) An evaluation mechanism is introduced to evaluate the effectiveness of the generated clothing trend images.

[0227] After generating an image, how to evaluate the quality of the image generation is a key step to ensure the output effect of the model. For image generation tasks, the evaluation can be performed from multiple dimensions, including structural similarity, image quality, industry attributes, etc. To this end, the present invention combines clothing generation images and uses multiple indicator dimensions for professional evaluation.

[0228] SSIM: Compares the structural similarity of the generated clothing design with the real design to ensure that the generated image is consistent with the design requirements in visual details;

[0229] FID: By calculating the distance between real clothing images and generated images in feature space, it ensures that the overall quality and diversity of generated images are consistent with the distribution of real clothing images;

[0230] IS: Evaluate the diversity of generated images and their relevance to recognizable clothing categories to ensure that the generated designs are not only accurate but also cover a wide range of styles.

[0231] (6.1) SSIM evaluation.

[0232] SSIM is a metric used to measure the similarity between two images, focusing on the structure, brightness, and contrast of the images.

[0233] Luminance: measures the average brightness of an image;

[0234] Contrast: measures the contrast changes of an image;

[0235] Structure: Measures the texture information and local structure of an image.

[0236] The SSIM index can be used to quantitatively evaluate the similarity between generated images and real images. In the field of clothing design, SSIM can effectively reflect the visual accuracy and consistency of generated images.

[0237] In the scenario of clothing image generation, the generated clothing image g and the real design image r can be used to perform SSIM calculation to analyze the structural similarity between the generated image g and the real image r.

[0238] The process of calculating SSIM is as follows.

[0239] Calculate the brightness L:

[0240]

[0241] in and is the average brightness of the two images in the window, and C1 is a small constant used to stabilize the denominator.

[0242] Calculate the contrast C:

[0243]

[0244] in and is the contrast (standard deviation) of the two images in the window, and C2 is a small constant.

[0245] Calculate the structure S:

[0246]

[0247] in is the covariance of the two images, and C3 is another small constant.

[0248] Calculate the SSIM value for each pair of images:

[0249]

[0250] Evaluation results: The performance of the generated model is evaluated based on the output SSIM value. Usually, the SSIM value is between 0 and 1. The closer the value is to 1, the more similar the generated image is to the real image in structure, indicating that the generated model works well.

[0251] (6.2) FID evaluation.

[0252] FID (Frechet Inception Distance) can be used to evaluate the generation effects of diffusion models and transformer models. It can also evaluate the quality and diversity of generated images by quantitatively comparing the output performance of the generated models.

[0253] Use FID to evaluate the steps:

[0254] Extract features: Input the real image and the generated image into the pre-trained Inception model, extract the features of the intermediate layer (usually use the output of a certain layer, such as the last fully connected layer), and record the mean and covariance of the generated image features and the real image features.

[0255] The extracted features are used to calculate the FID value.

[0256] First calculate the mean and covariance:

[0257] Calculate the mean of the real image features and covariance ;

[0258] Calculate the mean of the generated image features and covariance ;

[0259] Then apply the FID formula to calculate the FID value:

[0260]

[0261] in, is the distance between the means, is the trace of the covariance matrix.

[0262] Evaluation results: The FID value is usually between 0 and 100. The lower the FID value, the closer the quality of the generated image is to the distribution of real images. Generally speaking, an FID value below 10 is considered to be a high-quality generation.

[0263] (6.3) IS evaluation.

[0264] Inception Score is an indicator for evaluating the recognizability and diversity of generated images. The higher the IS value, the more diverse the generated images are and the more relevant they are to recognizable categories.

[0265] Image recognizability: whether the generated image can be clearly associated with a certain type of label, that is, whether the category of the image is clear and consistent;

[0266] Image diversity: Are the generated images diverse enough and cover a variety of styles or designs?

[0267] First, use the fine-tuned image diffusion model to generate clothing images, and input descriptions such as "winter black jacket", "casual style skirt", etc. The generated clothing image set is input into the pre-trained Inception v3 network. This network outputs the category prediction probability distribution of each image (that is, the output of the classifier). For each generated image x (x is used as the input of this step), calculate its classification probability distribution p(y|x), and then calculate the conditional entropy of the image:

[0268]

[0269] This measures how well the generated image matches a certain class label. The smaller the conditional entropy, the more clearly the generated image matches a certain class label.

[0270] Then, calculate the category probability distribution p(y) of all generated images, that is, the average value of the category distribution of all generated images:

[0271]

[0272] Here N is the number of generated images.

[0273] Finally, calculate the value of Inception Score :

[0274]

[0275] Among them, p(y|x) is the categorical distribution of the generated image x (that is, the probability distribution of the predicted category); p(y) is the marginal distribution of the categories of all generated images; KL stands for Kullback-Leibler divergence, which measures the difference between two probability distributions.

[0276] By calculating the KL divergence of the conditional entropy of each generated image and the distribution of all image categories, an Inception Score value is finally obtained. The IS value can be understood as follows:

[0277] A low IS value indicates that the generated image is either too monotonous or does not conform to common clothing category labels. The style or design may not be clear enough, or the generated image quality is poor.

[0278] A higher IS value indicates that the generated clothing images have strong diversity and recognizability, and conform to more common clothing categories (such as T-shirts, windbreakers, skirts, pants, etc.).

[0279] The method of the present invention is not only applicable to the trend prediction and product launch testing of clothing products, but can also be extended to more vertical industries or products that require visual trend prediction, as well as scenarios related to product design, testing, marketing, etc., such as e-commerce, automobiles, cultural and creative products, etc., and has a certain universality.

[0280] Furthermore, in the process of determining the strongly correlated feature parameters, different processing methods can be selected according to the actual situation and task characteristics.

[0281] 1. Feature extraction method: replace the TabTransformer model with other models or methods;

[0282] 2. Feature parameters: Redefine the importance of feature variables based on requirements and task characteristics, and select different sets of feature parameters.

[0283] Corresponding to the above-mentioned embodiment of the multimodal fashion trend prediction and evaluation method, the present invention also provides an embodiment of the multimodal fashion trend prediction and evaluation device.

[0284] See also Figure 4 The multimodal fashion trend prediction and evaluation device provided in the embodiment of the present invention includes one or more processors for implementing the multimodal fashion trend prediction and evaluation method in the above embodiment.

[0285] The embodiments of the multimodal fashion trend prediction and evaluation device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capabilities reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability where the multimodal fashion trend prediction and evaluation device of the present invention is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0286] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0287] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0288] Corresponding to the embodiment of the aforementioned multimodal fashion trend prediction and evaluation method, the embodiment of the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal fashion trend prediction and evaluation method as described above. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability where the multimodal fashion trend prediction and evaluation method provided in the embodiment of the present application is located, except Figure 5 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, which will not be described in detail, generally based on the actual functions of the device with data processing capabilities.

[0289] Corresponding to the embodiment of the aforementioned multimodal fashion trend prediction and evaluation method, an embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the multimodal fashion trend prediction and evaluation method in the aforementioned embodiment.

[0290] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0291] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0292] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A multimodal fashion trend prediction and evaluation method, characterized in that: The following steps are involved: (1) Based on the established full-volume trend feature parameters, the reference target image is labeled to obtain a full-volume feature parameter description; (2) Use the Transformer model to reduce the dimension of all feature parameters, extract strongly correlated feature parameters, and form a trend feature template; (3) Fine-tune CLIP based on trend feature templates to enhance cross-modal understanding capabilities and achieve semantic alignment of images and texts; (3.1) Establish a fine-tuned CLIP model structure, including an image encoder and a text encoder. These two encoders are pre-trained to make semantically related image and text features close in the same feature space. (3.2) Freeze encoder parameters: Freeze the parameters of the image encoder so that its parameters do not participate in training; (3.3) Add task-specific layers: Add a new output layer for the target task and perform fine-tuning learning; (3.4) Define the loss function: Use the cross entropy loss function to measure the gap between the predicted result and the true label; (3.5) Update only the parameters of the task-specific layers through back-propagation; evaluate the model on the validation set and adjust the hyperparameters until the model achieves the best performance on the task-specific layers; (4) Use the fine-tuned CLIP model to annotate the reference target image dataset and obtain an optimized feature parameter description, i.e., an image-text data pair; (5) Based on the image-text data pairs, the diffusion model is fine-tuned for the text-generated image type and the image-generated image type, and image generation is performed to obtain a clothing trend image; (5.1) By fine-tuning the prompts, freeze all parameters of the diffusion model, convert the input text prompts into vector embeddings, and then fine-tune these embeddings on the training data to adapt to the image generation requirements; (5.2) Insert a lightweight adapter into the diffusion model, keep the parameters of the diffusion model frozen, and perform fine-tuning for specific tasks to generate clothing trend images; (5.3) Introduce a similarity-controlled loss function; add a similarity control factor to the loss function to balance the similarity between the generated image and the input image; introduce the loss function with the similarity control factor into the fine-tuning process of the adapter, and update the adapter weights through the gradient to achieve similarity control between the generated image and the input image; (6) An evaluation mechanism is introduced to evaluate the effectiveness of the generated clothing trend images.

2. The multimodal fashion trend prediction and evaluation method according to claim 1, characterized in that: In the step (2), the Transformer model is used to perform attention weight analysis and SHAP value analysis on all feature parameters; the attention weight of each feature is extracted through attention weight analysis, and important features are selected based on these weights; The SHAP value is used to quantify the contribution of each feature to the model prediction and to rank the features according to their importance.

3. The multimodal fashion trend prediction and evaluation method according to claim 1, characterized in that: In the step (4), the reference target image data set is processed to obtain the patterns and relationships of the batch of image data sets of interest, reflecting user preferences and clothing fashion trends.

4. The multimodal fashion trend prediction and evaluation method according to claim 1, characterized in that: In the step (6), the generated clothing trend image is evaluated using the SSIM, FID and IS indicator dimensions; wherein: The SSIM indicator dimension is used to compare the structural similarity between the generated image and the real clothing image to ensure that the generated image is consistent with the design requirements in visual details; The FID indicator dimension is used to ensure that the overall quality and diversity of the generated images are consistent with the distribution of real clothing images by calculating the distance between the real clothing images and the generated images in the feature space; The IS indicator dimension is used to evaluate the diversity of generated images and their relevance to the identified clothing categories, ensuring that the generated designs are accurate and cover a variety of styles.

5. A multimodal fashion trend prediction and evaluation device, characterized in that: The method comprises one or more processors for implementing the multimodal fashion trend prediction and evaluation method according to any one of claims 1 to 4.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multimodal fashion trend prediction and evaluation method described in any one of claims 1-4 above.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal fashion trend prediction and evaluation method as described in any one of claims 1 to 4 is implemented.

Images