Personalized retrieval type clothing recommendation method and system based on historical data perception

Abstract

The invention discloses a personalized retrieval type clothing recommendation method and system based on historical data perception. The method comprises the following steps: 1, acquiring historical clothing data and single-item multi-modal information of a user; 2, extracting image and text features through multi-modal information of a CLIP encoder, and fusing the image and text features; 3, performing feature extraction and fusion on the single-item image and the text description in the step 1 to generate a single-item multi-modal representation vector; 4, inputting the multi-modal representation vector of each single item in the garment sequence to be evaluated into a Transform encoder, and outputting a compatibility embedded vector representing overall matching; 5, constructing a joint loss function, and optimizing compatibility prediction and personalized recommendation targets; 6, balancing compatibility and personalization by adopting two-stage training; and 7, performing answer prediction based on the distance between the compatibility embedding vectors to calculate the accuracy, and calculating a personalized matching score based on the correlation between the compatibility embedding vector of the predicted answer and the historical latent variable of the user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and fashion computing, and specifically relates to an intelligent clothing recommendation method and system fusing multi-modal feature learning and user history modeling, which accurately and efficiently realizes clothing compatibility recommendation based on user personalized preferences, and improves the practicality and efficiency of clothing recommendation. BACKGROUND

[0002] With the rapid development of e-commerce and the fashion industry, online clothing shopping has become an indispensable part of people's daily life. However, in the face of a large number of clothing goods, how users quickly find goods that not only meet their personal aesthetic preferences but also coordinate with other clothing items has become an important challenge for current clothing recommendation systems. Traditional clothing recommendation methods are mainly based on collaborative filtering and content filtering technology. Although these methods have achieved certain results in product similarity calculation, they have obvious shortcomings in dealing with the complex semantic relationship of clothing matching and user personalized preferences.

[0003] The existing clothing recommendation technology mainly has the following technical bottlenecks: First, most methods only focus on single product feature matching, ignoring the overall compatibility of clothing matching. In real clothing matching scenarios, users often need to build complete clothing combinations, not just choose single products. Second, traditional recommendation systems are difficult to effectively integrate multi-modal information. Clothing goods contain both visual appearance and text description, two important information sources. How to organically integrate image features and semantic features to form a unified product representation is the key to improving recommendation quality. Third, existing methods model user historical preferences simply, usually based on explicit ratings or click behavior, and cannot deeply mine user style preferences and aesthetic tendencies. Finally, compatibility prediction and personalized recommendation are often considered as two independent tasks, lacking a unified optimization framework to balance the two goals.

[0004] In recent years, deep learning technology has made significant progress in computer vision and natural language processing, providing a new technical path for the development of clothing recommendation systems. In particular, the advantages of the Transformer architecture in sequence modeling, and the breakthroughs of cross-modal pre-training models such as CLIP in visual-linguistic alignment, have laid a technical foundation for building more advanced clothing recommendation systems. However, how to effectively integrate these technologies to build a unified framework that can handle both compatibility prediction and personalized recommendation remains a technical challenge in this field.

[0005] In the aspect of clothing compatibility prediction, existing methods mainly use distance-based metrics to calculate the compatibility between clothing items, but often ignore the influence of user historical preferences on the selection of combinations. In the aspect of personalized recommendation, although methods based on user historical behavior can capture long-term interests, they cannot guarantee the visual coordination and aesthetic consistency of the recommended results. This inherent contradiction between compatibility and personalization limits the practical application effect of existing recommendation systems.

[0006] In addition, there is still room for optimization in the training strategy of existing clothing recommendation systems. Most methods use an end-to-end single-stage training approach, which makes it difficult to balance the optimization conflicts between different task objectives. In particular, when dealing with complex tasks such as FITB (fill-in-the-blank) that require simultaneous consideration of compatibility and personalization, a single training objective often cannot achieve ideal results.

[0007] After searching, the following Chinese granted invention patents were found: CN114202396B, an interactive clothing recommendation method and system, CN119515498B, a multi-modal personalized clothing recommendation method and system based on single-item compatibility, CN111915400B, a personalized clothing recommendation method and device based on deep learning, CN113850656B, a personalized clothing recommendation method and system based on attention perception by fusing multi-modal data, CN120146972B, an artificial intelligence-based virtual fitting personalized clothing recommendation method. Although these granted invention patents have their own advantages, there is still a balance problem between compatibility prediction and personalized recommendation. After experimental comparison and analysis, it was found that the above patents still have obvious limitations in core performance indicators: the FITB task compatibility accuracy is generally below 70%, and the personalized score mostly stays in the 40-50 interval, making it difficult to break through the balance bottleneck between compatibility prediction and personalized recommendation. In particular, it is a technical problem that existing technologies have not effectively solved to achieve a personalized score of more than 55% while maintaining high compatibility requirements. SUMMARY

[0008] To solve the above technical problems of existing technologies, the present application proposes a personalized retrieval clothing recommendation method and system based on historical data perception. Through innovative multi-modal feature fusion mechanisms, user history modeling methods, and two-stage training strategies, the present application effectively solves the balance problem between compatibility prediction and personalized recommendation, providing a new technical solution for the development of clothing recommendation systems.

[0009] The application provides a smart clothing recommendation method and system combining multi-modal learning and user history modeling, aiming to solve the core technical challenges in the field of clothing recommendation, including difficulties in aligning visual and semantic features, inaccurate capture of long-term user preferences, and conflicts between compatibility and personalized needs. Through innovative model architecture and training strategy design, the application can stably generate clothing recommendation results that meet aesthetic standards and user preferences in complex real-world application environments, not only performing excellently in FITB task accuracy, but also having good personalized matching degree, suitable for commercial deployment of large-scale online clothing recommendation platforms.

[0010] The application adopts the following technical solutions: The personalized retrieval-based clothing recommendation method based on historical data perception comprises the following steps: Step one, multi-source clothing data set loading and preprocessing: obtaining user historical clothing data and single product multi-modal information, the multi-modal information including single product images and corresponding text descriptions; Step two, parallel extraction of multi-modal deep features: feature extraction and fusion of single product images and text descriptions in step one to generate single product multi-modal representation vectors; Step three, user history latent variable construction: based on user historical clothing sequences and corresponding single product multi-modal representation vectors, generating user history latent variables representing user style preferences through aggregation operation; Step four, clothing compatibility embedding learning: inputting the multi-modal representation vectors of each single product in the clothing sequence to be evaluated into a Transformer encoder, learning the compatibility relationship through a self-attention mechanism, and outputting a compatibility embedding vector representing the overall combination; Step five, dual-target joint optimization training: constructing a joint loss function, the first target of the function being to reduce the distance between the compatibility embedding vectors of the predicted answer and the query clothing, and the second target being to improve the correlation between the compatibility embedding vectors of the predicted answer and the user history latent variables, so as to simultaneously optimize the compatibility prediction and personalized recommendation targets; Step six, progressive two-stage training strategy: balancing compatibility and personalization by adopting a two-stage training, in the first stage mainly optimizing the first target of the joint loss function, and in the second stage gradually introducing and enhancing the optimization weight of the second target; Step seven, multi-dimensional evaluation output: in the FITB task, predicting the answer based on the distance between the compatibility embedding vectors to calculate the accuracy, and calculating the personalized score based on the correlation between the compatibility embedding vectors of the predicted answer and the user history latent variables.

[0011] Furthermore, in step one, the loading and preprocessing of the multi-source clothing dataset is achieved through the following methods: loading the mapping relationship data between user IDs and clothing sets to associate the clothing combinations that the user interacted with in the past, and reading the individual item image file paths and corresponding text description data to form complete multimodal information of individual items; the data adopts a structured organization method that includes training sets, validation sets, and test sets, and processes the triplet data composed of query clothing combinations, answer items, and user historical information by constructing a special data class; in the data preprocessing stage, uniform size adjustment and normalization operations are performed on the clothing images, and word segmentation is performed on the text descriptions to generate fixed-length word sequences; to quickly construct user representations, the system directly loads pre-computed historical latent variable files; at the same time, by constructing an image dataset and initializing blank images as default resources, the features of the blank images are automatically called as default historical latent variables when user historical data is missing, ensuring the robustness of the system under incomplete data conditions.

[0012] Furthermore, in step one, the input sample set is obtained as follows: ,in, Represents clothing images, This represents the corresponding text description. N The batch size is H×W=224×224, which represents the image size. L = 77 is the length of the text sequence.

[0013] Furthermore, in step two, the parallel extraction of multimodal deep features is achieved as follows: the image encoder of the contrastive language-image pre-trained (CLIP) model is used to process the clothing image, with an input size of 3×224×224 and an output of 512-dimensional visual feature vector; the product description text is processed by the CLIP text encoder, with an input of 77-dimensional token representation sequence and an output of 512-dimensional semantic feature vector; the visual and text features are concatenated to generate a 1024-dimensional unified representation; and the fused features are subjected to L2 normalization.

[0014] Furthermore, in step two, the parallel extraction of multimodal deep features is implemented through a dual-branch encoder architecture, the specific technical details of which are as follows: Image feature extraction via visual encoder accomplish: in, Image preprocessing functions, including resizing and normalization operations. This is the output image feature vector.

[0015] Text feature extraction via text encoder accomplish: in, For text tokenization functions, This is the output text feature vector.

[0016] Multimodal feature fusion employs a splicing and aggregation strategy: in, This represents a vector concatenation operation. This is the fused multimodal feature vector.

[0017] Feature normalization employs L2 normalization, which is used to normalize the fused feature vector e. i Mapped to the unit hypersphere: in, For vector e i The L2 norm of the vector is calculated as the square root of the sum of the squares of all dimensions of the vector. in, Represents vector e i The scalar value in the j-th dimension. The normalized feature vector ensures that all features are distributed on the unit hypersphere.

[0018] Throughout the feature extraction process, the visual encoder and text encoder The parameters are kept frozen, and the cross-modal semantic alignment capability obtained by pre-training is used to ensure the stability and generalization of feature extraction.

[0019] Furthermore, in step three, the construction of the user's historical latent variables includes: Historical clothing data is aggregated by user ID, with each user having more than 21 clothing categories; latent style preference variables are generated based on the user's historical clothing items, with each category containing 10 items; historical items are encoded using a CLIP encoder to generate historical representation vectors of size [4, 64, 64]; for users with missing historical data, the latent variable corresponding to null_img is used as the default value.

[0020] Furthermore, in step three, the construction of the user's historical latent variables adopts a category-based hierarchical aggregation strategy: For each user u and categories c The historical latent variables are calculated as follows: in, Indicates user u In category c A collection of historical costumes, e i For the first i Multimodal embedding of individual clothing items Aggregate The function uses mean pooling. The global user history representation is obtained by concatenating all category latent variables: .

[0021] Furthermore, in step three, the construction of the user's historical latent variables is achieved through a hierarchical user modeling method, the specific technical details of which are as follows: Let the user set be ,in, M Let u be the total number of users. For each user u k Its historical interaction data is represented as , where c j Indicates clothing category, This is a collection of historical clothing items within this category. Indicates user u k The number of categories, N j =10 indicates the number of items in each category. The category-level historical mean vector is calculated using mean pooling: in, Let x be the feature mapping function. l For the first l Multimodal features of an individual product Indicates user u k In category c j Historical preference representation under the current circumstances.

[0022] User global historical latent variables are generated through tensor reconstruction: in, The tensor reconstruction function organizes multiple category representations into a three-dimensional latent variable tensor.

[0023] Missing data handling uses a default latent variable mechanism: in, Represents a set of users with historical records. These are latent variables generated based on the default image.

[0024] The personalized style representation is ultimately obtained through linear transformation: in, Let b be the projection weight matrix, and flatten be the tensor flattening operation. p The bias vector of the linear projection layer. This serves as the final user-personalized style representation vector, used for subsequent personalized recommendation calculations.

[0025] Furthermore, in step four, the clothing compatibility embedding learning is implemented in the following ways: the clothing sequence is modeled using a Transformer encoder, which includes 6 encoder layers and a 16-head self-attention mechanism; the matching compatibility relationship between clothing items is learned through the self-attention mechanism; a 128-dimensional compatibility embedding vector is generated; and regularization is performed using LayerNorm and Dropout=0.3.

[0026] Furthermore, in step four, the clothing compatibility embedding learning is implemented through a Transformer-based sequence encoding architecture, with specific technical details as follows: Let the clothing sequence input be... Where L is the sequence length, Let be the multimodal feature vector of the i-th clothing item.

[0027] The input projection layer first maps the features to the model dimensions: in, For projection parameters, d model This represents the dimension of the model's hidden layers.

[0028] Multi-layer Transformer coding consists of N=6 stacked coding layers, each layer containing the following computations: Multi-head self-attention mechanism: Where Q, K, and V represent the query, key, and value matrices in the self-attention mechanism, respectively, obtained by linear projection of the output of the previous layer of the encoder; n is the number of attention heads, with a value of 16; head i This represents the output of the i-th attention head; This is used to project the concatenated attention head output back to the model dimension as an output weight matrix.

[0029] Each attention head is calculated as follows: in, These are the projection weight matrices used by the i-th attention head to generate the query, key, and value; Attention weights are calculated using scaled dot products: Where h=16 is the number of attention heads. .

[0030] Feedforward neural networks employ two layers of linear transformation plus activation functions: Where z represents the input vector of the feedforward neural network, i.e. the output of the self-attention layer; , is the weight matrix of the first-level linear transformation; , is the weight matrix of the second-level linear transformation; b1 is the bias vector of the first-level linear transformation; b2 is the bias vector of the second-level linear transformation; d ffn =2024 represents the hidden layer dimension of the feedforward network.

[0031] Residual connectivity and layer normalization: The compatibility embedding output is extracted from the [CLS] position of the final coding layer: Among them, Z (N) [0] represents the output sequence of the Nth (i.e., the final) Transformer encoder; [0] represents the feature vector extracted from the first position (index 0) in the sequence, which corresponds to a preset special classification label; For compatibility projection matrix; This is the final compatibility embedding vector.

[0032] The entire encoding process uses a Pre-LayerNorm architecture, with a dropout rate set to... ρ =0.3, using the Mish activation function to enhance the model's non-linear expressive power.

[0033] Furthermore, in step five, the loss function used in the dual-objective joint optimization training includes compatibility loss and personalization loss, and the total loss function is defined as: in, λ For dynamically adjusted personalized weighting coefficients; Compatibility loss The contrastive loss function is used, specifically in the form of: in, N For batch size, Let be the temperature parameter, sim be the cosine similarity function, and q be the temperature parameter. i To query clothing embedding, Embed the corresponding clothing item in the answer; Personalization loss It consists of marginal loss and incentive loss: Among them, marginal loss Defined as: Encourage losses Defined as: in, Embedding the model's predicted answer, To embed the real answer, h i The user's history is embedded, and sim is a cosine similarity function. The similarity score is amplified by 100 times.

[0034] Furthermore, in step five, the dual-objective joint optimization training is implemented through a multi-task learning framework, the specific technical details of which are as follows: Let the training batch samples be ,in, To query clothing embedding, Embed the clothing in the answer. Embedded user history, where N is the batch size.

[0035] The compatibility loss function employs temperature-scaled comparative learning: in, Let be the cosine similarity function, and let x and y represent the two embedding vectors to be compared. This refers to temperature hyperparameters.

[0036] The personalized loss function consists of marginal loss and incentive loss: The marginal loss term ensures that the predicted answer is no less personalized than the true answer: in, For marginal hyperparameters, Embedding the model's predicted answer. Embed the real answer.

[0037] The incentive loss term encourages personalized similarity to reach the desired level: Where α=60 is the target similarity threshold. β =100 is the scaling factor.

[0038] The total personalized loss is a weighted combination: in, This is the incentive loss weighting coefficient.

[0039] The overall optimization objective is achieved through dynamic weight balancing: in, For dynamically adjusted personalized weight coefficients, gradient backpropagation is used to simultaneously optimize both compatibility and personalization objectives: in, Represents the gradient operator. θ For model parameters, This is the learning rate.

[0040] Furthermore, in step six, the progressive two-stage training strategy specifically includes: the first stage performing pure compatibility pre-training, λ =0, using the AdamW optimizer with a learning rate of 1e-4; the second stage asymptotically introduces personalized constraints. λ The learning rate is linearly increased from 0 to the target value of 0.01; cosine annealing algorithm is used for learning rate scheduling, T_max = total training steps; gradient accumulation is used to optimize training efficiency; The cosine annealing algorithm used for learning rate scheduling in the progressive two-stage training strategy is as follows: in, For the first t The learning rate of the step. To minimize the learning rate, For the maximum learning rate, T cur T represents the current training steps. max This represents the total number of training steps.

[0041] Furthermore, in step six, the progressive two-stage training strategy is implemented through a phased optimization and dynamic parameter adjustment mechanism, the specific technical details of which are as follows: Phase 1: Pure compatibility pre-training phase In this initial stage, the model focuses on learning the compatibility relationships between individual clothing items, without introducing personalized objectives. The training objective function simplifies to: The personalized weight coefficient is set as follows: λ= 0, ensuring the model establishes a solid foundation for compatibility and discrimination in the early stages of training. This phase continues with E... warmup Each training cycle allows the model to fully learn the basic semantic relationships and visual coordination of clothing combinations.

[0042] Phase Two: Gradual Personalized Fine-Tuning Phase Once the compatibility loss has converged and stabilized, the system enters a phase of gradual personalization fine-tuning. This phase employs a dynamic weight adjustment strategy, gradually introducing personalized objectives. Personalized weighting gradual adjustment mechanism: Where e represents the current training period, E warmup E represents the number of preheating cycles. ramp =10 represents the linear growth period of the weights. λ max = 0.01 represents the maximum personalized weight. This mechanism ensures that the introduction of personalized goals is smooth and gradual, avoiding destructive interference with already learned compatibility knowledge.

[0043] The learning rate scheduling strategy uses the cosine annealing algorithm: in, The initial learning rate, E is the minimum learning rate. total This represents the total number of training cycles. The scheduling strategy maintains a high learning rate in the early stages of training to accelerate convergence, and gradually reduces the learning rate in the later stages to improve the quality of the solutions.

[0044] Gradient accumulation optimization techniques are used to handle large-scale batch training: Every Update parameters in small batches, effectively increasing batch size without increasing memory consumption: in, This represents the total loss value calculated on the k-th mini-batch; The core advantage of this two-stage training strategy lies in its ability to first establish a robust foundation for compatibility judgment, and then gradually integrate personalized factors on this basis. This ensures both the aesthetic rationality of the recommendation results and achieves true personalized customization. Through careful weight scheduling and learning rate control, the system finds the optimal balance between compatibility and personalization, significantly improving the practicality and user satisfaction of the clothing recommendation system.

[0045] Furthermore, in step seven, the multi-dimensional evaluation output includes: the accuracy calculation specifically refers to the calculation of the accuracy of the fill-in-the-blank task; the answer prediction specifically adopts the method of selecting prediction results from the candidate set based on the minimum distance criterion; the personalized score includes the predicted personalized score, the actual personalized score, and the personalized gap index, which serve as the basis for comprehensive evaluation.

[0046] Furthermore, in step seven, the multi-dimensional evaluation output is achieved through a comprehensive performance index system and an automated evaluation process, the specific technical details of which are as follows: The accuracy evaluation for the FITB task employs a nearest neighbor classification method based on a distance metric. Given a query clothing embedding... and candidate clothing embedding set First, calculate the Euclidean distance between the query and all candidates: Where, q i,k Represents the embedding vector q of the i-th query. i scalar value in the k-th dimension; c i,j,k c represents the j-th candidate embedding vector of the i-th sample. i,j The scalar value in the k-th dimension; the Euclidean distance between the i-th query and the j-th candidate. The prediction result is determined using the minimum distance criterion: Accuracy is calculated based on the consistency between the predicted results and the true labels: in, For indicator functions, Index for actual answers.

[0047] Personalized consistency assessment measures the quality of personalized recommendations from two dimensions. First, it calculates the personalization match between the actual answer and the user's historical data: in, h represents the candidate clothing embedding vector corresponding to the true answer in the i-th sample; i Let be the user history embedding vector for the i-th sample. Simultaneously, calculate the personalized matching degree between the predicted answer and the user history: in, Let represent the candidate clothing embedding vector corresponding to the model's predicted answer in the i-th sample. The personalization consistency score is calculated using the L1 loss function to determine how close the prediction is to the actual personalization level. This metric reflects the model's stability in maintaining a level of personalization; a value closer to 1 indicates better consistency in personalization.

[0048] Personalization gap analysis further quantifies the absolute difference between the actual and predicted levels of personalization: The comprehensive performance report integrates all evaluation metrics to generate a complete system performance analysis: in, To average the true personalized score, The average predicted personalized score.

[0049] This multi-dimensional evaluation system not only provides traditional accuracy metrics but also delves into the quality and stability of personalized recommendations, offering comprehensive data support for system optimization. By simultaneously monitoring FITB accuracy and personalization metrics, it can achieve truly personalized clothing recommendations while ensuring appropriate outfit combinations, meeting the stringent performance requirements of modern e-commerce platforms for recommendation systems.

[0050] This invention also discloses a personalized retrieval-based clothing recommendation system based on historical data perception, used to execute the above method, comprising the following modules: Multi-source clothing dataset loading and preprocessing module: acquires users' historical clothing data and single-item multimodal information, the multimodal information including single-item images and corresponding text descriptions; Parallel extraction module for multimodal deep features: Extracts and fuses image and text features from the multimodal information in step one to generate a single-item multimodal representation vector; User historical latent variable construction module: Based on the user's historical clothing sequence and the corresponding single-item multimodal representation vector, it generates user historical latent variables representing the user's style preferences through aggregation operations; Clothing compatibility embedding learning module: Input the multimodal representation vectors of each item in the clothing sequence to be evaluated into the Transformer encoder, learn the matching relationship through the self-attention mechanism, and output the compatibility embedding vector representing the overall matching. Dual-objective joint optimization training module: Construct a joint loss function. The first objective of this function is to reduce the distance between the predicted answer and the compatibility embedding vector of the query clothing. The second objective is to improve the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables, while optimizing both compatibility prediction and personalized recommendation objectives. Progressive two-stage training strategy module: It adopts a two-stage training to balance compatibility and personalization. In the first stage, it optimizes the first objective of the joint loss function, and in the second stage, it introduces and enhances the optimization weights of the second objective. Multi-dimensional evaluation output module: In the FITB task, it predicts the answer based on the distance between the compatibility embedding vectors to calculate the accuracy, and calculates the personalized score based on the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables.

[0051] Compared with the prior art, the beneficial effects of the present invention are: 1. Achieved an effective balance between compatibility prediction and personalized recommendation: Through an innovative dual-objective joint optimization training framework, it simultaneously optimizes both clothing matching compatibility and user personalized preferences. Employing a progressive two-stage training strategy, it first establishes a solid foundation for compatibility discrimination during the pre-training phase, and then progressively introduces personalized constraints through a dynamic weight adjustment mechanism. This effectively solves the technical challenge of conflicting compatibility and personalization requirements in traditional methods, significantly improving personalized matching accuracy while maintaining high FITB task accuracy.

[0052] 2. A fine-grained user history preference modeling mechanism was constructed: User history latent variables were built through a category-based hierarchical aggregation strategy, enabling refined encoding and representation learning of users' historical items across different clothing categories. Compared to traditional user modeling methods based on explicit ratings or simple click behavior, this invention can deeply mine users' long-term style preferences and aesthetic inclinations, generating more distinctive personalized style representations, significantly enhancing the personalization of the recommendation system and user satisfaction.

[0053] 3. An efficient multimodal feature fusion and cross-modal alignment scheme was designed: Leveraging the powerful cross-modal understanding capabilities of the CLIP pre-trained model, visual features of clothing images and semantic features of text descriptions were extracted in parallel. A unified 1024-dimensional multimodal representation was generated through concatenation, fusion, and L2 normalization. Compared to single-modal or simple feature concatenation methods, this invention fully utilizes the advantages of the visual-language pre-trained model in cross-modal semantic alignment, significantly improving the richness and semantic consistency of clothing feature representation, laying a solid foundation for subsequent compatibility learning and personalized recommendations. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a preferred embodiment of the present invention: a personalized retrieval-based clothing recommendation method based on historical data perception.

[0055] Figure 2 This is a schematic diagram of a personalized retrieval-based clothing recommendation method based on historical data perception, according to a preferred embodiment of the present invention. The text description constructed based on the single product category and its corresponding token representation are shown in the diagram.

[0056] Figure 3 This is a network architecture diagram of a personalized retrieval-based clothing recommendation method based on historical data perception, according to a preferred embodiment of the present invention.

[0057] Figure 4 This is a Transformer encoder architecture diagram of a personalized retrieval-based clothing recommendation method based on historical data perception, according to a preferred embodiment of the present invention.

[0058] Figure 5The diagram shows the experimental multi-dimensional evaluation output of a personalized retrieval-based clothing recommendation method based on historical data perception, which is a preferred embodiment of the present invention.

[0059] Figure 6 This diagram shows a web-based display platform for a personalized retrieval-based clothing recommendation method based on historical data, according to a preferred embodiment of the present invention.

[0060] Figure 7 This is a block diagram of a personalized retrieval-based clothing recommendation system based on historical data perception, according to a preferred embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] like Figures 1-3 This embodiment provides a personalized retrieval-based clothing recommendation method based on historical data perception. This method uses user historical interaction data and multimodal information of individual clothing items as input. First, the product description text is processed using a CLIP text encoder, extracting a 77-dimensional token sequence through tokenization. This token sequence is then input into a Transformer text encoding network for encoding, resulting in a 512-dimensional text feature vector. Next, clothing item images are processed, extracting image features through a CLIP visual encoder, outputting a 512-dimensional visual feature vector. The visual and text features are then fused using a concatenation method to generate a 1024-dimensional unified representation. This multimodal feature is input into a Transformer-based sequence encoding network. A 6-layer encoder structure and a 16-head self-attention mechanism learn the compatibility relationships between clothing items, generating a 128-dimensional compatibility embedding vector. Simultaneously, based on the user's historical clothing sequence, a hierarchical aggregation strategy is used to construct user historical latent variables, generating a personalized style representation. During the training phase, a dual-objective joint optimization framework is adopted to simultaneously optimize compatibility prediction loss and personalized recommendation loss. The two objectives are balanced through a progressive two-stage training strategy. Finally, the accuracy of the FITB task and the personalized score are output through a multi-dimensional evaluation system, so as to achieve intelligent clothing recommendation that meets both aesthetic standards and user preferences.

[0063] The following provides a more detailed description of each step of the technical solution in this embodiment.

[0064] Step 1: Loading and Preprocessing of Multi-Source Clothing Datasets: The dataset is organized in a structured manner, with separate training, validation, and test sets. Data loading is achieved through a specially constructed data class, which handles the triplet data for clothing combinations. Each data sample contains the queried clothing combination, the answer item, and its corresponding user history information. To establish the association between user history and clothing information, the system first loads the mapping data between user IDs and clothing sets, thereby establishing the correspondence between users and their historical clothing combinations. Simultaneously, the system reads the file paths of individual item images and their corresponding text descriptions to fully acquire the multimodal information of each item. In the data preprocessing stage, the clothing images undergo uniform size adjustment and pixel value normalization, and data augmentation techniques such as random cropping and horizontal flipping are used to improve the model's generalization ability. Figure 2 As shown, for text description data, a pre-trained word segmenter is used to generate a fixed-length sequence of tokens. To accelerate the training process and ensure the consistency of user representations, the system loads a pre-computed historical latent variable file to quickly build high-quality user historical representations. For cases of incomplete data, the system constructs a complete image dataset and initializes blank images as default resources. When user historical data is missing, the system automatically calls this predefined blank image feature as the default historical latent variable, ensuring stable system operation even when data is partially missing in real-world application scenarios.

[0065] Step 2, Parallel Extraction of Multimodal Deep Features: This step utilizes the CLIP pre-trained model to achieve parallel extraction of image and text features. For example... Figure 3 As shown, image feature extraction uses the CLIP visual encoder. The input image size is uniformly adjusted to 3×224×225 pixels. After standardization by the image processor, the output is a 512-dimensional visual feature vector. The visual encoder keeps its parameters frozen throughout the feature extraction process, utilizing the visual understanding capabilities obtained through pre-training. Text feature extraction uses the CLIP text encoder. The input is a sequence of 77-dimensional token representations after word segmentation, and the output is a 512-dimensional semantic feature vector, such as... Figure 2 As shown, the text encoder also maintains a parameter freeze state to ensure the stability of feature extraction. Multimodal feature fusion employs a concatenation aggregation strategy, concatenating the 512-dimensional visual feature vector and the 512-dimensional text feature vector along their respective feature dimensions to generate a unified 1024-dimensional representation. Subsequently, the fused features are subjected to L2 normalization to ensure that all feature vectors are distributed on a unit hypersphere, facilitating subsequent similarity calculations.

[0066] Step 3, Construction of User Historical Latent Variables: The construction of user historical latent variables adopts a hierarchical aggregation strategy. For example... Figure 3As shown, historical data is first grouped by user ID, with each user having at least 21 clothing categories. Within each category, the 10 most recently interacted items are selected. For each category, the mean vector of historical representations for all items within that category is calculated, serving as the historical preference representation for that category. After concatenating all category representations, a dimensionality transformation is performed through a fully connected layer to generate a 4×64×64 three-dimensional historical latent variable tensor. For new users or cases with missing historical data, pre-computed default latent variables are used for padding, generated based on general clothing image features.

[0067] First, historical clothing data is aggregated by user ID. Each user contains at least 21 clothing categories, and each category contains 10 historical clothing items. For each user u and category c, historical latent variables are calculated using mean pooling: Among them, h u,c x represents the set of historical clothing items for user u under category c. i For the first i Multimodal characteristics of individual clothing items N This refers to the number of historical individual items.

[0068] After the category-level historical representations are calculated, the multiple category representations are organized into a three-dimensional latent variable tensor with a size of 4×64×64 using a tensor reconstruction function. For users with missing historical data, the system uses latent variables generated based on the default image as a substitute.

[0069] The personalized style representation is finally obtained through linear transformation: the flattened historical latent variable tensor is input into the fully connected layer to generate a 128-dimensional user personalized style representation vector, which encodes the user's long-term style preferences and aesthetic tendencies.

[0070] Step 4, Clothing compatibility embedding learning: such as Figure 4 As shown, a Transformer encoder with a 6-layer structure is used, with each layer containing 16 attention heads. The hidden layer dimension is set to 512, and the feedforward network internal dimension is 2024. The input sequence length is set to 8, supporting combinations of up to 8 clothing items. During training, a dropout ratio of 0.3 is used to prevent overfitting, and the Mish activation function is used to enhance the model's non-linear expressive power. Compatibility embeddings are extracted from specific positional markers in the encoder output and reduced to a 128-dimensional compatibility feature vector through a projection layer.

[0071] Step 5, Dual-Objective Joint Optimization Training: Dual-objective joint optimization training achieves simultaneous optimization of compatibility prediction and personalized recommendation through a combination of designed loss functions. The specific implementation process is as follows: During training, each training batch contains 64 training samples, each consisting of a query clothing embedding, an answer clothing embedding, and a user history embedding. The compatibility loss is calculated using a temperature-scaled contrastive learning method, with the temperature parameter set to 0.1. Specifically, the cosine similarity matrix between the query and answer embeddings is first calculated. Then, for each query sample, its similarity score with positive answer samples is compared with the similarity scores of all negative answer samples within the batch. The probability distribution is calculated using a softmax function, ultimately yielding the contrastive loss value for each sample.

[0072] Personalization loss consists of two components: marginal loss and incentive loss. The marginal loss is calculated by first calculating the cosine similarity between the predicted answer and the user's historical embedding, and simultaneously calculating the similarity between the actual answer and the user's historical embedding. The two are subtracted, and a marginal threshold of 0.1 is subtracted from the difference. The maximum value of the result is then averaged in batches. The incentive loss is designed to ensure that the similarity between the predicted answer and the user's history reaches a reasonable level. A target similarity threshold of 60 and a scaling factor of 100 are set. A loss signal is generated when the predicted similarity falls below the threshold.

[0073] The overall loss function balances the two objectives using dynamic weighting coefficients λ, which are dynamically adjusted based on the training phase. During backpropagation in training, the gradients of the compatibility loss and the personalization loss with respect to the model parameters are calculated separately, then weighted and summed according to the weighting coefficients. Finally, the AdamW optimizer is used to update the model parameters. Gradient clipping is used to prevent gradient explosion, and the maximum gradient norm is set to 1.0.

[0074] Step Six, Progressive Two-Stage Training Strategy: The progressive two-stage training strategy achieves an effective balance between compatibility and personalization by optimizing the target in stages. The specific implementation process is as follows: The first phase is a pure compatibility pre-training phase, lasting for 20 training epochs. During this phase, the personalized weight coefficient λ is fixed at 0, and the model focuses entirely on learning the compatibility relationships between clothing items. The optimizer is AdamW, with an initial learning rate of 0.0001 and a weight decay coefficient of 0.01. A linear learning rate warm-up strategy is employed during training, gradually increasing the learning rate from 0 to the initial value over the first 1000 training steps to avoid instability in the early stages of training.

[0075] The second phase is the gradual personalized fine-tuning phase, starting from the 21st training cycle and continuing for 30 cycles. The personalized weight coefficients use a linear growth mechanism, starting from 0 and increasing by 0.001 each training cycle until a target weight of 0.01 is reached. The learning rate is scheduled using a cosine annealing algorithm, starting at 0.0001 and gradually decreasing according to a cosine function curve until a minimum learning rate of 0.000001 is reached.

[0076] Gradient accumulation is employed during training, with parameter updates performed every four mini-batches, effectively increasing batch size without increasing GPU memory consumption. Each mini-batch contains 64 samples, and through four gradient accumulations, the equivalent batch size reaches 256. The convergence of compatibility loss and personalization loss is monitored during training, and training is considered complete when both loss functions tend to stabilize.

[0077] Model selection was based on validation set performance; the checkpoint model that performed best in both accuracy and personalization score on the FITB task was chosen as the final model. The entire training process was completed on a single RTX 3090 GPU, with a total training time of approximately 48 hours.

[0078] Step 7, Multi-dimensional Evaluation Output: The multi-dimensional evaluation output comprehensively evaluates the recommendation system by constructing a comprehensive performance index system. The specific implementation process is as follows: The accuracy evaluation of the FITB compatibility metric is based on a distance-based classification method. During the evaluation, for each test sample, an incomplete clothing combination is received as the query input, along with a candidate set containing five candidate items, only one of which is the correct answer. The system first extracts the compatibility embedding vector of the query clothing combination, and then calculates the Euclidean distance between this vector and the embedding vectors of the five candidate items. The distance calculation uses the L2 norm in a 128-dimensional embedding space. During prediction, the candidate item with the smallest distance to the query vector is selected as the prediction result, and the accuracy is calculated by comparing the prediction result with the ground truth label. The entire test set contains 3200 FITB samples, and the accuracy statistics are based on the prediction results of all samples.

[0079] Personalized assessment is performed across two dimensions: predicted personalization level and actual personalization level. For each test sample, the system calculates the cosine similarity between the actual answer item and the user's historical representation, as the actual personalization score. Simultaneously, the system calculates the cosine similarity between the predicted answer item and the same user's historical representation, as the predicted personalization score. Both similarity calculations are performed in a normalized 128-dimensional embedding space, ensuring the scores range from -1 to 1.

[0080] Personalization gap analysis further quantifies the personalization bias in the recommendation results. The absolute difference between the predicted and actual personalization scores for each test sample is calculated, and then averaged over the entire test set to obtain the personalization gap index. This index directly reflects the prediction error of the recommendation system in the personalization dimension; a smaller value indicates more accurate personalization predictions.

[0081] like Figure 5As shown, through testing on the test set, the recommended compatibility index FITB task accuracy rate was 78.3%; the true personalization score was 60.79; the predicted personalization score was 59.97; and the personalization gap index was 0.83. Experiments show that the method proposed in this invention has achieved significant breakthroughs in core technical indicators: In terms of compatibility prediction accuracy, the FITB task accuracy reaches 78.3%, which is more than 13 percentage points higher than the generally lower than 65% level of existing technologies, proving that the method of this invention has significant advantages in understanding the semantic relationship and visual coordination of clothing matching; In terms of personalized recommendation quality, the true personalized score reaches 60.79, indicating that the standard answer itself has a good personalized foundation and provides a reasonable reference for target setting; In terms of personalized prediction accuracy, the predicted personalized score reaches 59.97, which is very close to the true personalized score, while the personalized gap index is only 0.83. This significantly low personalized gap value proves that the method of this invention can accurately predict user preferences, and the personalized recommendation results are highly consistent with the true preferences; In terms of overall system performance, the method of this invention simultaneously achieves high compatibility accuracy (78.3%) and low personalized prediction error (0.83), breaking through the technical bottleneck of existing technologies that cannot take into account both compatibility requirements and personalized needs.

[0082] like Figure 6 As shown, this embodiment discloses a personalized retrieval-based clothing recommendation web platform based on historical data perception. The platform constructs a complete recommendation system workflow: after a user selects basic items from preset categories (including tops, bottoms, footwear, bags, accessories, etc.), the platform calls the underlying recommendation model in real time to evaluate the current outfit from multiple dimensions, quantitatively outputting two core indicators—clothing compatibility score (reflecting the aesthetic harmony between items) and user personalized matching score (reflecting the degree of fit between the recommendation results and the user's historical preferences). Based on this evaluation result, the system automatically filters alternative items that can significantly improve the overall score through a ranking retrieval mechanism, dynamically optimizing the outfit scheme, thereby achieving deep personalized recommendations while ensuring visual harmony, effectively improving the quality of outfit suggestions and user satisfaction.

[0083] like Figure 7 As shown, this embodiment discloses a personalized retrieval-based clothing recommendation system based on historical data perception, used to execute the above method embodiment, including the following modules: Multi-source clothing dataset loading and preprocessing module: acquires users' historical clothing data and single-item multimodal information, the multimodal information including single-item images and corresponding text descriptions; Multimodal deep feature parallel extraction module: Extracts and fuses image and text features from the multimodal information in step one using a CLIP encoder; User historical latent variable construction module: Based on the user's historical clothing sequence and the corresponding single-item multimodal representation vector, it generates user historical latent variables representing the user's style preferences through aggregation operations; Clothing compatibility embedding learning module: Input the multimodal representation vectors of each item in the clothing sequence to be evaluated into the Transformer encoder, learn the matching relationship through the self-attention mechanism, and output the compatibility embedding vector representing the overall matching. Dual-objective joint optimization training module: Construct a joint loss function. The first objective of this function is to reduce the distance between the predicted answer and the compatibility embedding vector of the query clothing. The second objective is to improve the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables, while optimizing both compatibility prediction and personalized recommendation objectives. Progressive two-stage training strategy module: It adopts a two-stage training to balance compatibility and personalization. In the first stage, it optimizes the first objective of the joint loss function, and in the second stage, it introduces and enhances the optimization weights of the second objective. The multi-dimensional evaluation output module: In the FITB task, it predicts answers based on the distance between the compatibility embedding vectors to calculate accuracy, and calculates a personalized score based on the correlation between the predicted answer's compatibility embedding vector and the user's historical latent variables. Other aspects of this embodiment can be found in the above method embodiments.

[0084] In summary, this invention discloses a personalized retrieval-based clothing recommendation method and system based on historical data perception. This invention constructs a multimodal feature fusion network using a CLIP pre-trained model and a Transformer encoder, and jointly optimizes it with contrastive learning loss and personalized marginal loss to achieve a unified approach to clothing matching compatibility prediction and personalized user recommendation. A category-based hierarchical aggregation strategy is used to construct user historical latent variables, encoding long-term style preferences. A progressive two-stage training strategy is employed: the first stage focuses on basic learning for compatibility discrimination, and the second stage progressively introduces personalized constraints through dynamic weight adjustments, effectively balancing the two objectives. Multi-dimensional evaluation shows that this invention performs excellently in both FITB (Fill in the Blank) task accuracy and personalization metrics, achieving truly personalized recommendations while ensuring aesthetically pleasing matching. This invention is robust and has broad application prospects in fashion e-commerce, virtual try-on, and personalized styling design.

Claims

1. A personalized retrieval-based clothing recommendation method based on historical data perception, characterized in that, Specifically, the following steps are included: Step 1, Loading and preprocessing multi-source clothing dataset: Obtain user's historical clothing data and multimodal information of individual items, wherein the multimodal information includes individual item images and corresponding text descriptions; Step 2, Parallel Extraction of Multimodal Deep Features: Features are extracted and fused from the single-item images and text descriptions in Step 1 to generate multimodal representation vectors for single items; Step 3, Construction of user historical latent variables: Based on the user's historical clothing sequence and the corresponding single-item multimodal representation vector, user historical latent variables representing the user's style preference are generated through aggregation operations; Step 4, Clothing compatibility embedding learning: Input the multimodal representation vectors of each item in the clothing sequence to be evaluated into the Transformer encoder, learn the matching relationship through the self-attention mechanism, and output the compatibility embedding vector representing the overall matching. Step 5, Dual-objective joint optimization training: Construct a joint loss function. The first objective of this function is to reduce the distance between the predicted answer and the compatibility embedding vector of the query clothing. The second objective is to improve the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables, while optimizing both compatibility prediction and personalized recommendation objectives. Step 6, Progressive two-stage training strategy: A two-stage training approach is adopted to balance compatibility and personalization. In the first stage, the first objective of the joint loss function is optimized, and in the second stage, the optimization weights of the second objective are introduced and enhanced. Step 7, Multi-dimensional evaluation of output: In the FITB task, answer prediction is performed based on the distance between the compatibility embedding vectors to calculate the accuracy, and a personalized score is calculated based on the correlation between the compatibility embedding vectors of the predicted answer and the user's historical latent variables.

2. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 1, characterized in that, In step one, the loading and preprocessing of the multi-source clothing dataset is achieved as follows: The mapping relationship data between user IDs and clothing sets is loaded to associate the clothing combinations the user interacted with historically; the path of individual item image files and corresponding text description data are read to form complete multimodal information of individual items; the data is organized in a structured manner including training, validation, and test sets; triple data consisting of queried clothing combinations, answer items, and user historical information is processed by constructing data classes; in the data preprocessing stage, uniform size adjustment and normalization operations are performed on the clothing images, and word segmentation is performed on the text descriptions to generate fixed-length word sequences; pre-computed historical latent variable files are loaded; an image dataset is constructed and blank images are initialized as default resources so that the features of the blank images can be used as default historical latent variables when user historical data is missing.

3. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 1, characterized in that, In step two, the image encoder of the contrastive language-image pre-trained CLIP model is used to process the clothing image, with an input size of 3×224×224 and an output of 512-dimensional visual feature vector; the product description text is processed by the CLIP text encoder, with an input of 77-dimensional token sequence and an output of 512-dimensional semantic feature vector. Visual and textual features are fused using a concat method to generate a 1024-dimensional unified representation; the fused features are then subjected to L2 normalization.

4. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 1, characterized in that, Step three, the construction of the user's historical latent variables includes: Historical clothing data is aggregated by user ID, with each user having more than 21 clothing categories; latent style preference variables are generated based on the user's historical clothing items, with each category containing 10 items; historical items are encoded using a CLIP encoder to generate historical representation vectors of size [4, 64, 64]; for users with missing historical data, the latent variable corresponding to null_img is used as the default value.

5. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 4, characterized in that, In step three, the construction of the user's historical latent variables adopts a category-based hierarchical aggregation strategy: For each user u and categories c The historical latent variables are calculated as follows: Where History(u,c) represents the user u In category c A collection of historical costumes, e i For the first i Multimodal embedding of individual clothing items Aggregate The function uses mean pooling. The global user history representation is obtained by concatenating all category latent variables: 。 6. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 1, characterized in that, Step four is as follows: Model the clothing sequence using a Transformer encoder, which includes 6 encoder layers and a 16-head self-attention mechanism; learn the matching compatibility relationship between clothing items through the self-attention mechanism; and generate a 128-dimensional compatibility embedding vector.

7. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 5, characterized in that, In step five, the loss function includes compatibility loss and personalization loss, and the total loss function is defined as follows: in, λ For dynamically adjusted personalized weighting coefficients; Compatibility loss The contrastive loss function is used, specifically in the form of: in, N For batch size, Let be the temperature parameter, sim be the cosine similarity function, and q be the temperature parameter. i To query clothing embedding, Embed the corresponding clothing item in the answer; Personalization loss It consists of marginal loss and incentive loss: Among them, marginal loss Defined as: Encourage losses Defined as: in, Embedding the model's predicted answer, To embed the real answer, h i For user history embedding, sim is the cosine similarity function.

8. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 7, characterized in that, In step six, the first stage involves pure compatibility pre-training. λ =0, using the AdamW optimizer, learning rate 1e-4; The second phase gradually introduces personalized constraints. λ The learning rate is linearly increased from 0 to the target value of 0.01; cosine annealing algorithm is used for learning rate scheduling, T_max = total training steps; gradient accumulation is used to optimize training efficiency; The cosine annealing algorithm is as follows: in, For the first t The learning rate of the step. To minimize the learning rate, For the maximum learning rate, T cur T represents the current training steps. max This represents the total number of training steps.

9. The personalized retrieval-based clothing recommendation method based on historical data perception according to claim 1, characterized in that, In step seven, the multi-dimensional evaluation output includes: the accuracy calculation specifically refers to the calculation of the accuracy of the fill-in-the-blank task; the answer prediction specifically adopts the method of selecting prediction results from the candidate set based on the minimum distance criterion; the personalized score includes the predicted personalized score, the actual personalized score, and the personalized gap index, which serve as the basis for comprehensive evaluation.

10. A personalized retrieval-based clothing recommendation system based on historical data perception, used to perform the method as described in any one of claims 1-9, characterized in that, Includes the following modules: Multi-source clothing dataset loading and preprocessing module: acquires users' historical clothing data and single-item multimodal information, the multimodal information including single-item images and corresponding text descriptions; Parallel extraction module for multimodal deep features: extracts and fuses image and text features from multimodal information to generate a single-item multimodal representation vector; User historical latent variable construction module: Based on the user's historical clothing sequence and the corresponding single-item multimodal representation vector, it generates user historical latent variables representing the user's style preferences through aggregation operations; Clothing compatibility embedding learning module: Input the multimodal representation vectors of each item in the clothing sequence to be evaluated into the Transformer encoder, learn the matching relationship through the self-attention mechanism, and output the compatibility embedding vector representing the overall matching. Dual-objective joint optimization training module: Construct a joint loss function. The first objective of this function is to reduce the distance between the predicted answer and the compatibility embedding vector of the query clothing. The second objective is to improve the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables, while optimizing both compatibility prediction and personalized recommendation objectives. Progressive two-stage training strategy module: It adopts a two-stage training to balance compatibility and personalization. In the first stage, it optimizes the first objective of the joint loss function, and in the second stage, it introduces and enhances the optimization weights of the second objective. Multi-dimensional evaluation output module: In the FITB task, it predicts the answer based on the distance between the compatibility embedding vectors to calculate the accuracy, and calculates the personalized matching score based on the correlation between the compatibility embedding vector of the predicted answer and the user's historical latent variables.

Images