A fashion preference prediction method and device

Abstract

The present application relates to the technical field of multi-modal data prediction, and discloses a fashion preference prediction method and device, which comprises obtaining multi-modal data including images, texts and user behavior time sequence, performing feature extraction to obtain image features, text features and time sequence features; constructing spatial features based on city embedding tables and corresponding regional culture label codes; aligning the dimensions of the spatial features and the time sequence features, fusing the aligned spatial features and the time sequence features by using a gated attention mechanism to obtain spatio-temporal fusion features; splicing the spatio-temporal fusion features with the image features and the text features to obtain multi-modal fusion features; decoupling the multi-modal fusion features to obtain material decoupling features, style decoupling features and scene decoupling features; and after feature splicing of the various decoupling features, performing Transform coding and MLP classification to obtain fashion preference prediction probability.

Description

Technical Field

[0001] This invention relates to the field of multimodal data prediction technology, and in particular to a method and apparatus for predicting fashion preferences. Background Technology

[0002] In industrial production, the fashion industry faces the dual challenges of upgrading consumer personalized demands and intensifying homogeneous competition. Building a product development system centered on accurately capturing user preferences and rapidly responding to market demands has become crucial for companies to break through competitive bottlenecks. At this point, consumer demand for clothing has shifted from basic functionality to scenario-based dressing and emotional value expression. E-commerce platforms and enterprise product development systems, as the core carriers connecting demand and products, urgently need reliable fashion preference prediction technology to support an efficient closed loop of "demand insight - product innovation - market delivery." The ability to accurately uncover deep user preferences and predict fashion trends directly determines the success rate of product development and market competitiveness for enterprises.

[0003] Collaborative filtering is a fundamental technology in fashion preference prediction. With the development of deep learning, collaborative filtering has entered a new stage of development, with Neural Collaborative Filtering (NCF) becoming a core technology. The core objective of NCF is to enhance the ability to model nonlinear interaction patterns between users and items. It uses an embedding layer combined with a multilayer perceptron (MLP) as its core architecture. First, user IDs and fashion item IDs are transformed into low-dimensional dense vector representations through the embedding layer. Then, these two types of vectors are fused to form an interaction feature vector, which is then input into an MLP containing multiple fully connected networks (such as a typical 512-256-128 structure) for nonlinear transformation. Finally, the probability of a user's preference for a target fashion item is output through a sigmoid activation function, thereby predicting the user's fashion preferences. To further optimize performance, NCF has spawned several improved models. For example, DeepCF introduces an attention mechanism to dynamically focus on key features for prediction; TA-NCF (Temporal Attention NCF) adds a temporal attention layer and sets a time decay factor, which can better combine the user's recent historical behavior sequence and consider the impact of different time dimensions of behavior on current preferences. However, existing Neural Collaborative Filtering (NCF) and its improved models have significant shortcomings. First, they lack multimodal information fusion capabilities, only modeling the interaction between user ID and fashion item ID, and cannot integrate multimodal data such as images and text. Even with a few improvements, the simple processing leads to the loss of fine-grained semantics, ignoring the local correlation between clothing details and user preferences. Second, they have poor semantic interpretability. The vectors generated by the embedding layer and the interaction features output by the MLP are all implicit vectors without explicit definitions, which cannot be associated with business attributes such as material and style, making it difficult to support business decisions.

[0004] Contrast-based learning-driven decoupling methods are an important emerging technology in the field of decoupling learning. Their core objective is to enhance the independence of decoupled features through contrastive learning mechanisms, thereby more accurately separating complex influencing factors of user preferences. The key logic of this type of method is to construct a cross-modal contrastive loss. By defining positive and negative samples, it forces multimodal data related to the same user (such as text and image comments) to be mapped to features closer together in the decoupling space, while similar item data from different users form distinguishable features further apart in the decoupling space. This achieves effective decoupling and semantic alignment of multimodal features. The CMCL (Cross-modal Contrastive Learning) model proposed by Sun et al. is a typical representative of this type of method. It adopts a dual-tower network architecture, extracting image and text features through corresponding encoders, and then combining the constructed cross-modal contrastive loss for model training. This effectively improves the decoupling effect and provides technical support for accurately separating influencing factors of user preferences and improving feature representation capabilities. However, existing contrastive learning-driven decoupling methods still have significant shortcomings. First, the decoupling dimensions lack explicit business definitions and do not incorporate fashion industry knowledge to guide the direction. The generated latent variables cannot be aligned with core business attributes such as materials, styles, and scenes, making it difficult for business personnel to understand and apply the decoupling features. Second, they rely on manually constructed negative sample pairs, and the quality is difficult to guarantee. They use a uniform temperature coefficient to adjust sample similarity, ignoring the different requirements of different attributes for contrast intensity, which affects the accuracy of decoupling.

[0005] Spatiotemporal decoupling technology aims to capture the spatiotemporal changes in user preferences, and its development has gone through different stages of exploration. In the early stages, this technology introduced time series analysis tools to mine the temporal evolution of user behavior, such as the LSTM model with a time decay factor designed by Tang et al., and the time series model built based on the Transformer self-attention mechanism. These models can effectively capture the changes in user preferences over time. With the development of technology, spatiotemporal decoupling and fusion have emerged in recent years. Among them, the STD (Spatiotemporal Decoupling) model proposed by Liu et al. is representative. This model decomposes the user's latent vector into two components corresponding to time-sensitive features (such as seasonal preferences) and space-sensitive features (such as regional popular elements) by designing a spatiotemporal decoupling regularization term, thereby achieving the decoupling of spatiotemporal features. In addition, the TA-NCF model has also explored spatiotemporal fusion. Based on the output of the time attention layer, it further splices regional features to integrate spatiotemporal information and provide support for a more comprehensive capture of the spatiotemporal dynamics of user preferences. However, existing spatiotemporal decoupling techniques also have significant shortcomings. First, early techniques focused only on capturing the temporal evolution of users' own behavior, without incorporating the influence of external spatial context (such as regional cultural events and climate fluctuations) on group preferences. Second, recent spatiotemporal fusion attempts (such as STD and TA-NCF models) have coarse spatial division granularity, mostly remaining at a large spatial level, and cannot accurately capture city-level cultural differences. Third, all methods treat spatiotemporal information as static features, without constructing dynamic control logic for spatiotemporal components and decoupling processes, making it unable to quickly respond to scenarios such as the surge in preference for quick-drying materials during the rainy season and the explosion in demand for ski clothing during the snow season, resulting in a decline in the interpretability of model attributes and predictive stability.

[0006] Among them, multimodal fusion technology is a key technology used to integrate multimodal data such as images, text, and time series in fashion preference prediction scenarios. Its core objective is to provide more comprehensive information support for preference prediction by effectively fusing the features of different types of data. This technology mainly includes two implementation paths: traditional feature concatenation and early fusion. Traditional feature concatenation first extracts features of each modality using corresponding tools, such as using ResNet-50 to extract 512-dimensional image features and BERT-base to extract 768-dimensional text features. Then, it combines spatiotemporal features to directly concatenate these features of different modalities into a higher-dimensional vector (e.g., a 1408-dimensional vector). This vector is then input into a fully connected layer for dimensionality reduction to adapt to the input requirements of subsequent classifiers. Early fusion focuses on integrating multimodal information during the feature extraction stage. This can be achieved in two ways: first, during image feature extraction (e.g., after the convolutional layer of ResNet-50), spatiotemporal location encoding is added through residual connections to initially fuse spatiotemporal information with image features; second, modality attention weights are designed to dynamically adjust the contribution of each modality feature according to the degree of influence of different modalities on the prediction results, thereby achieving early integration of multimodal information. Existing multimodal fusion technologies have significant shortcomings. First, traditional feature splicing methods suffer from information degradation due to the large differences in modal feature dimensions and semantic density, making it difficult to leverage the collaborative value of multimodal features. Second, early fusion methods suffer from fine-grained semantic loss, with global average pooling of images losing clothing details and text processing failing to capture implicit scene requirements, thus failing to preserve deep semantic relationships. Third, neither of these methods constructs a collaborative mechanism for the decoupling process, meaning the fused features cannot provide clear semantic support for attribute decoupling, making it difficult to achieve accurate separation of business attributes.

[0007] In summary, while current fashion preference prediction technologies have evolved with the development of big data and AI, they still have significant limitations: traditional methods often map users and items to single feature vectors, making it difficult to retain the fine-grained semantics of multimodal data such as images, text, and time series. They fail to fully capture the connection between details like fabric texture and cuff design and user aesthetics, and also struggle to analyze implicit scenario needs, resulting in vague preference profiles. Furthermore, they generally ignore the influence of spatiotemporal dimensions, failing to adapt to regular preference changes due to seasonal climate and regional culture, and are ill-suited to short-term fluctuations caused by holiday marketing and sudden trend events. Ultimately, the prediction results are out of sync with market demand, hindering the efficiency of enterprise product innovation. While some unsupervised decoupling methods attempt to separate complex factors in multimodal data, they fail to design decoupling directions in conjunction with the characteristics of the fashion industry, resulting in features lacking clear business semantics, making them difficult for business personnel to apply. Moreover, the lack of cross-modal semantic consistency constraints easily leads to misalignment between image and text features. Simultaneously, existing technologies often simply splice spatiotemporal information without constructing a dynamic adjustment mechanism for decoupling spatiotemporal and attribute information, further limiting prediction accuracy and practicality. While knowledge graphs in the fashion industry can structurally express the relationship between "material-style-scene", traditional construction methods rely on manual definition or simple tag matching, which makes it difficult to cover the deep relationship between design elements, spatiotemporal trends and user preferences. They also fail to effectively integrate with multimodal decoupling and spatiotemporal modeling, and cannot meet the needs of high-precision and controllable preference prediction in industrial production. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problems in the prior art that the inability to effectively integrate multimodal information easily leads to the loss of details, coarse spatial granularity, and failure to respond to market changes in a timely manner due to the lack of integration with spatiotemporal dynamics.

[0009] To address the aforementioned technical problems, this invention provides a fashion preference prediction method, comprising:

[0010] Acquire multimodal data including images, text, and time-series sequences of user behavior;

[0011] Feature extraction is performed on multimodal data to obtain image features, text features, and temporal features;

[0012] Spatial features are constructed based on the city embedding table and its corresponding regional cultural label coding.

[0013] Align the dimensions of spatial features with those of temporal features, and use a gating attention mechanism to fuse the aligned spatial features with temporal features to obtain spatiotemporal fusion features;

[0014] Spatiotemporal fusion features are concatenated with image and text features to obtain multimodal fusion features;

[0015] Decouple the multimodal fusion features to obtain material decoupling features, style decoupling features, and scene decoupling features;

[0016] After concatenating multiple decoupled features, the fashion preference prediction probability is obtained through Transformer encoding and MLP classification.

[0017] Preferably, feature extraction is performed on the multimodal data to obtain image features, including:

[0018] Convolution is performed on image data in multimodal data to obtain convolutional image features;

[0019] The convolutional image features are input into the improved MobileNetV3 to calculate and obtain the channel attention weights;

[0020] The image features are obtained by multiplying the convolutional image features element-wise with the channel attention weights;

[0021] The improved MobileNetV3 introduces an SE attention mechanism into the global averaging layer after the fifth convolutional block in the standard MobileNetV3.

[0022] Preferably, feature extraction is performed on the multimodal data to obtain text features, including:

[0023] WordPiece embedding is performed on text data in multimodal data to obtain the semantic vectors of each word;

[0024] The semantic vectors of each word are input into a 4-layer Transformer encoder to obtain the sentence-level attention weights of each word.

[0025] Text features are obtained by weighting and summing the sentence-level attention weights of each word with the word semantic vectors.

[0026] Preferably, feature extraction is performed on the multimodal data to obtain time-series features, including:

[0027] Embed the time series sequences of user behavior corresponding to each time step in the multimodal data to obtain the time series vector of each time step;

[0028] Bidirectional LSTM is used to compute the forward and backward hidden states of the temporal vectors at each time step;

[0029] The backward hidden state and the forward hidden state of the last time step are concatenated to obtain the temporal features.

[0030] Preferably, spatial features are constructed based on the city embedding table and its corresponding regional cultural label encoding, including:

[0031] Based on city ID and its corresponding embedding vector representation Construct a city embedding table; the embedding vector represents the commonalities of a city's geographical coordinates, economic level, population density, cultural attributes, and user behavior.

[0032] Obtain the unique heat vectors of various regional cultures. ;

[0033] Concatenate the embedded vector representation corresponding to each city ID and the unique regional cultural vector of its respective region. , obtain the concatenated vector;

[0034] Pass the concatenated vector through a fully connected layer to obtain spatial features. , is represented as: ;

[0035] in, Represents the ReLU activation function. This represents the weight matrix of the fully connected layer. This represents the bias vector of the fully connected layer.

[0036] Preferably, the spatial features and temporal features are aligned in dimension, and a gated attention mechanism is used to fuse the aligned spatial features and temporal features to obtain spatiotemporal fused features, including:

[0037] Based on learnable weight matrices and time-series features Spatial features Build a reset door Update door With candidate hidden state , respectively represented as:

[0038] , , ;

[0039] Based on the reset gate, update gate, and candidate hidden states, spatiotemporal fusion features are calculated. , is represented as:

[0040] ;

[0041] in, This represents the Sigmoid activation function. and This indicates the learnable weights for resetting the gate. and This indicates the learnable weights of the updated gate. and Represents the learnable weights of the candidate hidden states. This indicates resetting the gate vector. This indicates element-wise multiplication.

[0042] Preferably, the multimodal fusion features are decoupled to obtain material decoupling features, style decoupling features, and scene decoupling features, including:

[0043] The multimodal fusion features are input into the first fully connected structure, the second fully connected structure, and the third fully connected structure respectively to obtain the corresponding material decoupling features, style decoupling features, and scene decoupling features;

[0044] The first fully connected structure, the second fully connected structure, and the third fully connected structure all include three fully connected layers connected in series along the forward propagation direction.

[0045] Preferably, after decoupling the multimodal fusion features and obtaining material decoupling features, style decoupling features, and scene decoupling features, the process includes constructing a knowledge graph to guide the optimization of the fully connected structure, including:

[0046] Using clothing materials, styles, and applicable scenarios as entities, a fashion knowledge graph is constructed;

[0047] The TransE model is used to embed the triple relationship between any two entities in the fashion knowledge graph into a preset dimensional space to obtain the corresponding relationship feature vector;

[0048] Based on material decoupling features, style decoupling features, scene decoupling features, and their corresponding relational feature vectors, a knowledge alignment decoupling loss is constructed. , is represented as: ;

[0049] Based on entity anchor features, positive samples, and negative samples, a cross-modal contrastive loss is constructed. , is represented as: ;

[0050] Based on knowledge alignment loss and cross-modal contrast loss, backpropagation is used to update the parameters of the fully connected structure and obtain the optimized fully connected structure.

[0051] in, This refers to entities that belong to a material, style, or scene. Representing entities The decoupling characteristics, Representing entities relational feature vectors Represents cosine similarity. Representing entities Anchor point features, and Representing entities respectively Corresponding positive and negative samples, express Norm.

[0052] Preferably, after embedding the triple relationship between any two entities in the fashion knowledge graph into a preset dimensional space using the TransE model and obtaining the corresponding relationship feature vector, the process includes:

[0053] Based on fashion knowledge graph Positive sample relationships between various entities Relationship with negative samples Constructing knowledge graph embedding loss , is represented as: ;

[0054] The TransE model is trained based on knowledge graph embedding loss until the triples of positive samples are satisfied, the trained TransE model is obtained, and the relation feature vector is updated.

[0055] in, and These represent the head and tail entities of a positive sample, respectively. This represents the triplet relationship of positive samples. Indicates the boundary hyperparameters; and These represent the head and tail entities of the negative sample, respectively. This represents the triplet relationship of negative samples.

[0056] This embodiment provides an apparatus based on the fashion preference prediction method described above, comprising:

[0057] The multimodal feature extraction module is used to acquire multimodal data including images, text, and time-series sequences of user behavior; and to extract features from the multimodal data to obtain image features, text features, and time-series features.

[0058] The spatiotemporal dynamic modulation module is used to construct spatial features based on the city embedding table and its corresponding regional cultural label encoding; align the dimensions of spatial features with temporal features, and use a gating attention mechanism to fuse the aligned spatial features and temporal features to obtain spatiotemporal fusion features; and concatenate the spatiotemporal fusion features with image features and text features to obtain multimodal fusion features.

[0059] The knowledge-guided decoupling module is used to decouple multimodal fusion features and obtain material decoupling features, style decoupling features, and scene decoupling features.

[0060] The preference prediction module is used to concatenate multiple decoupled features, and then obtain the fashion preference prediction probability through Transformer encoding and MLP classification.

[0061] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0062] The fashion preference prediction method described in this invention extracts image features, text features, and temporal features from multimodal data, and combines them with a city embedding table to construct city-level spatial features, accurately depicting regional cultural differences. It utilizes a gating attention mechanism to fuse aligned spatial and temporal features, obtaining spatiotemporal fusion features that can quickly respond to seasonal and holiday events, achieving accurate and timely preference probability prediction. The spatiotemporal fusion features are then concatenated with image and text features to obtain multimodal fusion features, avoiding information degradation caused by early fusion, achieving cross-modal semantic complementarity and synergistic enhancement, adaptively responding to preference changes in spatiotemporally sensitive scenarios, and improving prediction stability and attribute decoupling accuracy. Finally, the multimodal fusion features are decoupled to obtain multiple corresponding decoupled features, which are then concatenated, encoded, and classified to obtain the fashion preference prediction probability.

[0063] In the image feature extraction stage, this invention precisely replaces the global average pooling layer of the 5th convolutional block in MobileNetV3 with the SE attention pooling layer, enhancing detailed features such as clothing texture and silhouette. In the text feature extraction stage, a sentence-level attention mechanism is added to the BERT-Tiny Transformer output layer, focusing on keywords related to commuting and relaxed fit. In the temporal feature extraction stage, a bidirectional LSTM is used to capture the periodic patterns of user behavior, fully preserving fine-grained semantic information in multimodal data. This invention achieves fine-grained multimodal feature extraction from multimodal data, providing multimodal feature support for preference prediction and ensuring prediction accuracy. This invention establishes an efficient multimodal fusion mechanism, fully integrating image, text, and temporal data, solving the problems of insufficient multimodal information fusion and loss of fine-grained semantics, accurately preserving fine-grained semantics, avoiding information degradation and loss, and providing comprehensive and high-quality feature support for fashion preference prediction.

[0064] This invention constructs a fashion knowledge graph based on the business attributes of clothing, such as material, style, and context, enhancing the semantic interpretability of the model and providing clear technical support for business scenarios such as product innovation and marketing decisions. Based on the triplet relationships between any two entities in the fashion knowledge graph, relational feature vectors are constructed to calculate knowledge alignment loss and cross-modal contrast loss. This forces the decoupled features to be precisely aligned with business attributes, while ensuring semantic consistency across multiple modalities such as images and text. This overcomes the limitations of traditional decoupling techniques, such as semantic ambiguity and disconnect from business needs, improving prediction accuracy and practicality. Attached Figure Description

[0065] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0066] Figure 1This is a flowchart of the steps in the fashion preference prediction method of the present invention;

[0067] Figure 2 This is a schematic diagram of the image feature extraction module;

[0068] Figure 3 This is a flowchart of the text feature extraction workflow;

[0069] Figure 4 This is a flowchart of the time-series feature extraction process;

[0070] Figure 5 This is a schematic diagram of multimodal fusion feature construction;

[0071] Figure 6 This is a schematic diagram of the decoupling feature acquisition process;

[0072] Figure 7 This is a flowchart for obtaining the probability of predicting fashion preferences. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0074] Reference Figure 1 The flowchart of the fashion preference prediction method of the present invention is shown below, and the specific steps are shown in S101 to S107.

[0075] S101: Acquire multimodal data including images, text, and time-series sequences of user behavior.

[0076] S102: Perform feature extraction on multimodal data to obtain image features, text features, and time-series features.

[0077] Reference Figure 2 The diagram shows the structure of the image feature extraction module, which extracts features from multimodal data to obtain image features. This includes: convolving the image data in the multimodal data to obtain convolutional image features; inputting the convolutional image features into an improved MobileNetV3 to calculate channel attention weights; and multiplying the convolutional image features element-wise with the channel attention weights to obtain image features. The improved MobileNetV3 introduces an SE attention mechanism into the global averaging layer after the fifth convolutional block in the standard MobileNetV3, using global information to calculate channel weights to optimize feature representation.

[0078] Specifically, the image feature extraction module adopts a MobileNetV3 structure combined with SE attention pooling. First, the image is input into a pre-trained MobileNetV3 module. Then, the SE attention mechanism is introduced into the global average pooling of the 5th convolutional block, using the formula... Calculate and obtain channel attention weights After Output 512-dimensional image features ;in, For convolution output, For global average pooling, It is a fully connected layer containing 128→32→128; For element-wise product, This represents the activation function. This represents the weighted feature map obtained by element-wise multiplication of the convolution output with the channel attention weights.

[0079] Reference Figure 3 The diagram shows the workflow for text feature extraction. Feature extraction is performed on multimodal data to obtain text features, including: WordPiece embedding of text data in multimodal data to obtain the semantic vector of each word; inputting the semantic vector of each word into a 4-layer Transformer encoder to obtain the sentence-level attention weight of each word; and performing a weighted sum based on the sentence-level attention weight of each word and the semantic vector of each word to obtain the text features.

[0080] Specifically, the text feature submodule is based on an improved BERT-Tiny. The text is embedded into 384-dimensional word vectors using WordPiece, and then input into a 4-layer Transformer. Calculate sentence-level attention weights After Output 256-dimensional text features ;in, Represents the learnable parameter matrix, For word semantic vectors, The dimension of the word semantic vector. =384.

[0081] Reference Figure 4 The diagram shows the workflow for temporal feature extraction. Feature extraction is performed on multimodal data to obtain temporal features, including: embedding the user behavior temporal sequence corresponding to each time step in the multimodal data to obtain the temporal vector of each time step; using bidirectional LSTM to calculate the forward hidden state and backward hidden state of the temporal vector of each time step; and concatenating the backward hidden state and the forward hidden state of the last time step to obtain the temporal features.

[0082] Specifically, the temporal feature submodule employs a bidirectional LSTM. The behavioral sequence is embedded into a 128-dimensional vector and then input into a bidirectional LSTM with 128 hidden units. The forward and backward hidden states are calculated separately to obtain the temporal features, including:

[0083] Forward hidden state: ; Represents a mapping function. This represents the forward weight matrix of the hidden state. express The 128-dimensional vector input at time step 1, This represents the forward transition matrix of the hidden state. express The forward hidden state at time 10:00 Represents the forward bias vector;

[0084] Backward hidden state: ; This represents the backward weight matrix of the hidden state. This represents the backward transition matrix of the hidden state. express The backward hidden state at time 10:00. Represents the backward bias vector;

[0085] Will and By splicing, 256-dimensional temporal features are obtained. .

[0086] This invention employs MobileNetV3 combined with SE attention pooling for image feature extraction, preserving local details such as clothing texture and cuffs, and outputting 512-dimensional image features. Based on an improved BERT-Tiny combined with sentence-level attention weights, it extracts 256-dimensional text features to capture implicit scene requirements. Finally, it uses a bidirectional LSTM to model user behavior sequences, outputting 256-dimensional temporal features. This invention achieves fine-grained multimodal feature extraction from multimodal data, providing multimodal feature support for preference prediction and ensuring prediction accuracy.

[0087] S103: Based on the city embedding table and its corresponding regional cultural label encoding, construct spatial features, including:

[0088] S103-1: Based on city ID and its corresponding embedding vector representation Construct a city embedding table; the embedding vector represents the commonalities of a city's geographical coordinates, economic level, population density, cultural attributes, and user behavior.

[0089] S103-2: Obtain the unique heat vectors of various regional cultures. ;

[0090] S103-3: Concatenate the embedded vector representations corresponding to each city ID and the unique regional cultural vectors of their respective areas. , obtain the concatenated vector;

[0091] S103-4: Pass the concatenated vector through a fully connected layer to obtain spatial features. , is represented as: ;

[0092] in, Represents the ReLU activation function. This represents the weight matrix of the fully connected layer. This represents the bias vector of the fully connected layer.

[0093] S104: Align the dimensions of spatial features with those of temporal features, and use a gated attention mechanism to fuse the aligned spatial and temporal features to obtain spatiotemporal fused features, including:

[0094] S104-1: Based on learnable weight matrices and temporal features Spatial features Build a reset door Update door With candidate hidden state , respectively represented as:

[0095] , , ;

[0096] S104-2: Calculate spatiotemporal fusion features based on reset gate, update gate, and candidate hidden states. , is represented as: ;

[0097] in, This represents the Sigmoid activation function. and This indicates the learnable weights for resetting the gate. and This indicates the learnable weights of the updated gate. and Represents the learnable weights of the candidate hidden states. This indicates resetting the gate vector. This indicates element-wise multiplication.

[0098] S105: Concatenate the spatiotemporal fusion features with image features and text features to obtain multimodal fusion features.

[0099] Reference Figure 5The diagram shown illustrates the construction of multimodal fusion features, including a spatial feature refinement unit (containing a city embedding table and regional cultural label encoding) and a temporal feature input unit (receiving...). Gated Spatiotemporal Attention (GSTA) fusion unit and multimodal stitching node.

[0100] Specifically, the spatial feature refinement unit first constructs an embedded table of 348 cities. Extract based on user city ID Combined with the unique regional cultural vectors ,pass Output 128-dimensional spatial features The GSTA unit will first... Linear upsizing to 256 dimensions to align with temporal features, then using a reset gate. Update door Candidate hidden state Computational spatiotemporal fusion features Finally and , spliced ​​into 1024-dimensional multimodal fusion features It is transmitted to the knowledge-guided decoupling layer through the channel.

[0101] This invention establishes an efficient multimodal fusion mechanism that fully integrates image, text, and time-series data, solving the problems of insufficient multimodal information fusion and loss of fine-grained semantics. It accurately preserves fine-grained semantics, avoids information degradation and loss, and provides comprehensive and high-quality feature support for fashion preference prediction.

[0102] S106: Decouple the multimodal fusion features to obtain material decoupling features, style decoupling features, and scene decoupling features, including:

[0103] The multimodal fusion features are input into the first fully connected structure, the second fully connected structure, and the third fully connected structure respectively to obtain the corresponding material decoupling features, style decoupling features, and scene decoupling features;

[0104] The first fully connected structure, the second fully connected structure, and the third fully connected structure all include three fully connected layers connected in series along the forward propagation direction.

[0105] Based on the above embodiments, in this embodiment of the invention, after decoupling the multimodal fusion features and obtaining material decoupling features, style decoupling features, and scene decoupling features, the method includes constructing a knowledge graph to guide the optimization of the fully connected structure, including:

[0106] Using clothing materials, styles, and applicable scenarios as entities, a fashion knowledge graph is constructed;

[0107] The TransE model is used to embed the triple relationship between any two entities in the fashion knowledge graph into a preset dimensional space to obtain the corresponding relationship feature vector;

[0108] Based on material decoupling features, style decoupling features, scene decoupling features, and their corresponding relational feature vectors, a knowledge alignment decoupling loss is constructed. , is represented as: ;

[0109] Based on entity anchor features, positive samples, and negative samples, a cross-modal contrastive loss is constructed. , is represented as: ;

[0110] Based on knowledge alignment loss and cross-modal contrast loss, backpropagation is used to update the parameters of the fully connected structure and obtain the optimized fully connected structure.

[0111] in, This refers to entities that belong to a material, style, or scene. Representing entities The decoupling characteristics, Representing entities relational feature vectors Represents cosine similarity. Representing entities Anchor point features, and Representing entities respectively Corresponding positive and negative samples, express Norm.

[0112] Based on the above embodiments, in this embodiment of the invention, after embedding the triple relationship between any two entities in the fashion knowledge graph into a preset dimensional space using the TransE model and obtaining the corresponding relationship feature vector, the process includes: based on the fashion knowledge graph... Positive sample relationships between various entities Relationship with negative samples Constructing knowledge graph embedding loss , is represented as: The TransE model is trained using knowledge graph embedding loss until the triples of positive samples meet the requirements. The trained TransE model is then obtained, and the relation feature vector is updated. and These represent the head and tail entities of a positive sample, respectively. This represents the triplet relationship of positive samples. Indicates the boundary hyperparameters; and These represent the head and tail entities of the negative sample, respectively. This represents the triplet relationship of negative samples.

[0113] Reference Figure 6 The diagram illustrates the decoupled feature acquisition process. Specifically, a FashionKG knowledge graph is first constructed, stored in "entity-relationship-entity" triples (containing 20 material categories, 15 style categories, and 10 scene categories). The TransE model is used to embed the triples into a 128-dimensional space. Training All three decoupling subnetworks are fully connected structures of "1024→512→256→128" (with the activation function LeakyReLU), and their outputs are respectively... (Material) (style), (Scenario); The loss calculation unit uses... Forced attribute alignment, while introducing cross-modal contrast loss. The loss result updates the sub-network parameters through the backpropagation channel, and the decoupled features are input into the preference prediction layer through the channel. Boundary hyperparameters , For positive sample triples, For negative sample triples, for Norm; for The corresponding entity vector; For cosine similarity, For anchor point features, For positive samples This is a negative sample.

[0114] This invention incorporates knowledge from the fashion industry to establish clear associations between technical features and business attributes such as materials, styles, and scenarios, enhancing the semantic interpretability of the model and providing clear technical support for business scenarios such as product innovation and marketing decisions. Simultaneously, this invention incorporates external spatial context to achieve fine-grained spatial characterization, constructing a dynamic control and collaboration mechanism that integrates spatiotemporal information with multimodal fusion and attribute decoupling. This adaptively responds to preference changes in spatiotemporally sensitive scenarios, improving prediction stability and the accuracy of attribute decoupling.

[0115] S107: After concatenating multiple decoupled features, the results are processed by Transformer encoding and MLP classification to obtain the fashion preference prediction probability, including:

[0116] After concatenating the material decoupling features, style decoupling features, and scene decoupling features, Transformer encoding is performed to obtain aggregated features;

[0117] The aggregated features are input into the MLP classifier, the hidden layer features are calculated and activated, and the fashion preference prediction probability is obtained.

[0118] Reference Figure 7 The diagram shown is a flowchart for obtaining the probability of fashion preference prediction; in preference prediction, the process first involves piecing together... , , 384-dimensional predictive features were obtained. Transformer encoder (2 layers, 4-head attention) Feature aggregation is performed, maintaining an output dimension of 384; the MLP classification network has a "384→256→128→1" structure, through... Calculate hidden layer features, and then... Output preference probability ; at the same time through Matching attribute labels, the final output is a preference probability and a "material + style + scene" combined label (e.g., "cotton + style 1 + workplace") through a dual-output channel. The classification loss is calculated using... Calculations are used for model parameter optimization. Among them, It is the Sigmoid activation function. for Medium entity vector, This is a real label; , and These represent the weights of the first, second, and output layers of the MLP, respectively. , and These represent the biases of the first, second, and output layers of the MLP, respectively.

[0119] This invention addresses the shortcomings of traditional fashion preference prediction technologies, such as insufficient multimodal information fusion and loss of fine-grained semantics, which prevent accurate capture of the correlation between clothing details and user scenario needs. It constructs a multimodal feature extraction and gated spatiotemporal attention fusion mechanism. SE attention preserves visual details such as clothing texture and silhouette, sentence-level attention parses implicit scenario needs in the text, and GSTA dynamically balances spatiotemporal feature weights, providing complete and accurate multimodal feature support for preference prediction. Furthermore, to address the shortcomings of existing decoupling techniques, such as semantic ambiguity and disconnect from business attributes, this invention introduces the FashionKG knowledge graph and cross-modal contrastive loss, using the TransE model to embed knowledge entities, combined with decoupling... The loss-forced decoupling features are aligned with the "material, style, and scene" attributes. At the same time, improved cosine similarity calculation is used to optimize cross-modal semantic consistency, which significantly improves the business interpretability and prediction accuracy of the decoupling dimension.

[0120] Based on the above embodiments, this invention employs a four-layer serial architecture consisting of a multimodal feature extraction layer, a spatiotemporal dynamic modulation layer, a knowledge-guided decoupling layer, and a preference prediction layer. It takes images (clothing pictures), text (product descriptions / user reviews), temporal sequences (user behavior sequences), spatial sequences (user city location), and a fashion knowledge graph as input. Through an end-to-end process of feature extraction, spatiotemporal fusion, attribute decoupling, and preference prediction, it outputs the probability of user preference for the target clothing, as well as explanations of attributes such as material, style, and scene. In this embodiment, the multimodal feature extraction layer consists of three sub-modules, which extract features from image features, text features, and temporal sequences, respectively. The output features of the three sub-modules are all accessed through independent channels to the spatiotemporal dynamic modulation layer. After multimodal feature fusion, they are transmitted to the knowledge-guided decoupling layer through channels. The decoupled features are then input to the preference prediction layer through channels, and the fashion preference probability and combined labels including material, style, and scene are output through dual output channels.

[0121] In this embodiment, the loop process structure includes data loading, feature extraction, spatiotemporal fusion, decoupling optimization, loss calculation, parameter update, and convergence judgment nodes. First, the FashionAI dataset (1.2 million images, 800,000 text entries, and 15 million interaction records) and spatiotemporal extended data are loaded, and the training / validation / test sets are divided in an 8:1:1 ratio (each set contains 25% of the data from all four seasons). Then, the multimodal feature extraction layer, spatiotemporal dynamic modulation layer, and knowledge-guided decoupling layer are sequentially called to generate decoupling features, and the calculation... Constructing joint losses The AdamW optimizer (initial learning rate 1e-4, decaying by 0.1 every 10 rounds, weight decaying by 1e-5) is used to backpropagate and update the parameters of each layer. Training stops when the validation set AUC increases by ≤0.001 for 5 consecutive rounds or when 50 rounds have been completed. Otherwise, the process returns to the feature extraction node and continues to loop. The training process log is stored in real time to the data storage node through the channel.

[0122] Specifically, when the fashion preference prediction method provided in this embodiment is integrated and deployed as a system, the user data input terminal (mobile phone / computer), the middle is the server cluster, and the right side is the result output terminal; the core hardware includes two NVIDIA A100 GPU servers (for model training and inference), a 10TB SSD data storage node (for storing datasets and training logs), and an embedded GPU Jetson AGX edge terminal (for real-time prediction); in the software modules, the data preprocessing module is deployed on the storage node (to implement image resizing and text segmentation), the model inference module is deployed on the GPU server (using TensorRT acceleration, with inference latency ≤200ms), and the knowledge graph management module is deployed on the MySQL database (providing... The query interface and the results display module are deployed on the edge terminal (visualizing preference probabilities and attribute labels). Each device is connected via 5G / Ethernet. The user terminal transmits data to the GPU server. After the server interacts with the storage node and the database, it sends the prediction results back to the output terminal. The data transmission is encapsulated in JSON format.

[0123] This invention provides a fashion preference prediction device, comprising:

[0124] The multimodal feature extraction module is used to acquire multimodal data including images, text, and time-series sequences of user behavior; and to extract features from the multimodal data to obtain image features, text features, and time-series features.

[0125] The spatiotemporal dynamic modulation module is used to construct spatial features based on the city embedding table and its corresponding regional cultural label encoding; align the dimensions of spatial features with temporal features, and use a gating attention mechanism to fuse the aligned spatial features and temporal features to obtain spatiotemporal fusion features; and concatenate the spatiotemporal fusion features with image features and text features to obtain multimodal fusion features.

[0126] The knowledge-guided decoupling module is used to decouple multimodal fusion features and obtain material decoupling features, style decoupling features, and scene decoupling features.

[0127] The preference prediction module is used to concatenate multiple decoupled features, and then obtain the fashion preference prediction probability through Transformer encoding and MLP classification.

[0128] The fashion preference prediction device in this embodiment is used to implement the aforementioned fashion preference prediction method. Therefore, the specific implementation of the fashion preference prediction device can be found in the embodiment section of the fashion preference prediction method above. For example, the multimodal feature extraction module is used to implement steps S101 and S102 in the fashion preference prediction method; the spatiotemporal dynamic modulation module is used to implement steps S103, S104 and S105 in the fashion preference prediction method; the knowledge-guided decoupling module and the preference prediction module are used to implement steps S106 and S107 in the fashion preference prediction method, respectively. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0129] Specifically, the multimodal feature extraction module is responsible for fine-grained feature extraction (including details such as texture, silhouette, and color) of clothing images (224×224 resolution, JPEG / PNG format), scene semantic parsing of user comment text (length ≤ 50 tokens) (such as "commuting wear", "loose and slimming"), trend pattern capture of user behavior time series (length ≤ 20), and spatial feature transformation of user city location data (covering 348 cities), completing the standardized preprocessing and unified format output of multimodal data.

[0130] Specifically, the spatiotemporal dynamic modulation module and the knowledge guidance decoupling module integrate a gated spatiotemporal attention (GSTA) unit, which can dynamically adjust the fusion weights of local spatial features (such as urban cultural style) and global temporal features (such as seasonal fashion trends); it interfaces with the FashionKG knowledge graph and supports TransE embedding training of "material-style-scene" triples, through comparative loss. The constraint decoupling process generates semantically clear decoupling feature vectors that are aligned with business attributes.

[0131] Specifically, preference prediction can aggregate decoupled features and output user preference probabilities (0-1 range) and attribute explanation labels including "material, style, and scene" through an MLP classification network.

[0132] This embodiment also includes a model training and evaluation module, supporting joint loss. The backpropagation training is used to calculate and visualize evaluation metrics such as AUC, NDCG@10, and attribute classification accuracy in real time, while also adapting to model compression (TensorRT quantization) and edge terminal deployment requirements.

[0133] In the image feature extraction stage, this invention precisely replaces the global average pooling layer of the 5th convolutional block in MobileNetV3 with the SE attention pooling layer, enhancing detailed features such as clothing texture and silhouette. In the text feature extraction stage, a sentence-level attention mechanism is added to the BERT-Tiny Transformer output layer, focusing on keywords related to commuting and relaxed living scenarios. In the temporal feature extraction stage, a bidirectional LSTM is used to capture the periodic patterns of user behavior, fully preserving fine-grained semantic information in multimodal data. This invention innovatively constructs a three-level processing mechanism including city embedding, cultural labels, and gating balance. First, a dedicated city embedding table covering 348 cities is established, associated with 8 types of regional cultural labels, generating fine-grained spatial features. Then, a GRU gating unit is introduced, dynamically adjusting the fusion weights of local spatial features (such as city style preferences) and global temporal trends (such as seasonal fashion changes) by resetting and updating the gates, solving the problem of inaccurate capture of regions and dynamic trends caused by the coarse spatial division and fixed spatiotemporal weights of traditional technologies. Based on the triple structure (containing 20 material types, 15 style types, and 10 scene types) obtained from the FashionKG knowledge graph, three parallel and independent fully connected decoupled sub-networks are designed. First, knowledge entity embedding is achieved through the TransE model. Then, a dual-loss constraint combining attribute alignment loss and cross-modal contrast loss forces precise alignment between decoupled features and business attributes, while ensuring semantic consistency across multiple modalities such as images and text. This overcomes the limitations of traditional decoupling techniques, such as semantic ambiguity and disconnect from business needs. This invention constructs a collaborative logic combining probability prediction and attribute interpretation. First, the decoupled material, style, and scene features are concatenated, and the user preference probability is output through an MLP network. Simultaneously, the decoupled features are semantically matched with knowledge graph entities to generate clear "material + style + scene" attribute labels, forming a dual output result. This addresses the pain point of traditional prediction models that only output probabilities and cannot explain the recommendation basis.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A fashion preference prediction method, characterized by, include: Acquire multimodal data including images, text, and time-series sequences of user behavior; Feature extraction is performed on multimodal data to obtain image features, text features, and temporal features, including: Convolution is performed on image data in multimodal data to obtain convolutional image features; the convolutional image features are input into an improved MobileNetV3 to calculate and obtain channel attention weights; the convolutional image features are multiplied element-wise with the channel attention weights to obtain image features; wherein, the improved MobileNetV3 introduces an SE attention mechanism into the global averaging layer after the fifth convolutional block in the standard MobileNetV3. WordPiece embedding is performed on the text data in the multimodal data to obtain the word semantic vector of each word; the word semantic vector of each word is input into a 4-layer Transformer encoder to obtain the sentence-level attention weight of each word; the sentence-level attention weight of each word and the word semantic vector are weighted and summed to obtain the text features; The user behavior time series corresponding to each time step in the multimodal data is embedded to obtain the time series vector of each time step; the forward hidden state and backward hidden state of the time series vector of each time step are calculated using bidirectional LSTM; the backward hidden state and forward hidden state of the last time step are concatenated to obtain the time series features. Based on the city embedding table and its corresponding regional cultural label encoding, spatial features are constructed, including: city ID and its corresponding embedding vector representation. A city embedding table is constructed; the embedding vector represents the commonalities of a city's geographical coordinates, economic level, population density, cultural attributes, and user behavior; and unique-hot vectors of various regional cultures are obtained. Concatenate the embedded vector representations corresponding to each city ID and the unique hot vectors of the regional culture of their respective areas. Obtain the concatenated vector; pass the concatenated vector through a fully connected layer to obtain spatial features. , is represented as: ;in, Represents the ReLU activation function. This represents the weight matrix of the fully connected layer. This represents the bias vector of the fully connected layer; Align the dimensions of spatial features with those of temporal features, and use a gating attention mechanism to fuse the aligned spatial features with temporal features to obtain spatiotemporal fusion features; Spatiotemporal fusion features are concatenated with image and text features to obtain multimodal fusion features; Decouple the multimodal fusion features to obtain material decoupling features, style decoupling features, and scene decoupling features; After concatenating multiple decoupled features, the fashion preference prediction probability is obtained through Transformer encoding and MLP classification.

2. The fashion preference prediction method according to claim 1, characterized in that, By aligning the dimensions of spatial and temporal features, and using a gated attention mechanism to fuse the aligned spatial and temporal features, spatiotemporal fusion features are obtained, including: based on learnable weight matrices, temporal features with spatial features , constructing a reset gate , an update gate with candidate hidden states , respectively ， ， ； The spatiotemporal fusion feature is calculated based on the reset gate, the update gate, and the candidate hidden state is represented as: ； in, This represents the Sigmoid activation function. and This indicates the learnable weights for resetting the gate. and This indicates the learnable weights of the updated gate. and Represents the learnable weights of the candidate hidden states. This indicates resetting the gate vector. This indicates element-wise multiplication. 3.The fashion preference prediction method of claim 1, wherein, Decouple the multimodal fusion features to obtain material decoupling features, style decoupling features, and scene decoupling features, including: The multimodal fusion features are input into the first fully connected structure, the second fully connected structure, and the third fully connected structure respectively to obtain the corresponding material decoupling features, style decoupling features, and scene decoupling features; The first fully connected structure, the second fully connected structure, and the third fully connected structure all include three fully connected layers connected in series along the forward propagation direction. 4.The fashion preference prediction method of claim 1, wherein, After decoupling the multimodal fusion features and obtaining material decoupling features, style decoupling features, and scene decoupling features, a knowledge graph is constructed to guide the optimization of the fully connected structure, including: Using clothing materials, styles, and applicable scenarios as entities, a fashion knowledge graph is constructed; The TransE model is used to embed the triple relationship between any two entities in the fashion knowledge graph into a preset dimensional space to obtain the corresponding relationship feature vector; Based on the material decoupling features, style decoupling features, scene decoupling features and corresponding relationship feature vectors, a knowledge alignment decoupling loss is constructed , is expressed as: ; Based on entity-based anchor features, positive samples and negative samples, a cross-modal contrast loss is constructed , is expressed as: ; Based on knowledge alignment loss and cross-modal contrast loss, backpropagation is used to update the parameters of the fully connected structure and obtain the optimized fully connected structure. in, This refers to entities that belong to a material, style, or scene. Representing entities The decoupling characteristics, Representing entities relational feature vectors Represents cosine similarity. Representing entities Anchor point features, and Representing entities respectively Corresponding positive and negative samples, express Norm. 5.The fashion preference prediction method of claim 4, wherein, By using the TransE model to embed the triplet relationship between any two entities in the fashion knowledge graph into a preset dimensional space, and obtaining the corresponding relationship feature vector, the following features are obtained: Based on fashion knowledge graph Positive sample relationships between various entities Relationship with negative samples Constructing knowledge graph embedding loss , represented as: ; The TransE model is trained based on knowledge graph embedding loss until the triples of positive samples are satisfied, the trained TransE model is obtained, and the relation feature vector is updated. in, and These represent the head and tail entities of a positive sample, respectively. This represents the triplet relationship of positive samples. Indicates the boundary hyperparameters; and These represent the head and tail entities of the negative sample, respectively. This represents the triplet relationship of negative samples.

6. An apparatus based on the fashion preference prediction method according to any one of claims 1 to 5, characterized in that, include: The multimodal feature extraction module is used to acquire multimodal data including images, text, and time-series sequences of user behavior. Feature extraction is performed on multimodal data to obtain image features, text features, and temporal features; The spatiotemporal dynamic modulation module is used to construct spatial features based on the city embedding table and its corresponding regional cultural label encoding; align the dimensions of spatial features with temporal features, and use a gating attention mechanism to fuse the aligned spatial features and temporal features to obtain spatiotemporal fusion features; and concatenate the spatiotemporal fusion features with image features and text features to obtain multimodal fusion features. The knowledge-guided decoupling module is used to decouple multimodal fusion features and obtain material decoupling features, style decoupling features, and scene decoupling features. The preference prediction module is used to concatenate multiple decoupled features, and then obtain the fashion preference prediction probability through Transformer encoding and MLP classification.

Images