A BPR clothing recommendation method and system based on multimodal information

Abstract

The present invention discloses a BPR clothing recommendation method and system considering multimodal information, which samples negative sample data in combination with clothing sub-classification source data and constructs triple data; extracts multimodal clothing features, and outputs a first-category visual-text compatible space feature set {f multi}, the second category of clothing name feature set {f name}、The third type of combined feature set {f multi}+{f name}; calculate the neighbor feature set; calculate the clothing multimodal relationship measurement matrix (MRM); calculate the BPR rationality clothing recommendation loss function; and use exposure reduction technology to reduce the MCAR bias in fashion datasets, thereby alleviating the impact of data sparsity. Compared with existing technologies, this invention can achieve a clothing recommendation model that balances rationality and accuracy.

Description

Technical Field

[0001] The present invention relates to a business-driven intelligent recommendation system, and in particular to a BPR clothing recommendation method based on multimodal information. Background Art

[0002] Clothing pairing is an important way to express one's self-image, taste, and style. However, clothing pairing is affected by various subjective and objective factors. When considering clothing for different seasons and occasions, it is necessary to comprehensively consider multiple attributes of clothing, such as cut, pattern, color, and fabric. At the same time, factors such as personal preferences, body shape, skin color, facial features, and cultural differences also affect clothing choices. In actual clothing recommendation scenarios, users want to understand the logic behind recommended clothing so that they can gradually form their own dressing experience based on these reasons or opinions. In addition, the current available clothing pairing datasets have a serious sparsity-induced exposure bias problem. Considering the above factors, a clothing recommendation system should have the following functions: (1) accurately understand user preferences; (2) recommend reasonable clothing; and (3) avoid the exposure bias problem caused by data sparsity.

[0003] Current mainstream recommendation systems are categorized as collaborative filtering-based, content-based, and hybrid. Collaborative filtering fully exploits user interaction history to make recommendations based on user or item similarity; however, it can suffer from a cold-start problem for new users or new products. With the rapid development of artificial intelligence (AI) technology, various visual, textual, and multimodal models have been proposed and widely used for content-based recommendations. However, these methods suffer from data sparsity and cannot accurately represent users with broad interests or no clear preferences. Due to the shortcomings of these two methods, hybrid recommendation approaches have become the mainstream approach in academia and industry.

[0004] Bayesian personalized ranking (BPR) can accurately rank implicit feedback data, but its interpretability is poor. To enhance the rationality of clothing recommendation models, various hybrid clothing recommendation methods based on BPR have been proposed. These mainly include using visual and textual features to enhance the rationality of model recommendations and using human-derived or probabilistic statistical rules to impart human fashion knowledge to clothing recommendation systems. Although these methods have significantly improved the accuracy of BPR-based clothing recommendation systems, several issues remain that need to be addressed:

[0005] 1) Accuracy and rationality are key metrics for clothing recommendation systems. BPR assumes that user / item preferences are independent of each other, making recommendation results uninterpretable. Interaction records between garments contain rich attribute and style correlations, but existing BPR-based clothing recommendation methods fail to fully utilize this information to enhance model interpretability. Furthermore, effectively incorporating interpretability factors into the BPR framework remains an unresolved issue.

[0006] 2) BPR assumes that interaction data is missing completely at random (MCAR), meaning that users have an equal chance of observing all items. However, fashion apparel datasets are often sparse, and this assumption does not hold. Existing BPR-based clothing recommendation methods do not fully consider this issue.

[0007] 3) In the fashion field, visual features convey information such as color and shape, while textual features provide information such as material and brand. The significant physical difference between the two cannot be achieved optimally through direct concatenation or multi-layer perceptron (MLP) fusion.

[0008] 4) When sampling negative samples, existing clothing recommendation methods often use random sampling of similar clothing or top-K negative sample sampling, which causes problems such as overly simple training samples or false negative samples.

[0009] References are as follows:

[0010] 1.X.Song, F.Feng, X.Han,

[0011] 2.X.Han,

[0012] 3.S.-I.Papadopoulos,C.Koutlis,S.Papadopoulos,and I.Kompatsiaris,“VICTOR:Visual incompatibility detection with transformers and fashion-specific contrastive pre-training”,Journal of Visual Communication and ImageRepresentation,vol.90,p.103741,2023.

[0013] 4.K.Damak,S.Khenissi,and O.Nasraoui,“Debiased explainable pairwiseranking from implicit feedback,”in Proceedings of the 15th ACM Conference onRecommender Systems,2021,pp.321-331.

[0014] 5.R.He and J.McAuley,“VBPR:Visual Bayesian Personalized Ranking fromImplicit Feedback”,in Proceedings of the AAAI conference on artificialintelligence,vol.30,no.1,2016.

[0015] 6.X.Song,X.Han,Y.Li,J.Chen,X.-S.Xu,and L.Nie,“GP-BPR:PersonalizedCompatibility Modeling for Clothing Matching”,in Proceedings of the 27th ACMinternational conference on multimedia,2019,pp.320-328.

[0016] 7. D. Sagar, J. Garg, P. Kansal, S. Bhalla, RR Shah, and Y. Yu, "PAI-BPR: Personalized Outfit Recommendation Scheme with Attribute-wiseInterpretability", in 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM). IEEE, 2020, pp. 221-230.

[0017] 8.Y.Lin, P.Ren, Z.Chen, Z.Ren, J.Ma, and M.De Rijke, "Explainable outfitrecommendation with joint outfit matching and comment generation", IEEE Transactions on Knowledge and Data Engineering, vol.32, no.8, pp.1502-1516, 2019.

[0018] 9. L. Yang, Y. Cui, Y. Xuan, C. Wang, S. Belongie, and D. Estrin, “Unbiasedoffline recommender evaluation for missing-not-at-random implicit feedback,” in Proceedings of the 12th ACM conference on recommender systems, 2018, pp.279-287. Summary of the Invention

[0019] In order to solve the problems of personalized differences and user demand understanding in the existing clothing recommendation methods, the present invention proposes a BPR clothing recommendation method and system based on multimodal information, and implements a BPR clothing recommendation strategy based on multimodal relationship measurement matrix and exposure bias mitigation.

[0020] The present invention is achieved by the following technical solutions:

[0021] A BPR clothing recommendation method based on multimodal information, the method comprising:

[0022] S1. Combine clothing sub-classification source data sampling to obtain negative sample data and construct triple data (selected clothing, paired positive sample clothing, unpaired negative sample clothing);

[0023] S2, extracting multimodal clothing features based on the triplet data of step S1;

[0024] S3, output three types of feature sets according to the multimodal clothing features of step S2, specifically: the first type of visual-text compatible space feature set {f multi}, the second category of clothing name feature set {f name}、The third type of combined feature set {f multi}+{f name}, where + represents feature concatenation;

[0025] S4, calculate and obtain the visual-text compatible spatial feature set of the first category {f multi} and the second category clothing name feature set {f name}’s neighbor feature set;

[0026] S5, calculating clothing multimodal relationship measurement matrix MRM;

[0027] S6. Combining the three-category feature set outputted in step S3 with the clothing multimodal relationship measurement matrix MRM calculated in step S5, and calculating a rational clothing recommendation loss function based on BPR;

[0028] S7. Obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and based on the trade-off result, reduce the MCAR bias in the fashion dataset through exposure bias reduction technology, thereby alleviating the impact of data sparsity.

[0029] A BPR clothing recommendation system based on multimodal information, including:

[0030] The negative sample sampling module 100 for combining clothing sub-classification is used to obtain negative sample data by combining clothing sub-classification source data and construct triplet data including selected clothing, paired positive sample clothing and unpaired negative sample clothing;

[0031] A multimodal feature extraction module 200 is used to extract multimodal clothing features based on triple data;

[0032] The feature combination module 300 is used to output three types of feature sets of the multimodal clothing features, including the first type of visual-text compatible space feature set {f multi}, the second category of clothing name feature set {f name}、The third type of combined feature set {f multi}+{f name}, where + represents feature concatenation;

[0033] Neighbor calculation module 400, used to calculate the first type of visual-text compatible spatial feature set {f multi} and the second category clothing name feature set {f name}’s neighbor feature set;

[0034] The clothing multimodal relationship measurement matrix MRM calculation module 500 is used to calculate the clothing multimodal relationship measurement matrix MRM; specifically, it includes counting the pairing records between clothing to obtain M_Inter, and combining the two neighbor feature sets calculated by the neighbor calculation module to obtain the MRM matrix according to three matrix combination methods;

[0035] A BPR-based rationality recommendation loss function calculation module 600 is used to calculate a BPR-based rationality clothing recommendation loss function based on a combination of a feature set and a clothing multimodal relationship measurement matrix MRM;

[0036] The data sparsity impact reduction module 700 is used to obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and reduce the MCAR bias in the fashion dataset through exposure bias reduction technology based on the trade-off result.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1) Detailed analysis and design of a multimodal relationship measurement matrix that comprehensively considers the combined relationship between clothing semantic relationship information and clothing historical interaction information. This matrix is ​​then applied to the BPR loss function in a reasonable and effective manner, enabling the clothing recommendation model to balance rationality and accuracy, making it more suitable for versatile and new clothing styles.

[0039] 2) Alleviate the exposure bias of fashion data by considering the impact of data sparsity. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an overall flow chart of a BPR clothing recommendation method based on multimodal information of the present invention;

[0041] Figure 2 Detailed flow chart of S2, S4 and S5;

[0042] Figure 3 This is a module diagram of a BPR clothing recommendation system based on multimodal information of the present invention;

[0043] Figure 4 A block diagram of a specific embodiment of the present invention;

[0044] Figure 5An example diagram of the loss function calculation module for BPR-based rationality recommendation;

[0045] Figure 6 This is an example diagram of the negative sample sampling module combined with clothing sub-classification;

[0046] Figure 7 are the 5 nearest neighbors calculated based on the visual-text compatible spatial features. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] Example 1

[0049] like Figures 1 to 2 As shown, the present invention provides a BPR clothing recommendation method based on multimodal information, comprising:

[0050] S1. Combine clothing sub-classification source data sampling to obtain negative sample data and construct triple data (selected clothing, paired positive sample clothing, unpaired negative sample clothing);

[0051] S2, extracting multimodal clothing features according to the triple data of step S1, wherein the triple data at least includes clothing images, corresponding texts and clothing names, such as Figure 2 As shown in (2a), the following specific steps are included:

[0052] S2.1. Extracting visual-text compatible spatial features f multi , wherein the visual-textual compatibility space feature f multi Contains clothing attribute-level information, which can clearly and unambiguously represent clothing color, shape, material, brand and other information, and can effectively capture the pairing style of clothing;

[0053] S2.2. Extract clothing name features f name ; Wherein, the clothing name feature f fname Contains historical clothing pairing information to provide more accurate and personalized recommendation results for the recommendation system;

[0054] S3, output the following three types of feature sets according to the multimodal clothing features of step S2, specifically: the first type of visual-text compatible space feature set {f multi}, the second category of clothing name feature set {f name}、The third type of combined feature set {f multi}+{f name}, where + represents feature concatenation;

[0055] S4, calculate and obtain the visual-text compatible spatial feature set of the first category {f multi} and the second category clothing name feature set {f name}’s neighbor feature set, such as Figure 2 As shown in (2b), the following specific steps are included:

[0056] S4.1, using the K nearest neighbor algorithm to calculate the visual-text compatible spatial feature set {f multi}nearly n neighbors, recorded as the first neighbor feature set

[0057] S4.2, using the cosine similarity method to calculate the similarity with the clothing name feature set {f name}nearly η neighbors, recorded as the second neighbor feature set

[0058] S5. Calculate the clothing multimodal relationship measurement matrix MRM, such as Figure 2 As shown in (2c), the following specific steps are included:

[0059] S5.1. Consider the selected garment as the user and the remaining available garments as the items to be selected. Count the garment pairing records in the dataset to obtain the garment interaction matrix M_Inter. The garment interaction matrix consists of interaction records, which are garment pairing records in the available fashion dataset. (For example, a pink coat has pairing records with jeans and a short skirt in the dataset.)

[0060] S5.2. Using the clothing interaction matrix M_Inter and the two neighbor feature sets from step S4, we obtain the clothing multimodal relationship measurement matrix MRM. The MRM matrix describes the interaction probability between the selected clothing item and its paired positive clothing neighbors. This means that for a selected clothing item, the more interactions it has with its paired neighbors, the higher the relationship score between the two items. By applying the MRM matrix to the clothing recommendation system, the model can balance the accuracy and rationality of recommendations while also avoiding issues like cold starts. This step proposes three matrix combinations:

[0061] Matrix combination method 1: combination With the clothing interaction matrix M_Inter, the semantic relationship measurement matrix M_m is calculated as the MRM matrix. M_m is based on f multi Calculation has the ability to accurately understand the semantics of clothing attributes and has a certain ability to capture preferences.

[0062] Matrix combination method 2: First, probability fusion and Combined with the M Inter matrix, the mixed relationship measurement matrix M_mn is calculated as the MRM matrix. M_mn is based on fmulti and f name Computation, with the ability to understand clothing attributes and capture preferences.

[0063] Matrix combination method 3: combination With the clothing interaction matrix M_Inter, the interaction affinity measurement matrix M_n is calculated. Based on f name Calculation can effectively predict paired clothing, thereby capturing clothing pairing preferences. After eliminating abnormal recommended clothing from M_n, the two matrices M_m and M_n are fused with a certain probability coefficient or a probability matrix calculated by a neural network to obtain the MRM matrix.

[0064] S6. Combine the three feature sets output from step S3 with the clothing multimodal relationship measurement matrix MRM calculated in step S5 to calculate a rational clothing recommendation loss function based on BPR, giving the clothing recommendation system the ability to understand fashion. This step can make up for the inherent limitation of BPR that ignores the correlation between clothing. The combination of the feature set and the clothing multimodal relationship measurement matrix MRM can be divided into the following ways:

[0065] Feature and matrix combination method 1: The MRM matrix calculated based on matrix combination method 1 acts on the first type of visual-text compatible space feature set {f multi}Calculated BPR loss function L bpr1 ;

[0066] Feature and matrix combination method 2: The MRM matrix calculated based on matrix combination method 1 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0067] Feature and matrix combination method 3: Based on the MRM matrix calculated by matrix combination method 1, it acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0068] Feature and matrix combination method 4: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 ;

[0069] Feature and matrix combination method 5: The MRM matrix calculated based on matrix combination method 2 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0070] Feature and matrix combination method 6: The MRM matrix calculated based on matrix combination method 2 acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0071] Feature and matrix combination method 7: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 ;

[0072] Feature and matrix combination method 8: The MRM matrix calculated based on matrix combination method 3 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0073] Feature and matrix combination method 9: The MRM matrix calculated based on matrix combination method 3 acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0074] Feature and matrix combination method 10: The MRM matrix calculated based on the matrix combination method 1 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0075] Feature and matrix combination method 11: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0076] Feature and matrix combination method 12: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0077] Feature and matrix combination method 13: Based on the M_m matrix calculated by matrix combination method 1, it acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 ; and, the M_n matrix calculated based on the matrix combination method 2 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 .

[0078] S7. Obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and based on the trade-off result, reduce the MCAR bias in the fashion dataset through exposure bias reduction technology, thereby alleviating the impact of data sparsity; this step alleviates the impact of exposure bias caused by data sparsity and ensures the rationality of the recommendation model; this step enables the clothing recommendation system to take into account both the rationality and accuracy of the recommendation results.

[0079] In the step 1, the specific processing of sampling negative sample data is described as follows: the clothing recommendation model usually randomly selects negative samples from similar clothing or adopts the top-K method to collect negative samples, which will cause the data set to be too simple or a false negative sample problem. The present invention makes full use of the unique subdivision characteristics of clothing and subdivides clothing into: coats, dresses, jeans, pants (non-jeans), sweaters, jumpsuits, shirts, shorts, and skirts. Secondly, the paired clothing of the selected clothing and the neighbors of the paired clothing are eliminated. Then, the top-K neighbors of the given clothing and the remaining similar clothing are calculated, and negative samples with higher difficulty are selected from them. The randomly sampled clothing and the negative samples sampled by the above process are mixed in a certain proportion to obtain higher quality triplet data.

[0080] Example 2

[0081] like Figure 2 As shown, the present invention provides a BPR clothing recommendation system based on multimodal information, including:

[0082] The negative sample sampling module 100 for clothing sub-classification is used to sample clothing sub-classification source data to generate negative sample data. This module fully utilizes the unique sub-classification characteristics of clothing and eliminates paired clothing and its neighbors. It then calculates the top-K neighbors of the given clothing and the remaining similar clothing, selecting high-difficulty negative samples from these. Randomly sampled clothing and negative samples sampled in the above process are mixed in a certain ratio to obtain high-quality triplet data (selected clothing, paired positive sample clothing, unpaired negative sample clothing).

[0083] The multimodal feature extraction module 200 is used to extract multimodal clothing features based on the triple data. The multimodal feature extraction module 200 further includes a visual-text compatible space (VT-CS) pre-training module 201 and a clothing name encoding module 202. The visual-text compatible space (VT-CS) pre-training module 201 is used to extract visual-text compatible space features f multi The clothing name encoding module 202 is used to extract clothing name features f name , the encoder can be an embedding, MLP or graph convolutional network.

[0084] The feature combination module 300 is used to output three types of feature sets of the multimodal clothing features, including the first type of visual-text compatible space feature set {f multi}, the second category of clothing name feature set {f name}、The third type of combined feature set {f multi}+{f name}, where + represents feature concatenation.

[0085] Neighbor calculation module 400, used to calculate the first type of visual-text compatible spatial feature set {f multi} and the second category clothing name feature set {f name}, specifically including using the K nearest neighbor algorithm to calculate the visual-text compatible spatial feature set {f multi}nearly n neighbors, recorded as the first neighbor feature set and use the cosine similarity method to calculate the clothing name feature set {f name}nearly η neighbors, recorded as the second neighbor feature set

[0086] The clothing multimodal relationship measurement matrix (MRM) calculation module 500 is used to calculate the clothing multimodal relationship measurement matrix (MRM). Specifically, this involves counting the pairing records between clothing items to obtain M_Inter, and combining it with the two neighbor feature sets calculated by the neighbor calculation module to obtain the MRM matrix using three matrix combinations. This matrix can help the clothing recommendation system better understand and utilize the semantic attribute relationships and interaction information between clothing items, thereby improving the accuracy and personalization of the system.

[0087] The BPR-based rationality recommendation loss function calculation module 600 is used to calculate the BPR-based rationality clothing recommendation loss function based on the combination of the feature set and the clothing multimodal relationship measurement matrix MRM; thereby giving the clothing recommendation system fashion understanding capabilities and significantly improving the rationality and transparency of the clothing recommendation model.

[0088] The data sparsity impact reduction module 700 is used to obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and reduces the MCAR bias in the fashion dataset through exposure bias reduction technology based on the trade-off result. This module enables the clothing recommendation system to take into account both the rationality and accuracy of the recommendation results.

[0089] The neighbor calculation module 400 further includes using a K-nearest neighbor algorithm to calculate the visual-text compatible spatial feature set {f multi}nearly n neighbors, recorded as the first neighbor feature set The cosine similarity method is used to calculate the similarity between the clothing name feature set {f name}nearly η neighbors, recorded as the second neighbor feature set

[0090] The clothing multimodal relationship measurement matrix MRM calculation module 500 further includes: counting the pairing records between clothing to obtain the clothing interaction matrix M_Inter; using the clothing interaction matrix M_Inter and the two neighbor feature sets of the neighbor calculation module 400 to obtain the clothing multimodal relationship measurement matrix MRM; the MRM matrix is ​​used to describe the interaction probability of the neighbors of the clothing paired with the selected clothing and the target clothing, and the three matrix combination methods are:

[0091] Matrix combination method 1: combination With the clothing interaction matrix M_Inter, the semantic relationship measurement matrix M_m is calculated as the MRM matrix;

[0092] Matrix combination method 2: First, probability fusion and Combined with the M_Inter matrix, the mixed relationship measurement matrix M_mn is calculated as the MRM matrix;

[0093] Matrix combination method 3: combination The interaction affinity measurement matrix M_n is calculated with the clothing interaction matrix M_Inter, and the two matrices M_m and M_n are fused with a certain probability coefficient or a probability matrix calculated by a neural network to obtain the MRM matrix.

[0094] The combination of the feature set of the BPR-based rationality recommendation loss function calculation module 600 and the clothing multimodal relationship measurement matrix MRM is as follows:

[0095] Feature and matrix combination method 1: The MRM matrix calculated based on matrix combination method 1 acts on the first type of visual-text compatible space feature set {f multi}Calculated BPR loss function L bpr1 ;

[0096] Feature and matrix combination method 2: The MRM matrix calculated based on matrix combination method 1 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0097] Feature and matrix combination method 3: Based on the MRM matrix calculated by matrix combination method 1, it acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0098] Feature and matrix combination method 4: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 ;

[0099] Feature and matrix combination method 5: The MRM matrix calculated based on matrix combination method 2 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0100] Feature and matrix combination method 6: The MRM matrix calculated based on matrix combination method 2 acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0101] Feature and matrix combination method 7: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category {f mu lti} calculated BPR loss function L bpr1 ;

[0102] Feature and matrix combination method 8: The MRM matrix calculated based on matrix combination method 3 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0103] Feature and matrix combination method 9: The MRM matrix calculated based on matrix combination method 3 acts on the combined feature set based on the third type {f multi}+{f name}Calculated BPR loss function L bpr3 ;

[0104] Feature and matrix combination method 10: The MRM matrix calculated based on the matrix combination method 1 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0105] Feature and matrix combination method 11: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0106] Feature and matrix combination method 12: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 And the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 ;

[0107] Feature and matrix combination method 13: Based on the M_m matrix calculated by matrix combination method 1, it acts on the visual-text compatible space feature set based on the first category {f multi}Calculated BPR loss function L bpr1 ; and, the M_n matrix calculated based on the matrix combination method 2 acts on the clothing name feature set based on the second category {f name}Calculated BPR loss function L bpr2 .

[0108] Among them, L bpr1 Refers to the loss function calculated based on the first type of features, L bpr2 Refers to the loss function calculated based on the second type of features, L bpr3 Refers to the loss function calculated based on the third type of features.

[0109] In summary, the present invention implements the following technical solutions:

[0110] 1) Fully utilizing clothing multimodal information, effectively combining clothing semantic relationship information and historical interaction information, designing a reasonable clothing recommendation loss function based on BPR, alleviating the impact of MCAR bias, and effectively obtaining high-quality multimodal features and negative samples, can provide accurate and reasonable clothing recommendation results;

[0111] 2) This paper introduces the MRM matrix and proposes three matrix combination methods by comprehensively considering the relationship between clothing semantic relationship information and clothing historical interaction information; making the BPR-based clothing recommendation system suitable for versatile styles of clothing and new clothing.

[0112] 3) This paper proposes a rationality loss function based on BPR, with 13 possible combinations. This method effectively overcomes the inherent limitation of BPR, which ignores the correlation between garments, improves the rationality and transparency of the clothing recommendation model, and empowers the clothing recommendation system with fashion understanding.

[0113] 4) The present invention observes that data sparsity affects the rationality and accuracy of the recommendation model. By considering the exposure tendency and association tendency between clothing, the impact of sparsity is alleviated, so that the BPR-based recommendation system can take into account both the rationality and accuracy of the recommendation results.

[0114] 5) Leveraging CLIP's powerful multimodal feature extraction capabilities, this paper employs a comparative image-text pre-training approach to effectively extract highly aligned visual and textual features from the clothing domain, avoiding ambiguity between visual and textual descriptions and fully leveraging the advantages of multimodal features. The model also boasts low computational complexity, meaning it only requires a single training step to function as a multimodal clothing feature extractor.

[0115] 6) The present invention makes full use of clothing subdivision categories, eliminates the selected clothing pairings and the neighbors of the paired clothing, and uses the top-K method to extract high-quality negative samples of the same category, and combines them with randomly sampled negative samples in a certain proportion to effectively construct high-quality triplet data.

[0116] The present invention comprehensively considers the combined relationship between clothing semantic relationship information and clothing historical interaction information; adopts a reasonable and effective way to apply the multimodal interaction matrix to the BPR loss function, so that the clothing recommendation model takes into account both rationality and accuracy, and is more suitable for all-match style clothing and new clothing; through the data sparsity impact mitigation module, the problem of fashion data exposure bias is alleviated. There is a significant difference in physical meaning between clothing visual features and text features. In order to give full play to the advantages of multimodal features, the present invention uses a comparative language-image pre-training method to extract visual-text compatible spatial features in the clothing field. The present invention fully combines clothing subdivision categories and general negative sample sampling methods to construct triple data (selected clothing, paired positive sample clothing, unpaired negative sample clothing).

[0117] like Figure 4The figure below is a block diagram of a specific embodiment of a BPR clothing recommendation method and system based on multimodal information. The "<" in the figure indicates compatibility, and an open direction indicates greater compatibility. This embodiment focuses on a user selecting a top, and the recommendation system provides reasonable bottom recommendations.

[0118] 1. Given a set of clothing pairing data, the upper left part is shown, denoted as (t, b + , b - ), where t represents the top, b + Indicates paired (interactive) download, b - Indicates unpaired (uninteracted) bottoms. Each piece of clothing corresponds to a 2D image and a text description of the clothing type, pattern, material and other attributes. Figure 6 The negative sample sampling module for clothing sub-classification is shown. According to the triple data, f is extracted through the multimodal feature module. multi Features and clothing name encoding features f name .

[0119] (1) The present invention adopts a visual encoder E that is consistent with the CLIP model v (.) and the text encoder Ec(.), which is trained contrastively to predict the correct pairing for the batch. Image-text pairs (I i , C i ) are processed by their respective encoders, where i represents a given uplink or downlink. Each encoder is then connected to a 2-layer fully connected (FC) network with the same output dimension to project the obtained features into a shared visual-textual feature space, as shown in Formula (1).

[0120]

[0121] Among them, W v and W c Represents the weights of the fully connected layers for images and text, with dimensions of [(512, 512), (512, 256)] and [(1024, 512), (512, 256)] respectively. and Denote the visual features and text features of a given garment i, respectively. The loss function is the average cross entropy between the predicted image-text pair and the target image-text pair, where the target pair is reflected by the main diagonal. Finally, the visual-text compatible space feature vector f is obtained multi .

[0122] (2) The clothing name encoding module consists of a two-layer perceptron (MLP), f name The output dimension is 50.

[0123] 2. This embodiment adopts the third type of combined feature set {f multi}+{f name}.

[0124] 3. Based on visual-text compatible spatial features f multi and the feature vector f of the clothing name name , use K nearest neighbor algorithm to calculate the given clothing f multi The 10 closest first neighbor feature sets Use the cosine similarity method to calculate the similarity with the given clothing f name The 10 most similar second neighbor feature sets In this embodiment, it is assumed that η=10. Figure 7 The 5 closest neighbors of a given garment are shown, demonstrating the effectiveness of features extracted by VT-CS.

[0125] 4. Calculate the top-bottom multimodal relationship measurement (MRM) matrix.

[0126] (1) Count the pairing records between the upper and lower parts to obtain the upper and lower part interaction matrix M_Inter.

[0127] (2) This paper defines MRM as the probability that a neighbor of a top and its paired bottom also has an interaction record. The semantic relationship measurement matrix M_m accurately understands the semantics of clothing attributes; the interaction affinity measurement matrix M_n effectively predicts interactive clothing; and the mixed relationship measurement matrix M_mn takes into account both understanding ability and prediction accuracy. The MRM matrix ranges from [0, 1], with larger values ​​indicating stronger interpretability between clothing items.

[0128] Matrix combination 1: The semantic relationship measurement matrix M_m is used as the clothing multimodal relationship measurement MRM.

[0129] Matrix combination 2: The mixed relationship measurement matrix M_mn is used as the clothing multimodal relationship measurement MRM.

[0130] Matrix combination 3: As shown in formulas (2-1) and (2-2), the abnormal recommended clothing of the interaction affinity measurement matrix M_n is eliminated. Here, clothing with an explainable score γ less than 0.05 is screened out, and the semantic relationship measurement matrix M_m and the interaction affinity measurement matrix M_n are fused with probabilities α and β, or with a probability matrix W based on neural network training. α and W β Fusion. This example uses a probabilistic fusion method and sets α = β = 1.0.

[0131] MRM=α×M_m+β×(M_n-γ) (2-1)

[0132] MRM=Wα ×M_m+W β ×(M_n-γ) (2-2)

[0133] 5. Reasonable clothing recommendation based on BPR. Figure 5 Figure 6 shows a schematic diagram of a BPR-based rationality recommendation loss function calculation module 600. This process enables the clothing recommendation system to generate a recommendation list for a top, with higher-ranked bottoms being more reasonable than lower-ranked bottoms. There are several ways to combine features and matrices:

[0134] The first 9 combinations are calculated using formulas (3)-(5):

[0135] L reason_bpr＝ M reason ×L bpr (3)

[0136]

[0137]

[0138] For the last four combinations, change formula (3) to formula (6):

[0139] L reason_bpr =M reason1 ×L bpr1 +λ×M reason2 ×L bpr2 (6)

[0140] Among them, L bpr According to the input characteristics, the value L bpr1 , L bpr2 and L bpr3 D is the training data, |D| represents the number of training data, θ Ω is a model with parameter Ω that quantifies the relationship between the top t and the bottom b. + The preference exceeds the degree of unpaired lower garment b-, is the Sigmoid function, λ is the weight penalty term, ||.|| 2 is L2 regularization, ft represents the upper feature vector, which can be f multi 、f name Or it can be spliced ​​together from the two. They represent the feature vectors of paired and unpaired sub-items respectively, and the calculation process is the same as f t The subscript tb in formula (4) + 、tb - Represents top and paired bottom and top and unpaired bottom, (t, b + , b - )∈D.

[0141] 6. Alleviate the impact of data sparsity. This example uses the inverse probability score (IPS) to alleviate the MCAR bias commonly found in fashion datasets.

[0142]

[0143] Y tb =O tb *R tb (8)

[0144] Among them, Y tb Measures the overall correlation between the top item t and the bottom item b. tb and R tb Respectively represent whether the given upper item t and the lower item b are visible to the upper item or related to the upper item. tb ~Ber(θ tb ), R rb ~Ber(γ tb ), Ber(.) is Bernoulli distribution.

[0145] In this example, the ninth parameter setting in step S5 of the technical solution was used. The VT-CS pre-training module was trained for 20 epochs, while the BPR-based rational clothing pairing model was trained for 40 epochs. Both modules used the Adam optimizer and a learning rate scheduler, with an initial learning rate of 1e-4 and a reduction rate of 0.1 after 10 epochs.

[0146] This example conducts experiments on the open source fashion dataset Exp-fashion (excluding abnormal data with less than 5 interactions), and compares the proposed model with several baseline models, including BPR, VBPR, TBPR (based on VBPR, replacing visual information with textual information), and VTBPR (based on VBPR, integrating textual information). For the above methods, this example uses ResNet18 to extract visual features and uses the CLIP text encoder to encode textual information. The rationality of the model recommendation is evaluated using weighted MEP (WMEP@K), which evaluates the rationality of the model recommendation by calculating the average expected position of relevant items in the first K=i0 recommendations. The weighting factor gives higher importance to more relevant or more ideal items. The recommendation accuracy of the model is evaluated using AUC. For convenience, the following is a brief overview of the method proposed in this example:

[0147] (1) Multimodal features: The model combines clothing name encoding features and visual-text compatible space features.

[0148] (2) Fusion MRM matrix: Based on multimodality, the MRM matrix is ​​incorporated into it, as defined in formulas (3)-(6).

[0149] (3) Remove MCAR bias: Based on the previous method, use formulas (7) and (8) to reduce the impact of exposure bias.

[0150] like Figure 6 As shown in the figure, in the example of the negative sample sampling module combined with clothing subclassification, for the clothing subclassification source data, the paired clothing and the neighbors of the paired clothing of the selected clothing are first eliminated, and then random sampling of similar samples and Top-K sampling of similar samples are performed, and finally probabilistic combination is performed.

[0151] Table 1 shows the AUC and WMEP@10 results. The proposed method achieves a 74.8% higher accuracy rate for removing MCAR bias than the BPR method. Furthermore, the proposed method achieves a higher WMEP@10 than the baseline method, demonstrating that the proposed clothing recommendation system can ensure the rationality of its recommendations while maintaining high accuracy.

[0152] As shown in Table 1, the AUC and WMEP@10 evaluation results of different algorithms on the open source fashion dataset are shown. The bold ones indicate the best results, and the underlined ones indicate the suboptimal results.

[0153] Table 1

[0154]

[0155]

[0156] It should be noted that although the present invention has been shown and described with reference to specific exemplary embodiments of the present invention, those skilled in the art should understand that the present invention is not limited to the above-mentioned embodiments and all kinds of changes to the present invention fall within the scope of protection of the present invention.

Claims

1. A BPR clothing recommendation method based on multimodal information, characterized in that: The method includes: S1. Combine clothing sub-classification source data sampling to obtain negative sample data, and construct triple data including selected clothing, paired positive sample clothing and unpaired negative sample clothing; S2, extracting multimodal clothing features based on the triplet data of step S1, S3, outputting three types of feature sets based on the multimodal clothing features of step S2, specifically: the first type of visual-text compatible spatial feature set , the second category of clothing name feature set , the third type of combined feature set ,in Represents feature splicing; S4. Calculate the visual-text compatible spatial feature set of the first category And the second category of clothing name feature set Neighbor feature set of ; S5. Calculating the clothing multimodal relationship measurement matrix MRM; further comprising: S5.

1. Count clothing pairing records and obtain the clothing interaction matrix M_Inter; S5.

2. Using the clothing interaction matrix M_Inter and the two neighbor feature sets in step S4, we obtain the clothing multimodal relationship measurement matrix MRM. The MRM matrix is ​​used to describe the interaction probability between the selected clothing and its paired positive sample clothing neighbors. The three matrix combinations are: Matrix combination method 1: combining the first neighbor feature set With the clothing interaction matrix M_Inter, the semantic relationship measurement matrix is ​​calculated As an MRM matrix; Matrix combination method 2: First, probability fusion of the first neighbor feature set and the second neighbor feature set , combined with the M_Inter matrix, the mixed relationship measurement matrix is ​​calculated As an MRM matrix; Matrix combination method 3: combining the second neighbor feature set With the clothing interaction matrix M_Inter, the interaction affinity measurement matrix is ​​calculated n, will and The two matrices are fused with a certain probability coefficient or a probability matrix calculated by a neural network to obtain the MRM matrix; S6, combining the three-category feature set outputted in step S3 and the clothing multimodal relationship measurement matrix MRM calculated in step S5, and calculating a rational clothing recommendation loss function based on BPR; S7. Obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and based on the trade-off result, reduce the MCAR bias in the fashion dataset through exposure reduction technology, thereby alleviating the impact of data sparsity.

2. The BPR clothing recommendation method based on multimodal information according to claim 1, characterized in that: The step S2 further comprises: S2.

1. Extracting visual-text compatible spatial features , wherein the visual-textual compatibility space feature Contains clothing attribute level information; S2.

2. Extract clothing name features ; Among them, the clothing name features Contains historical pairing information for clothing.

3. The BPR clothing recommendation method based on multimodal information according to claim 1, characterized in that: The step S2 further comprises: S4.

1. Calculate the feature set compatible with the visual-text space using the K-nearest neighbor algorithm Similar neighbors, recorded as the first neighbor feature set ; S4.

2. Use the cosine similarity method to calculate the feature set of the clothing name Similar neighbors, recorded as the second neighbor feature set .

4. The BPR clothing recommendation method based on multimodal information according to claim 1, characterized in that: The combination of the feature set of step S6 and the clothing multimodal relationship measurement matrix MRM includes: Feature and matrix combination method 1: The MRM matrix calculated based on matrix combination method 1 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 2: The MRM matrix calculated by matrix combination method 1 is applied to the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 3: Based on the MRM matrix calculated by matrix combination method 1, it acts on the combined feature set based on the third category Calculated BPR loss function ; Feature and matrix combination method 4: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 5: The MRM matrix calculated based on matrix combination method 2 acts on the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 6: The MRM matrix calculated based on matrix combination method 2 acts on the combined feature set based on the third category Computed ; Feature and matrix combination method 7: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 8: The MRM matrix calculated based on matrix combination method 3 is applied to the clothing name feature set based on the second category. Calculated BPR loss function ; Feature and Matrix Combination 9: The MRM matrix calculated based on Matrix Combination 3, acting on the combined feature set based on the third category Computed ; Feature and matrix combination method 10: The MRM matrix calculated based on matrix combination method 1 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 11: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category Computed and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 12: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category calculate and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 13: Based on the M_m matrix calculated by matrix combination method 1, it acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; and, calculated based on matrix combination method 2 n matrix, acting on the clothing name feature set of the second category Calculated BPR loss function .

5. A BPR clothing recommendation system based on multimodal information, characterized in that: include: A negative sample sampling module (100) for combining clothing sub-classification, for sampling clothing sub-classification source data to obtain negative sample data, and constructing triplet data including selected clothing, paired positive sample clothing, and unpaired negative sample clothing; A multimodal feature extraction module (200) is used to extract multimodal clothing features based on triplet data; A feature combination module (300) is used to output three types of feature sets of the multimodal clothing features, including a first type of visual-text compatible space feature set , the second category of clothing name feature set , the third type of combined feature set ,in Represents feature splicing; Neighbor calculation module (400), used to calculate and obtain the visual-text compatible spatial feature set of the first category And the second category of clothing name feature set Neighbor feature set of ; The clothing multimodal relationship measurement matrix MRM calculation module (500) is used to calculate the clothing multimodal relationship measurement matrix MRM; specifically, it includes counting the pairing records between clothing to obtain M_Inter, and combining the two neighbor feature sets calculated by the neighbor calculation module to obtain the MRM matrix according to three matrix combination methods; further includes: Count the clothing pairing records and obtain the clothing interaction matrix M_Inter; The clothing multimodal relationship measurement matrix MRM is obtained by using the clothing interaction matrix M_Inter and the two neighbor feature sets in step S4. The MRM matrix is ​​used to describe the interaction probability between the selected clothing and the paired positive sample clothing neighbors. The three matrix combinations are: Matrix combination method 1: combining the first neighbor feature set With the clothing interaction matrix M_Inter, the semantic relationship measurement matrix is ​​calculated As an MRM matrix; Matrix combination method 2: First, probability fusion of the first neighbor feature set and the second neighbor feature set , combined with the M_Inter matrix, the mixed relationship measurement matrix is ​​calculated As an MRM matrix; Matrix combination method 3: combining the second neighbor feature set With the clothing interaction matrix M_Inter, the interaction affinity measurement matrix is ​​calculated n, will and The two matrices are fused with a certain probability coefficient or a probability matrix calculated by a neural network to obtain the MRM matrix; A BPR rationality recommendation loss function calculation module (600) is used to calculate a BPR rationality clothing recommendation loss function based on a combination of a feature set and a clothing multimodal relationship measurement matrix MRM; The data sparsity impact reduction module (700) is used to obtain a trade-off result that comprehensively considers the exposure tendency of clothing and the association tendency between clothing, and reduce the MCAR bias in the fashion dataset through the exposure bias reduction technology based on the trade-off result.

6. A BPR clothing recommendation system considering multimodal information according to claim 5, characterized in that: The multimodal feature extraction module (200) further comprises a visual-text compatible space pre-training module (201) and a clothing name encoding module (202); the visual-text compatible space pre-training module (201) is used to extract visual-text compatible space features. , the visual-textual compatibility space feature Contains clothing attribute level information; the clothing name encoding module (202) is used to extract clothing name features , the clothing name features Contains historical pairing information for clothing.

7. A BPR clothing recommendation system considering multimodal information according to claim 5, characterized in that: The neighbor calculation module (400) further includes using a K-nearest neighbor algorithm to calculate the visual-text compatible spatial feature set. Similar neighbors, recorded as the first neighbor feature set ; Use the cosine similarity method to calculate the clothing name feature set Similar neighbors, recorded as the second neighbor feature set .

8. The BPR clothing recommendation system based on multimodal information according to claim 5, characterized in that: The clothing multimodal relationship measurement matrix MRM calculation module (500) further comprises: counting the pairing records between clothing to obtain the clothing interaction matrix M_Inter; using the clothing interaction matrix M_Inter and two neighbor feature sets, calculating the clothing multimodal relationship measurement matrix MRM; the MRM matrix is ​​used to describe the interaction probability between the selected clothing and the paired positive sample clothing neighbors, and the three matrix combination methods are: Feature and matrix combination method 1: The MRM matrix calculated based on matrix combination method 1 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 2: The MRM matrix calculated by matrix combination method 1 is applied to the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 3: Based on the MRM matrix calculated by matrix combination method 1, it acts on the combined feature set based on the third category Calculated BPR loss function ; Feature and matrix combination method 4: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 5: The MRM matrix calculated based on matrix combination method 2 acts on the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 6: The MRM matrix calculated based on matrix combination method 2 acts on the combined feature set based on the third category Computed ; Feature and matrix combination method 7: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; Feature and matrix combination method 8: The MRM matrix calculated based on matrix combination method 3 is applied to the clothing name feature set based on the second category. Calculated BPR loss function ; Feature and Matrix Combination 9: The MRM matrix calculated based on Matrix Combination 3, acting on the combined feature set based on the third category Computed ; Feature and matrix combination method 10: The MRM matrix calculated based on matrix combination method 1 acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 11: The MRM matrix calculated based on matrix combination method 2 acts on the visual-text compatible space feature set based on the first category calculate and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 12: The MRM matrix calculated based on matrix combination method 3 acts on the visual-text compatible space feature set based on the first category Computed and the clothing name feature set based on the second category Calculated BPR loss function ; Feature and matrix combination method 13: Based on the M_m matrix calculated by matrix combination method 1, it acts on the visual-text compatible space feature set based on the first category Calculated BPR loss function ; and, calculated based on matrix combination method 2 n matrix, acting on the clothing name feature set of the second category Calculated BPR loss function .

Images