Clothing matching recommendation method and system based on multiple modes

Abstract

The invention relates to the technical field of personalized garment recommendation, particularly provides a garment matching recommendation method and system based on multiple modes, and aims to solve the problem that in the prior art, the accuracy and personalization of garments recommended to a user by a garment recommendation system based on a single mode are low. According to the method, input text description and an original clothing image are respectively coded into text features and clothing image features; fusing the two features to form a fusion feature; and the multi-modal condition generative adversarial network generates a matched garment image set according to the fusion features, and screens out matched matched garment images in combination with user preference information. According to the method, the multi-modal data is input, and the text description and the original clothing image can be received and processed at the same time, so that the clothing matching recommendation system can more accurately understand the requirements of the user, the matching clothing which is higher in matching accuracy, better in quality, more in accordance with the requirements of the user and more personalized is generated, and the use experience of the user is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of personalized clothing recommendation, and in particular relates to a clothing matching recommendation method and system based on multimodality. Background Art

[0002] At present, clothing recommendation technology is basically based on a single-modal model (Image-to-Image), whose biggest advantage is that it can generate high-quality images and diverse data. However, the training process based on the single modality is complicated, which often requires a lot of computing resources and time. In addition, the single-modal model is prone to problems such as mode collapse, which makes the generator unable to effectively generate diverse clothing combinations, resulting in low accuracy of clothing recommendations and inability to make effective clothing recommendations for users.

[0003] To address these problems, this paper introduces a more complex multimodal input method to expand it into a multimodal model. This extension allows CGAN to establish associations between different modalities, thereby generating complex and high-quality multimodal data. Summary of the invention

[0004] The present invention provides a clothing matching recommendation method and system based on multimodality, aiming to solve the problem of low accuracy and personalization of clothing recommended to users by a clothing recommendation system based on single modality in the prior art.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a clothing matching recommendation method based on multimodality, comprising the following steps:

[0007] S100, receiving an input text description and an original clothing image;

[0008] S200, encoding the text description through a multi-head attention mechanism to obtain text features;

[0009] S300, encoding the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features;

[0010] S400, fusing the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature;

[0011] S500, inputting the fusion features into a multimodal conditional generative adversarial network, wherein the multimodal conditional generative adversarial network generates a set of matching clothing images according to the fusion features;

[0012] S600, calculating the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorting the matching clothing images in the matching clothing image set according to the similarity; obtaining user preference information, and screening out the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information.

[0013] A further solution: in the step S200, the multi-head attention mechanism transforms the text description into a query, a key and a value through h groups of linear projections;

[0014] The query, key and value are all input into the attention pool, and the outputs of the attention pool are spliced ​​together; the spliced ​​outputs of the attention pool are transformed through a linear projection to obtain text features.

[0015] Based on the above scheme, in the multi-head attention mechanism, the query, key and value are transformed through h groups of linear projections, which can capture the information in the text description from multiple angles, thereby focusing on and extracting different features in the text description.

[0016] In a further solution, in step S300, the multi-scale adaptive convolution network includes an adaptive convolution mechanism and a fusion multi-scale feature operation;

[0017] The adaptive convolution mechanism convolves the original clothing image to obtain a multi-scale feature map of the original clothing image;

[0018] The multi-scale feature fusion operation concatenates and fuses the multi-scale feature maps together to obtain the clothing image features.

[0019] Based on the above scheme, the multi-scale adaptive convolution network combines the adaptive convolution mechanism and the fusion multi-scale feature operation, which can improve the flexibility and adaptability of the convolution operation and the processing ability of different scales and complex image information, thereby improving the accuracy and flexibility of the clothing image feature extraction.

[0020] Further solution: The steps of obtaining the multi-scale feature map of the original clothing image are:

[0021] The adaptive convolution mechanism extracts the feature representation F(I) of the original clothing image through a convolution layer;

[0022] According to the feature representation F(I) of the original clothing image, an adaptive convolution kernel K is generated through a generative network θ(I) ;

[0023] Use the K θ(I)Perform a convolution operation on the original clothing image to obtain a multi-scale feature map F of the original clothing image θ (I);

[0024] Among them, using the K θ(I) The formula used for the convolution operation on the original clothing image is:

[0025] F θ (I) = I*K θ(I) ;

[0026] The I represents the original clothing image.

[0027] Based on the above scheme, the adaptive convolution kernel K is generated by the generative network θ(I) The adaptive convolution mechanism can dynamically adjust the shape and size of the convolution kernel according to the feature representation F(I) of the original clothing image, making the feature extraction more targeted and effectively improving the feature capture capability of the original clothing image.

[0028] Further solution: In the multi-scale feature fusion operation, a multi-scale feature map F of the original clothing image is extracted. θ Feature maps F of different scales in (I) σ1 (I) F σ2 (I) F σ3 (I)...F σn (I);

[0029] The extracted feature maps F of different scales σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) performing splicing and fusion to obtain the clothing image features;

[0030] Among them, the feature maps F of different scales are extracted σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) The formula used for splicing and fusion is:

[0031] F MSACN (I)=Fusion(F σ1 (I) F σ2 (I) F σ3 (I)...F σn (I));

[0032] The F MSACN (I) represents clothing image features; the Fusion() represents a splicing fusion operation.

[0033] Based on the above scheme, the extraction and fusion of multi-scale feature maps in the fusion multi-scale feature operation can significantly improve the quality and effect of the clothing image features.

[0034] Further solution: In the step S400, the text feature and the clothing image feature are fused through a cross-modal attention mechanism, and the step of obtaining the fused feature includes:

[0035] S401, defining the text features as The clothing image features are defined as Wherein, M represents the number of text features, and d t represents the dimension of each of the text features, N represents the number of clothing image features, and d p represents the dimension of each clothing image feature; R represents a real number set;

[0036] S402, using the clothing image feature as a query and the text feature as a key and a value, calculating the attention weight between the text feature and the clothing image feature;

[0037] S403 performs weighted averaging on the clothing image features according to the calculated attention weights, integrates the text features into the image features, and obtains the fusion features.

[0038] Based on the above scheme, the cross-modal attention mechanism effectively combines the text features and the clothing image features together, so that the multimodal conditional generative adversarial network can generate more accurate matching clothing images based on the fused features.

[0039] Further solution: In step S402, the clothing image feature is used as a query, the text feature is used as a key and a value, and the formula used to calculate the attention weight between the text feature and the clothing image feature is:

[0040]

[0041] Wherein, Q=PW Q A query matrix representing the image features, represents the weight matrix of the query, the d Q represents the dimension of the query; K = TW K A key matrix representing the text features, represents the key weight matrix, the d K Represents the dimension of the key; V = TW V A matrix of values ​​representing the features of the text, The weight matrix representing the value, the dV represents the dimension of the value; the R indicates that the elements of the matrix belong to the real number set; the softmax() indicates a normalization function;

[0042] In step S403, the clothing image features are weighted averaged according to the calculated attention weights, and the text features are integrated into the image features. The formula used to obtain the fusion features is:

[0043] P fused =Attention(P,T,T);

[0044] Among them, the P fused represents the fusion feature; the Attention() represents the attention weight, the P represents the clothing image feature; and the T represents the text feature.

[0045] Further solution: In the step S500, the multimodal conditional generative adversarial network is obtained through training; the multimodal conditional generative adversarial network generator and discriminator;

[0046] The steps of training the multimodal conditional generative adversarial network include:

[0047] Collecting training clothing images and text information, and inputting the training clothing images and text information into the multimodal conditional generative adversarial network;

[0048] The generator receives the training clothing image, generates a matching clothing image set according to the training clothing image, and inputs the matching clothing image set into the discriminator;

[0049] The discriminator determines whether the matching clothing image set matches the text information and the training clothing image;

[0050] If the judgment result is mismatch, the discriminator feeds back the judgment result to the generator, and the generator continues to generate a set of matching clothing images according to the judgment result; until the discriminator judges that the result is a match; and a trained multimodal conditional generative adversarial network is obtained.

[0051] Based on the above scheme, by training the multimodal conditional generative adversarial network, the multimodal conditional generative adversarial network is continuously learned and optimized, thereby improving the generation quality of the multimodal conditional generative adversarial network.

[0052] A further solution: the generator and the discriminator in the multimodal conditional generative adversarial network both adopt a perceptual adversarial loss function;

[0053] The loss calculation formula of the generator is:

[0054]

[0055] Among them, the z~p z (z) represents the noise distribution p z The random noise (z) sampled in c (c) represents the conditional distribution p c The condition information (c) sampled in ; the G(z, c) represents the matching clothing image generated by the generator according to the noise (z) and the condition (c); the D(G(z, c), c) represents the probability that the discriminator judges that the matching clothing image generated by the generator matches; the φ represents a pre-trained convolutional neural network for extracting features; the x target represents the target clothing image; the γ represents the weight coefficient of the perceptual loss;

[0056] The loss calculation formula of the discriminator is:

[0057]

[0058] Among them, the x~p data (x) represents a real sample (x) sampled from the text description and the original clothing image; represents the judgment of the discriminator on the sample (x) and the condition (c); Represents the discriminator's judgment on the matching clothing image G(z, c) and condition (c) generated by the generator.

[0059] Based on the above scheme, the perceptual adversarial loss function promotes the generator to generate more accurate and realistic recommended clothing images by utilizing the perceptual network, thereby further improving the quality of the recommended clothing images generated by the generator.

[0060] In a second aspect, the present invention provides a clothing collocation recommendation system based on multimodality, which is used to execute any clothing collocation recommendation method based on multimodality as described in the first aspect; the clothing collocation recommendation system comprises:

[0061] The multimodal data input module receives input text description and original clothing image; encodes the text description through a multi-head attention mechanism to obtain text features; encodes the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features;

[0062] A feature fusion module, which fuses the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature;

[0063] A multimodal conditional generative adversarial module, comprising a multimodal conditional generative adversarial network; the multimodal conditional generative adversarial network generates a plurality of matching clothing images according to the fusion features to obtain a matching clothing image set;

[0064] A user information input module, used to receive user preference information uploaded by users;

[0065] The clothing recommendation module calculates the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorts the matching clothing images in the matching clothing image set according to the similarity; obtains user preference information, and selects the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information in the user information input module.

[0066] The beneficial effects of the present invention are:

[0067] 1. The multimodal clothing matching recommendation system of the present invention is designed with multimodal data input, which can simultaneously receive and process the text description and the original clothing image, and fuse the two kinds of information together, so that the clothing matching recommendation system can understand the user's needs more accurately, thereby generating matching clothing with higher matching accuracy, better quality, more in line with user needs and more personalized, thereby improving the user experience.

[0068] 2. The present invention obtains the text features through a multi-head attention mechanism, and obtains the clothing image features through the multi-scale adaptive convolutional network, so that the clothing matching recommendation system can recognize and process text and images; the text features and the clothing image features are fused together through a cross-modal attention mechanism, so that the clothing matching recommendation system can better associate information of different modalities; the multimodal conditional generative adversarial network generates more accurate, better quality, and more matching matching clothing images according to the fused features; finally, combined with the user's preference information, the clothing matching recommendation system can recommend matching clothing that is more in line with the user's personal style, thereby improving the personalization of matching clothing, and thus helping users save time in choosing clothing and improving shopping experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1It is a flowchart of a clothing matching recommendation method based on multimodality in the present invention;

[0071] Figure 2 It is a schematic diagram of the workflow of the multi-head attention mechanism in the present invention;

[0072] Figure 3 It is a structural schematic diagram of a clothing matching recommendation system based on multimodality in the present invention;

[0073] Figure 4 is a schematic diagram of black suit pants input by the user in the case of the present invention;

[0074] Figure 5 It is a schematic diagram of a short-sleeved shirt generated by the clothing matching recommendation system in the case of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present invention.

[0076] Embodiment 1;

[0077] like Figure 1 As shown, this embodiment provides a clothing matching recommendation method based on multimodality, comprising the following steps:

[0078] S100, receiving an input text description and an original clothing image;

[0079] S200, encoding the text description through a multi-head attention mechanism to obtain text features;

[0080] S300, encoding the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features;

[0081] S400, fusing the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature;

[0082] S500, inputting the fusion features into a multimodal conditional generative adversarial network, wherein the multimodal conditional generative adversarial network generates a set of matching clothing images according to the fusion features;

[0083] S600, calculating the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorting the matching clothing images in the matching clothing image set according to the similarity; obtaining user preference information, and screening out the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information.

[0084] A more specific example of the above solution is that the text description can be in natural language, for example: the outdoor temperature today is 10 degrees.

[0085] like Figure 2 As shown, a more specific example of step S200 in the above scheme is: in step S200, the multi-head attention mechanism transforms the text description into a query (Q), a key (K) and a value (V) through h groups of linear projections (Linear);

[0086] The query, key and value are all input into the scaled dot-product attention, and the outputs of the scaled dot-product attention are concatenated together; the concatenated outputs of the scaled dot-product attention are transformed through a linear projection to obtain text features.

[0087] Furthermore, define the query key value Among them, the d Q represents the dimension of the query Q matrix; the d K represents the dimension of the bond matrix; the d V represents the dimension of the matrix V; said R represents that the elements of the matrix belong to the set of real numbers;

[0088] Each group of linear projections contains i (i=1, 2...h) attention heads, and the calculation method of each attention head hi is:

[0089]

[0090] Among them, the All represent learnable parameters. represents the query weight matrix of the i-th head; represents the key weight matrix of the i-th head; represents the value weight matrix of the i-th head; R represents a set of real numbers; f represents a function of attention aggregation;

[0091] The output of the spliced ​​attention convergence needs to be passed through a linear projection, which corresponds to the result of connecting h attention heads. Therefore, the learnable parameters of this linear projection are Among them, P o represents the output dimension after a linear projection; h represents the number of heads; P v represents the output dimension of each head after a linear projection; R represents a set of real numbers.

[0092] A more specific example of step S300 in the above solution is: in step S300, the multi-scale adaptive convolution network includes an adaptive convolution mechanism and a fusion multi-scale feature operation;

[0093] The adaptive convolution mechanism convolves the original clothing image to obtain a multi-scale feature map of the original clothing image;

[0094] The multi-scale feature fusion operation concatenates and fuses the multi-scale feature maps together to obtain the clothing image features.

[0095] On the basis of the above scheme, the adaptive convolution mechanism convolves the original clothing image to obtain a multi-scale feature map of the original clothing image in the following steps:

[0096] The adaptive convolution mechanism extracts the feature representation F(I) of the original clothing image through a convolution layer;

[0097] According to the feature representation F(I) of the original clothing image, an adaptive convolution kernel K is generated by a generative network (e.g., a small neural network) θ(I) ;

[0098] Use the K θ(I) Performing a convolution operation on the original clothing image to obtain a multi-scale feature map Fθ(I) of the original clothing image;

[0099] Among them, using the K θ(I) The formula used for the convolution operation on the original clothing image is:

[0100] F θ (I) = I*K θ(I) ;

[0101] The I represents the original clothing image.

[0102] Based on the above scheme, in the multi-scale feature fusion operation, a multi-scale feature map F of the original clothing image is extracted. θ Feature maps F of different scales in (I) σ1 (I) F σ2 (I) F σ3 (I)...F σn (I);

[0103] The extracted feature maps F of different scales σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) performing splicing and fusion to obtain the clothing image features;

[0104] Among them, the feature maps F of different scales are extracted σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) The formula used for splicing and fusion is:

[0105] F MSACN (I)=Fusion(F σ1 (I) F σ2 (I) F σ3 (I)...F σn (I));

[0106] The F MSACN (I) represents clothing image features; the Fusion() represents a splicing fusion operation.

[0107] A more specific example of step S400 in the above solution is: in step S400, the text feature and the clothing image feature are fused through a cross-modal attention mechanism to obtain the fused feature, including:

[0108] S401, defining the text features as The clothing image features are defined as Wherein, M represents the number of text features, and d t represents the dimension of each of the text features, N represents the number of clothing image features, and d p represents the dimension of each clothing image feature; R represents a real number set;

[0109] S402, using the clothing image feature as a query and the text feature as a key and a value, calculating the attention weight between the text feature and the clothing image feature;

[0110] S403 performs weighted averaging on the clothing image features according to the calculated attention weights, integrates the text features into the image features, and obtains the fusion features.

[0111] A more specific example of the above solution is: in step S402, the clothing image feature is used as a query, the text feature is used as a key and a value, and the formula used to calculate the attention weight between the text feature and the clothing image feature is:

[0112]

[0113] Wherein, Q=PW Q A query matrix representing the image features, represents the weight matrix of the query, the d Q represents the dimension of the query; K = TW K A key matrix representing the text features, represents the key weight matrix, the d K Represents the dimension of the key; V = TW V A matrix of values ​​representing the features of the text, The weight matrix representing the value, the d V represents the dimension of the value; the R indicates that the elements of the matrix belong to the real number set; the soffmax() indicates a normalization function;

[0114] In step S403, the clothing image features are weighted averaged according to the calculated attention weights, and the text features are integrated into the image features. The formula used to obtain the fusion features is:

[0115] P fused =Attention(P,T,T);

[0116] Among them, the P fused represents the fusion feature; the Attention() represents the attention weight, the P represents the clothing image feature; and the T represents the text feature.

[0117] It should be noted that, in addition to the above case where the clothing image feature is used as a query and the text feature is used as a key and value to obtain the fused feature, the text feature can also be used as a query and the image feature as a key and value to obtain the fused feature, the steps include:

[0118] The text features are defined as The clothing image features are defined as Wherein, M represents the number of text features, and d t represents the dimension of each of the text features, N represents the number of clothing image features, and d p represents the dimension of each clothing image feature; R represents a real number set;

[0119] Using the text feature as a query and the image feature as a key and a value, calculating an attention weight between the text feature and the clothing image feature;

[0120] According to the calculated attention weights, the clothing image features are weighted averaged, and the image features are integrated into the text features to obtain the fusion features.

[0121] Specifically, the text feature is used as a query, the image feature is used as a key and a value, and the formula used to calculate the attention weight between the text feature and the clothing image feature is:

[0122]

[0123] Wherein, Q = TW Q A query matrix representing the text features, represents the weight matrix of the query, the d Q Represents the dimension of the query; K = PW K A key matrix representing the image features, represents the key weight matrix, the d K Represents the dimension of the key; V = PW V A matrix of values ​​representing the features of the image, The weight matrix representing the value, the d V represents the dimension of the value; the R indicates that the elements of the matrix belong to the real number set; the softmax() indicates a normalization function;

[0124] According to the calculated attention weights, the clothing image features are weighted averaged, the image features are integrated into the text features, and the formula used to obtain the fusion features is:

[0125] T fused =Attention(T,P,P);

[0126] Among them, the T fused represents the fusion feature; the Attention() represents the attention weight, the P represents the clothing image feature; and the T represents the text feature.

[0127] A more specific example of step S500 in the above solution is: in step S500, the multimodal conditional generative adversarial network is obtained through training; the multimodal conditional generative adversarial network generator and discriminator;

[0128] The steps of training the multimodal conditional generative adversarial network include:

[0129] Collecting training clothing images and text information, and inputting the training clothing images and text information into the multimodal conditional generative adversarial network;

[0130] The generator receives the training clothing image, generates a matching clothing image set according to the training clothing image, and inputs the matching clothing image set into the discriminator;

[0131] The discriminator determines whether the matching clothing image set matches the text information and the training clothing image;

[0132] If the judgment result is mismatch, the discriminator feeds back the judgment result to the generator, and the generator continues to generate a set of matching clothing images according to the judgment result; until the discriminator judges that the result is a match; and a trained multimodal conditional generative adversarial network is obtained.

[0133] A more specific example of the above scheme is: the generator and the discriminator in the multimodal conditional generative adversarial network both adopt a perceptual adversarial loss function;

[0134] The loss calculation formula of the generator is:

[0135]

[0136] Among them, the z~p z (z) represents the noise distribution p z The random noise (z) sampled in c (c) represents the conditional distribution p c The condition information (c) sampled in ; the G(z, c) represents the matching clothing image generated by the generator according to the noise (z) and the condition (c); the D(G(z, c), c) represents the probability that the discriminator judges that the matching clothing image generated by the generator matches; the φ represents a pre-trained convolutional neural network for extracting features; the x target represents the target clothing image; the γ represents the weight coefficient of the perceptual loss;

[0137] The loss calculation formula of the discriminator is:

[0138]

[0139] Among them, the x~p data (x) represents a real sample (x) sampled from the text description and the original clothing image; represents the judgment of the discriminator on the sample (x) and the condition (c); Represents the discriminator's judgment on the matching clothing image G(z, c) and condition (c) generated by the generator.

[0140] A more specific example of step S600 in the above solution is: in step S600, the similarity between the matching clothing images in the matching clothing image set and the fusion feature is calculated by a cosine similarity measurement method.

[0141] Among them, the formula of the cosine similarity measurement method is: The A represents the feature of the matching clothing image; the B represents the fusion feature; the . represents the dot product; the ||A|| represents the norm of the feature of the matching clothing image, the The ||B|| represents the norm of the fusion feature, the

[0142] In addition, the N is set by the user and can be 1, 2, 3 or 4, etc.

[0143] It should be noted that, when the user preference information cannot be obtained, then in step S600, there is no need to filter the matching clothing images according to the user preference information; that is, the step content of the step S600 is to calculate the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and to sort the matching clothing images in the matching clothing image set according to the similarity, and to select the top N matching clothing images from the arranged matching clothing images.

[0144] The user preference information includes the user's personal information, browsing history of shopping websites or clothing purchase records, etc.

[0145] Embodiment 2:

[0146] like Figure 3 As shown, this embodiment provides a clothing collocation recommendation system based on multimodality, which is used to execute a clothing collocation recommendation method based on multimodality as described in Example 1; the clothing collocation recommendation system includes:

[0147] The multimodal data input module receives input text description and original clothing image; encodes the text description through a multi-head attention mechanism to obtain text features; encodes the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features;

[0148] A feature fusion module, which fuses the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature;

[0149] A multimodal conditional generative adversarial module, comprising a multimodal conditional generative adversarial network; the multimodal conditional generative adversarial network generates a plurality of matching clothing images according to the fusion features to obtain a matching clothing image set;

[0150] A user information input module, used to receive user preference information uploaded by users;

[0151] The clothing recommendation module calculates the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorts the matching clothing images in the matching clothing image set according to the similarity; obtains user preference information, and selects the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information in the user information input module.

[0152] Below, the present invention is further described in conjunction with a case:

[0153] Case content: The user inputs the following into the clothing matching recommendation system: Figure 4 The black suit pants and "the outdoor temperature today is 30 degrees" are shown, and the clothing matching recommendation system recommends matching clothing to the user.

[0154] The clothing matching recommendation system operation steps include:

[0155] Step 1: The multimodal data input module receives the input "Today's outdoor temperature is 30 degrees" (text description) and the image of black suit pants (original clothing image); obtains text features through a multi-head attention mechanism, and obtains clothing image features through a multi-scale adaptive convolutional network;

[0156] Step 2: The feature fusion module fuses the statement "Today's outdoor temperature is 30 degrees" with the image of black suit pants through the cross-modal attention mechanism to obtain the fusion feature;

[0157] Step 3: The multimodal conditional generation adversarial module generates the following Figure 5 The short sleeve shirt image shown;

[0158] Step 4: The clothing recommendation module calculates the similarity between the short-sleeved shirt and the fusion feature, and recommends the short-sleeved shirt image to the user.

[0159] The present invention is not limited to the above-mentioned optional implementation modes. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, all technical solutions that fall within the scope defined by the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A clothing matching recommendation method based on multimodality, characterized in that: The following steps are involved: S100, receiving an input text description and an original clothing image; S200, encoding the text description through a multi-head attention mechanism to obtain text features; S300, encoding the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features; S400, fusing the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature; S500, inputting the fusion features into a multimodal conditional generative adversarial network, wherein the multimodal conditional generative adversarial network generates a set of matching clothing images according to the fusion features; S600, calculating the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorting the matching clothing images in the matching clothing image set according to the similarity; obtaining user preference information, and screening out the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information.

2. The multimodal clothing matching recommendation method according to claim 1, characterized in that: In step S200, the multi-head attention mechanism transforms the text description into a query, a key, and a value through h groups of linear projections; The query, key and value are all input into the attention pool, and the outputs of the attention pool are spliced ​​together; the spliced ​​outputs of the attention pool are transformed through a linear projection to obtain text features.

3. The multimodal clothing matching recommendation method according to claim 1, characterized in that: In the step S300, the multi-scale adaptive convolution network includes an adaptive convolution mechanism and a fusion multi-scale feature operation; The adaptive convolution mechanism convolves the original clothing image to obtain a multi-scale feature map of the original clothing image; The multi-scale feature fusion operation concatenates and fuses the multi-scale feature maps together to obtain the clothing image features.

4. The multimodal clothing matching recommendation method according to claim 3, characterized in that: The steps of obtaining the multi-scale feature map of the original clothing image are: The adaptive convolution mechanism extracts the feature representation F(I) of the original clothing image through a convolution layer; According to the feature representation F(I) of the original clothing image, an adaptive convolution kernel K is generated through a generative network θ(I) ; Use the K θ(I) Perform a convolution operation on the original clothing image to obtain a multi-scale feature map F of the original clothing image θ (I); Among them, using the K θ(I) The formula used for the convolution operation on the original clothing image is: F θ (I)=I*K θ(I) ; The I represents the original clothing image.

5. The multimodal clothing matching recommendation method according to claim 4, characterized in that: In the multi-scale feature fusion operation, a multi-scale feature map F of the original clothing image is extracted. θ Feature maps F of different scales in (I) σ1 (I) F σ2 (I) F σ3 (I)...F σn (I); The extracted feature maps F of different scales σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) performing splicing and fusion to obtain the clothing image features; Among them, the feature maps F of different scales are extracted σ1 (I) F σ2 (I) F σ3 (I)...F σn (I) The formula used for splicing and fusion is: F MSACN (I)=Fusion(F σ1 (I)、F σ2 (I)、F σ3 (I)...F σn (I)); The F MSACN (I) represents clothing image features; the Fusion() represents a splicing fusion operation.

6. The multimodal clothing matching recommendation method according to claim 1, characterized in that: In step S400, the text feature and the clothing image feature are fused by a cross-modal attention mechanism to obtain the fused feature, which includes: S401, defining the text features as The clothing image features are defined as Wherein, M represents the number of text features, dt represents the dimension of each text feature, N represents the number of clothing image features, dp represents the dimension of each clothing image feature; R represents a real number set; S402, using the clothing image feature as a query and the text feature as a key and a value, calculating the attention weight between the text feature and the clothing image feature; S403 performs weighted averaging on the clothing image features according to the calculated attention weights, integrates the text features into the image features, and obtains the fusion features.

7. The multimodal clothing matching recommendation method according to claim 6, characterized in that: In step S402, the clothing image feature is used as a query, the text feature is used as a key and a value, and the attention weight between the text feature and the clothing image feature is calculated using the formula: Wherein, Q=PW Q A query matrix representing the image features, represents the weight matrix of the query, dQ represents the dimension of the query; K = TW K A key matrix representing the text features, represents the key weight matrix, the d K Represents the dimension of the key; V = TW v A matrix of values ​​representing the features of the text, The weight matrix representing the value, the dv represents the dimension of the value; the R represents that the elements of the matrix belong to the real number set; the softmax() represents the normalization function; In step S403, the clothing image features are weighted averaged according to the calculated attention weights, and the text features are integrated into the image features. The formula used to obtain the fusion features is: P fused =Attention(P,T,T); Among them, the P fused represents the fusion feature; the Attention() represents the attention weight, the P represents the clothing image feature; and the T represents the text feature.

8. The multimodal clothing matching recommendation method according to claim 1, characterized in that: In the step S500, the multimodal conditional generative adversarial network is obtained through training; The multimodal conditional generative adversarial network generator and discriminator; The steps of training the multimodal conditional generative adversarial network include: Collecting training clothing images and text information, and inputting the training clothing images and text information into the multimodal conditional generative adversarial network; The generator receives the training clothing image, generates a matching clothing image set according to the training clothing image, and inputs the matching clothing image set into the discriminator; The discriminator determines whether the matching clothing image set matches the text information and the training clothing image; If the judgment result is mismatch, the discriminator feeds back the judgment result to the generator, and the generator continues to generate a set of matching clothing images according to the judgment result; until the discriminator judges that the result is a match; and a trained multimodal conditional generative adversarial network is obtained.

9. The multimodal clothing matching recommendation method according to claim 8, characterized in that: The generator and the discriminator in the multimodal conditional generative adversarial network both adopt a perceptual adversarial loss function; The loss calculation formula of the generator is: Among them, the z~p z (z) represents the noise distribution p z The random noise (z) sampled in c (c) represents the condition information (c) sampled from the conditional distribution pc; G(z, c) represents the matching clothing image generated by the generator according to the noise (z) and the condition (c); D(G(z, c), c) represents the probability that the discriminator judges that the matching clothing image generated by the generator matches; φ represents a pre-trained convolutional neural network for extracting features; x target represents the target clothing image; the γ represents the weight coefficient of the perceptual loss; The loss calculation formula of the discriminator is: Among them, the x~p data (x) represents a real sample (x) sampled from the text description and the original clothing image; represents the judgment of the discriminator on the sample (x) and the condition (c); Represents the discriminator's judgment on the matching clothing image G(z, c) and condition (c) generated by the generator.

10. A clothing matching recommendation system based on multimodality, characterized in that: Used to execute a multimodal clothing matching recommendation method as described in any of claims 1-9; The clothing matching recommendation system comprises: The multimodal data input module receives input text description and original clothing image; encodes the text description through a multi-head attention mechanism to obtain text features; encodes the original clothing image through a multi-scale adaptive convolutional network to obtain clothing image features; A feature fusion module, which fuses the text feature and the clothing image feature through a cross-modal attention mechanism to obtain a fusion feature; A multimodal conditional generative adversarial module, comprising a multimodal conditional generative adversarial network; the multimodal conditional generative adversarial network generates a plurality of matching clothing images according to the fusion features to obtain a matching clothing image set; A user information input module, used to receive user preference information uploaded by users; The clothing recommendation module calculates the similarity between the matching clothing images in the matching clothing image set and the fusion feature, and sorts the matching clothing images in the matching clothing image set according to the similarity; obtains user preference information, and selects the top N matching clothing images that meet the user preference from the arranged matching clothing images according to the user preference information in the user information input module.

Images