Clothing matching recommendation method and device and training method for the device

Abstract

Embodiments of the present disclosure provide a method and device for recommending clothing combinations, as well as a training method for the device. The clothing combination recommendation method includes: acquiring images of multiple given clothing items and images of multiple candidate clothing item sets to be matched, each candidate clothing item set in the multiple candidate clothing item sets including one or more candidate clothing items; generating images of a target clothing item set of the same type as the multiple candidate clothing item sets based on the images of the multiple given clothing items; respectively calculating the similarity between the image of each candidate clothing item set in the multiple candidate clothing item sets and the images of the multiple given clothing items and the image of the target clothing item set; and recommending at least one candidate clothing item set in the multiple candidate clothing item sets based on the calculated similarity.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method and device for recommending clothing combinations, and a training method for the device. Background Art

[0002] In recent years, with the rapid development of the e-commerce industry, online shopping for apparel (tops, bottoms, shoes, hats, socks, gloves, scarves, ties, accessories, bags, umbrellas, etc.) has become increasingly popular, and the number of apparel items sold online has exploded. Users can browse through the vast selection of clothing images displayed in online stores and select matching outfits to create stylish outfits that showcase their personal style. When purchasing apparel, users may expect to find matching pieces in a single purchase to meet their style needs. However, not everyone is adept at matching items based on images. As a result, some users may return items purchased online that don't match. This not only increases the time and financial costs of shopping for users, but also increases operating costs for merchants. A fashion recommendation algorithm that can intelligently determine whether user-selected items are compatible and provide targeted matching recommendations could help users enjoy more enjoyable online shopping experiences.

[0003] With the rapid development of both the fashion industry and the field of artificial intelligence, many researchers have conducted extensive research on fashion matching algorithms. Fashion matching learning involves all aspects of artificial intelligence technology, including computer vision, natural language processing, and recommendation algorithms. As AI research deepens, more and more technologies are being applied to fashion matching learning. Summary of the Invention

[0004] The embodiments described herein provide a clothing matching recommendation method and apparatus, as well as a training method for the apparatus, an electronic device, and a computer-readable storage medium storing a computer program.

[0005] According to a first aspect of the present disclosure, a method for recommending clothing combinations is provided. The method comprises: acquiring images of a plurality of given clothing items and images of a plurality of candidate clothing item sets to be matched, each candidate clothing item set in the plurality of candidate clothing item sets including one or more candidate clothing items; generating images of a target clothing item set of the same type as the plurality of candidate clothing item sets based on the images of the plurality of given clothing items; calculating similarities between the images of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the image of the target clothing item set; and recommending at least one candidate clothing item set from the plurality of candidate clothing item sets based on the calculated similarities.

[0006] In some embodiments of the present disclosure, generating images of a target set of clothing items of the same type as the set of candidate clothing items based on the images of the multiple given clothing items includes: obtaining visual features of the images of the multiple given clothing items in multiple layers of the convolutional neural network through a convolutional neural network; fusing the obtained visual features based on a self-attention mechanism to obtain fused features; generating fused features based on the fused features through a variational autoencoder; and generating images of the target set of clothing items based on the fused features through a transposed convolutional neural network. The number of layers of the transposed convolutional neural network is the same as the number of layers of the convolutional neural network.

[0007] In some embodiments of the present disclosure, respectively calculating the similarity between the image of each candidate clothing item set in the multiple candidate clothing item sets and the images of the multiple given clothing items and the images of the target clothing item set includes, for each candidate clothing item set: obtaining, through a convolutional neural network, candidate visual features of the image of the candidate clothing item set in multiple layers of the convolutional neural network; and calculating the similarity between the image of the candidate clothing item set and the images of the multiple given clothing items and the images of the target clothing item set based on the candidate visual features, the visual features of the images of the multiple given clothing items in multiple layers of the convolutional neural network, and the hierarchical features of the image of the target clothing item set in multiple layers of the transposed convolutional neural network.

[0008] In some embodiments of the present disclosure, the similarity between the image of each candidate clothing item set and the images of the plurality of given clothing items and the image of the target clothing item set is calculated as:

[0009]

[0010] Among them, s h represents the similarity between the image of the candidate clothing item set and the images of the multiple given clothing items and the image of the target clothing item set, l represents the number of layers of the convolutional neural network, h i The matrix corresponding to the visual features of the image of the candidate clothing item set in the i-th layer of the convolutional neural network, g i Represents the matrix corresponding to the hierarchical features of the target clothing item set in the i-th layer of the transposed convolutional neural network, m represents the number of the given clothing items, x ij represents the matrix corresponding to the visual features of the image of the j-th given clothing item among the multiple given clothing items at the i-th layer of the convolutional neural network, and T represents a transpose operation.

[0011] According to a second aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the electronic device performs the steps of the method described in the first aspect of the present disclosure.

[0012] According to a third aspect of the present disclosure, a clothing matching recommendation device is provided. The clothing matching recommendation device includes: an image acquisition module configured to acquire images of multiple given clothing items and images of multiple candidate clothing item sets to be matched, each candidate clothing item set in the multiple candidate clothing item sets including one or more candidate clothing items; a target image generation module configured to generate images of a target clothing item set of the same type as the multiple candidate clothing item sets based on the images of the multiple given clothing items; a similarity calculation module configured to respectively calculate the similarity between the image of each candidate clothing item set in the multiple candidate clothing item sets and the images of the multiple given clothing items and the image of the target clothing item set; and a recommendation module configured to recommend at least one candidate clothing item set in the multiple candidate clothing item sets based on the calculated similarities.

[0013] According to a fourth aspect of the present disclosure, a training method is provided for training the clothing combination recommendation device according to the third aspect of the present disclosure. The training method comprises: acquiring images of a first set of clothing sets and images of a second set of clothing sets, each of the first set of clothing sets and the second set of clothing sets including a plurality of clothing items that coordinate with each other; in each round of training, using the images of the first set of clothing sets to train the clothing combination recommendation device to minimize a loss function of the clothing combination recommendation device; after each round of training, using the images of the second set of clothing sets to verify the recommendation accuracy of the clothing combination recommendation device; and repeatedly training the clothing combination recommendation device until a predetermined number of training rounds are completed or the recommendation accuracy converges.

[0014] In some embodiments of the present disclosure, using images of a first group of clothing sets to train a clothing matching recommendation device to minimize the loss function of the clothing matching recommendation device includes: taking each clothing set in the first group of clothing sets as a training clothing set; performing the following operations on the images of the training clothing sets: selecting one or more clothing items in the training clothing sets as a positive sample item set; taking clothing items other than the positive sample item set in the training clothing sets as a given clothing item set; selecting a clothing item set of the same type as the positive sample item set from clothing sets other than the training clothing sets in the first group of clothing sets as a negative sample item set; generating images of a reference clothing item set of the same type as the positive sample item set based on the images of the given clothing item set; calculating a first similarity between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set; calculating a second similarity between the images of the negative sample item set and the images of the given clothing item set and the images of the reference clothing item set; calculating the loss function of the clothing matching recommendation device based on the difference between the first similarity and the second similarity; and updating the parameters of the clothing matching recommendation device to minimize the loss function of the clothing matching recommendation device.

[0015] In some embodiments of the present disclosure, the loss function of the clothing matching recommendation device is also calculated based on the following sub-loss functions: a first sub-loss function calculated based on the difference between the images of the reference clothing item set and the images of the positive sample item set; and / or a second sub-loss function of the target image generation module.

[0016] In some embodiments of the present disclosure, using the images of the second group of clothing sets to verify the recommendation accuracy of the clothing matching recommendation device includes: taking each clothing set in the second group of clothing sets as a verification clothing set; performing the following operations on the images of the verification clothing sets: selecting one or more clothing items in the verification clothing set as a positive sample item set; taking the clothing items in the verification clothing set other than the positive sample item set as a given clothing item set; obtaining a clothing item set of the same type as the positive sample item set as a negative sample item set, and the negative sample item set does not match the given clothing item set; generating a parameter set of the same type as the positive sample item set based on the images of the given clothing item set. the images of the reference clothing item set; calculating a first verification similarity between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set; calculating a second verification similarity between the images of the negative sample item set and the images of the given clothing item set and the images of the reference clothing item set; and in response to the first verification similarity being greater than the second verification similarity, determining that the verification for the verification clothing set is passed; in response to the first verification similarity being less than or equal to the second verification similarity, determining that the verification for the verification clothing set is failed; and counting the number of clothing sets that have been verified in the second group of clothing sets to determine the recommendation accuracy of the clothing matching recommendation device.

[0017] According to a fifth aspect of the present disclosure, a training device is provided. The training device includes at least one processor and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the training device performs the steps of the training method according to the fourth aspect of the present disclosure.

[0018] According to a sixth aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program implements the steps of the method according to the first aspect of the present disclosure when executed by a processor.

[0019] According to a seventh aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program implements the steps of the training method according to the fourth aspect of the present disclosure when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure.

[0021] Figure 1 is an exemplary flow chart of a clothing matching recommendation method according to an embodiment of the present disclosure;

[0022] Figure 2 is an exemplary schematic diagram of a clothing matching recommendation method according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic block diagram of an electronic device for executing a clothing matching recommendation method according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic block diagram of a clothing collocation recommendation device according to an embodiment of the present disclosure;

[0025] Figure 5 is an exemplary flow chart of a training method for training an apparel matching recommendation device according to an embodiment of the present disclosure; and

[0026] Figure 6 3 is an exemplary schematic diagram of a training sub-process for training an apparatus for recommending clothing combinations according to an embodiment of the present disclosure.

[0027] Elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the subject matter of the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal manner unless otherwise explicitly defined herein. Terms such as "first" and "second" are used only to distinguish one component (or a portion of a component) from another component (or another portion of a component).

[0030] At present, the mainstream clothing matching recommendation methods are as follows: (1) Learning the overall matching relationship of clothing sets by studying the matching relationship of paired items, but this type of method only considers the matching relationship of paired items, and does not consider the matching relationship of multiple items; (2) Drawing on the methods of natural language processing, the clothing set is modeled as an ordered sequence, but the position of the clothing items has no effect on whether the clothing set is matched; (3) Considering clothing items or item categories as nodes in the graph, constructing a clothing matching network, and using graph neural network algorithms to learn matching relationships. The premise for the graph neural network algorithm to achieve better results is that it requires a large amount of contextual information related to the matching of clothing items, but it is difficult to meet the above prerequisites in reality.

[0031] Therefore, the present disclosure proposes a clothing matching recommendation method. Figure 1 An exemplary flowchart of a clothing collocation recommendation method according to an embodiment of the present disclosure is shown.

[0032] exist Figure 1At block S102, images of multiple given clothing items and images of multiple candidate clothing item sets to be matched are obtained. Each candidate clothing item set in the multiple candidate clothing item sets includes one or more candidate clothing items. In some embodiments of the present disclosure, the multiple given clothing items have a matching relationship. The multiple given clothing items are, for example, part of tops, bottoms, shoes, hats, socks, gloves, scarves, ties, accessories, bags, umbrellas, etc. The images of the multiple given clothing items may refer to a collection of images of each given clothing item. In some embodiments of the present disclosure, the number and type of candidate clothing items in each candidate clothing item set are the same. The type of candidate clothing item is different from the type of the given clothing item. The image of each candidate clothing item set may refer to a collection of images of each candidate clothing item in the candidate clothing item set. In some embodiments of the present disclosure, the image of the given clothing item is in a flat state with the clothing items in the image of the candidate clothing item set, and the image is clear and does not contain obvious impurities.

[0033] In some embodiments of the present disclosure, the image of a given clothing item and the images of the candidate clothing item set may be accompanied by information about the type of clothing item in the image. In other embodiments of the present disclosure, the type of clothing item in the image of the given clothing item and the images of the candidate clothing item set may be identified using image recognition technology.

[0034] Figure 2 FIG2 shows an exemplary schematic diagram of a clothing matching recommendation method according to an embodiment of the present disclosure. Figure 2 In the example, given clothing items include tops 201, bags 202, and shoes 203, each candidate clothing item set includes a candidate clothing item, and the type of the candidate clothing item is bottoms. Among them, the first candidate clothing item set includes skirts 204, and the second candidate clothing item set includes pants 205. Although in Figure 2 In the example, each candidate clothing item set includes only one candidate clothing item, but in some embodiments of the present disclosure, each candidate clothing item set may include two or more candidate clothing items, for example, bottoms, umbrellas, or gloves, etc.

[0035] At block S104, images of a target clothing item set of the same type as the plurality of candidate clothing item sets are generated based on the images of the plurality of given clothing items. Figure 2 In the example, each candidate clothing item set includes a candidate clothing item, and the type of the candidate clothing item is bottoms. Therefore, the clothing items included in the target clothing item set are bottoms. Figure 2 In the example of , an image of bottoms (skirt 206 ) is generated based on an image of top 201 , an image of bag 202 , and an image of shoes 203 .

[0036] In some embodiments of the present disclosure, the visual features of the images of the multiple given clothing items in the multiple layers of the convolutional neural network (CNN) can be obtained through a convolutional neural network. In one example, the size of the image of each given clothing item can be adjusted to, for example, (3×128×128), where 3 represents the three primary color dimensions of the image (red, green, and blue). The adjusted image is then input into the convolutional neural network to obtain the visual features of each given clothing item in the multiple layers of the convolutional neural network. Each given clothing item can be input into the same convolutional neural network, or into its own corresponding convolutional neural network. In the case where the convolutional neural network includes three layers, the matrices corresponding to the visual features of each given clothing item in each layer can be (512×8×8), (512×4×4), and (1024×1×1), respectively. The above-mentioned sizes of the matrices are exemplary and can also be set to other values. Assuming that the number of given clothing items is m, the visual features extracted for the m given clothing items are x1, x2, ..., x m-1 After splicing together the visual features of these given clothing items, we get the matrix X input .

[0037] Then, the visual features X obtained by the self-attention mechanism are fused input To obtain the fused feature x fuse In one example, the matrix X input Input the multi-head self-attention module based on the self-attention mechanism, and record the output of the multi-head self-attention module as matrix X atten . The matrix X atten Expand along the dimension of the number of items, and then concatenate along the dimension of the item features. Then, input the concatenated matrix into the fully connected layer to obtain the fused feature x fuse .

[0038] Then, the variational autoencoder is used based on the fused feature x fuse Generate fusion feature f g . Fusion feature f g It can be generated according to the following formula:

[0039] f g =ReLU(W z z+W fuse x fuse +b g ) (1)

[0040] Among them, ReLU represents the nonlinear activation function, z represents the latent variable, and W z Represents the weight matrix of the latent variable z, W fuse Represents x fuse The weight matrix, b grepresents the bias term.

[0041] The latent variable z can be calculated as follows:

[0042] z=μ+σ⊙ε,ε~N(0,1) (2)

[0043] Where μ+W μ x fuse +b μ (3), logσ 2 =W σ x fuse +b σ (4). In formula (3), W μ Represents x fuse The weight matrix, b μ In formula (4), W σ Represents x fuse The weight matrix, b σ represents the bias term.

[0044] In generating the fusion feature f g Afterwards, the transposed convolutional neural network can be used to transform the fusion feature f g Generate images of the target clothing item set I g Among them, the number of layers of the transposed convolutional neural network is the same as the number of layers of the convolutional neural network.

[0045] Image I of the target clothing item set g Can be expressed as:

[0046] I g =Deconv(f g ) (5)

[0047] Among them, Deconv represents a multi-layer transposed convolutional neural network. The input tensor of each layer of the transposed convolutional neural network can be used as the hierarchical feature of the layer. Figure 2 In the example, the input tensor size of the first layer (Deconv1) is (1024×1×1); the input tensor size of the second layer (Deconv2) is (512×4×4); and the input tensor size of the third layer (Deconv3) is (512×8×8). Before the transposed convolution operation, the hierarchical features of each layer are flattened and input into a fully connected layer, mapped into a vector of dimension 128 as the hierarchical features.

[0048] At box S106, the similarity between the image of each candidate clothing item set in the multiple candidate clothing item sets and the images of the multiple given clothing items and the images of the target clothing item set is calculated respectively. In some embodiments of the present disclosure, for each candidate clothing item set, candidate visual features of the image of the candidate clothing item set in multiple layers of the convolutional neural network are obtained through a convolutional neural network. The process of obtaining candidate visual features is similar to the process of obtaining visual features of the image of a given clothing item in multiple layers of the convolutional neural network, and will not be repeated here. Then, based on the candidate visual features, the visual features of the images of the multiple given clothing items in multiple layers of the convolutional neural network, and the hierarchical features of the image of the target clothing item set in multiple layers of the transposed convolutional neural network, the similarity between the image of the candidate clothing item set and the images of the multiple given clothing items and the images of the target clothing item set is calculated.

[0049] In some embodiments of the present disclosure, the similarity between the image of each candidate clothing item set and the images of the plurality of given clothing items and the image of the target clothing item set is calculated as:

[0050]

[0051] Among them, s h represents the similarity between the image of the candidate clothing item set and the images of the multiple given clothing items and the image of the target clothing item set, l represents the number of layers of the convolutional neural network, h i The matrix corresponding to the visual features of the image of the candidate clothing item set in the i-th layer of the convolutional neural network, g i Represents the matrix corresponding to the hierarchical features of the target clothing item set in the i-th layer of the transposed convolutional neural network, m represents the number of the given clothing items, x ij represents the matrix corresponding to the visual features of the image of the j-th given clothing item among the multiple given clothing items at the i-th layer of the convolutional neural network, and T represents a transpose operation.

[0052] At box S108, at least one candidate clothing item set from the multiple candidate clothing item sets is recommended based on the calculated similarity. In some embodiments of the present disclosure, the images of the multiple candidate clothing item sets can be displayed in descending order according to the calculated similarity. The image of the candidate clothing item set with the highest similarity can be displayed in the most prominent position. In some embodiments of the present disclosure, the candidate clothing item sets with the top s similarities can be screened out, and then the images of the s candidate clothing item sets can be displayed in descending order according to the similarity. Wherein, s is a positive integer less than the total number of candidate clothing item sets. In this way, users can browse to clothing item sets that match a given clothing item more quickly.

[0053] According to the clothing matching recommendation method of the embodiment of the present disclosure, the matching relationship of multiple clothing items is taken into consideration, and the many-to-one matching relationship is better learned by generating a matching clothing item set from multiple clothing items. Therefore, clothing matching recommendations can be made well without a large amount of contextual information related to the clothing items.

[0054] Figure 3 FIG. 3 is a schematic block diagram of an electronic device 300 for executing a clothing matching recommendation method according to an embodiment of the present disclosure. Figure 3 As shown, the electronic device 300 may include a processor 310 and a memory 320 storing a computer program. When the computer program is executed by the processor 310, the electronic device 300 may perform the following operations: Figure 1 The steps of the method 100 are shown. In one example, the electronic device 300 may be a computer device or a cloud computing node, or may be located in a GPU server.

[0055] The electronic device 300 may obtain images of multiple given clothing items and images of multiple candidate clothing item sets to be matched. Each candidate clothing item set in the multiple candidate clothing item sets includes one or more candidate clothing items. The electronic device 300 may generate images of a target clothing item set of the same type as the multiple candidate clothing item sets based on the images of the multiple given clothing items. The electronic device 300 may respectively calculate the similarity between the image of each candidate clothing item set in the multiple candidate clothing item sets and the images of the multiple given clothing items and the image of the target clothing item set. The electronic device 300 may recommend at least one candidate clothing item set in the multiple candidate clothing item sets based on the calculated similarity.

[0056] In an embodiment of the present disclosure, the processor 310 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 320 may be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0057] Furthermore, in embodiments of the present disclosure, electronic device 300 may also include an input device 330, such as a camera, keyboard, mouse, etc., for acquiring images of multiple given clothing items and images of multiple candidate clothing item sets to be matched. Furthermore, electronic device 300 may also include an output device 340, such as a display, for outputting the recommended candidate clothing item sets.

[0058] Figure 4A schematic block diagram of an apparatus 400 for recommending clothing combinations according to an embodiment of the present disclosure is shown. The apparatus 400 for recommending clothing combinations includes: an image acquisition module 410 , a target image generation module 420 , a similarity calculation module 430 , and a recommendation module 440 .

[0059] The image acquisition module 410 is configured to acquire images of a plurality of given clothing items and images of a plurality of candidate clothing item sets to be matched, each of which may include one or more candidate clothing items.

[0060] The target image generation module 420 is configured to generate images of a target clothing item set of the same type as the multiple candidate clothing item sets based on the images of the multiple given clothing items. The target image generation module 420 may include a feature fusion module, a variational autoencoder, and a transposed convolutional neural network. The feature fusion module can be used to obtain the fused feature x as described above. fuse The variational autoencoder can be used based on the fused feature x fuse Generate the fusion feature f as described above g The transposed convolutional neural network can be used to generate the fusion feature f g Generate the image I of the target clothing item set as described above g .

[0061] The similarity calculation module 430 is configured to respectively calculate the similarity between the image of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the image of the target clothing item set.

[0062] The recommendation module 440 is configured to recommend at least one candidate clothing item set from the plurality of candidate clothing item sets according to the calculated similarity.

[0063] Figure 5 An exemplary flow chart of a training method for training an apparatus for recommending clothing combinations according to an embodiment of the present disclosure is shown.

[0064] exist Figure 5 At block S502, images of a first set of clothing sets and an image of a second set of clothing sets are obtained. Each clothing set in the first set of clothing sets and the second set of clothing sets includes multiple clothing items that match each other. In some embodiments of the present disclosure, each clothing set includes at least four clothing items that match each other.

[0065] In some embodiments of the present disclosure, data on fashion matching sets can be crawled from a fashion matching community (e.g., the Polyvore website). Unqualified data in the crawled data can be cleaned to construct a data set for clothing sets. Unqualified data may include: blurred or distorted images, images containing a lot of debris, images of clothing items in a non-flattened state, and images of sets with less than four types of clothing (e.g., tops, bottoms, bags, shoes). Each element in the data set is an image of each clothing item in a complete set of clothing sets, as well as type information about the clothing item. The number and type of clothing items in each clothing set are consistent. The above-mentioned first group of clothing sets and second group of clothing sets can be obtained from the constructed data set. The first group of clothing sets and the second group of clothing sets can each include multiple clothing sets, and the number of clothing sets included in the first group of clothing sets can be the same as or different from the number of clothing sets included in the second group of clothing sets.

[0066] At block S504, in each round of training, the clothing combination recommendation apparatus is trained using the first set of clothing combination images to minimize the loss function of the clothing combination recommendation apparatus. In some embodiments of the present disclosure, each clothing combination in the first set of clothing combinations may be used as a training clothing combination to train the clothing combination recommendation apparatus. The process of training the clothing combination recommendation apparatus using a clothing combination can be considered a training sub-process within a training round. Figure 6 An exemplary schematic diagram illustrating a training sub-process for training an apparatus for recommending clothing combinations according to an embodiment of the present disclosure is shown.

[0067] In this training sub-process, one or more clothing items in the training clothing set are selected as the positive sample item set. Figure 6 In the example, the training clothing set includes: top 601, bag 602, shoes 603, and skirt 604. Skirt 604 in the training clothing set is selected as the positive sample item set. The clothing items in the training clothing set (top 601, bag 602, and shoes 603) other than the positive sample item set are used as the given clothing item set.

[0068] From the clothing sets in the first group of clothing sets, excluding the training clothing sets, select clothing item sets of the same type as the positive sample item set as the negative sample item set. Figure 6 In the example, the type of clothing items included in the positive sample item set is bottoms. Assume that the first set of clothing sets includes t i A clothing item of the type of bottoms can be randomly selected as a negative sample item set with equal probability. Since the negative sample item set is selected from clothing sets other than the training clothing set, the probability p of a clothing item of the type of bottoms being selected as a negative sample item set is i It can be calculated by the following formula:

[0069]

[0070] exist Figure 6 In the example of , the negative sample item set may be selected as pants 605, for example.

[0071] Then, based on the images of the given clothing item set, images of a reference clothing item set of the same type as the positive sample item set can be generated. The process of generating the images of the reference clothing item set is similar to the process of generating the images of the target clothing item set at block S104 and will not be repeated here. Figure 6 In the example of , the reference clothing item set may be generated as skirt 606, for example.

[0072] The first similarity s between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set can be calculated in parallel p , and calculate the second similarity s between the image of the negative sample item set and the image of the given clothing item set and the image of the reference clothing item set n .

[0073] The first similarity s p Can be calculated as:

[0074]

[0075] The second similarity s n Can be calculated as:

[0076]

[0077] Among them, l represents the number of layers of the convolutional neural network, p i Represents the matrix corresponding to the visual features of the image of the positive sample set in the i-th layer of the convolutional neural network, n i Represents the matrix corresponding to the visual features of the image of the negative sample set in the i-th layer of the convolutional neural network, g i represents the matrix corresponding to the hierarchical features of the image of the reference clothing item set in the i-th layer of the transposed convolutional neural network, m represents the number of items in the given clothing item set, x ij represents the matrix corresponding to the visual features of the image of the j-th given clothing item in the given clothing item set at the i-th layer of the convolutional neural network, and T represents the transpose operation.

[0078] In some embodiments of the present disclosure, the loss function of the clothing combination recommendation device can be calculated based on the difference between the first similarity and the second similarity. The difference between the first similarity and the second similarity can be expressed as:

[0079] out=s p -s n .

[0080] Bayesian Personalized Ranking (BPR) can be used as the loss function. BPR loss function L bpr Can be expressed as:

[0081] L bpr =-log(Sigmoid(out)).

[0082] Among them, the calculation process of Sigmoid is:

[0083]

[0084] In the training sub-process, the parameters of the clothing matching recommendation device can be updated to minimize the loss function of the clothing matching recommendation device. In one example, the parameters of the clothing matching recommendation device can be updated so that the first similarity s p Than the second similarity s n It gets bigger and bigger, thereby minimizing the loss function of the clothing matching recommendation device.

[0085] In some embodiments of the present disclosure, the loss function of the clothing matching recommendation device is further calculated based on the following sub-loss functions: an L2 loss function (which may be referred to as a first sub-loss function in this context) calculated based on the difference between the images of the reference clothing item set and the images of the positive sample item set, and a target image generation module 420 (in the variational autoencoder generating the fusion feature f g In the process of , the KL loss function (which can be called the second sub-loss function in this context) is used.

[0086] The L2 loss function can be calculated as follows:

[0087]

[0088] Among them, I p is the image of the positive sample item set, I g These are images of the generated reference clothing item set.

[0089] The KL loss function can be calculated as follows:

[0090]

[0091] Where K represents the dimensions of μ and σ. The total loss function of the clothing matching recommendation device can be optimized by adding the above loss functions together. The total loss function L can be calculated as follows:

[0092] L=λ1L2+λ2L KL +λ3L bpr .

[0093] Among them, λ1, λ2, and λ3 are pre-set hyperparameters.

[0094] During the training sub-process, the parameters of the clothing combination recommendation device may be updated to minimize the total loss function L of the clothing combination recommendation device.

[0095] At block S506 , after each round of training, the second set of clothing set images is used to verify the accuracy of the clothing combination recommendation device, for example, using a FitB (Fill in the Blank) method.

[0096] In some embodiments of the present disclosure, each clothing set in the second group of clothing sets can be used as a validation clothing set to validate the clothing combination recommendation device trained at block S504. One or more clothing items in the validation clothing set can be selected as a positive sample item set. In one example, the positive sample item set can include bottoms.

[0097] Then, the clothing items in the verification clothing set other than the positive sample item set are taken as the given clothing item set. In one example, the given clothing item set may include a top, a bag, and shoes.

[0098] A set of clothing items of the same type as the positive sample item set can be obtained as a negative sample item set. In the above example, the negative sample item set may include bottoms. The negative sample item set does not match the given clothing item set. The negative sample item set can be selected with equal probability from the clothing sets in the second set, excluding the verification clothing set.

[0099] Similar to the training process, images of a reference clothing item set of the same type as the positive sample item set can be generated based on the images of the given clothing item set. In the above example, the reference clothing item set may include bottoms. Then, a first verification similarity is calculated between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set. A second verification similarity is calculated between the images of the negative sample item set and the images of the given clothing item set and the images of the reference clothing item set. If the first verification similarity is greater than the second verification similarity, it can be determined that the verification for the verification clothing set has passed. If the first verification similarity is less than or equal to the second verification similarity, it can be determined that the verification for the verification clothing set has failed.

[0100] After verifying the clothing combination recommendation device using each clothing set in the second set of clothing sets, the number of clothing sets in the second set of clothing sets that passed verification can be counted to determine the recommendation accuracy of the clothing combination recommendation device. The recommendation accuracy can be equal to the number of clothing sets that passed verification divided by the total number of clothing sets in the second set of clothing sets.

[0101] At block S508 , it is determined whether the recommendation accuracy has converged. If the recommendation accuracy has not improved after multiple rounds (eg, 10 rounds) of training, it indicates that the recommendation accuracy has converged.

[0102] If the recommendation accuracy has not converged ("No" at block S508), then at block S510, it is determined whether a predetermined number of training rounds has been completed. The predetermined number of training rounds may be set based on empirical values, estimated based on expected training time, or considered based on resource usage of the computing device.

[0103] If the predetermined number of training rounds has not been completed ("No" at block S510), the process proceeds to block S504 to repeat the training of the clothing combination recommendation device. If the recommendation accuracy has converged ("Yes" at block S508), the process proceeds to block S512 to end the training. If the predetermined number of training rounds has been completed ("Yes" at block S510), the process proceeds to block S512 to end the training.

[0104] In some alternative embodiments of the present disclosure, the operation at block S508 may not be performed, that is, the training is terminated as soon as a predetermined number of training rounds are completed. In other alternative embodiments of the present disclosure, the operation at block S510 may not be performed, that is, regardless of the number of training rounds, the training is terminated only when the recommendation accuracy converges.

[0105] The clothing collocation recommendation device trained as above can make clothing collocation recommendations more accurately, thereby saving the user time in manual selection.

[0106] In other embodiments of the present disclosure, a computer-readable storage medium storing a computer program is further provided, wherein the computer program can achieve the following when executed by a processor: Figure 1 and Figure 5 The steps of the method are shown.

[0107] In summary, the clothing pairing recommendation method according to the embodiments of the present disclosure considers the matching relationships of multiple clothing items. By generating a matching clothing item set from multiple clothing items, it better learns many-to-one matching relationships. Therefore, it can effectively perform clothing pairing recommendations without requiring a large amount of contextual information related to the clothing items. Furthermore, by using the training method according to the embodiments of the present disclosure to train a clothing pairing recommendation device, the device can more accurately recommend clothing pairs, saving users time in manual selection.

[0108] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus and method according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0109] Unless the context clearly indicates otherwise, as used herein and in the appended claims, the singular includes the plural, and vice versa. Thus, when referring to the singular, the plural of the corresponding term is generally included. Similarly, the words "include" and "comprising" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it follows a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0110] Further aspects and scope of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended to be illustrative only and are not intended to limit the scope of the present application.

[0111] Several embodiments of the present disclosure have been described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The scope of protection of the present disclosure is defined by the appended claims.

Claims

1. A clothing matching recommendation method, comprising: Acquire images of a plurality of given clothing items and images of a plurality of candidate clothing item sets to be matched, wherein each candidate clothing item set in the plurality of candidate clothing item sets includes one or more candidate clothing items; generating, based on the images of the plurality of given clothing items, images of a target clothing item set of the same type as the plurality of candidate clothing item sets; respectively calculating similarities between the images of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the images of the target clothing item set; as well as recommending at least one candidate clothing item set from the plurality of candidate clothing item sets based on the calculated similarity; The step of respectively calculating the similarity between the image of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the image of the target clothing item set includes: for each candidate clothing item set: Obtaining, through the convolutional neural network, candidate visual features of the image of the candidate clothing item set in the multiple layers of the convolutional neural network; as well as Based on the candidate visual features, the visual features of the images of the multiple given clothing items in the multiple layers of the convolutional neural network, and the hierarchical features of the images of the target clothing item set in the multiple layers of the transposed convolutional neural network, the similarity between the images of the candidate clothing item set and the images of the multiple given clothing items and the images of the target clothing item set is calculated.

2. The clothing matching recommendation method according to claim 1, wherein: Generating images of a target clothing item set of the same type as the plurality of candidate clothing item sets based on the images of the plurality of given clothing items includes: Obtaining visual features of the images of the plurality of given clothing items in a plurality of layers of the convolutional neural network through a convolutional neural network; The acquired visual features are fused based on the self-attention mechanism to obtain the fused features; generating fused features based on the fused features through a variational autoencoder; and An image of the target clothing item set is generated based on the fusion features through a transposed convolutional neural network, wherein the number of layers of the transposed convolutional neural network is the same as the number of layers of the convolutional neural network.

3. The clothing matching recommendation method according to claim 1, wherein: The similarity between the images of each candidate clothing item set and the images of the plurality of given clothing items and the images of the target clothing item set is calculated as: Among them, s h represents the similarity between the image of the candidate clothing item set and the images of the multiple given clothing items and the image of the target clothing item set, l represents the number of layers of the convolutional neural network, h i The matrix representing the visual features of the candidate clothing item set in the i-th layer of the convolutional neural network, g i The matrix corresponding to the hierarchical features of the image of the target clothing item set at the i-th layer of the transposed convolutional neural network is represented, m represents the number of the given clothing items, x ij represents a matrix corresponding to the visual features of the image of the j-th given clothing item among the multiple given clothing items at the i-th layer of the convolutional neural network, and T represents a transpose operation.

4. An electronic device comprising: at least one processor; as well as at least one memory storing a computer program; Wherein, when the computer program is executed by the at least one processor, the electronic device executes the steps of the clothing matching recommendation method according to any one of claims 1 to 3.

5. A clothing matching recommendation device, comprising: an image acquisition module configured to acquire images of a plurality of given clothing items and images of a plurality of candidate clothing item sets to be matched, each candidate clothing item set of the plurality of candidate clothing item sets including one or more candidate clothing items; a target image generation module configured to generate images of a target clothing item set of the same type as the plurality of candidate clothing item sets based on the images of the plurality of given clothing items; a similarity calculation module configured to respectively calculate similarities between the image of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the image of the target clothing item set; as well as a recommendation module configured to recommend at least one candidate clothing item set from the plurality of candidate clothing item sets based on the calculated similarity; The step of respectively calculating the similarity between the image of each candidate clothing item set in the plurality of candidate clothing item sets and the images of the plurality of given clothing items and the image of the target clothing item set includes: for each candidate clothing item set: Obtaining, through the convolutional neural network, candidate visual features of the image of the candidate clothing item set in the multiple layers of the convolutional neural network; as well as Based on the candidate visual features, the visual features of the images of the multiple given clothing items in the multiple layers of the convolutional neural network, and the hierarchical features of the images of the target clothing item set in the multiple layers of the transposed convolutional neural network, the similarity between the images of the candidate clothing item set and the images of the multiple given clothing items and the images of the target clothing item set is calculated.

6. A training method for training the clothing combination recommendation device according to claim 5, the training method comprising: Acquire images of a first group of clothing sets and images of a second group of clothing sets, wherein each clothing set in the first group of clothing sets and the second group of clothing sets includes a plurality of clothing items that match each other; In each round of training, the clothing combination recommendation device is trained using the first set of clothing combination images to minimize a loss function of the clothing combination recommendation device; After each round of training, using the second set of clothing set images to verify the recommendation accuracy of the clothing combination recommendation device; as well as The clothing combination recommendation device is repeatedly trained until a predetermined number of training rounds are completed or the recommendation accuracy converges.

7. The training method according to claim 6, wherein: Training the clothing combination recommendation device using the first set of clothing set images to minimize a loss function of the clothing combination recommendation device includes: using each clothing set in the first group of clothing sets as a training clothing set; Perform the following operations on the image of the training clothing set: Select one or more clothing items in the training clothing set as a positive sample item set; The clothing items in the training clothing set except the positive sample item set are used as the given clothing item set; Selecting a set of clothing items of the same type as the positive sample item set from the clothing sets in the first group of clothing sets excluding the training clothing set as a negative sample item set; generating images of a reference clothing item set of the same type as the positive sample item set based on the images of the given clothing item set; Calculating a first similarity between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set; Calculating a second similarity between the images of the negative sample item set and the images of the given clothing item set and the images of the reference clothing item set; Calculating a loss function of the clothing combination recommendation device based on a difference between the first similarity and the second similarity; and The parameters of the clothing combination recommendation device are updated to minimize the loss function of the clothing combination recommendation device.

8. The training method according to claim 7, wherein: The loss function of the clothing combination recommendation device is also calculated based on the following sub-loss functions: A first sub-loss function calculated based on the difference between the images of the reference clothing item set and the images of the positive sample item set; and / or The second sub-loss function of the target image generation module.

9. The training method according to claim 6, wherein: Verifying the recommendation accuracy of the clothing combination recommendation device using the second set of clothing set images includes: using each clothing set in the second group of clothing sets as a verification clothing set; Perform the following operations on the image of the verification clothing set: Select one or more clothing items in the verification clothing set as a positive sample item set; The clothing items in the verification clothing set other than the positive sample item set are used as the given clothing item set; Obtaining a set of clothing items of the same type as the positive sample item set as a negative sample item set, wherein the negative sample item set does not match the given clothing item set; generating images of a reference clothing item set of the same type as the positive sample item set based on the images of the given clothing item set; Calculating a first verification similarity between the images of the positive sample item set and the images of the given clothing item set and the images of the reference clothing item set; Calculating a second verification similarity between the images of the negative sample item set and the images of the given clothing item set and the images of the reference clothing item set; and In response to the first verification similarity being greater than the second verification similarity, determining that verification of the verification clothing set is passed; In response to the first verification similarity being less than or equal to the second verification similarity, determining that verification of the verification apparel set has failed; and The number of clothing sets in the second group of clothing sets that have passed verification is counted to determine the recommendation accuracy of the clothing combination recommendation device.

Images