Information processing device, information processing method, and information processing program

Abstract

The similarity of the images showing clothing and accessories is determined taking into consideration the importance of each component element. [Solution] The information processing device of the present application is characterized by having an acquisition unit that acquires a reference image, which is an image of a wearer wearing an item of clothing and serves as a reference, and a comparison image, which is an image of a wearer wearing the item of clothing and is an image for comparison with the reference image; a judgment unit that judges the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance of each component of the image in evaluating the image of the wearer wearing the item of clothing; and an output unit that outputs the judgment result by the judgment unit.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is known a technique for determining the similarity of fashion items (clothing accessories). One example of such a technique is to calculate the similarity between each item in a full-body photograph in a reference photograph set and each item in a recommendation photograph set for each type of item, search the reference photograph set for full-body photographs that include items of the same type as an input item and that have a high similarity, extract items of a different type that are combined with items of the same type as the input item in the full-body photograph image obtained by the search, search the recommendation photograph set for a predetermined number of items in descending order of similarity to the items of the different type, and present the items obtained by the search as recommended items. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5476236 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-described techniques do not necessarily make it possible to determine the similarity of images showing clothing articles by taking into consideration the importance of each component element in the images.

[0005] For example, the above-mentioned technology simply calculates the similarity of items in an image using features extracted from the image, and does not necessarily allow for determining the similarity of images by taking into account the importance of each component in an image showing an item of clothing.

[0006] The present application has been made in view of the above, and aims to determine the similarity of images showing clothing and accessories by taking into consideration the importance of each component element in the images. [Means for solving the problem]

[0007] The information processing device of the present application is characterized by having an acquisition unit that acquires a reference image, which is an image of a wearer wearing an item of clothing, that is an image that serves as a reference, and a comparison image, which is an image of a wearer wearing the item of clothing, that is an image to be compared with the reference image; a judgment unit that judges the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance of each component of the image in evaluating the image of the wearer wearing the item of clothing; and an output unit that outputs the judgment result by the judgment unit. [Effects of the Invention]

[0008] According to one aspect of the embodiment, it is possible to determine the similarity of images showing clothing articles by taking into consideration the importance of each component element in the images. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a process for determining the similarity between a reference image and a comparative image. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of the information processing device 10 according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of the reference image database 31 according to the embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the importance database 32 according to the embodiment. [Figure 7]FIG. 7 is a diagram showing an example of the wearer information database 33 according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a procedure for information processing according to the embodiment. [Figure 9] FIG. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 10. As shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0011] (Embodiment) [1. Information Processing System Configuration] First, an information processing system 1 according to an embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment. As shown in FIG. 1, the information processing system 1 includes an information processing device 10 and a user terminal 100. The information processing device 10 and the user terminal 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 shown in FIG. 1 may include a plurality of information processing devices 10 and a plurality of user terminals 100.

[0012] The information processing device 10 is an information processing device that performs information processing to determine the similarity between a reference image, which is an image of a wearer wearing clothing items and serves as a reference, and a comparison image, which is an image of a wearer wearing the clothing items and is an image to be compared with the reference image, and output the determination result, and is realized, for example, by a server device, a cloud system, etc. For example, in the example shown in FIG. 2 , the information processing device 10 performs information processing to determine the similarity between a reference image, which is an image of a wearer wearing any one of clothing items (also called fashion items) such as clothing, footwear (also called shoes), headwear (e.g., caps, hats, etc.), ornaments (also called accessories), bags, etc., or an image of a wearer wearing a combination of multiple clothing items (also called coordination), and the comparison image, and output the determination result. In other words, clothing items are a concept that includes not only items worn, such as clothing, but also items owned by a user. In addition, clothing and accessories may include any object that has the function of decorating the wearer when photographed with the wearer, such as makeup (also called make-up), tableware, various interior items, studios, rooms, etc.

[0013] Furthermore, for example, the information processing device 10 provides an electronic commerce service for providing (selling) clothing and accessories. The information processing device 10 also provides a coordination service in which content (for example, still images, moving images, articles, etc.) showing coordination of clothing and accessories is submitted by users and provided to other users.

[0014] The information processing device 10 may have a function as a web server that provides websites related to e-commerce services or coordination services. The information processing device 10 may also be a device that distributes information to be displayed on applications related to e-commerce services or coordination services installed in the user terminal 100 to the user terminal 100. The information processing device 10 may also be a device that distributes the application data itself.

[0015] Furthermore, the information processing device 10 may function as a distribution device that distributes control information to the user terminal 100. Here, the control information is written in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the information processing device 10 may be regarded as the control information.

[0016] The user terminal 100 is an information processing device used by a user. Here, the user may be, for example, a wearer who wears an article of clothing or a person other than the wearer, such as the wearer's family, friend, or lover, or a salesperson at a store that provides the article of clothing to the wearer, etc. The user terminal 100 may be realized, for example, by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant). The user terminal 100 displays information distributed by the information processing device 10 or a server device that provides a predetermined service, using a web browser or an application. The example shown in FIG. 2 illustrates a case where the user terminal 100 is a smartphone.

[0017] [2. An example of information processing] Next, an example of information processing realized by the information processing device etc. according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of information processing according to this embodiment. In the following description, user terminals 100-1 to 100-N (N is an arbitrary natural number) will be described according to the user using the user terminal 100. For example, user terminal 100-1 is the user terminal 100 used by a user (user U1) identified by a user ID "UID#1". In the following description, user terminals 100-1 to 100-N will be referred to as user terminal 100 when there is no particular distinction between them. In the following description, user terminal 100 may be considered to be the same as the user. In other words, in the following description, user can also be read as user terminal 100.

[0018] First, the information processing device 10 acquires an image (reference image) of a wearer wearing an article of clothing or a combination of articles of clothing, or a wearer image, which is an image of the wearer himself, from the user terminal 100 (step S1). For example, the information processing device 10 acquires an image posted by the user to a coordination service as the reference image. Also, for example, the information processing device 10 acquires an image registered by the user to an e-commerce service as the reference image. As an example, the information processing device 10 acquires an image of a user (wearer) U1 wearing the article of clothing combination #1 from the user terminal 100-1 as the reference image Pa1.

[0019] The information processing device 10 also acquires a wearer image (e.g., a full-body image or a facial image of the wearer) to be used for the virtual try-on, along with a request for virtual try-on of the clothing item indicated by the reference image. For example, the information processing device 10 acquires information indicating that a selection operation has been performed on a reference image or a button corresponding to the reference image in a coordination service or an e-commerce service, as a request for virtual try-on of the clothing item indicated by the reference image. The information processing device 10 also acquires an image of the wearer captured by the user terminal 100 as the wearer image.

[0020] 2, it is assumed that a user (wearer) U2 has requested a virtual try-on of the combination #1 shown in the reference image Pa1. In this case, the information processing device 10 generates an image (comparison image) of the user U2 wearing the combination #1 based on the reference image Pa1 and the wearer image #1 showing the user U2 (step S2). For example, the information processing device 10 generates the comparison image based on the reference image Pa1, the wearer image #1, and a predetermined keyword (for example, a keyword such as "user U2 wearing the combination #1").

[0021] As a specific example, the information processing device 10 generates a comparison image by inputting the reference image Pa1, the wearer image #1, and a predetermined keyword to model #1, which has been trained to generate an image corresponding to an input keyword from an input image using an image generation method that uses a diffusion model such as stable diffusion. Note that the information processing device 10 can employ any model as model #1 as long as the model can generate an image with content corresponding to the keyword.

[0022] Here, various image generation AIs, such as diffusion models, generate images by gradually restoring an image containing at least a portion of random noise to an image containing the target indicated by the keyword. Therefore, when comparison images are generated using various image generation AIs, different images will be generated even when the same keyword is input due to the randomness of the random noise used by the diffusion model to generate images. Therefore, the information processing device 10 uses model #1 to generate multiple images of user U2 wearing combination #1. As a result, in the example shown in FIG. 2, comparison images Pb1 to Pb4 are generated.

[0023] Next, the information processing device 10 determines the importance of each component of the image in evaluating the image of the wearer wearing the clothing item (step S3). Here, it is assumed that the category (also called genre) of combination #1 is "casual." In this case, the information processing device 10 determines the importance of each component in the category "casual" based on reference image group #1 (in other words, a group of learning images) that belong to the category "casual" among reference images posted to a coordination service or reference images registered in an e-commerce service.

[0024] As a specific example, the information processing device 10 inputs a reference image belonging to a reference image group #1 and a masked image in which components of the reference image have been subjected to a predetermined masking process into model #2, which has been trained to output a score indicating an evaluation of the image when an image of a wearer wearing an article of clothing is input.The information processing device 10 determines the importance of each masked component based on the difference between the score of the reference image and the score of the masked image.As an example, if the score output by model #2 after masking a certain component is lower by a predetermined threshold or more than when masking is not performed, the information processing device 10 sets the importance of the component higher than the predetermined threshold.In other words, the greater the degree of decrease in the score after masking a certain component, the higher the importance value set by the information processing device 10 for that component.

[0025] Here, model #2 is trained to output a score indicating, for example, when a training image (e.g., an image posted to a coordination service or an image registered in an e-commerce service) is input, an evaluator's evaluation of the image (e.g., whether the combination of clothing items shown in the image suits (in other words, harmonious) or whether the clothing items or the combination of clothing items shown in the image is fashionable). For example, the information processing device 10 acquires a first training image and a second training image that is rated higher than the first training image (e.g., has a higher number of likes or a higher number of views). Then, the information processing device 10 trains model #2 using a training method such as backpropagation so that, when the first training image is input to model #2, model #2 outputs a score that is lower than the score that model #2 outputs when the second training image is input to model #2. In this way, any known technology can be applied to training model #2, and a learning method appropriately selected depending on the information used as training data may be used. For example, model #2 may be trained using various conventional machine learning techniques (e.g., supervised machine learning techniques such as SVM (Support Vector Machine)). Also, model #2 may be trained using deep learning techniques. For example, model #2 may be trained using various deep learning techniques such as RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network).

[0026] The information processing device 10 inputs the reference images belonging to the reference image group #1 into the model #2 that has been trained as described above, and obtains the score #1 (for example, the average value of the scores of the reference images belonging to the reference image group #1).

[0027] In addition, the information processing device 10 inputs a mask image into model #2, in which a predetermined masking process has been performed on components such as each area in a reference image belonging to reference image group #1, the shape of the clothing item shown in the reference image, and the color of the clothing item shown in the reference image, and obtains a score for the mask image.

[0028] To give a specific example, the information processing device 10 inputs mask image #1, which is a reference image in which the top area (in other words, the upper body of the wearer) has been masked (for example, filled in with black), into model #2 and obtains score #11 for mask image #1.

[0029] In addition, the information processing device 10 inputs mask image #2, which is obtained by masking the bottoms area of ​​the reference image (in other words, the area below the waist and above the ankles of the wearer), into model #2 and obtains score #12 for mask image #2.

[0030] Furthermore, the information processing device 10 inputs a mask image #3, which is obtained by masking the footwear area (in other words, the area below the ankle) of the reference image, into the model #2 and obtains a score #13 for the mask image #3.

[0031] In addition, the information processing device 10 inputs mask image #4 (in other words, an image that retains the image of the color of the clothing item while blurring the shape of each part such as the collar and sleeves) into model #2, where mask processing (for example, a process of filling each clothing item with the average color of each clothing item) has been performed on the shape of the clothing item shown in the reference image, and obtains score #14 for mask image #4.

[0032] In addition, the information processing device 10 inputs a mask image #5, which has been subjected to mask processing (e.g., processing to convert each clothing item to grayscale) for the colors of the clothing items shown in the reference image, into model #2 and obtains a score #15 for mask image #5.

[0033] Then, the information processing device 10 determines the importance #1 of the tops area in the "casual" category based on the difference between the score #1 and the score #11. For example, the information processing device 10 determines the importance #1 to be higher as the difference between the score #1 and the score #11 increases.

[0034] Similarly, the information processing device 10 determines the importance #2 of the bottoms area in the "casual" category based on the difference between score #1 and score #12. The information processing device 10 determines the importance #3 of the footwear area in the "casual" category based on the difference between score #1 and score #13. The information processing device 10 determines the importance #4 of the shape in the "casual" category based on the difference between score #1 and score #14. The information processing device 10 determines the importance #5 of the color in the "casual" category based on the difference between score #1 and score #15.

[0035] In this way, by performing a masking process on a certain component in an image showing a clothing item of a specified category and determining whether the similarity with the original image changes, it is possible to determine whether the component was important in that category.

[0036] Next, the information processing device 10 determines the similarity (in other words, the degree of similarity) between the reference image Pa1 and the comparison images Pb1 to Pb4 based on the similarity between the components of the reference image Pa1 and the components of the comparison images Pb1 to Pb4 and the importance of each component (step S4).

[0037] An example of a process for determining the similarity between a reference image and a comparison image will now be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a process for determining the similarity between a reference image and a comparison image. In the example of Fig. 3, it is assumed that the similarity between a reference image Pa1 and a comparison image Pb1 is determined.

[0038] For example, the information processing device 10 determines the similarity between a top area AR11 in the reference image Pa1 and a top area AR21 in the comparison image Pb1 using any image recognition technology. As a specific example, the information processing device 10 weights the similarity of the shape of the subject (top) in areas AR11 and AR21 with importance #4, and further weights the similarity of the color of the subject in areas AR11 and AR21 with importance #5, thereby determining similarity #1 between areas AR11 and AR21.

[0039] Furthermore, the information processing device 10 determines the similarity between bottoms region AR12 in the reference image Pa1 and bottoms region AR22 in the comparison image Pb1 using any image recognition technology. As a specific example, the information processing device 10 weights the similarity of the shape of the subject (bottoms) in regions AR12 and AR22 with importance #4, and further weights the similarity of the color of the subject in regions AR12 and AR22 with importance #5, thereby determining similarity #2 between regions AR12 and AR22.

[0040] Furthermore, the information processing device 10 determines the similarity between a footwear area AR13 in the reference image Pa1 and a footwear area AR23 in the comparison image Pb1 using any image recognition technology. As a specific example, the information processing device 10 weights the similarity of the shape of the photographed subject (footwear) in areas AR13 and AR23 with importance #4, and further weights the similarity of the color of the photographed subject in areas AR13 and AR23 with importance #5, thereby determining similarity #3 between areas AR13 and AR23.

[0041] Then, the information processing device 10 weights similarity #1 by importance #1, similarity #2 by importance #2, and similarity #3 by importance #3, and determines the similarity between the reference image Pa1 and the comparison image Pb1.

[0042] The information processing device 10 performs the same process as above to determine the similarity between the reference image Pa1 and the comparative image Pb2, the similarity between the reference image Pa1 and the comparative image Pb3, and the similarity between the reference image Pa1 and the comparative image Pb4.

[0043] Returning to FIG. 2, the explanation continues. Next, the information processing device 10 outputs the determination result to the user terminal 100-2 (step S5). For example, the information processing device 10 outputs, via a coordination service or an e-commerce service, an image of the comparison images Pb1 to Pb4 whose similarity to the reference image Pa1 is equal to or greater than a predetermined threshold, as an image of the user U2 virtually trying on the combination #1. Alternatively, the information processing device 10 may output, from the comparison images Pb1 to Pb4, the image that has the highest similarity to the reference image Pa1. Alternatively, the information processing device 10 may output the comparison images Pb1 to Pb4 in descending order of similarity to the reference image Pa1.

[0044] Next, the information processing device 10 acquires the annotation result from the user terminal 100-2 (step S6). For example, the information processing device 10 acquires the annotation result indicating whether the combination of clothing items shown in the comparison image output in step S5 suits the user. The information processing device 10 may also acquire the annotation result indicating whether the clothing item or the combination of clothing items shown in the comparison image output in step S5 is fashionable. Then, the information processing device 10 trains the model #2 using the output comparison image and the annotation result.

[0045] As described above, the information processing device 10 according to the embodiment determines the importance of each component element of an image of a wearer wearing an article of clothing in evaluating the image, and determines the similarity between a reference image and a comparison image based on the determined importance. That is, the information processing device 10 according to the embodiment can determine the similarity of the images by taking into account the importance of each component element in an image showing an article of clothing.

[0046] Furthermore, conventionally, when a diffusion model is used to generate an image of a person wearing a predetermined clothing item, a random noise image is generated, and therefore an image of a person wearing the desired clothing item is not necessarily obtained. Therefore, the information processing device 10 according to the embodiment determines the similarity between a reference image showing the desired clothing item and a comparison image generated using the diffusion model, and can select and output an image showing a person wearing a clothing item similar to the desired clothing item from the generated comparison images.

[0047] [3. Other processing examples] The above-described process is merely an example, and the information processing device 10 may perform various processes using various information. In this regard, examples are listed below.

[0048] [3-1. Determining the importance of each occasion] In the example of FIG. 2, the information processing device 10 may determine the importance for each occasion on which the clothing item is worn. For example, if information indicating the occasion "date" is linked to reference image Pa1 in a coordination service or e-commerce service, the information processing device 10 determines the importance of each component element for the occasion "date" based on a group of reference images linked to the information indicating the occasion "date" in the coordination service or e-commerce service. To give a more specific example, the information processing device 10 may prepare a model similar to model #2 for each occasion using learning images prepared for each occasion, and output an appropriate virtual try-on image by determining the importance using model #2 corresponding to the occasion specified by the user.

[0049] [3-2. Components] The image components are not limited to those described above and may be any. For example, the image components may be the patterns or designs of clothing items. In such cases, the information processing device 10 determines the importance of the patterns or designs based on a mask image obtained by masking the clothing items shown in the image by painting them with a predetermined color (for example, the color that occupies the largest proportion of the clothing items). Then, the information processing device 10 determines whether to output a comparison image to the user based on whether the patterns or designs with high importance are similar to the reference image.

[0050] Furthermore, each region, which is a component of an image, is not limited to the above, and may be set arbitrarily. For example, each region may be a region obtained by dividing the image into a predetermined number of regions with a predetermined area (for example, regions obtained by dividing the image into two regions, vertically or horizontally, or regions obtained by dividing the image into a grid pattern (for example, into four or eight divisions)). Furthermore, each region may be a region corresponding to a part of an article of clothing, such as a collar or a hem.

[0051] Furthermore, the image components may be set for each category of clothing items or each occasion for wearing the clothing items. For example, the information processing device 10 may determine the importance for each combination of at least one of the category and the occasion and the component, and determine the similarity between the reference image and the comparison image based on the determined importance.

[0052] [3-3. Outputting comparison images] 2, the information processing device 10 may output, to an evaluator (annotator) who evaluates the combination of clothing and accessories, images among the comparison images Pb1 to Pb4 whose similarity to the reference image Pa1 is equal to or greater than a predetermined threshold value. Then, the information processing device 10 may train model #2 on the output images and the evaluation results by the evaluator.

[0053] Here, as described above, Model #2 is trained based on training images showing combinations of clothing items and evaluations of the training images, and is therefore capable of estimating whether combinations of multiple clothing items sold on e-commerce services, etc., match (are harmonious). Using such Model #2, it is possible to suggest other clothing items (e.g., bags and accessories that match a combination of a dress and shoes) that match clothing items that a user is planning to purchase or that the user owns. Training Model #2 requires images of combinations of multiple clothing items that have been evaluated as matching (e.g., evaluations such as likes) or that can be reasonably estimated to be evaluated as such (i.e., images as positive examples), and images that have been evaluated as not matching (or that have been evaluated as matching by fewer people or with fewer evaluations of matching than images evaluated as matching) (i.e., images as negative examples).

[0054] Preparing a large number of such training images is time-consuming. Therefore, there is a desire to use a diffusion model or the like to generate images to be used for training Model #2. However, Model #2 is required to learn the "degree of match between combinations of multiple clothing items" rather than the "evaluation of the clothing items themselves" or the "similarity of the clothing items themselves." In such a case, the diffusion model itself generates images containing randomness without considering the degree of match between the clothing items that generate the images.

[0055] Therefore, the information processing device 10 performs the above-described process on the posted reference image and the comparison image generated using the diffusion model, and selects an image among the comparison images whose similarity to the reference image is equal to or greater than a predetermined threshold as a training image. The information processing device 10 then provides the reference image and the training image to an annotator, and obtains an evaluation of which image shows a combination of multiple clothing items that is a better match. Thereafter, if the reference image is evaluated as a better match, the information processing device 10 may train model #2 so that when the reference image is input to model #2, it outputs a higher score than when the training image is input. Also, if the training image is evaluated as a better match, the information processing device 10 may train model #2 so that when the training image is input to model #2, it outputs a higher score than when the reference image is input. This may improve the accuracy of model #2.

[0056] [3-4. About the comparison images] In the example of Fig. 2, the comparison image is not limited to an image generated for virtual try-on, and may be any image. For example, the comparison image may be an image posted by a user to a coordination service or an image registered in an e-commerce service. In such a case, the information processing device 10 may output to the user via the coordination service, e-commerce service, etc., a comparison image whose similarity to an image (reference image) selected or viewed by the user in the coordination service, e-commerce service, etc. is equal to or greater than a predetermined threshold.

[0057] [3-5. Determining Importance] 2, the information processing device 10 may determine the importance for each level of clothing category (in other words, the ratio of each category). For example, the information processing device 10 determines the importance for each component element in "Casual: 7, Natural: 3" or "Mode: 6, Casual: 4".

[0058] [3-6. Determining importance based on categories] In the example of Fig. 2, the information processing device 10 may determine the importance of each component of an image of a wearer in evaluating the image of the wearer based on a change in category when each component is masked. For example, the information processing device 10 generates model #3 that has been trained to output the category of the clothing item indicated in an image of a wearer wearing a clothing item when the image is input. As a specific example, the information processing device 10 uses data (e.g., data posted to a coordination service or data registered in an e-commerce service) in which an image of a wearer wearing a clothing item is associated with the category of the clothing item indicated in the image as training data, and generates model #3 by training to output the category associated with the image when an image of the training data is input.

[0059] Then, the information processing device 10 inputs into model #3 a reference image belonging to reference image group #1 whose category is "casual" and a mask image in which a predetermined masking process has been performed on the components of the reference image, and determines the importance of each component that has been masked based on the difference (degree of similarity) between the category of the reference image and the category of the mask image.

[0060] As an example, the information processing device 10 inputs a mask image obtained by masking a certain component #1 of a reference image into model #3, and if the output category differs from the output category (i.e., "casual") obtained by inputting the reference image on which the mask image was based into model #3, it determines that component #1 is an important component in the category "casual" and sets the importance of component #1 higher than a predetermined threshold.

[0061] In addition, the information processing device 10 inputs a mask image obtained by masking component #1 of the reference image into model #3, and inputs the output category into model #3 based on the reference image on which the mask image was based, and if the output category matches, it determines that component #1 is an unimportant component in the category "casual," and sets the importance of component #1 to less than a predetermined threshold.

[0062] In other words, if performing mask processing on a component of the reference image changes the category, the information processing device 10 sets a high value of importance for that component, and if performing mask processing on a component of the reference image does not change the category, the information processing device 10 sets a low value of importance for that component.

[0063] In addition, the information processing device 10 may input a mask image obtained by masking component #1 of a reference image into model #3, and even if the output category is different from the category obtained by inputting the reference image on which the mask image was based into model #3 and outputting it, the more similar the two categories are (for example, the closer their positions are in the distributed representation space), the lower the importance may be set, and the more dissimilar the categories are, the higher the importance may be set.

[0064] Furthermore, the information processing device 10 may set the importance of component #1 higher the more reference images belonging to the reference image group #1 that are in a different category from the mask image that has been subjected to mask processing for component #1, and may set the importance of component #1 lower the fewer reference images there are.

[0065] In this way, the information processing device 10 determines the importance of each component in the category “casual.” Then, the information processing device 10 determines the similarity between the reference image Pa1 and the comparison image Pb1 based on the similarity between the components of the reference image Pa1 and the components of the comparison image Pb1, and the importance of each component in the category “casual.”

[0066] The information processing device 10 may train model #3 so that, when an image of a wearer wearing an item of clothing is input, the information processing device 10 outputs the ratio of categories of the item of clothing indicated in the image (for example, "casual: 7, natural: 3"). If the ratio of categories changes when masking a certain component of the reference image (for example, when the ratio of each category changes by a predetermined threshold or more, or when ratios of different categories are output), the information processing device 10 may set a high value of importance for the component, and if the ratio of categories does not change when masking a certain component of the reference image (for example, when the ratios of each category are the same, or when the ratios of each category do not change by a predetermined threshold or more), the information processing device 10 may set a low value of importance for the component.

[0067] Furthermore, in the above configuration, the “category” may be “occasion.” Even when the “category” is “occasion,” the same process as above is performed to determine the importance of each component element for each occasion.

[0068] 4. Configuration of Information Processing Device Next, the configuration of the information processing device 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 10 according to an embodiment. As shown in Fig. 4, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0069] (Regarding the communication unit 20) The communication unit 20 is realized by, for example, a network interface card (NIC), etc. The communication unit 20 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the user terminal 100, etc.

[0070] (Regarding the storage unit 30) The storage unit 30 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in Fig. 4, the storage unit 30 has a reference image database 31, an importance database 32, a wearer information database 33, and a model database 34.

[0071] (Regarding Reference Image Database 31) The reference image database 31 stores various information related to reference images. An example of the information stored in the reference image database 31 will now be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the reference image database 31 according to the embodiment. In the example of FIG. 5, the reference image database 31 has items such as "reference image ID," "reference image information," "category," and "occasion."

[0072] "Reference image ID" indicates identification information for identifying the reference image. "Reference image information" indicates information related to the reference image, and for example, an image of a wearer wearing an item of clothing (reference image) is stored. "Category" indicates the category to which the item of clothing shown in the reference image belongs. "Occasion" indicates the occasion on which the item of clothing shown in the reference image is worn. "Component information" indicates the components of the reference image. "Importance" indicates the importance of the component.

[0073] That is, Figure 5 shows an example in which the reference image information of the reference image identified by the reference image ID "PID#1" is "Reference Image Information #1", the category of the clothing item indicated by the reference image is "Category #1", and the occasion on which the clothing item is worn is "Occasion #1".

[0074] (About Importance Database 32) The importance database 32 stores various information related to the importance of each component element of an image in the evaluation of an image of a wearer wearing an article of clothing. An example of information stored in the importance database 32 will now be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the importance database 32 according to the embodiment. In the example of FIG. 6, the importance database 32 has items such as "category," "occasion," "component information," and "importance."

[0075] "Category" indicates the category of the clothing item. "Occasion" indicates the occasion on which the clothing item is worn. "Component information" indicates the components of the image. "Importance" indicates the importance of the component.

[0076] That is, Figure 6 shows an example in which the component of an image showing an item of clothing that belongs to the category "Category #1" and is to be worn on "Occasion #1" is "Component Information #1," and the importance of the component in "Category #1" and "Occasion #1" is "Importance #1."

[0077] (About wearer information database 33) The wearer information database 33 stores various types of information related to the wearer. Here, an example of information stored in the wearer information database 33 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the wearer information database 33 according to the embodiment. In the example of FIG. 7, the wearer information database 33 has items such as "wearer ID," "attribute information," "wearer image," "purchase history," and "browsing history."

[0078] "Wearer ID" indicates identification information for identifying the wearer. "Attribute information" indicates the attributes of the wearer. "Wearer image" indicates an image of the wearer themselves. "Purchase history" indicates the wearer's purchase history in e-commerce services, etc. "Browse history" indicates the wearer's browsing history in e-commerce services, coordination services, etc.

[0079] That is, Figure 7 shows an example in which the attribute information of a wearer identified by wearer ID "UID#1" is "attribute information #1", the wearer image is "wearer image #1", the purchase history is "purchase history #1", and the browsing history is "browsing history #1".

[0080] (About Model Database 34) The model database 34 stores a model trained to generate an image corresponding to an input keyword from an input image. The model database 34 also stores a model trained to output a score indicating an evaluation of an image of a wearer wearing a clothing item when the image is input. The model database 34 also stores a model trained to output a category of the clothing item indicated by the image when the image of a wearer wearing the clothing item is input. The model database 34 also stores a model trained to output an occasion for the clothing item indicated by the image when the image of a wearer wearing the clothing item is input.

[0081] (Regarding the control unit 40) The control unit 40 is a controller, and is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the information processing device 10 using RAM as a work area. The control unit 40 is also a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in FIG. 4 , the control unit 40 according to the embodiment has an acquisition unit 41, a determination unit 42, a determination unit 43, and an output unit 44, and realizes or executes the functions and actions of information processing described below.

[0082] (Regarding the acquisition unit 41) The acquisition unit 41 acquires a reference image, which is an image of a wearer wearing a clothing item, and a comparison image, which is an image of a wearer wearing the clothing item and is an image to be compared with the reference image. For example, in the example of FIG. 2, the acquisition unit 41 acquires an image posted by the wearer to the coordination service as the reference image and stores it in the reference image database 31. The acquisition unit 41 also acquires an image of the wearer to be used in virtual try-on, and acquires an image generated based on the wearer image as the comparison image.

[0083] The acquisition unit 41 may also acquire a reference image, which is an image of a wearer wearing a combination of multiple clothing items, and a comparison image, which is an image of a wearer wearing a combination of multiple clothing items, to be compared with the reference image. For example, in the example of FIG. 2, the acquisition unit 41 acquires an image of a wearer wearing clothing combination #1 as the reference image Pa1. The acquisition unit 41 also acquires comparison images Pb1 to Pb4 of the wearer wearing the combination of clothing items.

[0084] The acquisition unit 41 may also acquire a reference image and comparative images, which are images generated based on the reference image and are images of a wearer wearing an accessory corresponding to the accessory shown in the reference image. For example, in the example of Fig. 2, the acquisition unit 41 acquires the reference image Pa1 and comparative images Pb1-Pb4, which are generated based on the reference image Pa1 and show the user U2 wearing the combination #1.

[0085] The acquisition unit 41 may also acquire a reference image and comparative images that are images of a wearer different from the wearer shown in the reference image. For example, in the example of Fig. 2, the acquisition unit 41 acquires a reference image Pa1 that is an image of a wearer U1 wearing clothing item combination #1, and comparative images Pb1 to Pb4 that show a user U2 wearing combination #1.

[0086] (Regarding the decision unit 42) The determination unit 42 inputs a reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process into a model that has been trained to output a score indicating an evaluation of an image of a wearer wearing an article of clothing when the image is input, and determines the importance of each masked component based on the difference between the score of the reference image and the score of the mask image. For example, in the example of Fig. 2, the determination unit 42 refers to the model database 34, and inputs a reference image belonging to the reference image group #1 and a mask image in which components of the reference image have been subjected to a predetermined masking process into model #2 that has been trained to output a score indicating an evaluation of an image when an image of a wearer wearing an article of clothing is input, and determines the importance of each masked component based on the difference between the score of the reference image and the score of the mask image, and stores the importance in the importance database 32.

[0087] Furthermore, the determination unit 42 may determine the importance of the masked component to be higher as the difference between the score of the reference image and the score of the mask image increases. For example, in the example of Fig. 2, the determination unit 42 determines the importance #1 to be higher as the difference between score #1 and score #11 increases.

[0088] The determining unit 42 may also determine the importance for each category of clothing and accessories. For example, in the example of Fig. 2, the determining unit 42 determines the importance for each component element in the category "casual."

[0089] The determination unit 42 may also determine the importance for each occasion that the clothing item is worn in. For example, in the example of Fig. 2, the determination unit 42 determines the importance for each component element for the occasion "date."

[0090] Furthermore, the determination unit 42 may input a reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model that has been trained to output the category of the clothing item indicated in an image of a wearer wearing the clothing item when the image is input, and determine the importance of each masked component based on the difference between the category of the clothing item indicated in the reference image and the category of the clothing item indicated in the mask image. For example, in the example of Fig. 2, the determination unit 42 refers to the model database 34, and inputs a reference image belonging to the reference image group #1 and a mask image in which components of the reference image have been subjected to a predetermined masking process to model #3 that has been trained to output the category of the clothing item indicated in an image of a wearer wearing the clothing item when the image is input, and determines the importance of each masked component based on the difference between the category of the clothing item indicated in the reference image and the category of the clothing item indicated in the mask image, and stores the importance in the importance database 32.

[0091] Furthermore, the determination unit 42 may input a reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model that has been trained to output the occasion of the clothing item indicated in an image of a wearer wearing the clothing item when the image is input, and determine the importance of each masked component based on the difference between the occasion of the clothing item indicated in the reference image and the occasion of the clothing item indicated in the mask image. For example, in the example of Fig. 2, the determination unit 42 refers to the model database 34, and inputs a reference image belonging to reference image group #1 and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model that has been trained to output the occasion of the clothing item indicated in the image when an image of a wearer wearing the clothing item is input, and determines the importance of each masked component based on the difference between the occasion of the clothing item indicated in the reference image and the occasion of the clothing item indicated in the mask image, and stores the importance in the importance database 32.

[0092] (Regarding the determination unit 43) The determination unit 43 determines the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image, and the importance of each constituent element of the image in evaluating an image of a wearer wearing an apparel item. For example, in the example of Fig. 2, the determination unit 43 refers to the importance database 32 and determines the similarity between the reference image Pa1 and the comparison images Pb1 to Pb4 based on the similarity between the constituent elements of the reference image Pa1 and the constituent elements of the comparison images Pb1 to Pb4 and the importance of each constituent element.

[0093] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between each region of the reference image and each region of the comparison image, and the importance of each region of the image in evaluating an image of a wearer wearing an apparel item. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between each region, such as a top region, a bottom region, or a footwear region, in the reference image Pa1 and each region, such as a top region, a bottom region, or a footwear region, in the comparison image Pb1, according to the importance of each region, and determines the similarity between the reference image Pa1 and the comparison image Pb1.

[0094] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between the color of the clothing item shown in the reference image and the color of the clothing item shown in the comparison image, and the importance of each color of the clothing item. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between the color of the clothing item shown in the reference image Pa1 and the color of the clothing item shown in the comparison image Pb1 using importance #5, and determines the similarity between the reference image Pa1 and the comparison image Pb1.

[0095] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between the shape of the clothing item shown in the reference image and the shape of the clothing item shown in the comparison image, and the importance of each clothing item shape. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between the shape of the clothing item shown in the reference image Pa1 and the shape of the clothing item shown in the comparison image Pb1 using importance #4, and determines the similarity between the reference image Pa1 and the comparison image Pb1.

[0096] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparative image based on the similarity between the pattern of the clothing item shown in the reference image and the pattern of the clothing item shown in the comparative image, and the importance of each pattern of the clothing item. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between the pattern of the clothing item shown in the reference image Pa1 and the pattern of the clothing item shown in the comparative image Pb1 according to the importance of the pattern in the category "casual," and determines the similarity between the reference image Pa1 and the comparative image Pb1.

[0097] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparative image based on the similarity between the pattern of the clothing item shown in the reference image and the pattern of the clothing item shown in the comparative image, and the importance of each pattern of the clothing item. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between the pattern of the clothing item shown in the reference image Pa1 and the pattern of the clothing item shown in the comparative image Pb1 according to the importance of the pattern in the category "casual," and determines the similarity between the reference image Pa1 and the comparative image Pb1.

[0098] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between each clothing item shown in the reference image and each clothing item shown in the comparison image, and the importance of each clothing item in a combination of multiple clothing items. For example, in the example of Fig. 2, the determination unit 43 weights the similarity between the shape of the clothing item shown in the reference image Pa1 and the shape of the clothing item shown in the comparison image Pb1 using importance #4, and determines the similarity between the reference image Pa1 and the comparison image Pb1.

[0099] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance determined by the determination unit 42. For example, in the example of FIG. 2, the determination unit 43 determines the similarity between the reference image Pa1 and the comparison image Pb1 based on the importance determined by the determination unit 42.

[0100] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparative image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparative image, and the importance in the category of the clothing items indicated by the reference image and the comparative image. For example, in the example of Fig. 2, the determination unit 43 determines the similarity between the reference image Pa1 and the comparative image Pb1 based on the similarity between the constituent elements of the reference image Pa1 and the constituent elements of the comparative image Pb1, and the importance of each constituent element in the category "casual."

[0101] Furthermore, the determination unit 43 may determine the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance of the clothing items represented by the reference image and the comparison image in the occasion for wearing them. For example, in the example of Fig. 2, the determination unit 43 determines the similarity between the reference image Pa1 and the comparison image Pb1 based on the similarity between the components of the reference image Pa1 and the components of the comparison image Pb1, and the importance of each component in the occasion "date."

[0102] (Regarding the output unit 44) The output unit 44 outputs the determination result by the determination unit 43. For example, in the example of Fig. 2, the output unit 44 outputs the determination result to the user terminal 100-2.

[0103] Furthermore, the output unit 44 may output, from among the comparison images, images whose similarity to the reference image satisfies a predetermined condition. For example, in the example of Fig. 2, the output unit 44 outputs, via the coordination service, an image from among the comparison images Pb1 to Pb4 whose similarity to the reference image Pa1 is equal to or greater than a predetermined threshold, as an image of user U2 virtually trying on combination #1. Furthermore, the information processing device 10 outputs, from among the comparison images Pb1 to Pb4, the image that is most similar to the reference image Pa1.

[0104] Furthermore, when an image of a wearer wearing an article of clothing is input, the output unit 44 may output, as information for training a model that has been trained to output information indicating an evaluation of the image, images of the comparison images whose similarity to the comparison image is equal to or greater than a predetermined threshold. For example, in the example of Fig. 2, the output unit 44 outputs, from the comparison images Pb1 to Pb4, images whose similarity to the reference image Pa1 is equal to or greater than a predetermined threshold.

[0105] The model to which information is output by the output unit 44 is not limited to a model such as model #2 that has been trained to output a score indicating an evaluation of an image of a wearer wearing an item of clothing when the image is input, but may be any model that is capable of evaluating an image of a wearer wearing an item of clothing.

[0106] For example, when an image of a wearer wearing clothing items and an instruction sentence instructing the output unit 44 to output whether the combination of clothing items shown in the image suits them, or whether the clothing items and combination of clothing items shown in the image are fashionable, are input, the output unit 44 may output information to a model that has been trained to output an answer (e.g., text information such as "it suits you" or "it doesn't suit you"). Note that such a model may be a large-scale language model (LLM) that performs natural language processing, such as a generative pre-trained transformer (GPT) or a transformer, that has been trained to handle multimodal input.

[0107] [5. Information Processing Flow] The procedure of information processing of the information processing device 10 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the procedure of information processing according to the embodiment.

[0108] 8, the information processing device 10 determines whether or not it has acquired a reference image, which is an image of a wearer wearing an apparel item and serves as a reference, and a comparison image, which is an image of a wearer wearing the apparel item and is an image to be compared with the reference image (step S101). If the reference image and the comparison image have not been acquired (step S101; No), the information processing device 10 waits until it acquires the reference image and the comparison image.

[0109] On the other hand, if the reference image and the comparison image are acquired (step S101; Yes), the information processing device 10 determines the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image and the importance of each component of the image (step S102). Subsequently, the information processing device 10 outputs the determination result (step S103) and ends the process.

[0110] [6. Modifications] The above-described embodiment is merely an example, and various modifications and applications are possible.

[0111] [6-1. Processing mode] Of the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, and conversely, all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information, including the processing procedures, specific names, various data, and parameters shown in the above text and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0112] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0113] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0114] 6-2. Processing by the user terminal 100 The user terminal 100 may execute all or part of the processes described in the above embodiments.

[0115] [7. Effects] As described above, the information processing device 10 according to the embodiment includes an acquisition unit 41, a determination unit 42, a judgment unit 43, and an output unit 44. The acquisition unit 41 acquires a reference image, which is an image of a wearer wearing clothing items and serves as a reference, and a comparison image, which is an image of a wearer wearing the clothing items and is to be compared with the reference image. The acquisition unit 41 also acquires a reference image, which is an image of a wearer wearing a combination of multiple clothing items and serves as a reference, and a comparison image, which is an image of a wearer wearing the combination of multiple clothing items and is to be compared with the reference image. The determination unit 42 inputs the reference image and a masked image, in which components of the reference image have been subjected to a predetermined masking process, to a model trained to output a score indicating an evaluation of an image of a wearer wearing clothing items when the image is input, and determines the importance of each masked component based on the difference between the score of the reference image and the score of the masked image. The determination unit 42 also determines the importance of the masked component to be higher the greater the difference between the score of the reference image and the score of the masked image. The determination unit 42 inputs a reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model trained to output the category of the clothing item indicated in an image of a wearer wearing the clothing item when the image is input. The determination unit 42 then inputs the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model trained to output the occasion of the clothing item indicated in the image when the image of a wearer wearing the clothing item is input. The determination unit 42 then inputs the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model trained to output the occasion of the clothing item indicated in the image when the image of a wearer wearing the clothing item is input. The determination unit 42 then inputs the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model trained to output the occasion of the clothing item indicated in the image. The determination unit 43 then determines the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image and the importance of each component of the image in evaluating the image of the wearer wearing the clothing item.Furthermore, the determination unit 43 determines the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance determined by the determination unit 42. The output unit 44 outputs the determination result by the determination unit 43. Furthermore, the output unit 44 outputs, from among the comparison images, images whose similarity to the reference image satisfies a predetermined condition.

[0116] As a result, the information processing device 10 of the embodiment can determine the importance of each component of an image when evaluating an image of a wearer wearing a clothing item, and can determine the similarity between the reference image and the comparison image based on the determined importance, thereby determining the similarity of the images by taking into account the importance of each component in the image showing the clothing item.

[0117] In the information processing device 10 according to the embodiment, for example, the acquisition unit 41 acquires a reference image and a comparison image, which is an image generated based on the reference image and is an image of a wearer wearing a clothing item corresponding to the clothing item shown in the reference image. The acquisition unit 41 also acquires the reference image and a comparison image, which is an image of a wearer different from the wearer shown in the reference image.

[0118] As a result, the information processing device 10 according to the embodiment can select and output an image from the generated images showing a person wearing an item of clothing similar to the desired item of clothing during virtual fitting, thereby improving the accuracy of the information to be output.

[0119] Furthermore, in the information processing device 10 according to the embodiment, for example, the determination unit 42 determines the importance for each category of clothing items. Then, the judgment unit 43 determines the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance for the category of clothing items indicated by the reference image and the comparison image. Also, the determination unit 42 determines the importance for each occasion in which the clothing items are worn. Then, the judgment unit 43 determines the similarity between the reference image and the comparison image based on the similarity between the components of the reference image and the components of the comparison image, and the importance for the occasion in which the clothing items indicated by the reference image and the comparison image are worn.

[0120] As a result, the information processing device 10 according to the embodiment can determine the importance for each category of clothing item and each occasion for wearing the clothing item, and determine the similarity between the reference image and the comparison image, thereby improving the accuracy of the information to be output.

[0121] Furthermore, in the information processing device 10 according to the embodiment, for example, the determination unit 43 determines the similarity between the reference image and the comparison image based on the similarity between the color of the clothing item shown in the reference image and the color of the clothing item shown in the comparison image, and the importance of each color of the clothing item. The determination unit 43 also determines the similarity between the reference image and the comparison image based on the similarity between the shape of the clothing item shown in the reference image and the shape of the clothing item shown in the comparison image, and the importance of each shape of the clothing item. The determination unit 43 also determines the similarity between the reference image and the comparison image based on the similarity between the pattern of the clothing item shown in the reference image and the pattern of the clothing item shown in the comparison image, and the importance of each pattern of the clothing item. The determination unit 43 also determines the similarity between the reference image and the comparison image based on the similarity between the design of the clothing item shown in the reference image and the design of the clothing item shown in the comparison image, and the importance of each design of the clothing item. In addition, the judgment unit 43 judges the similarity between the reference image and the comparison image based on the similarity between each clothing item shown in the reference image and each clothing item shown in the comparison image, as well as the importance of each clothing item in the combination of multiple clothing items.

[0122] As a result, the information processing device 10 according to the embodiment can determine the similarity between the reference image and the comparison image based on the importance determined for each of the various components, thereby improving the accuracy of the information to be output.

[0123] Furthermore, in the information processing device 10 according to the embodiment, for example, when an image of a wearer wearing an article of clothing is input, the output unit 44 outputs, as information for training a model that has been trained to output information indicating an evaluation of the image, an image among the comparison images whose similarity to the comparison image is equal to or greater than a predetermined threshold.

[0124] As a result, the information processing device 10 according to the embodiment can output information used for learning the model, thereby improving the accuracy of the model.

[0125] [8. Hardware Configuration] The information processing device 10 according to each of the above-described embodiments is realized, for example, by a computer 1000 configured as shown in Fig. 9. The following description will be given taking the information processing device 10 as an example. Fig. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 10. The computer 1000 has a CPU 1100, a ROM 1200, a RAM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0126] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1200 or the HDD 1400. The ROM 1200 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0127] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a communication network 500 (corresponding to the network N in the embodiment) and sends the data to the CPU 1100, and also transmits data generated by the CPU 1100 to other devices via the communication network 500.

[0128] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0129] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1300. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1300 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0130] For example, when the computer 1000 functions as the information processing device 10, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1300 to realize the functions of the control unit 40. The HDD 1400 also stores various data in the storage device of the information processing device 10. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0131] [9. Other] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have been modified and improved in various ways based on the knowledge of those skilled in the art.

[0132] Furthermore, the information processing device 10 described above can flexibly change its configuration, for example, by calling an external platform or the like using an API (Application Programming Interface) or network computing, depending on the function.

[0133] Furthermore, the term "unit" in the claims can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0134] 10. Information processing equipment 20 Communications Department 30 Storage section 31 Reference Image Database 32 Importance Database 33 Wearer information database 34 Model Database 40 Control Unit 41 Acquisition Department 42 Decision Section 43 Judgment section 44 Output section 100 user terminals

Claims

1. an acquisition unit that acquires a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, and is an image to be compared with the reference image; a determination unit that, when a training image that is an image of a wearer wearing an article of clothing is input, outputs a score indicating an evaluation of the training image, inputs the reference image and a masked image in which components of the reference image have been subjected to a predetermined masking process to the model that has been trained to output a score indicating an evaluation of the training image, and determines the importance in the evaluation of the reference image for each masked component based on the difference between the score of the reference image and the score of the masked image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

2. An acquisition unit that acquires a reference image, which is an image of a wearer wearing an article of clothing, and a comparison image, which is an image of a wearer wearing the article of clothing, for comparison with the reference image; a determination unit that, when a training image that is an image of a wearer wearing an article of clothing is input, outputs a score indicating an evaluation of the training image, inputs the reference image and a mask image in which a predetermined masking process is performed on an area of ​​the reference image to a trained model, and determines the importance in the evaluation of the reference image for each area that has been masked based on the difference between the score of the reference image and the score of the mask image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

3. An acquisition unit that acquires a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination unit that, when a training image that is an image of a wearer wearing an accessory is input, inputs the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process to a model that has been trained to output the category of the accessory indicated by the training image, and determines the importance in evaluating the reference image for each masked component based on the difference between the category of the accessory indicated by the reference image and the category of the accessory indicated by the mask image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

4. An acquisition unit that acquires a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination unit that, when a training image that is an image of a wearer wearing an accessory is input, inputs the reference image and a mask image in which a predetermined mask process is performed on an area of ​​the reference image to a model that has been trained to output the category of the accessory indicated by the training image, and determines the importance in evaluating the reference image for each masked area based on the difference between the category of the accessory indicated by the reference image and the category of the accessory indicated by the mask image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

5. An acquisition unit that acquires a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination unit that, when a learning image that is an image of a wearer wearing an accessory is input, inputs the reference image and a mask image in which a predetermined mask process is performed on components of the reference image to a model that has been trained to output the occasion of the accessory indicated in the learning image, and determines the importance in evaluating the reference image for each masked component based on the difference between the occasion of the accessory indicated in the reference image and the occasion of the accessory indicated in the mask image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

6. An acquisition unit that acquires a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination unit that, when a learning image that is an image of a wearer wearing an accessory is input, inputs the reference image and a mask image in which a predetermined mask process is performed on an area of ​​the reference image to a model that has been trained to output the occasion of the accessory indicated in the learning image, and determines the importance in evaluating the reference image for each area that has been masked based on the difference between the occasion of the accessory indicated in the reference image and the occasion of the accessory indicated in the mask image; a determination unit that determines the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination unit; an output unit that outputs a determination result by the determination unit; An information processing device comprising:

7. 1. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determining step of inputting a reference image and a masked image in which components of the reference image have been subjected to a predetermined masking process into a model that has been trained to output a score indicating an evaluation of the reference image when a training image that is an image of a wearer wearing an article of clothing is input, and determining the importance in the evaluation of the reference image for each masked component based on the difference between the score of the reference image and the score of the masked image; a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

8. an acquisition step of acquiring a reference image, which is an image of a wearer wearing the clothing item and serves as a reference image, and a comparison image, which is an image of a wearer wearing the clothing item and is an image to be compared with the reference image; a determination procedure in which, when a training image which is an image of a wearer wearing an article of clothing is input, a model which has been trained to output a score indicating an evaluation of the training image is input, the reference image and a masked image in which components of the reference image have been subjected to a predetermined masking process are input to the model, and the importance in the evaluation of the reference image is determined for each masked component based on the difference between the score of the reference image and the score of the masked image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

9. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determining step of inputting a reference image and a masked image in which a predetermined masking process has been performed on an area of ​​the reference image into a model that has been trained to output a score indicating an evaluation of the reference image when the model is trained to output a score indicating an evaluation of the reference image, the masked image being an image of a wearer wearing an article of clothing; and determining the importance of each masked area in the evaluation of the reference image based on the difference between the score of the reference image and the score of the masked image. a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

10. An acquisition procedure for acquiring a reference image, which is an image of a wearer wearing the clothing item and serves as a reference, and a comparison image, which is an image of a wearer wearing the clothing item and is an image for comparison with the reference image; a determination procedure in which, when a training image which is an image of a wearer wearing an article of clothing is input, a model which has been trained to output a score indicating an evaluation of the training image is input, the reference image and a mask image in which a predetermined masking process has been performed on an area of ​​the reference image are input to the model, and the importance in the evaluation of the reference image is determined for each area which has been masked based on the difference between the score of the reference image and the score of the mask image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

11. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination step of inputting the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process into a model that has been trained to output the category of the clothing item indicated by a training image of a wearer wearing the clothing item when the training image is input, and determining the importance in the evaluation of the reference image for each masked component based on the difference between the category of the clothing item indicated by the reference image and the category of the clothing item indicated by the mask image; a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

12. An acquisition step of acquiring a reference image, which is an image of a wearer wearing the clothing item and serves as a reference, and a comparison image, which is an image of a wearer wearing the clothing item and is an image for comparison with the reference image; a determination procedure in which, when a training image that is an image of a wearer wearing an accessory is input, a model that has been trained to output the category of the accessory indicated by the training image is input, and the reference image and a mask image in which components of the reference image have been subjected to a predetermined masking process are input to the model, and the importance of each masked component in the evaluation of the reference image is determined based on the difference between the category of the accessory indicated by the reference image and the category of the accessory indicated by the mask image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

13. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination step of inputting the reference image and a mask image in which a predetermined masking process has been performed on an area of ​​the reference image into a model that has been trained to output the category of the clothing item indicated by a training image of a wearer wearing the clothing item when the training image is input, and determining the importance in the evaluation of the reference image for each masked area based on the difference between the category of the clothing item indicated by the reference image and the category of the clothing item indicated by the mask image; a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

14. An acquisition step of acquiring a reference image, which is an image of a wearer wearing the clothing item and serves as a reference, and a comparison image, which is an image of a wearer wearing the clothing item and is an image for comparison with the reference image; a determination procedure in which, when a training image that is an image of a wearer wearing an accessory is input, a model that has been trained to output the category of the accessory indicated by the training image is input, the reference image and a mask image in which a predetermined mask process has been performed on an area of ​​the reference image are input, and the importance in evaluating the reference image is determined for each masked area based on the difference between the category of the accessory indicated by the reference image and the category of the accessory indicated by the mask image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

15. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination step of inputting the reference image and a mask image in which a predetermined masking process has been performed on components of the reference image into a model that has been trained to output the occasion of the clothing item indicated in the training image when a training image that is an image of a wearer wearing the clothing item is input, and determining the importance in evaluating the reference image for each masked component based on the difference between the occasion of the clothing item indicated in the reference image and the occasion of the clothing item indicated in the mask image; a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

16. An acquisition step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination procedure in which, when a learning image which is an image of a wearer wearing an accessory is input, a model which has been trained to output the occasion of the accessory indicated in the learning image is input, the reference image and a mask image in which a predetermined mask processing has been performed on components of the reference image are input, and the importance of each masked component in the evaluation of the reference image is determined based on the difference between the occasion of the accessory indicated in the reference image and the occasion of the accessory indicated in the mask image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the constituent elements of the reference image and the constituent elements of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

17. A computer-implemented information processing method, comprising: an acquiring step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination step of inputting the reference image and a mask image in which a predetermined masking process has been performed on an area of ​​the reference image into a model that has been trained to output the occasion of the clothing item indicated in the training image when a training image that is an image of a wearer wearing the clothing item is input, and determining the importance in the evaluation of the reference image for each area that has been masked based on the difference between the occasion of the clothing item indicated in the reference image and the occasion of the clothing item indicated in the mask image; a determining step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined in the determining step; an output step of outputting the determination result obtained by the determination step; An information processing method comprising:

18. An acquisition step of acquiring a reference image, which is an image of a wearer wearing the clothing item, and a comparison image, which is an image of a wearer wearing the clothing item, for comparison with the reference image; a determination procedure in which, when a training image which is an image of a wearer wearing an accessory is input, a model which has been trained to output the occasion of the accessory indicated in the training image is input, the reference image and a mask image in which a predetermined mask process has been performed on an area of ​​the reference image are input, and the importance of each masked area in the evaluation of the reference image is determined based on the difference between the occasion of the accessory indicated in the reference image and the occasion of the accessory indicated in the mask image; a determination step of determining the similarity between the reference image and the comparison image based on the similarity between the region of the reference image and the region of the comparison image and the importance determined by the determination step; an output step for outputting a determination result obtained by the determination step; An information processing program characterized by causing a computer to execute the above.

Images