Cosmetics trial system, trial method and program

Abstract

To provide a cosmetic trial system, trial method and program enabling a check of an appearance of virtually trying out another item without removing the applied item in a photographing environment where the application of a certain item has been photographed.SOLUTION: A selection receiving unit 101 of a cosmetics trial system receives selection input to select a first item that a user tries out. An application image acquisition unit 102 acquires an application image obtained by capturing the user after applying a second item. A specification unit 103 specifies an area where the second item is applied, in the acquired application image. A conversion calculation unit 106 calculates a conversion for converting the second color associated with the second item into a color of pixels in the specified area. A trial image acquisition unit 107 acquires a trial image by replacing the color of the pixels in the specified area with a color obtained by applying the calculated conversion to the first color associated with the first item. An output unit 108 outputs the trial image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a cosmetic trial system, a trial method, and a program. [Background technology]

[0002] There are known techniques for virtually applying makeup to a user's facial image. For example, Patent Document 1 discloses a technique for generating a face image from a face image of a user with makeup applied, from which the makeup has been removed, and applying a selected makeup style to the face image from which the makeup has been removed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-36784 Summary of the Invention [Problem to be solved by the invention]

[0004] While the above-described techniques allow users to virtually try on various items without actually removing the makeup, they require processing to generate a facial image with the makeup removed. However, there is a demand for a simpler method for checking the appearance of virtually trying on various cosmetic items without the need for processing to generate a facial image with the makeup removed.

[0005] The present invention is intended to solve the above-mentioned problems, and aims to provide a cosmetic trial system, trial method, and program that allows a user to virtually try on another item in a photographed environment in which the application of a certain item is photographed, without removing the applied item. [Means for solving the problem]

[0006] A cosmetic trial system according to a first aspect of the present invention comprises: a selection receiving unit that receives a selection input from a user to select a first item to be tried; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a conversion calculation unit that calculates a conversion for converting the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing the color of the pixel in the identified region in the acquired application image with a color obtained by applying the determined transformation to the acquired first color; an output unit that outputs the acquired trial image; The present invention is characterized by comprising:

[0007] In addition, in the cosmetic trial system according to the above aspect, The identification unit identifies the area where the first item is to be applied as the area where the second item is applied in the application image. It is characterized by:

[0008] In addition, in the cosmetic trial system according to the above aspect, the second color acquisition unit acquires candidate items that are candidates for the second item from the purchase history of the user, and acquires, from among the colors of the candidate items, a color that is closest to a color of a pixel in the specified area as the second color; The candidate item having the closest color is designated as the second item. It is characterized by:

[0009] In addition, in the cosmetic trial system according to the above aspect, a designation receiving unit that acquires candidate items that are candidates for the second item from the purchase history of the user, presents a list of the candidate items, and receives a designation input that designates any of the candidate items in the list; The candidate item related to the specified input is set as the second item. It is characterized by:

[0010] In addition, in the cosmetic trial system according to the above aspect, The identification unit identifies the area where the second item is to be applied as the area where the second item is applied in the application image. It is characterized by:

[0011] In addition, in the cosmetic trial system according to the above aspect, the first color acquisition unit acquires the first color associated with the first item by referring to a catalog database in which information on colors associated with items is registered; The second color acquisition unit refers to the catalog database and acquires a second color associated with the second item. It is characterized by:

[0012] In addition, in the cosmetic trial system according to the above aspect, The catalog database contains items from different manufacturers, The first item and the second item are items from different manufacturers. It is characterized by:

[0013] In addition, in the cosmetic trial system according to the above aspect, The trial image acquired by the application image acquisition unit is a still image, the conversion calculation unit determines the conversion by creating a filter that converts the acquired second color into a color of a pixel in the identified region on a pixel-by-pixel basis; The trial image acquisition unit converts the acquired first color on a pixel-by-pixel basis using the filter, and acquires the trial image by replacing the color of each pixel of the identified region in the acquired application image with the color converted on a pixel-by-pixel basis. It is characterized by:

[0014] In addition, in the cosmetic trial system according to the above aspect, the test image acquired by the application image acquisition unit is a frame image included in a moving image, the conversion calculation unit determines a representative color for the specified region and generates a filter that converts the acquired second color into the representative color, thereby determining the conversion; the application image acquisition unit converts the acquired first color using the filter, and applies the converted color to the identified area in the acquired application image to acquire the trial image; The output unit outputs the acquired trial image as a frame image included in a moving image. It is characterized by:

[0015] In addition, in the cosmetic trial system according to the above aspect, The conversion calculation unit determines, as the representative color, a color represented by an average value, a median value, or a mode value of pixel values ​​in the specified area. It is characterized by:

[0016] In addition, in the cosmetic trial system according to the above aspect, The conversion calculation unit clusters pixel values ​​of the identified region and determines a color represented by a mean value, a median value, or a mode value of the largest cluster as the representative color. It is characterized by:

[0017] A trial method according to a second aspect of the present invention comprises: A cosmetic trial method executed by a cosmetic trial system, The trial system comprises: accepting a selection input from the user to select a first item to try; acquiring an application image capturing a state of the user applying the second item; Identifying an area where the second item is applied in the acquired application image; obtaining a first color associated with the first item; obtaining a second color associated with the second item; determining a transformation that converts the obtained second color to a color of a pixel in the identified region; a trial image is obtained by replacing the color of the pixel in the identified region in the obtained application image with a color obtained by applying the obtained transformation to the obtained first color; outputting the acquired trial image; It is characterized by:

[0018] A program according to a third aspect of the present invention comprises: Computer, a selection receiving unit that receives a selection input from a user to select a first item to be tried; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit that calculates a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing the color of the pixel in the identified region in the acquired application image with a color obtained by applying the determined transformation to the acquired first color; an output unit for outputting the acquired trial image; The present invention is characterized in that it functions as a

[0019] The program may be recorded on a non-transitory recording medium. The non-transitory recording medium can be distributed or sold independently of the computer. Here, a non-transitory recording medium refers to a tangible recording medium. Examples of non-transitory recording media include compact discs, flexible disks, hard disks, magneto-optical disks, digital video disks, magnetic tapes, and semiconductor memories. A transitory recording medium refers to the transmission medium (propagation signal) itself. Examples of transitory recording media include electrical signals, optical signals, and electromagnetic waves. A temporary storage area is an area for temporarily storing data and programs, such as volatile memory such as RAM (Random Access Memory). [Effects of the Invention]

[0020] According to the present invention, it is possible to provide a cosmetic trial system, trial method, and program that allows a user to virtually try on another item in a photographed environment in which the application of a certain item is photographed, without removing the applied item. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram showing the relationship between a cosmetic trial system according to a first embodiment, a catalog database, and an online market server. [Figure 2] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Figure 3] 1 is a diagram showing the functional configuration of a cosmetic product trial system according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an application image according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a trial image according to the first embodiment. [Figure 6] 1 is a diagram for explaining a trial process executed by a cosmetic trial system according to the first embodiment. FIG. [Figure 7]FIG. 10 is a diagram showing the functional configuration of a cosmetic trial system according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a list according to the second embodiment. [Figure 9] FIG. 10 is a diagram for explaining a trial process executed by a cosmetic trial system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] (1. Overall structure) A cosmetics trial system 1 according to an embodiment of the present invention is a system that presents a virtual trial of cosmetics to a user. As shown in Fig. 1, the cosmetics trial system 1 includes a server device 100 and a terminal device 200, and the server device 100, the terminal device 200, the catalog database 300, and the online market server 400 are all communicatively connected via a computer communication network 500 such as the Internet.

[0023] The server device 100 is a device that executes various processes for presenting a virtual try-on of makeup. The server device 100 provides various information in response to a request from the terminal device 200.

[0024] The terminal device 200 is a terminal that captures an image of a user trying on makeup and presents the user virtually trying on the makeup. For example, the terminal device 200 is a smartphone, a tablet computer, or the like. For example, an app for virtually trying on makeup (hereinafter simply referred to as "app") is installed in the terminal device 200. The terminal device 200 is operated by the user himself or herself, or, when used in a store, by a store staff member.

[0025] The catalog database 300 is a database in which information about items is registered. An item is a cosmetic product that a user applies to the face or body. For example, items include lipstick, blush, eye shadow, hair color, nail color, etc. Color information associated with an item is registered in the catalog database 300 as information about the item. The items whose information is registered in the catalog database 300 are items handled in the online marketplace managed by the online marketplace server 400. Therefore, items from different manufacturers are registered in the catalog database 300. The color information associated with an item may be registered in a database separate from the catalog database 300.

[0026] The online market server 400 is a server that manages the online market. In response to a request from the cosmetic trial system 1, the online market server 400 provides the cosmetic trial system 1 with information about the user's purchase history.

[0027] (2. Hardware configuration of information processing device) FIG. 2 shows an example of the hardware configuration of the information processing device 10 in which the server device 100 and the terminal device 200 are realized.

[0028] 2, the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a recording medium 14, an output device 15, a communication device 16, and an input device 17. The components are connected to each other via a bus 18.

[0029] The CPU 11 controls the overall operation of the information processing device 10, and is connected to each component to exchange control signals and data.

[0030] The ROM 12 stores operating programs and various data necessary for controlling the overall operation of the information processing device 10.

[0031] The RAM 13 is used to temporarily store data and programs, and stores programs and data read from the recording medium 14, as well as other data necessary for communication.

[0032] The recording medium 14 is composed of a hard disk, a flash memory, or the like, and records data to be processed by the information processing device 10.

[0033] The output device 15 includes a display device such as an LCD (Liquid Crystal Display) and an audio output device such as a speaker. The output device 15 outputs, for example, data output from the CPU 11 under the control of the CPU 11.

[0034] The communication device 16 includes a communication interface for connecting the information processing device 10 to a computer communication network such as the Internet, and communicates with other information processing devices and the like via the communication device 16 .

[0035] The input device 17 includes input devices such as buttons, a keyboard, a touch panel, a camera, a microphone, etc. The input device 17 receives an operation input from the user of the information processing device 10 and outputs a signal corresponding to the received operation input to the CPU 11.

[0036] (3. Functional configuration of the cosmetic trial system of embodiment 1) The functional configuration of the cosmetic trial system 1 of the first embodiment will be described with reference to FIG.

[0037] The cosmetic trial system 1 of the first embodiment functionally comprises a selection receiving unit 101, an application image acquisition unit 102, an identification unit 103, a first color acquisition unit 104, a second color acquisition unit 105, a conversion calculation unit 106, a trial image acquisition unit 107, and an output unit 108. These functional units may be realized by the server device 100 and the terminal device 200 working together, or may be realized by the server device 100 or the terminal device 200 alone, or may be realized by the server device 100 alone.

[0038] The following describes an example in which the above functional units are realized by the server device 100 and the terminal device 200 working together to present a virtual try-on of an item as an online service. Specifically, the server device 100 realizes the functions of the identification unit 103, the second color acquisition unit 105, the conversion calculation unit 106, and the trial image acquisition unit 107, and the terminal device 200 realizes the functions of the selection acceptance unit 101, the application image acquisition unit 102, the first color acquisition unit 104, and the output unit 108.

[0039] 4 shows an example of an app that runs on the terminal device 200. The app 600 includes an area 601 in which an application image or a trial image (described later) is displayed, a field 602 for displaying a first item selected by the user, a field 603 for displaying the color of the first item, a field 604 for displaying a second item applied by the user, and a field 605 for displaying the color of the second item.

[0040] The first item is an item that the user virtually tries on. The second item is an item that is applied to the user and is applied to the same area as the first item. App 600 shows the virtual application of the first item to the user's area where the second item has been applied.

[0041] Here, the first item and the second item may be items from different manufacturers, as long as the color information of the items is registered in the catalog database 300.

[0042] For example, a user can use the app 600 to view an image of themselves wearing lipstick from "Maker X" and see how it would look if they were virtually trying on lipstick from "Maker Y."

[0043] The selection receiving unit 101 receives a selection input from the user to select a first item to be tried out. The selection receiving unit 101 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0044] For example, when the user 700 selects, by tapping or the like, the field 602 in which the first item is displayed in the app 600, the selection receiving unit 101 displays a list of items registered in the catalog database 300. Then, when the user 700 selects "Lipstick 1" from the list, the selection receiving unit 101 receives the selection input of "Lipstick 1" and transmits information about "Lipstick 1" as the first item to the server device 100. The selection receiving unit 101 also displays "Lipstick 1" in the field 602 in which the first item of the app 600 is displayed.

[0045] The application image acquisition unit 102 acquires an application image in which a user applies the second item. The application image acquisition unit 102 is realized by the CPU 11, the communication device 16, and the input device 17.

[0046] The application image is an image of a user to whom the second item has been applied.

[0047] Examples of application images are shown in area 601 of Fig. 4. Application image 601-1 shows a state in which lip balm, which is the second item, has been applied to lips 701 of user 700.

[0048] For example, when a user 700 uses the camera of the terminal device 200 to display and photograph an application image 601-1 in the area 601 as shown in FIG. 4, the application image acquisition unit 102 acquires the application image 601-1 and transmits information about the application image 601-1 to the server device 100.

[0049] Here, the application image acquired by the application image acquisition unit 102 is a still image or a frame image included in a moving image. The application image acquisition unit 102 may acquire the still image or the moving image while the application is running, or may acquire it from a library of images acquired in the past.

[0050] For example, the application image acquisition unit 102 acquires a live video of a user taking a selfie of themselves applying an item they own at home. Alternatively, the application image acquisition unit 102 acquires a live video of a user applying an item at a store. Alternatively, the application image acquisition unit 102 acquires a still image of the user applying an item from the library of the terminal device 200.

[0051] The identification unit 103 identifies an area where the second item is applied in the acquired application image. The identification unit 103 is realized by the CPU 11.

[0052] Specifically, the identifying unit 103 identifies the area where the first item should be applied as the area where the second item is applied in the application image.

[0053] For example, the identification unit 103 performs facial recognition on the application image transmitted from the terminal device 200 and identifies the area in the application image where the second item is applied. For example, if the first item is lipstick, the area to be applied is the lips, so the identification unit 103 identifies the area of ​​lips 701 in the application image 601-1 as the area where the second item is applied. Alternatively, if the first item is eye shadow, the area to be applied is the eyelids, so the identification unit 103 identifies the area of ​​the eyelids in the application image 601-1 as the area where the second item is applied.

[0054] The first color acquisition unit 104 acquires a first color associated with the first item. The first color acquisition unit 104 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0055] The first color is the color of the first item that the user virtually tries on.

[0056] Specifically, the first color acquisition unit 104 refers to the catalog database 300 to acquire the first color associated with the first item.

[0057] For example, when the user 700 selects, by tapping or the like, the field 603 in the app 600 where the first color is displayed, the first color acquisition unit 104 displays a list of color variations for "Lip 1" registered in the catalog database 300. For example, the list may include "Color 1-1" to "Color 1-7." When the user 700 selects "Color 1-3" from the list, the first color acquisition unit 104 accepts the selection input of "Color 1-3" and transmits information about "Color 1-3" to the server device 100 as the first color. In addition, the first color acquisition unit 104 displays "Color 1-3" in the field 603 in the app 600 where the first color is displayed.

[0058] The second color acquisition unit 105 acquires a second color associated with the second item. The second color acquisition unit 105 is realized by the CPU 11 and the communication device 16.

[0059] The second color is the color of the second item that the user applies to themselves.

[0060] Specifically, the second color acquisition unit 105 acquires candidate items for the second item from the user's purchase history, and acquires, from among the colors of the candidate items, a color that is closest to the color of the pixels in the identified area as the second color. Here, the second color acquisition unit 105 refers to the catalog database 300 to acquire the second color associated with the second item.

[0061] For example, the second color acquisition unit 105 acquires the purchase history of the user 700 from the online market server 400. The second color acquisition unit 105 acquires, from among the items included in the acquired purchase history of the user 700, items in the same category as the first item as candidate items. If the second color acquisition unit 105 acquires, as candidate items, "Lipstick 2," "Lipstick 3," and "Lipstick 4," which are items in the same category as "Lipstick 1," then the second color acquisition unit 105 refers to the catalog database 300 to acquire colors associated with each of the multiple candidate items. If the second color acquisition unit 105 acquires "Color 2-1" to "Color 2-5" associated with "Lip 2," "Color 3-1" to "Color 3-4" associated with "Lip 3," and "Color 4-1" to "Color 4-6" associated with "Lip 4," it identifies the color from among "Color 2-1" to "Color 2-5," "Color 3-1" to "Color 3-4," and "Color 4-1" to "Color 4-6" that is closest to the color of the pixel of lip 701 in application image 601-1 as the second color.

[0062] For example, if the color closest to the color of the pixel of lips 701 in application image 601-1 is "color 2-1," second color acquisition section 105 identifies "color 2-1" as the second color.

[0063] The server device 100 then selects the candidate item with the closest color as the second item.

[0064] For example, the server device 100 sets "Lip 2" associated with the second color "Color 2-1" as the second item. In this case, as shown in FIG. 4, "Lip 2" is displayed in the field 604 where the second item of the app 600 is displayed, and "Color 2-1" is displayed in the field 605 where the second color is displayed.

[0065] The user can modify the automatically set second item and second color. When field 604 is selected by tapping or the like, it displays a list of items that are registered in catalog database 300 and are in the same category as the first item. The user can select a second item from the list. When field 605 is selected by tapping or the like, it displays a list of colors that are registered in catalog database 300 for the item displayed in field 604. The user can select a second color from the list.

[0066] The conversion calculation unit 106 calculates a conversion for converting the acquired second color into the color of the pixel in the specified region. The conversion calculation unit 106 is realized by the CPU 11.

[0067] For example, the transformation calculation unit 106 calculates a transformation that transforms the second color "Color 2-1" of the second item "Lip 2" into the color of the pixels in the area of ​​lips 701 identified in the application image 601-1.

[0068] Specifically, when the application image acquired by the application image acquisition unit 102 is a still image, the conversion calculation unit 106 calculates the conversion by creating a filter that converts the acquired second color into the color of the pixels in the specified area on a pixel-by-pixel basis.

[0069] For example, if the application image 601-1 is a still image, the conversion calculation unit 106 creates a filter that converts the second color "color 2-1" into the color i of pixel i (i is 1 or more and N or less, N is the number of pixels included in the area) in the lip 701 area, i.e., a filter that multiplies the R value, G value, and B value by k_ri, k_gi, and k_bi.

[0070] Alternatively, if the application image acquired by the application image acquisition unit 102 is a frame image included in a video, the conversion calculation unit 106 determines the representative color for the identified area and calculates the conversion by creating a filter that converts the acquired second color into the representative color.

[0071] For example, if application image 601-1 is a frame image included in a moving image, conversion calculation unit 106 first obtains the representative color of the lip 701 area.

[0072] Specifically, the conversion calculation unit 106 determines the color represented by the average, median, or mode of the pixel values ​​in the identified region as the representative color.

[0073] For example, the conversion calculation unit 106 calculates the average value, median value, or mode value of each of the R value, G value, and B value for pixel i in the region of the lips 701. The conversion calculation unit 106 sets the color obtained by combining the average value, median value, or mode value of the calculated R value, G value, and B value as the representative color of the identified region. Alternatively, the conversion calculation unit 106 sets the mode value of the combination of the R value, G value, and B value of pixel i as the representative color.

[0074] Alternatively, the conversion calculation unit 106 clusters the pixel values ​​of the identified region, and determines the color represented by the average value, median value, or mode value of the largest cluster as the representative color.

[0075] For example, the conversion calculation unit 106 clusters three-dimensional vectors represented by R, G, and B values ​​for pixel i in the lip 701 area using the k-means method, and determines the color represented by the average, median, or mode of the R, G, and B values ​​of the three-dimensional vectors included in the largest cluster as the representative color. Alternatively, the conversion calculation unit 106 determines the color represented by the R, G, and B values ​​at the center of gravity of the largest cluster as the representative color.

[0076] The conversion calculation unit 106 creates a filter that converts the second color "color 2-1" into the representative color of the area of ​​the lips 701, that is, a filter that multiplies the R value, G value, and B value by k_r, k_g, and k_b, respectively.

[0077] The trial image acquisition unit 107 acquires a trial image by replacing the color of the pixel in the identified region in the acquired application image with a color obtained by applying the determined transformation to the acquired first color. The trial image acquisition unit 107 is realized by the CPU 11 and the communication device 16.

[0078] Specifically, when the application image acquired by the application image acquisition unit 102 is a still image, the trial image acquisition unit 107 converts the acquired first color on a pixel-by-pixel basis using a filter, and acquires a trial image by replacing the color of each pixel in a specified area in the acquired application image with the color converted on a pixel-by-pixel basis.

[0079] For example, when the application image 601-1 is a still image, the trial image acquisition unit 107 obtains a converted color i for pixel i using a filter that multiplies the R value, B value, and G value of the first color "Color 1-3" by k_ri, k_gi, and k_bi. Then, the trial image acquisition unit 107 generates a trial image 601-2 in which the color i of pixel i in the region of the lips 701 in the application image 601-1 is replaced with the converted color i, and transmits information about the trial image 601-2 to the terminal device 200.

[0080] When the application image acquired by the application image acquisition unit 102 is a frame image included in a moving image, the trial image acquisition unit 107 converts the acquired first color using a filter and acquires a trial image by applying the converted color to a specified area in the acquired application image.

[0081] For example, if the application image 601-1 is a frame image included in a moving image, the trial image acquisition unit 107 obtains the converted color using a filter that multiplies the R value, B value, and G value of the first color "Color 1-3" by k_r, k_g, and k_b. Then, the trial image acquisition unit 107 generates a trial image 601-2 in which the color i of pixel i in the lip 701 area in the application image 601-1 is replaced with the converted color, and transmits information about the trial image 601-2 to the terminal device 200.

[0082] The output unit 108 outputs the acquired trial image. The output unit 108 and the selection receiving unit 101 are realized by the CPU 11, the output device 15, and the communication device 16.

[0083] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the app 600, as shown in FIG. 5. For example, if the application image 601-1 is a still image showing the user 700 applying "Lip 2," the trial image 601-2 is an image showing the user 700's lips 701 in the application image 601-1 virtually applying "Lip 1" in "Colors 1-3." For example, if the user selects an image from the library showing the user's application of "Lip 2" and selects "Lip 1" in "Colors 1-3," an image is displayed in the area 601 in which "Lip 1" in "Colors 1-3" has been applied to the area in the library where "Lip 2" has been applied.

[0084] Furthermore, if the application image acquired by the application image acquisition unit 102 is a frame image included in a moving image, the output unit 108 outputs the acquired trial image as a frame image included in the moving image.

[0085] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the app 600 as a frame image of a video. If the application image 601-1 is a frame image of a live video showing the user 700 applying "Lip 2," the trial image 601-2 is a frame image showing the virtual application of "Lip 1" in "Colors 1-3" to the lips 701 of the user 700 in the application image 601-1. For example, if the user takes a selfie video of themselves applying their own "Lip 2" and selects "Lip 1" in "Colors 1-3," a video of "Lip 1" in "Colors 1-3" being applied to the area where "Lip 2" was applied is displayed in real time in the area 601.

[0086] (4. Operation of the Cosmetic Trial System of Embodiment 1) The operation of the cosmetics trial system 1 of this embodiment will be described with reference to Fig. 6. For example, when the application 600 is launched on the terminal device 200, the cosmetics trial system 1 starts the trial process shown in Fig. 6.

[0087] The selection receiving unit 101 receives a selection input from the user to select a first item to be tried (step S101).

[0088] For example, when the user 700 selects, by tapping or the like, the field 602 in which the first item is displayed in the app 600, the selection receiving unit 101 displays a list of items registered in the catalog database 300, and when the user 700 selects “Lip 1” from the list, the selection receiving unit 101 accepts the selection input of “Lip 1” and transmits information about “Lip 1” to the server device 100 as the first item.

[0089] The application image acquisition unit 102 acquires an application image capturing a state of the user applying the second item (Step S102).

[0090] For example, when a user 700 uses the camera of the terminal device 200 to display and photograph an application image 601-1 in the area 601 as shown in FIG. 4, the application image acquisition unit 102 acquires the application image 601-1 and transmits information about the application image 601-1 to the server device 100.

[0091] The identifying unit 103 identifies the area in the acquired application image where the first item should be applied as the area where the second item is applied (step S103).

[0092] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area where the first item “Lip 1” should be applied, as the area where the second item is applied.

[0093] The first color acquisition unit 104 refers to the catalog database 300 and acquires the first color associated with the first item (step S104).

[0094] For example, when the user 700 selects the field 603 in the app 600 where the first color is displayed by tapping or the like, the first color acquisition unit 104 displays a list of color variations for "Lip 1" registered in the catalog database 300, and when the user 700 selects "Color 1-3" from the list, the first color acquisition unit 104 accepts the selection input of "Color 1-3" and transmits information about "Color 1-3" to the server device 100 as the first color.

[0095] The second color acquisition unit 105 refers to the catalog database 300 and acquires the second color associated with the second item (step S105).

[0096] For example, the second color acquisition unit 105 acquires candidate items "Lip 2," "Lip 3," and "Lip 4" that are candidates for the second item based on the purchase history of the user 700 acquired from the online market server 400, and acquires candidate items "Color 2-1" to "Color 2-5," "Color 3-1" to "Color 3-4," and "Color 4-1" to "Color 4-6" registered in the catalog database 300.The second color acquisition unit 105 then automatically identifies, as the second color, the color "Color 2-1" that is closest to the color of the pixel of the lips 701 in the application image 601-1 among "Color 2-1" to "Color 2-5," "Color 3-1" to "Color 3-4," and "Color 4-1" to "Color 4-6."

[0097] The conversion calculation unit 106 calculates a conversion that converts the acquired second color into the color of the pixel in the specified region (step S106).

[0098] For example, the transformation calculation unit 106 calculates a transformation that transforms the second color "color 2-1" into the color of the pixels in the region of the lips 701 identified in the application image 601-1.

[0099] The trial image acquisition unit 107 acquires a trial image by replacing the color of the pixels in the identified area in the acquired application image with a color obtained by applying the determined transformation to the acquired first color (step S107).

[0100] For example, the trial image acquisition unit 107 uses a filter that multiplies the R value, B value, and G value of the first color "Color 1-3" by k_ri, k_gi, and k_bi to obtain the converted color i for pixel i, generates trial image 601-2 in which the color i of pixel i in the area of ​​the lips 701 in the application image 601-1 is replaced with the converted color i, and transmits information about the trial image 601-2 to the terminal device 200.

[0101] The output unit 108 outputs the acquired trial image (step S108).

[0102] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the application 600 as shown in FIG.

[0103] According to this embodiment, in a photographing environment where the second item has been applied, it is possible to confirm the virtual trial use of the first item without wiping off the applied portion of the second item. This makes it possible to easily compare the color of an item that has actually been applied with the color of an item that has not actually been applied in the same photographing environment.

[0104] Furthermore, according to this embodiment, it is possible to identify the items and colors that a user has applied based on the user's purchase history, thereby eliminating the need for the user to search for items that the user has applied currently or in the past.

[0105] Furthermore, according to this embodiment, it is possible to virtually try out items from different manufacturers that are registered in the catalog database. This allows users to compare the colors of items from different manufacturers that are applied to the same area. Furthermore, since the items registered in the catalog database are items available on the online market, users can virtually try out items from various manufacturers and consider purchasing them.

[0106] Furthermore, according to this embodiment, it is possible to compare the color of an item in a still image or video captured in the past, or in a live video, with the color of an item that has not actually been painted. This allows the user to compare the color of an item that has actually been painted with the color of an item that has not actually been painted in various shooting environments.

[0107] Furthermore, according to this embodiment, by applying a representative color to a moving image such as a live video, it is possible to check in real time how an item is virtually tried on. Furthermore, according to this embodiment, by determining the representative color through clustering, it is possible to determine the representative color, for example, in an area where glossy lipstick is applied, excluding the highlights, and to determine an appropriate color as the representative color.

[0108] (5. Functional configuration of the cosmetic trial system of embodiment 2) The functional configuration of the cosmetic trial system 1 of the second embodiment will be described with reference to FIG.

[0109] The cosmetic trial system 1 of embodiment 2 functionally comprises a selection receiving unit 101, an application image acquisition unit 102, a determination unit 103, a first color acquisition unit 104, a second color acquisition unit 105, a conversion calculation unit 106, a trial image acquisition unit 107, an output unit 108, and a designation receiving unit 109. The following describes an example in which the above functional units are realized by a server device 100 and a terminal device 200 working together to present a virtual trial of an item as an online service. Specifically, the server device 100 realizes the functions of the determination unit 103, the second color acquisition unit 105, the conversion calculation unit 106, and the trial image acquisition unit 107, and the terminal device 200 realizes the functions of the selection receiving unit 101, the application image acquisition unit 102, the first color acquisition unit 104, the output unit 108, and the designation receiving unit 109. In the following, functions that differ from the functions of the trial system 1 of the first embodiment will be described.

[0110] The specification receiving unit 109 obtains candidate items for the second item from the user's purchase history, presents a list of candidate items, and receives a specification input specifying any candidate item in the list. The specification receiving unit 109 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0111] For example, when the user 700 selects, by tapping or the like, a field 604 in which the second item is displayed in the app 600, the designation receiving unit 109 acquires the purchase history of the user 700 from the online market server 400. Next, the designation receiving unit 109 acquires items included in the acquired purchase history of the user 700 as candidate items. The designation receiving unit 109 displays a list 606 of candidate items, as shown in FIG. 8 . Then, when the user 700 selects “Lipstick 2” from the list as the second item, the designation receiving unit 109 accepts the designation input of “Lipstick 2” and receives the information of “Lipstick 2” from the server device 100.

[0112] The server device 100 sets the candidate item related to the specified input as the second item.

[0113] For example, the server device 100 sets "Lipstick 2" as the second item. The designation receiving unit 109 also displays "Lipstick 2" in the field 604 in which the second item of the application 600 is displayed.

[0114] The identifying unit 103 identifies the area where the second item is to be applied as the area where the second item is applied in the application image.

[0115] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area where the second item “Lip 2” should be applied, as the area where the second item is applied.

[0116] The second color acquisition unit 105 refers to the catalog database 300 and acquires the second color associated with the second item designated by the user.

[0117] For example, the second color acquisition unit 105 refers to the catalog database 300 to acquire "Color 2-1" to "Color 2-5" associated with the second item "Lip 2." Then, the second color acquisition unit 105 identifies, from among "Color 2-1" to "Color 2-5," the color closest to the color of the pixel of lips 701 in application image 601-1 as the second color.

[0118] (6. Operation of the Cosmetics Trial System of Embodiment 2) The operation of the cosmetics trial system 1 of this embodiment will be described with reference to Fig. 9. For example, when the application 600 is launched on the terminal device 200, the cosmetics trial system 1 starts the trial process shown in Fig. 9. Note that the processes of steps S201 and S205 to S209 are the same as the processes of steps S102, S106, S101, S104, S107, and S108 in the flowchart of Fig. 6.

[0119] In step S201, when the application image acquisition unit 102 acquires an application image, the designation receiving unit 109 acquires candidate items that are candidates for the second item from the user's purchase history, presents a list of candidate items, and receives designation input to designate any candidate item in the list (step S202).

[0120] For example, the designation receiving unit 109 displays a list 606 of candidate items in the application 600 of the terminal device 200 as shown in FIG. 8, and receives designation input information of “Lip 2” from the application 600 of the terminal device 200.

[0121] The identifying unit 103 identifies the area where the second item is to be applied as the area where the second item is applied in the application image (step S203).

[0122] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area where the second item "Lip 2" should be applied, as the area where the second item "Lip 2" is applied.

[0123] The second color acquisition unit 105 acquires a second color associated with the second item (step S204).

[0124] For example, when the second color acquisition unit 105 refers to the catalog database 300 and acquires "Color 2-1" to "Color 2-5" associated with "Lip 2," it automatically identifies, as the second color, "Color 2-1" from among "Color 2-1" to "Color 2-5," which is closest to the color of the pixel of lip 701 in application image 601-1.

[0125] In step S204, when the second color acquisition unit 105 acquires the second color, the processes from step S205 onwards are executed.

[0126] According to this embodiment, by presenting a list of candidate items that the user has applied based on the user's purchase history, it is possible to save the user the trouble of searching for items that the user has applied currently or in the past. Furthermore, by having the user select from a list of items generated based on the user's purchase history, it is possible to allow the user to accurately select the items that they have applied.

[0127] (7. Variations) Although the embodiment of the present invention has been described above, various modifications and applications are possible in carrying out the present invention.

[0128] In the above embodiment, the server device 100 and the terminal device 200 cooperate to realize each functional unit and present a virtual try-on of an item as an online service. However, this is not limiting. For example, an app running on the terminal device 200 may independently realize each functional unit and present a virtual try-on of an item offline. In this case, the app downloads information from the catalog database 300 in advance. Furthermore, the identification unit 103 performs facial recognition using a library, such as an augmented reality (AR) library, an artificial intelligence (AI) library, or an image recognition library, that the terminal device 200 has, to identify the area in the application image where the second item is applied.

[0129] In the above embodiment, the server device 100 realizes the functions of the identification unit 103, second color acquisition unit 105, conversion calculation unit 106, and trial image acquisition unit 107, and the terminal device 200 realizes the functions of the selection reception unit 101, application image acquisition unit 102, first color acquisition unit 104, and output unit 108, but this is not limited to this. The functional units realized by the server device 100 and the terminal device 200 can be changed as appropriate.

[0130] 6 shows a flowchart of the trial process in the first embodiment, but the order of the processes is not limited to this. For example, the process of step S104 may be performed in any order as long as it is performed after the process of step S101 and before the process of step S107.

[0131] In the second embodiment, the second color acquisition unit 105 automatically identifies the second color associated with the second item specified by the user by referring to the catalog database 300, but the present invention is not limited to this. For example, the second color acquisition unit 105 may present the user with a list of color variations associated with the second item registered in the catalog database 300, and acquire a color selected from the list as the second color.

[0132] 9 shows a flowchart of the trial process in the second embodiment, the order of the processes is not limited to this. For example, the processes of steps S206 and S207 may be performed before the process of step S202.

[0133] In addition, by applying an operating program that defines the operation of the cosmetic trial system 1 according to the above embodiment to an existing personal computer or information terminal device, it is possible to make the personal computer or information terminal device function as the cosmetic trial system 1 according to the embodiment.

[0134] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]

[0135] According to the present invention, it is possible to provide a cosmetic trial system, trial method, and program that allows a user to virtually try on another item in a photographed environment in which the application of a certain item is photographed, without removing the applied item. [Explanation of symbols]

[0136] 1. Cosmetics trial system 10. Information processing equipment 11 CPU 12 ROM 13 RAM 14 Recording media 15 Output Devices 16. Communication Devices 17 Input Devices 18 Bus 100 Server device 101 Selection Reception Department 102 Application image acquisition unit 103 Specific section 104 1st color acquisition section 105 2nd color acquisition section 106 Conversion calculation unit 107 Trial image acquisition unit 108 Output section 109 Designated Reception Department 200 Terminal Device 300 Catalog Database 400 Electronic Market Server 500 Computer Network 600 apps 601 area 601-1 Application image 601-2 Trial image Columns 602 to 605 606 List 700 users 701 Lips

Claims

1. a selection receiving unit that receives a selection input from a user to select a first item to be tried; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a conversion calculation unit that calculates a conversion for converting the acquired second color into a color of a pixel in the specified region; a trial image acquisition unit that acquires a trial image by replacing the color of the pixel in the identified region in the acquired application image with a color obtained by applying the determined transformation to the acquired first color; an output unit that outputs the acquired trial image; A cosmetic trial system comprising:

2. The specifying unit specifies the area to which the first item is to be applied as the area to which the second item is applied in the application image.

2. The cosmetic product trial system according to claim 1.

3. the second color acquisition unit acquires candidate items that are candidates for the second item from the purchase history of the user, and acquires, from among colors of the candidate items, a color that is closest to a color of a pixel in the specified area as the second color; The candidate item having the closest color is designated as the second item.

3. The cosmetic product trial system according to claim 2.

4. a designation receiving unit that acquires candidate items that are candidates for the second item from the purchase history of the user, presents a list of the candidate items, and receives a designation input that designates any of the candidate items in the list; The candidate item related to the specified input is set as the second item.

2. The cosmetic product trial system according to claim 1.

5. The specifying unit specifies the area to which the second item is to be applied as the area where the second item is applied in the application image.

5. The cosmetic product trial system according to claim 4.

6. the first color acquisition unit acquires the first color associated with the first item by referring to a catalog database in which information on colors associated with items is registered; The second color acquisition unit refers to the catalog database and acquires a second color associated with the second item.

2. The cosmetic product trial system according to claim 1.

7. The catalog database contains items from different manufacturers, The first item and the second item are items from different manufacturers.

7. The cosmetic product trial system according to claim 6.

8. The trial image acquired by the application image acquisition unit is a still image, the conversion calculation unit determines the conversion by creating a filter that converts the acquired second color into a color of a pixel in the identified region on a pixel-by-pixel basis; The trial image acquisition unit converts the acquired first color on a pixel-by-pixel basis using the filter, and acquires the trial image by replacing the color of each pixel in the specified region in the acquired application image with the color converted on a pixel-by-pixel basis.

8. The cosmetic product trial system according to claim 1.

9. the test image acquired by the application image acquisition unit is a frame image included in a moving image, the conversion calculation unit determines a representative color for the specified region and generates a filter for converting the acquired second color into the representative color, thereby determining the conversion; the application image acquisition unit converts the acquired first color using the filter, and applies the converted color to the identified area in the acquired application image to acquire the trial image; The output unit outputs the acquired trial image as a frame image included in a moving image.

8. The cosmetic product trial system according to claim 1.

10. The conversion calculation unit determines a color represented by an average value, a median value, or a mode value of pixel values ​​in the specified area as the representative color.

10. The cosmetic product trial system according to claim 9.

11. The conversion calculation unit clusters pixel values ​​of the identified region and determines a color represented by a mean value, a median value, or a mode value of the largest cluster as the representative color.

10. The cosmetic product trial system according to claim 9.

12. A cosmetic trial method executed by a cosmetic trial system, The trial system comprises: accepting a selection input from the user selecting a first item to sample; acquiring an application image capturing a state of the user applying the second item; Identifying an area where the second item is applied in the acquired application image; obtaining a first color associated with the first item; obtaining a second color associated with the second item; determining a transformation that converts the obtained second color to a color of a pixel in the identified region; a trial image is obtained by replacing the color of the pixel in the identified region in the obtained application image with a color obtained by applying the obtained transformation to the obtained first color; outputting the acquired trial image; A cosmetic trial system characterized by:

13. Computer, a selection receiving unit that receives a selection input from a user to select a first item to be tried; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit that calculates a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing the color of the pixel in the identified region in the acquired application image with a color obtained by applying the determined transformation to the acquired first color; an output unit for outputting the acquired trial image; A program characterized by functioning as

Images