Cosmetics trial system, trial method and program

Abstract

To provide a cosmetic trial system, trial method, and program that enable confirmation of a state in which a certain item is virtually tried on another item in a photographing environment in which the state in which the item is applied is photographed, without removing the applied item. [Solution] A selection receiving unit 101 of a cosmetic trial system receives a selection input for selecting a first item to be tried by a user. An application image acquisition unit 102 acquires an application image captured of a user applying a second item. An identification unit 103 identifies an area in the acquired application image where the second item is applied. A transformation calculation unit 106 determines a transformation for converting a second color associated with the second item into a color of pixels in the identified area. A trial image acquisition unit 107 acquires a trial image by replacing the color of pixels in the identified area with a color obtained by applying the determined transformation to a first color associated with the first item. An output unit 108 outputs the trial image.

Description

[Technical field]

[0001] The present invention relates to a cosmetic product trial system, a trial method, and a program. [Background technology]

[0002] There is known a technique for virtually applying makeup to a face image of a user. For example, Patent Document 1 discloses a technique for generating a face image from which makeup has been removed from a face image of a user on which makeup has been applied, and applying a selected makeup style to the face image from which makeup has been removed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-36784 A Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned technology allows users to virtually try on various items without actually removing the makeup, but requires a process for generating a face image with the makeup removed. On the other hand, there is a demand for a simpler method for checking the appearance of virtually trying on various makeup items without requiring a process for generating a face image with the makeup removed.

[0005] The present invention is devised to solve the above-mentioned problems, and aims to provide a cosmetic trial system, trial method, and program that makes it possible to check the virtual trial of another item in the same shooting environment where the application of a certain item is photographed, without removing the applied item. [Means for solving the problem]

[0006] A cosmetic trial system according to a first aspect of the present invention comprises: a selection receiving unit that receives a selection input for selecting a first item to be tried by a user; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit that calculates a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing a color of a pixel in the specified region in the acquired application image with a color obtained by applying the obtained transformation to the acquired first color; an output unit that outputs the acquired trial image; The present invention is characterized by comprising:

[0007] In addition, in the cosmetic trial system according to the above aspect, The identification unit identifies an area to which the first item is to be applied as an area to which the second item is applied in the application image. It is characterized by:

[0008] In addition, in the cosmetic trial system according to the above aspect, the second color acquisition unit acquires candidate items that are candidates for the second item from the purchase history of the user, and acquires, from among colors of the candidate items, a color that is closest to a color of a pixel in the specified area as the second color; The candidate item having the closest color is designated as the second item. It is characterized by:

[0009] In addition, in the cosmetic trial system according to the above aspect, a designation receiving unit configured to obtain candidate items that are candidates for the second item from a purchase history of the user, present a list of the candidate items, and receive a designation input for designating any one of the candidate items in the list; The candidate item related to the specified input is set as the second item. It is characterized by:

[0010] In addition, in the cosmetic trial system according to the above aspect, The identification unit identifies an area to which the second item is to be applied as an area to which the second item is applied in the application image. It is characterized by:

[0011] In addition, in the cosmetic trial system according to the above aspect, the first color acquisition unit acquires a first color associated with the first item by referring to a catalog database in which information on colors associated with items is registered; The second color acquisition unit refers to the catalog database to acquire a second color associated with the second item. It is characterized by:

[0012] In addition, in the cosmetic trial system according to the above aspect, The catalog database includes items from different manufacturers; The first item and the second item are items from different manufacturers. It is characterized by:

[0013] In addition, in the cosmetic trial system according to the above aspect, The trial image acquired by the application image acquisition unit is a still image, the conversion calculation unit determines the conversion by creating a filter that converts the acquired second color into a color of a pixel of the identified region on a pixel-by-pixel basis; The trial image acquisition unit converts the acquired first color on a pixel-by-pixel basis using the filter, and acquires the trial image by replacing the color of each pixel of the specified area in the acquired application image with the color converted on a pixel-by-pixel basis. It is characterized by:

[0014] In addition, in the cosmetic trial system according to the above aspect, The trial image acquired by the application image acquisition unit is a frame image included in a moving image, the conversion calculation unit determines a representative color for the specified region, and generates a filter for converting the acquired second color into the representative color, thereby determining the conversion; the application image acquisition unit converts the acquired first color using the filter, and acquires the trial image by applying the converted color to the identified area in the acquired application image; The output unit outputs the acquired trial image as a frame image included in a moving image. It is characterized by:

[0015] In addition, in the cosmetic trial system according to the above aspect, The conversion calculation unit obtains a color represented by an average value, a median value, or a mode value of pixel values ​​in the specified area as the representative color. It is characterized by:

[0016] In addition, in the cosmetic trial system according to the above aspect, The conversion calculation unit performs clustering of pixel values ​​in the specified region, and obtains a color represented by a mean value, a median value, or a mode value of the largest cluster as the representative color. It is characterized by:

[0017] The trial method according to the second aspect of the present invention comprises: A cosmetic product trial method executed by a cosmetic product trial system, comprising: The trial system comprises: accepting a selection input from a user to select a first item to try; acquiring an application image capturing a state of the user applying the second item; Identifying an area where the second item is applied in the acquired application image; Obtaining a first color associated with the first item; obtaining a second color associated with the second item; determining a transformation that converts the obtained second color to a color of a pixel in the identified region; obtaining a trial image by replacing a color of a pixel in the identified region in the obtained application image with a color obtained by applying the obtained transformation to the obtained first color; outputting the acquired trial image; It is characterized by:

[0018] A program according to a third aspect of the present invention comprises: Computer, a selection receiving unit that receives a selection input for selecting a first item to be tried by a user; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit for calculating a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing a color of a pixel in the specified region in the acquired application image with a color obtained by applying the obtained transformation to the acquired first color; an output unit for outputting the acquired trial image; The present invention is characterized in that it functions as a

[0019] The above program may be recorded on a non-transitory recording medium. The non-transitory recording medium can be distributed and sold independently of the computer. Here, the non-transitory recording medium refers to a tangible recording medium. Examples of the non-transitory recording medium include a compact disc, a flexible disc, a hard disk, a magneto-optical disk, a digital video disk, a magnetic tape, and a semiconductor memory. In addition, the transitory recording medium refers to the transmission medium (propagation signal) itself. Examples of the transitory recording medium include an electric signal, an optical signal, and an electromagnetic wave. In addition, the temporary storage area is an area for temporarily storing data and programs, and is, for example, a volatile memory such as a RAM (Random Access Memory). Effect of the Invention

[0020] According to the present invention, it is possible to provide a cosmetic trial system, trial method, and program that make it possible to confirm the virtual trial use of another item in a shooting environment in which the application of a certain item is photographed, without removing the applied item. [Brief description of the drawings]

[0021] [Figure 1] 1 is a diagram showing the relationship between a cosmetic trial system according to a first embodiment, a catalog database, and an online market server. [Diagram 2] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Diagram 3] 1 is a diagram showing a functional configuration of a cosmetic product trial system according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an application image according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a trial image according to the first embodiment. [Figure 6] 2 is a diagram for explaining a trial process executed by the cosmetic product trial system according to the first embodiment. FIG. [Figure 7]FIG. 11 is a diagram showing the functional configuration of a cosmetic product trial system according to a second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a list according to the second embodiment. [Figure 9] 11 is a diagram for explaining a trial process executed by a cosmetic product trial system according to a second embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] (1. Overall Configuration) A cosmetics trial system 1 according to an embodiment of the present invention is a system that presents a user with a virtual trial of cosmetics. As shown in Fig. 1, the cosmetics trial system 1 includes a server device 100 and a terminal device 200, and the server device 100, the terminal device 200, the catalog database 300, and the electronic market server 400 are communicatively connected to each other via a computer communication network 500 such as the Internet.

[0023] The server device 100 is a device that executes various processes for presenting a virtual try-on of makeup. The server device 100 provides various information in response to a request from the terminal device 200.

[0024] The terminal device 200 is a terminal that captures an image of a user trying on makeup and presents the user's virtual try-on of the makeup. For example, the terminal device 200 is a smartphone, a tablet computer, or the like. For example, an application for virtually trying on makeup (hereinafter simply referred to as an "application") is installed in the terminal device 200. The terminal device 200 is operated by the user himself or herself, or, when used in a store, is operated by a store staff member.

[0025] Catalog database 300 is a database in which information about items is registered. An item is a cosmetic product that a user applies to the face or body. For example, items include lipstick, blush, eye shadow, hair color, nail color, etc. In catalog database 300, color information associated with an item is registered as information about the item. Furthermore, the items whose information is registered in catalog database 300 are items handled in the online market managed by online market server 400. Therefore, items from different manufacturers are registered in catalog database 300. Note that color information associated with an item may be registered in a database other than catalog database 300.

[0026] The online market server 400 is a server that manages the online market. In response to a request from the cosmetic product trial system 1, the online market server 400 provides the cosmetic product trial system 1 with information on the user's purchase history.

[0027] (2. Hardware configuration of information processing device) FIG. 2 shows an example of a hardware configuration of an information processing device 10 in which the server device 100 and the terminal device 200 are realized.

[0028] 2, the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a recording medium 14, an output device 15, a communication device 16, and an input device 17. The components are connected to each other via a bus 18.

[0029] The CPU 11 controls the overall operation of the information processing device 10, and is connected to each of the components to exchange control signals and data.

[0030] In the ROM 12, operating programs and various data necessary for controlling the overall operation of the information processing device 10 are recorded.

[0031] The RAM 13 is for temporarily storing data and programs, and holds programs and data read from the recording medium 14, as well as other data necessary for communication.

[0032] The recording medium 14 is composed of a hard disk, a flash memory, or the like, and records data to be processed by the information processing device 10.

[0033] The output device 15 includes a display device such as an LCD (Liquid Crystal Display) and an audio output device such as a speaker. The output device 15 outputs, for example, data output from the CPU 11 under the control of the CPU 11.

[0034] The communication device 16 includes a communication interface for connecting the information processing device 10 to a computer communication network such as the Internet, and communicates with other information processing devices and the like via the communication device 16.

[0035] The input device 17 includes input devices such as a button, a keyboard, a touch panel, a camera, a microphone, etc. The input device 17 receives an operation input from a user of the information processing device 10, and outputs a signal corresponding to the received operation input to the CPU 11.

[0036] (3. Functional configuration of the cosmetic trial system of embodiment 1) The functional configuration of the cosmetic product trial system 1 of the first embodiment will be described with reference to FIG.

[0037] The cosmetic product trial system 1 of the first embodiment functionally comprises a selection receiving unit 101, an application image acquisition unit 102, an identification unit 103, a first color acquisition unit 104, a second color acquisition unit 105, a conversion calculation unit 106, a trial image acquisition unit 107, and an output unit 108. These functional units may be realized by the server device 100 and the terminal device 200 working together, or may be realized by the server device 100 or the terminal device 200 alone, or may be realized by the server device 100 alone.

[0038] In the following, an example will be described in which the above functional units are realized by the server device 100 and the terminal device 200 in cooperation with each other, and a virtual trial of an item is presented as an online service. Specifically, the server device 100 realizes the functions of the identification unit 103, the second color acquisition unit 105, the conversion calculation unit 106, and the trial image acquisition unit 107, and the terminal device 200 realizes the functions of the selection reception unit 101, the application image acquisition unit 102, the first color acquisition unit 104, and the output unit 108.

[0039] 4 shows an example of an application that runs on the terminal device 200. The application 600 includes an area 601 in which an application image or a trial image, which will be described later, a field 602 for displaying a first item selected by the user, a field 603 for displaying the color of the first item, a field 604 for displaying a second item applied by the user, and a field 605 for displaying the color of the second item.

[0040] The first item is an item that is virtually applied to the user, and the second item is an item that is applied to the user and to the same area as the first item. The app 600 shows the virtual application of the first item to the user's area where the second item is applied.

[0041] Here, the first item and the second item may be items from different manufacturers as long as the color information of the items is registered in the catalog database 300.

[0042] For example, a user can use app 600 to see what a virtual try on of "Maker Y" lipstick would look like on an image of themselves wearing "Maker X" lipstick.

[0043] The selection receiving unit 101 receives a selection input for selecting a first item to be tried by a user. The selection receiving unit 101 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0044] For example, when the user 700 selects, by tapping or the like, the field 602 in which the first item is displayed in the app 600, the selection receiving unit 101 displays a list of items registered in the catalog database 300. Then, when the user 700 selects "Lipstick 1" from the list, the selection receiving unit 101 accepts the selection input of "Lipstick 1" and transmits information on "Lipstick 1" as the first item to the server device 100. Also, the selection receiving unit 101 displays "Lipstick 1" in the field 602 in which the first item of the app 600 is displayed.

[0045] The application image acquisition unit 102 acquires an application image in which a user applies the second item. The application image acquisition unit 102 is realized by the CPU 11, the communication device 16, and the input device 17.

[0046] The application image is an image taken of a user to which the second item has been applied.

[0047] Examples of application images are shown in area 601 of Fig. 4. Application image 601-1 shows a state in which lip balm, which is a second item, has been applied to lips 701 of user 700.

[0048] For example, when a user 700 uses the camera of the terminal device 200 to display and capture an application image 601-1 in the area 601 as shown in FIG. 4, the application image acquisition unit 102 acquires the application image 601-1 and transmits information about the application image 601-1 to the server device 100.

[0049] Here, the application image acquired by the application image acquisition unit 102 is a still image or a frame image included in a moving image. The application image acquisition unit 102 may acquire the still image or the moving image while the application is running, or may acquire the image from a library acquired in the past.

[0050] For example, the application image acquisition unit 102 acquires a live video of a user taking a selfie of the user applying an item at home. Alternatively, the application image acquisition unit 102 acquires a live video of the user applying an item at a store. Alternatively, the application image acquisition unit 102 acquires a still image of the user applying an item from the library of the terminal device 200.

[0051] The identification unit 103 identifies an area where the second item is applied in the acquired application image. The identification unit 103 is realized by the CPU 11.

[0052] Specifically, the identifying unit 103 identifies the area to which the first item should be applied as the area to which the second item is applied in the application image.

[0053] For example, the identification unit 103 performs face recognition on the application image transmitted from the terminal device 200, and identifies the area in the application image where the second item is applied. For example, when the first item is lipstick, the area to be applied is the lips, so the identification unit 103 identifies the area of ​​the lips 701 in the application image 601-1 as the area to be applied with the second item. Alternatively, when the first item is eyeshadow, the area to be applied is the eyelids, so the identification unit 103 identifies the area of ​​the eyelids in the application image 601-1 as the area to be applied with the second item.

[0054] The first color acquisition unit 104 acquires a first color associated with the first item. The first color acquisition unit 104 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0055] The first color is the color of the first item that the user virtually tries on.

[0056] Specifically, the first color acquiring unit 104 refers to the catalog database 300 to acquire the first color associated with the first item.

[0057] For example, when the user 700 selects, by tapping or the like, the field 603 in which the first color is displayed in the app 600, the first color acquisition unit 104 displays a list of color variations of "Lip 1" registered in the catalog database 300. For example, the list includes "Color 1-1" to "Color 1-7." When the user 700 selects "Color 1-3" from the list, the first color acquisition unit 104 accepts the selection input of "Color 1-3" and transmits information of "Color 1-3" as the first color to the server device 100. In addition, the first color acquisition unit 104 displays "Color 1-3" in the field 603 in which the first color of the app 600 is displayed.

[0058] The second color acquisition unit 105 acquires a second color associated with the second item. The second color acquisition unit 105 is realized by the CPU 11 and the communication device 16.

[0059] The second color is the color of a second item that the user applies to themselves.

[0060] Specifically, the second color acquisition unit 105 acquires candidate items that are candidates for the second item from the user's purchase history, and acquires, from among the colors of the candidate items, a color that is closest to the color of the pixels in the specified area as the second color. Here, the second color acquisition unit 105 refers to the catalog database 300 to acquire the second color associated with the second item.

[0061] For example, the second color acquisition unit 105 acquires the purchase history of the user 700 from the electronic market server 400. The second color acquisition unit 105 acquires, as candidate items, items in the same category as the first item from among the items included in the acquired purchase history of the user 700. If the second color acquisition unit 105 acquires, as candidate items, items "Lipstick 2", "Lipstick 3", and "Lipstick 4" that are in the same category as "Lipstick 1", then the second color acquisition unit 105 refers to the catalog database 300 to acquire colors associated with each of the multiple candidate items. If the second color acquisition unit 105 acquires "Color 2-1" through "Color 2-5" associated with "Lip 2," "Color 3-1" through "Color 3-4" associated with "Lip 3," and "Color 4-1" through "Color 4-6" associated with "Lip 4," it identifies the color among "Color 2-1" through "Color 2-5," "Color 3-1" through "Color 3-4," and "Color 4-1" through "Color 4-6" that is closest to the color of the pixel of lips 701 in application image 601-1 as the second color.

[0062] For example, if the color closest to the color of the pixel of lips 701 in application image 601-1 is "color 2-1," second color acquisition section 105 identifies "color 2-1" as the second color.

[0063] Then, the server device 100 determines the candidate item having the closest color as the second item.

[0064] For example, the server device 100 sets "lips 2" associated with the second color "color 2-1" as the second item. In this case, as shown in Fig. 4, "lips 2" is displayed in the field 604 in which the second items of the app 600 are displayed, and "color 2-1" is displayed in the field 605 in which the second colors are displayed.

[0065] The user can modify the automatically set second item and second color. When field 604 is selected by tapping or the like, it displays a list of items that are registered in catalog database 300 and are in the same category as the first item. The user can select the second item from the list. When field 605 is selected by tapping or the like, it displays a list of colors that are registered in catalog database 300 for the item displayed in field 604. The user can select a second color from the list.

[0066] The conversion calculation unit 106 obtains a conversion for converting the acquired second color into the color of the pixel in the specified region. The conversion calculation unit 106 is realized by the CPU 11.

[0067] For example, transformation calculation unit 106 determines a transformation that transforms the second color "Color 2-1" of the second item "Lip 2" into the color of the pixels in the region of lips 701 identified in application image 601-1.

[0068] Specifically, when the application image acquired by the application image acquisition unit 102 is a still image, the conversion calculation unit 106 calculates the conversion by creating a filter that converts the acquired second color into the color of the pixels in the specified area on a pixel-by-pixel basis.

[0069] For example, if the application image 601-1 is a still image, the conversion calculation unit 106 creates a filter that converts the second color "color 2-1" into the color i of pixel i (i is 1 or more and N or less, N is the number of pixels included in the region) in the lips 701 area, i.e., a filter that multiplies the R value, G value, and B value by k_ri, k_gi, and k_bi.

[0070] Alternatively, when the application image acquired by the application image acquisition unit 102 is a frame image included in a moving image, the conversion calculation unit 106 determines a representative color for the identified area and calculates the conversion by creating a filter that converts the acquired second color into the representative color.

[0071] For example, when application image 601-1 is a frame image included in a moving image, conversion calculation section 106 first obtains the representative color of the lips 701 area.

[0072] Specifically, the conversion calculation unit 106 determines a color represented by the average, median, or mode of pixel values ​​in the identified region as the representative color.

[0073] For example, the conversion calculation unit 106 calculates the average value, median value, or mode value of each of the R value, G value, and B value for pixel i in the region of the lips 701. The conversion calculation unit 106 sets a color that combines the average value, median value, or mode value of the calculated R value, G value, and B value as the representative color of the identified region. Alternatively, the conversion calculation unit 106 sets the mode value of the combination of the R value, G value, and B value of pixel i as the representative color.

[0074] Alternatively, the conversion calculation unit 106 clusters the pixel values ​​in the identified region, and determines the color represented by the average value, median value, or mode value of the largest cluster as the representative color.

[0075] For example, the conversion calculation unit 106 clusters three-dimensional vectors represented by R, G, and B values ​​for pixel i in the region of the lips 701 using the k-means method, and determines a color represented by the average, median, or mode of the R, G, and B values ​​of the three-dimensional vectors included in the largest cluster as a representative color, or determines a color represented by the R, G, and B values ​​of the center of gravity of the largest cluster as a representative color.

[0076] The conversion calculation unit 106 creates a filter that converts the second color "color 2-1" into the representative color of the area of ​​the lips 701, that is, a filter that multiplies the R value, G value, and B value by k_r, k_g, and k_b, respectively.

[0077] The trial image acquisition unit 107 acquires a trial image by replacing the color of the pixel in the specified region in the acquired application image with a color obtained by applying the obtained transformation to the acquired first color. The trial image acquisition unit 107 is realized by the CPU 11 and the communication device 16.

[0078] Specifically, when the application image acquired by the application image acquisition unit 102 is a still image, the trial image acquisition unit 107 converts the acquired first color on a pixel-by-pixel basis using a filter, and acquires a trial image by replacing the color of each pixel in a specified area in the acquired application image with the color converted on a pixel-by-pixel basis.

[0079] For example, when the application image 601-1 is a still image, the trial image acquisition unit 107 obtains a converted color i for pixel i using a filter that multiplies the R value, B value, and G value of the first color "color 1-3" by k_ri, k_gi, and k_bi. Then, the trial image acquisition unit 107 generates a trial image 601-2 in which the color i of pixel i in the region of the lips 701 in the application image 601-1 is replaced with the converted color i, and transmits information about the trial image 601-2 to the terminal device 200.

[0080] When the application image acquired by the application image acquisition unit 102 is a frame image included in a moving image, the trial image acquisition unit 107 converts the acquired first color using a filter and acquires a trial image by applying the converted color to a specified area in the acquired application image.

[0081] For example, when the application image 601-1 is a frame image included in a moving image, the trial image acquisition unit 107 obtains a converted color using a filter that multiplies the R value, B value, and G value of the first color "color 1-3" by k_r, k_g, and k_b. Then, the trial image acquisition unit 107 generates a trial image 601-2 in which the color i of pixel i in the area of ​​the lips 701 in the application image 601-1 is replaced with the converted color, and transmits information about the trial image 601-2 to the terminal device 200.

[0082] The output unit 108 outputs the acquired trial image. The output unit 108 and the selection receiving unit 101 are realized by the CPU 11, the output device 15, and the communication device 16.

[0083] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the application 600 as shown in FIG. 5. For example, when the application image 601-1 is a still image showing the user 700 applying "lip 2", the trial image 601-2 is an image showing the state where "color 1-3" of "lip 1" is virtually applied to the lips 701 of the user 700 in the application image 601-1. For example, when the user selects an image showing the state where the user has applied "lip 2" from the library and selects "lip 1" of "color 1-3", an image is displayed in the area 601 in which "lip 1" of "color 1-3" is applied to the area where "lip 2" has been applied in the image of the library.

[0084] Furthermore, when the application image acquired by the application image acquisition section 102 is a frame image included in a moving image, the output section 108 outputs the acquired trial image as a frame image included in the moving image.

[0085] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the application 600 as a frame image of a moving image. When the application image 601-1 is a frame image of a live moving image showing the user 700 applying "Lip 2", the trial image 601-2 is a frame image showing the user 700's lips 701 in the application image 601-1 virtually applying "Color 1-3" of "Lip 1". For example, when the user takes a selfie video of the user applying "Lip 2" and selects "Lip 1" in "Color 1-3", a moving image is displayed in the area 601 in real time, in which "Lip 1" in "Color 1-3" is applied to the area where "Lip 2" was applied.

[0086] (4. Operation of the Cosmetic Trial System of the First Embodiment) The operation of the cosmetic product trial system 1 of this embodiment will be described with reference to Fig. 6. For example, when the application 600 is launched on the terminal device 200, the cosmetic product trial system 1 starts the trial process shown in Fig. 6.

[0087] The selection receiving unit 101 receives a selection input by the user to select a first item to be tried (step S101).

[0088] For example, when a user 700 selects, by tapping or otherwise, a field 602 in which a first item is displayed in the app 600, the selection receiving unit 101 displays a list of items registered in the catalog database 300, and when the user 700 selects “Lip 1” from the list, the selection receiving unit 101 accepts the selection input of “Lip 1” and transmits information about “Lip 1” as the first item to the server device 100.

[0089] The application image acquisition section 102 acquires an application image capturing a picture of the user applying the second item (step S102).

[0090] For example, when a user 700 uses the camera of the terminal device 200 to display and capture an application image 601-1 in the area 601 as shown in FIG. 4, the application image acquisition unit 102 acquires the application image 601-1 and transmits information about the application image 601-1 to the server device 100.

[0091] The identification unit 103 identifies an area in the acquired application image where the first item should be applied as an area where the second item is applied (step S103).

[0092] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area where the first item “Lip 1” should be applied, as the area where the second item is applied.

[0093] The first color acquisition unit 104 refers to the catalog database 300 to acquire a first color associated with the first item (Step S104).

[0094] For example, when the user 700 selects, by tapping or otherwise, the field 603 in which the first color is displayed in the app 600, the first color acquisition unit 104 displays a list of color variations of “Lip 1” registered in the catalog database 300, and when the user 700 selects “Color 1-3” from the list, the first color acquisition unit 104 accepts the selection input of “Color 1-3” and transmits information of “Color 1-3” as the first color to the server device 100.

[0095] The second color acquisition unit 105 refers to the catalog database 300 to acquire a second color associated with the second item (Step S105).

[0096] For example, the second color acquisition unit 105 acquires candidate items "Lip 2", "Lip 3", and "Lip 4" that are candidates for the second item based on the purchase history of the user 700 acquired from the electronic market server 400, and acquires candidate items "Color 2-1" to "Color 2-5", "Color 3-1" to "Color 3-4", and "Color 4-1" to "Color 4-6" registered in the catalog database 300.The second color acquisition unit 105 then automatically identifies, as the second color, the color "Color 2-1", which is closest to the color of the pixel of the lips 701 in the application image 601-1, from among "Color 2-1" to "Color 2-5", "Color 3-1" to "Color 3-4", and "Color 4-1" to "Color 4-6".

[0097] The transformation calculation unit 106 calculates a transformation for transforming the acquired second color into the color of the pixel in the identified region (step S106).

[0098] For example, the transformation calculation unit 106 determines a transformation that transforms the second color "color 2-1" into the color of the pixels in the region of the lips 701 identified in the application image 601-1.

[0099] The trial image acquisition unit 107 acquires a trial image by replacing the color of the pixel in the identified area in the acquired application image with a color obtained by applying the determined transformation to the acquired first color (step S107).

[0100] For example, the trial image acquisition unit 107 uses a filter that multiplies the R value, B value, and G value of the first color “Color 1-3” by k_ri, k_gi, and k_bi to obtain a converted color i for pixel i, generates a trial image 601-2 in which the color i of pixel i in the area of ​​the lips 701 in the application image 601-1 is replaced with the converted color i, and transmits information about the trial image 601-2 to the terminal device 200.

[0101] The output unit 108 outputs the acquired trial image (step S108).

[0102] For example, the output unit 108 displays the trial image 601-2 transmitted by the server device 100 in the area 601 of the application 600 as shown in FIG.

[0103] According to this embodiment, in a photographing environment where the second item is applied, it is possible to confirm the appearance of a virtual trial of the first item without wiping off the part where the second item is applied. This makes it possible to easily compare the color of an item that is actually applied with the color of an item that is not actually applied in the same photographing environment.

[0104] Furthermore, according to the present embodiment, the items and colors applied by the user can be identified based on the user's purchase history, which can save the user the trouble of searching for items that the user has applied currently or in the past.

[0105] Furthermore, according to this embodiment, it is possible to virtually try items from different manufacturers that are registered in the catalog database. This allows the color tones of items from different manufacturers that are applied to the same area to be compared. Furthermore, since the items registered in the catalog database are items that are available in the electronic market, the user can virtually try items from various manufacturers and consider purchasing them.

[0106] Furthermore, according to this embodiment, it is possible to compare the color of an item in a still image or video captured in the past, or in a video captured live, with the color of an item that has not actually been painted. This allows the user to compare the color of an item that has actually been painted in various shooting environments with the color of an item that has not actually been painted.

[0107] In addition, according to this embodiment, by applying a representative color to a moving image such as a live video, it is possible to check the virtual try-on of an item in real time. Furthermore, according to this embodiment, by obtaining a representative color through clustering, it is possible to determine a representative color excluding a highlight part in an area where a glossy lipstick is applied, and to obtain an appropriate color as the representative color.

[0108] (5. Functional configuration of the cosmetic trial system of the second embodiment) The functional configuration of the cosmetic product trial system 1 of the second embodiment will be described with reference to FIG.

[0109] The cosmetic product trial system 1 of the second embodiment functionally comprises a selection receiving unit 101, an application image acquiring unit 102, a determination unit 103, a first color acquiring unit 104, a second color acquiring unit 105, a conversion calculation unit 106, a trial image acquiring unit 107, an output unit 108, and a designation accepting unit 109. In the following, a case will be described in which the above functional units are realized by the server device 100 and the terminal device 200 working together to present a virtual trial of an item as an online service. Specifically, the server device 100 realizes the functions of the determination unit 103, the second color acquiring unit 105, the conversion calculation unit 106, and the trial image acquiring unit 107, and the terminal device 200 realizes the functions of the selection accepting unit 101, the application image acquiring unit 102, the first color acquiring unit 104, the output unit 108, and the designation accepting unit 109. In the following, functions that are different from the functions of the trial system 1 of the first embodiment will be described.

[0110] The specification receiving unit 109 obtains candidate items that are candidates for the second item from the user's purchase history, presents a list of the candidate items, and receives a specification input for specifying any of the candidate items in the list. The specification receiving unit 109 is realized by the CPU 11, the output device 15, the communication device 16, and the input device 17.

[0111] For example, when the user 700 selects, by tapping or the like, a field 604 in which the second item is displayed in the app 600, the designation receiving unit 109 acquires the purchase history of the user 700 from the online market server 400. Next, the designation receiving unit 109 acquires items included in the acquired purchase history of the user 700 as candidate items. The designation receiving unit 109 displays a list 606 of candidate items as shown in FIG. 8. Then, when the user 700 selects “Lipstick 2” from the list as the second item, the designation receiving unit 109 accepts the designation input of “Lipstick 2” and accepts the information of “Lipstick 2” in the server device 100.

[0112] The server device 100 determines the candidate item related to the specified input as the second item.

[0113] For example, the server device 100 sets "Lipstick 2" as the second item. Furthermore, the designation receiving unit 109 displays "Lipstick 2" in the field 604 in which the second item of the application 600 is displayed.

[0114] The identifying unit 103 identifies the area to which the second item is to be applied as the area to which the second item is applied in the application image.

[0115] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area to which the second item “Lip 2” should be applied, as the area to which the second item is applied.

[0116] The second color acquisition unit 105 refers to the catalog database 300 to acquire a second color associated with the second item designated by the user.

[0117] For example, the second color acquisition unit 105 refers to the catalog database 300 to acquire "Color 2-1" to "Color 2-5" associated with the second item "Lip 2." Then, the second color acquisition unit 105 specifies, as the second color, the color that is closest to the color of the pixel of lips 701 in application image 601-1 among "Color 2-1" to "Color 2-5."

[0118] (6. Operation of the Cosmetic Trial System of the Second Embodiment) The operation of the cosmetics trial system 1 of this embodiment will be described with reference to Fig. 9. For example, when the application 600 is launched on the terminal device 200, the cosmetics trial system 1 starts the trial process shown in Fig. 9. Note that the processes of steps S201, and S205 to S209 are the same as the processes of steps S102, S106, S101, S104, S107, and S108 in the flowchart of Fig. 6.

[0119] In step S201, when the application image acquisition unit 102 acquires an application image, the designation receiving unit 109 acquires candidate items that are candidates for the second item from the user's purchase history, presents a list of the candidate items, and accepts a designation input to designate any of the candidate items in the list (step S202).

[0120] For example, the designation receiving unit 109 displays a list 606 of candidate items in the application 600 of the terminal device 200 as shown in FIG. 8, and receives information of a designation input of “Lip 2” from the application 600 of the terminal device 200.

[0121] The identifying unit 103 identifies the area to which the second item is to be applied as the area to which the second item is applied in the application image (step S203).

[0122] For example, the identification unit 103 performs facial recognition on the application image 601-1 transmitted from the terminal device 200, and identifies the area of ​​the lips 701, which is the area where the second item “Lip 2” should be applied, as the area where the second item “Lip 2” is applied.

[0123] The second color acquiring unit 105 acquires a second color associated with the second item (Step S204).

[0124] For example, when the second color acquisition unit 105 refers to the catalog database 300 and acquires "Color 2-1" to "Color 2-5" associated with "Lip 2," it automatically identifies, among "Color 2-1" to "Color 2-5," the color "Color 2-1" that is closest to the color of the pixel of lips 701 in application image 601-1, as the second color.

[0125] When the second color acquisition unit 105 acquires the second color in step S204, the processes in and after step S205 are executed.

[0126] According to the present embodiment, a list of candidate items applied by the user is presented based on the user's purchase history, thereby eliminating the need for the user to search for items that have been applied currently or in the past. Also, by having the user select from a list of items generated based on the user's purchase history, the user can accurately select the items to be applied.

[0127] (7. Modifications) Although the embodiment of the present invention has been described above, various modifications and applications are possible in carrying out the present invention.

[0128] In the above embodiment, the server device 100 and the terminal device 200 cooperate to realize each functional unit and present a state in which an item is virtually tried as an online service, but the present invention is not limited to this. For example, an application running on the terminal device 200 may independently realize each functional unit and present a state in which an item is virtually tried offline. In this case, the application downloads information of the catalog database 300 in advance. In addition, the identification unit 103 performs face recognition using a library such as an AR (Augmented Reality) library, an AI (Artificial Intelligence) library, or an image recognition library that the terminal device 200 has, and identifies an area in the application image where the second item is applied.

[0129] In the above embodiment, the server device 100 realizes the functions of the identification unit 103, the second color acquisition unit 105, the conversion calculation unit 106, and the trial image acquisition unit 107, and the terminal device 200 realizes the functions of the selection reception unit 101, the application image acquisition unit 102, the first color acquisition unit 104, and the output unit 108, but this is not limited to the above. The functional units realized by the server device 100 and the terminal device 200 can be changed as appropriate.

[0130] In the above-mentioned first embodiment, the order of the trial process is not limited to the flowchart shown in Fig. 6. For example, the process of step S104 may be performed in any order as long as it is performed after the process of step S101 and before the process of step S107.

[0131] In the above-mentioned second embodiment, the second color acquisition unit 105 refers to the catalog database 300 and automatically identifies the second color associated with the second item specified by the user, but the present invention is not limited to this. For example, the second color acquisition unit 105 may present the user with a list of color variations associated with the second item registered in the catalog database 300, and acquire a color selected from the list as the second color.

[0132] In the second embodiment, the order of the trial process is not limited to the flowchart shown in Fig. 9. For example, the processes in steps S206 and S207 may be performed before the process in step S202.

[0133] In addition, by applying an operating program that specifies the operation of the cosmetic trial system 1 according to the above embodiment to an existing personal computer or information terminal device, it is also possible to cause the personal computer or information terminal device to function as the cosmetic trial system 1 according to the embodiment.

[0134] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]

[0135] According to the present invention, it is possible to provide a cosmetic trial system, trial method, and program that make it possible to confirm the virtual trial use of another item in a shooting environment in which the application of a certain item is photographed, without removing the applied item. [Explanation of symbols]

[0136] 1. Cosmetics trial system 10. Information processing device 11 CPU 12 ROM 13 RAM 14 Recording media 15 Output Devices 16 Communication Devices 17 Input Devices 18 Bus 100 Server device 101 Selection Reception Department 102 Coating image acquisition unit 103 Specific part 104 1st color acquisition section 105 2nd color acquisition section 106 Conversion calculation unit 107 Trial image acquisition unit 108 Output section 109 Designated Reception Department 200 Terminal Equipment 300 Catalog Database 400 Electronic Market Server 500 Computer Network 600 Apps 601 area 601-1 Coating image 601-2 Trial Image Columns 602 to 605 606 List 700 users 701 Lips

Claims

1. a selection receiving unit that receives a selection input for selecting a first item to be tried by a user; an application image acquisition unit that acquires an application image capturing a state in which the user applies the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit that calculates a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing a color of a pixel in the specified region in the acquired application image with a color obtained by applying the obtained transformation to the acquired first color; an output unit that outputs the acquired trial image; A cosmetic trial system comprising:

2. The identification unit identifies an area to which the first item is to be applied as an area to which the second item is applied in the application image.

2. The cosmetic product trial system according to claim 1.

3. the second color acquisition unit acquires candidate items that are candidates for the second item from a purchase history of the user, and acquires, from among colors of the candidate items, a color that is closest to a color of a pixel in the specified area as the second color; The candidate item having the closest color is designated as the second item.

3. The cosmetic product trial system according to claim 2.

4. a designation receiving unit configured to obtain candidate items that are candidates for the second item from a purchase history of the user, present a list of the candidate items, and receive a designation input for designating any one of the candidate items in the list; The candidate item related to the specified input is set as the second item.

2. The cosmetic product trial system according to claim 1.

5. The identification unit identifies an area to which the second item is to be applied as an area to which the second item is applied in the application image.

5. The cosmetic product trial system according to claim 4.

6. the first color acquisition unit acquires a first color associated with the first item by referring to a catalog database in which information on colors associated with items is registered; The second color acquisition unit refers to the catalog database to acquire a second color associated with the second item.

2. The cosmetic product trial system according to claim 1.

7. The catalog database includes items from different manufacturers; The first item and the second item are items from different manufacturers.

7. The cosmetic product trial system according to claim 6.

8. The trial image acquired by the application image acquisition unit is a still image, the conversion calculation unit determines the conversion by creating a filter that converts the acquired second color into a color of a pixel of the identified region on a pixel-by-pixel basis; The trial image acquisition unit converts the acquired first color on a pixel-by-pixel basis using the filter, and acquires the trial image by replacing the color of each pixel of the specified area in the acquired application image with the color converted on a pixel-by-pixel basis.

8. The cosmetic product trial system according to claim 1,

9. The trial image acquired by the application image acquisition unit is a frame image included in a moving image, the conversion calculation unit determines a representative color for the specified region, and generates a filter for converting the acquired second color into the representative color, thereby determining the conversion; the application image acquisition unit converts the acquired first color using the filter, and acquires the trial image by applying the converted color to the identified area in the acquired application image; The output unit outputs the acquired trial image as a frame image included in a moving image.

8. The cosmetic product trial system according to claim 1,

10. The conversion calculation unit obtains a color represented by an average value, a median value, or a mode value of pixel values ​​in the specified area as the representative color.

10. The cosmetic product trial system according to claim 9.

11. The conversion calculation unit performs clustering of pixel values ​​in the specified region, and determines a color represented by a mean value, a median value, or a mode value of the largest cluster as the representative color.

10. The cosmetic product trial system according to claim 9.

12. A cosmetic product trial method executed by a cosmetic product trial system, comprising: The trial system comprises: accepting a selection input from a user selecting a first item to sample; acquiring an application image capturing a state of the user applying the second item; Identifying an area where the second item is applied in the acquired application image; Obtaining a first color associated with the first item; Obtaining a second color associated with the second item; determining a transformation that converts the obtained second color into a color of a pixel in the identified region; obtaining a trial image by replacing a color of a pixel in the identified region in the obtained application image with a color obtained by applying the obtained transformation to the obtained first color; outputting the acquired trial image; A method for trying out cosmetics, comprising:

13. Computer, a selection receiving unit that receives a selection input for selecting a first item to be tried by a user; an application image acquisition unit that acquires an application image capturing a state of the user applying the second item; an identification unit that identifies an area where the second item is applied in the acquired application image; a first color acquisition unit that acquires a first color associated with the first item; a second color acquisition unit that acquires a second color associated with the second item; a transformation calculation unit for calculating a transformation for transforming the acquired second color into a color of a pixel in the identified region; a trial image acquisition unit that acquires a trial image by replacing a color of a pixel in the specified region in the acquired application image with a color obtained by applying the obtained transformation to the acquired first color; an output unit for outputting the acquired trial image; A program characterized by causing the program to function as a

Images