Eye Makeup Product Recommendation Program, Method, Device, and System

Abstract

The present invention provides a storage medium storing a program for recommending eye makeup products suitable for a user. The program causes an information processing apparatus to function as the following respective parts: an image acquisition part that acquires an image of a periorbital part including the user's eyes; a feature extraction part that extracts features from the periorbital part including the eyes; a product recommendation part that recommends products for eye makeup according to the features; and a display part that displays the recommended products.

Description

Technical Field

[0001] The present invention relates to an eye makeup product recommendation program, method, apparatus, and system. Background Art

[0002] Conventionally, beauty professionals such as makeup artists or beauty consultants select cosmetics suitable for makeup requesters based on their long - accumulated experience (for example, an optical illusion effect such as a certain makeup method making the eyes appear larger. Refer to Non - Patent Document 1).

[0003] [[Prior Art Documents]]

[0004] [[Non - Patent Documents]]

[0005] Non - Patent Document 1: FRAGRANCE JOURNAL, March 2013 issue, pages 55 - 61 Summary of the Invention

[0006] [[Problems to be Solved by the Invention]]

[0007] However, not all users who want to buy cosmetics necessarily have the opportunity to get help from beauty professionals to select cosmetics. Sometimes, users can only select cosmetics by themselves (for example, when users buy cosmetics through an EC website, etc.). And eye makeup products have various colors and functions, making it particularly difficult for users to select suitable products by themselves.

[0008] In view of this, an object of one embodiment of the present invention is to recommend suitable eye makeup products to users.

[0009] [[Means for Solving the Above - Mentioned Problems]]

[0010] One aspect of the present invention is a storage medium storing a program that causes an information processing apparatus to function as the following components: an image acquisition unit that acquires an image of a periorbital part including a user's eyes; a feature extraction unit that extracts features from the periorbital part including the eyes; a product recommendation unit that recommends eye makeup products based on the features; and a display unit that displays the recommended products.

[0011] [[Effects of the Invention]]

[0012] According to the present invention, it is possible to recommend suitable eye makeup products for users. Brief Description of the Drawings

[0013] Figure 1 is an example of an overall structure diagram of one embodiment of the present invention (Embodiment 1).

[0014] Figure 2 is an example of a functional block diagram of a user terminal of one embodiment of the present invention (Embodiment 1).

[0015] Figure 3 This is a diagram for feature extraction (eyeshadow, eyes open) to illustrate an embodiment of the present invention.

[0016] Figure 4 This is a diagram for feature extraction (eyeshadow, eyes closed) to illustrate an embodiment of the present invention.

[0017] Figure 5 This is a diagram for feature extraction (mascara and eyeliner) to illustrate an embodiment of the present invention.

[0018] Figure 6 This is a flowchart showing an example of product recommendation processing according to an embodiment of the present invention.

[0019] Figure 7 This is a flowchart showing an example of feature extraction processing (eyeshadow) according to an embodiment of the present invention.

[0020] Figure 8 This is a flowchart showing an example of feature extraction processing (mascara and eyeliner) according to an embodiment of the present invention.

[0021] Figure 9 This is an example of the overall structure diagram of an embodiment of the present invention (Embodiment 2).

[0022] Figure 10 An example of the functional block diagram of the eye makeup product recommendation system according to an embodiment of the present invention (Embodiment 2).

[0023] Figure 11 This is a block diagram showing an example of the hardware structure of the user terminal and the server according to an embodiment of the present invention. Detailed Embodiment

[0024] Hereinafter, embodiments of the present invention will be described with reference to the drawings. It is assumed that eye makeup products (hereinafter also referred to as eye makeup products) suitable for user 11 are recommended to user 11. The cases where user 11 receives product recommendations when the user terminal 10 is not connected to the server 20 (Embodiment 1) and when the user terminal 10 is connected to the server 20 (Embodiment 2) will be described separately.

[0025] <Embodiment 1>

[0026] Hereinafter, Embodiment 1 will be described.

[0027] <Overall Structure Diagram>

[0028] Figure 1This is an example of the overall structure diagram of an embodiment of the present invention (Embodiment 1). In <Embodiment 1>, user 11 receives recommendations for eye makeup products suitable for user 11 through user terminal 10 (user terminal 10 is not connected to server 20).

[0029] User terminal 10 is a terminal used by user 11 who receives recommendations for eye makeup products. For example, user 11 can use the smartphone application installed in user terminal 10 to receive recommendations for eye makeup products. User terminal 10 is, for example, any computer (also referred to as an information processing device) such as a smartphone, a tablet terminal, or a personal computer.

[0030] <Functional Structure>

[0031] Figure 2 This is an example of the functional block diagram of user terminal 10 of an embodiment of the present invention (Embodiment 1). User terminal 10 includes an image acquisition unit 101, a feature extraction unit 102, a product recommendation unit 103, a display unit 104, and a feature - product correspondence storage unit 105. User terminal 10 functions as image acquisition unit 101, feature extraction unit 102, product recommendation unit 103, and display unit 104 by executing a program (for example, a smartphone application). The following is a description of each respectively.

[0032] Image acquisition unit 101 acquires an image of the eye area including user 11's eyes. For example, image acquisition unit 101 acquires an image taken by the built - in camera of user terminal 10. Here, the image of the eye area including user 11's eyes can be an image of the face including user 11's eyes or an image that only captures the eye area including user 11's eyes.

[0033] Here, an image of the eye area including user 11's eyes is described. In this specification, the eye area including user 11's eyes refers to user 11's eyes and the area around the eyes. The image of the eye area including user 11's eyes can include two types of images: an image taken when user 11 has open eyes and an image taken when user 11 has closed eyes.

[0034] Feature extraction unit 102 extracts features from the eye area including user 11's eyes. The following respectively describes the cases where the recommended eye makeup products are eyeshadow, mascara, and eyeliner. It should be noted that the following feature points are only examples, and other feature points can also be used.

[0035] <Case of Eyeshadow>

[0036] Reference Figure 3 (The state where User 11 has open eyes) and Figure 4 (The state where User 11 has closed eyes) are used to illustrate the method of extracting features from the periorbital region including the eyes of User 11.

[0037] Figure 3 is a diagram for illustrating feature extraction (eyeshadow, open-eye case) of an embodiment of the present invention. The feature extraction unit 102 calculates the brightness (Value) of the black eyeball in the eyes of User 11.

[0038] The feature extraction unit 102, for example, extracts the feature points of the left eye ( Figure 3 (6) to (11) of Figure 3 ) and the feature points of the right eye ( Figure 3 (17) to (22) of

[0039] ) from the image taken in the state where User 11 has open eyes as shown. (6) and (17) represent the inner canthi (the ends close to the nose). (7), (8), (10), (11) and (18), (19), (21), (22) represent the boundaries between the black eyeball, the white eyeball and the surrounding part of the eye. (9) and (20) represent the outer canthi (the ends close to the ears).

[0040] For example, for the left eye, the feature extraction unit 102 generates an inscribed rectangle of the quadrilateral formed by the connection line of (7) and (8), the connection line of (8) and (10), the connection line of (10) and (11), and the connection line of (11) and (7). The feature extraction unit 102 transforms the RGB values of each pixel in the generated rectangular area into L*a*b* values. The feature extraction unit 102 obtains the minimum value of the transformed L* value (brightness). GND

[0041] Figure 4 is a diagram for illustrating feature extraction (eyeshadow, closed-eye case) of an embodiment of the present invention. The feature extraction unit 102 calculates the brightness of the eyelids (inner canthus part and central part) of User 11.

[0042] The feature extraction unit 102, for example, extracts from Figure 4 ​In the image captured with the user 11's eyes closed as shown, the feature points of the left eye ( Figure 4 (6) to (11)) and the feature points of the right eye ( Figure 4 (17) to (22)) are extracted. Figure 4 (6) to (11) and (17) to (22) of Figure 3 are the corresponding feature points to

[0043] (6) to (11) and (17) to (22) of Figure 4 Furthermore, the feature extraction unit 102 extracts, for example, the feature points of the left eyebrow ( Figure 4 (1) to (5)) and the feature points of the right eyebrow ( Figure 4 (12) to (16)) from the image captured with the user 11's eyes closed as shown. (1) and (12) represent one end on the inner corner of the eye side, and (5) and (16) represent one end on the outer corner of the eye side. (2), (3), and (4) are the points that divide the connecting curve between (1) and (5) into four equal parts. (13), (14), and (15) are the points that divide the connecting curve between (12) and (16) into four equal parts.

[0044] <Brightness of the inner corner part>

[0045] For example, for the left eye, the feature extraction unit 102 makes a line connection between (1) of the eyebrow and (6) of the eye, and calculates the distance and the midpoint between the two points. The feature extraction unit 102 generates a rectangle with a side length equal to 1 / 3 of the distance between the two points (i.e., the distance between (1) and (6)) with the calculated midpoint as the center. The feature extraction unit 102 transforms the RGB values of each pixel in the generated rectangular area into L*a*b* values. The feature extraction unit 102 calculates the average value of the transformed L* value (brightness), and takes the average value of the L* value as "L_EYE IN ".

[0046] Similarly for the right eye, the feature extraction unit 102 makes a line connection between (12) of the eyebrow and (17) of the eye, and calculates the distance and the midpoint between the two points. The feature extraction unit 102 generates a rectangle with a side length equal to 1 / 3 of the distance between the two points (i.e., the distance between (12) and (17)) with the calculated midpoint as the center. The feature extraction unit 102 transforms the RGB values of each pixel in the generated rectangular area into L*a*b* values. The feature extraction unit 102 calculates the average value of the transformed L* value (brightness), and takes the average value of the L* value as "L_EYE IN ".

[0047] <Brightness of the central part>

[0048] For example, for the left eye, the feature extraction unit 102 makes a line connection between (4) of the eyebrows and (7) of the eyes, and calculates the distance between the two points and the midpoint. With the calculated midpoint as the center, the feature extraction unit 102 generates a rectangle with a side length equal to 1 / 3 of the distance between the two points (i.e., the distance between (4) and (7)). The feature extraction unit 102 transforms the RGB values of each pixel in the generated rectangular area into L*a*b* values. The feature extraction unit 102 calculates the average value of the transformed L* value (brightness), and uses the average value of the L* value as "L_EYE CNT ".

[0049] Similarly for the right eye, the feature extraction unit 102 makes a line connection between (15) of the eyebrows and (18) of the eyes, and calculates the distance between the two points and the midpoint. With the calculated midpoint as the center, the feature extraction unit 102 generates a rectangle with a side length equal to 1 / 3 of the distance between the two points (i.e., the distance between (15) and (18)). The feature extraction unit 102 transforms the RGB values of each pixel in the generated rectangular area into L*a*b* values. The feature extraction unit 102 calculates the average value of the transformed L* value (brightness), and uses the average value of the L* value as "L_EYE CNT ".

[0050] As described above, the feature extraction unit 102 calculates the brightness of the eyelids (the central part and the inner corner part) of the user 11 for the right eye and the left eye, and the brightness of the black eyeballs in the eyes of the user 11. The feature extraction unit 102 substitutes the calculated brightness into (Equation 1) to calculate the brightness of the right eyelid and the left eyelid. Here, in addition to using a constant for the denominator as shown in (Equation 1) (it is "100" in (Equation 1)), a variable can also be used (for example, instead of "100", use "L_EYE CNT ").

[0051]

[0052] As described above, based on the average value of the brightness of the central part of the user 11's eyelids (i.e., assumed to be the brightest part of the eyelids) and the brightness of the inner corner part (i.e., assumed to be the darkest part of the eyelids), the brightness of each eyelid is calculated.

[0053] The feature extraction unit 102 calculates the average value of the brightness of the right eyelid and the left eyelid obtained from the above (Equation 1). The average value of the brightness of the right eyelid and the left eyelid is used to select the products recommended to the user 11.

[0054] Here, an explanation is given regarding the brightness of the eyelids. The method for calculating the eyelid brightness is not limited to the above method. For example, the feature extraction unit 102 can find the minimum value of the V value of the HSV values of each pixel in the rectangular area of the black eyeball in the eye, and use the minimum value of the V value of the HSV values as "L_PPL GND”. In addition, the feature extraction unit 102 can calculate the average value of the V values of the HSV values of each pixel within the rectangular region of the inner corner part of the eyelid, and use the average value of the V values of the HSV values as "L_EYE" IN ”. Further, the feature extraction unit 102 can also calculate the average value of the V values of the HSV values of each pixel within the rectangular region of the central part of the eyelid, and use the V value of the HSV values as "L_EYE" CNT ”. Furthermore, it is also possible to use the luminance value of any one of the RGB values (for example, the average value of the G values on the G frame or the B values on the B frame).

[0055] In addition, in a manner of adding to the brightness of the above-mentioned eyelid or replacing the brightness of the above-mentioned eyelid, the feature extraction unit 102 can also extract the eyelid chroma (Chroma, vividness) such as the S value of the HSV value as a feature used when selecting a product to be recommended to the user 11.

[0056] <Regarding the cases of mascara and eyeliner>

[0057] Refer to Figure 5 to describe a method for extracting features from the periorbital part including the eyes of the user 11.

[0058] Figure 5 is a diagram for explaining feature extraction (in the cases of mascara and eyeliner) according to an embodiment of the present invention. The feature extraction unit 102 calculates the ratio of the vertical height to the horizontal width of the eyes of the user 11.

[0059] The feature extraction unit 102, for example, extracts the feature points of the left eye ( Figure 5 (6) to (11) of Figure 5 ) and the feature points of the right eye ( Figure 5 (17) to (22) of

[0060] from an image taken in the state where the user 11 has open eyes as shown in

[0061] The same applies to the right eye. The feature extraction unit 102 generates a line (line 1) that intersects the center line perpendicularly and passes through (18) or (19). In addition, the feature extraction unit 102 generates a line (line 2) that intersects the center line perpendicularly and passes through (21) or (22). The feature extraction unit 102 takes the shortest distance between line 1 and line 2 as the vertical height of the right eye of the user 11, and takes the distance between the two points (17) and (20) as the horizontal width of the right eye of the user 11.

[0062] As described above, the feature extraction unit 102 calculates the vertical height and the horizontal width of the eyes of the user 11 for the right eye and the left eye, and calculates the ratio of the vertical height to the horizontal width. For example, based on the aspect ratio of the eyes, it is determined whether the eyes of the user 11 are round eyes or phoenix eyes.

[0063] The feature extraction unit 102 also calculates the average value of the ratio of the vertical height to the horizontal width of the right eye calculated above and the ratio of the vertical height to the horizontal width of the left eye. The average value of the ratio of the vertical height to the horizontal width of the right eye and the ratio of the vertical height to the horizontal width of the left eye is used for the selection process of the products recommended to the user 11.

[0064] In the feature-product correspondence storage unit 105, the feature information extracted from the eye peripheral part including the user's eyes and one or more product information determined by beauty professionals such as makeup artists or beauty consultants to be suitable for users with such features are stored in an associated manner.

[0065] Here, products can also be recommended by multiple beauty professionals such as makeup artists or beauty consultants. In this case, by setting so that the user 11 can specify the desired makeup artist, etc. in the user terminal 10, the product recommendation unit 103 will refer to the corresponding relationship of the specified makeup artist, etc.

[0066] For example, in the feature-product correspondence storage unit 105, various color eyeshadow information suitable for the user 11 with a relatively high eyelid brightness (for example, the average value of the right eyelid brightness and the left eyelid brightness is above the threshold) and various color eyeshadow information suitable for the user 11 with a relatively low eyelid brightness (for example, the average value of the right eyelid brightness and the left eyelid brightness is less than the threshold) are stored.

[0067] For example, in the feature-product correspondence storage unit 105, various types of mascara and eyeliner information suitable for the user 11 with round eyes (for example, the average value of the ratio of the vertical height to the horizontal width of the right eye and the ratio of the vertical height to the horizontal width of the left eye is less than the threshold) and various types of mascara and eyeliner information suitable for the user 11 with phoenix eyes (for example, the average value of the ratio of the vertical height to the horizontal width of the right eye and the ratio of the vertical height to the horizontal width of the left eye is above the threshold) are stored.

[0068] In addition, the characteristic information of multiple users can be collected in advance, and values that can classify these users into four categories (i.e., round eyes with higher eyelid brightness, phoenix eyes with higher eyelid brightness, round eyes with lower eyelid brightness, phoenix eyes with lower eyelid brightness) are used as thresholds (i.e., the threshold for judging high or low eyelid brightness, the threshold for judging round eyes or phoenix eyes).

[0069] Based on the characteristics extracted by the feature extraction unit 102, the product recommendation unit 103 recommends products for eye makeup. Specifically, the product recommendation unit 103 refers to the corresponding relationship stored in the feature - product correspondence storage unit 105 and selects products corresponding to the characteristics extracted by the feature extraction unit 102 (for example, eyeshadow of a color suitable for user 11 or mascara of a type suitable for user 11 (such as mascara for lengthening eyelashes, mascara for thickening eyelashes, mascara for densifying eyelashes, etc.)).

[0070] <User preferences>

[0071] In one embodiment of the present invention, suitable products can be recommended to the user according to the preferences specified by user 11. Specifically, the image acquisition unit 101 acquires the preferences specified by the user (for example, image selection that can evoke the impression after makeup, responses to questionnaires) together with the image. The product recommendation unit 103 selects products corresponding to the user preferences and the characteristics extracted by the feature extraction unit 102 by referring to the corresponding relationship stored in the feature - product correspondence storage unit 105. In this case, user preferences, characteristics extracted from the periorbital part including the user's eyes, and products that are judged by beauty professionals such as makeup artists or beauty consultants to be suitable for users with such characteristics are associated and stored in the feature - product correspondence storage unit 105.

[0072] <Combination of eyeshadow and mascara>

[0073] In one embodiment of the present invention, products that take into account the combination of eyeshadow and mascara can be recommended. Specifically, from the combinations of various-color eyeshadows suitable for user 11 with a relatively high eyelid brightness and various types of mascaras suitable for user 11 with round eyes, the combinations of eyes and mascaras recommended by makeup artists and the application methods are recommended to user 11 as products. In addition, from the combinations of various-color eyeshadows suitable for user 11 with a relatively high eyelid brightness and various types of mascaras suitable for user 11 with phoenix eyes, the combinations of eyeshadow and mascara recommended by makeup artists and the application methods are recommended to user 11 as products. In addition, from the combinations of various-color eyeshadows suitable for user 11 with a relatively low eyelid brightness and various types of mascaras suitable for user 11 with round eyes, the combinations of eyeshadow and mascara recommended by makeup artists and the application methods are recommended to user 11 as products. In addition, from the combinations of various-color eyeshadows suitable for user 11 with a relatively low eyelid brightness and various types of mascaras suitable for user 11 with phoenix eyes, the combinations of eyeshadow and mascara recommended by makeup artists and the application methods are recommended to user 11 as products. Here, it can also be a combination of eyeshadow, mascara, and eyeliner.

[0074] The display unit 104 presents the products recommended by the product recommendation unit 103 to user 11. Specifically, the display unit 104 displays information such as the product names, descriptions, prices, and product images of the products recommended by the product recommendation unit 103 on the screen of the user terminal 10. In addition, in one embodiment of the present invention, user 11 can also purchase the displayed products through the network.

[0075] Figure 6 It is a flowchart showing an example of product recommendation processing according to one embodiment of the present invention.

[0076] In step 11 (S11), the image acquisition unit 101 acquires an image of the periorbital region including the eyes of user 11.

[0077] In step 12 (S12), the feature extraction unit 102 extracts features from the periorbital region including the eyes in the image acquired in S11.

[0078] In step 13 (S13), the product recommendation unit 103 recommends products for eye makeup based on the features extracted in S12.

[0079] In step 14 (S14), the display unit 104 displays the products recommended in S13.

[0080] Figure 7 It is a flowchart showing an example of feature extraction processing (in the case of eyeshadow) according to one embodiment of the present invention.

[0081] In step 21 (S21), the feature extraction unit 102 extracts feature points ((6) to (11) of the left eye, (17) to (22) of the right eye, (1) to (5) of the left eyebrow, and (12) to (16) of the right eyebrow) from the images captured while the user 11 has open eyes and the images captured while the user 11 has closed eyes.

[0082] In step 22 (S22), the feature extraction unit 102 calculates the brightness of the central part of the user 11's eyelids, the brightness of the inner corner part of the user 11's eyelids, and the brightness of the black eyeballs inside the user 11's eyes based on the feature points extracted in S21.

[0083] In step 23 (S23), the feature extraction unit 102 substitutes the brightness values calculated in S22 into the above (Equation 1) to calculate the brightness of the right eyelid and the brightness of the left eyelid.

[0084] In step 24 (S24), the feature extraction unit 102 calculates the average value of the right eyelid brightness and the left eyelid brightness calculated in S23. The average value of the right eyelid brightness and the left eyelid brightness is used to select products to be recommended to the user 11.

[0085] Figure 8 It is a flowchart showing an example of the feature extraction process (in the case of mascara and eyeliner) according to an embodiment of the present invention.

[0086] In step 31 (S31), the feature extraction unit 102 extracts feature points ((6) to (11) of the left eye and (17) to (22) of the right eye) from the pattern captured while the user 11 has open eyes.

[0087] In step 32 (S32), the feature extraction unit 102 calculates the vertical height and the horizontal width of the eyes based on the feature points extracted in S31.

[0088] In step 33 (S33), the feature extraction unit 102 calculates the ratio of the vertical height to the horizontal width of the eyes calculated in S32.

[0089] In step 34 (S34), the feature extraction unit 102 calculates the average value of the ratio of the vertical height to the horizontal width of the right eye and the ratio of the vertical height to the horizontal width of the left eye calculated in S33. The average value of the ratio of the vertical height to the horizontal width of the right eye and the ratio of the vertical height to the horizontal width of the left eye is used to select products to be recommended to the user 11.

[0090] <Embodiment 2>

[0091] Hereinafter, Embodiment 2 will be described.

[0092] <Overall Structure Diagram>

[0093] Figure 9 This is an example of the overall structure diagram of an embodiment of the present invention (Embodiment 2). In <Embodiment 2>, the user 11 connects the user terminal 10 to the server 20 and receives recommendations for eye makeup products suitable for the user 11.

[0094] As Figure 9 shown, the eye makeup product recommendation system (also referred to as an information processing system) 1 includes a user terminal 10 and a server 20. The user terminal 10 and the server 20 are connected in a communicable manner via an arbitrary network 30. Hereinafter, each will be described separately.

[0095] The user terminal 10 is a terminal used by the user 11 who receives recommendations for eye makeup products. For example, the user 11 can use the smartphone application installed in the user terminal 10 to connect to the server 20 and receive recommendations for eye makeup products. The user terminal 10 is, for example, an arbitrary computer (also referred to as an information processing device) such as a smartphone, a tablet terminal, or a personal computer.

[0096] The server 20 executes processing for recommending suitable eye makeup products to the user 11 according to a request from the user terminal 10. The server 20 is composed of one or more computers (also referred to as information processing devices).

[0097] <Functional Structure>

[0098] Figure 10 This is an example of the functional block diagram of the eye makeup product recommendation system of an embodiment of the present invention (Embodiment 2). In an embodiment of the present invention, the server 20 may include at least one of the image acquisition unit 101, the feature extraction unit 102, the product recommendation unit 103, the display unit 104, and the feature - product correspondence storage unit 105 described in <Embodiment 1>. In <Embodiment 2>, the user terminal 10 includes the image acquisition unit 101 and the display unit 104, and the server 20 includes the feature extraction unit 102, the product recommendation unit 103, and the feature - product correspondence storage unit 105. The following will be described in detail.

[0099] The user terminal 10 includes an image acquisition unit 101, an image transmission unit 106, a product information reception unit 109, and a display unit 104. Hereinafter, each will be described separately.

[0100] The image acquisition unit 101 is the same as that in <Embodiment 1>, so the description is omitted.

[0101] The image transmission unit 106 transmits the image acquired by the image acquisition unit 101 to the server 20.

[0102] The product information reception unit 109 receives information on the products recommended by the server 20 from the server 20.

[0103] The display unit 104 displays the product information received by the product information receiving unit 109 on the screen of the user terminal 10.

[0104] The server 20 includes an image receiving unit 107, a feature extraction unit 102, a product recommendation unit 103, a product information sending unit 108, and a feature-product correspondence storage unit 105. Hereinafter, each of these will be described.

[0105] The image receiving unit 107 receives the image acquired by the image acquisition unit 101 of the user terminal 10 from the user terminal 10.

[0106] The feature extraction unit 102 is the same as that in <Embodiment 1>, so the description thereof is omitted.

[0107] The product recommendation unit 103 is the same as that in <Embodiment 1>, so the description thereof is omitted.

[0108] The feature-product correspondence storage unit 105 is the same as that in <Embodiment 1>, so the description thereof is omitted.

[0109] The product information sending unit 108 sends the product information recommended by the product recommendation unit 103 to the user terminal 10.

[0110] As described above, in one embodiment of the present invention, the server 20 may include at least one of the image acquisition unit 101, the feature extraction unit 102, the product recommendation unit 103, the display unit 104, and the feature-product correspondence storage unit 105 described in <Embodiment 1>.

[0111] <Effect>

[0112] As described above, in one embodiment of the present invention, it is possible to recommend eye makeup products that are judged by beauty professionals such as makeup artists or beauty consultants to be suitable for users with such features based on the features extracted from the periorbital area including the user's eyes. For example, it is possible to recommend eye shadow of an appropriate color based on the brightness of the eyelid (specifically, the value calculated based on the average of the brightness of the central part of the eyelid and the brightness of the inner corner part of the eye and the brightness of the darkest part of the black eyeball inside the eye), and recommend an appropriate type of mascara or eyeliner and its application method based on round eyes or phoenix eyes (specifically, the aspect ratio of the eyes).

[0113] <Hardware Structure>

[0114] Figure 11FIG. 0 is a block diagram showing an example of the hardware configuration of an information processing apparatus (i.e., the user terminal 10 and the server 20) according to an embodiment of the present invention. The user terminal 10 and the server 20 include a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003. The CPU 1001, the ROM 1002, and the RAM 1003 form a so-called computer.

[0115] In addition, the user terminal 10 and the server 20 include an auxiliary storage device 1004, a display device 1005, an operation device 1006, an I / F (Interface) device 1007, and a drive device 1008. Here, each hardware of the user terminal 10 and the server 20 is connected via a bus B.

[0116] The CPU 1001 is an arithmetic device for executing various programs installed in the auxiliary storage device 1004.

[0117] The ROM 1002 is a non-volatile memory. The ROM 1002 functions as a main storage device for storing various programs, data, etc. required when the CPU 1001 executes various programs installed in the auxiliary storage device 1004. Specifically, the ROM 1002 functions as a main storage device for storing a boot program such as BIOS (Basic Input / Output System) or EFI (Extensible Firmware Interface).

[0118] The RAM 1003 is a volatile memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The RAM 1003 functions as a main storage device for providing a working area expanded when the CPU 1001 executes various programs installed in the auxiliary storage device 1004.

[0119] The auxiliary storage device 1004 is an auxiliary storage device for storing various programs and information used when executing various programs.

[0120] The display device 1005 is a display device for displaying the internal state of the user terminal 10 and the server 20.

[0121] The operation device 1006 is an input device for an administrator of the user terminal 10 and the server 20 to input various instructions to the user terminal 10 and the server 20.

[0122] The I / F device 1007 is a communication device for connecting to the network 30 and communicating with the user terminal 10 and the server 20.

[0123] The drive device 1008 is a device for setting the storage medium 1009. The storage medium 1009 mentioned here includes media such as CD-ROM, floppy disk, and optical disk that can record information optically, electrically, or magnetically. In addition, the storage medium 1009 can also include semiconductor memories such as EPROM (Erasable Programmable Read Only Memory) and flash memory that record information electrically.

[0124] In addition, regarding various programs installed in the auxiliary storage device 1004, for example, they are installed by setting the allocated storage medium 1009 with the drive device 1008 and reading various programs recorded in the storage medium 1009 by the drive device 1008. Or, various programs installed in the auxiliary storage device 1004 can also be installed by using the I / F device 1007 to download them from other networks outside the network 30.

[0125] The embodiments of the present invention have been described in detail above, but the present invention is not limited to the above specific embodiments, and various deformations and changes can be made within the scope of the gist of the present invention described in the claims.

[0126] Symbol Explanation

[0127] 1 Eye Makeup Product Recommendation System

[0128] 10 User Terminal

[0129] 11 User

[0130] 20 Server

[0131] 30 Network

[0132] 101 Image Acquisition Unit

[0133] 102 Feature Extraction Unit

[0134] 103 Product Recommendation Unit

[0135] 104 Display Unit

[0136] 105 Feature-Product Correspondence Storage Unit

[0137] 106 Image Sending Unit

[0138] 107 Image Receiving Unit

[0139] 108 Product Information Sending Unit

[0140] 109 Commodity Information Receiving Department

Claims

1. A program product for causing an information processing device to function as the following components: An image acquisition unit that acquires an image of a periorbital region including a user's eyes, the image including an image taken while the user has open eyes and an image taken while the user has closed eyes; A feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the brightness of the black eyeball in the user's eyes, and the brightness of the central part of the user's eyelids and the brightness of the inner corner part of the eyelids; A product recommendation unit that recommends eyeshadows suitable for the user based on the brightness of the black eyeball and the average of the brightness of the central part of the eyelids and the brightness of the inner corner part of the eyelids; and A display unit that displays the recommended eyeshadows.

2. The program product according to claim 1, wherein the brightness of the black eyeball is extracted from an image taken while the user has open eyes, and the brightness of the central part of the eyelids and the brightness of the inner corner part of the eyelids are extracted from an image taken while the user has closed eyes, the features further include the chroma of the user's eyelids, the product recommendation unit recommends eyeshadows suitable for the user based on the brightness and chroma of the eyelids, the brightness of the eyelids is calculated based on the brightness of the central part of the eyelids, the brightness of the inner corner part of the eyelids, and the brightness of the black eyeball.

3. The program product according to claim 1 or 2, wherein the product recommendation unit recommends a product suitable for the user based on the correspondence between the features and a product suitable for a user having the features.

4. The program product according to claim 1 or 2, wherein the product recommendation unit recommends a product suitable for the user based on the features and the preferences specified by the user.

5. A program product for causing an information processing device to function as the following components: An image acquisition unit that acquires an image of a periorbital region including a user's eyes, the image being an image taken while the user has open eyes; A feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the ratio of the vertical height to the horizontal width of the user's eyes; A product recommendation unit that recommends mascara and eyeliner suitable for the user based on the ratio of the vertical height to the horizontal width of the eyes; and A display unit that displays the recommended mascara and eyeliner, wherein the horizontal width of the eyes refers to the distance between two points, namely, a feature point representing the inner corner of the user's eyes and a feature point representing the outer corner of the user's eyes in the image, the vertical height of the eyes refers to the shortest distance between two lines generated based on four feature points of the boundary between the black eyeball, the white eyeball, and the surrounding part of the eyes of the user in the image, one of the two lines intersects perpendicularly with the midline and passes through any one of the two feature points closer to the eyebrows among the four feature points, The other of the two lines intersects the midline perpendicularly and passes over any one of the two feature points among the four feature points that are far from the eyebrows. The product recommendation unit determines whether the ratio of the vertical width to the horizontal width of the user's eyes is above a specified threshold, and recommends mascara and eyeliner suitable for the user based on the determination result.

6. The program product according to claim 5, wherein The product recommendation unit recommends products suitable for the user according to the correspondence between the features and the products suitable for users with such features.

7. The program product according to claim 5, wherein The product recommendation unit recommends products suitable for the user according to the features and the preferences specified by the user.

8. An eye makeup product recommendation method, which is executed by an information processing device and includes the following steps: Obtain an image of the eye area including the user's eyes, the image including an image taken when the user has open eyes and an image taken when the user has closed eyes; Extract features from the eye area including the eyes in the image, the features including the brightness of the black eyeball in the user's eyes, and the brightness of the central part of the user's eyelids and the brightness of the inner corner part of the eyelids; Recommend eyeshadow suitable for the user according to the brightness of the black eyeball and the average value of the brightness of the central part of the eyelids and the brightness of the inner corner part of the eyelids; and Display the recommended eyeshadow.

9. An eye makeup product recommendation method, which is executed by an information processing device and includes the following steps: Obtain an image of the eye area including the user's eyes, the image being an image taken when the user has open eyes; Extract features from the eye area including the eyes in the image, the features including the ratio of the vertical height to the horizontal width of the user's eyes; Recommend mascara and eyeliner suitable for the user according to the ratio of the vertical height to the horizontal width of the eyes; and Display the recommended mascara and eyeliner, wherein The horizontal width of the eyes refers to the distance between two points between the feature point representing the inner corner of the user's eyes and the feature point representing the outer corner of the user's eyes in the image. The vertical height of the eyes refers to the shortest distance between two lines generated based on four feature points of the boundary between the black eyeball, white eyeball and the surrounding part of the user's eyes in the image. One of the two lines intersects the midline perpendicularly and passes over any one of the two feature points among the four feature points that are close to the eyebrows. The other of the two lines intersects the midline perpendicularly and passes over any one of the two feature points among the four feature points that are far from the eyebrows. When making the recommendation, it is determined whether the ratio of the vertical width to the horizontal width of the user's eyes is above a specified threshold, and based on the determination result, mascara and eyeliner suitable for the user are recommended.

10. An information processing device includes: An image acquisition unit that acquires an image of a periorbital region including a user's eyes, the image including an image taken while the user has their eyes open and an image taken while the user has their eyes closed; A feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the brightness of the black eyeball in the user's eye, and the brightness of the central part of the user's eyelid and the brightness of the inner corner part of the eyelid; A product recommendation unit that recommends eyeshadow suitable for the user based on the brightness of the black eyeball and the average of the brightness of the central part of the eyelid and the brightness of the inner corner part of the eyelid; and A display unit that displays the recommended eyeshadow.

11. An information processing apparatus, comprising: An image acquisition unit that acquires an image of a periorbital region including a user's eyes, the image being an image taken while the user has their eyes open; A feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the ratio of the vertical height to the horizontal width of the user's eyes; A product recommendation unit that recommends mascara and eyeliner suitable for the user based on the ratio of the vertical height to the horizontal width of the eyes; and A display unit that displays the recommended mascara and eyeliner, wherein The horizontal width of the eyes refers to the distance between two points, namely, the feature point representing the inner corner of the user's eyes and the feature point representing the outer corner of the user's eyes in the image; The vertical height of the eyes refers to the shortest distance between two lines generated based on four feature points of the boundary between the black eyeball, the white eyeball, and the surrounding part of the eye in the image; One of the two lines intersects perpendicularly with the midline and passes through any one of the two feature points closer to the eyebrows among the four feature points; The other of the two lines intersects perpendicularly with the midline and passes through any one of the two feature points farther from the eyebrows among the four feature points; The product recommendation unit determines whether the ratio of the vertical width to the horizontal width of the user's eyes is above a specified threshold, and based on the result of the determination, recommends mascara and eyeliner suitable for the user.

12. An information processing system, which includes a user terminal and a server, and comprises: An image acquisition unit that acquires an image of a periorbital region including a user's eyes, the image including an image taken while the user has their eyes open and an image taken while the user has their eyes closed; A feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the brightness of the black eyeball in the user's eye, and the brightness of the central part of the user's eyelid and the brightness of the inner corner part of the eyelid; A product recommendation unit that recommends eyeshadow suitable for the user based on the brightness of the black eyeball and the average of the brightness of the central part of the eyelid and the brightness of the inner corner part of the eyelid; and A display unit that displays the recommended eyeshadow.

13. An information processing system, which includes a user terminal and a server, and has: an image acquisition unit that acquires an image of a periorbital region including the user's eyes, the image being an image captured in a state where the user has open eyes; a feature extraction unit that extracts features from the periorbital region including the eyes in the image, the features including the ratio of the vertical height to the horizontal width of the user's eyes; a product recommendation unit that recommends mascara and eyeliner suitable for the user according to the ratio of the vertical height to the horizontal width of the eyes; and a display unit that displays the recommended mascara and eyeliner, wherein the horizontal width of the eyes refers to the distance between two points, namely, a feature point representing the inner canthus of the user's eyes and a feature point representing the outer canthus of the user's eyes, in the image; the vertical height of the eyes refers to the shortest distance between two lines generated based on four feature points representing the boundaries between the black eyeballs, white eyeballs and the surrounding parts of the user's eyes in the image; one of the two lines intersects perpendicularly with the midline and passes through any one of the two feature points closer to the eyebrows among the four feature points; the other of the two lines intersects perpendicularly with the midline and passes through any one of the two feature points farther from the eyebrows among the four feature points; the product recommendation unit determines whether the ratio of the vertical width to the horizontal width of the user's eyes is above a specified threshold, and recommends mascara and eyeliner suitable for the user according to the result of the determination.

Images