Information processing device, information processing method, and information processing program

Abstract

This will further improve usability for users who refer to other people's makeup looks. [Solution] The information processing device according to the present invention comprises a determination unit and an application unit, wherein the determination unit determines the application method when applying a predetermined decoration in the face image of a decorator to the face image of a user, based on a comparison result between the face image of the decorator and the face image of the user, and the application unit applies the predetermined decoration to the face image of the user in the application method determined by the determination unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, techniques for assisting a user in easily trying desired makeup have been known. For example, Patent Document 1 discloses a technique for providing a face image obtained by applying makeup information generated from a face image of a poster who has applied a predetermined makeup selected by the user to the face image of the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, if makeup information generated from the face image of a poster is applied to the face image of a user without considering the difference between the user's face and the poster's face, there is a risk of providing a face image with a sense of incongruity. For this reason, there has been room for further improvement in user-friendliness for applying makeup information of others to the face image of a user.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of determining a makeup mode for applying makeup in a face image of another person who has applied makeup to a face image of a user based on the face image of the user and the face image of the other person, and applying makeup to the face image of the user in the determined makeup mode.

Means for Solving the Problems

[0006] The information processing device according to the present invention is characterized by having a determination unit that determines the application method when applying a predetermined decoration in the face image of a decorator to the face image of a user, based on a comparison result between the face image of the decorator and the face image of the user, and an application unit that applies the predetermined decoration to the face image of the user in the application method determined by the determination unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to promote further improvements in usability for users who refer to the makeup of others. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows the necessity of blending the eyebrows according to the embodiment. [Figure 3] Figure 3 shows an overview of the eyebrow blending process according to this embodiment. [Figure 4] Figure 4 shows a specific example (1) of eyebrow blending according to the embodiment. [Figure 5] Figure 5 shows a specific example (2) of eyebrow blending according to the embodiment. [Figure 6] Figure 6 shows an overview of the eyebrow blending correction according to the embodiment. [Figure 7] Figure 7 shows an overview of the cheek application process according to this embodiment. [Figure 8] Figure 8 shows an example of the configuration of an information processing system according to an embodiment. [Figure 9] Figure 9 shows an example of the configuration of a terminal device according to an embodiment. [Figure 10] Figure 10 shows an example of the configuration of an information processing device according to an embodiment. [Figure 11] Figure 11 shows an example of a user information storage unit according to an embodiment. [Figure 12]Figure 12 shows an example of a user decoration information storage unit according to the embodiment. [Figure 13] Figure 13 is a flowchart showing the processing procedure according to the embodiment. [Figure 14] Figure 14 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant descriptions are omitted.

[0010] [1. An example of information processing] First, an example of information processing according to the embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiment, and is executed by the information processing device 100. In Figure 1, an example will be given in which a predetermined makeup style is applied to the user's face image, based on the user's face image and a face image of another person with predetermined makeup applied, which the user has selected.

[0011] The information processing device 100 shown in Figure 1 is an information processing device that is connected to terminal devices 10a, 10b, ..., 10n of multiple users Ua, Ub, ..., Un via a network N (see Figure 8) via wired or wireless connections, enabling communication between them, and provides various types of information to users Ua, Ub, ..., Un. This is implemented, for example, by one or more servers or cloud systems. Hereinafter, users Ua, Ub, ..., Un may be referred to as User U, and terminal devices 10a, 10b, ..., 10n may be referred to as Terminal Device 10. In addition, users in general may be referred to as User U. Furthermore, "makeup" may be referred to as "decoration," "makeup target" such as eyebrows may be referred to as "decoration target," "makeup (makeup) style" may be referred to as "decoration style," and "another person who has applied makeup" may be referred to as "decorator." Furthermore, the facial image of the user to whom the specified decoration is applied may be without makeup, or it may have base makeup (application of makeup base, foundation, concealer, face powder, etc.) or full makeup (application of base makeup plus face color (blush, highlighter, contouring, etc.), eyeshadow, eyeliner, eyebrow makeup, mascara, lipstick, etc.). Also, the facial image of the user to whom the specified decoration is applied may or may not have point makeup applied to the object to be decorated. In addition, the makeup applied by the decorator may be of any type. For example, it may only be eyebrow makeup.

[0012] In this application, "decorative form" refers to, for example, a makeup form using a specified cosmetic product, and "decorative information indicated by the decorative form" refers to information about the color of the makeup area which is the makeup form, or information about the skin color around the makeup area in addition to the makeup form.

[0013] The terminal device 10 is a smart device such as a smartphone or a tablet terminal used by the user U, and is an information processing device capable of communicating with an information processing device such as an arbitrary server via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). Further, the terminal device 10 has a screen such as a liquid crystal display, and has a screen having a touch panel function, and accepts various operations on display data such as content, such as a tap operation, a slide operation, and a scroll operation, from the user U using a finger, a stylus, or the like. Note that an operation performed on the area of the screen where the content is displayed may also be regarded as an operation on the content. Further, the terminal device 10 may be not only a smart device but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0014] The information processing device 100 is an information processing device that cooperates with the terminal device 10 of each user U and provides API (Application Programming Interface) services and various data for the terminal device 10 of each user U with respect to various applications (hereinafter referred to as apps), etc., and is realized by a computer, a cloud system, or the like.

[0015] Further, the information processing device 100 may be an information processing device that provides some kind of web service online for the terminal device 10 of each user U. For example, the information processing device 100 may provide services such as an Internet connection service, a search service, an e-commerce service, and an SNS (Social Networking Service) as web services. In reality, the information processing device 100 may cooperate with various servers that provide the above web services and mediate the web services, or may be in charge of the processing of the web services.

[0016] Furthermore, the information processing device 100 can acquire user information about user U. For example, the information processing device 100 acquires information about user U's attributes, such as gender, age, and residential area. The information processing device 100 then stores and manages information about user U's attributes along with identification information (such as user ID) that identifies user U. Note that user attributes are not mandatory and may be estimated using various estimation techniques.

[0017] Incidentally, if a user wants to try out the makeup style seen in a face image posted by a celebrity or other influencer whose follower count exceeds a certain threshold on social media, they may need to search for the brand, product name, and color number of the cosmetics used in that makeup on social media, and gather information from reviews and other sources. Also, in recent years, it has become a trend for users to edit and correct their own faces using apps, focusing on specific facial features.

[0018] Therefore, the information processing device 100 according to the embodiment determines the application method when applying a predetermined decoration from the decorator's face image to the user's face image based on the decorator's face image and the user's face image, and provides the user with the user's face image in which the predetermined decoration has been applied according to the determined application method.

[0019] [1-1. Obtain user Un's face image] As shown in Figure 1, first, the information processing device 100 acquires a facial image of user Un from the terminal device 10n (step S1). User Un's facial image may be without makeup, or with base makeup or full makeup applied. Point makeup may also be applied. For example, the terminal device 10n transmits information about the facial image obtained when user Un takes a picture of their own face using a predetermined selfie function to the information processing device 100.

[0020] [1-2. Obtain information on the facial image of the decorator] Next, the information processing device 100 obtains information on a face image selected by user Un from among face images posted by or registered by decorators, such as celebrities or general posters, whose number of followers exceeds a predetermined threshold on social media (step S2). The information on the face image selected by user Un includes, for example, the face image selected by user Un and a partial image of that face image that includes the decoration target and its area specified by user Un. In Figure 1, the information processing device 100 obtains the face image selected by user Un from, for example, face image IMa1 registered or posted by user Ua, face image IMb1 registered or posted by user Ub, etc., in content CT1. Content CT1 is, for example, content on a predetermined service that manages the face images of registrants or posters. Note that the face images that user Un can select may include multiple face images registered or posted by a single user. Also, for example, face images posted by posters that user Un follows may be selected preferentially. The following explanation assumes that user Un has selected face image IMa1 from content CT1.

[0021] [1-3. Determining the object to be decorated] Next, the information processing device 100 determines the object to be decorated (step S3). For example, based on the range specified by user Un on the face image IMa1, the information processing device 100 uses a predetermined contour extraction method to extract makeup parts such as lips (lipstick, etc.), upper or lower eyelids (eyeshadow, eyeliner, etc.), eyebrows (eyeshadow, eyeliner, eyebrow makeup, mascara, etc.), or cheeks (cheeks, highlighter, etc.) from the range specified by user Un on the face image IMa1. For example, the information processing device 100 extracts skin color information (e.g., RGB (Red-Green-Blue Color model)) from the face image IMa1 using a predetermined method and extracts each part using color difference. Alternatively, the information processing device 100 may use a model that has learned the position of makeup parts using a large number of face images annotated with the positions of makeup parts, and extract each part from the face image IMa1, etc. The information processing device 100 may, independently of user operation, cut out a predetermined number of makeup parts and determine multiple decorative objects.

[0022] Regarding the extraction of makeup parts, there are areas where it is difficult to isolate only the parts with makeup applied. For example, for eyebrows, the information processing device 100 extracts the makeup parts by including the parts without makeup (skin) so that the parts with makeup are included. In the case of eyebrows, the hair growth may be sparse, making it difficult to clearly distinguish the boundary between the eyebrows with makeup and the surrounding skin without makeup, so the information processing device 100 extracts the eyebrows and the surrounding skin separately. For example, the information processing device 100 extracts about 4 pixels as the skin area around the eyebrows. For example, for lips, eyelids, cheeks, etc., since the boundary between the makeup color and skin is relatively clear, the information processing device 100 may extract the makeup parts as they are using a predetermined contour extraction method. In addition, the information processing device 100 also extracts the contour of the face from the face image IMa1.

[0023] Next, the information processing device 100 extracts skin tone information from user Un's face image using a predetermined method and extracts parts such as eyebrows, lips, eyelids, and cheeks to be decorated using color differences. The information processing device 100 may also use a model to extract parts such as eyebrows, lips, eyelids, and cheeks from user Un's face image. For example, the information processing device 100 determines the parts of user Un's face image that correspond to the makeup parts by comparing the color information and / or positional information within the face between the makeup parts extracted from face image IMa1 and the parts of the decoration object extracted from user Un's face image. Alternatively, the information processing device 100 may determine the parts of user Un's face image that correspond to the makeup parts by comparing the skin tone information in a predetermined area related to the makeup parts with the skin tone information in a predetermined area related to the parts of the decoration object. Furthermore, the information processing device 100 may determine the parts of the user Un's face image that correspond to the makeup parts by comparing skin tones, for example, skin tones related to the HSV (Hue-Saturation-Value Color model) as skin tone information. The information processing device 100 also extracts the facial contours from the user Un's face image.

[0024] Furthermore, the information processing device 100 may, for example, determine the position of the cheeks in the user Un's face image and determine the cheek area to be decorated that corresponds to the cheek part in the face image IMa1, based on a predetermined part balance and skin color derived from the facial contour extracted from the user Un's face image, and the positions of the eyebrows, lips, and eyelids to be decorated.

[0025] [1-4. Determination of the application method of decorative information indicated by the decorative method] Next, the information processing device 100 determines the application method of the decorative information indicated by the decorative pattern (step S4). First, the eyebrows will be described as the object of decoration. Figure 2 is a diagram showing the necessity of blending the eyebrows according to the embodiment. In Figure 2, it is assumed that the information processing device 100 places the eyebrow part 200 cut out from the face image IMa1 onto the skin 201 of the eyebrow area corresponding to the eyebrow part 200 in the face image of user Un, who is trying on the product. Here, although the eyebrows have shapes such as an arch shape, for convenience, they will be described as having a rectangular shape. Since it is not possible to clearly separate the boundary between the eyebrows and the surrounding skin, the information processing device 100 cuts out the eyebrow part 200, which includes the eyebrow portion 200b and the skin portion 200a of the eyebrow part, from the face image IMa1, and places the cut-out eyebrow part 200 in a predetermined position on the skin of user Un, who is trying on the product. However, if the eyebrow part 200 extracted from the face image IMa1 is simply placed on user Un's skin 202, there is a difference in color between the skin portion 200a of the eyebrow part and the color of user Un's skin 202, resulting in an unnatural appearance due to the lack of color blending around the eyebrows.

[0026] To resolve this incongruity, it becomes necessary to correct the skin color included in the extracted eyebrow part. Figure 3 is a diagram showing an overview of eyebrow blending according to the embodiment. As shown in Figure 3, the eyebrow part 300 extracted from the face image IMa1 includes the skin part 300a of the eyebrow part and the eyebrow part 300b of the eyebrow part. The information processing device 100 first calculates the offset 301. The offset 301 is obtained by subtracting the average skin color 302 of the skin part 300a of the eyebrow part from the average skin color around the eyebrows of user Un, who is trying on the model. For example, if the color of the eyebrow part 300b of the eyebrow part is darker than the color of the skin part 300a of the eyebrow part, the information processing device 100 sets the average color of the skin part 302 to the average color of the skin part 302, which is lighter than the median color of the skin part 300a of the eyebrow part (Pattern A). On the other hand, if the color of the eyebrow 300b of the eyebrow part is lighter than the color of the skin 300a of the eyebrow part, the information processing device 100 sets the average value of the skin color 302 of the eyebrow part to the average value of the skin color 300a of the eyebrow part (Pattern B).

[0027] Next, the information processing device 100 calculates the eyebrow blending color 303. The eyebrow blending color 303 is obtained by adding a predetermined adjusted offset to the color of the eyebrow part 300 (the respective colors of the skin part 300a and the eyebrow part 300b. More specifically, the skin part 300a also has different colors depending on its position, and similarly the eyebrow part 300b also has different colors depending on its position). The predetermined adjusted offset is obtained by dividing the color of the eyebrow part 300 by the average value 302 of the skin color of the eyebrow part, and then multiplying the result by the offset. The information processing device 100 then places the eyebrow part 304, to which the eyebrow blending color has been applied, onto the skin 305 of user Un, who is trying it on. Here, the information processing device 100 adds a predetermined adjusted offset to the color of the eyebrow part 300 because there is a risk that the boundary between the skin part 300a and the eyebrow part 300b may not be distinguishable. However, if the boundary can be distinguished, the predetermined adjusted offset may be added only to the skin part 300a of the eyebrow part. The information processing device 100 calculates the offset and color blending for each color, for example, RGB or HSV.

[0028] Next, specific examples of eyebrow blending according to the embodiment will be described. Figure 4 shows a specific example (1) of eyebrow blending according to the embodiment. As shown in Figure 4, in the case of pattern B, the eyebrow blending color 401 is darker than in pattern A. On the other hand, in the case of pattern A, the eyebrow blending color 402 is lighter than in pattern B. Figure 5 shows a specific example (2) of eyebrow blending according to the embodiment. As shown in Figure 5, in the case of pattern B, the degree of eyebrow blending is low, and the color of the eyebrow 500b of the eyebrow part cut out from face image IMa1 is almost the same as the color of the eyebrow 502b of the eyebrow part. However, around the eyebrow 502b of the eyebrow part placed on user Un's skin 504, the skin 502a of the eyebrow part may be slightly noticeable. On the other hand, in the case of pattern A, the degree of eyebrow blending is high, and the skin 503a of the eyebrow part blends better with user Un's skin 505. However, the color of the eyebrow 501b of the eyebrow part extracted from the face image IMa1 may become darker than the color of the eyebrow 503b of the eyebrow part after eyebrow blending. In this case, by applying eyebrow blending only to the skin 501a of the eyebrow part, it is possible to avoid the color of the eyebrow 503b of the eyebrow part becoming darker than the color of the eyebrow 501b of the eyebrow part. The information processing device 100 does not have to perform the blending process if the color of the eyebrow 501b of the eyebrow part extracted from the face image IMa1 and the color of the eyebrow 503b of the eyebrow part after blending differ by more than a predetermined threshold (if there is a risk of causing a sense of incongruity such as the makeup being different from what was chosen). In other words, the information processing device 100 may determine the application method of the decorative information indicated by the decorative form based on the comparison result between the color of the eyebrow 501b of the eyebrow part extracted from the face image IMa1 and the color of the eyebrow 503b of the eyebrow part after eyebrow blending.

[0029] In determining the application method described above, the color of the eyebrows of the eyebrow part will be adjusted along with the adjustment of the skin color of the eyebrow part. Therefore, the information processing device 100 may, for example, pre-identify the color of the area around the center of the eyebrows of the eyebrow part cut out from the face image IMa1, and if the adjustment range of that color is greater than or equal to a predetermined threshold, multiply the eyebrow part that has been blended (color corrected) by a predetermined adjustment value to correct it. Figure 6 is a diagram showing an overview of the eyebrow blending correction according to the embodiment. As shown in Figure 6, the information processing device 100, for example, multiplies the entire eyebrow part by a predetermined adjustment value according to the color of the eyebrows of the eyebrow part that has been blended to correct it. For example, if the skin color of the eyebrow part 601a is #FFDCBC (R:255, G:220, B:188) and the eyebrow color of the eyebrow part 601b is #8E5624 (R:142, G:86, B:36), the information processing device 100 multiplies each of the RGB values ​​by an adjustment value of 0.95 to correct the skin color of the eyebrow part 602a to #F1D1B4 (R:242, G:209, B:179) and the eyebrow color of the eyebrow part 602b to #875222 (R:135, G:82, B:34). In this case, the difference in color between the eyebrow colors 601b and 602b is not very noticeable. Furthermore, if, for example, the color of the skin 603a of the eyebrow part is #FFDCBC (R:255, G220, B188) and the color of the eyebrow 603b of the eyebrow part is #F0D488 (R:240, G212, B136), the information processing device 100 multiplies each of the RGB values ​​by an adjustment value of 0.95 to correct the color of the skin 604a of the eyebrow part to #F1D1B4 (R:242, G:209, B:179) and the color of the eyebrow 604b of the eyebrow part to #E4C981 (R:228, G:201, B:129). In this case, a slight difference in color can be perceived between the eyebrow colors 603b and 604b. In this case, it is thought that by multiplying by, for example, 0.93 as an adjustment value, it is possible to match the skin tone of the eyebrow part to the skin tone of the try-on user (User Un) while avoiding over-adjusting the eyebrow color of the eyebrow part.

[0030] In this way, the information processing device 100 determines the application method when applying the eyebrow parts extracted from the face image IMa1 to the face image of user Un. Note that the above eyebrow blending is merely an example and is not limited to the method of eyebrow blending described above.

[0031] Here, for each makeup part, there are parts where the difference between the makeup area and the skin area is clear, and parts where it is not. For example, for lip parts, cheek parts, and eyeshadow parts, the makeup area is clearer than that of eyebrow parts, so instead of including the skin area around the makeup part when cutting out the makeup part, the color difference from the skin tone information of the surrounding skin area is used to cut out each makeup part. The cut-out makeup parts are then subjected to a predetermined color blending process and placed on the designated area of ​​the user Un's face image. This prevents the unnatural appearance of having a sticker stuck on.

[0032] The following description will focus on blush as the object of decoration. Figure 7 shows an overview of the blush application according to the embodiment. As shown in Figure 7, the information processing device 100 extracts skin tone information in a predetermined way based on the range specified by user Un on the face image IMa1, for example, and cuts out the blush part 700 using color difference. The information processing device 100 then determines the application method based on the blush part 700 cut out from the face image IMa1, the predetermined position of the cheeks in user Un's face image, the skin tone of user Un's entire face, and the skin tone of the entire face in face image IMa1.

[0033] The information processing device 100 first calculates the offset 701. The offset 701 is obtained by subtracting the average skin color of the entire face of the face image IMa1 from the average skin color of the entire face of the user Un, who is trying on the garment. The skin color of the entire face refers to the skin color excluding the eyebrows, eyes, and mouth (702a, 702b). The information processing device 100, for example, applies the skin color of user Un's face in the face image of user Un, who is trying on the garment, to a predetermined Gaussian distribution and excludes colors that are more than one standard deviation away from the mean of the distribution. The information processing device 100 also applies the skin color of the face in the face image IMa1 to a predetermined Gaussian mixing model and selects the one that is closest to human skin, excluding other colors. In this way, if there are unusual colors in the skin, those unusual colors can be excluded.

[0034] Next, the information processing device 100 calculates the blush blending color 703. The blush blending color 703 is obtained by adding a predetermined adjusted offset to the color of the blush part 700. The predetermined adjusted offset is obtained by multiplying the offset by the result of dividing the color of the blush part 700 by the average value of the skin color of the entire face in the face image IMa1.

[0035] Then, the information processing device 100 applies a color-corrected cheek part 704 to the cheek position in a predetermined facial image of user Un, and places the cheek part 704 onto user Un's skin 705.

[0036] Next, the information processing device 100 applies the same lip blending and eyelid blending processes to the lip and eyelid parts as to the cheek parts, and determines the application method.

[0037] [1-5. Apply the decorative information indicated by the decorative style to the user Un's face image.] Next, the information processing device 100 applies the decorative information indicated by the decorative pattern to the user Un's face image (step S5). Specifically, the information processing device 100 places the makeup parts specified by user Un on the face image IMa1, which have been cut out from the face image IMa1 and color-corrected, onto a predetermined position on user Un's face image.

[0038] Furthermore, if the lips are to be used as the object of decoration, the information processing device 100 may, for example, cut out a lip part from the face image IMa1, apply color correction to it, place it on a predetermined lip area in the user Un's face image, and then use a known edge blending technique to blend the edges of the lip part with user Un's skin. The same applies to cheek parts and eyelid parts.

[0039] [1-6. Provide a facial image to which the decorative information indicated by the decorative design is applied.] Next, the information processing device 100 provides user Un with a face image to which the decorative information indicated by the decorative pattern has been applied (step S6). Specifically, the information processing device 100 applies makeup parts, which have been extracted from the face image IMa1 and color-corrected, to user Un's face image and transmits the resulting image to user Un's terminal device 10n.

[0040] In this way, the information processing device 100 can determine the application method for applying a predetermined decoration from the decorator's face image to the user's face image based on a comparison between the decorator's face image and the user's face image, apply the predetermined decoration to the user's face image in the determined application method, and provide the user with the user's face image to which the predetermined decoration has been applied.

[0041] [2. Example of an information processing system configuration] Next, the configuration of the information processing system 1, which includes the information processing device 100 according to the embodiment, will be described using Figure 8. Figure 8 is a diagram showing an example of the configuration of the information processing system according to the embodiment. As shown in Figure 8, the information processing system 1 according to the embodiment includes a terminal device 10 and an information processing device 100. These various devices are connected to each other via a network N, either by wire or wireless communication. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0042] Furthermore, the number of devices included in the information processing system 1 shown in Figure 8 is not limited to those illustrated. For example, in Figure 8, only two terminal devices 10 are shown for the sake of simplicity, but this is merely an example and not limiting; there may be three or more.

[0043] Terminal device 10 is an information processing device used by user U. For example, terminal device 10 may be a smart device such as a smartphone or tablet, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV equipment with communication functions, a car navigation system, a wearable device such as a smartwatch or head-mounted display, or smart glasses. Alternatively, terminal device 10 may be a house or building compatible with the Internet of Things (IoT), a car, home appliances, electronic devices, etc.

[0044] Furthermore, the terminal device 10 can connect to the network N via wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation: 5th Generation Mobile Communication System), or via short-range wireless communication such as Bluetooth (registered trademark) and Wi-Fi (Local Area Network), and communicate with the information processing device 100.

[0045] The information processing device 100 may be, for example, a computer such as a PC or blade server, or a mainframe or workstation. The information processing device 100 may also be implemented through cloud computing.

[0046] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be described using Figure 9. Figure 9 is a diagram showing an example of the configuration of a terminal device according to the embodiment. As shown in Figure 9, the terminal device 10 comprises a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0047] (Communications Section 11) The communication unit 11 is connected to the network N (see Figure 8) by wire or wireless connection and transmits and receives information to and from the information processing device 100 via the network N. For example, the communication unit 11 can be implemented using a NIC (Network Interface Card) or an antenna.

[0048] (Display section 12) The display unit 12 is a display device that displays various information such as location information. For example, the display unit 12 may be a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 may also be a touch panel display, but is not limited to this.

[0049] (Input section 13) The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons for inputting characters, numbers, etc. The input unit 13 may also be an input / output port (I / O port) or a USB (Universal Serial Bus) port. If the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may also be a microphone that receives voice input from the user U. The microphone may be wireless.

[0050] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection can be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices or wireless devices. In the example shown in Figure 9, the sensor unit 20 includes an optical sensor 21 and an image sensor (camera) 22.

[0051] The sensors 21-22 described above are merely examples and not limiting. In other words, the sensor unit 20 may include other sensors in addition to or instead of sensors 21-22.

[0052] The light sensor 20 detects the illuminance around the terminal device 10. The image sensor 21 captures an image of the area around the terminal device 10.

[0053] (Control Unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM, input / output ports, and various circuits. Alternatively, the control unit 30 may be composed of hardware such as an integrated circuit (ASIC) or FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31, a reception unit 32, and a processing unit 33.

[0054] (Transmitter 31) The transmission unit 31 can transmit various information, such as information input by the user U using the input unit 13, various information detected by sensors 21-28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 determined by the positioning unit 14, to the information processing device 100 via the communication unit 11.

[0055] (Receiver 32) The receiving unit 32 can receive various types of information provided by the information processing device 100, as well as requests for various types of information from the information processing device 100, via the communication unit 11.

[0056] (Processing 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output and display various information transmitted by the transmission unit 31 and various information received from the information processing device 100 by the reception unit 32 to the display unit 12.

[0057] (Storage unit 40) The storage unit 40 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDD (Hard Disk Drive), SSD (Solid State Drive), and optical discs. Various programs and various data are stored in this storage unit 40.

[0058] [4. Example of Information Processing Device Configuration] Next, the configuration of the information processing device 100 according to the embodiment will be described using Figure 10. Figure 10 is a diagram showing an example of the configuration of the information processing device according to the embodiment. As shown in Figure 10, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0059] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N (see Figure 8) by wire or wireless connection.

[0060] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDDs, SSDs, and optical discs. As shown in Figure 10, the storage unit 120 has a user information storage unit 121 and a user customization information storage unit 122.

[0061] (User information storage unit 121) The user information storage unit 121 stores user information about user U. For example, the user information storage unit 121 stores various information such as user U's attributes. Figure 11 is a diagram showing an example of a user information storage unit according to an embodiment. In the example shown in Figure 11, the user information storage unit 121 has items such as "User ID (Identifier)", "Age", "Gender", "Home", "Workplace", and "Interests".

[0062] "User ID" indicates identification information for identifying user U. Note that "User ID" may be user U's contact information (telephone number, email address, etc.) or identification information for identifying user U's terminal device 10.

[0063] Furthermore, "Age" indicates the age of User U, identified by the User ID. Note that "Age" may be information indicating User U's specific age (e.g., 35 years old), or information indicating User U's age group (e.g., 30s), or "Age" may be information indicating User U's date of birth, or information indicating User U's generation (e.g., born in the 1980s). Furthermore, "Gender" indicates the gender of User U, identified by the User ID.

[0064] Furthermore, "Home" indicates the location information of user U's home, which is identified by the user ID. In the example shown in Figure 11, "Home" is represented by an abstract code such as "LC11," but it could also be latitude and longitude information, etc. Also, for example, "Home" could be a regional name or address.

[0065] Furthermore, "Workplace" indicates the location information of the workplace (or school in the case of a student) of user U, identified by the user ID. In the example shown in Figure 11, "Workplace" is illustrated with an abstract code such as "LC12," but it may also be latitude and longitude information, etc. Also, for example, "Workplace" may be a regional name or address.

[0066] Furthermore, "Interests" indicate the interests of User U, identified by the User ID. In other words, "Interests" indicate the subjects of high interest to User U, identified by the User ID. For example, "Interests" may be search queries (keywords) entered into a search engine by User U. In the example shown in Figure 11, one "Interest" is shown for each User U, but there may be multiple.

[0067] For example, in the example shown in Figure 11, user U, identified by user ID "U1", is in their 20s and is male. Also, for example, user U, identified by user ID "U1", has their home address at "LC11". Furthermore, for example, user U, identified by user ID "U1", has their workplace at "LC12". Finally, for example, user U, identified by user ID "U1", is interested in "sports".

[0068] In the example shown in Figure 11, abstract values ​​such as "U1," "LC11," and "LC12" are used to illustrate the information, but it is assumed that "U1," "LC11," and "LC12" actually store specific strings, numbers, or other information. In the following diagrams relating to other information, abstract values ​​may also be used to illustrate the information.

[0069] The user information storage unit 121 is not limited to the above and may store various types of information depending on the purpose. For example, the user information storage unit 121 may store various types of information about user U's terminal device 11. In addition, the user information storage unit 121 may store information about user U's demographic, psychographic, geographic, and behavioral attributes.

[0070] (User-Decorated Information Storage Unit 122) The user decoration information storage unit 122 stores decoration information related to user U. For example, the user decoration information storage unit 122 stores various information such as user U's face image and decoration information. Figure 12 is a diagram showing an example of the user decoration information storage unit according to the embodiment. In the example shown in Figure 12, the user decoration information storage unit 122 has items such as "User ID (Identifier)", "Face Image", and "Decoration Information".

[0071] "User ID" indicates identification information for identifying user U. Note that "User ID" may be user U's contact information (e.g., email address) or identification information for identifying user U's terminal device 10.

[0072] Furthermore, the "face image" refers to the face image of user U, identified by the user ID. The "face image" is, for example, a face image of user U sent by user U.

[0073] Furthermore, "decoration information" includes information such as the facial image of the decorator selected by user U, the object of decoration, and the application method of the decoration information indicated by the decoration method.

[0074] For example, in the example shown in Figure 12, the face image of user U, identified by user ID "U1", is "Face Image #1", and the decoration information of user U, identified by user ID "U1", is "Decoration Information #1".

[0075] In the example shown in Figure 12, abstract information such as "Face Image #1" is stored in "Face Image" and "Decoration Information #1" is stored in "Decoration Information." However, in reality, image data, the object to be decorated, and the application method of the decoration information indicated by the decoration method are stored. In addition, "Face Image" may store, for example, the URL where the image data is located or the file path name indicating the storage location.

[0076] (Control unit 130) Returning to Figure 10, let's continue the explanation. The control unit 130 is a controller, and is realized by various programs (corresponding to an example of an information processing program) stored in the internal memory of the information processing device 100, such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array), executing them using a memory area such as RAM as the working area. In the example shown in Figure 10, the control unit 130 has a determination unit 131, an acquisition unit 132, an application unit 133, and a provision unit 134.

[0077] (Decision Section 131) The determination unit 131 determines the object to be decorated. Specifically, the determination unit 131 compares the color information and / or positional information within the face of the makeup parts cut out from a predetermined face image and the object to be decorated extracted from the user U's face image, thereby determining the object to be decorated in the user U's face image that corresponds to the makeup parts.

[0078] Furthermore, the determination unit 131 may determine the part of the user U's face image that corresponds to the makeup part by comparing skin tone information in a predetermined area related to the makeup part with skin tone information in a predetermined area related to the part to be decorated.

[0079] Furthermore, the determination unit 131 may determine the parts of the user U's face image that correspond to the makeup parts by comparing skin tones, for example, using skin tones related to HSV as skin color information.

[0080] Furthermore, the determination unit 131 may determine the position of the cheeks in the user U's face image and determine the cheek area to be decorated that corresponds to the cheek parts in the predetermined face image, based on the facial contour extracted from the user U's face image, a predetermined part balance based on the positions of the eyebrows, lips, and eyelid parts to be decorated, and the skin color.

[0081] Furthermore, the determination unit 131 may, for example, use a model that has learned the positions of makeup parts using a large number of face images annotated with the positions of the makeup parts, and extract each part from the face images, etc. In addition, the determination unit 131 may cut out a predetermined number of makeup parts and determine multiple decorative objects, without relying on user operation.

[0082] The determination unit 131 determines the application method of the decorative information indicated by the decorative style. Specifically, the determination unit 131 adjusts an offset value obtained by subtracting the average skin color of the eyebrow parts extracted from the face image from the average skin color of the eyebrows of user U by multiplying this offset value by the result of dividing the color of the eyebrow parts by the average skin color of the eyebrow parts, and then adds the adjusted offset value to the color of the eyebrow parts to determine the color-corrected eyebrow parts as the application style.

[0083] Furthermore, the determination unit 131 adjusts an offset value obtained by subtracting the average skin color of the entire face extracted from the face image from the average skin color of the entire face of user U, by multiplying this offset value by the result of dividing the color of the cheek parts by the average skin color of the entire face in the face image, and then adds the adjusted offset value to the color of the cheek parts to determine the color-corrected cheek parts as the applicable configuration.

[0084] Furthermore, the determination unit 131 may correct the color of the decorative information by multiplying the application method of the decorative information indicated by the decorative configuration by a predetermined adjustment value.

[0085] (Acquisition part 132) The acquisition unit 132 acquires user information about user U via the communication unit 110. For example, the acquisition unit 132 acquires identification information (such as user ID), location information, and attribute information of user U from user U's terminal device 10. The acquisition unit 132 may also acquire identification information and attribute information of user U when user U is registered. The acquisition unit 132 then registers the user information in the user information storage unit 121 of the storage unit 120.

[0086] Furthermore, the acquisition unit 132 acquires a facial image of user U. Specifically, the acquisition unit 132 acquires a facial image of user U from user U's terminal device 10 via the communication unit 110.

[0087] Furthermore, the acquisition unit 132 acquires information on the facial image of a person who has been given a predetermined decoration. Specifically, the acquisition unit 132 acquires information on the facial image of a person who has been given a predetermined decoration, selected by user U, from user U's terminal device 10 via the communication unit 110.

[0088] Furthermore, the acquisition unit 132 acquires the object to be decorated. Specifically, the acquisition unit 132 acquires the object to be decorated, which has been determined by the determination unit 131 from a predetermined face image selected by the user U, via the communication unit 110.

[0089] Furthermore, the acquisition unit 132 acquires decoration information. Specifically, the acquisition unit 132 acquires decoration information applied to user U via the communication unit 110. Then, the acquisition unit 132 registers the decoration information applied to user U in the user decoration information storage unit 122 of the storage unit 120.

[0090] Furthermore, the acquisition unit 132 acquires search queries entered by user U. For example, when user U enters a search query into a search engine or the like and performs a keyword search, the acquisition unit 132 acquires the search query via the communication unit 110. In other words, the acquisition unit 132 acquires keywords entered by user U into the search box of a search engine, website, or app via the communication unit 110.

[0091] (Applicable part 133) The application unit 133 applies the decorative information indicated by the decorative pattern to the user U's face image. Specifically, the application unit 133 places the makeup parts specified by user U on a predetermined face image, which have been cut out from the predetermined face image and color-corrected, onto a predetermined position on user U's face image.

[0092] Furthermore, the application unit 133 may cut out a makeup part from a predetermined face image, apply color correction to it, place it on a predetermined area of ​​the user U's face image, and then use a known edge blending technique to blend the edges of the makeup part with the user U's skin.

[0093] (Provider 134) The providing unit 134 provides user U with a facial image to which the decorative information indicated by the decorative pattern has been applied. Specifically, the providing unit 134 transmits to user U's terminal device 10n, via the communication unit 110, an image to which the predetermined decorative pattern has been applied to user U's facial image in the application pattern determined by the determination unit 131.

[0094] [5. Processing Procedure] Next, the processing procedure by the information processing device 100 according to the embodiment will be described using Figure 13. Figure 13 is a flowchart of the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 130 of the information processing device 100.

[0095] As shown in Figure 13, the acquisition unit 132 of the information processing device 100 acquires a facial image of user U from user U's terminal device 10 via the communication unit 110 (step S101).

[0096] Next, the acquisition unit 132 of the information processing device 100 acquires information about the face image of the decorator selected by user U via the communication unit 110 (step S102).

[0097] Next, the determination unit 131 of the information processing device 100 determines the object to be decorated (step S103).

[0098] Next, the determination unit 131 of the information processing device 100 determines the application method of the decorative information indicated by the decorative pattern (step S104).

[0099] Next, the application unit 133 of the information processing device 100 applies the decorative information indicated by the decorative pattern to the user U's face image (step S105).

[0100] Next, the information processing device 100's provisioning unit 134 provides the user U's terminal device 10, via the communication unit 110, with the decorative information indicated by the decorative pattern applied to the user U's face image (step S106).

[0101] [6. Variant Example] The terminal device 10 and information processing device 100 described above may be implemented in various different forms other than those described in the above embodiment. Therefore, modifications of the embodiment will be described below.

[0102] In the above embodiment, some or all of the processing performed by the information processing device 100 may actually be performed by the terminal device 10. For example, the processing may be completed in a standalone manner (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to be equipped with the functions of the information processing device 100 in the above embodiment. Furthermore, in the above embodiment, since the terminal device 10 is in cooperation with the information processing device 100, from the perspective of the user U, it appears as if the processing of the information processing device 100 is also being performed by the terminal device 10. In other words, from another perspective, it can be said that the terminal device 10 is equipped with the information processing device 100.

[0103] For example, the terminal device 10 may execute the processing of the information processing device 100 in the above embodiment as processing on an installed application. In other words, the processing of the information processing device 100 in the above embodiment may be processing on an application installed on the terminal device 10.

[0104] [7. Effects] As described above, the information processing device (terminal device 10 and information processing device 100) according to the present invention includes a determination unit 131 that determines the application method when applying a predetermined decoration in the face image of a decorator to the face image of a user, based on a comparison result between the face image of the decorator and the face image of the user, and an application unit 133 that applies the predetermined decoration to the face image of the user in the application method determined by the determination unit 131.

[0105] The information processing device according to the present invention further includes an acquisition unit 132 that acquires a facial image of the user and also acquires a facial image selected by the user from among the facial images of the decorator.

[0106] The determination unit 131 determines the application method when applying a predetermined decoration from the decorator's face image to the user's face image, based on a comparison between the color in the decorator's face image and the color in the user's face image.

[0107] The determination unit 131 determines the application method when applying a predetermined decoration from the decorator's face image to the user's face image, based on the difference between the color information in the decorator's face image and the color information in the user's face image.

[0108] The color in the face image of the decorator refers to the color in a predetermined decorative area of ​​the decorator's face image, and the color in the face image of the user refers to the color in a predetermined decorative area of ​​the user's face image.

[0109] The color in the face image of the person who applied the decoration is the skin tone in a predetermined decorative area of ​​the person who applied the decoration, and the color in the face image of the user is the skin tone in a predetermined decorative area of ​​the user's face image.

[0110] The system further includes a providing unit 134 that provides the user with a facial image of the user to which the predetermined decoration has been applied in the application mode determined by the determining unit 131.

[0111] Through any or a combination of the above-described processes, the information processing device according to the present invention can further improve usability for users who refer to the makeup of others.

[0112] [8. Hardware Configuration] Furthermore, the terminal device 10 and information processing device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 14. The following explanation will use the information processing device 100 as an example. Figure 14 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0113] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0114] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0115] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0116] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0117] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0118] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0119] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0120] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0121] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, the arithmetic unit 1030 of computer 1000 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, the arithmetic unit 1030 of computer 1000 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0122] [9. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0123] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0124] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0125] For example, the information processing device 100 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via APIs (Application Programming Interfaces) or network computing, allowing for flexible configuration changes.

[0126] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0127] Furthermore, the terms "section," "module," and "unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0128] 1. Information Processing System 10 Terminal devices 11 Communications Department 12 Display section 13 Input section 14 Positioning Unit 20 Sensor section 30 Control Unit 40 Storage section 100 Information Processing Devices 110 Communications Department 120 Storage section 121 User Information Storage Unit 122 User Customization Information Storage Unit 130 Control Unit 131 Decision Section 132 Acquisition Department 133 Application section 134 Provision Department

Claims

1. A determination unit that determines the application method when applying a predetermined decoration to a user's face image in a region of the decorator's face image that includes a decorative object with a predetermined decoration applied to it, A providing unit provides the user's face image with the predetermined decoration applied in the application mode determined by the determining unit, Equipped with, The determination unit corrects the color information of the predetermined decoration in the area of ​​the decorator's face image according to the difference between the color information in the area of ​​the user's face image and the color information in the area of ​​the decorator's face image, and determines the application method. An information processing device characterized by the following:

2. Acquisition unit: Acquires the user's face image and also acquires the face image selected by the user from among the face images of the decorator. The information processing apparatus according to claim 1, further comprising:

3. The application unit applies the predetermined decorations included in the region, extracted from the decorator's face image, to the user's face image in the application mode determined by the determination unit. The information processing device according to claim 2, further comprising the features described above.

4. A method of information processing performed by a computer, A determination step of determining the application method when applying a predetermined decoration to a user's face image in a region of the decorator's face image that includes a decorative object with a predetermined decoration applied to it, A provisioning step in which the predetermined decoration is applied to the user's face image and provided in the application mode determined in the aforementioned determination step, Includes, The determination step involves correcting the color information in the area of ​​the predetermined decoration in the decorator's face image according to the difference between the color information in the area of ​​the user's face image and the color information in the area of ​​the decorator's face image, and then determining the application method. An information processing method characterized by the following:

5. A determination procedure for determining the application method when applying a predetermined decoration to a user's face image in a region of the decorator's face image that includes a decorative object with a predetermined decoration applied to it, A provisioning procedure which involves applying the predetermined decoration to the user's face image and providing it in the application mode determined by the aforementioned determination procedure, Have the computer run it, The aforementioned determination procedure corrects the color information in the area of ​​the predetermined decoration in the decorator's face image according to the difference between the color information in the area of ​​the user's face image and the color information in the area of ​​the decorator's face image, and determines the application mode. An information processing program characterized by the following features.

Images