System

Abstract

A system is provided.SOLUTION: Camera means for a user to photograph himself / herself, image analysis means for analyzing a photographed image, proposal means for proposing an optimal makeup method to the user on the basis of an analysis result, display means for displaying a proposed makeup procedure in a moving image, a still image, or a text format, management means for managing cosmetic information possessed by the user, determination means for determining whether a cosmetic necessary for the proposed makeup procedure has been possessed or needs to be purchased, purchase means for linking the cosmetic that needs to be purchased to a shopping site, and advice means for providing additional advice after photographing the user; Selection means able to select an advice from a professional cosmetic expert.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern consumers use various information sources to learn how to apply makeup, but often find it difficult to find a makeup method that suits them. It is also difficult to keep track of the cosmetics they own and use them appropriately. Furthermore, there is a need for a system that allows them to easily purchase new cosmetics and a method to effectively receive feedback after applying makeup. There is a need to provide a system that solves these issues and allows users to efficiently and effectively find and apply the makeup method that is best for them. [Means for solving the problem]

[0005] The present invention provides a system that includes a camera means for allowing a user to photograph themselves, an image analysis means for analyzing the photographed image, a suggestion means for suggesting the optimal makeup regimen for the user based on the analysis results, a display means for displaying the suggested makeup regimen in video, still image, or text format, a management means for managing information about cosmetics owned by the user, a determination means for determining whether the cosmetics required for the suggested makeup regimen are already owned or need to be purchased, a purchase means for linking the cosmetics required to a shopping site, an advice means for providing additional advice after the user's photograph is taken, and a selection means for selecting advice from professional makeup experts. This allows users to easily find the makeup regimen that best suits them, appropriately use the cosmetics they own, and easily purchase the necessary cosmetics. Furthermore, they can effectively receive feedback after applying their makeup.

[0006] "User" refers to an individual who uses the system to photograph their face and receive suggestions on how to apply makeup.

[0007] "Camera means" refers to equipment or functions for photographing the user's face.

[0008] "Image analysis means" refers to algorithms or software that process captured image data to analyze facial features and environmental data.

[0009] "Proposal means" refers to a function or system that calculates and presents the optimal makeup method to the user based on the analysis results.

[0010] "Display means" refers to a device or function that provides the user with suggested makeup procedures in video, still image, or text format.

[0011] "Management means" refers to a system or database that records and manages information about cosmetics owned by users.

[0012] "Determination means" refers to a function or system that determines whether the cosmetics required for the proposed makeup procedure are already in the user's possession or need to be purchased.

[0013] "Purchase methods" refers to functions and systems that provide links to shopping sites so that users can easily purchase the cosmetics they need.

[0014] "Advice means" refers to a function or system that evaluates the user's condition after makeup application and provides additional advice.

[0015] "Selection method" refers to a function or system that allows users to choose advice from a professional makeup expert instead of advice from AI. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that allows a user to photograph their own face and suggests the optimal makeup application method. This system includes a camera, an image analysis unit, a suggestion unit, a display unit, a management unit, a judgment unit, a purchasing unit, an advice unit, and a selection unit. The specific operation of each unit and the program processing for that purpose are explained in natural language below.

[0038] 1. Camera equipment and photography

[0039] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[0040] 2. Image Analysis

[0041] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup methods that are suitable for the user.

[0042] 3. Makeup tips

[0043] The server uses the analyzed data to calculate the optimal makeup application, taking into account the user's current cosmetics and personal information. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0044] 4. Display of makeup procedures

[0045] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[0046] 5. Cosmetics management and purchase suggestions

[0047] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[0048] 6. Post-makeup evaluation and advice

[0049] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[0050] 7. Professional makeup expert advice

[0051] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[0052] The above is the specific operation of each means in this system and the processing contents of the program.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] (User) activates the smartphone camera and takes a picture of their own face. By opening the app and pressing the "take a picture" button, the camera is activated and the user takes a picture of their own face.

[0056] Step 2:

[0057] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[0058] Step 3:

[0059] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[0060] Step 4:

[0061] The server checks the user's cosmetics possession status, accesses the user's cosmetics database, and retrieves the cosmetics possessed by the user.

[0062] Step 5:

[0063] The server will suggest the optimal makeup application. Based on facial feature data, environmental data, and information on the cosmetics you own, the AI ​​algorithm will generate optimal makeup suggestions. For example, if the summer sun is strong, the server will suggest makeup that emphasizes UV protection.

[0064] Step 6:

[0065] The server generates the proposed makeup steps in video, still images, and text format. It records the makeup steps as a video, captures important steps as still images, and summarizes the detailed steps in text.

[0066] Step 7:

[0067] The server sends the generated makeup instructions to the device. The generated videos, still images, and text data are sent to the device and presented in a visually easy-to-understand format.

[0068] Step 8:

[0069] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[0070] Step 9:

[0071] The device checks the user's cosmetic information, checks whether the user already has the cosmetics required for the proposed makeup routine, and adds them to the list if they are missing.

[0072] Step 10:

[0073] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[0074] Step 11:

[0075] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[0076] Step 12:

[0077] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[0078] Step 13:

[0079] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[0080] Step 14:

[0081] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[0082] Step 15:

[0083] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[0084] Step 16:

[0085] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[0086] Step 17:

[0087] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[0088] Step 18:

[0089] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[0090] Step 19:

[0091] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[0092] Step 20:

[0093] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conventional cosmetics recommendation systems lack the ability to suggest optimal makeup applications based on the user's facial features and environmental data. Furthermore, there is no way for users to verify the consistency of the recommended cosmetics with the cosmetics they currently own, making the process of purchasing the necessary cosmetics cumbersome. Furthermore, the lack of a mechanism for providing evaluations of the makeup application or additional advice makes it difficult for users to improve their makeup techniques.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes a photographing device means for allowing the user to photograph themselves, an image analysis device means for analyzing the photographed image, a suggestion device means for suggesting the optimal makeup regimen for the user based on the analysis results, a display device means for displaying the suggested makeup regimen in video, still image, and text format, a management function means for managing information on cosmetics owned by the user, a determination device means for determining whether the cosmetics required for the suggested makeup regimen are already owned or need to be purchased, a purchase device means for linking cosmetics that need to be purchased to an electronic shopping site, an advice device means for providing additional advice to the user after photographing, and a selection device means for selecting advice from experts. This makes it possible to suggest the optimal makeup regimen based on the user's characteristics and environmental data, simplifies the process of checking the compatibility of the necessary cosmetics and purchasing them, and enables the provision of evaluations after makeup application and additional advice.

[0099] The "photographing device means" is a function including a device that allows the user to take a picture of their own face.

[0100] The "image analysis device means" is a function that analyzes the captured image and extracts the user's facial features and environmental data.

[0101] The "suggestion device means" is a function that provides the user with the most suitable makeup method based on the analysis results.

[0102] The "display device" is a function that displays the proposed makeup procedure to the user in the form of video, still images, or text.

[0103] The "management function means" is a function for managing the cosmetic information owned by the user.

[0104] The "determination device means" is a function that determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased.

[0105] The "purchase device means" is a function that links cosmetics that need to be purchased to an electronic commerce site.

[0106] The "advice device means" is a function that provides additional advice to the user after taking a photo.

[0107] The "selection device means" is a function that allows the user to select advice from experts.

[0108] This invention is a system that allows a user to take a picture of their own face and suggests the best makeup application method. The system includes the following means:

[0109] photographing device means

[0110] Image analysis device means

[0111] Proposed device means

[0112] display means

[0113] management function means

[0114] Judgment device means

[0115] Purchasing Device Means

[0116] Advice Device Means

[0117] Selector means

[0118] Users take a photo of their face using the camera on their smartphone or tablet device. To do this, they press the "take a photo" button through the app, which activates the camera and takes a photo from an appropriate position. The device then sends the captured image data to the server. The server then uses an algorithm to analyze the received image data. Specifically, the server analyzes the user's facial features (e.g., face shape, skin tone, eye position) and environmental data (e.g., season, weather, temperature). Based on this analytical data, the system prepares to suggest makeup methods that are suitable for the user.

[0119] The server then uses the analyzed data to calculate the optimal makeup application, taking into account the user's current makeup habits and personal characteristics. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0120] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as "First, apply moisturizing cream, then apply liquid foundation," as well as playing back the specific makeup steps in video format.

[0121] The terminal manages information about cosmetics owned by the user. The determination device checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the terminal adds them to a list and provides a link to a shopping site, allowing the user to purchase the required cosmetics with one click.

[0122] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis, the app evaluates the makeup and generates additional advice. For example, it might provide specific feedback like, "Your eyeliner is a little too thick, so if you make it a little thinner it will balance things out better."

[0123] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. In this way, the user can receive more professional advice.

[0124] For example, the following prompts can be fed into a generative AI model to get further advice:

[0125] "I'm a woman in my 20s. I'd like to know how to apply makeup during the day in the summer. My skin tends to be dry, so I want to make my body look slimmer. The cosmetics I currently use are sunscreen, liquid foundation, powder foundation, brush-on bronzer, mascara, eyeliner, and lip tint."

[0126] This invention allows users to easily find the best makeup method for their face, and efficiently select and purchase the necessary cosmetics. In addition, users can improve their makeup techniques by receiving further advice through evaluations after makeup application.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Program processing flow

[0129] Step 1: Capture a photo of the user's face

[0130] Input: Smartphone or tablet camera

[0131] Output: Image data of the captured face

[0132] Specific behavior:

[0133] 1. The user launches the app.

[0134] 2. The user presses the "take a photo" button.

[0135] 3. The device starts the camera and the user takes a picture of their face in an appropriate position.

[0136] 4. The device generates image data of the captured face.

[0137] Step 2: Sending image data

[0138] Input: Image data of a photographed face

[0139] Output: Image data sent to the server

[0140] Specific behavior:

[0141] 1. The device sends the captured image data to the server.

[0142] 2. The server receives the image data through a receiving protocol.

[0143] Step 3: Image analysis

[0144] Input: Image data sent to the server

[0145] Output: Facial feature data, environmental data

[0146] Specific behavior:

[0147] 1. The server applies an image analysis algorithm (e.g., a deep learning model).

[0148] 2. Analyze facial features (e.g., face shape, skin tone, eye position).

[0149] 3. Reference environmental data (e.g., season, weather, temperature) to complement the analysis results.

[0150] Step 4: Makeup tips

[0151] Input: Facial feature data and environmental data

[0152] Output: Data on optimal makeup application

[0153] Specific behavior:

[0154] 1. The server uses the analysis data to calculate the optimal makeup application method in comparison with the cosmetic information it has.

[0155] 2. Consider user characteristics, environmental data, trend information, past makeup patterns, etc.

[0156] 3. Generate data on optimal makeup application methods.

[0157] Step 5: Send and view your cosmetic procedure

[0158] Input: Data on optimal makeup application

[0159] Output: Makeup procedure (video, still image, text)

[0160] Specific behavior:

[0161] 1. The server sends the generated makeup procedure to the terminal.

[0162] 2. The terminal analyzes the received instructions and displays them on the user interface.

[0163] 3. Provide makeup instructions in the form of video playback, still image display, and text display.

[0164] Step 6: Cosmetics management and purchasing suggestions

[0165] Input: Cosmetic information owned by the user, suggested makeup procedures

[0166] Output: List of cosmetics to purchase, purchase link

[0167] Specific behavior:

[0168] 1. The device references the user's cosmetics database and matches the cosmetics required for the proposed makeup procedure.

[0169] 2. If you don't have any necessary cosmetics, make a list of them.

[0170] 3. Provide a link to an e-commerce site for the cosmetics you need to purchase.

[0171] Step 7: Post-makeup evaluation and advice

[0172] Input: Image data of face after makeup

[0173] Output: Makeup evaluation data, additional advice

[0174] Specific behavior:

[0175] 1. The user takes another photo of their face after applying makeup.

[0176] 2. The device sends the new image to the server.

[0177] 3. The server analyzes the image again and generates a makeup evaluation.

[0178] 4. Generate additional advice and send it to the device.

[0179] Step 8: Expert advice

[0180] Input: User selection, facial image data

[0181] Output: Expert advice

[0182] Specific behavior:

[0183] 1. The user selects advice from a professional makeup expert.

[0184] 2. The device notifies the server of its choice.

[0185] 3. The server sends the user's image data to the expert.

[0186] 4. The expert creates customized advice and sends it to the device via the server.

[0187] 5. Users receive expert advice.

[0188] (Application example 1)

[0189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] Conventional makeup advice systems are primarily designed for use on digital devices and lack the ability to provide real-time product recommendations and inventory checks in a physical store environment. Furthermore, users often have to put in significant effort to find the perfect cosmetics in-store, and sometimes certain products are out of stock, creating an inconvenient shopping experience. A new system is needed to address these issues.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0192] In this invention, the server includes a photographing means for the user to photograph themselves, an analysis means for analyzing the photographed image, a suggestion means for suggesting the optimal makeup method to the user based on the analysis results, a display means for displaying the suggested makeup procedure in video, still image, or text format, a management means for managing information on items owned by the user, a determination means for determining whether the user already owns or needs to purchase items required for the suggested makeup procedure, a purchase means for linking items that need to be purchased to a shopping site, an advice means for providing additional advice to the user after photographing, an inventory confirmation and guidance means for checking inventory in the store and guiding the user to the location of items, and a feedback means for providing interactive feedback linked to the real world. This allows the user to find cosmetics in the store in real time and receive detailed suggestions on the optimal makeup method.

[0193] The "photographing means" is a device that allows the user to photograph their own face.

[0194] The "analysis means" is a device or software that analyzes the captured image data and extracts the user's facial features and environmental data.

[0195] The "suggestion means" is a device or software that provides the user with the most suitable makeup method based on the data obtained by the analysis means.

[0196] The "display means" is a device or software that presents the suggested makeup procedure to the user in the form of video, still images, or text.

[0197] The "management means" refers to a device or software that centrally manages information about items (cosmetics) owned by the user.

[0198] The "determination means" is a device or software that determines whether the user already possesses the items necessary for the proposed cosmetic procedure or whether new purchases are required.

[0199] "Purchase Instrument" refers to a device or software that allows the user to purchase the suggested items as needed through a link to a shopping site.

[0200] An "advice means" is a device or software that provides additional makeup advice to a user after the user has taken a selfie.

[0201] The "inventory checking and guidance means" is a device or software that checks the inventory in the store and guides the user to the location of the required item.

[0202] A "feedback tool" is a device or software that provides real-time interactive feedback and evaluations to users after they have actually used a product.

[0203] This invention provides a system that allows users to use smart glasses in a physical store and receive recommendations on optimal makeup application methods. The system includes a photographing unit, an analysis unit, a recommendation unit, a display unit, a management unit, a determination unit, a purchasing unit, an advice unit, an inventory confirmation and guidance unit, and a feedback unit.

[0204] First, the user puts on the smart glasses in a physical store and takes a picture of their face. The image captured by this photographing means is sent to the analyzing means, which then analyzes facial features and environmental data using machine learning models such as TensorFlow. This analyzed data is then used by the suggesting means to suggest the most suitable makeup application to the user.

[0205] The suggestion unit suggests the most suitable cosmetics based on the user's facial features and environmental data. For example, it may recommend the most suitable foundation or eyeshadow depending on the user's skin tone, the season, and the lighting conditions in the store. The suggestion is displayed on the smart glasses' display via the display unit in the form of video, still images, or text.

[0206] The management means manages information about cosmetics owned by the user through the smart glasses and related applications. The determination means determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased. If necessary, the purchase means provides a link to a related online shopping site to facilitate easy purchase.

[0207] Furthermore, the inventory check and guidance module checks the inventory in the physical store and guides the user to the location of the necessary items. This information is linked to the store's real-time inventory database. After the user actually applies the makeup, the feedback module provides real-time interactive evaluation and additional makeup advice.

[0208] As a concrete example, imagine a user puts on smart glasses in a physical store and scans their face. The system analyzes the user's face and suggests the best foundation and eyeshadow based on their skin tone. These suggestions are played back as a video on the glasses' display, along with detailed instructions on how to use the product. The system also displays the shelf location, allowing users to easily find the product without getting lost. Furthermore, with one click, users can access an online shopping site for the product they need, and if it's not in stock at the store, they can have it ordered from a nearby store or online store.

[0209] The system allows users to receive detailed real-time suggestions on how to best apply makeup through the smart glasses, and also significantly improves the in-store shopping experience.

[0210] Examples of prompts powered by generative AI models include:

[0211] "Develop an app that takes a picture of the user's face with a camera in smart glasses and suggests the best makeup look for them, taking into account their facial features, skin tone, eye position, and in-store environmental data (lighting, climate, temperature). It also checks the stock availability of cosmetics used in the suggested makeup look and provides real-time feedback to the user."

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1:

[0214] The user puts on the smart glasses in a physical store and has their face photographed.

[0215] Input: Image data of the user's face captured by the camera built into the smart glasses.

[0216] Action: The capture device activates the camera and prompts the user to position their face appropriately.

[0217] Output: The captured image data is stored in the smart glasses.

[0218] Step 2:

[0219] The captured image is sent to a server for analysis.

[0220] Input: Image data of the user's face.

[0221] How it works: Image data is sent from the smart glasses to a server, and the analysis means analyzes the image using a machine learning model such as TensorFlow.

[0222] Output: Facial feature data and environmental data (skin tone, eye position, lighting conditions, etc.) are generated.

[0223] Step 3:

[0224] Based on the analysis results, the suggestion method calculates the optimal makeup method.

[0225] Input: Facial feature data and environmental data.

[0226] How it works: The proposed method uses a generative AI model to calculate the optimal makeup routine for the user.

[0227] Output: Makeup application suggestion data (video, still images, text format) is generated.

[0228] Step 4:

[0229] The generated makeup procedure is displayed on the smart glasses.

[0230] Input: Makeup method suggestion data.

[0231] Operation: The display means displays makeup instructions in video, still images, and text format on the display of the smart glasses.

[0232] Output: A state in which the user can visually confirm the makeup procedure.

[0233] Step 5:

[0234] The cosmetic information possessed by the user is confirmed by the management means.

[0235] Input: A list of cosmetics owned by the user.

[0236] Operation: The control unit compares the list of cosmetics on hand with the cosmetics required for the proposed makeup procedure.

[0237] Output: The result of determining whether the item is already in possession or needs to be purchased.

[0238] Step 6:

[0239] Link missing cosmetics to purchasing options.

[0240] Input: A list of cosmetics that need to be purchased according to the determination means.

[0241] Operation: The purchasing instrument generates and provides a link to an online shopping site to the user.

[0242] Output: Purchase link information.

[0243] Step 7:

[0244] Check inventory in store and provide location of items.

[0245] Input: List of cosmetics you need to buy.

[0246] Operation: The inventory check and guidance tool checks the store's real-time inventory database and guides the user to shelf locations.

[0247] Output: Inventory check results and shelf location information.

[0248] Step 8:

[0249] After the user actually applies the makeup, the user is given an evaluation and additional advice through a feedback means.

[0250] Input: Image data of the user's face after makeup application.

[0251] How it works: The feedback mechanism analyzes the image again and generates a makeup rating and additional recommendations.

[0252] Output: Evaluation results and further advice.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] The present invention combines a system that allows a user to take a picture of their own face and suggests the optimal makeup application method with an emotion recognition means that recognizes the user's emotions. This system includes a camera means, an image analysis means, a suggestion means, a display means, a management means, a judgment means, a purchasing means, an advice means, a selection means, and an emotion recognition means. The specific operation of each means and the program processing for that purpose are explained in natural language below.

[0255] 1. Camera equipment and photography

[0256] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[0257] 2. Image Analysis

[0258] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup techniques that are suitable for the user.

[0259] 3. Emotion recognition

[0260] Furthermore, the image analysis means acquires facial expression data of the user, which is then analyzed by the emotion recognition means. The emotion recognition means identifies the current emotion (e.g., joy, sadness, anger, etc.) from the user's facial expression.

[0261] 4. Makeup tips

[0262] Based on the analyzed data, the server compares it with the information on the cosmetics the user owns and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also trend information, past makeup patterns, and emotional data obtained through emotion recognition. For example, if the server recognizes that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0263] 5. Display of makeup procedures

[0264] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[0265] 6. Cosmetics management and purchase suggestions

[0266] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[0267] 7. Post-makeup evaluation and advice

[0268] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[0269] 8. Professional makeup expert advice

[0270] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[0271] The above is the specific operation of each means in this system and the processing contents of the program.

[0272] The processing flow will be explained below.

[0273] Step 1:

[0274] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user opens the app and presses the "take a picture" button to activate the camera, and then positions themselves appropriately to take a picture.

[0275] Step 2:

[0276] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[0277] Step 3:

[0278] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[0279] Step 4:

[0280] The server checks the user's cosmetics possession status. It accesses the user's cosmetics database and retrieves the cosmetics possessed by the user. This data is taken into account in conjunction with the analysis results.

[0281] Step 5:

[0282] The (server) uses emotion recognition means to analyze the user's emotions. It analyzes the user's facial expressions from the analyzed image data and identifies their current emotional state (e.g., joy, sadness, anger, etc.).

[0283] Step 6:

[0284] The server will then suggest the optimal makeup application based on all analysis results (facial features, environmental data, and emotional data). Using an AI algorithm, it will compare the results with the user's current cosmetics and generate specific makeup steps. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0285] Step 7:

[0286] The server prepares the generated makeup steps in video, still images, and text format and sends them to the device. The server records the makeup steps as video, captures important steps as still images, and summarizes the detailed steps in text.

[0287] Step 8:

[0288] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[0289] Step 9:

[0290] The device checks the cosmetics information it has, checks whether the cosmetics required for the proposed makeup procedure are already in possession, and adds them to the list if they are missing.

[0291] Step 10:

[0292] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[0293] Step 11:

[0294] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[0295] Step 12:

[0296] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[0297] Step 13:

[0298] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[0299] Step 14:

[0300] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[0301] Step 15:

[0302] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[0303] Step 16:

[0304] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[0305] Step 17:

[0306] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[0307] Step 18:

[0308] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[0309] Step 19:

[0310] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[0311] Step 20:

[0312] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[0313] Example 2

[0314] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] Conventional makeup suggestion systems only consider the user's facial features and environmental data, which means they are unable to suggest makeup methods that match the user's current emotional state or mood of the day. Furthermore, there are limited ways to display makeup steps, and they lack a format that is visually easy for users to understand. Furthermore, if the required cosmetics are not available, users must manually search and purchase them, which is a time-consuming process.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0317] In this invention, the server includes an image analysis means for analyzing the captured image, an emotion recognition means for acquiring and analyzing the user's facial expression data by the image analysis means, and a suggestion means for suggesting the most suitable makeup method to the user based on the analysis results. This makes it possible to suggest the most suitable makeup method to the user taking into consideration the user's facial features, environmental data, and the user's emotional state.

[0318] "Imaging means" refers to a device or equipment that allows a user to take a picture of their own face.

[0319] "Image analysis means" refers to a device or method that analyzes captured image data and extracts facial features and other relevant information.

[0320] "Emotion recognition means" refers to a device or method for identifying the emotional state of a user based on facial expression data acquired by image analysis means.

[0321] The "suggestion means" refers to a device or method that suggests the most suitable makeup method to the user based on the analysis results.

[0322] "Display means" refers to a device or method that visually presents the suggested makeup procedure to the user in the form of video, still images, or text.

[0323] "Information management means" refers to a device or method for managing information about cosmetics owned by a user.

[0324] "Determination means" refers to a device or method for confirming whether the cosmetics required for the proposed makeup procedure are already in possession.

[0325] "Purchase method" refers to a method that provides a link to an online retail site to facilitate purchases by users when they are out of the required cosmetics.

[0326] "Advice means" refers to a device or method that allows a user to receive additional advice after taking a photo.

[0327] "Selection means" refers to a device or method that allows a user to select advice from experts.

[0328] The present invention is a system that allows a user to take a picture of their own face and suggests the optimal makeup method. Furthermore, by combining it with an emotion recognition means that recognizes the user's emotions, it is possible to suggest makeup methods that correspond to the user's emotional state. This system includes an imaging means, an image analysis means, an emotion recognition means, a suggestion means, a display means, an information management means, a judgment means, a purchasing means, an advice means, and a selection means.

[0329] Imaging means

[0330] The user activates the camera on their smartphone or tablet device and presses the "take a photo" button to take a picture of their own face, which generates image data.

[0331] Image analysis methods

[0332] The device sends the captured image data to a server. The server uses an algorithm to analyze the received image data. For example, image analysis libraries such as OpenCV or TensorFlow are used to analyze facial features (e.g., facial contours, skin tone, eye position, etc.). Environmental data (e.g., weather, season, temperature, etc.) is also collected and used to analyze the data.

[0333] emotion recognition means

[0334] The image analysis means acquires the user's facial expression data, which is then analyzed by the emotion recognition means. An emotion recognition algorithm is used to identify the user's emotional state (e.g., joy, sadness, anger, etc.), often using an Emotion API or a deep learning model.

[0335] Proposal means

[0336] Based on the analysis results, the server compares the user's cosmetics with their current makeup product information and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also emotional data obtained through emotion recognition. For example, if the server detects that the user looks tired, it will suggest a lighter makeup look that will give a refreshing feeling.

[0337] Display means

[0338] The server prepares the generated makeup instructions in video, still images, and text format and sends them to the device. The device then displays the received information to the user. For example, the video might say, "First, apply moisturizing cream," while simultaneously displaying the same part in text.

[0339] Information management means

[0340] The device manages the cosmetics information of the user, checks the information database of the cosmetics the user owns, and determines whether the user has the necessary items.

[0341] Judgment means

[0342] Check whether the cosmetics required for the proposed makeup procedure are already in your possession and notify the user if any required items are missing.

[0343] Purchase method

[0344] If the required cosmetics are missing, the device will assist the user in the purchase process by providing a link to an online retail site, allowing the user to easily purchase the missing cosmetics.

[0345] Advice tools

[0346] After the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis results, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little too thick. Next time, try drawing it thinner."

[0347] Selection method

[0348] The user can select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server.

[0349] Specific examples and prompts for the generative AI model

[0350] For example, when a user uses this system to get ready in the morning, they follow the steps below.

[0351] 1. The user launches the app and takes a photo of their face.

[0352] 2. The server analyzes facial features and environmental data, and then recognizes emotions from the user's facial expressions.

[0353] 3. If the user feels tired, the server will suggest refreshing makeup using lighter colors.

[0354] 4. Apply makeup while watching the video to see the suggested makeup steps.

[0355] 5. If you are missing the cosmetics you need, proceed with the purchase.

[0356] 6. After your makeup is applied, you will receive evaluation and advice on how to further refine your look.

[0357] Example prompt for a generative AI model:

[0358] "Please explain how a system analyzes a user's facial photo and suggests the best makeup application. Please also explain in detail how it combines emotion recognition and suggests lighter makeup colors if the user wants to feel refreshed."

[0359] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0360] Step 1:

[0361] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user then presses the "take a picture" button through the app, which activates the camera. This inputs image data of the face.

[0362] Step 2:

[0363] The device sends the captured image data to the server. The image data is uploaded, and the server receives it. The input image data is then processed, and image analysis is ready.

[0364] Step 3:

[0365] The server analyzes the received image data. Using a facial recognition algorithm (e.g., OpenCV or TensorFlow), it analyzes facial features (facial contours, skin tone, eye position, etc.) and environmental data (weather information, season, etc.). This analysis outputs the user's facial feature data.

[0366] Step 4:

[0367] The server acquires the user's facial expression data using image analysis means and analyzes it using emotion recognition means. The user's emotions (e.g., joy, sadness, anger, etc.) are identified using the Emotion API and deep learning models. Emotion data is output from the input facial expression data.

[0368] Step 5:

[0369] The server calculates the optimal makeup application method (using AI models such as Keras) based on the analysis results (facial feature data and emotional data). The calculation takes into account the user's features, environmental data, and emotional data. For example, if the user is recognized as tired, a refreshing makeup look using lighter colors will be suggested. The output is suggested data for the optimal makeup application method.

[0370] Step 6:

[0371] The server generates a proposed makeup procedure in the form of video, still images, and text, and sends it to the device. The device then displays the received makeup procedure to the user. For example, a video might say, "First, apply moisturizing cream," while simultaneously displaying that part in text. The input is the proposed data, and the output is the display data of the makeup procedure.

[0372] Step 7:

[0373] The device manages the user's cosmetics information and checks whether the user has the cosmetics necessary for the proposed makeup procedure. It checks the managed cosmetics database and outputs a list of missing cosmetics.

[0374] Step 8:

[0375] The terminal assists the user in purchasing the missing cosmetics. It provides the user with a link to an online retail site, allowing them to easily purchase the necessary cosmetics. The terminal notifies the user by saying, "You are missing this lipstick. You can purchase it here." The missing cosmetics list is input, and the online link data is output.

[0376] Step 9:

[0377] After the user finishes applying makeup, they take a picture of themselves again. The user takes a new image, and the device sends it to the server. The input is the new image data.

[0378] Step 10:

[0379] The server analyzes the image again and generates makeup evaluation and advice based on the new analysis results. For example, it may generate specific feedback such as, "Your eyeliner is a little thick, so next time, try drawing it thinner." The input is the new image data, and the output is the evaluation and advice data.

[0380] Step 11:

[0381] When a user selects advice from a professional makeup expert, the device notifies the server of the selection. The server sends the user's image data to the expert, who then creates customized advice based on the image. The data is then sent to the device via the server. The input is the user's selection information and image data, and the output is the expert's advice.

[0382] (Application example 2)

[0383] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] Conventional makeup suggestion systems suggest optimal makeup methods based on the user's facial features and environmental data. However, these systems do not take into account the user's current emotional state, making it difficult to provide makeup suggestions that match the user's feelings and mood. Furthermore, because the system does not suggest makeup methods that correspond to emotions, it is difficult to increase user satisfaction.

[0385] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the user's face, an analysis means for analyzing the captured image, an emotion recognition means for recognizing emotions, and a suggestion means for suggesting an optimal makeup method to the user based on the analysis results and emotion data. This makes it possible to suggest a makeup method based on the user's emotional state.

[0386] "Photography means" refers to a device or function that allows a user to take pictures or videos of themselves.

[0387] "Analysis means" refers to a device or function for analyzing facial features and environmental information based on captured image and video data.

[0388] The "suggestion means" is a device or function that calculates and suggests the most suitable makeup method for the user based on the analyzed data.

[0389] The "display means" is a device or function for displaying the proposed makeup procedure to the user in the form of video, still images, or text.

[0390] The "management means" refers to a device or function for centrally managing information about cosmetics owned by a user.

[0391] The "determination means" is a device or function that determines whether the cosmetics required for the proposed makeup procedure are in the user's possession or need to be purchased.

[0392] "Purchase means" refers to a device or function that links cosmetics to be purchased to an online purchasing site, allowing users to easily purchase them.

[0393] An "advice means" is a device or function that provides additional advice or evaluation to the user after the user has finished applying makeup.

[0394] The "selection means" is a device or function that allows the user to select and receive advice from an expert.

[0395] "Emotion recognition means" refers to a device or function for analyzing and recognizing the current emotion of the user from their facial expression.

[0396] The present invention is a system that allows users to take a picture of their own face, analyzes the image data, and provides optimal makeup methods, and also adds a function to recognize the user's emotions. This system takes into account the user's facial features, environmental information, and current emotional state to propose more detailed and personalized makeup methods.

[0397] Hardware and software used

[0398] This system uses the following hardware and software:

[0399] Photography method: Use the built-in camera on a smartphone or tablet device or a webcam connected to a computer.

[0400] Analysis method: DeepFace library for analyzing facial features and environmental information.

[0401] Emotion recognizer: An emotion recognition algorithm to recognize user emotions (e.g., a Python emotion recognition library).

[0402] Suggestion method: A makeup suggestion algorithm that runs on the server side. Data is exchanged via REST API.

[0403] Display means: Smartphone or tablet display, and dedicated mobile application.

[0404] Management means: A database (e.g., SQLite) for managing the cosmetic data owned by the user.

[0405] Determination method: An algorithm to determine whether the required cosmetics are in possession.

[0406] Purchase method: A function to link the cosmetics to be purchased to an online shop (e.g., integration with a shopping API).

[0407] Advice tool: An algorithm that analyzes re-photograph data after makeup application and provides additional advice.

[0408] Selection tool: A user interface that allows the selection of expert advice.

[0409] System operation explanation

[0410] Filming method

[0411] Users take a photo of their face using the camera on their smartphone or tablet device by launching the camera app and pressing the "take photo" button.

[0412] Analysis means

[0413] The device sends the captured image data to a server and analyzes it using the DeepFace library to extract facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.).

[0414] emotion recognition means

[0415] The image analysis means acquires facial expression data of the user, and the emotion recognition means analyzes this to identify the user's current emotion (for example, joy, sadness, anger, etc.).

[0416] Proposal means

[0417] The server then uses the analysis data to suggest makeup applications. These suggestions take into account not only the user's facial features and environmental data, but also their emotional data. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0418] Display means

[0419] The server sends the generated makeup instructions in the form of video, still images, and text to the device, which then displays them to the user. The display includes detailed instructions in video, still images, and text.

[0420] Examples of concrete examples and prompts

[0421] Below are some example prompts to be input to the generative AI model:

[0422] Develop a system that takes a picture of a user's face, analyzes the image to recognize emotions, and suggests the best makeup application based on the user's emotions. Follow these steps:

[0423] 1. Activate the camera and take a picture of the user's face.

[0424] 2. Analyze the captured image to identify facial features and emotions (happiness, sadness, anger, etc.).

[0425] 3. Send facial features and emotion data to an external API and receive the optimal makeup application.

[0426] 4. The received makeup method is displayed to the user.

[0427] As described above, the system for implementing this invention is a multifunctional system that proposes the optimal makeup method taking into consideration the user's facial features and emotions. This system allows the user to apply makeup that suits their emotional state, thereby increasing satisfaction.

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1:

[0430] The user activates the camera on their smartphone or tablet and takes a picture of their own face. At this time, the camera is activated by pressing the "shoot" button in the application, and the user takes a picture in an appropriate position. The input is the user's facial image, which is output as image data.

[0431] Step 2:

[0432] The device receives the captured image data and sends it to the server. The input is the image data, which is sent to the server via the network. The output is the image data received by the server.

[0433] Step 3:

[0434] The server analyzes the received image data. For this analysis, it uses the DeepFace library to analyze the user's facial features (face shape, skin tone, eye position, etc.). The input is the received image data, and the output is facial feature data. Specifically, the DeepFace library maps each point on the face and extracts feature data.

[0435] Step 4:

[0436] The server then uses emotion recognition means to recognize the user's emotions from the analyzed feature data. This process uses an emotion recognition algorithm (for example, a Python emotion recognition library). The input is facial feature data, and the output is emotion data. Specifically, it analyzes changes in facial expression (for example, eyebrow movement, mouth corner position) to identify emotions such as joy, sadness, and anger.

[0437] Step 5:

[0438] The server then uses the analyzed data (facial features and emotion data) to suggest the most suitable makeup application method for the user. This proposal is made using a makeup application suggestion algorithm on the server. The input is facial feature data and emotion data, and the output is suggested data for makeup procedures. For example, if the emotion is recognized as "fatigue," the server will suggest a makeup method to make the user's face look brighter.

[0439] Step 6:

[0440] The server prepares the generated makeup steps in video, still image, and text formats and sends them to the terminal. The input is the proposed makeup steps data, and the output is the terminal that receives it. Specifically, the server converts the makeup steps into multiple formats (video file, still image file, text file) and sends them all at once to the terminal.

[0441] Step 7:

[0442] The device then displays the received makeup instructions to the user. The input is the received makeup instructions data, and the output is a visual representation of the makeup instructions that the user can confirm. Specifically, the device application plays the video and highlights important parts with still images and text.

[0443] Step 8:

[0444] The user applies makeup according to the suggested makeup steps, then takes a photo of their face again. The input is the face image after makeup, and the output is the image data. Specifically, the user starts the camera again and presses the "take another photo" button to take a photo of their face.

[0445] Step 9:

[0446] The device sends the face image data after makeup application to the server, which then analyzes it again. The input is the re-photographed image data, and the output is the analysis results. Specifically, the server executes the process described above again to evaluate the face after makeup application.

[0447] Step 10:

[0448] The server generates additional advice based on the reanalysis results and sends it to the terminal. The input is the reanalysis results, and the output is additional advice data. Specific actions include providing specific feedback such as "Draw your eyeliner a little thinner" if the eyeliner is too thick.

[0449] Step 11:

[0450] The terminal displays the received additional advice to the user. The input is the received additional advice data, and the output is an advice display that the user can visually confirm. Specifically, the advice text or a still image is displayed on the terminal screen.

[0451] The above are the specific processing steps for carrying out the present invention. The user can receive personalized makeup suggestions that take into consideration their emotions, which can improve satisfaction.

[0452] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0453] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0454] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0455] [Second embodiment]

[0456] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0457] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0458] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0459] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0460] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0462] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0463] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0464] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0465] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0466] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0467] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0468] The present invention is a system that allows a user to photograph their own face and suggests the optimal makeup application method. This system includes a camera, an image analysis unit, a suggestion unit, a display unit, a management unit, a judgment unit, a purchasing unit, an advice unit, and a selection unit. The specific operation of each unit and the program processing for that purpose are explained in natural language below.

[0469] 1. Camera equipment and photography

[0470] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[0471] 2. Image Analysis

[0472] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup methods that are suitable for the user.

[0473] 3. Makeup tips

[0474] The server uses the analyzed data to calculate the optimal makeup application, taking into account the user's current cosmetics and personal information. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0475] 4. Display of makeup procedures

[0476] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[0477] 5. Cosmetics management and purchase suggestions

[0478] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[0479] 6. Post-makeup evaluation and advice

[0480] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[0481] 7. Professional makeup expert advice

[0482] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[0483] The above is the specific operation of each means in this system and the processing contents of the program.

[0484] The processing flow will be explained below.

[0485] Step 1:

[0486] (User) activates the smartphone camera and takes a picture of their own face. By opening the app and pressing the "take a picture" button, the camera is activated and the user takes a picture of their own face.

[0487] Step 2:

[0488] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[0489] Step 3:

[0490] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[0491] Step 4:

[0492] The server checks the user's cosmetics possession status, accesses the user's cosmetics database, and retrieves the cosmetics possessed by the user.

[0493] Step 5:

[0494] The server will suggest the optimal makeup application. Based on facial feature data, environmental data, and information on the cosmetics you own, the AI ​​algorithm will generate optimal makeup suggestions. For example, if the summer sun is strong, the server will suggest makeup that emphasizes UV protection.

[0495] Step 6:

[0496] The server generates the proposed makeup steps in video, still images, and text format. It records the makeup steps as a video, captures important steps as still images, and summarizes the detailed steps in text.

[0497] Step 7:

[0498] The server sends the generated makeup instructions to the device. The generated videos, still images, and text data are sent to the device and presented in a visually easy-to-understand format.

[0499] Step 8:

[0500] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[0501] Step 9:

[0502] The device checks the user's cosmetic information, checks whether the user already has the cosmetics required for the proposed makeup routine, and adds them to the list if they are missing.

[0503] Step 10:

[0504] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[0505] Step 11:

[0506] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[0507] Step 12:

[0508] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[0509] Step 13:

[0510] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[0511] Step 14:

[0512] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[0513] Step 15:

[0514] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[0515] Step 16:

[0516] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[0517] Step 17:

[0518] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[0519] Step 18:

[0520] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[0521] Step 19:

[0522] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[0523] Step 20:

[0524] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[0525] Example 1

[0526] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0527] Conventional cosmetics recommendation systems lack the ability to suggest optimal makeup applications based on the user's facial features and environmental data. Furthermore, there is no way for users to verify the consistency of the recommended cosmetics with the cosmetics they currently own, making the process of purchasing the necessary cosmetics cumbersome. Furthermore, the lack of a mechanism for providing evaluations of the makeup application or additional advice makes it difficult for users to improve their makeup techniques.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0529] In this invention, the server includes a photographing device means for allowing the user to photograph themselves, an image analysis device means for analyzing the photographed image, a suggestion device means for suggesting the optimal makeup regimen for the user based on the analysis results, a display device means for displaying the suggested makeup regimen in video, still image, and text format, a management function means for managing information on cosmetics owned by the user, a determination device means for determining whether the cosmetics required for the suggested makeup regimen are already owned or need to be purchased, a purchase device means for linking cosmetics that need to be purchased to an electronic shopping site, an advice device means for providing additional advice to the user after photographing, and a selection device means for selecting advice from experts. This makes it possible to suggest the optimal makeup regimen based on the user's characteristics and environmental data, simplifies the process of checking the compatibility of the necessary cosmetics and purchasing them, and enables the provision of evaluations after makeup application and additional advice.

[0530] The "photographing device means" is a function including a device that allows the user to take a picture of their own face.

[0531] The "image analysis device means" is a function that analyzes the captured image and extracts the user's facial features and environmental data.

[0532] The "suggestion device means" is a function that provides the user with the most suitable makeup method based on the analysis results.

[0533] The "display device" is a function that displays the proposed makeup procedure to the user in the form of video, still images, or text.

[0534] The "management function means" is a function for managing the cosmetic information owned by the user.

[0535] The "determination device means" is a function that determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased.

[0536] The "purchase device means" is a function that links cosmetics that need to be purchased to an electronic commerce site.

[0537] The "advice device means" is a function that provides additional advice to the user after taking a photo.

[0538] The "selection device means" is a function that allows the user to select advice from experts.

[0539] This invention is a system that allows a user to take a picture of their own face and suggests the best makeup application method. The system includes the following means:

[0540] photographing device means

[0541] Image analysis device means

[0542] Proposed device means

[0543] display means

[0544] management function means

[0545] Judgment device means

[0546] Purchasing Device Means

[0547] Advice Device Means

[0548] Selector means

[0549] Users take a photo of their face using the camera on their smartphone or tablet device. To do this, they press the "take a photo" button through the app, which activates the camera and takes a photo from an appropriate position. The device then sends the captured image data to the server. The server then uses an algorithm to analyze the received image data. Specifically, the server analyzes the user's facial features (e.g., face shape, skin tone, eye position) and environmental data (e.g., season, weather, temperature). Based on this analytical data, the system prepares to suggest makeup methods that are suitable for the user.

[0550] The server then uses the analyzed data to calculate the optimal makeup application, taking into account the user's current makeup habits and personal characteristics. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0551] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as "First, apply moisturizing cream, then apply liquid foundation," as well as playing back the specific makeup steps in video format.

[0552] The terminal manages information about cosmetics owned by the user. The determination device checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the terminal adds them to a list and provides a link to a shopping site, allowing the user to purchase the required cosmetics with one click.

[0553] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis, the app evaluates the makeup and generates additional advice. For example, it might provide specific feedback like, "Your eyeliner is a little too thick, so if you make it a little thinner it will balance things out better."

[0554] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. In this way, the user can receive more professional advice.

[0555] For example, the following prompts can be fed into a generative AI model to get further advice:

[0556] "I'm a woman in my 20s. I'd like to know how to apply makeup during the day in the summer. My skin tends to be dry, so I want to make my body look slimmer. The cosmetics I currently use are sunscreen, liquid foundation, powder foundation, brush-on bronzer, mascara, eyeliner, and lip tint."

[0557] This invention allows users to easily find the best makeup method for their face, and efficiently select and purchase the necessary cosmetics. In addition, users can improve their makeup techniques by receiving further advice through evaluations after makeup application.

[0558] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0559] Program processing flow

[0560] Step 1: Capture a photo of the user's face

[0561] Input: Smartphone or tablet camera

[0562] Output: Image data of the captured face

[0563] Specific behavior:

[0564] 1. The user launches the app.

[0565] 2. The user presses the "take a photo" button.

[0566] 3. The device starts the camera and the user takes a picture of their face in an appropriate position.

[0567] 4. The device generates image data of the captured face.

[0568] Step 2: Sending image data

[0569] Input: Image data of a photographed face

[0570] Output: Image data sent to the server

[0571] Specific behavior:

[0572] 1. The device sends the captured image data to the server.

[0573] 2. The server receives the image data through a receiving protocol.

[0574] Step 3: Image analysis

[0575] Input: Image data sent to the server

[0576] Output: Facial feature data, environmental data

[0577] Specific behavior:

[0578] 1. The server applies an image analysis algorithm (e.g., a deep learning model).

[0579] 2. Analyze facial features (e.g., face shape, skin tone, eye position).

[0580] 3. Reference environmental data (e.g., season, weather, temperature) to complement the analysis results.

[0581] Step 4: Makeup tips

[0582] Input: Facial feature data and environmental data

[0583] Output: Data on optimal makeup application

[0584] Specific behavior:

[0585] 1. The server uses the analysis data to calculate the optimal makeup application method in comparison with the cosmetic information it has.

[0586] 2. Consider user characteristics, environmental data, trend information, past makeup patterns, etc.

[0587] 3. Generate data on optimal makeup application methods.

[0588] Step 5: Send and view your cosmetic procedure

[0589] Input: Data on optimal makeup application

[0590] Output: Makeup procedure (video, still image, text)

[0591] Specific behavior:

[0592] 1. The server sends the generated makeup procedure to the terminal.

[0593] 2. The terminal analyzes the received instructions and displays them on the user interface.

[0594] 3. Provide makeup instructions in the form of video playback, still image display, and text display.

[0595] Step 6: Cosmetics management and purchasing suggestions

[0596] Input: Cosmetic information owned by the user, suggested makeup procedures

[0597] Output: List of cosmetics to purchase, purchase link

[0598] Specific behavior:

[0599] 1. The device references the user's cosmetics database and matches the cosmetics required for the proposed makeup procedure.

[0600] 2. If you don't have any necessary cosmetics, make a list of them.

[0601] 3. Provide a link to an e-commerce site for the cosmetics you need to purchase.

[0602] Step 7: Post-makeup evaluation and advice

[0603] Input: Image data of face after makeup

[0604] Output: Makeup evaluation data, additional advice

[0605] Specific behavior:

[0606] 1. The user takes another photo of their face after applying makeup.

[0607] 2. The device sends the new image to the server.

[0608] 3. The server analyzes the image again and generates a makeup evaluation.

[0609] 4. Generate additional advice and send it to the device.

[0610] Step 8: Expert advice

[0611] Input: User selection, facial image data

[0612] Output: Expert advice

[0613] Specific behavior:

[0614] 1. The user selects advice from a professional makeup expert.

[0615] 2. The device notifies the server of its choice.

[0616] 3. The server sends the user's image data to the expert.

[0617] 4. The expert creates customized advice and sends it to the device via the server.

[0618] 5. Users receive expert advice.

[0619] (Application example 1)

[0620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] Conventional makeup advice systems are primarily designed for use on digital devices and lack the ability to provide real-time product recommendations and inventory checks in a physical store environment. Furthermore, users often have to put in significant effort to find the perfect cosmetics in-store, and sometimes certain products are out of stock, creating an inconvenient shopping experience. A new system is needed to address these issues.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0623] In this invention, the server includes a photographing means for the user to photograph themselves, an analysis means for analyzing the photographed image, a suggestion means for suggesting the optimal makeup method to the user based on the analysis results, a display means for displaying the suggested makeup procedure in video, still image, or text format, a management means for managing information on items owned by the user, a determination means for determining whether the user already owns or needs to purchase items required for the suggested makeup procedure, a purchase means for linking items that need to be purchased to a shopping site, an advice means for providing additional advice to the user after photographing, an inventory confirmation and guidance means for checking inventory in the store and guiding the user to the location of items, and a feedback means for providing interactive feedback linked to the real world. This allows the user to find cosmetics in the store in real time and receive detailed suggestions on the optimal makeup method.

[0624] The "photographing means" is a device that allows the user to photograph their own face.

[0625] The "analysis means" is a device or software that analyzes the captured image data and extracts the user's facial features and environmental data.

[0626] The "suggestion means" is a device or software that provides the user with the most suitable makeup method based on the data obtained by the analysis means.

[0627] The "display means" is a device or software that presents the suggested makeup procedure to the user in the form of video, still images, or text.

[0628] The "management means" refers to a device or software that centrally manages information about items (cosmetics) owned by the user.

[0629] The "determination means" is a device or software that determines whether the user already possesses the items necessary for the proposed cosmetic procedure or whether new purchases are required.

[0630] "Purchase Instrument" refers to a device or software that allows the user to purchase the suggested items as needed through a link to a shopping site.

[0631] An "advice means" is a device or software that provides additional makeup advice to a user after the user has taken a selfie.

[0632] The "inventory checking and guidance means" is a device or software that checks the inventory in the store and guides the user to the location of the required item.

[0633] A "feedback tool" is a device or software that provides real-time interactive feedback and evaluations to users after they have actually used a product.

[0634] This invention provides a system that allows users to use smart glasses in a physical store and receive recommendations on optimal makeup application methods. The system includes a photographing unit, an analysis unit, a recommendation unit, a display unit, a management unit, a determination unit, a purchasing unit, an advice unit, an inventory confirmation and guidance unit, and a feedback unit.

[0635] First, the user puts on the smart glasses in a physical store and takes a picture of their face. The image captured by this photographing means is sent to the analyzing means, which then analyzes facial features and environmental data using machine learning models such as TensorFlow. This analyzed data is then used by the suggesting means to suggest the most suitable makeup application to the user.

[0636] The suggestion unit suggests the most suitable cosmetics based on the user's facial features and environmental data. For example, it may recommend the most suitable foundation or eyeshadow depending on the user's skin tone, the season, and the lighting conditions in the store. The suggestion is displayed on the smart glasses' display via the display unit in the form of video, still images, or text.

[0637] The management means manages information about cosmetics owned by the user through the smart glasses and related applications. The determination means determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased. If necessary, the purchase means provides a link to a related online shopping site to facilitate easy purchase.

[0638] Furthermore, the inventory check and guidance module checks the inventory in the physical store and guides the user to the location of the necessary items. This information is linked to the store's real-time inventory database. After the user actually applies the makeup, the feedback module provides real-time interactive evaluation and additional makeup advice.

[0639] As a concrete example, imagine a user puts on smart glasses in a physical store and scans their face. The system analyzes the user's face and suggests the best foundation and eyeshadow based on their skin tone. These suggestions are played back as a video on the glasses' display, along with detailed instructions on how to use the product. The system also displays the shelf location, allowing users to easily find the product without getting lost. Furthermore, with one click, users can access an online shopping site for the product they need, and if it's not in stock at the store, they can have it ordered from a nearby store or online store.

[0640] The system allows users to receive detailed real-time suggestions on how to best apply makeup through the smart glasses, and also significantly improves the in-store shopping experience.

[0641] Examples of prompts powered by generative AI models include:

[0642] "Develop an app that takes a picture of the user's face with a camera in smart glasses and suggests the best makeup look for them, taking into account their facial features, skin tone, eye position, and in-store environmental data (lighting, climate, temperature). It also checks the stock availability of cosmetics used in the suggested makeup look and provides real-time feedback to the user."

[0643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0644] Step 1:

[0645] The user puts on the smart glasses in a physical store and has their face photographed.

[0646] Input: Image data of the user's face captured by the camera built into the smart glasses.

[0647] Action: The capture device activates the camera and prompts the user to position their face appropriately.

[0648] Output: The captured image data is stored in the smart glasses.

[0649] Step 2:

[0650] The captured image is sent to a server for analysis.

[0651] Input: Image data of the user's face.

[0652] How it works: Image data is sent from the smart glasses to a server, and the analysis means analyzes the image using a machine learning model such as TensorFlow.

[0653] Output: Facial feature data and environmental data (skin tone, eye position, lighting conditions, etc.) are generated.

[0654] Step 3:

[0655] Based on the analysis results, the suggestion method calculates the optimal makeup method.

[0656] Input: Facial feature data and environmental data.

[0657] How it works: The proposed method uses a generative AI model to calculate the optimal makeup routine for the user.

[0658] Output: Makeup application suggestion data (video, still images, text format) is generated.

[0659] Step 4:

[0660] The generated makeup procedure is displayed on the smart glasses.

[0661] Input: Makeup method suggestion data.

[0662] Operation: The display means displays makeup instructions in video, still images, and text format on the display of the smart glasses.

[0663] Output: A state in which the user can visually confirm the makeup procedure.

[0664] Step 5:

[0665] The cosmetic information possessed by the user is confirmed by the management means.

[0666] Input: A list of cosmetics owned by the user.

[0667] Operation: The control unit compares the list of cosmetics on hand with the cosmetics required for the proposed makeup procedure.

[0668] Output: The result of determining whether the item is already in possession or needs to be purchased.

[0669] Step 6:

[0670] Link missing cosmetics to purchasing options.

[0671] Input: A list of cosmetics that need to be purchased according to the determination means.

[0672] Operation: The purchasing instrument generates and provides a link to an online shopping site to the user.

[0673] Output: Purchase link information.

[0674] Step 7:

[0675] Check inventory in store and provide location of items.

[0676] Input: List of cosmetics you need to buy.

[0677] Operation: The inventory check and guidance tool checks the store's real-time inventory database and guides the user to shelf locations.

[0678] Output: Inventory check results and shelf location information.

[0679] Step 8:

[0680] After the user actually applies the makeup, the user is given an evaluation and additional advice through a feedback means.

[0681] Input: Image data of the user's face after makeup application.

[0682] How it works: The feedback mechanism analyzes the image again and generates a makeup rating and additional recommendations.

[0683] Output: Evaluation results and further advice.

[0684] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0685] The present invention combines a system that allows a user to take a picture of their own face and suggests the optimal makeup application method with an emotion recognition means that recognizes the user's emotions. This system includes a camera means, an image analysis means, a suggestion means, a display means, a management means, a judgment means, a purchasing means, an advice means, a selection means, and an emotion recognition means. The specific operation of each means and the program processing for that purpose are explained in natural language below.

[0686] 1. Camera equipment and photography

[0687] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[0688] 2. Image Analysis

[0689] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup techniques that are suitable for the user.

[0690] 3. Emotion recognition

[0691] Furthermore, the image analysis means acquires facial expression data of the user, which is then analyzed by the emotion recognition means. The emotion recognition means identifies the current emotion (e.g., joy, sadness, anger, etc.) from the user's facial expression.

[0692] 4. Makeup tips

[0693] Based on the analyzed data, the server compares it with the information on the cosmetics the user owns and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also trend information, past makeup patterns, and emotional data obtained through emotion recognition. For example, if the server recognizes that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0694] 5. Display of makeup procedures

[0695] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[0696] 6. Cosmetics management and purchase suggestions

[0697] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[0698] 7. Post-makeup evaluation and advice

[0699] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[0700] 8. Professional makeup expert advice

[0701] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[0702] The above is the specific operation of each means in this system and the processing contents of the program.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user opens the app and presses the "take a picture" button to activate the camera, and then positions themselves appropriately to take a picture.

[0706] Step 2:

[0707] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[0708] Step 3:

[0709] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[0710] Step 4:

[0711] The server checks the user's cosmetics possession status. It accesses the user's cosmetics database and retrieves the cosmetics possessed by the user. This data is taken into account in conjunction with the analysis results.

[0712] Step 5:

[0713] The (server) uses emotion recognition means to analyze the user's emotions. It analyzes the user's facial expressions from the analyzed image data and identifies their current emotional state (e.g., joy, sadness, anger, etc.).

[0714] Step 6:

[0715] The server will then suggest the optimal makeup application based on all analysis results (facial features, environmental data, and emotional data). Using an AI algorithm, it will compare the results with the user's current cosmetics and generate specific makeup steps. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0716] Step 7:

[0717] The server prepares the generated makeup steps in video, still images, and text format and sends them to the device. The server records the makeup steps as video, captures important steps as still images, and summarizes the detailed steps in text.

[0718] Step 8:

[0719] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[0720] Step 9:

[0721] The device checks the cosmetics information it has, checks whether the cosmetics required for the proposed makeup procedure are already in possession, and adds them to the list if they are missing.

[0722] Step 10:

[0723] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[0724] Step 11:

[0725] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[0726] Step 12:

[0727] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[0728] Step 13:

[0729] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[0730] Step 14:

[0731] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[0732] Step 15:

[0733] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[0734] Step 16:

[0735] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[0736] Step 17:

[0737] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[0738] Step 18:

[0739] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[0740] Step 19:

[0741] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[0742] Step 20:

[0743] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[0744] Example 2

[0745] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] Conventional makeup suggestion systems only consider the user's facial features and environmental data, which means they are unable to suggest makeup methods that match the user's current emotional state or mood of the day. Furthermore, there are limited ways to display makeup steps, and they lack a format that is visually easy for users to understand. Furthermore, if the required cosmetics are not available, users must manually search and purchase them, which is a time-consuming process.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0748] In this invention, the server includes an image analysis means for analyzing the captured image, an emotion recognition means for acquiring and analyzing the user's facial expression data by the image analysis means, and a suggestion means for suggesting the most suitable makeup method to the user based on the analysis results. This makes it possible to suggest the most suitable makeup method to the user taking into consideration the user's facial features, environmental data, and the user's emotional state.

[0749] "Imaging means" refers to a device or equipment that allows a user to take a picture of their own face.

[0750] "Image analysis means" refers to a device or method that analyzes captured image data and extracts facial features and other relevant information.

[0751] "Emotion recognition means" refers to a device or method for identifying the emotional state of a user based on facial expression data acquired by image analysis means.

[0752] The "suggestion means" refers to a device or method that suggests the most suitable makeup method to the user based on the analysis results.

[0753] "Display means" refers to a device or method that visually presents the suggested makeup procedure to the user in the form of video, still images, or text.

[0754] "Information management means" refers to a device or method for managing information about cosmetics owned by a user.

[0755] "Determination means" refers to a device or method for confirming whether the cosmetics required for the proposed makeup procedure are already in possession.

[0756] "Purchase method" refers to a method that provides a link to an online retail site to facilitate purchases by users when they are out of the required cosmetics.

[0757] "Advice means" refers to a device or method that allows a user to receive additional advice after taking a photo.

[0758] "Selection means" refers to a device or method that allows a user to select advice from experts.

[0759] The present invention is a system that allows a user to take a picture of their own face and suggests the optimal makeup method. Furthermore, by combining it with an emotion recognition means that recognizes the user's emotions, it is possible to suggest makeup methods that correspond to the user's emotional state. This system includes an imaging means, an image analysis means, an emotion recognition means, a suggestion means, a display means, an information management means, a judgment means, a purchasing means, an advice means, and a selection means.

[0760] Imaging means

[0761] The user activates the camera on their smartphone or tablet device and presses the "take a photo" button to take a picture of their own face, which generates image data.

[0762] Image analysis methods

[0763] The device sends the captured image data to a server. The server uses an algorithm to analyze the received image data. For example, image analysis libraries such as OpenCV or TensorFlow are used to analyze facial features (e.g., facial contours, skin tone, eye position, etc.). Environmental data (e.g., weather, season, temperature, etc.) is also collected and used to analyze the data.

[0764] emotion recognition means

[0765] The image analysis means acquires the user's facial expression data, which is then analyzed by the emotion recognition means. An emotion recognition algorithm is used to identify the user's emotional state (e.g., joy, sadness, anger, etc.), often using an Emotion API or a deep learning model.

[0766] Proposal means

[0767] Based on the analysis results, the server compares the user's cosmetics with their current makeup product information and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also emotional data obtained through emotion recognition. For example, if the server detects that the user looks tired, it will suggest a lighter makeup look that will give a refreshing feeling.

[0768] Display means

[0769] The server prepares the generated makeup instructions in video, still images, and text format and sends them to the device. The device then displays the received information to the user. For example, the video might say, "First, apply moisturizing cream," while simultaneously displaying the same part in text.

[0770] Information management means

[0771] The device manages the cosmetics information of the user, checks the information database of the cosmetics the user owns, and determines whether the user has the necessary items.

[0772] Judgment means

[0773] Check whether the cosmetics required for the proposed makeup procedure are already in your possession and notify the user if any required items are missing.

[0774] Purchase method

[0775] If the required cosmetics are missing, the device will assist the user in the purchase process by providing a link to an online retail site, allowing the user to easily purchase the missing cosmetics.

[0776] Advice tools

[0777] After the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis results, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little too thick. Next time, try drawing it thinner."

[0778] Selection method

[0779] The user can select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server.

[0780] Specific examples and prompts for the generative AI model

[0781] For example, when a user uses this system to get ready in the morning, they follow the steps below.

[0782] 1. The user launches the app and takes a photo of their face.

[0783] 2. The server analyzes facial features and environmental data, and then recognizes emotions from the user's facial expressions.

[0784] 3. If the user feels tired, the server will suggest refreshing makeup using lighter colors.

[0785] 4. Apply makeup while watching the video to see the suggested makeup steps.

[0786] 5. If you are missing the cosmetics you need, proceed with the purchase.

[0787] 6. After your makeup is applied, you will receive evaluation and advice on how to further refine your look.

[0788] Example prompt for a generative AI model:

[0789] "Please explain how a system analyzes a user's facial photo and suggests the best makeup application. Please also explain in detail how it combines emotion recognition and suggests lighter makeup colors if the user wants to feel refreshed."

[0790] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0791] Step 1:

[0792] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user then presses the "take a picture" button through the app, which activates the camera. This inputs image data of the face.

[0793] Step 2:

[0794] The device sends the captured image data to the server. The image data is uploaded, and the server receives it. The input image data is then processed, and image analysis is ready.

[0795] Step 3:

[0796] The server analyzes the received image data. Using a facial recognition algorithm (e.g., OpenCV or TensorFlow), it analyzes facial features (facial contours, skin tone, eye position, etc.) and environmental data (weather information, season, etc.). This analysis outputs the user's facial feature data.

[0797] Step 4:

[0798] The server acquires the user's facial expression data using image analysis means and analyzes it using emotion recognition means. The user's emotions (e.g., joy, sadness, anger, etc.) are identified using the Emotion API and deep learning models. Emotion data is output from the input facial expression data.

[0799] Step 5:

[0800] The server calculates the optimal makeup application method (using AI models such as Keras) based on the analysis results (facial feature data and emotional data). The calculation takes into account the user's features, environmental data, and emotional data. For example, if the user is recognized as tired, a refreshing makeup look using lighter colors will be suggested. The output is suggested data for the optimal makeup application method.

[0801] Step 6:

[0802] The server generates a proposed makeup procedure in the form of video, still images, and text, and sends it to the device. The device then displays the received makeup procedure to the user. For example, a video might say, "First, apply moisturizing cream," while simultaneously displaying that part in text. The input is the proposed data, and the output is the display data of the makeup procedure.

[0803] Step 7:

[0804] The device manages the user's cosmetics information and checks whether the user has the cosmetics necessary for the proposed makeup procedure. It checks the managed cosmetics database and outputs a list of missing cosmetics.

[0805] Step 8:

[0806] The terminal assists the user in purchasing the missing cosmetics. It provides the user with a link to an online retail site, allowing them to easily purchase the necessary cosmetics. The terminal notifies the user by saying, "You are missing this lipstick. You can purchase it here." The missing cosmetics list is input, and the online link data is output.

[0807] Step 9:

[0808] After the user finishes applying makeup, they take a picture of themselves again. The user takes a new image, and the device sends it to the server. The input is the new image data.

[0809] Step 10:

[0810] The server analyzes the image again and generates makeup evaluation and advice based on the new analysis results. For example, it may generate specific feedback such as, "Your eyeliner is a little thick, so next time, try drawing it thinner." The input is the new image data, and the output is the evaluation and advice data.

[0811] Step 11:

[0812] When a user selects advice from a professional makeup expert, the device notifies the server of the selection. The server sends the user's image data to the expert, who then creates customized advice based on the image. The data is then sent to the device via the server. The input is the user's selection information and image data, and the output is the expert's advice.

[0813] (Application example 2)

[0814] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] Conventional makeup suggestion systems suggest optimal makeup methods based on the user's facial features and environmental data. However, these systems do not take into account the user's current emotional state, making it difficult to provide makeup suggestions that match the user's feelings and mood. Furthermore, because the system does not suggest makeup methods that correspond to emotions, it is difficult to increase user satisfaction.

[0816] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the user's face, an analysis means for analyzing the captured image, an emotion recognition means for recognizing emotions, and a suggestion means for suggesting an optimal makeup method to the user based on the analysis results and emotion data. This makes it possible to suggest a makeup method based on the user's emotional state.

[0817] "Photography means" refers to a device or function that allows a user to take pictures or videos of themselves.

[0818] "Analysis means" refers to a device or function for analyzing facial features and environmental information based on captured image and video data.

[0819] The "suggestion means" is a device or function that calculates and suggests the most suitable makeup method for the user based on the analyzed data.

[0820] The "display means" is a device or function for displaying the proposed makeup procedure to the user in the form of video, still images, or text.

[0821] The "management means" refers to a device or function for centrally managing information about cosmetics owned by a user.

[0822] The "determination means" is a device or function that determines whether the cosmetics required for the proposed makeup procedure are in the user's possession or need to be purchased.

[0823] "Purchase means" refers to a device or function that links cosmetics to be purchased to an online purchasing site, allowing users to easily purchase them.

[0824] An "advice means" is a device or function that provides additional advice or evaluation to the user after the user has finished applying makeup.

[0825] The "selection means" is a device or function that allows the user to select and receive advice from an expert.

[0826] "Emotion recognition means" refers to a device or function for analyzing and recognizing the current emotion of the user from their facial expression.

[0827] The present invention is a system that allows users to take a picture of their own face, analyzes the image data, and provides optimal makeup methods, and also adds a function to recognize the user's emotions. This system takes into account the user's facial features, environmental information, and current emotional state to propose more detailed and personalized makeup methods.

[0828] Hardware and software used

[0829] This system uses the following hardware and software:

[0830] Photography method: Use the built-in camera on a smartphone or tablet device or a webcam connected to a computer.

[0831] Analysis method: DeepFace library for analyzing facial features and environmental information.

[0832] Emotion recognizer: An emotion recognition algorithm to recognize user emotions (e.g., a Python emotion recognition library).

[0833] Suggestion method: A makeup suggestion algorithm that runs on the server side. Data is exchanged via REST API.

[0834] Display means: Smartphone or tablet display, and dedicated mobile application.

[0835] Management means: A database (e.g., SQLite) for managing the cosmetic data owned by the user.

[0836] Determination method: An algorithm to determine whether the required cosmetics are in possession.

[0837] Purchase method: A function to link the cosmetics to be purchased to an online shop (e.g., integration with a shopping API).

[0838] Advice tool: An algorithm that analyzes re-photograph data after makeup application and provides additional advice.

[0839] Selection tool: A user interface that allows the selection of expert advice.

[0840] System operation explanation

[0841] Filming method

[0842] Users take a photo of their face using the camera on their smartphone or tablet device by launching the camera app and pressing the "take photo" button.

[0843] Analysis means

[0844] The device sends the captured image data to a server and analyzes it using the DeepFace library to extract facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.).

[0845] emotion recognition means

[0846] The image analysis means acquires facial expression data of the user, and the emotion recognition means analyzes this to identify the user's current emotion (for example, joy, sadness, anger, etc.).

[0847] Proposal means

[0848] The server then uses the analysis data to suggest makeup applications. These suggestions take into account not only the user's facial features and environmental data, but also their emotional data. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[0849] Display means

[0850] The server sends the generated makeup instructions in the form of video, still images, and text to the device, which then displays them to the user. The display includes detailed instructions in video, still images, and text.

[0851] Examples of concrete examples and prompts

[0852] Below are some example prompts to be input to the generative AI model:

[0853] Develop a system that takes a picture of a user's face, analyzes the image to recognize emotions, and suggests the best makeup application based on the user's emotions. Follow these steps:

[0854] 1. Activate the camera and take a picture of the user's face.

[0855] 2. Analyze the captured image to identify facial features and emotions (happiness, sadness, anger, etc.).

[0856] 3. Send facial features and emotion data to an external API and receive the optimal makeup application.

[0857] 4. The received makeup method is displayed to the user.

[0858] As described above, the system for implementing this invention is a multifunctional system that proposes the optimal makeup method taking into consideration the user's facial features and emotions. This system allows the user to apply makeup that suits their emotional state, thereby increasing satisfaction.

[0859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0860] Step 1:

[0861] The user activates the camera on their smartphone or tablet and takes a picture of their own face. At this time, the camera is activated by pressing the "shoot" button in the application, and the user takes a picture in an appropriate position. The input is the user's facial image, which is output as image data.

[0862] Step 2:

[0863] The device receives the captured image data and sends it to the server. The input is the image data, which is sent to the server via the network. The output is the image data received by the server.

[0864] Step 3:

[0865] The server analyzes the received image data. For this analysis, it uses the DeepFace library to analyze the user's facial features (face shape, skin tone, eye position, etc.). The input is the received image data, and the output is facial feature data. Specifically, the DeepFace library maps each point on the face and extracts feature data.

[0866] Step 4:

[0867] The server then uses emotion recognition means to recognize the user's emotions from the analyzed feature data. This process uses an emotion recognition algorithm (for example, a Python emotion recognition library). The input is facial feature data, and the output is emotion data. Specifically, it analyzes changes in facial expression (for example, eyebrow movement, mouth corner position) to identify emotions such as joy, sadness, and anger.

[0868] Step 5:

[0869] The server then uses the analyzed data (facial features and emotion data) to suggest the most suitable makeup application method for the user. This proposal is made using a makeup application suggestion algorithm on the server. The input is facial feature data and emotion data, and the output is suggested data for makeup procedures. For example, if the emotion is recognized as "fatigue," the server will suggest a makeup method to make the user's face look brighter.

[0870] Step 6:

[0871] The server prepares the generated makeup steps in video, still image, and text formats and sends them to the terminal. The input is the proposed makeup steps data, and the output is the terminal that receives it. Specifically, the server converts the makeup steps into multiple formats (video file, still image file, text file) and sends them all at once to the terminal.

[0872] Step 7:

[0873] The device then displays the received makeup instructions to the user. The input is the received makeup instructions data, and the output is a visual representation of the makeup instructions that the user can confirm. Specifically, the device application plays the video and highlights important parts with still images and text.

[0874] Step 8:

[0875] The user applies makeup according to the suggested makeup steps, then takes a photo of their face again. The input is the face image after makeup, and the output is the image data. Specifically, the user starts the camera again and presses the "take another photo" button to take a photo of their face.

[0876] Step 9:

[0877] The device sends the face image data after makeup application to the server, which then analyzes it again. The input is the re-photographed image data, and the output is the analysis results. Specifically, the server executes the process described above again to evaluate the face after makeup application.

[0878] Step 10:

[0879] The server generates additional advice based on the reanalysis results and sends it to the terminal. The input is the reanalysis results, and the output is additional advice data. Specific actions include providing specific feedback such as "Draw your eyeliner a little thinner" if the eyeliner is too thick.

[0880] Step 11:

[0881] The terminal displays the received additional advice to the user. The input is the received additional advice data, and the output is an advice display that the user can visually confirm. Specifically, the advice text or a still image is displayed on the terminal screen.

[0882] The above are the specific processing steps for carrying out the present invention. The user can receive personalized makeup suggestions that take into consideration their emotions, which can improve satisfaction.

[0883] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0884] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0885] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0886] [Third embodiment]

[0887] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0888] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0889] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0890] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0891] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0892] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0893] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0894] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0895] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0896] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0897] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0898] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0899] The present invention is a system that allows a user to photograph their own face and suggests the optimal makeup application method. This system includes a camera, an image analysis unit, a suggestion unit, a display unit, a management unit, a judgment unit, a purchasing unit, an advice unit, and a selection unit. The specific operation of each unit and the program processing for that purpose are explained in natural language below.

[0900] 1. Camera equipment and photography

[0901] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[0902] 2. Image Analysis

[0903] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup methods that are suitable for the user.

[0904] 3. Makeup tips

[0905] The server uses the analyzed data to calculate the optimal makeup application, taking into account the user's current cosmetics and personal information. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0906] 4. Display of makeup procedures

[0907] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[0908] 5. Cosmetics management and purchase suggestions

[0909] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[0910] 6. Post-makeup evaluation and advice

[0911] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[0912] 7. Professional makeup expert advice

[0913] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[0914] The above is the specific operation of each means in this system and the processing contents of the program.

[0915] The processing flow will be explained below.

[0916] Step 1:

[0917] (User) activates the smartphone camera and takes a picture of their own face. By opening the app and pressing the "take a picture" button, the camera is activated and the user takes a picture of their own face.

[0918] Step 2:

[0919] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[0920] Step 3:

[0921] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[0922] Step 4:

[0923] The server checks the user's cosmetics possession status, accesses the user's cosmetics database, and retrieves the cosmetics possessed by the user.

[0924] Step 5:

[0925] The server will suggest the optimal makeup application. Based on facial feature data, environmental data, and information on the cosmetics you own, the AI ​​algorithm will generate optimal makeup suggestions. For example, if the summer sun is strong, the server will suggest makeup that emphasizes UV protection.

[0926] Step 6:

[0927] The server generates the proposed makeup steps in video, still images, and text format. It records the makeup steps as a video, captures important steps as still images, and summarizes the detailed steps in text.

[0928] Step 7:

[0929] The server sends the generated makeup instructions to the device. The generated videos, still images, and text data are sent to the device and presented in a visually easy-to-understand format.

[0930] Step 8:

[0931] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[0932] Step 9:

[0933] The device checks the user's cosmetic information, checks whether the user already has the cosmetics required for the proposed makeup routine, and adds them to the list if they are missing.

[0934] Step 10:

[0935] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[0936] Step 11:

[0937] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[0938] Step 12:

[0939] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[0940] Step 13:

[0941] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[0942] Step 14:

[0943] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[0944] Step 15:

[0945] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[0946] Step 16:

[0947] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[0948] Step 17:

[0949] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[0950] Step 18:

[0951] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[0952] Step 19:

[0953] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[0954] Step 20:

[0955] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[0956] Example 1

[0957] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] Conventional cosmetics recommendation systems lack the ability to suggest optimal makeup applications based on the user's facial features and environmental data. Furthermore, there is no way for users to verify the consistency of the recommended cosmetics with the cosmetics they currently own, making the process of purchasing the necessary cosmetics cumbersome. Furthermore, the lack of a mechanism for providing evaluations of the makeup application or additional advice makes it difficult for users to improve their makeup techniques.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0960] In this invention, the server includes a photographing device means for allowing the user to photograph themselves, an image analysis device means for analyzing the photographed image, a suggestion device means for suggesting the optimal makeup regimen for the user based on the analysis results, a display device means for displaying the suggested makeup regimen in video, still image, and text format, a management function means for managing information on cosmetics owned by the user, a determination device means for determining whether the cosmetics required for the suggested makeup regimen are already owned or need to be purchased, a purchase device means for linking cosmetics that need to be purchased to an electronic shopping site, an advice device means for providing additional advice to the user after photographing, and a selection device means for selecting advice from experts. This makes it possible to suggest the optimal makeup regimen based on the user's characteristics and environmental data, simplifies the process of checking the compatibility of the necessary cosmetics and purchasing them, and enables the provision of evaluations after makeup application and additional advice.

[0961] The "photographing device means" is a function including a device that allows the user to take a picture of their own face.

[0962] The "image analysis device means" is a function that analyzes the captured image and extracts the user's facial features and environmental data.

[0963] The "suggestion device means" is a function that provides the user with the most suitable makeup method based on the analysis results.

[0964] The "display device" is a function that displays the proposed makeup procedure to the user in the form of video, still images, or text.

[0965] The "management function means" is a function for managing the cosmetic information owned by the user.

[0966] The "determination device means" is a function that determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased.

[0967] The "purchase device means" is a function that links cosmetics that need to be purchased to an electronic commerce site.

[0968] The "advice device means" is a function that provides additional advice to the user after taking a photo.

[0969] The "selection device means" is a function that allows the user to select advice from experts.

[0970] This invention is a system that allows a user to take a picture of their own face and suggests the best makeup application method. The system includes the following means:

[0971] photographing device means

[0972] Image analysis device means

[0973] Proposed device means

[0974] display means

[0975] management function means

[0976] Judgment device means

[0977] Purchasing Device Means

[0978] Advice Device Means

[0979] Selector means

[0980] Users take a photo of their face using the camera on their smartphone or tablet device. To do this, they press the "take a photo" button through the app, which activates the camera and takes a photo from an appropriate position. The device then sends the captured image data to the server. The server then uses an algorithm to analyze the received image data. Specifically, the server analyzes the user's facial features (e.g., face shape, skin tone, eye position) and environmental data (e.g., season, weather, temperature). Based on this analytical data, the system prepares to suggest makeup methods that are suitable for the user.

[0981] The server then uses the analyzed data to calculate the optimal makeup application, taking into account the user's current makeup habits and personal characteristics. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[0982] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as "First, apply moisturizing cream, then apply liquid foundation," as well as playing back the specific makeup steps in video format.

[0983] The terminal manages information about cosmetics owned by the user. The determination device checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the terminal adds them to a list and provides a link to a shopping site, allowing the user to purchase the required cosmetics with one click.

[0984] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis, the app evaluates the makeup and generates additional advice. For example, it might provide specific feedback like, "Your eyeliner is a little too thick, so if you make it a little thinner it will balance things out better."

[0985] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. In this way, the user can receive more professional advice.

[0986] For example, the following prompts can be fed into a generative AI model to get further advice:

[0987] "I'm a woman in my 20s. I'd like to know how to apply makeup during the day in the summer. My skin tends to be dry, so I want to make my body look slimmer. The cosmetics I currently use are sunscreen, liquid foundation, powder foundation, brush-on bronzer, mascara, eyeliner, and lip tint."

[0988] This invention allows users to easily find the best makeup method for their face, and efficiently select and purchase the necessary cosmetics. In addition, users can improve their makeup techniques by receiving further advice through evaluations after makeup application.

[0989] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0990] Program processing flow

[0991] Step 1: Capture a photo of the user's face

[0992] Input: Smartphone or tablet camera

[0993] Output: Image data of the captured face

[0994] Specific behavior:

[0995] 1. The user launches the app.

[0996] 2. The user presses the "take a photo" button.

[0997] 3. The device starts the camera and the user takes a picture of their face in an appropriate position.

[0998] 4. The device generates image data of the captured face.

[0999] Step 2: Sending image data

[1000] Input: Image data of a photographed face

[1001] Output: Image data sent to the server

[1002] Specific behavior:

[1003] 1. The device sends the captured image data to the server.

[1004] 2. The server receives the image data through a receiving protocol.

[1005] Step 3: Image analysis

[1006] Input: Image data sent to the server

[1007] Output: Facial feature data, environmental data

[1008] Specific behavior:

[1009] 1. The server applies an image analysis algorithm (e.g., a deep learning model).

[1010] 2. Analyze facial features (e.g., face shape, skin tone, eye position).

[1011] 3. Reference environmental data (e.g., season, weather, temperature) to complement the analysis results.

[1012] Step 4: Makeup tips

[1013] Input: Facial feature data and environmental data

[1014] Output: Data on optimal makeup application

[1015] Specific behavior:

[1016] 1. The server uses the analysis data to calculate the optimal makeup application method in comparison with the cosmetic information it has.

[1017] 2. Consider user characteristics, environmental data, trend information, past makeup patterns, etc.

[1018] 3. Generate data on optimal makeup application methods.

[1019] Step 5: Send and view your cosmetic procedure

[1020] Input: Data on optimal makeup application

[1021] Output: Makeup procedure (video, still image, text)

[1022] Specific behavior:

[1023] 1. The server sends the generated makeup procedure to the terminal.

[1024] 2. The terminal analyzes the received instructions and displays them on the user interface.

[1025] 3. Provide makeup instructions in the form of video playback, still image display, and text display.

[1026] Step 6: Cosmetics management and purchasing suggestions

[1027] Input: Cosmetic information owned by the user, suggested makeup procedures

[1028] Output: List of cosmetics to purchase, purchase link

[1029] Specific behavior:

[1030] 1. The device references the user's cosmetics database and matches the cosmetics required for the proposed makeup procedure.

[1031] 2. If you don't have any necessary cosmetics, make a list of them.

[1032] 3. Provide a link to an e-commerce site for the cosmetics you need to purchase.

[1033] Step 7: Post-makeup evaluation and advice

[1034] Input: Image data of face after makeup

[1035] Output: Makeup evaluation data, additional advice

[1036] Specific behavior:

[1037] 1. The user takes another photo of their face after applying makeup.

[1038] 2. The device sends the new image to the server.

[1039] 3. The server analyzes the image again and generates a makeup evaluation.

[1040] 4. Generate additional advice and send it to the device.

[1041] Step 8: Expert advice

[1042] Input: User selection, facial image data

[1043] Output: Expert advice

[1044] Specific behavior:

[1045] 1. The user selects advice from a professional makeup expert.

[1046] 2. The device notifies the server of its choice.

[1047] 3. The server sends the user's image data to the expert.

[1048] 4. The expert creates customized advice and sends it to the device via the server.

[1049] 5. Users receive expert advice.

[1050] (Application example 1)

[1051] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1052] Conventional makeup advice systems are primarily designed for use on digital devices and lack the ability to provide real-time product recommendations and inventory checks in a physical store environment. Furthermore, users often have to put in significant effort to find the perfect cosmetics in-store, and sometimes certain products are out of stock, creating an inconvenient shopping experience. A new system is needed to address these issues.

[1053] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1054] In this invention, the server includes a photographing means for the user to photograph themselves, an analysis means for analyzing the photographed image, a suggestion means for suggesting the optimal makeup method to the user based on the analysis results, a display means for displaying the suggested makeup procedure in video, still image, or text format, a management means for managing information on items owned by the user, a determination means for determining whether the user already owns or needs to purchase items required for the suggested makeup procedure, a purchase means for linking items that need to be purchased to a shopping site, an advice means for providing additional advice to the user after photographing, an inventory confirmation and guidance means for checking inventory in the store and guiding the user to the location of items, and a feedback means for providing interactive feedback linked to the real world. This allows the user to find cosmetics in the store in real time and receive detailed suggestions on the optimal makeup method.

[1055] The "photographing means" is a device that allows the user to photograph their own face.

[1056] The "analysis means" is a device or software that analyzes the captured image data and extracts the user's facial features and environmental data.

[1057] The "suggestion means" is a device or software that provides the user with the most suitable makeup method based on the data obtained by the analysis means.

[1058] The "display means" is a device or software that presents the suggested makeup procedure to the user in the form of video, still images, or text.

[1059] The "management means" refers to a device or software that centrally manages information about items (cosmetics) owned by the user.

[1060] The "determination means" is a device or software that determines whether the user already possesses the items necessary for the proposed cosmetic procedure or whether new purchases are required.

[1061] "Purchase Instrument" refers to a device or software that allows the user to purchase the suggested items as needed through a link to a shopping site.

[1062] An "advice means" is a device or software that provides additional makeup advice to a user after the user has taken a selfie.

[1063] The "inventory checking and guidance means" is a device or software that checks the inventory in the store and guides the user to the location of the required item.

[1064] A "feedback tool" is a device or software that provides real-time interactive feedback and evaluations to users after they have actually used a product.

[1065] This invention provides a system that allows users to use smart glasses in a physical store and receive recommendations on optimal makeup application methods. The system includes a photographing unit, an analysis unit, a recommendation unit, a display unit, a management unit, a determination unit, a purchasing unit, an advice unit, an inventory confirmation and guidance unit, and a feedback unit.

[1066] First, the user puts on the smart glasses in a physical store and takes a picture of their face. The image captured by this photographing means is sent to the analyzing means, which then analyzes facial features and environmental data using machine learning models such as TensorFlow. This analyzed data is then used by the suggesting means to suggest the most suitable makeup application to the user.

[1067] The suggestion unit suggests the most suitable cosmetics based on the user's facial features and environmental data. For example, it may recommend the most suitable foundation or eyeshadow depending on the user's skin tone, the season, and the lighting conditions in the store. The suggestion is displayed on the smart glasses' display via the display unit in the form of video, still images, or text.

[1068] The management means manages information about cosmetics owned by the user through the smart glasses and related applications. The determination means determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased. If necessary, the purchase means provides a link to a related online shopping site to facilitate easy purchase.

[1069] Furthermore, the inventory check and guidance module checks the inventory in the physical store and guides the user to the location of the necessary items. This information is linked to the store's real-time inventory database. After the user actually applies the makeup, the feedback module provides real-time interactive evaluation and additional makeup advice.

[1070] As a concrete example, imagine a user puts on smart glasses in a physical store and scans their face. The system analyzes the user's face and suggests the best foundation and eyeshadow based on their skin tone. These suggestions are played back as a video on the glasses' display, along with detailed instructions on how to use the product. The system also displays the shelf location, allowing users to easily find the product without getting lost. Furthermore, with one click, users can access an online shopping site for the product they need, and if it's not in stock at the store, they can have it ordered from a nearby store or online store.

[1071] The system allows users to receive detailed real-time suggestions on how to best apply makeup through the smart glasses, and also significantly improves the in-store shopping experience.

[1072] Examples of prompts powered by generative AI models include:

[1073] "Develop an app that takes a picture of the user's face with a camera in smart glasses and suggests the best makeup look for them, taking into account their facial features, skin tone, eye position, and in-store environmental data (lighting, climate, temperature). It also checks the stock availability of cosmetics used in the suggested makeup look and provides real-time feedback to the user."

[1074] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1075] Step 1:

[1076] The user puts on the smart glasses in a physical store and has their face photographed.

[1077] Input: Image data of the user's face captured by the camera built into the smart glasses.

[1078] Action: The capture device activates the camera and prompts the user to position their face appropriately.

[1079] Output: The captured image data is stored in the smart glasses.

[1080] Step 2:

[1081] The captured image is sent to a server for analysis.

[1082] Input: Image data of the user's face.

[1083] How it works: Image data is sent from the smart glasses to a server, and the analysis means analyzes the image using a machine learning model such as TensorFlow.

[1084] Output: Facial feature data and environmental data (skin tone, eye position, lighting conditions, etc.) are generated.

[1085] Step 3:

[1086] Based on the analysis results, the suggestion method calculates the optimal makeup method.

[1087] Input: Facial feature data and environmental data.

[1088] How it works: The proposed method uses a generative AI model to calculate the optimal makeup routine for the user.

[1089] Output: Makeup application suggestion data (video, still images, text format) is generated.

[1090] Step 4:

[1091] The generated makeup procedure is displayed on the smart glasses.

[1092] Input: Makeup method suggestion data.

[1093] Operation: The display means displays makeup instructions in video, still images, and text format on the display of the smart glasses.

[1094] Output: A state in which the user can visually confirm the makeup procedure.

[1095] Step 5:

[1096] The cosmetic information possessed by the user is confirmed by the management means.

[1097] Input: A list of cosmetics owned by the user.

[1098] Operation: The control unit compares the list of cosmetics on hand with the cosmetics required for the proposed makeup procedure.

[1099] Output: The result of determining whether the item is already in possession or needs to be purchased.

[1100] Step 6:

[1101] Link missing cosmetics to purchasing options.

[1102] Input: A list of cosmetics that need to be purchased according to the determination means.

[1103] Operation: The purchasing instrument generates and provides a link to an online shopping site to the user.

[1104] Output: Purchase link information.

[1105] Step 7:

[1106] Check inventory in store and provide location of items.

[1107] Input: List of cosmetics you need to buy.

[1108] Operation: The inventory check and guidance tool checks the store's real-time inventory database and guides the user to shelf locations.

[1109] Output: Inventory check results and shelf location information.

[1110] Step 8:

[1111] After the user actually applies the makeup, the user is given an evaluation and additional advice through a feedback means.

[1112] Input: Image data of the user's face after makeup application.

[1113] How it works: The feedback mechanism analyzes the image again and generates a makeup rating and additional recommendations.

[1114] Output: Evaluation results and further advice.

[1115] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1116] The present invention combines a system that allows a user to take a picture of their own face and suggests the optimal makeup application method with an emotion recognition means that recognizes the user's emotions. This system includes a camera means, an image analysis means, a suggestion means, a display means, a management means, a judgment means, a purchasing means, an advice means, a selection means, and an emotion recognition means. The specific operation of each means and the program processing for that purpose are explained in natural language below.

[1117] 1. Camera equipment and photography

[1118] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[1119] 2. Image Analysis

[1120] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup techniques that are suitable for the user.

[1121] 3. Emotion recognition

[1122] Furthermore, the image analysis means acquires facial expression data of the user, which is then analyzed by the emotion recognition means. The emotion recognition means identifies the current emotion (e.g., joy, sadness, anger, etc.) from the user's facial expression.

[1123] 4. Makeup tips

[1124] Based on the analyzed data, the server compares it with the information on the cosmetics the user owns and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also trend information, past makeup patterns, and emotional data obtained through emotion recognition. For example, if the server recognizes that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1125] 5. Display of makeup procedures

[1126] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[1127] 6. Cosmetics management and purchase suggestions

[1128] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[1129] 7. Post-makeup evaluation and advice

[1130] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[1131] 8. Professional makeup expert advice

[1132] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[1133] The above is the specific operation of each means in this system and the processing contents of the program.

[1134] The processing flow will be explained below.

[1135] Step 1:

[1136] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user opens the app and presses the "take a picture" button to activate the camera, and then positions themselves appropriately to take a picture.

[1137] Step 2:

[1138] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[1139] Step 3:

[1140] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[1141] Step 4:

[1142] The server checks the user's cosmetics possession status. It accesses the user's cosmetics database and retrieves the cosmetics possessed by the user. This data is taken into account in conjunction with the analysis results.

[1143] Step 5:

[1144] The (server) uses emotion recognition means to analyze the user's emotions. It analyzes the user's facial expressions from the analyzed image data and identifies their current emotional state (e.g., joy, sadness, anger, etc.).

[1145] Step 6:

[1146] The server will then suggest the optimal makeup application based on all analysis results (facial features, environmental data, and emotional data). Using an AI algorithm, it will compare the results with the user's current cosmetics and generate specific makeup steps. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1147] Step 7:

[1148] The server prepares the generated makeup steps in video, still images, and text format and sends them to the device. The server records the makeup steps as video, captures important steps as still images, and summarizes the detailed steps in text.

[1149] Step 8:

[1150] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[1151] Step 9:

[1152] The device checks the cosmetics information it has, checks whether the cosmetics required for the proposed makeup procedure are already in possession, and adds them to the list if they are missing.

[1153] Step 10:

[1154] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[1155] Step 11:

[1156] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[1157] Step 12:

[1158] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[1159] Step 13:

[1160] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[1161] Step 14:

[1162] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[1163] Step 15:

[1164] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[1165] Step 16:

[1166] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[1167] Step 17:

[1168] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[1169] Step 18:

[1170] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[1171] Step 19:

[1172] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[1173] Step 20:

[1174] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[1175] Example 2

[1176] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1177] Conventional makeup suggestion systems only consider the user's facial features and environmental data, which means they are unable to suggest makeup methods that match the user's current emotional state or mood of the day. Furthermore, there are limited ways to display makeup steps, and they lack a format that is visually easy for users to understand. Furthermore, if the required cosmetics are not available, users must manually search and purchase them, which is a time-consuming process.

[1178] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1179] In this invention, the server includes an image analysis means for analyzing the captured image, an emotion recognition means for acquiring and analyzing the user's facial expression data by the image analysis means, and a suggestion means for suggesting the most suitable makeup method to the user based on the analysis results. This makes it possible to suggest the most suitable makeup method to the user taking into consideration the user's facial features, environmental data, and the user's emotional state.

[1180] "Imaging means" refers to a device or equipment that allows a user to take a picture of their own face.

[1181] "Image analysis means" refers to a device or method that analyzes captured image data and extracts facial features and other relevant information.

[1182] "Emotion recognition means" refers to a device or method for identifying the emotional state of a user based on facial expression data acquired by image analysis means.

[1183] The "suggestion means" refers to a device or method that suggests the most suitable makeup method to the user based on the analysis results.

[1184] "Display means" refers to a device or method that visually presents the suggested makeup procedure to the user in the form of video, still images, or text.

[1185] "Information management means" refers to a device or method for managing information about cosmetics owned by a user.

[1186] "Determination means" refers to a device or method for confirming whether the cosmetics required for the proposed makeup procedure are already in possession.

[1187] "Purchase method" refers to a method that provides a link to an online retail site to facilitate purchases by users when they are out of the required cosmetics.

[1188] "Advice means" refers to a device or method that allows a user to receive additional advice after taking a photo.

[1189] "Selection means" refers to a device or method that allows a user to select advice from experts.

[1190] The present invention is a system that allows a user to take a picture of their own face and suggests the optimal makeup method. Furthermore, by combining it with an emotion recognition means that recognizes the user's emotions, it is possible to suggest makeup methods that correspond to the user's emotional state. This system includes an imaging means, an image analysis means, an emotion recognition means, a suggestion means, a display means, an information management means, a judgment means, a purchasing means, an advice means, and a selection means.

[1191] Imaging means

[1192] The user activates the camera on their smartphone or tablet device and presses the "take a photo" button to take a picture of their own face, which generates image data.

[1193] Image analysis methods

[1194] The device sends the captured image data to a server. The server uses an algorithm to analyze the received image data. For example, image analysis libraries such as OpenCV or TensorFlow are used to analyze facial features (e.g., facial contours, skin tone, eye position, etc.). Environmental data (e.g., weather, season, temperature, etc.) is also collected and used to analyze the data.

[1195] emotion recognition means

[1196] The image analysis means acquires the user's facial expression data, which is then analyzed by the emotion recognition means. An emotion recognition algorithm is used to identify the user's emotional state (e.g., joy, sadness, anger, etc.), often using an Emotion API or a deep learning model.

[1197] Proposal means

[1198] Based on the analysis results, the server compares the user's cosmetics with their current makeup product information and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also emotional data obtained through emotion recognition. For example, if the server detects that the user looks tired, it will suggest a lighter makeup look that will give a refreshing feeling.

[1199] Display means

[1200] The server prepares the generated makeup instructions in video, still images, and text format and sends them to the device. The device then displays the received information to the user. For example, the video might say, "First, apply moisturizing cream," while simultaneously displaying the same part in text.

[1201] Information management means

[1202] The device manages the cosmetics information of the user, checks the information database of the cosmetics the user owns, and determines whether the user has the necessary items.

[1203] Judgment means

[1204] Check whether the cosmetics required for the proposed makeup procedure are already in your possession and notify the user if any required items are missing.

[1205] Purchase method

[1206] If the required cosmetics are missing, the device will assist the user in the purchase process by providing a link to an online retail site, allowing the user to easily purchase the missing cosmetics.

[1207] Advice tools

[1208] After the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis results, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little too thick. Next time, try drawing it thinner."

[1209] Selection method

[1210] The user can select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server.

[1211] Specific examples and prompts for the generative AI model

[1212] For example, when a user uses this system to get ready in the morning, they follow the steps below.

[1213] 1. The user launches the app and takes a photo of their face.

[1214] 2. The server analyzes facial features and environmental data, and then recognizes emotions from the user's facial expressions.

[1215] 3. If the user feels tired, the server will suggest refreshing makeup using lighter colors.

[1216] 4. Apply makeup while watching the video to see the suggested makeup steps.

[1217] 5. If you are missing the cosmetics you need, proceed with the purchase.

[1218] 6. After your makeup is applied, you will receive evaluation and advice on how to further refine your look.

[1219] Example prompt for a generative AI model:

[1220] "Please explain how a system analyzes a user's facial photo and suggests the best makeup application. Please also explain in detail how it combines emotion recognition and suggests lighter makeup colors if the user wants to feel refreshed."

[1221] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user then presses the "take a picture" button through the app, which activates the camera. This inputs image data of the face.

[1224] Step 2:

[1225] The device sends the captured image data to the server. The image data is uploaded, and the server receives it. The input image data is then processed, and image analysis is ready.

[1226] Step 3:

[1227] The server analyzes the received image data. Using a facial recognition algorithm (e.g., OpenCV or TensorFlow), it analyzes facial features (facial contours, skin tone, eye position, etc.) and environmental data (weather information, season, etc.). This analysis outputs the user's facial feature data.

[1228] Step 4:

[1229] The server acquires the user's facial expression data using image analysis means and analyzes it using emotion recognition means. The user's emotions (e.g., joy, sadness, anger, etc.) are identified using the Emotion API and deep learning models. Emotion data is output from the input facial expression data.

[1230] Step 5:

[1231] The server calculates the optimal makeup application method (using AI models such as Keras) based on the analysis results (facial feature data and emotional data). The calculation takes into account the user's features, environmental data, and emotional data. For example, if the user is recognized as tired, a refreshing makeup look using lighter colors will be suggested. The output is suggested data for the optimal makeup application method.

[1232] Step 6:

[1233] The server generates a proposed makeup procedure in the form of video, still images, and text, and sends it to the device. The device then displays the received makeup procedure to the user. For example, a video might say, "First, apply moisturizing cream," while simultaneously displaying that part in text. The input is the proposed data, and the output is the display data of the makeup procedure.

[1234] Step 7:

[1235] The device manages the user's cosmetics information and checks whether the user has the cosmetics necessary for the proposed makeup procedure. It checks the managed cosmetics database and outputs a list of missing cosmetics.

[1236] Step 8:

[1237] The terminal assists the user in purchasing the missing cosmetics. It provides the user with a link to an online retail site, allowing them to easily purchase the necessary cosmetics. The terminal notifies the user by saying, "You are missing this lipstick. You can purchase it here." The missing cosmetics list is input, and the online link data is output.

[1238] Step 9:

[1239] After the user finishes applying makeup, they take a picture of themselves again. The user takes a new image, and the device sends it to the server. The input is the new image data.

[1240] Step 10:

[1241] The server analyzes the image again and generates makeup evaluation and advice based on the new analysis results. For example, it may generate specific feedback such as, "Your eyeliner is a little thick, so next time, try drawing it thinner." The input is the new image data, and the output is the evaluation and advice data.

[1242] Step 11:

[1243] When a user selects advice from a professional makeup expert, the device notifies the server of the selection. The server sends the user's image data to the expert, who then creates customized advice based on the image. The data is then sent to the device via the server. The input is the user's selection information and image data, and the output is the expert's advice.

[1244] (Application example 2)

[1245] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1246] Conventional makeup suggestion systems suggest optimal makeup methods based on the user's facial features and environmental data. However, these systems do not take into account the user's current emotional state, making it difficult to provide makeup suggestions that match the user's feelings and mood. Furthermore, because the system does not suggest makeup methods that correspond to emotions, it is difficult to increase user satisfaction.

[1247] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the user's face, an analysis means for analyzing the captured image, an emotion recognition means for recognizing emotions, and a suggestion means for suggesting an optimal makeup method to the user based on the analysis results and emotion data. This makes it possible to suggest a makeup method based on the user's emotional state.

[1248] "Photography means" refers to a device or function that allows a user to take pictures or videos of themselves.

[1249] "Analysis means" refers to a device or function for analyzing facial features and environmental information based on captured image and video data.

[1250] The "suggestion means" is a device or function that calculates and suggests the most suitable makeup method for the user based on the analyzed data.

[1251] The "display means" is a device or function for displaying the proposed makeup procedure to the user in the form of video, still images, or text.

[1252] The "management means" refers to a device or function for centrally managing information about cosmetics owned by a user.

[1253] The "determination means" is a device or function that determines whether the cosmetics required for the proposed makeup procedure are in the user's possession or need to be purchased.

[1254] "Purchase means" refers to a device or function that links cosmetics to be purchased to an online purchasing site, allowing users to easily purchase them.

[1255] An "advice means" is a device or function that provides additional advice or evaluation to the user after the user has finished applying makeup.

[1256] The "selection means" is a device or function that allows the user to select and receive advice from an expert.

[1257] "Emotion recognition means" refers to a device or function for analyzing and recognizing the current emotion of the user from their facial expression.

[1258] The present invention is a system that allows users to take a picture of their own face, analyzes the image data, and provides optimal makeup methods, and also adds a function to recognize the user's emotions. This system takes into account the user's facial features, environmental information, and current emotional state to propose more detailed and personalized makeup methods.

[1259] Hardware and software used

[1260] This system uses the following hardware and software:

[1261] Photography method: Use the built-in camera on a smartphone or tablet device or a webcam connected to a computer.

[1262] Analysis method: DeepFace library for analyzing facial features and environmental information.

[1263] Emotion recognizer: An emotion recognition algorithm to recognize user emotions (e.g., a Python emotion recognition library).

[1264] Suggestion method: A makeup suggestion algorithm that runs on the server side. Data is exchanged via REST API.

[1265] Display means: Smartphone or tablet display, and dedicated mobile application.

[1266] Management means: A database (e.g., SQLite) for managing the cosmetic data owned by the user.

[1267] Determination method: An algorithm to determine whether the required cosmetics are in possession.

[1268] Purchase method: A function to link the cosmetics to be purchased to an online shop (e.g., integration with a shopping API).

[1269] Advice tool: An algorithm that analyzes re-photograph data after makeup application and provides additional advice.

[1270] Selection tool: A user interface that allows the selection of expert advice.

[1271] System operation explanation

[1272] Filming method

[1273] Users take a photo of their face using the camera on their smartphone or tablet device by launching the camera app and pressing the "take photo" button.

[1274] Analysis means

[1275] The device sends the captured image data to a server and analyzes it using the DeepFace library to extract facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.).

[1276] emotion recognition means

[1277] The image analysis means acquires facial expression data of the user, and the emotion recognition means analyzes this to identify the user's current emotion (for example, joy, sadness, anger, etc.).

[1278] Proposal means

[1279] The server then uses the analysis data to suggest makeup applications. These suggestions take into account not only the user's facial features and environmental data, but also their emotional data. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1280] Display means

[1281] The server sends the generated makeup instructions in the form of video, still images, and text to the device, which then displays them to the user. The display includes detailed instructions in video, still images, and text.

[1282] Examples of concrete examples and prompts

[1283] Below are some example prompts to be input to the generative AI model:

[1284] Develop a system that takes a picture of a user's face, analyzes the image to recognize emotions, and suggests the best makeup application based on the user's emotions. Follow these steps:

[1285] 1. Activate the camera and take a picture of the user's face.

[1286] 2. Analyze the captured image to identify facial features and emotions (happiness, sadness, anger, etc.).

[1287] 3. Send facial features and emotion data to an external API and receive the optimal makeup application.

[1288] 4. The received makeup method is displayed to the user.

[1289] As described above, the system for implementing this invention is a multifunctional system that proposes the optimal makeup method taking into consideration the user's facial features and emotions. This system allows the user to apply makeup that suits their emotional state, thereby increasing satisfaction.

[1290] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1291] Step 1:

[1292] The user activates the camera on their smartphone or tablet and takes a picture of their own face. At this time, the camera is activated by pressing the "shoot" button in the application, and the user takes a picture in an appropriate position. The input is the user's facial image, which is output as image data.

[1293] Step 2:

[1294] The device receives the captured image data and sends it to the server. The input is the image data, which is sent to the server via the network. The output is the image data received by the server.

[1295] Step 3:

[1296] The server analyzes the received image data. For this analysis, it uses the DeepFace library to analyze the user's facial features (face shape, skin tone, eye position, etc.). The input is the received image data, and the output is facial feature data. Specifically, the DeepFace library maps each point on the face and extracts feature data.

[1297] Step 4:

[1298] The server then uses emotion recognition means to recognize the user's emotions from the analyzed feature data. This process uses an emotion recognition algorithm (for example, a Python emotion recognition library). The input is facial feature data, and the output is emotion data. Specifically, it analyzes changes in facial expression (for example, eyebrow movement, mouth corner position) to identify emotions such as joy, sadness, and anger.

[1299] Step 5:

[1300] The server then uses the analyzed data (facial features and emotion data) to suggest the most suitable makeup application method for the user. This proposal is made using a makeup application suggestion algorithm on the server. The input is facial feature data and emotion data, and the output is suggested data for makeup procedures. For example, if the emotion is recognized as "fatigue," the server will suggest a makeup method to make the user's face look brighter.

[1301] Step 6:

[1302] The server prepares the generated makeup steps in video, still image, and text formats and sends them to the terminal. The input is the proposed makeup steps data, and the output is the terminal that receives it. Specifically, the server converts the makeup steps into multiple formats (video file, still image file, text file) and sends them all at once to the terminal.

[1303] Step 7:

[1304] The device then displays the received makeup instructions to the user. The input is the received makeup instructions data, and the output is a visual representation of the makeup instructions that the user can confirm. Specifically, the device application plays the video and highlights important parts with still images and text.

[1305] Step 8:

[1306] The user applies makeup according to the suggested makeup steps, then takes a photo of their face again. The input is the face image after makeup, and the output is the image data. Specifically, the user starts the camera again and presses the "take another photo" button to take a photo of their face.

[1307] Step 9:

[1308] The device sends the face image data after makeup application to the server, which then analyzes it again. The input is the re-photographed image data, and the output is the analysis results. Specifically, the server executes the process described above again to evaluate the face after makeup application.

[1309] Step 10:

[1310] The server generates additional advice based on the reanalysis results and sends it to the terminal. The input is the reanalysis results, and the output is additional advice data. Specific actions include providing specific feedback such as "Draw your eyeliner a little thinner" if the eyeliner is too thick.

[1311] Step 11:

[1312] The terminal displays the received additional advice to the user. The input is the received additional advice data, and the output is an advice display that the user can visually confirm. Specifically, the advice text or a still image is displayed on the terminal screen.

[1313] The above are the specific processing steps for carrying out the present invention. The user can receive personalized makeup suggestions that take into consideration their emotions, which can improve satisfaction.

[1314] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1315] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1316] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1317] [Fourth embodiment]

[1318] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1319] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1320] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1321] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1322] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1323] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1324] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1325] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1326] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1327] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1328] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1329] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1330] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1331] The present invention is a system that allows a user to photograph their own face and suggests the optimal makeup application method. This system includes a camera, an image analysis unit, a suggestion unit, a display unit, a management unit, a judgment unit, a purchasing unit, an advice unit, and a selection unit. The specific operation of each unit and the program processing for that purpose are explained in natural language below.

[1332] 1. Camera equipment and photography

[1333] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[1334] 2. Image Analysis

[1335] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup methods that are suitable for the user.

[1336] 3. Makeup tips

[1337] The server uses the analyzed data to calculate the optimal makeup application, taking into account the user's current cosmetics and personal information. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[1338] 4. Display of makeup procedures

[1339] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[1340] 5. Cosmetics management and purchase suggestions

[1341] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[1342] 6. Post-makeup evaluation and advice

[1343] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[1344] 7. Professional makeup expert advice

[1345] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[1346] The above is the specific operation of each means in this system and the processing contents of the program.

[1347] The processing flow will be explained below.

[1348] Step 1:

[1349] (User) activates the smartphone camera and takes a picture of their own face. By opening the app and pressing the "take a picture" button, the camera is activated and the user takes a picture of their own face.

[1350] Step 2:

[1351] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[1352] Step 3:

[1353] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[1354] Step 4:

[1355] The server checks the user's cosmetics possession status, accesses the user's cosmetics database, and retrieves the cosmetics possessed by the user.

[1356] Step 5:

[1357] The server will suggest the optimal makeup application. Based on facial feature data, environmental data, and information on the cosmetics you own, the AI ​​algorithm will generate optimal makeup suggestions. For example, if the summer sun is strong, the server will suggest makeup that emphasizes UV protection.

[1358] Step 6:

[1359] The server generates the proposed makeup steps in video, still images, and text format. It records the makeup steps as a video, captures important steps as still images, and summarizes the detailed steps in text.

[1360] Step 7:

[1361] The server sends the generated makeup instructions to the device. The generated videos, still images, and text data are sent to the device and presented in a visually easy-to-understand format.

[1362] Step 8:

[1363] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[1364] Step 9:

[1365] The device checks the user's cosmetic information, checks whether the user already has the cosmetics required for the proposed makeup routine, and adds them to the list if they are missing.

[1366] Step 10:

[1367] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[1368] Step 11:

[1369] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[1370] Step 12:

[1371] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[1372] Step 13:

[1373] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[1374] Step 14:

[1375] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[1376] Step 15:

[1377] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[1378] Step 16:

[1379] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[1380] Step 17:

[1381] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[1382] Step 18:

[1383] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[1384] Step 19:

[1385] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[1386] Step 20:

[1387] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[1388] Example 1

[1389] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] Conventional cosmetics recommendation systems lack the ability to suggest optimal makeup applications based on the user's facial features and environmental data. Furthermore, there is no way for users to verify the consistency of the recommended cosmetics with the cosmetics they currently own, making the process of purchasing the necessary cosmetics cumbersome. Furthermore, the lack of a mechanism for providing evaluations of the makeup application or additional advice makes it difficult for users to improve their makeup techniques.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1392] In this invention, the server includes a photographing device means for allowing the user to photograph themselves, an image analysis device means for analyzing the photographed image, a suggestion device means for suggesting the optimal makeup regimen for the user based on the analysis results, a display device means for displaying the suggested makeup regimen in video, still image, and text format, a management function means for managing information on cosmetics owned by the user, a determination device means for determining whether the cosmetics required for the suggested makeup regimen are already owned or need to be purchased, a purchase device means for linking cosmetics that need to be purchased to an electronic shopping site, an advice device means for providing additional advice to the user after photographing, and a selection device means for selecting advice from experts. This makes it possible to suggest the optimal makeup regimen based on the user's characteristics and environmental data, simplifies the process of checking the compatibility of the necessary cosmetics and purchasing them, and enables the provision of evaluations after makeup application and additional advice.

[1393] The "photographing device means" is a function including a device that allows the user to take a picture of their own face.

[1394] The "image analysis device means" is a function that analyzes the captured image and extracts the user's facial features and environmental data.

[1395] The "suggestion device means" is a function that provides the user with the most suitable makeup method based on the analysis results.

[1396] The "display device" is a function that displays the proposed makeup procedure to the user in the form of video, still images, or text.

[1397] The "management function means" is a function for managing the cosmetic information owned by the user.

[1398] The "determination device means" is a function that determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased.

[1399] The "purchase device means" is a function that links cosmetics that need to be purchased to an electronic commerce site.

[1400] The "advice device means" is a function that provides additional advice to the user after taking a photo.

[1401] The "selection device means" is a function that allows the user to select advice from experts.

[1402] This invention is a system that allows a user to take a picture of their own face and suggests the best makeup application method. The system includes the following means:

[1403] photographing device means

[1404] Image analysis device means

[1405] Proposed device means

[1406] display means

[1407] management function means

[1408] Judgment device means

[1409] Purchasing Device Means

[1410] Advice Device Means

[1411] Selector means

[1412] Users take a photo of their face using the camera on their smartphone or tablet device. To do this, they press the "take a photo" button through the app, which activates the camera and takes a photo from an appropriate position. The device then sends the captured image data to the server. The server then uses an algorithm to analyze the received image data. Specifically, the server analyzes the user's facial features (e.g., face shape, skin tone, eye position) and environmental data (e.g., season, weather, temperature). Based on this analytical data, the system prepares to suggest makeup methods that are suitable for the user.

[1413] The server then uses the analyzed data to calculate the optimal makeup application, taking into account the user's current makeup habits and personal characteristics. This calculation takes into account not only the user's characteristics and environmental data, but also trends and past makeup patterns. For example, if the summer sun is strong, the server will suggest sunscreen and cool-toned eyeshadow to protect against UV rays.

[1414] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as "First, apply moisturizing cream, then apply liquid foundation," as well as playing back the specific makeup steps in video format.

[1415] The terminal manages information about cosmetics owned by the user. The determination device checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the terminal adds them to a list and provides a link to a shopping site, allowing the user to purchase the required cosmetics with one click.

[1416] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis, the app evaluates the makeup and generates additional advice. For example, it might provide specific feedback like, "Your eyeliner is a little too thick, so if you make it a little thinner it will balance things out better."

[1417] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. In this way, the user can receive more professional advice.

[1418] For example, the following prompts can be fed into a generative AI model to get further advice:

[1419] "I'm a woman in my 20s. I'd like to know how to apply makeup during the day in the summer. My skin tends to be dry, so I want to make my body look slimmer. The cosmetics I currently use are sunscreen, liquid foundation, powder foundation, brush-on bronzer, mascara, eyeliner, and lip tint."

[1420] This invention allows users to easily find the best makeup method for their face, and efficiently select and purchase the necessary cosmetics. In addition, users can improve their makeup techniques by receiving further advice through evaluations after makeup application.

[1421] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1422] Program processing flow

[1423] Step 1: Capture a photo of the user's face

[1424] Input: Smartphone or tablet camera

[1425] Output: Image data of the captured face

[1426] Specific behavior:

[1427] 1. The user launches the app.

[1428] 2. The user presses the "take a photo" button.

[1429] 3. The device starts the camera and the user takes a picture of their face in an appropriate position.

[1430] 4. The device generates image data of the captured face.

[1431] Step 2: Sending image data

[1432] Input: Image data of a photographed face

[1433] Output: Image data sent to the server

[1434] Specific behavior:

[1435] 1. The device sends the captured image data to the server.

[1436] 2. The server receives the image data through a receiving protocol.

[1437] Step 3: Image analysis

[1438] Input: Image data sent to the server

[1439] Output: Facial feature data, environmental data

[1440] Specific behavior:

[1441] 1. The server applies an image analysis algorithm (e.g., a deep learning model).

[1442] 2. Analyze facial features (e.g., face shape, skin tone, eye position).

[1443] 3. Reference environmental data (e.g., season, weather, temperature) to complement the analysis results.

[1444] Step 4: Makeup tips

[1445] Input: Facial feature data and environmental data

[1446] Output: Data on optimal makeup application

[1447] Specific behavior:

[1448] 1. The server uses the analysis data to calculate the optimal makeup application method in comparison with the cosmetic information it has.

[1449] 2. Consider user characteristics, environmental data, trend information, past makeup patterns, etc.

[1450] 3. Generate data on optimal makeup application methods.

[1451] Step 5: Send and view your cosmetic procedure

[1452] Input: Data on optimal makeup application

[1453] Output: Makeup procedure (video, still image, text)

[1454] Specific behavior:

[1455] 1. The server sends the generated makeup procedure to the terminal.

[1456] 2. The terminal analyzes the received instructions and displays them on the user interface.

[1457] 3. Provide makeup instructions in the form of video playback, still image display, and text display.

[1458] Step 6: Cosmetics management and purchasing suggestions

[1459] Input: Cosmetic information owned by the user, suggested makeup procedures

[1460] Output: List of cosmetics to purchase, purchase link

[1461] Specific behavior:

[1462] 1. The device references the user's cosmetics database and matches the cosmetics required for the proposed makeup procedure.

[1463] 2. If you don't have any necessary cosmetics, make a list of them.

[1464] 3. Provide a link to an e-commerce site for the cosmetics you need to purchase.

[1465] Step 7: Post-makeup evaluation and advice

[1466] Input: Image data of face after makeup

[1467] Output: Makeup evaluation data, additional advice

[1468] Specific behavior:

[1469] 1. The user takes another photo of their face after applying makeup.

[1470] 2. The device sends the new image to the server.

[1471] 3. The server analyzes the image again and generates a makeup evaluation.

[1472] 4. Generate additional advice and send it to the device.

[1473] Step 8: Expert advice

[1474] Input: User selection, facial image data

[1475] Output: Expert advice

[1476] Specific behavior:

[1477] 1. The user selects advice from a professional makeup expert.

[1478] 2. The device notifies the server of its choice.

[1479] 3. The server sends the user's image data to the expert.

[1480] 4. The expert creates customized advice and sends it to the device via the server.

[1481] 5. Users receive expert advice.

[1482] (Application example 1)

[1483] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1484] Conventional makeup advice systems are primarily designed for use on digital devices and lack the ability to provide real-time product recommendations and inventory checks in a physical store environment. Furthermore, users often have to put in significant effort to find the perfect cosmetics in-store, and sometimes certain products are out of stock, creating an inconvenient shopping experience. A new system is needed to address these issues.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1486] In this invention, the server includes a photographing means for the user to photograph themselves, an analysis means for analyzing the photographed image, a suggestion means for suggesting the optimal makeup method to the user based on the analysis results, a display means for displaying the suggested makeup procedure in video, still image, or text format, a management means for managing information on items owned by the user, a determination means for determining whether the user already owns or needs to purchase items required for the suggested makeup procedure, a purchase means for linking items that need to be purchased to a shopping site, an advice means for providing additional advice to the user after photographing, an inventory confirmation and guidance means for checking inventory in the store and guiding the user to the location of items, and a feedback means for providing interactive feedback linked to the real world. This allows the user to find cosmetics in the store in real time and receive detailed suggestions on the optimal makeup method.

[1487] The "photographing means" is a device that allows the user to photograph their own face.

[1488] The "analysis means" is a device or software that analyzes the captured image data and extracts the user's facial features and environmental data.

[1489] The "suggestion means" is a device or software that provides the user with the most suitable makeup method based on the data obtained by the analysis means.

[1490] The "display means" is a device or software that presents the suggested makeup procedure to the user in the form of video, still images, or text.

[1491] The "management means" refers to a device or software that centrally manages information about items (cosmetics) owned by the user.

[1492] The "determination means" is a device or software that determines whether the user already possesses the items necessary for the proposed cosmetic procedure or whether new purchases are required.

[1493] "Purchase Instrument" refers to a device or software that allows the user to purchase the suggested items as needed through a link to a shopping site.

[1494] An "advice means" is a device or software that provides additional makeup advice to a user after the user has taken a selfie.

[1495] The "inventory checking and guidance means" is a device or software that checks the inventory in the store and guides the user to the location of the required item.

[1496] A "feedback tool" is a device or software that provides real-time interactive feedback and evaluations to users after they have actually used a product.

[1497] This invention provides a system that allows users to use smart glasses in a physical store and receive recommendations on optimal makeup application methods. The system includes a photographing unit, an analysis unit, a recommendation unit, a display unit, a management unit, a determination unit, a purchasing unit, an advice unit, an inventory confirmation and guidance unit, and a feedback unit.

[1498] First, the user puts on the smart glasses in a physical store and takes a picture of their face. The image captured by this photographing means is sent to the analyzing means, which then analyzes facial features and environmental data using machine learning models such as TensorFlow. This analyzed data is then used by the suggesting means to suggest the most suitable makeup application to the user.

[1499] The suggestion unit suggests the most suitable cosmetics based on the user's facial features and environmental data. For example, it may recommend the most suitable foundation or eyeshadow depending on the user's skin tone, the season, and the lighting conditions in the store. The suggestion is displayed on the smart glasses' display via the display unit in the form of video, still images, or text.

[1500] The management means manages information about cosmetics owned by the user through the smart glasses and related applications. The determination means determines whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased. If necessary, the purchase means provides a link to a related online shopping site to facilitate easy purchase.

[1501] Furthermore, the inventory check and guidance module checks the inventory in the physical store and guides the user to the location of the necessary items. This information is linked to the store's real-time inventory database. After the user actually applies the makeup, the feedback module provides real-time interactive evaluation and additional makeup advice.

[1502] As a concrete example, imagine a user puts on smart glasses in a physical store and scans their face. The system analyzes the user's face and suggests the best foundation and eyeshadow based on their skin tone. These suggestions are played back as a video on the glasses' display, along with detailed instructions on how to use the product. The system also displays the shelf location, allowing users to easily find the product without getting lost. Furthermore, with one click, users can access an online shopping site for the product they need, and if it's not in stock at the store, they can have it ordered from a nearby store or online store.

[1503] The system allows users to receive detailed real-time suggestions on how to best apply makeup through the smart glasses, and also significantly improves the in-store shopping experience.

[1504] Examples of prompts powered by generative AI models include:

[1505] "Develop an app that takes a picture of the user's face with a camera in smart glasses and suggests the best makeup look for them, taking into account their facial features, skin tone, eye position, and in-store environmental data (lighting, climate, temperature). It also checks the stock availability of cosmetics used in the suggested makeup look and provides real-time feedback to the user."

[1506] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1507] Step 1:

[1508] The user puts on the smart glasses in a physical store and has their face photographed.

[1509] Input: Image data of the user's face captured by the camera built into the smart glasses.

[1510] Action: The capture device activates the camera and prompts the user to position their face appropriately.

[1511] Output: The captured image data is stored in the smart glasses.

[1512] Step 2:

[1513] The captured image is sent to a server for analysis.

[1514] Input: Image data of the user's face.

[1515] How it works: Image data is sent from the smart glasses to a server, and the analysis means analyzes the image using a machine learning model such as TensorFlow.

[1516] Output: Facial feature data and environmental data (skin tone, eye position, lighting conditions, etc.) are generated.

[1517] Step 3:

[1518] Based on the analysis results, the suggestion method calculates the optimal makeup method.

[1519] Input: Facial feature data and environmental data.

[1520] How it works: The proposed method uses a generative AI model to calculate the optimal makeup routine for the user.

[1521] Output: Makeup application suggestion data (video, still images, text format) is generated.

[1522] Step 4:

[1523] The generated makeup procedure is displayed on the smart glasses.

[1524] Input: Makeup method suggestion data.

[1525] Operation: The display means displays makeup instructions in video, still images, and text format on the display of the smart glasses.

[1526] Output: A state in which the user can visually confirm the makeup procedure.

[1527] Step 5:

[1528] The cosmetic information possessed by the user is confirmed by the management means.

[1529] Input: A list of cosmetics owned by the user.

[1530] Operation: The control unit compares the list of cosmetics on hand with the cosmetics required for the proposed makeup procedure.

[1531] Output: The result of determining whether the item is already in possession or needs to be purchased.

[1532] Step 6:

[1533] Link missing cosmetics to purchasing options.

[1534] Input: A list of cosmetics that need to be purchased according to the determination means.

[1535] Operation: The purchasing instrument generates and provides a link to an online shopping site to the user.

[1536] Output: Purchase link information.

[1537] Step 7:

[1538] Check inventory in store and provide location of items.

[1539] Input: List of cosmetics you need to buy.

[1540] Operation: The inventory check and guidance tool checks the store's real-time inventory database and guides the user to shelf locations.

[1541] Output: Inventory check results and shelf location information.

[1542] Step 8:

[1543] After the user actually applies the makeup, the user is given an evaluation and additional advice through a feedback means.

[1544] Input: Image data of the user's face after makeup application.

[1545] How it works: The feedback mechanism analyzes the image again and generates a makeup rating and additional recommendations.

[1546] Output: Evaluation results and further advice.

[1547] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1548] The present invention combines a system that allows a user to take a picture of their own face and suggests the optimal makeup application method with an emotion recognition means that recognizes the user's emotions. This system includes a camera means, an image analysis means, a suggestion means, a display means, a management means, a judgment means, a purchasing means, an advice means, a selection means, and an emotion recognition means. The specific operation of each means and the program processing for that purpose are explained in natural language below.

[1549] 1. Camera equipment and photography

[1550] First, the user activates the camera on their smartphone or tablet and takes a picture of their face. The camera is activated by pressing the "shoot" button through the app, and the user positions themselves appropriately to take the picture.

[1551] 2. Image Analysis

[1552] Next, the (device) sends the captured image data to the (server). The (server) uses an algorithm to analyze the received image data. Specifically, it analyzes facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.). Based on this analysis data, it prepares to suggest makeup techniques that are suitable for the user.

[1553] 3. Emotion recognition

[1554] Furthermore, the image analysis means acquires facial expression data of the user, which is then analyzed by the emotion recognition means. The emotion recognition means identifies the current emotion (e.g., joy, sadness, anger, etc.) from the user's facial expression.

[1555] 4. Makeup tips

[1556] Based on the analyzed data, the server compares it with the information on the cosmetics the user owns and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also trend information, past makeup patterns, and emotional data obtained through emotion recognition. For example, if the server recognizes that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1557] 5. Display of makeup procedures

[1558] The server prepares the generated makeup steps in the form of video, still images, and text, and sends them to the device. The device then displays the received information to the user. This display includes detailed instructions such as playing back specific makeup steps in video, and emphasizing important points with still images and text. For example, the steps may be explained in detail, such as "First, apply moisturizing cream, then apply liquid foundation."

[1559] 6. Cosmetics management and purchase suggestions

[1560] The (terminal) manages information about cosmetics owned by the user. The (determination means) checks whether the cosmetics required for the proposed makeup procedure are already owned. If the required cosmetics are not owned, the (terminal) adds them to a list and provides a link to a shopping site. This allows the (user) to purchase the necessary cosmetics with one click.

[1561] 7. Post-makeup evaluation and advice

[1562] Once the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the results of this analysis, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little thick, so if you make it a little thinner it will balance things out better."

[1563] 8. Professional makeup expert advice

[1564] The user can also select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server. This allows the user to receive more professional advice.

[1565] The above is the specific operation of each means in this system and the processing contents of the program.

[1566] The processing flow will be explained below.

[1567] Step 1:

[1568] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user opens the app and presses the "take a picture" button to activate the camera, and then positions themselves appropriately to take a picture.

[1569] Step 2:

[1570] The (terminal) sends the captured image data to the server. Once the captured image is ready to be sent to the server via the Internet, the device executes the transmission. Once the transmission is complete, the device waits for a response from the server.

[1571] Step 3:

[1572] The server analyzes the received image data and uses an image analysis algorithm to detect facial features (face shape, skin tone, eye position, etc.). It also calls an environmental data API to obtain the current season, weather, and temperature.

[1573] Step 4:

[1574] The server checks the user's cosmetics possession status. It accesses the user's cosmetics database and retrieves the cosmetics possessed by the user. This data is taken into account in conjunction with the analysis results.

[1575] Step 5:

[1576] The (server) uses emotion recognition means to analyze the user's emotions. It analyzes the user's facial expressions from the analyzed image data and identifies their current emotional state (e.g., joy, sadness, anger, etc.).

[1577] Step 6:

[1578] The server will then suggest the optimal makeup application based on all analysis results (facial features, environmental data, and emotional data). Using an AI algorithm, it will compare the results with the user's current cosmetics and generate specific makeup steps. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1579] Step 7:

[1580] The server prepares the generated makeup steps in video, still images, and text format and sends them to the device. The server records the makeup steps as video, captures important steps as still images, and summarizes the detailed steps in text.

[1581] Step 8:

[1582] The device displays the received makeup instructions to the user. The app plays a video of the makeup instructions and visually displays still images and text. For example, while the video is playing, the device displays text such as "First, apply moisturizing cream."

[1583] Step 9:

[1584] The device checks the cosmetics information it has, checks whether the cosmetics required for the proposed makeup procedure are already in possession, and adds them to the list if they are missing.

[1585] Step 10:

[1586] (Device) displays the cosmetics that are in short supply by linking with a shopping site, and displays a purchase link for the cosmetics that are in short supply, allowing users to purchase them with one click.

[1587] Step 11:

[1588] The user follows the suggested makeup steps to apply makeup. Follow the steps to complete the makeup.

[1589] Step 12:

[1590] (User) takes a photo of themselves again after applying makeup. After the makeup is complete, they turn on the camera and take another photo of their face.

[1591] Step 13:

[1592] (Device) sends a new image to the server. Sends the image data after makeup to the server and requests a re-evaluation.

[1593] Step 14:

[1594] The server analyzes the image after makeup is applied and evaluates it. It then analyzes the image again to evaluate the makeup condition. For example, it checks the thickness and evenness of the eyeliner.

[1595] Step 15:

[1596] The server generates additional advice and sends it to the device. For example, it might say, "Your eyeliner is a little thick, so if you make it a little thinner it will look more balanced."

[1597] Step 16:

[1598] (Device) displays additional advice to the user. The app displays additional feedback and advice to help the user adjust their makeup.

[1599] Step 17:

[1600] Users also have the option to choose advice from a professional makeup expert by selecting "Get Professional Advice" from the app options.

[1601] Step 18:

[1602] The device notifies the server of this selection, and the server sends the user's image data to the expert, who then creates customized advice based on the image and sends it back to the server.

[1603] Step 19:

[1604] (Server) receives advice from experts and sends it to the terminal. Receive customized advice from experts.

[1605] Step 20:

[1606] The user applies makeup based on professional advice and makes final adjustments, allowing the user to receive expert advice and achieve the makeup look that best suits them.

[1607] Example 2

[1608] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1609] Conventional makeup suggestion systems only consider the user's facial features and environmental data, which means they are unable to suggest makeup methods that match the user's current emotional state or mood of the day. Furthermore, there are limited ways to display makeup steps, and they lack a format that is visually easy for users to understand. Furthermore, if the required cosmetics are not available, users must manually search and purchase them, which is a time-consuming process.

[1610] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1611] In this invention, the server includes an image analysis means for analyzing the captured image, an emotion recognition means for acquiring and analyzing the user's facial expression data by the image analysis means, and a suggestion means for suggesting the most suitable makeup method to the user based on the analysis results. This makes it possible to suggest the most suitable makeup method to the user taking into consideration the user's facial features, environmental data, and the user's emotional state.

[1612] "Imaging means" refers to a device or equipment that allows a user to take a picture of their own face.

[1613] "Image analysis means" refers to a device or method that analyzes captured image data and extracts facial features and other relevant information.

[1614] "Emotion recognition means" refers to a device or method for identifying the emotional state of a user based on facial expression data acquired by image analysis means.

[1615] The "suggestion means" refers to a device or method that suggests the most suitable makeup method to the user based on the analysis results.

[1616] "Display means" refers to a device or method that visually presents the suggested makeup procedure to the user in the form of video, still images, or text.

[1617] "Information management means" refers to a device or method for managing information about cosmetics owned by a user.

[1618] "Determination means" refers to a device or method for confirming whether the cosmetics required for the proposed makeup procedure are already in possession.

[1619] "Purchase method" refers to a method that provides a link to an online retail site to facilitate purchases by users when they are out of the required cosmetics.

[1620] "Advice means" refers to a device or method that allows a user to receive additional advice after taking a photo.

[1621] "Selection means" refers to a device or method that allows a user to select advice from experts.

[1622] The present invention is a system that allows a user to take a picture of their own face and suggests the optimal makeup method. Furthermore, by combining it with an emotion recognition means that recognizes the user's emotions, it is possible to suggest makeup methods that correspond to the user's emotional state. This system includes an imaging means, an image analysis means, an emotion recognition means, a suggestion means, a display means, an information management means, a judgment means, a purchasing means, an advice means, and a selection means.

[1623] Imaging means

[1624] The user activates the camera on their smartphone or tablet device and presses the "take a photo" button to take a picture of their own face, which generates image data.

[1625] Image analysis methods

[1626] The device sends the captured image data to a server. The server uses an algorithm to analyze the received image data. For example, image analysis libraries such as OpenCV or TensorFlow are used to analyze facial features (e.g., facial contours, skin tone, eye position, etc.). Environmental data (e.g., weather, season, temperature, etc.) is also collected and used to analyze the data.

[1627] emotion recognition means

[1628] The image analysis means acquires the user's facial expression data, which is then analyzed by the emotion recognition means. An emotion recognition algorithm is used to identify the user's emotional state (e.g., joy, sadness, anger, etc.), often using an Emotion API or a deep learning model.

[1629] Proposal means

[1630] Based on the analysis results, the server compares the user's cosmetics with their current makeup product information and calculates the optimal makeup application. This calculation takes into account not only the user's characteristics and environmental data, but also emotional data obtained through emotion recognition. For example, if the server detects that the user looks tired, it will suggest a lighter makeup look that will give a refreshing feeling.

[1631] Display means

[1632] The server prepares the generated makeup instructions in video, still images, and text format and sends them to the device. The device then displays the received information to the user. For example, the video might say, "First, apply moisturizing cream," while simultaneously displaying the same part in text.

[1633] Information management means

[1634] The device manages the cosmetics information of the user, checks the information database of the cosmetics the user owns, and determines whether the user has the necessary items.

[1635] Judgment means

[1636] Check whether the cosmetics required for the proposed makeup procedure are already in your possession and notify the user if any required items are missing.

[1637] Purchase method

[1638] If the required cosmetics are missing, the device will assist the user in the purchase process by providing a link to an online retail site, allowing the user to easily purchase the missing cosmetics.

[1639] Advice tools

[1640] After the user has finished applying their makeup, they take another photo of themselves. The device then sends the new image to the server, which analyzes it again. Based on the analysis results, the system generates an evaluation of the makeup and additional advice. For example, the system provides specific feedback such as, "Your eyeliner is a little too thick. Next time, try drawing it thinner."

[1641] Selection method

[1642] The user can select advice from a professional makeup expert. In this case, the device notifies the server of the selection, and the server sends the user's image data to the expert. The expert creates customized makeup advice based on the image and sends it to the device via the server.

[1643] Specific examples and prompts for the generative AI model

[1644] For example, when a user uses this system to get ready in the morning, they follow the steps below.

[1645] 1. The user launches the app and takes a photo of their face.

[1646] 2. The server analyzes facial features and environmental data, and then recognizes emotions from the user's facial expressions.

[1647] 3. If the user feels tired, the server will suggest refreshing makeup using lighter colors.

[1648] 4. Apply makeup while watching the video to see the suggested makeup steps.

[1649] 5. If you are missing the cosmetics you need, proceed with the purchase.

[1650] 6. After your makeup is applied, you will receive evaluation and advice on how to further refine your look.

[1651] Example prompt for a generative AI model:

[1652] "Please explain how a system analyzes a user's facial photo and suggests the best makeup application. Please also explain in detail how it combines emotion recognition and suggests lighter makeup colors if the user wants to feel refreshed."

[1653] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1654] Step 1:

[1655] The user activates the camera on their smartphone or tablet and takes a picture of their face. The user then presses the "take a picture" button through the app, which activates the camera. This inputs image data of the face.

[1656] Step 2:

[1657] The device sends the captured image data to the server. The image data is uploaded, and the server receives it. The input image data is then processed, and image analysis is ready.

[1658] Step 3:

[1659] The server analyzes the received image data. Using a facial recognition algorithm (e.g., OpenCV or TensorFlow), it analyzes facial features (facial contours, skin tone, eye position, etc.) and environmental data (weather information, season, etc.). This analysis outputs the user's facial feature data.

[1660] Step 4:

[1661] The server acquires the user's facial expression data using image analysis means and analyzes it using emotion recognition means. The user's emotions (e.g., joy, sadness, anger, etc.) are identified using the Emotion API and deep learning models. Emotion data is output from the input facial expression data.

[1662] Step 5:

[1663] The server calculates the optimal makeup application method (using AI models such as Keras) based on the analysis results (facial feature data and emotional data). The calculation takes into account the user's features, environmental data, and emotional data. For example, if the user is recognized as tired, a refreshing makeup look using lighter colors will be suggested. The output is suggested data for the optimal makeup application method.

[1664] Step 6:

[1665] The server generates a proposed makeup procedure in the form of video, still images, and text, and sends it to the device. The device then displays the received makeup procedure to the user. For example, a video might say, "First, apply moisturizing cream," while simultaneously displaying that part in text. The input is the proposed data, and the output is the display data of the makeup procedure.

[1666] Step 7:

[1667] The device manages the user's cosmetics information and checks whether the user has the cosmetics necessary for the proposed makeup procedure. It checks the managed cosmetics database and outputs a list of missing cosmetics.

[1668] Step 8:

[1669] The terminal assists the user in purchasing the missing cosmetics. It provides the user with a link to an online retail site, allowing them to easily purchase the necessary cosmetics. The terminal notifies the user by saying, "You are missing this lipstick. You can purchase it here." The missing cosmetics list is input, and the online link data is output.

[1670] Step 9:

[1671] After the user finishes applying makeup, they take a picture of themselves again. The user takes a new image, and the device sends it to the server. The input is the new image data.

[1672] Step 10:

[1673] The server analyzes the image again and generates makeup evaluation and advice based on the new analysis results. For example, it may generate specific feedback such as, "Your eyeliner is a little thick, so next time, try drawing it thinner." The input is the new image data, and the output is the evaluation and advice data.

[1674] Step 11:

[1675] When a user selects advice from a professional makeup expert, the device notifies the server of the selection. The server sends the user's image data to the expert, who then creates customized advice based on the image. The data is then sent to the device via the server. The input is the user's selection information and image data, and the output is the expert's advice.

[1676] (Application example 2)

[1677] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1678] Conventional makeup suggestion systems suggest optimal makeup methods based on the user's facial features and environmental data. However, these systems do not take into account the user's current emotional state, making it difficult to provide makeup suggestions that match the user's feelings and mood. Furthermore, because the system does not suggest makeup methods that correspond to emotions, it is difficult to increase user satisfaction.

[1679] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing an image of the user's face, an analysis means for analyzing the captured image, an emotion recognition means for recognizing emotions, and a suggestion means for suggesting an optimal makeup method to the user based on the analysis results and emotion data. This makes it possible to suggest a makeup method based on the user's emotional state.

[1680] "Photography means" refers to a device or function that allows a user to take pictures or videos of themselves.

[1681] "Analysis means" refers to a device or function for analyzing facial features and environmental information based on captured image and video data.

[1682] The "suggestion means" is a device or function that calculates and suggests the most suitable makeup method for the user based on the analyzed data.

[1683] The "display means" is a device or function for displaying the proposed makeup procedure to the user in the form of video, still images, or text.

[1684] The "management means" refers to a device or function for centrally managing information about cosmetics owned by a user.

[1685] The "determination means" is a device or function that determines whether the cosmetics required for the proposed makeup procedure are in the user's possession or need to be purchased.

[1686] "Purchase means" refers to a device or function that links cosmetics to be purchased to an online purchasing site, allowing users to easily purchase them.

[1687] An "advice means" is a device or function that provides additional advice or evaluation to the user after the user has finished applying makeup.

[1688] The "selection means" is a device or function that allows the user to select and receive advice from an expert.

[1689] "Emotion recognition means" refers to a device or function for analyzing and recognizing the current emotion of the user from their facial expression.

[1690] The present invention is a system that allows users to take a picture of their own face, analyzes the image data, and provides optimal makeup methods, and also adds a function to recognize the user's emotions. This system takes into account the user's facial features, environmental information, and current emotional state to propose more detailed and personalized makeup methods.

[1691] Hardware and software used

[1692] This system uses the following hardware and software:

[1693] Photography method: Use the built-in camera on a smartphone or tablet device or a webcam connected to a computer.

[1694] Analysis method: DeepFace library for analyzing facial features and environmental information.

[1695] Emotion recognizer: An emotion recognition algorithm to recognize user emotions (e.g., a Python emotion recognition library).

[1696] Suggestion method: A makeup suggestion algorithm that runs on the server side. Data is exchanged via REST API.

[1697] Display means: Smartphone or tablet display, and dedicated mobile application.

[1698] Management means: A database (e.g., SQLite) for managing the cosmetic data owned by the user.

[1699] Determination method: An algorithm to determine whether the required cosmetics are in possession.

[1700] Purchase method: A function to link the cosmetics to be purchased to an online shop (e.g., integration with a shopping API).

[1701] Advice tool: An algorithm that analyzes re-photograph data after makeup application and provides additional advice.

[1702] Selection tool: A user interface that allows the selection of expert advice.

[1703] System operation explanation

[1704] Filming method

[1705] Users take a photo of their face using the camera on their smartphone or tablet device by launching the camera app and pressing the "take photo" button.

[1706] Analysis means

[1707] The device sends the captured image data to a server and analyzes it using the DeepFace library to extract facial features (face shape, skin tone, eye position, etc.) and environmental data (season, weather, temperature, etc.).

[1708] emotion recognition means

[1709] The image analysis means acquires facial expression data of the user, and the emotion recognition means analyzes this to identify the user's current emotion (for example, joy, sadness, anger, etc.).

[1710] Proposal means

[1711] The server then uses the analysis data to suggest makeup applications. These suggestions take into account not only the user's facial features and environmental data, but also their emotional data. For example, if the server detects that the user looks tired, it will suggest a refreshing makeup look using lighter colors.

[1712] Display means

[1713] The server sends the generated makeup instructions in the form of video, still images, and text to the device, which then displays them to the user. The display includes detailed instructions in video, still images, and text.

[1714] Examples of concrete examples and prompts

[1715] Below are some example prompts to be input to the generative AI model:

[1716] Develop a system that takes a picture of a user's face, analyzes the image to recognize emotions, and suggests the best makeup application based on the user's emotions. Follow these steps:

[1717] 1. Activate the camera and take a picture of the user's face.

[1718] 2. Analyze the captured image to identify facial features and emotions (happiness, sadness, anger, etc.).

[1719] 3. Send facial features and emotion data to an external API and receive the optimal makeup application.

[1720] 4. The received makeup method is displayed to the user.

[1721] As described above, the system for implementing this invention is a multifunctional system that proposes the optimal makeup method taking into consideration the user's facial features and emotions. This system allows the user to apply makeup that suits their emotional state, thereby increasing satisfaction.

[1722] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1723] Step 1:

[1724] The user activates the camera on their smartphone or tablet and takes a picture of their own face. At this time, the camera is activated by pressing the "shoot" button in the application, and the user takes a picture in an appropriate position. The input is the user's facial image, which is output as image data.

[1725] Step 2:

[1726] The device receives the captured image data and sends it to the server. The input is the image data, which is sent to the server via the network. The output is the image data received by the server.

[1727] Step 3:

[1728] The server analyzes the received image data. For this analysis, it uses the DeepFace library to analyze the user's facial features (face shape, skin tone, eye position, etc.). The input is the received image data, and the output is facial feature data. Specifically, the DeepFace library maps each point on the face and extracts feature data.

[1729] Step 4:

[1730] The server then uses emotion recognition means to recognize the user's emotions from the analyzed feature data. This process uses an emotion recognition algorithm (for example, a Python emotion recognition library). The input is facial feature data, and the output is emotion data. Specifically, it analyzes changes in facial expression (for example, eyebrow movement, mouth corner position) to identify emotions such as joy, sadness, and anger.

[1731] Step 5:

[1732] The server then uses the analyzed data (facial features and emotion data) to suggest the most suitable makeup application method for the user. This proposal is made using a makeup application suggestion algorithm on the server. The input is facial feature data and emotion data, and the output is suggested data for makeup procedures. For example, if the emotion is recognized as "fatigue," the server will suggest a makeup method to make the user's face look brighter.

[1733] Step 6:

[1734] The server prepares the generated makeup steps in video, still image, and text formats and sends them to the terminal. The input is the proposed makeup steps data, and the output is the terminal that receives it. Specifically, the server converts the makeup steps into multiple formats (video file, still image file, text file) and sends them all at once to the terminal.

[1735] Step 7:

[1736] The device then displays the received makeup instructions to the user. The input is the received makeup instructions data, and the output is a visual representation of the makeup instructions that the user can confirm. Specifically, the device application plays the video and highlights important parts with still images and text.

[1737] Step 8:

[1738] The user applies makeup according to the suggested makeup steps, then takes a photo of their face again. The input is the face image after makeup, and the output is the image data. Specifically, the user starts the camera again and presses the "take another photo" button to take a photo of their face.

[1739] Step 9:

[1740] The device sends the face image data after makeup application to the server, which then analyzes it again. The input is the re-photographed image data, and the output is the analysis results. Specifically, the server executes the process described above again to evaluate the face after makeup application.

[1741] Step 10:

[1742] The server generates additional advice based on the reanalysis results and sends it to the terminal. The input is the reanalysis results, and the output is additional advice data. Specific actions include providing specific feedback such as "Draw your eyeliner a little thinner" if the eyeliner is too thick.

[1743] Step 11:

[1744] The terminal displays the received additional advice to the user. The input is the received additional advice data, and the output is an advice display that the user can visually confirm. Specifically, the advice text or a still image is displayed on the terminal screen.

[1745] The above are the specific processing steps for carrying out the present invention. The user can receive personalized makeup suggestions that take into consideration their emotions, which can improve satisfaction.

[1746] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1747] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1748] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1749] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1750] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1751] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1752] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1753] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1754] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1755] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1756] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1757] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1758] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1759] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1760] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1761] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1762] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1763] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1764] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1765] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1766] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1767] The following is further disclosed regarding the above embodiment.

[1768] (Claim 1)

[1769] camera means for allowing a user to photograph themselves;

[1770] image analysis means for analyzing the captured image;

[1771] A proposal method that suggests the best makeup method for the user based on the analysis results;

[1772] a display means for displaying the proposed makeup procedure in the form of a video, a still image, or a text;

[1773] A management means for managing cosmetic information possessed by the user;

[1774] A determination means for determining whether the cosmetics required for the proposed makeup procedure are already in possession or need to be purchased;

[1775] A purchasing method that links to a shopping site for cosmetics that need to be purchased,

[1776] an advice means for providing additional advice to the user after the image is captured;

[1777] A selection of options that allow you to choose advice from professional makeup experts,

[1778] A system including:

[1779] (Claim 2)

[1780] The system of claim 1, wherein the image analysis means analyzes the user's facial features and environmental data.

[1781] (Claim 3)

[1782] 2. The system according to claim 1, wherein the display means provides the suggested makeup procedure in the form of a video playback, a still image display, or a text display.

[1783] "Example 1"

[1784] (Claim 1)

[1785] a photographing device means for allowing a user to photograph himself / herself;

[1786] an image analysis device means for analyzing the captured image;

[1787] a suggestion device means for suggesting an optimal makeup method to the user based on the analysis results;

[1788] a display device for displaying the proposed makeup procedure in the form of a moving image, a still image, or a text;

[1789] A management function means for managing cosmetic information owned by the user;

[1790] A determination device means for determining whether the cosmetics required for the proposed makeup procedure are already owned or need to be purchased;

[1791] a purchasing device means for linking the cosmetics to be purchased to an electronic commerce site;

[1792] an advice device means for providing additional advice to the user after the image is captured;

[1793] a selection device means for selecting advice from experts;

[1794] A system including:

[1795] (Claim 2)

[1796] 10. The system of claim 1, wherein said image analyzer means analyzes the user's facial features and environmental data.

[1797] (Claim 3)

[1798] 2. The system according to claim 1, wherein the display device provides the suggested makeup procedure in the form of a video playback, a still image display, or a text display.

[1799] "Application Example 1"

[1800] (Claim 1)

[1801] a photographing means for the user to photograph himself / herself;

[1802] analysis means for analyzing the captured image;

[1803] A proposal method that suggests the best makeup method for the user based on the analysis results;

[1804] a display means for displaying the proposed makeup procedure in the form of a video, a still image, or a text;

[1805] A management means for managing information on items possessed by a user;

[1806] A determination means for determining whether the items required for the proposed cosmetic procedure are already in possession or need to be purchased;

[1807] a purchasing means for linking the item to a shopping site;

[1808] an advice means for providing additional advice to the user after the image is captured;

[1809] an inventory confirmation and guidance means for checking inventory in the store and providing information on the location of the item;

[1810] A feedback means for providing interactive feedback linked to the real world;

[1811] A system including:

[1812] (Claim 2)

[1813] 10. The system of claim 1, wherein the analyzing means analyzes the user's facial features and environmental data.

[1814] (Claim 3)

[1815] 2. The system according to claim 1, wherein the display means provides the suggested makeup procedure in the form of a video playback, a still image display, or a text display.

[1816] "Example 2: Combining Emotion Engines"

[1817] (Claim 1)

[1818] an imaging means for allowing a user to photograph himself / herself;

[1819] image analysis means for analyzing the captured image;

[1820] emotion recognition means for acquiring and analyzing facial expression data of a user by an image analysis means;

[1821] A suggestion means for suggesting the most suitable makeup method to the user based on the analysis results;

[1822] a display means for displaying the proposed makeup procedure in the form of a moving image, a still image, or a text;

[1823] an information management means for managing cosmetic information possessed by a user;

[1824] a determining means for determining whether the cosmetics required for the proposed makeup procedure are already in possession;

[1825] a purchasing method that links to an online retail site for the cosmetic product you need to purchase;

[1826] an advice means for providing additional advice to the user after the image has been captured;

[1827] A choice of expert advice options and

[1828] A system including: 【...

Claims

1. camera means for allowing a user to photograph themselves; image analysis means for analyzing the captured image; A proposal method that suggests the best makeup method for the user based on the analysis results; a display means for displaying the proposed makeup procedure in the form of a video, a still image, or a text; A management means for managing cosmetic information possessed by the user; A determination means for determining whether the cosmetics required for the proposed makeup procedure are already in possession or need to be purchased; A purchasing method that links to a shopping site for cosmetics that need to be purchased, an advice means for providing additional advice to the user after the image is captured; A selection of options that allow you to choose advice from professional makeup experts, A system including:

2. The system of claim 1, wherein said image analysis means analyzes the user's facial features and environmental data.

3. 2. The system according to claim 1, wherein said display means provides the proposed makeup procedure in the form of a video playback, a still image display, or a text display.

Images