System

Abstract

A system is provided.SOLUTION: A system comprising: means for registering information of a user and storing the user information in a database; means for capturing an image of a facial surface of the user and analyzing the image to generate facial condition data; means for proposing an optimal cosmetic product and a makeup method based on the user information and the facial condition data; and means for giving a step-by-step instruction for makeup based on the proposed cosmetic product and makeup method.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by this invention is to provide a system that suggests cosmetics that are best suited to a user's preferences and skin condition, and helps them recreate professional makeup looks, thereby reducing the waste of unnecessary cosmetics and improving user satisfaction. Specifically, the goal is to suggest an appropriate makeup look based on the user's facial condition each morning, enabling anyone to easily achieve professional makeup looks. [Means for solving the problem]

[0005] The system of the present invention includes the following means.

[0006] means for registering user information and storing the user information in a database;

[0007] means for capturing an image of a user's face and analyzing the image to generate facial condition data;

[0008] means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data;

[0009] The system includes a means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method.

[0010] The system may further include means for displaying makeup instruction information generated based on the proposed cosmetics and makeup methods and prompting the user to apply makeup in accordance with the instructions. It may also include means for collecting user feedback on the proposed cosmetics and makeup methods and recording the feedback in the database. This allows for makeup suggestions and instructions tailored to individual users, reduces cosmetic waste, and increases user satisfaction.

[0011] "User information" is personal data including the user's allergy information, usage history, preferred colors and ingredients, and the like.

[0012] A "database" is a system that centrally stores data such as user information and analysis results, and allows for efficient access and management as needed.

[0013] "Facial condition" is information indicating the state of the user's face on that day, such as dark circles under the eyes, skin tone, degree of tanning, and blemishes.

[0014] "Image analysis" is the process of analyzing image data, such as facial photographs, to identify specific features or conditions.

[0015] "Cosmetics suggestion" refers to the act of selecting and presenting the most suitable type and brand of cosmetics based on the user's information and facial condition.

[0016] "Makeup methods" refers to information that provides guidance on specific steps and techniques for applying makeup using specific cosmetics.

[0017] "Step-by-step instruction" is assistance that provides detailed step-by-step instructions for applying makeup, allowing the user to apply the makeup by following those instructions.

[0018] "Suggested cosmetics and makeup methods" refer to cosmetics and their usage methods selected by the analysis and proposal system based on each individual user's information and facial condition.

[0019] "Feedback" refers to information about the user's evaluation, impressions, and usage of the system and the proposed cosmetics and makeup methods. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] An embodiment for implementing the system of the present invention will be described.

[0042] The system has an interface that allows users to register and manage their information, analyze their facial condition, suggest optimal makeup and cosmetics, and provide instructions on how to apply makeup.

[0043] First, the user installs the smartphone app and enters information about allergies and cosmetics they have used in the past (favorite cosmetics and cosmetics that did not suit them). The smartphone app formats this information and sends it to the server to be stored in a database.

[0044] The server stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0045] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is then sent to the AI ​​Mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​Mirror, which analyzes information such as dark circles, skin tone, sun exposure, and blemishes.

[0046] The server receives the analysis results from the AI ​​mirror, combines them with information stored in the user's database, and runs an algorithm to generate the optimal cosmetics and makeup routine. Using the AI ​​model, it suggests a set of cosmetics and makeup routines optimized for the user's facial condition.

[0047] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0048] Users can also provide feedback on the suggested cosmetics and makeup techniques through the smartphone app, which the server then stores in a database to improve the accuracy of future suggestions.

[0049] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest the optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition.

[0050] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0051] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0055] Step 2:

[0056] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0057] Step 3:

[0058] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0059] Step 4:

[0060] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0061] Step 5:

[0062] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0063] Step 6:

[0064] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0065] Step 7:

[0066] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0067] Step 8:

[0068] The user begins applying makeup based on the suggestions. The device (AI mirror) guides the user by displaying specific steps for applying makeup using the suggested cosmetics in real time, step by step.

[0069] Step 9:

[0070] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0071] Step 10:

[0072] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions.

[0073] In this way, the system suggests the optimal cosmetics and makeup methods based on the user's individual preferences and facial condition, helping the user easily recreate professional makeup looks.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] In today's society, recreating professional makeup looks at any time in a busy daily life is a difficult challenge for many users. Choosing the optimal cosmetics based on individual allergies and skin conditions can be complicated, and incorrect choices can cause skin problems. Furthermore, inappropriate makeup instruction can make it difficult to achieve the desired results. To solve these problems, a system is needed that is easy for users to use and provides optimal cosmetics and makeup methods tailored to individual needs.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques using a generative AI model based on the user information and facial condition data, means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques, and means for collecting user feedback on the suggested cosmetics and makeup techniques and recording the feedback in the database. This allows users to easily recreate professional makeup looks even in their busy daily lives and select cosmetics optimized for their individual needs. Furthermore, the system can be continuously improved based on user feedback, resulting in more accurate suggestions.

[0079] "User information" refers to individual profile data that a user registers in the system, including allergy information and information about cosmetics used in the past.

[0080] A "database" is a system installed inside a server for systematically storing and managing user information and feedback information.

[0081] "Facial condition data" refers to information such as dark circles, skin tone, tan, and blemishes analyzed by the AI ​​mirror.

[0082] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal cosmetics and makeup methods based on user information and facial condition data.

[0083] A "makeup method" refers to a set of instructions including steps for using and applying cosmetic products that are optimized for the user's facial condition and individual needs.

[0084] "Step-by-step instruction" refers to providing users with sequential instructions for each step of makeup application through the AI ​​mirror and smartphone app.

[0085] "Feedback" refers to comments and evaluation information provided by users regarding the proposed cosmetics and makeup methods.

[0086] An "AI-equipped display device" refers to a display device that has built-in artificial intelligence to display makeup instruction information to users in real time.

[0087] The present invention relates to a system that registers and manages user information, analyzes facial condition, suggests optimal cosmetics and makeup techniques, and provides step-by-step makeup instruction. The system is composed of the following elements: a user, a terminal, and a server.

[0088] First, the user installs a dedicated app on a device such as a smartphone or tablet. Using this app, the user enters information about their allergies and the cosmetics they have used in the past. Specifically, this information may include nut or shellfish allergies, or whether item X from brand A was good but item Y from brand B was not. This information is then formatted and sent to the server via a secure protocol (e.g., HTTPS).

[0089] The server stores the received user information in a database. The database is in SQL or NoSQL format and is managed as profile data for each user. This makes it possible to recommend optimal cosmetics based on the user's allergy information and past usage history.

[0090] Every morning, users take a photo of their face using a smartphone app. The device (smartphone app) then sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes to generate facial condition data. The analysis results are then sent back to the server.

[0091] The server combines the received facial condition data with user information stored in a database and uses a generative AI model (such as Azure AI, Google AI, or IBM Watson) to suggest optimal cosmetics and makeup applications, such as foundation, eyeshadow, and lipstick that are free of nuts and shellfish.

[0092] The device (AI mirror and smartphone app) provides detailed instructions based on the proposed makeup method. Specifically, the AI ​​mirror provides step-by-step makeup instructions to the user in real time, such as "Next, apply eyeshadow" or "Apply foundation evenly." The user can then follow these instructions to apply makeup.

[0093] Users can also provide feedback on the proposed cosmetics and makeup techniques through a smartphone app. This feedback is sent to the server and stored in a database. This allows the server to continuously incorporate user feedback and improve the accuracy of future suggestions.

[0094] As a specific example, a user may register allergies such as "nut allergy" and "shellfish allergy" and enter information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will use this information to suggest the optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients. Furthermore, the AI ​​mirror will provide real-time instructions on how to use the suggested cosmetics, and the user will follow those instructions to apply their makeup.

[0095] An example prompt is:

[0096] "To generate makeup suggestions for the AI ​​Specialist, please use the following information to suggest the most suitable cosmetics for the user from a database. User information: Nut allergy, shellfish allergy. Past cosmetics usage history: Brand A's item X worked well, but Brand B's item Y did not. Face photo analysis results: Dark circles under the eyes, normal skin tone, slight blemishes."

[0097] The above is a specific embodiment for carrying out the present invention. This system allows users to easily recreate professional makeup looks every day and reduces the waste of unnecessary cosmetics.

[0098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0099] Step 1:

[0100] Users install the smartphone app and launch it. They then enter information about allergies and cosmetics they have used in the past. For example, they can enter information such as "nut allergy," "shellfish allergy," "item X from brand A was good," or "item Y from brand B didn't suit me."

[0101] Input: User's allergy information and cosmetic use history

[0102] Output: Formatted user information

[0103] Step 2:

[0104] The device (smartphone app) formats the information entered by the user. In this step, the data format is organized and converted into a format that is easy for the server to receive and analyze. The formatted information is then sent to the server using a secure protocol (e.g., HTTPS).

[0105] Input: Raw information entered by the user

[0106] Output: User information formatted for easy reception by the server

[0107] Step 3:

[0108] The server stores the received user information in a database. The database is managed in SQL or NoSQL format and is saved as profile data for each user. In this process, the user information is securely registered in the database.

[0109] Input: formatted user information

[0110] Output: User profile data in the database

[0111] Step 4:

[0112] Every morning, users take a photo of their face using a smartphone app and their smartphone's camera.

[0113] Input: User's face photo

[0114] Output: Photographed face photo data

[0115] Step 5:

[0116] The device (smartphone app) takes a photo of your face and sends it to the AI ​​mirror. When sending, the photo must be of the correct resolution and format.

[0117] Input: Photographed face photo data

[0118] Output: Facial photo data sent to AI Mirror

[0119] Step 6:

[0120] The device (AI Mirror) analyzes the facial condition based on the received facial photo, including dark circles, skin tone, tan, and blemishes, and generates an analysis result.

[0121] Input: Facial photo data sent to AI Mirror

[0122] Output: Facial condition data

[0123] Step 7:

[0124] The server receives the analysis results sent by the AI ​​mirror and combines them with the user's information in the database. Using a generative AI model, it generates the optimal cosmetics and makeup routine based on the user's facial condition. For example, it suggests foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0125] Input: Facial condition data and user information from the database

[0126] Output: Recommendations for optimal cosmetics and makeup application

[0127] Step 8:

[0128] The devices (AI mirror and smartphone app) provide detailed instructions based on the generated makeup method. The AI ​​mirror provides step-by-step makeup instructions to the user in real time, providing guidance such as "Next, apply eyeshadow" and "Apply foundation evenly."

[0129] Input: Recommendations for optimal cosmetics and makeup techniques

[0130] Output: Step-by-step makeup instruction information

[0131] Step 9:

[0132] Users provide feedback on the proposed cosmetics and makeup methods through a smartphone app, which is then sent to a server and stored in a database.

[0133] Input: User feedback

[0134] Output: Feedback information stored in a database

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] While conventional systems can suggest the best cosmetics and makeup methods for a user's specific facial condition, they face the challenge of making it difficult to check and try on the suggestions in real time. Another issue is that the methods for providing detailed instructions on the steps of the suggested makeup methods are limited, making it difficult for users to accurately replicate them. Furthermore, there are also issues with systems that are inadequate in collecting user feedback and incorporating that information into future suggestions.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques based on the user information and facial condition data, means for allowing the user to virtually try on the suggested cosmetics in real time, and means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques. This allows the user to check and try on the optimal cosmetics suggestions in real time, and accurately understand and execute the makeup steps. Feedback can also be collected to improve the accuracy of the suggestions.

[0140] "User information" refers to information including allergy information entered by the user, information about cosmetics used in the past, individual preferences, skin characteristics, and the like.

[0141] "Database" refers to an information system for storing and managing user information and facial condition data.

[0142] "Facial condition data" is data generated by analyzing an image of a face, and includes information such as dark circles, skin tone, blemishes, and degree of tanning.

[0143] "Cosmetics" are products such as foundation, eye shadow, lipstick, etc. that are used to enhance the aesthetic appearance of a user's face.

[0144] "Makeup method" refers to the steps and techniques for applying makeup to the face using cosmetics.

[0145] "Virtual try-on" is a technology that simulates applying suggested cosmetics to the user's face in real time, allowing them to see how they will look without actually applying them.

[0146] "Feedback" is information in which a user transmits opinions and evaluations about cosmetics and makeup methods used.

[0147] "Step-by-step instruction" is a process that provides step-by-step instructions on how to use cosmetics and how to apply makeup, helping users to apply makeup correctly.

[0148] The following describes an embodiment of the present invention. The system of the present invention uses a smartphone app, a server, and AI analysis technology to propose optimal cosmetics and makeup methods based on the user's facial condition, and is equipped with virtual try-on and step-by-step instruction functions.

[0149] First, the user installs the smartphone app and enters information about their allergies and the cosmetics they have used in the past. This information is formatted and stored in a cloud database (e.g., Firebase). The server then manages the received user information individually. Based on the information stored in this database, the app can suggest the most suitable cosmetics based on the user's allergies and past usage history.

[0150] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is sent to an AI analysis engine (e.g., TensorFlow Lite, OpenCV) and analyzed for information such as dark circles, skin tone, sun exposure, and blemishes. The smartphone app then sends the photo to the AI ​​analysis engine and receives the results.

[0151] The server receives the analysis results from the AI ​​analysis engine, combines them with information stored in the user's database, and runs an algorithm to generate optimal cosmetics and makeup techniques, which are then presented to the user in real time.

[0152] Furthermore, the proposed cosmetics are applied to the user's face in real time using virtual try-on features (e.g., OpenCV, camera API), allowing users to see how the products will look without actually applying them.

[0153] Based on the proposed makeup application, step-by-step instructions are provided. The smartphone app uses real-time video and audio to provide detailed instructions that users can follow. Based on the suggestions, users can also provide feedback, which is recorded in a database to improve the accuracy of future suggestions.

[0154] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​analysis engine analyzes the results as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition. The system also allows users to virtually try on the suggested cosmetics, and if they are satisfied, they can purchase them immediately.

[0155] Prompt Sentence Examples

[0156] markdown

[0157] Prompt statement

[0158] User information: Nut allergy, cosmetics used in the past: Brand A, item X (good), Brand B, item Y (unsuitable)

[0159] Take a photo of your face and output the analysis results: dark circles, normal skin tone, slight blemishes

[0160] Based on the analysis results generated, the app will suggest the best cosmetics and makeup techniques.

[0161] Simply apply the suggested items to your face virtually and provide a picture.

[0162] Provide step-by-step instructions for users to proceed with their makeup.

[0163] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0164] Program processing steps

[0165] Step 1:

[0166] Users enter information about their allergies and cosmetics they have used in the past into a smartphone app.

[0167] Input: Allergy information (e.g., nut allergy, shellfish allergy), past cosmetic use information (e.g., item X from brand A is fine, but item Y from brand B is incompatible)

[0168] Data processing: The input information is formatted into JSON format and sent to the server.

[0169] Output: Formatted user information (JSON)

[0170] Step 2:

[0171] The server stores the received user information in a cloud database (e.g., Firebase).

[0172] Input: User information (JSON)

[0173] Data processing: Save as an entry in a database and associate with an individual user ID.

[0174] Output: Database entries

[0175] Step 3:

[0176] A user takes a photo of their face every morning using a smartphone app.

[0177] Input: User's face photo

[0178] Data processing: Save the image in the correct format and send it to the AI ​​analysis engine.

[0179] Output: Sent face photo (image)

[0180] Step 4:

[0181] An AI analysis engine (e.g., TensorFlow Lite, OpenCV) analyzes the facial photo and generates facial condition data.

[0182] Input: Face photo (image)

[0183] Data processing: Analyzes facial dark circles, skin tone, blemishes, etc. and converts the results into JSON format.

[0184] Output: Facial condition data (JSON)

[0185] Step 5:

[0186] The server runs an algorithm that generates optimal cosmetics and makeup methods based on facial condition data and user information.

[0187] Input: Facial condition data (JSON), user information

[0188] Data processing: Analyze data using an AI model and suggest the most suitable cosmetics and makeup techniques.

[0189] Output: Suggested cosmetics and makeup methods (list and step-by-step instructions)

[0190] Step 6:

[0191] The smartphone app applies the suggested cosmetics to the user's face in real time using a virtual try-on function (e.g., OpenCV, camera API).

[0192] Input: Suggested cosmetics (list), user's face photo (image)

[0193] Data processing: Using a virtual try-on algorithm, images of cosmetics applied to the user's face are generated in real time.

[0194] Output: Virtual try-on image

[0195] Step 7:

[0196] The smartphone app provides real-time step-by-step instructions based on the proposed makeup application method, providing detailed instructions using video and audio.

[0197] Input: Suggested makeup method (step-by-step instructions)

[0198] Data processing: Generate step-by-step video and audio guides and display and play them to the user in real time.

[0199] Output: Video and audio guide

[0200] Step 8:

[0201] Users provide feedback on the results of their makeup through a smartphone app, and the server records it in a database.

[0202] Input: User feedback (text or rating)

[0203] Data processing: Feedback information is stored in a database so that it can be reflected in future proposals.

[0204] Output: Feedback stored in a database

[0205] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0206] An embodiment for implementing the system of the present invention will be described.

[0207] The system is equipped with an interface that registers and manages user information, analyzes facial condition, suggests optimal makeup and cosmetics, provides instructions on how to apply makeup, and an emotion engine that recognizes the user's emotions.

[0208] First, the user installs the smartphone app and, when they launch it for the first time, enters information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like). The smartphone app formats this information and sends it to the server. The server then stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0209] Next, each morning, the user takes a photo of their face using a smartphone app. The photo is sent to the AI ​​mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes. The server receives the analysis results from the AI ​​mirror and combines them with information stored in the user's database to generate recommendations for the optimal cosmetic set and makeup method for the user.

[0210] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0211] This system also incorporates an emotion engine that recognizes the user's emotions. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if the user is feeling down, it will suggest refreshing makeup that will cheer them up. In this way, the system dynamically adjusts the suggestions according to the user's emotional state, thereby improving user satisfaction.

[0212] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," and the emotion engine further recognizes that the user is "feeling stressed," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients, based on the user's data, facial condition, and emotion data, as well as relaxing makeup to relieve stress.

[0213] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0214] The user's emotions recognized by the emotion engine are also recorded in a database and used to suggest makeup looks for the next time onward, enabling optimal suggestions to be made in response to fluctuations in the user's emotional state.

[0215] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics. Furthermore, by making suggestions based on the user's emotions, further improvements in satisfaction can be expected.

[0216] The processing flow will be explained below.

[0217] Step 1:

[0218] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0219] Step 2:

[0220] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0221] Step 3:

[0222] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0223] Step 4:

[0224] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0225] Step 5:

[0226] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0227] Step 6:

[0228] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0229] Step 7:

[0230] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0231] Step 8:

[0232] The user begins applying makeup based on the suggested cosmetics and makeup methods. The device (AI mirror) guides the user by displaying specific makeup steps using the suggested cosmetics in real time, step by step.

[0233] Step 9:

[0234] While the user is applying makeup, the emotion engine analyzes the user's facial expressions and voice to generate emotion data, for example, determining whether the user is smiling or feeling stressed through a camera or microphone.

[0235] Step 10:

[0236] The server receives the emotion data generated by the emotion engine. Based on this, it adjusts the initial cosmetics and makeup suggestions and re-suggests the most optimal makeup method. For example, if the user is tired, it suggests makeup that has a refreshing effect.

[0237] Step 11:

[0238] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0239] Step 12:

[0240] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions. It also records the user's emotions recognized by the emotion engine in a database and uses this information to improve future makeup suggestions.

[0241] In this way, the system suggests the optimal cosmetics and makeup techniques based on the user's individual preferences, facial condition, and even emotions, helping them easily recreate professional makeup looks. Furthermore, the suggestions are constantly adjusted based on the user's latest condition and emotions, providing a high level of satisfaction.

[0242] Example 2

[0243] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] Conventional makeup systems rely on the user's qualitative senses and feedback, making it difficult to provide optimal recommendations based on scientific evidence or an individual's emotional state. Furthermore, because they do not take into account the user's past usage history or emotions, they are unable to provide optimal makeup recommendations for individual needs.

[0245] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial state data, means for suggesting optimal cosmetics and makeup techniques based on the user information and the facial state data, means for providing makeup instructions based on the suggested cosmetics and makeup techniques, and means for analyzing emotions from the user's facial expressions and voice, saving the emotion data in a database, and dynamically adjusting the makeup suggestions. This makes it possible to suggest optimal makeup based on the user's past usage history and emotional state, and to provide a satisfying makeup experience that meets the needs of each individual user.

[0246] "User information" refers to personal allergy information entered by the user, history information on cosmetics used in the past, and the like.

[0247] "Database" refers to a storage device for storing and managing user information, facial state data, emotion data, and other related information.

[0248] "Facial condition data" refers to information about facial condition such as dark circles under the eyes, skin tone, degree of tanning, and blemishes, which is analyzed from a facial photograph.

[0249] "Cosmetics" refers to items used in makeup, such as foundation, eye shadow, and lipstick.

[0250] "Suggestion" refers to the process of presenting the user with the most suitable cosmetics and makeup methods based on the user information and facial condition data.

[0251] "Emotional data" refers to information about the emotional state of a user analyzed from facial expressions and voice.

[0252] "Makeup procedure instruction" refers to the process of guiding a user through specific makeup steps based on the proposed cosmetics and makeup method.

[0253] "User feedback" refers to information such as impressions and improvements provided by users after completing their makeup.

[0254] MODE FOR CARRYING OUT THE INVENTION

[0255] The present invention provides a makeup suggestion system that registers and manages user information, analyzes facial conditions, suggests optimal cosmetics and makeup methods, provides makeup instruction, and includes an emotion engine that recognizes the user's emotions. Hereinafter, an embodiment of the present invention will be described.

[0256] User information registration and management

[0257] First, the user installs the smartphone app and, when launching it for the first time, enters allergy information and information about cosmetics used in the past. At this time, the user enters allergy information such as "nut allergy" or "shellfish allergy" as well as a history of cosmetics use, such as "Brand A's item X was good" or "Brand B's item Y did not suit me." The smartphone app formats this information and sends it to the server.

[0258] The server stores the received user information in a database and manages it individually. Based on this stored information, it can respond to the user's allergies and suggest cosmetics based on their past usage history.

[0259] Analysis of facial condition

[0260] Next, every morning, users take a photo of their face using a smartphone app, which then sends the photo to the AI ​​Mirror for analysis. The AI ​​Mirror then analyzes the photo to determine whether they have dark circles, skin tone, sun exposure, or blemishes.

[0261] The device (smartphone app) sends the captured facial photo to the AI ​​Mirror, which then sends the analysis results to the server. The AI ​​Mirror's data includes specific information such as "there are dark circles under the eyes," "the skin tone is normal," and "there are some blemishes."

[0262] Proposal of optimal cosmetics and makeup methods

[0263] The server compares the analysis results received from the AI ​​mirror with the user's database information to generate the optimal cosmetic set and makeup method. This includes foundation, eye shadow, lipstick, etc. that do not contain nuts or shellfish ingredients. It also takes into account emotional data analyzed by the emotion engine and suggests makeup methods that correspond to the user's emotions. For example, if the user is "feeling stressed," it will suggest relaxing makeup.

[0264] Makeup instruction

[0265] Next, the device (AI mirror and smartphone app) will provide step-by-step instructions to the user based on the generated makeup method, displaying detailed instructions and guiding the user, such as, "First, apply this concealer under your eyes to hide dark circles."

[0266] The user follows the instructions to apply the makeup. After completing the makeup, the user provides feedback on the completed makeup, which is then stored in a database and reflected in future suggestions.

[0267] Optimization by Emotion Engine

[0268] The system also incorporates an emotion engine. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions it recommends based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if they are feeling down, it will suggest refreshing makeup that will cheer them up. In this way, dynamically adjusting the suggestions according to the user's emotional state increases user satisfaction.

[0269] Examples and prompts

[0270] As a concrete example, we will explain the process when a user uses the system for the first time. The user installs the smartphone app and enters their "nut allergy" and "shellfish allergy." They also enter their past cosmetic use history, such as "Item X from brand A was good, but item Y from brand B didn't suit me."

[0271] The next morning, the user takes a photo of their face and sends it to the AI ​​mirror. The AI ​​mirror analyzes the photo and determines whether they have dark circles under their eyes, normal skin tone, or slight blemishes, and sends the results to the server. If the emotion engine determines that the user is feeling stressed, the server will suggest relaxing makeup, including foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0272] An example prompt is, "I have nut and shellfish allergies. In the past, item X from brand A worked well for me, but item Y from brand B did not. I would like you to analyze a photo of my face this morning and suggest the best cosmetics and makeup techniques. I am currently feeling stressed."

[0273] The above is an embodiment of the present invention. By using this system, users can easily recreate professional makeup looks every day, providing a highly satisfying makeup experience that meets the needs of each individual user.

[0274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0275] Program processing flow

[0276] Step 1: Register user information

[0277] 1. Input: The user installs the smartphone app and enters information about allergies and cosmetics used in the past.

[0278] Examples: "Nut allergy", "Shellfish allergy", cosmetics used in the past: "Item X from brand A was good", "Item Y from brand B didn't suit me".

[0279] 2. How it works: The smartphone app properly formats the entered information and sends it to the server.

[0280] 3. Output: The server stores the received user information in a database, providing the basic data for future cosmetic recommendations.

[0281] Step 2: Analyze your facial condition

[0282] 1. Input: Every morning, the user takes a photo of their face using a smartphone app.

[0283] 2. How it works: The smartphone app sends the captured facial photo to the AI ​​mirror.

[0284] 3. How it works: The AI ​​mirror analyzes the received facial photo and generates facial condition data, such as whether the person has dark circles under their eyes, whether their skin tone is normal, or whether they have some blemishes.

[0285] 4. Output: The AI ​​mirror sends the analysis results to the server and stores them in the user's database.

[0286] Step 3: Recommendations for the best cosmetics and makeup techniques

[0287] 1. Input: The server performs analysis based on facial condition data and the user's database information (allergy information and cosmetic use history).

[0288] 2. Operation: The server combines facial condition and user information to generate optimal cosmetics and makeup methods, such as recommending foundation, eye shadow, and lipstick that are free of nuts and shellfish ingredients.

[0289] 3. How it works: The emotion engine analyzes emotional data from the user's facial expressions and voice and adjusts the suggestions.

[0290] 4. Output: The server sends the optimal cosmetic set and makeup method for the user to the device (smartphone app and AI mirror).

[0291] Step 4: Makeup Instructions

[0292] 1. Input: Proposal content sent from the server (cosmetics set and makeup method).

[0293] 2. Operation: The device (smartphone app and AI mirror) displays the suggested cosmetics and makeup methods to the user, and provides detailed step-by-step instructions for use. For example, "First, apply this concealer under your eyes to hide dark circles."

[0294] 3. Output: The user follows the instructions to apply makeup.

[0295] Step 5: Collect user feedback and sentiment data

[0296] 1. Input: User feedback after makeup is completed.

[0297] 2. Action: The user enters feedback about the makeup through the smartphone app. For example, "This makeup was very good" or "This eyeshadow didn't suit me."

[0298] 3. Operation: The emotion engine analyzes emotion data from the user's facial expressions and voice.

[0299] 4. Output: The server stores the collected feedback and sentiment data in a database, which will be used to improve future suggestions.

[0300] Specific steps in the overall flow

[0301] These processing steps allow users to easily receive recommendations for appropriate cosmetics and makeup techniques each day, and follow the instructions to apply their makeup. The system is continuously optimized, as user feedback and emotional data are reflected in the next recommendations.

[0302] (Application example 2)

[0303] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] In today's world, it is important to suggest optimal cosmetics and makeup techniques based on each individual user's skin condition and emotional state. However, no previous technology existed that could perform real-time facial image analysis and emotional analysis of the user and then suggest and instruct specific makeup techniques based on the results. Furthermore, there was insufficient support for applying the suggested makeup techniques in physical stores, resulting in a lack of assistance for users in selecting the most suitable products. Furthermore, there was also a lack of a mechanism for accumulating user feedback in a database and reflecting it in future suggestions.

[0305] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0306] In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial condition data, means for analyzing the user's emotions and further optimizing cosmetics and makeup methods based on the emotion data, and means for displaying the suggested cosmetics and makeup methods on a smart mirror and smart glasses in a physical store to guide the user. This makes it possible to suggest optimal cosmetics and makeup methods based on the user's skin condition and emotional state, and to provide specific makeup instructions in the physical store in real time.

[0307] "User information" refers to data including personal information about the user, allergy information, and a history of past cosmetic use.

[0308] "Database" refers to a collection of information for storing and managing user information, feedback information, etc.

[0309] "Facial condition data" refers to data regarding the condition of the user's face, such as skin tone, dark circles, and blemishes, analyzed based on a photograph of the user's face.

[0310] "Emotion data" is data relating to the emotional state of the user acquired by analyzing the user's facial expressions and voice.

[0311] "Cosmetics" refers to products used for makeup and skin care, including foundation, eye shadow, lipstick, etc.

[0312] "Makeup method" refers to specific makeup procedures and techniques suggested to the user.

[0313] "Suggestion" refers to a recommendation of optimal cosmetics and makeup techniques generated based on user information, facial condition data, and emotion data.

[0314] "Step-by-step instruction" refers to the process of teaching a user how to apply makeup step by step.

[0315] "Brick and mortar store" refers to a retail store that has a physical presence and where consumers can visit and purchase products.

[0316] A "smart mirror" is a highly functional mirror equipped with a camera and display that analyzes facial condition and provides makeup guidance.

[0317] "Smart glasses" are glasses-type devices that are worn by a user and have the function of displaying information for makeup guidance.

[0318] "Feedback" refers to information about the evaluations and impressions provided by users regarding the proposed cosmetics and makeup methods.

[0319] "Server" refers to a centralized device that processes and stores user information, facial condition data, and emotion data over a network.

[0320] The following describes an embodiment of the present invention. This system registers individual user information, and proposes optimal cosmetics and makeup methods based on the analysis of the user's facial condition and emotions, providing makeup guidance in real time.

[0321] User information registration and management

[0322] Users enter their personal information (such as allergy information and past history of cosmetic use) into a smartphone application. The application then sends the collected information to a server to store in a database. The server then records and manages the user information in the database. This database serves as the basis for making optimal recommendations based on the user's past usage and allergy information.

[0323] Facial condition and emotion analysis

[0324] The user stands in front of a smart mirror installed in a physical store and takes a photo of their face. The smart mirror uses its built-in camera to capture an image of their face and analyzes it. An AI model running on TensorFlow and Keras is used for the analysis to obtain facial condition data such as the user's skin tone, dark circles, and the presence or absence of blemishes. At the same time, emotional data is also analyzed from the user's facial expressions and voice. An emotion recognition engine is used for this emotional analysis.

[0325] Generate optimized proposals

[0326] The server compares the user's facial condition data and emotion data with a user information database to generate the optimal cosmetics and makeup method. The server generates a list of cosmetics and makeup procedures that are optimal for the presented conditions and sends them to the smart mirror and smart glasses.

[0327] Providing real-time makeup guides

[0328] When a user wears smart glasses and applies makeup in front of a smart mirror, the glasses display suggested cosmetics and makeup steps. Audio instructions are also provided. This real-time guide is implemented using Python libraries (e.g., pyttsx3).

[0329] Collecting and incorporating user feedback

[0330] After the user has completed their makeup application, they can submit feedback about the proposed cosmetics and makeup techniques through the application. This feedback information is sent to the server and recorded in a database. This feedback is then reflected in future suggestions, enabling more accurate suggestions to be made.

[0331] Examples of concrete examples and prompts

[0332] For example, a user puts on smart glasses and stands in front of a smart mirror in a brick-and-mortar store. The smart mirror takes a photo of the user's face and analyzes their skin tone, presence of dark circles, and their emotions. Based on the results of this analysis and the user's allergy information, the system suggests the most suitable cosmetics (e.g., foundation from brand X, lipstick from brand Y). Below is an example of a prompt to be input into the generative AI model.

[0333] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[0334] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0335] Step 1:

[0336] Users enter personal information into a smartphone app, including allergy information and a history of past cosmetic use. The app formats this information and sends it to a server. The input data, which includes allergy information and past cosmetic use history, is sent to the server and stored in a database.

[0337] Step 2:

[0338] A user stands in front of a smart mirror installed in a brick-and-mortar store. The smart mirror uses a built-in camera to capture an image of the user's face. This image data is input and sent to an AI model for analysis. The output is facial condition data, including information on skin tone, dark circles, blemishes, etc.

[0339] Step 3:

[0340] The server analyzes the captured facial image using an AI model (using TensorFlow and Keras) to generate condition data. This analysis is achieved by evaluating each facial parameter through the AI ​​model. The input is the facial image, and the output is facial condition data.

[0341] Step 4:

[0342] The smart mirror also uses the user's voice and facial expressions to analyze emotional data in an emotion engine. This data is analyzed by an emotion recognition engine. The input is the user's voice and facial expressions, and the output is emotional data.

[0343] Step 5:

[0344] The server combines the user information database, facial condition data, and emotional data to suggest optimal cosmetics and makeup methods based on a generative AI model. During this process, each piece of data is collated, and the generative AI model calculates the optimal combination. The input is user information, facial condition data, and emotional data, and the output is a list of suggested cosmetics and makeup methods.

[0345] Step 6:

[0346] The server sends the suggested cosmetics and makeup techniques to the smart mirror and smart glasses. The smart glasses then display step-by-step instructions on how to use the cosmetics and how to apply makeup to the user. They also provide audio guidance to support the user. The input is the suggested information, and the output is guidance information for the user.

[0347] Step 7:

[0348] After the user completes their makeup, they submit feedback through the application. This feedback is stored on the server and reflected in future suggestions. The input is the user's feedback, and the output is the improved accuracy of the suggestions.

[0349] As a concrete example, the following prompt sentences can be input to a generative AI model:

[0350] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[0351] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0352] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0353] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0354] [Second embodiment]

[0355] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0356] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0357] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0358] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0359] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0360] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0361] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0362] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0365] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0366] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0367] An embodiment for implementing the system of the present invention will be described.

[0368] The system has an interface that allows users to register and manage their information, analyze their facial condition, suggest optimal makeup and cosmetics, and provide instructions on how to apply makeup.

[0369] First, the user installs the smartphone app and enters information about allergies and cosmetics they have used in the past (favorite cosmetics and cosmetics that did not suit them). The smartphone app formats this information and sends it to the server to be stored in a database.

[0370] The server stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0371] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is then sent to the AI ​​Mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​Mirror, which analyzes information such as dark circles, skin tone, sun exposure, and blemishes.

[0372] The server receives the analysis results from the AI ​​mirror, combines them with information stored in the user's database, and runs an algorithm to generate the optimal cosmetics and makeup routine. Using the AI ​​model, it suggests a set of cosmetics and makeup routines optimized for the user's facial condition.

[0373] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0374] Users can also provide feedback on the suggested cosmetics and makeup techniques through the smartphone app, which the server then stores in a database to improve the accuracy of future suggestions.

[0375] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest the optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition.

[0376] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0377] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics.

[0378] The processing flow will be explained below.

[0379] Step 1:

[0380] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0381] Step 2:

[0382] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0383] Step 3:

[0384] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0385] Step 4:

[0386] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0387] Step 5:

[0388] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0389] Step 6:

[0390] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0391] Step 7:

[0392] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0393] Step 8:

[0394] The user begins applying makeup based on the suggestions. The device (AI mirror) guides the user by displaying specific steps for applying makeup using the suggested cosmetics in real time, step by step.

[0395] Step 9:

[0396] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0397] Step 10:

[0398] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions.

[0399] In this way, the system suggests the optimal cosmetics and makeup methods based on the user's individual preferences and facial condition, helping the user easily recreate professional makeup looks.

[0400] Example 1

[0401] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0402] In today's society, recreating professional makeup looks at any time in a busy daily life is a difficult challenge for many users. Choosing the optimal cosmetics based on individual allergies and skin conditions can be complicated, and incorrect choices can cause skin problems. Furthermore, inappropriate makeup instruction can make it difficult to achieve the desired results. To solve these problems, a system is needed that is easy for users to use and provides optimal cosmetics and makeup methods tailored to individual needs.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0404] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques using a generative AI model based on the user information and facial condition data, means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques, and means for collecting user feedback on the suggested cosmetics and makeup techniques and recording the feedback in the database. This allows users to easily recreate professional makeup looks even in their busy daily lives and select cosmetics optimized for their individual needs. Furthermore, the system can be continuously improved based on user feedback, resulting in more accurate suggestions.

[0405] "User information" refers to individual profile data that a user registers in the system, including allergy information and information about cosmetics used in the past.

[0406] A "database" is a system installed inside a server for systematically storing and managing user information and feedback information.

[0407] "Facial condition data" refers to information such as dark circles, skin tone, tan, and blemishes analyzed by the AI ​​mirror.

[0408] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal cosmetics and makeup methods based on user information and facial condition data.

[0409] A "makeup method" refers to a set of instructions including steps for using and applying cosmetic products that are optimized for the user's facial condition and individual needs.

[0410] "Step-by-step instruction" refers to providing users with sequential instructions for each step of makeup application through the AI ​​mirror and smartphone app.

[0411] "Feedback" refers to comments and evaluation information provided by users regarding the proposed cosmetics and makeup methods.

[0412] An "AI-equipped display device" refers to a display device that has built-in artificial intelligence to display makeup instruction information to users in real time.

[0413] The present invention relates to a system that registers and manages user information, analyzes facial condition, suggests optimal cosmetics and makeup techniques, and provides step-by-step makeup instruction. The system is composed of the following elements: a user, a terminal, and a server.

[0414] First, the user installs a dedicated app on a device such as a smartphone or tablet. Using this app, the user enters information about their allergies and the cosmetics they have used in the past. Specifically, this information may include nut or shellfish allergies, or whether item X from brand A was good but item Y from brand B was not. This information is then formatted and sent to the server via a secure protocol (e.g., HTTPS).

[0415] The server stores the received user information in a database. The database is in SQL or NoSQL format and is managed as profile data for each user. This makes it possible to recommend optimal cosmetics based on the user's allergy information and past usage history.

[0416] Every morning, users take a photo of their face using a smartphone app. The device (smartphone app) then sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes to generate facial condition data. The analysis results are then sent back to the server.

[0417] The server combines the received facial condition data with user information stored in a database and uses a generative AI model (such as Azure AI, Google AI, or IBM Watson) to suggest optimal cosmetics and makeup applications, such as foundation, eyeshadow, and lipstick that are free of nuts and shellfish.

[0418] The device (AI mirror and smartphone app) provides detailed instructions based on the proposed makeup method. Specifically, the AI ​​mirror provides step-by-step makeup instructions to the user in real time, such as "Next, apply eyeshadow" or "Apply foundation evenly." The user can then follow these instructions to apply makeup.

[0419] Users can also provide feedback on the proposed cosmetics and makeup techniques through a smartphone app. This feedback is sent to the server and stored in a database. This allows the server to continuously incorporate user feedback and improve the accuracy of future suggestions.

[0420] As a specific example, a user may register allergies such as "nut allergy" and "shellfish allergy" and enter information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will use this information to suggest the optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients. Furthermore, the AI ​​mirror will provide real-time instructions on how to use the suggested cosmetics, and the user will follow those instructions to apply their makeup.

[0421] An example prompt is:

[0422] "To generate makeup suggestions for the AI ​​Specialist, please use the following information to suggest the most suitable cosmetics for the user from a database. User information: Nut allergy, shellfish allergy. Past cosmetics usage history: Brand A's item X worked well, but Brand B's item Y did not. Face photo analysis results: Dark circles under the eyes, normal skin tone, slight blemishes."

[0423] The above is a specific embodiment for carrying out the present invention. This system allows users to easily recreate professional makeup looks every day and reduces the waste of unnecessary cosmetics.

[0424] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0425] Step 1:

[0426] Users install the smartphone app and launch it. They then enter information about allergies and cosmetics they have used in the past. For example, they can enter information such as "nut allergy," "shellfish allergy," "item X from brand A was good," or "item Y from brand B didn't suit me."

[0427] Input: User's allergy information and cosmetic use history

[0428] Output: Formatted user information

[0429] Step 2:

[0430] The device (smartphone app) formats the information entered by the user. In this step, the data format is organized and converted into a format that is easy for the server to receive and analyze. The formatted information is then sent to the server using a secure protocol (e.g., HTTPS).

[0431] Input: Raw information entered by the user

[0432] Output: User information formatted for easy reception by the server

[0433] Step 3:

[0434] The server stores the received user information in a database. The database is managed in SQL or NoSQL format and is saved as profile data for each user. In this process, the user information is securely registered in the database.

[0435] Input: formatted user information

[0436] Output: User profile data in the database

[0437] Step 4:

[0438] Every morning, users take a photo of their face using a smartphone app and their smartphone's camera.

[0439] Input: User's face photo

[0440] Output: Photographed face photo data

[0441] Step 5:

[0442] The device (smartphone app) takes a photo of your face and sends it to the AI ​​mirror. When sending, the photo must be of the correct resolution and format.

[0443] Input: Photographed face photo data

[0444] Output: Facial photo data sent to AI Mirror

[0445] Step 6:

[0446] The device (AI Mirror) analyzes the facial condition based on the received facial photo, including dark circles, skin tone, tan, and blemishes, and generates an analysis result.

[0447] Input: Facial photo data sent to AI Mirror

[0448] Output: Facial condition data

[0449] Step 7:

[0450] The server receives the analysis results sent by the AI ​​mirror and combines them with the user's information in the database. Using a generative AI model, it generates the optimal cosmetics and makeup routine based on the user's facial condition. For example, it suggests foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0451] Input: Facial condition data and user information from the database

[0452] Output: Recommendations for optimal cosmetics and makeup application

[0453] Step 8:

[0454] The devices (AI mirror and smartphone app) provide detailed instructions based on the generated makeup method. The AI ​​mirror provides step-by-step makeup instructions to the user in real time, providing guidance such as "Next, apply eyeshadow" and "Apply foundation evenly."

[0455] Input: Recommendations for optimal cosmetics and makeup techniques

[0456] Output: Step-by-step makeup instruction information

[0457] Step 9:

[0458] Users provide feedback on the proposed cosmetics and makeup methods through a smartphone app, which is then sent to a server and stored in a database.

[0459] Input: User feedback

[0460] Output: Feedback information stored in a database

[0461] (Application example 1)

[0462] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0463] While conventional systems can suggest the best cosmetics and makeup methods for a user's specific facial condition, they face the challenge of making it difficult to check and try on the suggestions in real time. Another issue is that the methods for providing detailed instructions on the steps of the suggested makeup methods are limited, making it difficult for users to accurately replicate them. Furthermore, there are also issues with systems that are inadequate in collecting user feedback and incorporating that information into future suggestions.

[0464] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0465] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques based on the user information and facial condition data, means for allowing the user to virtually try on the suggested cosmetics in real time, and means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques. This allows the user to check and try on the optimal cosmetics suggestions in real time, and accurately understand and execute the makeup steps. Feedback can also be collected to improve the accuracy of the suggestions.

[0466] "User information" refers to information including allergy information entered by the user, information about cosmetics used in the past, individual preferences, skin characteristics, and the like.

[0467] "Database" refers to an information system for storing and managing user information and facial condition data.

[0468] "Facial condition data" is data generated by analyzing an image of a face, and includes information such as dark circles, skin tone, blemishes, and degree of tanning.

[0469] "Cosmetics" are products such as foundation, eye shadow, lipstick, etc. that are used to enhance the aesthetic appearance of a user's face.

[0470] "Makeup method" refers to the steps and techniques for applying makeup to the face using cosmetics.

[0471] "Virtual try-on" is a technology that simulates applying suggested cosmetics to the user's face in real time, allowing them to see how they will look without actually applying them.

[0472] "Feedback" is information in which a user transmits opinions and evaluations about cosmetics and makeup methods used.

[0473] "Step-by-step instruction" is a process that provides step-by-step instructions on how to use cosmetics and how to apply makeup, helping users to apply makeup correctly.

[0474] The following describes an embodiment of the present invention. The system of the present invention uses a smartphone app, a server, and AI analysis technology to propose optimal cosmetics and makeup methods based on the user's facial condition, and is equipped with virtual try-on and step-by-step instruction functions.

[0475] First, the user installs the smartphone app and enters information about their allergies and the cosmetics they have used in the past. This information is formatted and stored in a cloud database (e.g., Firebase). The server then manages the received user information individually. Based on the information stored in this database, the app can suggest the most suitable cosmetics based on the user's allergies and past usage history.

[0476] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is sent to an AI analysis engine (e.g., TensorFlow Lite, OpenCV) and analyzed for information such as dark circles, skin tone, sun exposure, and blemishes. The smartphone app then sends the photo to the AI ​​analysis engine and receives the results.

[0477] The server receives the analysis results from the AI ​​analysis engine, combines them with information stored in the user's database, and runs an algorithm to generate optimal cosmetics and makeup techniques, which are then presented to the user in real time.

[0478] Furthermore, the proposed cosmetics are applied to the user's face in real time using virtual try-on features (e.g., OpenCV, camera API), allowing users to see how the products will look without actually applying them.

[0479] Based on the proposed makeup application, step-by-step instructions are provided. The smartphone app uses real-time video and audio to provide detailed instructions that users can follow. Based on the suggestions, users can also provide feedback, which is recorded in a database to improve the accuracy of future suggestions.

[0480] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​analysis engine analyzes the results as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition. The system also allows users to virtually try on the suggested cosmetics, and if they are satisfied, they can purchase them immediately.

[0481] Prompt Sentence Examples

[0482] markdown

[0483] Prompt statement

[0484] User information: Nut allergy, cosmetics used in the past: Brand A, item X (good), Brand B, item Y (unsuitable)

[0485] Take a photo of your face and output the analysis results: dark circles, normal skin tone, slight blemishes

[0486] Based on the analysis results generated, the app will suggest the best cosmetics and makeup techniques.

[0487] Simply apply the suggested items to your face virtually and provide a picture.

[0488] Provide step-by-step instructions for users to proceed with their makeup.

[0489] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0490] Program processing steps

[0491] Step 1:

[0492] Users enter information about their allergies and cosmetics they have used in the past into a smartphone app.

[0493] Input: Allergy information (e.g., nut allergy, shellfish allergy), past cosmetic use information (e.g., item X from brand A is fine, but item Y from brand B is incompatible)

[0494] Data processing: The input information is formatted into JSON format and sent to the server.

[0495] Output: Formatted user information (JSON)

[0496] Step 2:

[0497] The server stores the received user information in a cloud database (e.g., Firebase).

[0498] Input: User information (JSON)

[0499] Data processing: Save as an entry in a database and associate with an individual user ID.

[0500] Output: Database entries

[0501] Step 3:

[0502] A user takes a photo of their face every morning using a smartphone app.

[0503] Input: User's face photo

[0504] Data processing: Save the image in the correct format and send it to the AI ​​analysis engine.

[0505] Output: Sent face photo (image)

[0506] Step 4:

[0507] An AI analysis engine (e.g., TensorFlow Lite, OpenCV) analyzes the facial photo and generates facial condition data.

[0508] Input: Face photo (image)

[0509] Data processing: Analyzes facial dark circles, skin tone, blemishes, etc. and converts the results into JSON format.

[0510] Output: Facial condition data (JSON)

[0511] Step 5:

[0512] The server runs an algorithm that generates optimal cosmetics and makeup methods based on facial condition data and user information.

[0513] Input: Facial condition data (JSON), user information

[0514] Data processing: Analyze data using an AI model and suggest the most suitable cosmetics and makeup techniques.

[0515] Output: Suggested cosmetics and makeup methods (list and step-by-step instructions)

[0516] Step 6:

[0517] The smartphone app applies the suggested cosmetics to the user's face in real time using a virtual try-on function (e.g., OpenCV, camera API).

[0518] Input: Suggested cosmetics (list), user's face photo (image)

[0519] Data processing: Using a virtual try-on algorithm, images of cosmetics applied to the user's face are generated in real time.

[0520] Output: Virtual try-on image

[0521] Step 7:

[0522] The smartphone app provides real-time step-by-step instructions based on the proposed makeup application method, providing detailed instructions using video and audio.

[0523] Input: Suggested makeup method (step-by-step instructions)

[0524] Data processing: Generate step-by-step video and audio guides and display and play them to the user in real time.

[0525] Output: Video and audio guide

[0526] Step 8:

[0527] Users provide feedback on the results of their makeup through a smartphone app, and the server records it in a database.

[0528] Input: User feedback (text or rating)

[0529] Data processing: Feedback information is stored in a database so that it can be reflected in future proposals.

[0530] Output: Feedback stored in a database

[0531] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0532] An embodiment for implementing the system of the present invention will be described.

[0533] The system is equipped with an interface that registers and manages user information, analyzes facial condition, suggests optimal makeup and cosmetics, provides instructions on how to apply makeup, and an emotion engine that recognizes the user's emotions.

[0534] First, the user installs the smartphone app and, when they launch it for the first time, enters information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like). The smartphone app formats this information and sends it to the server. The server then stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0535] Next, each morning, the user takes a photo of their face using a smartphone app. The photo is sent to the AI ​​mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes. The server receives the analysis results from the AI ​​mirror and combines them with information stored in the user's database to generate recommendations for the optimal cosmetic set and makeup method for the user.

[0536] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0537] This system also incorporates an emotion engine that recognizes the user's emotions. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if the user is feeling down, it will suggest refreshing makeup that will cheer them up. In this way, the system dynamically adjusts the suggestions according to the user's emotional state, thereby improving user satisfaction.

[0538] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," and the emotion engine further recognizes that the user is "feeling stressed," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients, based on the user's data, facial condition, and emotion data, as well as relaxing makeup to relieve stress.

[0539] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0540] The user's emotions recognized by the emotion engine are also recorded in a database and used to suggest makeup looks for the next time onward, enabling optimal suggestions to be made in response to fluctuations in the user's emotional state.

[0541] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics. Furthermore, by making suggestions based on the user's emotions, further improvements in satisfaction can be expected.

[0542] The processing flow will be explained below.

[0543] Step 1:

[0544] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0545] Step 2:

[0546] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0547] Step 3:

[0548] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0549] Step 4:

[0550] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0551] Step 5:

[0552] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0553] Step 6:

[0554] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0555] Step 7:

[0556] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0557] Step 8:

[0558] The user begins applying makeup based on the suggested cosmetics and makeup methods. The device (AI mirror) guides the user by displaying specific makeup steps using the suggested cosmetics in real time, step by step.

[0559] Step 9:

[0560] While the user is applying makeup, the emotion engine analyzes the user's facial expressions and voice to generate emotion data, for example, determining whether the user is smiling or feeling stressed through a camera or microphone.

[0561] Step 10:

[0562] The server receives the emotion data generated by the emotion engine. Based on this, it adjusts the initial cosmetics and makeup suggestions and re-suggests the most optimal makeup method. For example, if the user is tired, it suggests makeup that has a refreshing effect.

[0563] Step 11:

[0564] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0565] Step 12:

[0566] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions. It also records the user's emotions recognized by the emotion engine in a database and uses this information to improve future makeup suggestions.

[0567] In this way, the system suggests the optimal cosmetics and makeup techniques based on the user's individual preferences, facial condition, and even emotions, helping them easily recreate professional makeup looks. Furthermore, the suggestions are constantly adjusted based on the user's latest condition and emotions, providing a high level of satisfaction.

[0568] Example 2

[0569] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Conventional makeup systems rely on the user's qualitative senses and feedback, making it difficult to provide optimal recommendations based on scientific evidence or an individual's emotional state. Furthermore, because they do not take into account the user's past usage history or emotions, they are unable to provide optimal makeup recommendations for individual needs.

[0571] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial state data, means for suggesting optimal cosmetics and makeup techniques based on the user information and the facial state data, means for providing makeup instructions based on the suggested cosmetics and makeup techniques, and means for analyzing emotions from the user's facial expressions and voice, saving the emotion data in a database, and dynamically adjusting the makeup suggestions. This makes it possible to suggest optimal makeup based on the user's past usage history and emotional state, and to provide a satisfying makeup experience that meets the needs of each individual user.

[0572] "User information" refers to personal allergy information entered by the user, history information on cosmetics used in the past, and the like.

[0573] "Database" refers to a storage device for storing and managing user information, facial state data, emotion data, and other related information.

[0574] "Facial condition data" refers to information about facial condition such as dark circles under the eyes, skin tone, degree of tanning, and blemishes, which is analyzed from a facial photograph.

[0575] "Cosmetics" refers to items used in makeup, such as foundation, eye shadow, and lipstick.

[0576] "Suggestion" refers to the process of presenting the user with the most suitable cosmetics and makeup methods based on the user information and facial condition data.

[0577] "Emotional data" refers to information about the emotional state of a user analyzed from facial expressions and voice.

[0578] "Makeup procedure instruction" refers to the process of guiding a user through specific makeup steps based on the proposed cosmetics and makeup method.

[0579] "User feedback" refers to information such as impressions and improvements provided by users after completing their makeup.

[0580] MODE FOR CARRYING OUT THE INVENTION

[0581] The present invention provides a makeup suggestion system that registers and manages user information, analyzes facial conditions, suggests optimal cosmetics and makeup methods, provides makeup instruction, and includes an emotion engine that recognizes the user's emotions. Hereinafter, an embodiment of the present invention will be described.

[0582] User information registration and management

[0583] First, the user installs the smartphone app and, when launching it for the first time, enters allergy information and information about cosmetics used in the past. At this time, the user enters allergy information such as "nut allergy" or "shellfish allergy" as well as a history of cosmetics use, such as "Brand A's item X was good" or "Brand B's item Y did not suit me." The smartphone app formats this information and sends it to the server.

[0584] The server stores the received user information in a database and manages it individually. Based on this stored information, it can respond to the user's allergies and suggest cosmetics based on their past usage history.

[0585] Analysis of facial condition

[0586] Next, every morning, users take a photo of their face using a smartphone app, which then sends the photo to the AI ​​Mirror for analysis. The AI ​​Mirror then analyzes the photo to determine whether they have dark circles, skin tone, sun exposure, or blemishes.

[0587] The device (smartphone app) sends the captured facial photo to the AI ​​Mirror, which then sends the analysis results to the server. The AI ​​Mirror's data includes specific information such as "there are dark circles under the eyes," "the skin tone is normal," and "there are some blemishes."

[0588] Proposal of optimal cosmetics and makeup methods

[0589] The server compares the analysis results received from the AI ​​mirror with the user's database information to generate the optimal cosmetic set and makeup method. This includes foundation, eye shadow, lipstick, etc. that do not contain nuts or shellfish ingredients. It also takes into account emotional data analyzed by the emotion engine and suggests makeup methods that correspond to the user's emotions. For example, if the user is "feeling stressed," it will suggest relaxing makeup.

[0590] Makeup instruction

[0591] Next, the device (AI mirror and smartphone app) will provide step-by-step instructions to the user based on the generated makeup method, displaying detailed instructions and guiding the user, such as, "First, apply this concealer under your eyes to hide dark circles."

[0592] The user follows the instructions to apply the makeup. After completing the makeup, the user provides feedback on the completed makeup, which is then stored in a database and reflected in future suggestions.

[0593] Optimization by Emotion Engine

[0594] The system also incorporates an emotion engine. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions it recommends based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if they are feeling down, it will suggest refreshing makeup that will cheer them up. In this way, dynamically adjusting the suggestions according to the user's emotional state increases user satisfaction.

[0595] Examples and prompts

[0596] As a concrete example, we will explain the process when a user uses the system for the first time. The user installs the smartphone app and enters their "nut allergy" and "shellfish allergy." They also enter their past cosmetic use history, such as "Item X from brand A was good, but item Y from brand B didn't suit me."

[0597] The next morning, the user takes a photo of their face and sends it to the AI ​​mirror. The AI ​​mirror analyzes the photo and determines whether they have dark circles under their eyes, normal skin tone, or slight blemishes, and sends the results to the server. If the emotion engine determines that the user is feeling stressed, the server will suggest relaxing makeup, including foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0598] An example prompt is, "I have nut and shellfish allergies. In the past, item X from brand A worked well for me, but item Y from brand B did not. I would like you to analyze a photo of my face this morning and suggest the best cosmetics and makeup techniques. I am currently feeling stressed."

[0599] The above is an embodiment of the present invention. By using this system, users can easily recreate professional makeup looks every day, providing a highly satisfying makeup experience that meets the needs of each individual user.

[0600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0601] Program processing flow

[0602] Step 1: Register user information

[0603] 1. Input: The user installs the smartphone app and enters information about allergies and cosmetics used in the past.

[0604] Examples: "Nut allergy", "Shellfish allergy", cosmetics used in the past: "Item X from brand A was good", "Item Y from brand B didn't suit me".

[0605] 2. How it works: The smartphone app properly formats the entered information and sends it to the server.

[0606] 3. Output: The server stores the received user information in a database, providing the basic data for future cosmetic recommendations.

[0607] Step 2: Analyze your facial condition

[0608] 1. Input: Every morning, the user takes a photo of their face using a smartphone app.

[0609] 2. How it works: The smartphone app sends the captured facial photo to the AI ​​mirror.

[0610] 3. How it works: The AI ​​mirror analyzes the received facial photo and generates facial condition data, such as whether the person has dark circles under their eyes, whether their skin tone is normal, or whether they have some blemishes.

[0611] 4. Output: The AI ​​mirror sends the analysis results to the server and stores them in the user's database.

[0612] Step 3: Recommendations for the best cosmetics and makeup techniques

[0613] 1. Input: The server performs analysis based on facial condition data and the user's database information (allergy information and cosmetic use history).

[0614] 2. Operation: The server combines facial condition and user information to generate optimal cosmetics and makeup methods, such as recommending foundation, eye shadow, and lipstick that are free of nuts and shellfish ingredients.

[0615] 3. How it works: The emotion engine analyzes emotional data from the user's facial expressions and voice and adjusts the suggestions.

[0616] 4. Output: The server sends the optimal cosmetic set and makeup method for the user to the device (smartphone app and AI mirror).

[0617] Step 4: Makeup Instructions

[0618] 1. Input: Proposal content sent from the server (cosmetics set and makeup method).

[0619] 2. Operation: The device (smartphone app and AI mirror) displays the suggested cosmetics and makeup methods to the user, and provides detailed step-by-step instructions for use. For example, "First, apply this concealer under your eyes to hide dark circles."

[0620] 3. Output: The user follows the instructions to apply makeup.

[0621] Step 5: Collect user feedback and sentiment data

[0622] 1. Input: User feedback after makeup is completed.

[0623] 2. Action: The user enters feedback about the makeup through the smartphone app. For example, "This makeup was very good" or "This eyeshadow didn't suit me."

[0624] 3. Operation: The emotion engine analyzes emotion data from the user's facial expressions and voice.

[0625] 4. Output: The server stores the collected feedback and sentiment data in a database, which will be used to improve future suggestions.

[0626] Specific steps in the overall flow

[0627] These processing steps allow users to easily receive recommendations for appropriate cosmetics and makeup techniques each day, and follow the instructions to apply their makeup. The system is continuously optimized, as user feedback and emotional data are reflected in the next recommendations.

[0628] (Application example 2)

[0629] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] In today's world, it is important to suggest optimal cosmetics and makeup techniques based on each individual user's skin condition and emotional state. However, no previous technology existed that could perform real-time facial image analysis and emotional analysis of the user and then suggest and instruct specific makeup techniques based on the results. Furthermore, there was insufficient support for applying the suggested makeup techniques in physical stores, resulting in a lack of assistance for users in selecting the most suitable products. Furthermore, there was also a lack of a mechanism for accumulating user feedback in a database and reflecting it in future suggestions.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0632] In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial condition data, means for analyzing the user's emotions and further optimizing cosmetics and makeup methods based on the emotion data, and means for displaying the suggested cosmetics and makeup methods on a smart mirror and smart glasses in a physical store to guide the user. This makes it possible to suggest optimal cosmetics and makeup methods based on the user's skin condition and emotional state, and to provide specific makeup instructions in the physical store in real time.

[0633] "User information" refers to data including personal information about the user, allergy information, and a history of past cosmetic use.

[0634] "Database" refers to a collection of information for storing and managing user information, feedback information, etc.

[0635] "Facial condition data" refers to data regarding the condition of the user's face, such as skin tone, dark circles, and blemishes, analyzed based on a photograph of the user's face.

[0636] "Emotion data" is data relating to the emotional state of the user acquired by analyzing the user's facial expressions and voice.

[0637] "Cosmetics" refers to products used for makeup and skin care, including foundation, eye shadow, lipstick, etc.

[0638] "Makeup method" refers to specific makeup procedures and techniques suggested to the user.

[0639] "Suggestion" refers to a recommendation of optimal cosmetics and makeup techniques generated based on user information, facial condition data, and emotion data.

[0640] "Step-by-step instruction" refers to the process of teaching a user how to apply makeup step by step.

[0641] "Brick and mortar store" refers to a retail store that has a physical presence and where consumers can visit and purchase products.

[0642] A "smart mirror" is a highly functional mirror equipped with a camera and display that analyzes facial condition and provides makeup guidance.

[0643] "Smart glasses" are glasses-type devices that are worn by a user and have the function of displaying information for makeup guidance.

[0644] "Feedback" refers to information about the evaluations and impressions provided by users regarding the proposed cosmetics and makeup methods.

[0645] "Server" refers to a centralized device that processes and stores user information, facial condition data, and emotion data over a network.

[0646] The following describes an embodiment of the present invention. This system registers individual user information, and proposes optimal cosmetics and makeup methods based on the analysis of the user's facial condition and emotions, providing makeup guidance in real time.

[0647] User information registration and management

[0648] Users enter their personal information (such as allergy information and past history of cosmetic use) into a smartphone application. The application then sends the collected information to a server to store in a database. The server then records and manages the user information in the database. This database serves as the basis for making optimal recommendations based on the user's past usage and allergy information.

[0649] Facial condition and emotion analysis

[0650] The user stands in front of a smart mirror installed in a physical store and takes a photo of their face. The smart mirror uses its built-in camera to capture an image of their face and analyzes it. An AI model running on TensorFlow and Keras is used for the analysis to obtain facial condition data such as the user's skin tone, dark circles, and the presence or absence of blemishes. At the same time, emotional data is also analyzed from the user's facial expressions and voice. An emotion recognition engine is used for this emotional analysis.

[0651] Generate optimized proposals

[0652] The server compares the user's facial condition data and emotion data with a user information database to generate the optimal cosmetics and makeup method. The server generates a list of cosmetics and makeup procedures that are optimal for the presented conditions and sends them to the smart mirror and smart glasses.

[0653] Providing real-time makeup guides

[0654] When a user wears smart glasses and applies makeup in front of a smart mirror, the glasses display suggested cosmetics and makeup steps. Audio instructions are also provided. This real-time guide is implemented using Python libraries (e.g., pyttsx3).

[0655] Collecting and incorporating user feedback

[0656] After the user has completed their makeup application, they can submit feedback about the proposed cosmetics and makeup techniques through the application. This feedback information is sent to the server and recorded in a database. This feedback is then reflected in future suggestions, enabling more accurate suggestions to be made.

[0657] Examples of concrete examples and prompts

[0658] For example, a user puts on smart glasses and stands in front of a smart mirror in a brick-and-mortar store. The smart mirror takes a photo of the user's face and analyzes their skin tone, presence of dark circles, and their emotions. Based on the results of this analysis and the user's allergy information, the system suggests the most suitable cosmetics (e.g., foundation from brand X, lipstick from brand Y). Below is an example of a prompt to be input into the generative AI model.

[0659] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[0660] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0661] Step 1:

[0662] Users enter personal information into a smartphone app, including allergy information and a history of past cosmetic use. The app formats this information and sends it to a server. The input data, which includes allergy information and past cosmetic use history, is sent to the server and stored in a database.

[0663] Step 2:

[0664] A user stands in front of a smart mirror installed in a brick-and-mortar store. The smart mirror uses a built-in camera to capture an image of the user's face. This image data is input and sent to an AI model for analysis. The output is facial condition data, including information on skin tone, dark circles, blemishes, etc.

[0665] Step 3:

[0666] The server analyzes the captured facial image using an AI model (using TensorFlow and Keras) to generate condition data. This analysis is achieved by evaluating each facial parameter through the AI ​​model. The input is the facial image, and the output is facial condition data.

[0667] Step 4:

[0668] The smart mirror also uses the user's voice and facial expressions to analyze emotional data in an emotion engine. This data is analyzed by an emotion recognition engine. The input is the user's voice and facial expressions, and the output is emotional data.

[0669] Step 5:

[0670] The server combines the user information database, facial condition data, and emotional data to suggest optimal cosmetics and makeup methods based on a generative AI model. During this process, each piece of data is collated, and the generative AI model calculates the optimal combination. The input is user information, facial condition data, and emotional data, and the output is a list of suggested cosmetics and makeup methods.

[0671] Step 6:

[0672] The server sends the suggested cosmetics and makeup techniques to the smart mirror and smart glasses. The smart glasses then display step-by-step instructions on how to use the cosmetics and how to apply makeup to the user. They also provide audio guidance to support the user. The input is the suggested information, and the output is guidance information for the user.

[0673] Step 7:

[0674] After the user completes their makeup, they submit feedback through the application. This feedback is stored on the server and reflected in future suggestions. The input is the user's feedback, and the output is the improved accuracy of the suggestions.

[0675] As a concrete example, the following prompt sentences can be input to a generative AI model:

[0676] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[0677] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0678] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0679] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0680] [Third embodiment]

[0681] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0682] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0683] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0684] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0685] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0686] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0687] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0688] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0689] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0690] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0691] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0692] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0693] An embodiment for implementing the system of the present invention will be described.

[0694] The system has an interface that allows users to register and manage their information, analyze their facial condition, suggest optimal makeup and cosmetics, and provide instructions on how to apply makeup.

[0695] First, the user installs the smartphone app and enters information about allergies and cosmetics they have used in the past (favorite cosmetics and cosmetics that did not suit them). The smartphone app formats this information and sends it to the server to be stored in a database.

[0696] The server stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0697] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is then sent to the AI ​​Mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​Mirror, which analyzes information such as dark circles, skin tone, sun exposure, and blemishes.

[0698] The server receives the analysis results from the AI ​​mirror, combines them with information stored in the user's database, and runs an algorithm to generate the optimal cosmetics and makeup routine. Using the AI ​​model, it suggests a set of cosmetics and makeup routines optimized for the user's facial condition.

[0699] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0700] Users can also provide feedback on the suggested cosmetics and makeup techniques through the smartphone app, which the server then stores in a database to improve the accuracy of future suggestions.

[0701] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest the optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition.

[0702] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0703] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics.

[0704] The processing flow will be explained below.

[0705] Step 1:

[0706] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0707] Step 2:

[0708] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0709] Step 3:

[0710] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0711] Step 4:

[0712] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0713] Step 5:

[0714] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0715] Step 6:

[0716] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0717] Step 7:

[0718] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0719] Step 8:

[0720] The user begins applying makeup based on the suggestions. The device (AI mirror) guides the user by displaying specific steps for applying makeup using the suggested cosmetics in real time, step by step.

[0721] Step 9:

[0722] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0723] Step 10:

[0724] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions.

[0725] In this way, the system suggests the optimal cosmetics and makeup methods based on the user's individual preferences and facial condition, helping the user easily recreate professional makeup looks.

[0726] Example 1

[0727] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0728] In today's society, recreating professional makeup looks at any time in a busy daily life is a difficult challenge for many users. Choosing the optimal cosmetics based on individual allergies and skin conditions can be complicated, and incorrect choices can cause skin problems. Furthermore, inappropriate makeup instruction can make it difficult to achieve the desired results. To solve these problems, a system is needed that is easy for users to use and provides optimal cosmetics and makeup methods tailored to individual needs.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0730] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques using a generative AI model based on the user information and facial condition data, means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques, and means for collecting user feedback on the suggested cosmetics and makeup techniques and recording the feedback in the database. This allows users to easily recreate professional makeup looks even in their busy daily lives and select cosmetics optimized for their individual needs. Furthermore, the system can be continuously improved based on user feedback, resulting in more accurate suggestions.

[0731] "User information" refers to individual profile data that a user registers in the system, including allergy information and information about cosmetics used in the past.

[0732] A "database" is a system installed inside a server for systematically storing and managing user information and feedback information.

[0733] "Facial condition data" refers to information such as dark circles, skin tone, tan, and blemishes analyzed by the AI ​​mirror.

[0734] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal cosmetics and makeup methods based on user information and facial condition data.

[0735] A "makeup method" refers to a set of instructions including steps for using and applying cosmetic products that are optimized for the user's facial condition and individual needs.

[0736] "Step-by-step instruction" refers to providing users with sequential instructions for each step of makeup application through the AI ​​mirror and smartphone app.

[0737] "Feedback" refers to comments and evaluation information provided by users regarding the proposed cosmetics and makeup methods.

[0738] An "AI-equipped display device" refers to a display device that has built-in artificial intelligence to display makeup instruction information to users in real time.

[0739] The present invention relates to a system that registers and manages user information, analyzes facial condition, suggests optimal cosmetics and makeup techniques, and provides step-by-step makeup instruction. The system is composed of the following elements: a user, a terminal, and a server.

[0740] First, the user installs a dedicated app on a device such as a smartphone or tablet. Using this app, the user enters information about their allergies and the cosmetics they have used in the past. Specifically, this information may include nut or shellfish allergies, or whether item X from brand A was good but item Y from brand B was not. This information is then formatted and sent to the server via a secure protocol (e.g., HTTPS).

[0741] The server stores the received user information in a database. The database is in SQL or NoSQL format and is managed as profile data for each user. This makes it possible to recommend optimal cosmetics based on the user's allergy information and past usage history.

[0742] Every morning, users take a photo of their face using a smartphone app. The device (smartphone app) then sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes to generate facial condition data. The analysis results are then sent back to the server.

[0743] The server combines the received facial condition data with user information stored in a database and uses a generative AI model (such as Azure AI, Google AI, or IBM Watson) to suggest optimal cosmetics and makeup applications, such as foundation, eyeshadow, and lipstick that are free of nuts and shellfish.

[0744] The device (AI mirror and smartphone app) provides detailed instructions based on the proposed makeup method. Specifically, the AI ​​mirror provides step-by-step makeup instructions to the user in real time, such as "Next, apply eyeshadow" or "Apply foundation evenly." The user can then follow these instructions to apply makeup.

[0745] Users can also provide feedback on the proposed cosmetics and makeup techniques through a smartphone app. This feedback is sent to the server and stored in a database. This allows the server to continuously incorporate user feedback and improve the accuracy of future suggestions.

[0746] As a specific example, a user may register allergies such as "nut allergy" and "shellfish allergy" and enter information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will use this information to suggest the optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients. Furthermore, the AI ​​mirror will provide real-time instructions on how to use the suggested cosmetics, and the user will follow those instructions to apply their makeup.

[0747] An example prompt is:

[0748] "To generate makeup suggestions for the AI ​​Specialist, please use the following information to suggest the most suitable cosmetics for the user from a database. User information: Nut allergy, shellfish allergy. Past cosmetics usage history: Brand A's item X worked well, but Brand B's item Y did not. Face photo analysis results: Dark circles under the eyes, normal skin tone, slight blemishes."

[0749] The above is a specific embodiment for carrying out the present invention. This system allows users to easily recreate professional makeup looks every day and reduces the waste of unnecessary cosmetics.

[0750] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] Users install the smartphone app and launch it. They then enter information about allergies and cosmetics they have used in the past. For example, they can enter information such as "nut allergy," "shellfish allergy," "item X from brand A was good," or "item Y from brand B didn't suit me."

[0753] Input: User's allergy information and cosmetic use history

[0754] Output: Formatted user information

[0755] Step 2:

[0756] The device (smartphone app) formats the information entered by the user. In this step, the data format is organized and converted into a format that is easy for the server to receive and analyze. The formatted information is then sent to the server using a secure protocol (e.g., HTTPS).

[0757] Input: Raw information entered by the user

[0758] Output: User information formatted for easy reception by the server

[0759] Step 3:

[0760] The server stores the received user information in a database. The database is managed in SQL or NoSQL format and is saved as profile data for each user. In this process, the user information is securely registered in the database.

[0761] Input: formatted user information

[0762] Output: User profile data in the database

[0763] Step 4:

[0764] Every morning, users take a photo of their face using a smartphone app and their smartphone's camera.

[0765] Input: User's face photo

[0766] Output: Photographed face photo data

[0767] Step 5:

[0768] The device (smartphone app) takes a photo of your face and sends it to the AI ​​mirror. When sending, the photo must be of the correct resolution and format.

[0769] Input: Photographed face photo data

[0770] Output: Facial photo data sent to AI Mirror

[0771] Step 6:

[0772] The device (AI Mirror) analyzes the facial condition based on the received facial photo, including dark circles, skin tone, tan, and blemishes, and generates an analysis result.

[0773] Input: Facial photo data sent to AI Mirror

[0774] Output: Facial condition data

[0775] Step 7:

[0776] The server receives the analysis results sent by the AI ​​mirror and combines them with the user's information in the database. Using a generative AI model, it generates the optimal cosmetics and makeup routine based on the user's facial condition. For example, it suggests foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0777] Input: Facial condition data and user information from the database

[0778] Output: Recommendations for optimal cosmetics and makeup application

[0779] Step 8:

[0780] The devices (AI mirror and smartphone app) provide detailed instructions based on the generated makeup method. The AI ​​mirror provides step-by-step makeup instructions to the user in real time, providing guidance such as "Next, apply eyeshadow" and "Apply foundation evenly."

[0781] Input: Recommendations for optimal cosmetics and makeup techniques

[0782] Output: Step-by-step makeup instruction information

[0783] Step 9:

[0784] Users provide feedback on the proposed cosmetics and makeup methods through a smartphone app, which is then sent to a server and stored in a database.

[0785] Input: User feedback

[0786] Output: Feedback information stored in a database

[0787] (Application example 1)

[0788] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0789] While conventional systems can suggest the best cosmetics and makeup methods for a user's specific facial condition, they face the challenge of making it difficult to check and try on the suggestions in real time. Another issue is that the methods for providing detailed instructions on the steps of the suggested makeup methods are limited, making it difficult for users to accurately replicate them. Furthermore, there are also issues with systems that are inadequate in collecting user feedback and incorporating that information into future suggestions.

[0790] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0791] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques based on the user information and facial condition data, means for allowing the user to virtually try on the suggested cosmetics in real time, and means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques. This allows the user to check and try on the optimal cosmetics suggestions in real time, and accurately understand and execute the makeup steps. Feedback can also be collected to improve the accuracy of the suggestions.

[0792] "User information" refers to information including allergy information entered by the user, information about cosmetics used in the past, individual preferences, skin characteristics, and the like.

[0793] "Database" refers to an information system for storing and managing user information and facial condition data.

[0794] "Facial condition data" is data generated by analyzing an image of a face, and includes information such as dark circles, skin tone, blemishes, and degree of tanning.

[0795] "Cosmetics" are products such as foundation, eye shadow, lipstick, etc. that are used to enhance the aesthetic appearance of a user's face.

[0796] "Makeup method" refers to the steps and techniques for applying makeup to the face using cosmetics.

[0797] "Virtual try-on" is a technology that simulates applying suggested cosmetics to the user's face in real time, allowing them to see how they will look without actually applying them.

[0798] "Feedback" is information in which a user transmits opinions and evaluations about cosmetics and makeup methods used.

[0799] "Step-by-step instruction" is a process that provides step-by-step instructions on how to use cosmetics and how to apply makeup, helping users to apply makeup correctly.

[0800] The following describes an embodiment of the present invention. The system of the present invention uses a smartphone app, a server, and AI analysis technology to propose optimal cosmetics and makeup methods based on the user's facial condition, and is equipped with virtual try-on and step-by-step instruction functions.

[0801] First, the user installs the smartphone app and enters information about their allergies and the cosmetics they have used in the past. This information is formatted and stored in a cloud database (e.g., Firebase). The server then manages the received user information individually. Based on the information stored in this database, the app can suggest the most suitable cosmetics based on the user's allergies and past usage history.

[0802] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is sent to an AI analysis engine (e.g., TensorFlow Lite, OpenCV) and analyzed for information such as dark circles, skin tone, sun exposure, and blemishes. The smartphone app then sends the photo to the AI ​​analysis engine and receives the results.

[0803] The server receives the analysis results from the AI ​​analysis engine, combines them with information stored in the user's database, and runs an algorithm to generate optimal cosmetics and makeup techniques, which are then presented to the user in real time.

[0804] Furthermore, the proposed cosmetics are applied to the user's face in real time using virtual try-on features (e.g., OpenCV, camera API), allowing users to see how the products will look without actually applying them.

[0805] Based on the proposed makeup application, step-by-step instructions are provided. The smartphone app uses real-time video and audio to provide detailed instructions that users can follow. Based on the suggestions, users can also provide feedback, which is recorded in a database to improve the accuracy of future suggestions.

[0806] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​analysis engine analyzes the results as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition. The system also allows users to virtually try on the suggested cosmetics, and if they are satisfied, they can purchase them immediately.

[0807] Prompt Sentence Examples

[0808] markdown

[0809] Prompt statement

[0810] User information: Nut allergy, cosmetics used in the past: Brand A, item X (good), Brand B, item Y (unsuitable)

[0811] Take a photo of your face and output the analysis results: dark circles, normal skin tone, slight blemishes

[0812] Based on the analysis results generated, the app will suggest the best cosmetics and makeup techniques.

[0813] Simply apply the suggested items to your face virtually and provide a picture.

[0814] Provide step-by-step instructions for users to proceed with their makeup.

[0815] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0816] Program processing steps

[0817] Step 1:

[0818] Users enter information about their allergies and cosmetics they have used in the past into a smartphone app.

[0819] Input: Allergy information (e.g., nut allergy, shellfish allergy), past cosmetic use information (e.g., item X from brand A is fine, but item Y from brand B is incompatible)

[0820] Data processing: The input information is formatted into JSON format and sent to the server.

[0821] Output: Formatted user information (JSON)

[0822] Step 2:

[0823] The server stores the received user information in a cloud database (e.g., Firebase).

[0824] Input: User information (JSON)

[0825] Data processing: Save as an entry in a database and associate with an individual user ID.

[0826] Output: Database entries

[0827] Step 3:

[0828] A user takes a photo of their face every morning using a smartphone app.

[0829] Input: User's face photo

[0830] Data processing: Save the image in the correct format and send it to the AI ​​analysis engine.

[0831] Output: Sent face photo (image)

[0832] Step 4:

[0833] An AI analysis engine (e.g., TensorFlow Lite, OpenCV) analyzes the facial photo and generates facial condition data.

[0834] Input: Face photo (image)

[0835] Data processing: Analyzes facial dark circles, skin tone, blemishes, etc. and converts the results into JSON format.

[0836] Output: Facial condition data (JSON)

[0837] Step 5:

[0838] The server runs an algorithm that generates optimal cosmetics and makeup methods based on facial condition data and user information.

[0839] Input: Facial condition data (JSON), user information

[0840] Data processing: Analyze data using an AI model and suggest the most suitable cosmetics and makeup techniques.

[0841] Output: Suggested cosmetics and makeup methods (list and step-by-step instructions)

[0842] Step 6:

[0843] The smartphone app applies the suggested cosmetics to the user's face in real time using a virtual try-on function (e.g., OpenCV, camera API).

[0844] Input: Suggested cosmetics (list), user's face photo (image)

[0845] Data processing: Using a virtual try-on algorithm, images of cosmetics applied to the user's face are generated in real time.

[0846] Output: Virtual try-on image

[0847] Step 7:

[0848] The smartphone app provides real-time step-by-step instructions based on the proposed makeup application method, providing detailed instructions using video and audio.

[0849] Input: Suggested makeup method (step-by-step instructions)

[0850] Data processing: Generate step-by-step video and audio guides and display and play them to the user in real time.

[0851] Output: Video and audio guide

[0852] Step 8:

[0853] Users provide feedback on the results of their makeup through a smartphone app, and the server records it in a database.

[0854] Input: User feedback (text or rating)

[0855] Data processing: Feedback information is stored in a database so that it can be reflected in future proposals.

[0856] Output: Feedback stored in a database

[0857] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0858] An embodiment for implementing the system of the present invention will be described.

[0859] The system is equipped with an interface that registers and manages user information, analyzes facial condition, suggests optimal makeup and cosmetics, provides instructions on how to apply makeup, and an emotion engine that recognizes the user's emotions.

[0860] First, the user installs the smartphone app and, when they launch it for the first time, enters information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like). The smartphone app formats this information and sends it to the server. The server then stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[0861] Next, each morning, the user takes a photo of their face using a smartphone app. The photo is sent to the AI ​​mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes. The server receives the analysis results from the AI ​​mirror and combines them with information stored in the user's database to generate recommendations for the optimal cosmetic set and makeup method for the user.

[0862] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[0863] This system also incorporates an emotion engine that recognizes the user's emotions. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if the user is feeling down, it will suggest refreshing makeup that will cheer them up. In this way, the system dynamically adjusts the suggestions according to the user's emotional state, thereby improving user satisfaction.

[0864] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," and the emotion engine further recognizes that the user is "feeling stressed," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients, based on the user's data, facial condition, and emotion data, as well as relaxing makeup to relieve stress.

[0865] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[0866] The user's emotions recognized by the emotion engine are also recorded in a database and used to suggest makeup looks for the next time onward, enabling optimal suggestions to be made in response to fluctuations in the user's emotional state.

[0867] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics. Furthermore, by making suggestions based on the user's emotions, further improvements in satisfaction can be expected.

[0868] The processing flow will be explained below.

[0869] Step 1:

[0870] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[0871] Step 2:

[0872] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[0873] Step 3:

[0874] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[0875] Step 4:

[0876] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[0877] Step 5:

[0878] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[0879] Step 6:

[0880] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[0881] Step 7:

[0882] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[0883] Step 8:

[0884] The user begins applying makeup based on the suggested cosmetics and makeup methods. The device (AI mirror) guides the user by displaying specific makeup steps using the suggested cosmetics in real time, step by step.

[0885] Step 9:

[0886] While the user is applying makeup, the emotion engine analyzes the user's facial expressions and voice to generate emotion data, for example, determining whether the user is smiling or feeling stressed through a camera or microphone.

[0887] Step 10:

[0888] The server receives the emotion data generated by the emotion engine. Based on this, it adjusts the initial cosmetics and makeup suggestions and re-suggests the most optimal makeup method. For example, if the user is tired, it suggests makeup that has a refreshing effect.

[0889] Step 11:

[0890] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[0891] Step 12:

[0892] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions. It also records the user's emotions recognized by the emotion engine in a database and uses this information to improve future makeup suggestions.

[0893] In this way, the system suggests the optimal cosmetics and makeup techniques based on the user's individual preferences, facial condition, and even emotions, helping them easily recreate professional makeup looks. Furthermore, the suggestions are constantly adjusted based on the user's latest condition and emotions, providing a high level of satisfaction.

[0894] Example 2

[0895] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] Conventional makeup systems rely on the user's qualitative senses and feedback, making it difficult to provide optimal recommendations based on scientific evidence or an individual's emotional state. Furthermore, because they do not take into account the user's past usage history or emotions, they are unable to provide optimal makeup recommendations for individual needs.

[0897] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial state data, means for suggesting optimal cosmetics and makeup techniques based on the user information and the facial state data, means for providing makeup instructions based on the suggested cosmetics and makeup techniques, and means for analyzing emotions from the user's facial expressions and voice, saving the emotion data in a database, and dynamically adjusting the makeup suggestions. This makes it possible to suggest optimal makeup based on the user's past usage history and emotional state, and to provide a satisfying makeup experience that meets the needs of each individual user.

[0898] "User information" refers to personal allergy information entered by the user, history information on cosmetics used in the past, and the like.

[0899] "Database" refers to a storage device for storing and managing user information, facial state data, emotion data, and other related information.

[0900] "Facial condition data" refers to information about facial condition such as dark circles under the eyes, skin tone, degree of tanning, and blemishes, which is analyzed from a facial photograph.

[0901] "Cosmetics" refers to items used in makeup, such as foundation, eye shadow, and lipstick.

[0902] "Suggestion" refers to the process of presenting the user with the most suitable cosmetics and makeup methods based on the user information and facial condition data.

[0903] "Emotional data" refers to information about the emotional state of a user analyzed from facial expressions and voice.

[0904] "Makeup procedure instruction" refers to the process of guiding a user through specific makeup steps based on the proposed cosmetics and makeup method.

[0905] "User feedback" refers to information such as impressions and improvements provided by users after completing their makeup.

[0906] MODE FOR CARRYING OUT THE INVENTION

[0907] The present invention provides a makeup suggestion system that registers and manages user information, analyzes facial conditions, suggests optimal cosmetics and makeup methods, provides makeup instruction, and includes an emotion engine that recognizes the user's emotions. Hereinafter, an embodiment of the present invention will be described.

[0908] User information registration and management

[0909] First, the user installs the smartphone app and, when launching it for the first time, enters allergy information and information about cosmetics used in the past. At this time, the user enters allergy information such as "nut allergy" or "shellfish allergy" as well as a history of cosmetics use, such as "Brand A's item X was good" or "Brand B's item Y did not suit me." The smartphone app formats this information and sends it to the server.

[0910] The server stores the received user information in a database and manages it individually. Based on this stored information, it can respond to the user's allergies and suggest cosmetics based on their past usage history.

[0911] Analysis of facial condition

[0912] Next, every morning, users take a photo of their face using a smartphone app, which then sends the photo to the AI ​​Mirror for analysis. The AI ​​Mirror then analyzes the photo to determine whether they have dark circles, skin tone, sun exposure, or blemishes.

[0913] The device (smartphone app) sends the captured facial photo to the AI ​​Mirror, which then sends the analysis results to the server. The AI ​​Mirror's data includes specific information such as "there are dark circles under the eyes," "the skin tone is normal," and "there are some blemishes."

[0914] Proposal of optimal cosmetics and makeup methods

[0915] The server compares the analysis results received from the AI ​​mirror with the user's database information to generate the optimal cosmetic set and makeup method. This includes foundation, eye shadow, lipstick, etc. that do not contain nuts or shellfish ingredients. It also takes into account emotional data analyzed by the emotion engine and suggests makeup methods that correspond to the user's emotions. For example, if the user is "feeling stressed," it will suggest relaxing makeup.

[0916] Makeup instruction

[0917] Next, the device (AI mirror and smartphone app) will provide step-by-step instructions to the user based on the generated makeup method, displaying detailed instructions and guiding the user, such as, "First, apply this concealer under your eyes to hide dark circles."

[0918] The user follows the instructions to apply the makeup. After completing the makeup, the user provides feedback on the completed makeup, which is then stored in a database and reflected in future suggestions.

[0919] Optimization by Emotion Engine

[0920] The system also incorporates an emotion engine. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions it recommends based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if they are feeling down, it will suggest refreshing makeup that will cheer them up. In this way, dynamically adjusting the suggestions according to the user's emotional state increases user satisfaction.

[0921] Examples and prompts

[0922] As a concrete example, we will explain the process when a user uses the system for the first time. The user installs the smartphone app and enters their "nut allergy" and "shellfish allergy." They also enter their past cosmetic use history, such as "Item X from brand A was good, but item Y from brand B didn't suit me."

[0923] The next morning, the user takes a photo of their face and sends it to the AI ​​mirror. The AI ​​mirror analyzes the photo and determines whether they have dark circles under their eyes, normal skin tone, or slight blemishes, and sends the results to the server. If the emotion engine determines that the user is feeling stressed, the server will suggest relaxing makeup, including foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[0924] An example prompt is, "I have nut and shellfish allergies. In the past, item X from brand A worked well for me, but item Y from brand B did not. I would like you to analyze a photo of my face this morning and suggest the best cosmetics and makeup techniques. I am currently feeling stressed."

[0925] The above is an embodiment of the present invention. By using this system, users can easily recreate professional makeup looks every day, providing a highly satisfying makeup experience that meets the needs of each individual user.

[0926] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0927] Program processing flow

[0928] Step 1: Register user information

[0929] 1. Input: The user installs the smartphone app and enters information about allergies and cosmetics used in the past.

[0930] Examples: "Nut allergy", "Shellfish allergy", cosmetics used in the past: "Item X from brand A was good", "Item Y from brand B didn't suit me".

[0931] 2. How it works: The smartphone app properly formats the entered information and sends it to the server.

[0932] 3. Output: The server stores the received user information in a database, providing the basic data for future cosmetic recommendations.

[0933] Step 2: Analyze your facial condition

[0934] 1. Input: Every morning, the user takes a photo of their face using a smartphone app.

[0935] 2. How it works: The smartphone app sends the captured facial photo to the AI ​​mirror.

[0936] 3. How it works: The AI ​​mirror analyzes the received facial photo and generates facial condition data, such as whether the person has dark circles under their eyes, whether their skin tone is normal, or whether they have some blemishes.

[0937] 4. Output: The AI ​​mirror sends the analysis results to the server and stores them in the user's database.

[0938] Step 3: Recommendations for the best cosmetics and makeup techniques

[0939] 1. Input: The server performs analysis based on facial condition data and the user's database information (allergy information and cosmetic use history).

[0940] 2. Operation: The server combines facial condition and user information to generate optimal cosmetics and makeup methods, such as recommending foundation, eye shadow, and lipstick that are free of nuts and shellfish ingredients.

[0941] 3. How it works: The emotion engine analyzes emotional data from the user's facial expressions and voice and adjusts the suggestions.

[0942] 4. Output: The server sends the optimal cosmetic set and makeup method for the user to the device (smartphone app and AI mirror).

[0943] Step 4: Makeup Instructions

[0944] 1. Input: Proposal sent from the server (cosmetics set and makeup method).

[0945] 2. Operation: The device (smartphone app and AI mirror) displays the suggested cosmetics and makeup methods to the user, and provides detailed step-by-step instructions for use. For example, "First, apply this concealer under your eyes to hide dark circles."

[0946] 3. Output: The user follows the instructions to apply makeup.

[0947] Step 5: Collect user feedback and sentiment data

[0948] 1. Input: User feedback after makeup is completed.

[0949] 2. Action: The user enters feedback about the makeup through the smartphone app. For example, "This makeup was very good" or "This eyeshadow didn't suit me."

[0950] 3. Operation: The emotion engine analyzes emotion data from the user's facial expressions and voice.

[0951] 4. Output: The server stores the collected feedback and sentiment data in a database, which will be used to improve future suggestions.

[0952] Specific steps in the overall flow

[0953] These processing steps allow users to easily receive recommendations for appropriate cosmetics and makeup techniques each day, and follow the instructions to apply their makeup. The system is continuously optimized, as user feedback and emotional data are reflected in the next recommendations.

[0954] (Application example 2)

[0955] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0956] In today's world, it is important to suggest optimal cosmetics and makeup techniques based on each individual user's skin condition and emotional state. However, no previous technology existed that could perform real-time facial image analysis and emotional analysis of the user and then suggest and instruct specific makeup techniques based on the results. Furthermore, there was insufficient support for applying the suggested makeup techniques in physical stores, resulting in a lack of assistance for users in selecting the most suitable products. Furthermore, there was also a lack of a mechanism for accumulating user feedback in a database and reflecting it in future suggestions.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0958] In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial condition data, means for analyzing the user's emotions and further optimizing cosmetics and makeup methods based on the emotion data, and means for displaying the suggested cosmetics and makeup methods on a smart mirror and smart glasses in a physical store to guide the user. This makes it possible to suggest optimal cosmetics and makeup methods based on the user's skin condition and emotional state, and to provide specific makeup instructions in the physical store in real time.

[0959] "User information" refers to data including personal information about the user, allergy information, and a history of past cosmetic use.

[0960] "Database" refers to a collection of information for storing and managing user information, feedback information, etc.

[0961] "Facial condition data" refers to data regarding the condition of the user's face, such as skin tone, dark circles, and blemishes, analyzed based on a photograph of the user's face.

[0962] "Emotion data" is data relating to the emotional state of the user acquired by analyzing the user's facial expressions and voice.

[0963] "Cosmetics" refers to products used for makeup and skin care, including foundation, eye shadow, lipstick, etc.

[0964] "Makeup method" refers to specific makeup procedures and techniques suggested to the user.

[0965] "Suggestion" refers to a recommendation of optimal cosmetics and makeup techniques generated based on user information, facial condition data, and emotion data.

[0966] "Step-by-step instruction" refers to the process of teaching a user how to apply makeup step by step.

[0967] "Brick and mortar store" refers to a retail store that has a physical presence and where consumers can visit and purchase products.

[0968] A "smart mirror" is a highly functional mirror equipped with a camera and display that analyzes facial condition and provides makeup guidance.

[0969] "Smart glasses" are glasses-type devices that are worn by a user and have the function of displaying information for makeup guidance.

[0970] "Feedback" refers to information about the evaluations and impressions provided by users regarding the proposed cosmetics and makeup methods.

[0971] "Server" refers to a centralized device that processes and stores user information, facial condition data, and emotion data over a network.

[0972] The following describes an embodiment of the present invention. This system registers individual user information, and proposes optimal cosmetics and makeup methods based on the analysis of the user's facial condition and emotions, providing makeup guidance in real time.

[0973] User information registration and management

[0974] Users enter their personal information (such as allergy information and past history of cosmetic use) into a smartphone application. The application then sends the collected information to a server to store in a database. The server then records and manages the user information in the database. This database serves as the basis for making optimal recommendations based on the user's past usage and allergy information.

[0975] Facial condition and emotion analysis

[0976] The user stands in front of a smart mirror installed in a physical store and takes a photo of their face. The smart mirror uses its built-in camera to capture an image of their face and analyzes it. An AI model running on TensorFlow and Keras is used for the analysis to obtain facial condition data such as the user's skin tone, dark circles, and the presence or absence of blemishes. At the same time, emotional data is also analyzed from the user's facial expressions and voice. An emotion recognition engine is used for this emotional analysis.

[0977] Generate optimized proposals

[0978] The server compares the user's facial condition data and emotion data with a user information database to generate the optimal cosmetics and makeup method. The server generates a list of cosmetics and makeup procedures that are optimal for the presented conditions and sends them to the smart mirror and smart glasses.

[0979] Providing real-time makeup guides

[0980] When a user wears smart glasses and applies makeup in front of a smart mirror, the glasses display suggested cosmetics and makeup steps. Audio instructions are also provided. This real-time guide is implemented using Python libraries (e.g., pyttsx3).

[0981] Collecting and incorporating user feedback

[0982] After the user has completed their makeup application, they can submit feedback about the proposed cosmetics and makeup techniques through the application. This feedback information is sent to the server and recorded in a database. This feedback is then reflected in future suggestions, enabling more accurate suggestions to be made.

[0983] Examples of concrete examples and prompts

[0984] For example, a user puts on smart glasses and stands in front of a smart mirror in a brick-and-mortar store. The smart mirror takes a photo of the user's face and analyzes their skin tone, presence of dark circles, and their emotions. Based on the results of this analysis and the user's allergy information, the system suggests the most suitable cosmetics (e.g., foundation from brand X, lipstick from brand Y). Below is an example of a prompt to be input into the generative AI model.

[0985] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[0986] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0987] Step 1:

[0988] Users enter personal information into a smartphone app, including allergy information and a history of past cosmetic use. The app formats this information and sends it to a server. The input data, which includes allergy information and past cosmetic use history, is sent to the server and stored in a database.

[0989] Step 2:

[0990] A user stands in front of a smart mirror installed in a brick-and-mortar store. The smart mirror uses a built-in camera to capture an image of the user's face. This image data is input and sent to an AI model for analysis. The output is facial condition data, including information on skin tone, dark circles, blemishes, etc.

[0991] Step 3:

[0992] The server analyzes the captured facial image using an AI model (using TensorFlow and Keras) to generate condition data. This analysis is achieved by evaluating each facial parameter through the AI ​​model. The input is the facial image, and the output is facial condition data.

[0993] Step 4:

[0994] The smart mirror also uses the user's voice and facial expressions to analyze emotional data in an emotion engine. This data is analyzed by an emotion recognition engine. The input is the user's voice and facial expressions, and the output is emotional data.

[0995] Step 5:

[0996] The server combines the user information database, facial condition data, and emotional data to suggest optimal cosmetics and makeup methods based on a generative AI model. During this process, each piece of data is collated, and the generative AI model calculates the optimal combination. The input is user information, facial condition data, and emotional data, and the output is a list of suggested cosmetics and makeup methods.

[0997] Step 6:

[0998] The server sends the suggested cosmetics and makeup techniques to the smart mirror and smart glasses. The smart glasses then display step-by-step instructions on how to use the cosmetics and how to apply makeup to the user. They also provide audio guidance to support the user. The input is the suggested information, and the output is guidance information for the user.

[0999] Step 7:

[1000] After the user completes their makeup, they submit feedback through the application. This feedback is stored on the server and reflected in future suggestions. The input is the user's feedback, and the output is the improved accuracy of the suggestions.

[1001] As a concrete example, the following prompt sentences can be input to a generative AI model:

[1002] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[1003] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1004] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1005] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1006] [Fourth embodiment]

[1007] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1008] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1009] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1010] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1011] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1012] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1013] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1014] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1015] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1016] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1017] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1018] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1019] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1020] An embodiment for implementing the system of the present invention will be described.

[1021] The system has an interface that allows users to register and manage their information, analyze their facial condition, suggest optimal makeup and cosmetics, and provide instructions on how to apply makeup.

[1022] First, the user installs the smartphone app and enters information about allergies and cosmetics they have used in the past (favorite cosmetics and cosmetics that did not suit them). The smartphone app formats this information and sends it to the server to be stored in a database.

[1023] The server stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[1024] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is then sent to the AI ​​Mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​Mirror, which analyzes information such as dark circles, skin tone, sun exposure, and blemishes.

[1025] The server receives the analysis results from the AI ​​mirror, combines them with information stored in the user's database, and runs an algorithm to generate the optimal cosmetics and makeup routine. Using the AI ​​model, it suggests a set of cosmetics and makeup routines optimized for the user's facial condition.

[1026] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[1027] Users can also provide feedback on the suggested cosmetics and makeup techniques through the smartphone app, which the server then stores in a database to improve the accuracy of future suggestions.

[1028] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest the optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition.

[1029] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[1030] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics.

[1031] The processing flow will be explained below.

[1032] Step 1:

[1033] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[1034] Step 2:

[1035] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[1036] Step 3:

[1037] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[1038] Step 4:

[1039] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[1040] Step 5:

[1041] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[1042] Step 6:

[1043] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[1044] Step 7:

[1045] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[1046] Step 8:

[1047] The user begins applying makeup based on the suggestions, and the device (AI mirror) guides the user by displaying specific steps for applying makeup using the suggested cosmetics in real time, step by step.

[1048] Step 9:

[1049] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[1050] Step 10:

[1051] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions.

[1052] In this way, the system suggests the optimal cosmetics and makeup methods based on the user's individual preferences and facial condition, helping the user easily recreate professional makeup looks.

[1053] Example 1

[1054] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1055] In today's society, recreating professional makeup looks at any time in a busy daily life is a difficult challenge for many users. Choosing the optimal cosmetics based on individual allergies and skin conditions can be complicated, and incorrect choices can cause skin problems. Furthermore, inappropriate makeup instruction can make it difficult to achieve the desired results. To solve these problems, a system is needed that is easy for users to use and provides optimal cosmetics and makeup methods tailored to individual needs.

[1056] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1057] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques using a generative AI model based on the user information and facial condition data, means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques, and means for collecting user feedback on the suggested cosmetics and makeup techniques and recording the feedback in the database. This allows users to easily recreate professional makeup looks even in their busy daily lives and select cosmetics optimized for their individual needs. Furthermore, the system can be continuously improved based on user feedback, resulting in more accurate suggestions.

[1058] "User information" refers to individual profile data that a user registers in the system, including allergy information and information about cosmetics used in the past.

[1059] A "database" is a system installed inside a server for systematically storing and managing user information and feedback information.

[1060] "Facial condition data" refers to information such as dark circles, skin tone, tan, and blemishes analyzed by the AI ​​mirror.

[1061] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal cosmetics and makeup methods based on user information and facial condition data.

[1062] A "makeup method" refers to a set of instructions including steps for using and applying cosmetic products that are optimized for the user's facial condition and individual needs.

[1063] "Step-by-step instruction" refers to providing users with sequential instructions for each step of makeup application through the AI ​​mirror and smartphone app.

[1064] "Feedback" refers to comments and evaluation information provided by users regarding the proposed cosmetics and makeup methods.

[1065] An "AI-equipped display device" refers to a display device that has built-in artificial intelligence to display makeup instruction information to users in real time.

[1066] The present invention relates to a system that registers and manages user information, analyzes facial condition, suggests optimal cosmetics and makeup techniques, and provides step-by-step makeup instruction. The system is composed of the following elements: a user, a terminal, and a server.

[1067] First, the user installs a dedicated app on a device such as a smartphone or tablet. Using this app, the user enters information about their allergies and the cosmetics they have used in the past. Specifically, this information may include nut or shellfish allergies, or whether item X from brand A was good but item Y from brand B was not. This information is then formatted and sent to the server via a secure protocol (e.g., HTTPS).

[1068] The server stores the received user information in a database. The database is in SQL or NoSQL format and is managed as profile data for each user. This makes it possible to recommend optimal cosmetics based on the user's allergy information and past usage history.

[1069] Every morning, users take a photo of their face using a smartphone app. The device (smartphone app) then sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes to generate facial condition data. The analysis results are then sent back to the server.

[1070] The server combines the received facial condition data with user information stored in a database and uses a generative AI model (such as Azure AI, Google AI, or IBM Watson) to suggest optimal cosmetics and makeup applications, such as foundation, eyeshadow, and lipstick that are free of nuts and shellfish.

[1071] The device (AI mirror and smartphone app) provides detailed instructions based on the proposed makeup method. Specifically, the AI ​​mirror provides step-by-step makeup instructions to the user in real time, such as "Next, apply eyeshadow" or "Apply foundation evenly." The user can then follow these instructions to apply makeup.

[1072] Users can also provide feedback on the proposed cosmetics and makeup techniques through a smartphone app. This feedback is sent to the server and stored in a database. This allows the server to continuously incorporate user feedback and improve the accuracy of future suggestions.

[1073] As a specific example, a user may register allergies such as "nut allergy" and "shellfish allergy" and enter information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," the system will use this information to suggest the optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients. Furthermore, the AI ​​mirror will provide real-time instructions on how to use the suggested cosmetics, and the user will follow those instructions to apply their makeup.

[1074] An example prompt is:

[1075] "To generate makeup suggestions for the AI ​​Specialist, please use the following information to suggest the most suitable cosmetics for the user from a database. User information: Nut allergy, shellfish allergy. Past cosmetics usage history: Brand A's item X worked well, but Brand B's item Y did not. Face photo analysis results: Dark circles under the eyes, normal skin tone, slight blemishes."

[1076] The above is a specific embodiment for carrying out the present invention. This system allows users to easily recreate professional makeup looks every day and reduces the waste of unnecessary cosmetics.

[1077] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1078] Step 1:

[1079] Users install the smartphone app and launch it. They then enter information about allergies and cosmetics they have used in the past. For example, they can enter information such as "nut allergy," "shellfish allergy," "item X from brand A was good," or "item Y from brand B didn't suit me."

[1080] Input: User's allergy information and cosmetic use history

[1081] Output: Formatted user information

[1082] Step 2:

[1083] The device (smartphone app) formats the information entered by the user. In this step, the data format is organized and converted into a format that is easy for the server to receive and analyze. The formatted information is then sent to the server using a secure protocol (e.g., HTTPS).

[1084] Input: Raw information entered by the user

[1085] Output: User information formatted for easy reception by the server

[1086] Step 3:

[1087] The server stores the received user information in a database. The database is managed in SQL or NoSQL format and is saved as profile data for each user. In this process, the user information is securely registered in the database.

[1088] Input: formatted user information

[1089] Output: User profile data in the database

[1090] Step 4:

[1091] Every morning, users take a photo of their face using a smartphone app and their smartphone's camera.

[1092] Input: User's face photo

[1093] Output: Photographed face photo data

[1094] Step 5:

[1095] The device (smartphone app) takes a photo of your face and sends it to the AI ​​mirror. When sending, the photo must be of the correct resolution and format.

[1096] Input: Photographed face photo data

[1097] Output: Facial photo data sent to AI Mirror

[1098] Step 6:

[1099] The device (AI Mirror) analyzes the facial condition based on the received facial photo, including dark circles, skin tone, tan, and blemishes, and generates an analysis result.

[1100] Input: Facial photo data sent to AI Mirror

[1101] Output: Facial condition data

[1102] Step 7:

[1103] The server receives the analysis results sent by the AI ​​mirror and combines them with the user's information in the database. Using a generative AI model, it generates the optimal cosmetics and makeup routine based on the user's facial condition. For example, it suggests foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[1104] Input: Facial condition data and user information from the database

[1105] Output: Recommendations for optimal cosmetics and makeup application

[1106] Step 8:

[1107] The devices (AI mirror and smartphone app) provide detailed instructions based on the generated makeup method. The AI ​​mirror provides step-by-step makeup instructions to the user in real time, providing guidance such as "Next, apply eyeshadow" and "Apply foundation evenly."

[1108] Input: Recommendations for optimal cosmetics and makeup techniques

[1109] Output: Step-by-step makeup instruction information

[1110] Step 9:

[1111] Users provide feedback on the proposed cosmetics and makeup methods through a smartphone app, which is then sent to a server and stored in a database.

[1112] Input: User feedback

[1113] Output: Feedback information stored in a database

[1114] (Application example 1)

[1115] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1116] While conventional systems can suggest the best cosmetics and makeup methods for a user's specific facial condition, they face the challenge of making it difficult to check and try on the suggestions in real time. Another issue is that the methods for providing detailed instructions on the steps of the suggested makeup methods are limited, making it difficult for users to accurately replicate them. Furthermore, there are also issues with systems that are inadequate in collecting user feedback and incorporating that information into future suggestions.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1118] In this invention, the server includes means for registering user information and storing the user information in a database, means for capturing images of the user's face and analyzing the images to generate facial condition data, means for suggesting optimal cosmetics and makeup techniques based on the user information and facial condition data, means for allowing the user to virtually try on the suggested cosmetics in real time, and means for providing step-by-step makeup instruction based on the suggested cosmetics and makeup techniques. This allows the user to check and try on the optimal cosmetics suggestions in real time, and accurately understand and execute the makeup steps. Feedback can also be collected to improve the accuracy of the suggestions.

[1119] "User information" refers to information including allergy information entered by the user, information about cosmetics used in the past, individual preferences, skin characteristics, and the like.

[1120] "Database" refers to an information system for storing and managing user information and facial condition data.

[1121] "Facial condition data" is data generated by analyzing an image of a face, and includes information such as dark circles, skin tone, blemishes, and degree of tanning.

[1122] "Cosmetics" are products such as foundation, eye shadow, lipstick, etc. that are used to enhance the aesthetic appearance of a user's face.

[1123] "Makeup method" refers to the steps and techniques for applying makeup to the face using cosmetics.

[1124] "Virtual try-on" is a technology that simulates applying suggested cosmetics to the user's face in real time, allowing them to see how they will look without actually applying them.

[1125] "Feedback" is information in which a user transmits opinions and evaluations about cosmetics and makeup methods used.

[1126] "Step-by-step instruction" is a process that provides step-by-step instructions on how to use cosmetics and how to apply makeup, helping users to apply makeup correctly.

[1127] The following describes an embodiment of the present invention. The system of the present invention uses a smartphone app, a server, and AI analysis technology to propose optimal cosmetics and makeup methods based on the user's facial condition, and is equipped with virtual try-on and step-by-step instruction functions.

[1128] First, the user installs the smartphone app and enters information about their allergies and the cosmetics they have used in the past. This information is formatted and stored in a cloud database (e.g., Firebase). The server then manages the received user information individually. Based on the information stored in this database, the app can suggest the most suitable cosmetics based on the user's allergies and past usage history.

[1129] Next, every morning, the user takes a photo of their face using a smartphone app. This photo is sent to an AI analysis engine (e.g., TensorFlow Lite, OpenCV) and analyzed for information such as dark circles, skin tone, sun exposure, and blemishes. The smartphone app then sends the photo to the AI ​​analysis engine and receives the results.

[1130] The server receives the analysis results from the AI ​​analysis engine, combines them with information stored in the user's database, and runs an algorithm to generate optimal cosmetics and makeup techniques, which are then presented to the user in real time.

[1131] Furthermore, the proposed cosmetics are applied to the user's face in real time using virtual try-on features (e.g., OpenCV, camera API), allowing users to see how the products will look without actually applying them.

[1132] Based on the proposed makeup application, step-by-step instructions are provided. The smartphone app uses real-time video and audio to provide detailed instructions that users can follow. Based on the suggestions, users can also provide feedback, which is recorded in a database to improve the accuracy of future suggestions.

[1133] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the user's face is taken and the AI ​​analysis engine analyzes the results as "dark circles," "normal skin tone," and "slight blemishes," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nut or shellfish ingredients based on the user's data and facial condition. The system also allows users to virtually try on the suggested cosmetics, and if they are satisfied, they can purchase them immediately.

[1134] Prompt Sentence Examples

[1135] markdown

[1136] Prompt statement

[1137] User information: Nut allergy, cosmetics used in the past: Brand A, item X (good), Brand B, item Y (unsuitable)

[1138] Take a photo of your face and output the analysis results: dark circles, normal skin tone, slight blemishes

[1139] Based on the analysis results generated, the app will suggest the best cosmetics and makeup techniques.

[1140] Simply apply the suggested items to your face virtually and provide a picture.

[1141] Provide step-by-step instructions for users to proceed with their makeup.

[1142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1143] Program processing steps

[1144] Step 1:

[1145] Users enter information about their allergies and cosmetics they have used in the past into a smartphone app.

[1146] Input: Allergy information (e.g., nut allergy, shellfish allergy), past cosmetic use information (e.g., item X from brand A is fine, but item Y from brand B is incompatible)

[1147] Data processing: The input information is formatted into JSON format and sent to the server.

[1148] Output: Formatted user information (JSON)

[1149] Step 2:

[1150] The server stores the received user information in a cloud database (e.g., Firebase).

[1151] Input: User information (JSON)

[1152] Data processing: Save as an entry in a database and associate with an individual user ID.

[1153] Output: Database entries

[1154] Step 3:

[1155] A user takes a photo of their face every morning using a smartphone app.

[1156] Input: User's face photo

[1157] Data processing: Save the image in the correct format and send it to the AI ​​analysis engine.

[1158] Output: Sent face photo (image)

[1159] Step 4:

[1160] An AI analysis engine (e.g., TensorFlow Lite, OpenCV) analyzes the facial photo and generates facial condition data.

[1161] Input: Face photo (image)

[1162] Data processing: Analyzes facial dark circles, skin tone, blemishes, etc. and converts the results into JSON format.

[1163] Output: Facial condition data (JSON)

[1164] Step 5:

[1165] The server runs an algorithm that generates optimal cosmetics and makeup methods based on facial condition data and user information.

[1166] Input: Facial condition data (JSON), user information

[1167] Data processing: Analyze data using an AI model and suggest the most suitable cosmetics and makeup techniques.

[1168] Output: Suggested cosmetics and makeup methods (list and step-by-step instructions)

[1169] Step 6:

[1170] The smartphone app applies the suggested cosmetics to the user's face in real time using a virtual try-on function (e.g., OpenCV, camera API).

[1171] Input: Suggested cosmetics (list), user's face photo (image)

[1172] Data processing: Using a virtual try-on algorithm, images of cosmetics applied to the user's face are generated in real time.

[1173] Output: Virtual try-on image

[1174] Step 7:

[1175] The smartphone app provides real-time step-by-step instructions based on the proposed makeup application method, providing detailed instructions using video and audio.

[1176] Input: Suggested makeup method (step-by-step instructions)

[1177] Data processing: Generate step-by-step video and audio guides and display and play them to the user in real time.

[1178] Output: Video and audio guide

[1179] Step 8:

[1180] Users provide feedback on the results of their makeup through a smartphone app, and the server records it in a database.

[1181] Input: User feedback (text or rating)

[1182] Data processing: Feedback information is stored in a database so that it can be reflected in future proposals.

[1183] Output: Feedback stored in a database

[1184] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1185] An embodiment for implementing the system of the present invention will be described.

[1186] The system is equipped with an interface that registers and manages user information, analyzes facial condition, suggests optimal makeup and cosmetics, provides instructions on how to apply makeup, and an emotion engine that recognizes the user's emotions.

[1187] First, the user installs the smartphone app and, when they launch it for the first time, enters information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like). The smartphone app formats this information and sends it to the server. The server then stores the received user information in a database and manages it individually. The information stored in this database makes it possible to address the user's allergies and suggest the most suitable cosmetics based on their past usage history.

[1188] Next, each morning, the user takes a photo of their face using a smartphone app. The photo is sent to the AI ​​mirror, where it analyzes their facial condition. The device (smartphone app) sends the photo to the AI ​​mirror, which analyzes information such as dark circles, skin tone, tanning, and blemishes. The server receives the analysis results from the AI ​​mirror and combines them with information stored in the user's database to generate recommendations for the optimal cosmetic set and makeup method for the user.

[1189] Next, step-by-step instructions are given for applying makeup based on the suggested cosmetics and makeup techniques. The device (AI mirror and smartphone app) provides detailed instructions on how to use the suggested cosmetics, guiding the user step by step. The user can then follow the instructions to apply the makeup.

[1190] This system also incorporates an emotion engine that recognizes the user's emotions. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if the user is feeling down, it will suggest refreshing makeup that will cheer them up. In this way, the system dynamically adjusts the suggestions according to the user's emotional state, thereby improving user satisfaction.

[1191] As a specific example, let's say a user registers "nut allergy" and "shellfish allergy" as allergy information, and enters information about cosmetics they have used in the past, such as "Item X from brand A was good, but item Y from brand B didn't suit me." If a photo of the face is taken and the AI ​​mirror analyzes it as "dark circles," "normal skin tone," and "slight blemishes," and the emotion engine further recognizes that the user is "feeling stressed," the system will suggest optimal foundation, eye shadow, and lipstick that do not contain nuts or shellfish ingredients, based on the user's data, facial condition, and emotion data, as well as relaxing makeup to relieve stress.

[1192] The AI ​​mirror then provides real-time instructions on how to apply the suggested cosmetics, and the user follows those instructions to apply the makeup. When the user provides feedback on the finished makeup, that information is saved in a database and reflected in future suggestions.

[1193] The user's emotions recognized by the emotion engine are also recorded in a database and used to suggest makeup looks for the next time onward, enabling optimal suggestions to be made in response to fluctuations in the user's emotional state.

[1194] As described above, by using this system, users can easily recreate professional makeup looks every day and reduce the waste of unnecessary cosmetics. Furthermore, by making suggestions based on the user's emotions, further improvements in satisfaction can be expected.

[1195] The processing flow will be explained below.

[1196] Step 1:

[1197] Users install the smartphone app and, when they launch it for the first time, enter information about their allergies and cosmetics they have used in the past (those they liked and those they didn't like).

[1198] Step 2:

[1199] The device (smartphone app) organizes the information entered by the user and sends it to a server, including allergy information, usage history, and other personal data.

[1200] Step 3:

[1201] The server stores the received user information in a database and manages it for each user, making it possible to suggest cosmetics based on the user's allergies and select the most suitable cosmetics based on past usage history.

[1202] Step 4:

[1203] Every morning, users open the smartphone app and take a photo of their face, which is then sent to the AI ​​mirror to analyze their facial condition for the day.

[1204] Step 5:

[1205] The device (smartphone app) takes a photo of your face and sends it to the AI ​​Mirror, which analyzes the image and generates facial condition data such as dark circles, skin tone, tan level, and blemishes.

[1206] Step 6:

[1207] The server receives facial condition data from the AI ​​mirror, combines it with user information, and analyzes it to generate recommendations for the optimal cosmetic set and makeup method for each user.

[1208] Step 7:

[1209] The server sends the generated cosmetic set and makeup method suggestions to the smartphone app, which then displays them to the user.

[1210] Step 8:

[1211] The user begins applying makeup based on the suggested cosmetics and makeup methods. The device (AI mirror) guides the user by displaying specific makeup steps using the suggested cosmetics in real time, step by step.

[1212] Step 9:

[1213] While the user is applying makeup, the emotion engine analyzes the user's facial expressions and voice to generate emotion data, for example, determining whether the user is smiling or feeling stressed through a camera or microphone.

[1214] Step 10:

[1215] The server receives the emotion data generated by the emotion engine. Based on this, it adjusts the initial cosmetics and makeup suggestions and re-suggests the most optimal makeup method. For example, if the user is tired, it suggests makeup that has a refreshing effect.

[1216] Step 11:

[1217] After completing the makeup application, the user provides feedback on the proposed cosmetics and makeup method via a smartphone app.

[1218] Step 12:

[1219] The server records the received feedback in a database and uses this information to improve the accuracy of future suggestions. It also records the user's emotions recognized by the emotion engine in a database and uses this information to improve future makeup suggestions.

[1220] In this way, the system suggests the optimal cosmetics and makeup techniques based on the user's individual preferences, facial condition, and even emotions, helping them easily recreate professional makeup looks. Furthermore, the suggestions are constantly adjusted based on the user's latest condition and emotions, providing a high level of satisfaction.

[1221] Example 2

[1222] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] Conventional makeup systems rely on the user's qualitative senses and feedback, making it difficult to provide optimal recommendations based on scientific evidence or an individual's emotional state. Furthermore, because they do not take into account the user's past usage history or emotions, they are unable to provide optimal makeup recommendations for individual needs.

[1224] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial state data, means for suggesting optimal cosmetics and makeup techniques based on the user information and the facial state data, means for providing makeup instructions based on the suggested cosmetics and makeup techniques, and means for analyzing emotions from the user's facial expressions and voice, saving the emotion data in a database, and dynamically adjusting the makeup suggestions. This makes it possible to suggest optimal makeup based on the user's past usage history and emotional state, and to provide a satisfying makeup experience that meets the needs of each individual user.

[1225] "User information" refers to personal allergy information entered by the user, history information on cosmetics used in the past, and the like.

[1226] "Database" refers to a storage device for storing and managing user information, facial state data, emotion data, and other related information.

[1227] "Facial condition data" refers to information about facial condition such as dark circles under the eyes, skin tone, degree of tanning, and blemishes, which is analyzed from a facial photograph.

[1228] "Cosmetics" refers to items used in makeup, such as foundation, eye shadow, and lipstick.

[1229] "Suggestion" refers to the process of presenting the user with the most suitable cosmetics and makeup methods based on the user information and facial condition data.

[1230] "Emotional data" refers to information about the emotional state of a user analyzed from facial expressions and voice.

[1231] "Makeup procedure instruction" refers to the process of guiding a user through specific makeup steps based on the proposed cosmetics and makeup method.

[1232] "User feedback" refers to information such as impressions and improvements provided by users after completing their makeup.

[1233] MODE FOR CARRYING OUT THE INVENTION

[1234] The present invention provides a makeup suggestion system that registers and manages user information, analyzes facial conditions, suggests optimal cosmetics and makeup methods, provides makeup instruction, and includes an emotion engine that recognizes the user's emotions. Hereinafter, an embodiment of the present invention will be described.

[1235] User information registration and management

[1236] First, the user installs the smartphone app and, when launching it for the first time, enters allergy information and information about cosmetics used in the past. At this time, the user enters allergy information such as "nut allergy" or "shellfish allergy" as well as a history of cosmetics use, such as "Brand A's item X was good" or "Brand B's item Y did not suit me." The smartphone app formats this information and sends it to the server.

[1237] The server stores the received user information in a database and manages it individually. Based on this stored information, it can respond to the user's allergies and suggest cosmetics based on their past usage history.

[1238] Analysis of facial condition

[1239] Next, every morning, users take a photo of their face using a smartphone app, which then sends the photo to the AI ​​Mirror for analysis. The AI ​​Mirror then analyzes the photo to determine whether they have dark circles, skin tone, sun exposure, or blemishes.

[1240] The device (smartphone app) sends the captured facial photo to the AI ​​Mirror, which then sends the analysis results to the server. The AI ​​Mirror's data includes specific information such as "there are dark circles under the eyes," "the skin tone is normal," and "there are some blemishes."

[1241] Proposal of optimal cosmetics and makeup methods

[1242] The server compares the analysis results received from the AI ​​mirror with the user's database information to generate the optimal cosmetic set and makeup method. This includes foundation, eye shadow, lipstick, etc. that do not contain nuts or shellfish ingredients. It also takes into account emotional data analyzed by the emotion engine and suggests makeup methods that correspond to the user's emotions. For example, if the user is "feeling stressed," it will suggest relaxing makeup.

[1243] Makeup instruction

[1244] Next, the device (AI mirror and smartphone app) will provide step-by-step instructions to the user based on the generated makeup method, displaying detailed instructions and guiding the user, such as, "First, apply this concealer under your eyes to hide dark circles."

[1245] The user follows the instructions to apply the makeup. After completing the makeup, the user provides feedback on the completed makeup, which is then stored in a database and reflected in future suggestions.

[1246] Optimization by Emotion Engine

[1247] The system also incorporates an emotion engine. It analyzes emotions from the user's facial expressions and voice and sends this data to the server. The server then further optimizes the cosmetics and makeup suggestions it recommends based on the emotional data recognized by the emotion engine. For example, if the user is tired, it will suggest makeup that will make them look brighter, and if they are feeling down, it will suggest refreshing makeup that will cheer them up. In this way, dynamically adjusting the suggestions according to the user's emotional state increases user satisfaction.

[1248] Examples and prompts

[1249] As a concrete example, we will explain the process when a user uses the system for the first time. The user installs the smartphone app and enters their "nut allergy" and "shellfish allergy." They also enter their past cosmetic use history, such as "Item X from brand A was good, but item Y from brand B didn't suit me."

[1250] The next morning, the user takes a photo of their face and sends it to the AI ​​mirror. The AI ​​mirror analyzes the photo and determines whether they have dark circles under their eyes, normal skin tone, or slight blemishes, and sends the results to the server. If the emotion engine determines that the user is feeling stressed, the server will suggest relaxing makeup, including foundation, eyeshadow, and lipstick that are free of nuts and shellfish ingredients.

[1251] An example prompt is, "I have nut and shellfish allergies. In the past, item X from brand A worked well for me, but item Y from brand B did not. I would like you to analyze a photo of my face this morning and suggest the best cosmetics and makeup techniques. I am currently feeling stressed."

[1252] The above is an embodiment of the present invention. By using this system, users can easily recreate professional makeup looks every day, providing a highly satisfying makeup experience that meets the needs of each individual user.

[1253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1254] Program processing flow

[1255] Step 1: Register user information

[1256] 1. Input: The user installs the smartphone app and enters information about allergies and cosmetics used in the past.

[1257] Examples: "Nut allergy", "Shellfish allergy", cosmetics used in the past: "Item X from brand A was good", "Item Y from brand B didn't suit me".

[1258] 2. How it works: The smartphone app properly formats the entered information and sends it to the server.

[1259] 3. Output: The server stores the received user information in a database, providing the basic data for future cosmetic recommendations.

[1260] Step 2: Analyze your facial condition

[1261] 1. Input: Every morning, the user takes a photo of their face using a smartphone app.

[1262] 2. How it works: The smartphone app sends the captured facial photo to the AI ​​mirror.

[1263] 3. How it works: The AI ​​mirror analyzes the received facial photo and generates facial condition data, such as whether the person has dark circles under their eyes, whether their skin tone is normal, or whether they have some blemishes.

[1264] 4. Output: The AI ​​mirror sends the analysis results to the server and stores them in the user's database.

[1265] Step 3: Recommendations for the best cosmetics and makeup techniques

[1266] 1. Input: The server performs analysis based on facial condition data and the user's database information (allergy information and cosmetic use history).

[1267] 2. Operation: The server combines facial condition and user information to generate optimal cosmetics and makeup methods, such as recommending foundation, eye shadow, and lipstick that are free of nuts and shellfish ingredients.

[1268] 3. How it works: The emotion engine analyzes emotional data from the user's facial expressions and voice and adjusts the suggestions.

[1269] 4. Output: The server sends the optimal cosmetic set and makeup method for the user to the device (smartphone app and AI mirror).

[1270] Step 4: Makeup Instructions

[1271] 1. Input: Proposal content sent from the server (cosmetics set and makeup method).

[1272] 2. Operation: The device (smartphone app and AI mirror) displays the suggested cosmetics and makeup methods to the user, and provides detailed step-by-step instructions for use. For example, "First, apply this concealer under your eyes to hide dark circles."

[1273] 3. Output: The user follows the instructions to apply makeup.

[1274] Step 5: Collect user feedback and sentiment data

[1275] 1. Input: User feedback after makeup is completed.

[1276] 2. Action: The user enters feedback about the makeup through the smartphone app. For example, "This makeup was very good" or "This eyeshadow didn't suit me."

[1277] 3. Operation: The emotion engine analyzes emotion data from the user's facial expressions and voice.

[1278] 4. Output: The server stores the collected feedback and sentiment data in a database, which will be used to improve future suggestions.

[1279] Specific steps in the overall flow

[1280] These processing steps allow users to easily receive recommendations for appropriate cosmetics and makeup techniques each day, and follow the instructions to apply their makeup. The system is continuously optimized, as user feedback and emotional data are reflected in the next recommendations.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] In today's world, it is important to suggest optimal cosmetics and makeup techniques based on each individual user's skin condition and emotional state. However, no previous technology existed that could perform real-time facial image analysis and emotional analysis of the user and then suggest and instruct specific makeup techniques based on the results. Furthermore, there was insufficient support for applying the suggested makeup techniques in physical stores, resulting in a lack of assistance for users in selecting the most suitable products. Furthermore, there was also a lack of a mechanism for accumulating user feedback in a database and reflecting it in future suggestions.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1285] In this invention, the server includes means for registering user information and saving the user information in a database, means for taking an image of the user's face and analyzing the image to generate facial condition data, means for analyzing the user's emotions and further optimizing cosmetics and makeup methods based on the emotion data, and means for displaying the suggested cosmetics and makeup methods on a smart mirror and smart glasses in a physical store to guide the user. This makes it possible to suggest optimal cosmetics and makeup methods based on the user's skin condition and emotional state, and to provide specific makeup instructions in the physical store in real time.

[1286] "User information" refers to data including personal information about the user, allergy information, and a history of past cosmetic use.

[1287] "Database" refers to a collection of information for storing and managing user information, feedback information, etc.

[1288] "Facial condition data" refers to data regarding the condition of the user's face, such as skin tone, dark circles, and blemishes, analyzed based on a photograph of the user's face.

[1289] "Emotion data" is data relating to the emotional state of the user acquired by analyzing the user's facial expressions and voice.

[1290] "Cosmetics" refers to products used for makeup and skin care, including foundation, eye shadow, lipstick, etc.

[1291] "Makeup method" refers to specific makeup procedures and techniques suggested to the user.

[1292] "Suggestion" refers to a recommendation of optimal cosmetics and makeup techniques generated based on user information, facial condition data, and emotion data.

[1293] "Step-by-step instruction" refers to the process of teaching a user how to apply makeup step by step.

[1294] "Brick and mortar store" refers to a retail store that has a physical presence and where consumers can visit and purchase products.

[1295] A "smart mirror" is a highly functional mirror equipped with a camera and display that analyzes facial condition and provides makeup guidance.

[1296] "Smart glasses" are glasses-type devices that are worn by a user and have the function of displaying information for makeup guidance.

[1297] "Feedback" refers to information about the evaluations and impressions provided by users regarding the proposed cosmetics and makeup methods.

[1298] "Server" refers to a centralized device that processes and stores user information, facial condition data, and emotion data over a network.

[1299] The following describes an embodiment of the present invention. This system registers individual user information, and proposes optimal cosmetics and makeup methods based on the analysis of the user's facial condition and emotions, providing makeup guidance in real time.

[1300] User information registration and management

[1301] Users enter their personal information (such as allergy information and past history of cosmetic use) into a smartphone application. The application then sends the collected information to a server to store in a database. The server then records and manages the user information in the database. This database serves as the basis for making optimal recommendations based on the user's past usage and allergy information.

[1302] Facial condition and emotion analysis

[1303] The user stands in front of a smart mirror installed in a physical store and takes a photo of their face. The smart mirror uses its built-in camera to capture an image of their face and analyzes it. An AI model running on TensorFlow and Keras is used for the analysis to obtain facial condition data such as the user's skin tone, dark circles, and the presence or absence of blemishes. At the same time, emotional data is also analyzed from the user's facial expressions and voice. An emotion recognition engine is used for this emotional analysis.

[1304] Generate optimized proposals

[1305] The server compares the user's facial condition data and emotion data with a user information database to generate the optimal cosmetics and makeup method. The server generates a list of cosmetics and makeup procedures that are optimal for the presented conditions and sends them to the smart mirror and smart glasses.

[1306] Providing real-time makeup guides

[1307] When a user wears smart glasses and applies makeup in front of a smart mirror, the glasses display suggested cosmetics and makeup steps. Audio instructions are also provided. This real-time guide is implemented using Python libraries (e.g., pyttsx3).

[1308] Collecting and incorporating user feedback

[1309] After the user has completed their makeup application, they can submit feedback about the proposed cosmetics and makeup techniques through the application. This feedback information is sent to the server and recorded in a database. This feedback is then reflected in future suggestions, enabling more accurate suggestions to be made.

[1310] Examples of concrete examples and prompts

[1311] For example, a user puts on smart glasses and stands in front of a smart mirror in a brick-and-mortar store. The smart mirror takes a photo of the user's face and analyzes their skin tone, presence of dark circles, and their emotions. Based on the results of this analysis and the user's allergy information, the system suggests the most suitable cosmetics (e.g., foundation from brand X, lipstick from brand Y). Below is an example of a prompt to be input into the generative AI model.

[1312] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[1313] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1314] Step 1:

[1315] Users enter personal information into a smartphone app, including allergy information and a history of past cosmetic use. The app formats this information and sends it to a server. The input data, which includes allergy information and past cosmetic use history, is sent to the server and stored in a database.

[1316] Step 2:

[1317] A user stands in front of a smart mirror installed in a brick-and-mortar store. The smart mirror uses a built-in camera to capture an image of the user's face. This image data is input and sent to an AI model for analysis. The output is facial condition data, including information on skin tone, dark circles, blemishes, etc.

[1318] Step 3:

[1319] The server analyzes the captured facial image using an AI model (using TensorFlow and Keras) to generate condition data. This analysis is achieved by evaluating each facial parameter through the AI ​​model. The input is the facial image, and the output is facial condition data.

[1320] Step 4:

[1321] The smart mirror also uses the user's voice and facial expressions to analyze emotional data in an emotion engine. This data is analyzed by an emotion recognition engine. The input is the user's voice and facial expressions, and the output is emotional data.

[1322] Step 5:

[1323] The server combines the user information database, facial condition data, and emotional data to suggest optimal cosmetics and makeup methods based on a generative AI model. During this process, each piece of data is collated, and the generative AI model calculates the optimal combination. The input is user information, facial condition data, and emotional data, and the output is a list of suggested cosmetics and makeup methods.

[1324] Step 6:

[1325] The server sends the suggested cosmetics and makeup techniques to the smart mirror and smart glasses. The smart glasses then display step-by-step instructions on how to use the cosmetics and how to apply makeup to the user. They also provide audio guidance to support the user. The input is the suggested information, and the output is guidance information for the user.

[1326] Step 7:

[1327] After the user completes their makeup, they submit feedback through the application. This feedback is stored on the server and reflected in future suggestions. The input is the user's feedback, and the output is the improved accuracy of the suggestions.

[1328] As a concrete example, the following prompt sentences can be input to a generative AI model:

[1329] I have a nut allergy and a shellfish allergy. In the past, I used "Brand A's Item X" and it worked well, but "Brand B's Item Y" didn't work for me. I currently have dark circles under my eyes and my skin tone is normal. I have some blemishes and am feeling stressed. Please suggest the best cosmetics and relaxing makeup for me.

[1330] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1331] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1332] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1333] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1334] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1335] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1336] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1337] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1338] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1339] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1340] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1341] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1342] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1343] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1344] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1345] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1346] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1347] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1348] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1349] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1350] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1351] The following is further disclosed regarding the above embodiment.

[1352] (Claim 1)

[1353] means for registering user information and storing the user information in a database;

[1354] means for capturing an image of a user's face and analyzing the image to generate facial condition data;

[1355] means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data;

[1356] A means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method;

[1357] A system including:

[1358] (Claim 2)

[1359] The system according to claim 1, further comprising means for displaying makeup instruction information generated based on the proposed cosmetics and makeup method, and prompting the user to apply makeup in accordance with the instructions.

[1360] (Claim 3)

[1361] 10. The system of claim 1, further comprising means for collecting user feedback regarding suggested cosmetics and makeup methods and recording said feedback in said database.

[1362] "Example 1"

[1363] (Claim 1)

[1364] means for registering user information and storing the user information in a database;

[1365] means for capturing an image of a user's face and analyzing the image to generate facial condition data;

[1366] means for suggesting optimal cosmetics and makeup methods using a generative AI model based on the user information and the facial condition data;

[1367] A means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method;

[1368] means for collecting user feedback on the proposed cosmetics and makeup methods and recording said feedback in said database;

[1369] A system including:

[1370] (Claim 2)

[1371] 10. The system of claim 1, further comprising: means for using an AI-equipped display device to display makeup instruction information generated based on the suggested cosmetics and makeup method and to prompt the user to proceed with makeup application in accordance with the instructions.

[1372] (Claim 3)

[1373] 10. The system of claim 1, further comprising: updating information in the database based on the proposed cosmetics and makeup methods.

[1374] "Application Example 1"

[1375] (Claim 1)

[1376] means for registering user information and storing the user information in a database;

[1377] means for capturing an image of a user's face and analyzing the image to generate facial condition data;

[1378] means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data;

[1379] A means for virtual try-on of the proposed cosmetics in real time;

[1380] A means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method;

[1381] A system including:

[1382] (Claim 2)

[1383] The system according to claim 1, further comprising means for displaying makeup instruction information generated based on the proposed cosmetics and makeup method, and prompting the user to apply makeup in accordance with the instructions.

[1384] (Claim 3)

[1385] 10. The system of claim 1, further comprising means for collecting user feedback regarding suggested cosmetics and makeup methods and recording said feedback in said database.

[1386] "Example 2: Combining Emotion Engines"

[1387] (Claim 1)

[1388] means for registering user information and storing the user information in a database;

[1389] means for capturing an image of a user's face and analyzing the image to generate facial state data;

[1390] means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data;

[1391] a means for providing instructions on makeup procedures based on the proposed cosmetics and makeup methods;

[1392] means for analyzing emotions from the user's facial expressions and voice, storing the emotion data in a database, and dynamically adjusting makeup suggestions;

[1393] A system including:

[1394] (Claim 2)

[1395] The system according to claim 1, further comprising means for displaying makeup instruction information generated based on the proposed cosmetics and makeup method, and prompting the user to apply makeup in accordance with the instructions.

[1396] (Claim 3)

[1397] 10. The system of claim 1, further comprising means for collecting user feedback regarding suggested cosmetics and makeup methods and recording said feedback in said database.

[1398] "Application example 2 when combining emotion engines"

[1399] (Claim 1)

[1400] means for registering user information and storing the user information in a database;

[1401] means for capturing an image of a user's face and analyzing the image to generate facial condition data;

[1402] means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data;

[1403] A means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method;

[1404] means for analyzing the user's emotions and further optimizing cosmetics and makeup methods based on the emotion data;

[1405] A means for displaying the suggested cosmetics and makeup methods on a smart mirror and smart glasses in a physical store to guide the user;

[1406] A system including:

[1407] (Claim 2)

[1408] The system according to claim 1, further comprising means for displaying makeup instruction information generated based on the proposed cosmetics and makeup method, and prompting the user to apply makeup in accordance with the instructions.

[1409] (Claim 3)

[1410] 10. The system of claim 1, further comprising means for collecting user feedback regarding suggested cosmetics and makeup methods and recording said feedback in said database. [Explanation of symbols]

[1411] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for registering user information and storing the user information in a database; means for capturing an image of a user's face and analyzing the image to generate facial condition data; means for suggesting optimal cosmetics and makeup methods based on the user information and the facial condition data; A means for providing step-by-step makeup instruction based on the proposed cosmetics and makeup method; A system including:

2. The system according to claim 1 , further comprising means for displaying makeup instruction information generated based on the proposed cosmetics and makeup method, and urging the user to apply makeup in accordance with the instructions.

3. The system of claim 1 , further comprising means for collecting user feedback regarding suggested cosmetics and makeup methods and recording said feedback in said database.

Images