System

Abstract

A system is provided.SOLUTION: A means for selecting a specific scene by a user, a means for capturing a face image of the user and receiving image data thereof, a means for analyzing the received face image data and generating an ideal face, a means for analyzing a difference between the ideal face and a current face and generating a specific makeup procedure, a means for recommending a skin care product and cosmetics based on a skin condition of the user, and a means for displaying a face image of the user in real time; A system comprising: means for providing a mirror function for performing a makeup procedure; means for receiving and analyzing feedback from a user; and means for evolving a AI model using the analyzed feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, with the increase in remote meetings, both men and women have become more conscious of their appearance. However, many users lack knowledge of proper makeup techniques and skin care, leading to a lack of self-confidence. In addition, it is difficult to check the makeup application in real time, and there is a lack of concrete help to achieve the ideal appearance. [Means for solving the problem]

[0005] The present invention provides an environment in which users can easily learn and apply makeup and skin care that suits them, through a system that includes means for the user to select a specific scene, means for taking an image of the user's face and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the user's current face and generating specific makeup procedures, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for executing makeup procedures, means for receiving and analyzing feedback from the user, and means for evolving an AI model using the analyzed feedback data.

[0006] The "scene selection means" is a means for the user to select a scene according to the purpose, such as a remote conference or a daily outing.

[0007] The "face image capturing means" is a means for capturing a face image of the user and acquiring the image data.

[0008] The "face image receiving means" is a means for receiving photographed face image data.

[0009] The "image analysis means" is a means for analyzing the received facial image data and analyzing the current facial features of the user.

[0010] The "ideal face generating means" is a means for generating an ideal face according to the scene.

[0011] The "difference analysis means" is a means for analyzing the difference between an ideal face and a current face and identifying the gap.

[0012] The "makeup procedure generation means" is a means for generating a specific makeup procedure based on the difference analysis results.

[0013] The "skin care product recommendation means" is a means for recommending an appropriate skin care product based on the user's skin condition.

[0014] "Cosmetics recommendation means" refers to a means for selecting and recommending cosmetics suitable for a particular occasion.

[0015] The "real-time mirror function" is a function that displays an image of the user's face in real time, allowing the user to carry out makeup procedures.

[0016] The "feedback receiving means" is a means for receiving feedback from a user.

[0017] A "feedback analysis means" is a means for analyzing received feedback data and evolving the AI ​​model based on that data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system allows users to select a specific scene, such as a remote meeting or a daily outing, generate an ideal face for that scene, analyze the gap between that and their current face, and suggest specific makeup steps and appropriate skincare products and cosmetics. Using this system, users can learn makeup techniques that suit them, select skincare products according to their skin condition, and check the effects in real time.

[0040] The system consists of the following elements:

[0041] Scene selection method

[0042] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as a remote meeting or everyday outing. This information is sent from the device to the server.

[0043] Facial image capturing means

[0044] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[0045] Facial image receiving means

[0046] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[0047] Image analysis methods

[0048] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene, generating the ideal face.

[0049] Differential analysis means

[0050] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0051] Makeup procedure generation means

[0052] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[0053] Skincare product recommendation tool and cosmetic recommendation tool

[0054] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[0055] Real-time mirror function

[0056] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[0057] Feedback receiving means and feedback analyzing means

[0058] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0059] By combining these elements, users can easily learn and apply the appropriate makeup and skincare products. Real-time confirmation is also possible, resulting in higher satisfaction. Specifically, the system can suggest makeup steps and skincare products to create a professional look suitable for remote meetings, and the suggestions can be confirmed in real time while being applied.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user launches an application.

[0063] Action: The device launches the application and displays the scene selection screen.

[0064] Step 2:

[0065] The user selects a scene.

[0066] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[0067] Step 3:

[0068] The device activates the camera function.

[0069] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[0070] Step 4:

[0071] The user takes a picture of their face and saves the image.

[0072] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[0073] Step 5:

[0074] The terminal transmits the facial image data to the server.

[0075] Operation: The stored facial image data is encrypted and securely sent to the server.

[0076] Step 6:

[0077] The server receives the facial image data.

[0078] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[0079] Step 7:

[0080] The server analyzes the facial image and generates the ideal face.

[0081] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[0082] Step 8:

[0083] The server analyzes the differences between the ideal face and the current face.

[0084] How it works: Compare the generated ideal face with your current face and analyze the gap.

[0085] Step 9:

[0086] The server generates the make procedure.

[0087] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[0088] Step 10:

[0089] The server sends the make procedure to the terminal.

[0090] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[0091] Step 11:

[0092] The device will display the makeup instructions.

[0093] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[0094] Step 12:

[0095] The server evaluates the user's skin condition.

[0096] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[0097] Step 13:

[0098] Servers recommend skin care products and cosmetics.

[0099] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[0100] Step 14:

[0101] The device will activate the real-time mirror function.

[0102] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[0103] Step 15:

[0104] The user performs the make procedure.

[0105] Action: The user performs the suggested makeup steps while using a real-time mirror.

[0106] Step 16:

[0107] The device will display a feedback screen.

[0108] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[0109] Step 17:

[0110] The user submits feedback.

[0111] Action: The user enters feedback and presses the submit button.

[0112] Step 18:

[0113] The terminal transmits the feedback data to the server.

[0114] Action: Feedback data is encrypted and sent to the server.

[0115] Step 19:

[0116] The server analyzes the feedback data.

[0117] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[0118] The above are the detailed processing steps of the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skin care, allowing them to approach remote meetings and daily life with confidence.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] In modern society, there is a demand for instantly preparing the perfect look for various situations, such as remote meetings or outings. However, it is not easy for individual users to know what kind of makeup will bring them closer to their ideal face. It is also difficult to select the appropriate skincare products and cosmetics depending on their skin condition. Furthermore, there is no way to check the effects of makeup in real time as they are applied. A system that can solve these issues is needed.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for allowing a user to select a specific scene, means for capturing a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for executing the makeup routine, means for receiving and analyzing feedback from the user, and means for evolving an artificial intelligence model using the analyzed feedback data. This allows users to learn makeup techniques that suit them, select skin care products according to their skin condition, and proceed while checking the effects in real time.

[0124] "Specific scenes" refer to situations where users need to dress appropriately for the occasion, such as remote meetings or everyday outings.

[0125] "User's face image" refers to photographic data of the user's face taken using the camera of the terminal.

[0126] "Image data" refers to digital data that includes a facial image captured by a user.

[0127] An "ideal face" refers to a facial image that models the user's appearance that is considered optimal for a particular selected scene.

[0128] "Difference" refers to the result of calculating the difference between the current facial image and the ideal facial image.

[0129] "Makeup procedure" refers to the specific makeup method that a user should follow to get closer to their ideal face.

[0130] "Skin condition" refers to an evaluation including the dryness, oil content, health, etc. of the user's skin.

[0131] "Skin care products" refers to beauty products such as creams and lotions used to maintain and improve the health of the skin.

[0132] "Cosmetics" refers to all cosmetics used in makeup (e.g., eye shadow, foundation).

[0133] The "mirror function" refers to a function that uses the device's camera to display the user's face in real time, allowing them to check the progress of their makeup.

[0134] "Feedback" refers to information entered by users about their satisfaction and opinions after using the product.

[0135] An "artificial intelligence model" refers to an algorithm that evolves based on data about the user's makeup and skincare.

[0136] This system allows users to select a specific scene, generate an ideal face for that scene, analyze the gap between the ideal face and their current face, and suggest specific makeup procedures and appropriate skin care products and cosmetics. By using this system, users can learn makeup techniques that suit them, select skin care products according to their skin condition, and check the effects in real time.

[0137] The system consists of the following elements:

[0138] Scene selection method

[0139] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0140] Facial image capturing means

[0141] After the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved in the app. The captured facial image data is encrypted and sent from the device to the server.

[0142] Facial image receiving means

[0143] The server stores the received facial images and inputs them into the analysis engine.

[0144] Image analysis methods

[0145] The server uses OpenCV and deep learning models (e.g., TensorFlow) to analyze the user's current facial features based on the received facial image data. It also runs AI algorithms to generate an ideal face model for each scene, generating the ideal face. For example, if the user selects "Remote Meeting," an ideal face with a professional appearance will be generated.

[0146] Differential analysis means

[0147] The server compares the generated ideal face with the current face and analyzes the gap. An AI algorithm calculates the difference and generates specific makeup steps to close the gap, such as how to apply eyeshadow or choose foundation.

[0148] Makeup procedure generation means

[0149] The server generates specific makeup instructions based on the analysis results. The generated makeup instructions are sent to the device, where the user can review them. The specific makeup instructions are created using a generative AI model (e.g., GPT-4). An example of a prompt to input to the generative AI model is, "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products."

[0150] Skincare product recommendation tool and cosmetic recommendation tool

[0151] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. For example, information such as "Use this moisturizing cream for dry skin" is sent to the terminal, allowing the user to select the optimal skin care and cosmetics.

[0152] Real-time mirror function

[0153] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can perform the suggested makeup steps and check the results. By tapping the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed in real time.

[0154] Feedback receiving means and feedback analyzing means

[0155] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The server analyzes the received feedback data and uses it to evolve the AI ​​model. The user enters their satisfaction after applying the makeup, and the data is analyzed and used to improve future makeup suggestions.

[0156] By combining these elements, users can easily learn the proper makeup and skincare products to create a look that suits the real-life situation, and the ability to see the results in real time provides a high level of satisfaction.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1:

[0159] The user launches the app and a scene selection screen appears. The user taps to select the desired scene from options such as remote meeting or everyday outing. The input includes the user's scene selection information. This information is sent from the device to the server. Specifically, the user taps the app icon to launch the app, and then taps the "Remote Meeting" button from the multiple scene selection buttons displayed on the home screen.

[0160] Input: User selected scene information

[0161] Output: Transfer of scene information from the device to the server

[0162] Step 2:

[0163] After the user selects a scene, the device launches the camera function and prompts the user to take a photo of their face. The user uses the camera to take a photo of their face and then reviews and saves the image within the app. The input is the user's face image. The captured face image data is encrypted and sent from the device to the server. Specifically, the device automatically launches the camera app, displays a message saying "Please take a photo of your face," and the user taps the shutter button to take a photo of their face. When the user presses the "Save" button, the image is encrypted and sent to the server.

[0164] Input: User's face image

[0165] Output: Sending encrypted facial image data to the server

[0166] Step 3:

[0167] The server stores the received facial image data and inputs it into the analysis engine. The input is encrypted facial image data. Based on this, the system analyzes the user's current facial features and generates an ideal face model for the scene. Specifically, it uses OpenCV and TensorFlow to analyze facial features and generates the ideal face using AI algorithms.

[0168] Input: Encrypted facial image data

[0169] Output: User's facial feature analysis results, ideal face model

[0170] Step 4:

[0171] The server compares the generated ideal face with the user's current face and analyzes the gap. The inputs are the user's current facial features and the ideal face model. Based on these differences, the server generates specific makeup steps to help the user achieve their ideal face. Specifically, the AI ​​algorithm calculates the optimal makeup steps (e.g., applying a thin layer of eyeshadow around the eyes) to close the gap.

[0172] Input: Current facial features, ideal face model

[0173] Output: Makeup steps to help the user achieve their ideal face

[0174] Step 5:

[0175] The generated makeup instructions are sent to the device, where the user can review them. The input is the makeup instructions. The generated makeup instructions are created using a generative AI model (e.g., GPT-4). Specifically, the server inputs the prompt "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products." into the generative AI model, and then sends the results to the device.

[0176] Input: Generated Make steps

[0177] Output: Display the make steps on the terminal for the user to review.

[0178] Step 6:

[0179] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. The input is the user's skin condition data. The server recommends specific skin care products and cosmetics based on the skin analysis results. For example, information such as "Use this moisturizing cream for dry skin" is sent to the device.

[0180] Input: User's skin condition data

[0181] Output: Recommended skincare products and cosmetic information

[0182] Step 7:

[0183] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can apply the suggested makeup steps and check the results. The input is a real-time image of the face. Specifically, when the user taps the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed.

[0184] Input: Real-time facial video

[0185] Output: Facial image projected on a real-time mirror

[0186] Step 8:

[0187] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The input is the user's feedback data. The server analyzes the received feedback data and uses it to evolve the AI ​​model. Specifically, the user enters their satisfaction after applying the makeup into the device, and the data is analyzed and used to improve future makeup suggestions.

[0188] Input: User feedback data

[0189] Output: Analyzed feedback data, evolved AI model

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] Compared to face-to-face services, makeup advice in virtual stores often falls short, making it difficult for users to achieve professional makeup in the comfort of their own homes. It is also often difficult for users to determine effective makeup techniques and appropriate skincare products on their own. Furthermore, existing systems lack the ability to check the effects of makeup in real time or to use feedback to improve AI models. Therefore, there is a need for a system that can provide highly accurate makeup advice in virtual stores and improve user satisfaction.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes: a means for allowing a user to select a specific scene; a means for capturing a facial image of the user and receiving the image data; a means for analyzing the received facial image data and generating an ideal face; a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; a means for recommending skin care products and cosmetics based on the user's skin condition; a means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; a means for receiving and analyzing feedback from the user; a means for evolving an AI model using the analyzed feedback data; a means for providing makeup advice in a virtual store; and a means for confirming and executing the suggested makeup routine in real time. This allows users to receive professional makeup advice from the comfort of their own home and select effective makeup techniques and skin care products. Furthermore, the ability to check the makeup effects in real time significantly increases user satisfaction.

[0195] "User" refers to an individual who uses this system to receive makeup advice tailored to a specific scene.

[0196] A "specific scene" refers to a specific situation or occasion that the user desires, such as a remote meeting, a party event, or everyday outings.

[0197] "Facial image" refers to image data of a user's face that the user provides to the system.

[0198] "Means for receiving" refers to a method or device for the system to import facial image data and other information provided by the user into a server or the like.

[0199] "Means for analyzing" includes techniques and algorithms for analyzing a user's facial features based on facial image data received by the system.

[0200] An "ideal face" refers to a model that generates the optimal facial condition and appearance for a specific scene desired by the user.

[0201] "Difference" refers to the difference that exists between the current user's facial features and the generated ideal facial features.

[0202] The "makeup procedure" refers to the specific makeup method or procedure that the user follows to achieve an ideal face.

[0203] "Skin Care Products" refers to cosmetics and related products for skin care that are recommended based on the user's skin condition.

[0204] "Cosmetics" refers to all cosmetics that users use for makeup.

[0205] The "mirror function" refers to a function that allows users to project their own face on the screen in real time and check the makeup steps as they are performed.

[0206] "Feedback" refers to information about the user's experience and satisfaction with the system after using it.

[0207] "Means for checking in real time" refers to a method or device that allows a user to check the results of a proposed makeup routine in real time while performing the routine.

[0208] "Virtual Store" means an online makeup advice and skin care product recommendation service provided via the Internet.

[0209] An embodiment of the present invention is a system for a virtual store where users can receive professional makeup advice from the comfort of their own home. The system provides makeup procedures and appropriate skin care products tailored to specific scenes, and users can check and implement the effects in real time.

[0210] Hardware and software used

[0211] Hardware: Terminal devices such as smartphones, PCs, and cameras

[0212] Software: Python, Flask, OpenCV, Requests, Heroku, AWS (EC2, S3)

[0213] Overall system flow

[0214] 1. Scene selection:

[0215] The user launches the application and selects a specific scene, such as a remote meeting, a party event, or a daily outing, and this information is sent from the device to the server.

[0216] 2. Facial image capture:

[0217] When the user selects a scene, the device activates the camera function and prompts the user to take a face image. The user uses the camera to take a picture of their face, and the image is saved.

[0218] 3. Receipt and analysis of image data:

[0219] The captured facial image data is encrypted and sent from the device to a server. The server's analysis engine receives this data and analyzes the user's facial features. It also runs an AI algorithm to generate an ideal face model for each scene, generating the ideal face.

[0220] 4. Differential analysis:

[0221] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0222] 5. Makeup procedure and skin care product suggestions:

[0223] Based on the analysis results, the server generates specific makeup procedures for the eyes, skin, lips, etc. and sends them to the device. It also recommends appropriate skin care products and cosmetics based on the user's skin condition. This information can be viewed on the device.

[0224] 6. Real-time mirror function:

[0225] The device uses the front camera to display a real-time image of the user's face, allowing the user to see the results of the proposed makeup application in real time.

[0226] 7. Feedback Receipt and Analysis:

[0227] After the makeup application is complete, the device displays a questionnaire screen to the user about their experience and satisfaction, and collects their feedback. This data is sent to and stored on a server and used to improve the AI ​​model.

[0228] Examples and prompts

[0229] Specific examples

[0230] When a user wants to join a remote meeting, they select "Remote Meeting" from the scene selection menu and take a photo of their face. The server then suggests a professional look suitable for the remote meeting and applies makeup while checking it in real time. During this process, the user can purchase the suggested skin care products and provide feedback to improve the system.

[0231] Prompt Sentence Examples

[0232] "Analyze the facial features in this image and suggest the best makeup routine for a remote meeting."

[0233] As described above, this system enables users to receive highly accurate makeup advice and check and apply it in real time, significantly improving satisfaction in the virtual store.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] The user launches the application and a scene selection screen appears. The user selects a specific scene, such as a remote meeting, a party event, or a daily outing. This scene information is sent from the device to the server, which then obtains input to determine the next step based on the user's selection.

[0237] Step 2:

[0238] When the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved within the app. Once the face image is saved, the device encrypts the data before sending it to the server.

[0239] Step 3:

[0240] The server receives the encrypted facial image data, decrypts it, and inputs it into the analysis engine. The analysis engine extracts features from the image and analyzes the user's current facial features. Specifically, it uses AI algorithms to analyze the facial contours, position of the eyes, nose, and mouth, as well as the condition of the skin.

[0241] Step 4:

[0242] The analysis engine runs an AI algorithm to generate an ideal face model for each scene. The algorithm uses the prompt "Please analyze the facial features in this image and suggest the best makeup routine for a remote meeting" as input. The server then obtains the ideal face model as output.

[0243] Step 5:

[0244] The server analyzes the differences between the user's current face and the generated ideal face. This process compares the facial contours, the relative positions of each feature, and color tones to identify any gaps. Based on the results of this analysis, the server generates specific makeup instructions to help the user achieve their ideal face.

[0245] Step 6:

[0246] The server then recommends appropriate skincare products and cosmetics based on the user's skin condition along with the generated makeup instructions. This information is then sent to the device, where the user can view the specific makeup instructions and recommended skincare products.

[0247] Step 7:

[0248] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the real-time mirror function to check the results of the proposed makeup steps as they are applied.

[0249] Step 8:

[0250] After the makeup application is complete, the device displays a survey screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0251] This allows makeup advice to be given in the virtual store with high accuracy, realizing a system that improves user satisfaction.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] The present invention is a system that allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between that and the user's current face, and suggests specific makeup routines, skin care products, and cosmetics. The present invention also incorporates an emotion engine that recognizes the user's emotions, and has the function of analyzing the user's real-time emotions to adjust the makeup routine and skin care product recommendations, allowing users to learn and apply makeup routines that suit them and see the results in real time.

[0254] The system consists of the following elements:

[0255] Scene selection method

[0256] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0257] Facial image capturing means

[0258] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[0259] Facial image receiving means

[0260] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[0261] Image analysis methods

[0262] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[0263] Differential analysis means

[0264] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0265] Makeup procedure generation means

[0266] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[0267] Skincare product recommendation tool and cosmetic recommendation tool

[0268] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[0269] Real-time mirror function

[0270] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[0271] Emotion Engine

[0272] The device uses an emotion engine to analyze the user's real-time emotions based on facial video and voice data, and the analysis results are sent to a server where they are used to adjust makeup routines and skin care product recommendations.

[0273] Feedback receiving means and feedback analyzing means

[0274] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0275] Specific examples

[0276] A user wants to put on makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup instructions (for example, "shape your eyebrows" or "use a light beige foundation") are generated and sent to the device. The user can check the instructions on the device and use the real-time mirror function to apply their makeup.

[0277] Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides feedback and adjustments accordingly. After the makeup is complete, the user can send feedback, and the server will use that data to improve the AI ​​model and suggest more effective makeup procedures for the next time.

[0278] As described above, by providing detailed support that also takes the user's emotions into consideration, users can easily achieve their ideal appearance.

[0279] The processing flow will be explained below.

[0280] Step 1:

[0281] The user launches an application.

[0282] Action: The device launches the application and displays the scene selection screen.

[0283] Step 2:

[0284] The user selects a scene.

[0285] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[0286] Step 3:

[0287] The device activates the camera function.

[0288] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[0289] Step 4:

[0290] The user takes a picture of their face and saves the image.

[0291] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[0292] Step 5:

[0293] The terminal transmits the facial image data to the server.

[0294] Operation: The stored facial image data is encrypted and securely sent to the server.

[0295] Step 6:

[0296] The server receives the facial image data.

[0297] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[0298] Step 7:

[0299] The server analyzes the facial image and generates the ideal face.

[0300] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[0301] Step 8:

[0302] The server analyzes the differences between the ideal face and the current face.

[0303] How it works: Compare the generated ideal face with your current face and analyze the gap.

[0304] Step 9:

[0305] The server generates the make procedure.

[0306] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[0307] Step 10:

[0308] The server sends the make procedure to the terminal.

[0309] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[0310] Step 11:

[0311] The device will display the makeup instructions.

[0312] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[0313] Step 12:

[0314] The server evaluates the user's skin condition.

[0315] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[0316] Step 13:

[0317] Servers recommend skin care products and cosmetics.

[0318] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[0319] Step 14:

[0320] The device will activate the real-time mirror function.

[0321] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[0322] Step 15:

[0323] The user performs the make procedure.

[0324] Action: The user performs the suggested makeup steps while using a real-time mirror.

[0325] Step 16:

[0326] The emotion engine analyzes the user's emotions.

[0327] How it works: The device collects the user's facial expressions and voice data and uses an emotion engine to analyze emotions in real time.

[0328] Step 17:

[0329] The server receives and adjusts the emotion data.

[0330] How it works: The server adjusts makeup routines and skin care product recommendations based on the emotion data it receives.

[0331] Step 18:

[0332] The device will display a feedback screen.

[0333] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[0334] Step 19:

[0335] The user submits feedback.

[0336] Action: The user enters feedback and presses the submit button.

[0337] Step 20:

[0338] The terminal transmits the feedback data to the server.

[0339] Action: Feedback data is encrypted and sent to the server.

[0340] Step 21:

[0341] The server analyzes the feedback data.

[0342] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[0343] The above are the detailed processing steps for combining the emotion engine with the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skincare routine, allowing them to approach remote meetings and daily life with confidence.

[0344] Example 2

[0345] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional makeup instruction systems are unable to consider users' emotions or real-time reactions, and only provide general advice on specific makeup steps and skin care product recommendations. This makes it difficult for users to receive specific and effective instruction to achieve their ideal appearance.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for the user to select a specific scene; means for capturing a facial image of the user and receiving the image data; means for analyzing the received facial image data to generate an ideal face; means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; means for recommending skin care products and cosmetics based on the user's skin condition; means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; means for analyzing real-time emotions based on the user's facial image and voice and adjusting the makeup routine and skin care product recommendations; means for receiving and analyzing feedback from the user; and means for evolving the generative AI model using the analyzed feedback data. This enables detailed makeup instruction tailored to the user's emotions and real-time situation, and recommendations of skin care products and cosmetics tailored to the user's individual needs.

[0348] "Scene selection means" refers to a means by which a user selects a particular scene.

[0349] The "face image capturing means" refers to a means for capturing a face image of the user.

[0350] The "image data receiving means" refers to a means for receiving face image data sent from a terminal on the server side.

[0351] The "ideal face generating means" refers to a means for generating an ideal face based on received face image data.

[0352] "Difference analysis means" refers to means for analyzing the difference between an ideal face and a current face.

[0353] The "makeup procedure generation means" refers to a means for generating a specific makeup procedure based on the difference analysis results.

[0354] "Recommendation means" refers to a means for recommending skin care products and cosmetics based on the user's skin condition.

[0355] The "mirror function providing means" refers to a means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures.

[0356] "Emotion analysis means" refers to a means for analyzing real-time emotions based on facial image and voice data of a user.

[0357] "Feedback receiving means" refers to a means for receiving feedback from a user.

[0358] "Feedback analysis means" refers to means for analyzing received feedback data.

[0359] "Generative AI model" refers to an artificial intelligence model that generates ideal faces and provides makeup instructions.

[0360] This system allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between the current face and the ideal face, and provides specific makeup procedures, skin care products, and cosmetics. Furthermore, this system has the function of analyzing the user's real-time emotions and adjusting the makeup procedures and skin care product recommendations.

[0361] The system consists of the following components:

[0362] Hardware and Software Configuration

[0363] User device: A smartphone or tablet with a built-in camera, display, and microphone.

[0364] Server: A high-performance computer server used to analyze facial image data and emotional data, and generate ideal facial and makeup procedures.

[0365] Software: A software stack including image processing libraries such as OpenCV and Dlib, emotion recognition APIs, and generative AI models.

[0366] Specific processing explanation

[0367] 1. Application launch and scene selection

[0368] On the device: When the user launches the application, a scene selection screen appears. The user selects a scene, such as a remote meeting, work, a date, or a casual outing. This information is sent to the server.

[0369] 2. Taking a facial image

[0370] Device: When the user selects a scene, the camera function is activated and the user is prompted to take a facial image. The user uses the camera to take a facial image, then reviews and saves the image.

[0371] 3. Sending and receiving facial image data

[0372] Terminal: The captured facial image data is encrypted for security purposes and sent to the server.

[0373] Server: Decodes the received facial image data and stores it in a database.

[0374] 4. Facial feature analysis and ideal face model generation

[0375] Server: Based on the received facial image data, it uses image processing libraries such as OpenCV and Dlib to analyze the user's current facial features, and then uses generative AI models to generate an ideal face model for the scene.

[0376] 5. Difference analysis and generation of specific make-up steps

[0377] Server: Compares the difference between the ideal face and the current face and analyzes the gap. Based on the analyzed data, it generates specific makeup instructions to help the user achieve their ideal face.

[0378] 6. Skin care and cosmetic product recommendations

[0379] Server: Based on facial image analysis, evaluates the user's skin condition and recommends the most suitable skincare products and cosmetics. This information is also sent to the device and displayed to the user.

[0380] 7. Executing the real-time mirror function

[0381] Device: Uses the front camera to display a real-time image of the user's face, allowing the user to see the effects of the suggested makeup steps as they are performed.

[0382] 8. Emotional analysis and feedback regulation

[0383] Device: Analyzes facial video and audio data using an emotion analysis API, and transmits the user's real-time emotions to the server.

[0384] Server: Based on the analysis results, it adjusts makeup routines and skin care product recommendations in real time.

[0385] 9. Feedback Collection and Analysis

[0386] Device: After the makeup application is completed, the user is asked to fill out a survey about their experience and satisfaction.

[0387] User: Enter and submit feedback.

[0388] Server: Stores and analyzes the received feedback data, evolves the generative AI model based on that data, and reflects it the next time it is used.

[0389] Specific examples

[0390] A user wants to apply makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for "Remote Meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "Shape your eyebrows" or "Use a light beige foundation") are generated and sent to the device. The user checks the steps on the device and uses the real-time mirror function to apply the makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides appropriate feedback and adjustments. After completing the makeup, the user submits feedback, and the server uses that data to evolve the generative AI model and suggest a more customized makeup step the next time they use it.

[0391] Prompt Sentence Examples

[0392] "I would like the AI ​​to automate the process of comparing my current face with the ideal face for remote meeting situations and suggesting specific makeup steps. I would also like the facial image data to be encrypted before transmission, and appropriate skin care products and cosmetics to be recommended based on the analysis results."

[0393] The above is a detailed description of the preferred embodiment of the invention, which allows users to easily achieve their ideal appearance.

[0394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0395] Step 1: Launch the application and select a scene

[0396] On the device: When the user launches the application, a scene selection screen appears, displaying multiple options (e.g., remote meeting, work, date, everyday outing, etc.) as input.

[0397] User: Selects the desired scene and enters the information into the device.

[0398] Terminal: Sends the user's selection to the server. The input data to the server is the scene information selected by the user. The output data is the scene information sent to the server.

[0399] Step 2: Capture a facial image

[0400] Terminal: When a scene is selected, the terminal activates the camera function and prompts the user to take a facial image. The input at this stage is the scene information that triggers the camera activation.

[0401] User: Take a picture of your face with the camera and view and save the image within the app.

[0402] Terminal: The captured face image data is saved. The output data is the captured face image data.

[0403] Step 3: Encrypt and transmit facial image data

[0404] Terminal: Encrypts the stored facial image data to ensure security. The input data is facial image data, which is then encrypted.

[0405] Terminal: Sends the encrypted facial image data to the server. The encrypted facial image data is sent to the server and becomes the output data.

[0406] Step 4: Receiving and storing facial image data

[0407] Server: Receives the encrypted facial image data and decrypts it. The input data is the encrypted facial image data.

[0408] Server: Saves the decoded facial image data in the database. The decoded facial image data is saved as output.

[0409] Step 5: Facial feature analysis and ideal face model generation

[0410] Server: Analyzes the user's current facial features based on the received facial image data. This process uses image processing libraries such as OpenCV and Dlib. The input data is the decoded facial image data.

[0411] Server: Uses a generative AI model to generate an ideal face model based on scene information. The input data is the decoded face image data and scene information, and the output data is the generated ideal face model.

[0412] Step 6: Difference analysis and generation of specific make procedures

[0413] Server: Compares and analyzes the differences between the ideal face and the current face. A facial landmark detection algorithm is used for the analysis. The input data is the current facial features and the ideal face model.

[0414] Server: Generates specific make procedures based on the difference analysis results. The generated make procedures become the output data.

[0415] Step 7: Skincare and cosmetic product recommendations

[0416] Server: Evaluates the user's skin condition based on the results of facial image analysis and recommends appropriate skin care products and cosmetics. The input data is the results of facial image analysis.

[0417] Server: Sends the recommendation information to the terminal and displays it to the user. The information on the recommended skin care products and cosmetics is the output data.

[0418] Step 8: Run the real-time mirror function

[0419] Terminal: Uses the front camera to display the user's face image in real time. The camera is activated when the user performs the makeup procedure. The input data is the trigger to start the makeup procedure.

[0420] User: Perform the proposed makeup steps and check their effectiveness.

[0421] Terminal: Real-time facial image is displayed to the user, which becomes the output data.

[0422] Step 9: Emotional analysis and feedback adjustment

[0423] Terminal: Analyzes real-time emotions using an emotion analysis API based on the user's facial video and audio data. The input data is real-time facial video and audio data.

[0424] Server: Receives the analysis results and adjusts makeup routines and skin care product recommendations in real time. The output data is the adjusted makeup routine and skin care product information.

[0425] Step 10: Collect and analyze feedback

[0426] Terminal: After the user has finished applying makeup, a questionnaire about the user's experience and satisfaction is displayed. The input data is the trigger for finishing the makeup.

[0427] User: Enter and submit feedback.

[0428] Server: Stores and analyzes the received feedback data. The analysis results are used to evolve the generative AI model and are reflected the next time it is used. The output data is the evolved generative AI model.

[0429] (Application example 2)

[0430] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Conventional makeup and skincare recommendation systems are not optimized for the user's emotions or specific situations, limiting their ability to improve the user experience. Furthermore, users often lack real-time feedback and adjustments, preventing them from accurately executing the suggested steps. Furthermore, when using smart glasses or other devices, their interfaces are often not optimized, resulting in a lack of user convenience.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0433] In this invention, the server includes a means for capturing a user's facial image and receiving the image data, a means for analyzing the received facial image data and generating an ideal face, a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, a means for analyzing the user's emotions and adjusting the recommendations, an interface displayed on the smart glasses, a means for receiving and analyzing feedback from the user, and a means for evolving the AI ​​model using the analyzed feedback data. This enables recommendations of optimal makeup routines and products according to the user's specific emotions and situations. Furthermore, real-time adjustments and feedback improve the user experience and enable intuitive operation using the smart glasses.

[0434] The "means for the user to select a specific scene" is an operation interface that allows the user to select a desired scene according to a specific situation such as a remote meeting, a date, or a party.

[0435] "Means for capturing an image of a user's face and receiving the image data" refers to a device or system for capturing an image of a user's face using smart glasses or a camera device and receiving the data to an application or server.

[0436] The "means for analyzing the received facial image data and generating an ideal face" refers to an AI algorithm and a system for executing the algorithm, which uses the received facial image data to generate an ideal face suitable for a specific scene.

[0437] The "means for analyzing the difference between the ideal face and the current face and generating specific makeup procedures" is a system for comparing and analyzing the gap between the current face and the generated ideal face, and obtaining specific makeup procedures to fill the gap.

[0438] The "means for recommending skin care products and cosmetics based on the user's skin condition" is a system for analyzing the user's skin condition and suggesting optimal skin care products and cosmetics.

[0439] "Means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures" refers to a function that displays a real-time image of the user's face through the display of smart glasses or a device, allowing the user to check the procedure while applying makeup.

[0440] The "means for analyzing the user's emotions and adjusting the recommended content" is a function that analyzes the user's emotions from their facial expressions and voice, and dynamically adjusts the makeup steps and recommended content based on that.

[0441] The "interface means displayed on the smart glasses" refers to a user interface for displaying information such as makeup procedures, recommended products, and real-time feedback on the display of the smart glasses.

[0442] The "means for receiving and analyzing feedback from users" is a system for collecting feedback from users regarding their experience and satisfaction with the product after they have finished applying their makeup, and analyzing that data.

[0443] The "means for evolving the AI ​​model using the analyzed feedback data" refers to a system that improves and evolves the AI ​​model based on feedback data collected from users, enabling it to suggest more appropriate makeup procedures and products the next time the model is used.

[0444] This invention is a system that generates an ideal face for a specific scene, analyzes the gap between the ideal face and the current face, and suggests makeup routines and skin care products. It also improves the user experience by dynamically adjusting the recommendations using emotion analysis. A detailed description of a system for implementing this invention is provided below.

[0445] The system includes means for a user to select a specific scene, means for taking a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating specific makeup steps, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for performing makeup steps, means for analyzing the user's emotions and adjusting the recommended content, interface means displayed on the smart glasses, means for receiving and analyzing feedback from the user, and means for evolving the AI ​​model using the analyzed feedback data.

[0446] The server runs an AI algorithm to generate an ideal face based on the scene selected by the user. For example, for a remote meeting, an algorithm is used to generate a natural and professional appearance. The user's facial image is captured using the camera in the smart glasses, and the image data is encrypted and sent to the server. The server analyzes the received facial image data and extracts the current facial features.

[0447] The server then analyzes the differences between the ideal face and the current face and generates specific makeup steps to fill the gap. Based on the user's skin condition, skincare products and cosmetics are also suggested. This information is displayed in real time on the smart glasses' display, allowing the user to apply makeup while viewing it. Additionally, a real-time mirror function allows users to check their own face on the display as they apply their makeup.

[0448] An emotion engine using facial expression and voice recognition is used to analyze user emotions. This emotion engine analyzes the user's emotional state in real time and adjusts recommendations based on the results. For example, if the user appears dissatisfied, it will suggest alternative products or procedures to improve satisfaction.

[0449] After the makeup application is complete, the system receives feedback from the user and analyzes the data. This feedback data is used to improve the AI ​​model, allowing it to make more appropriate suggestions the next time the product is used.

[0450] As a specific example, a user wants to apply makeup for a remote meeting and launches the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face using the smart glasses' camera. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "shape your eyebrows" or "use a light beige foundation") are generated and displayed on the smart glasses' display. The user checks these as they apply their makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with their makeup, it provides appropriate feedback and adjustments. After completing their makeup, the user submits feedback, and the server uses that data to evolve the AI ​​model and suggest more effective makeup steps the next time they use the app.

[0451] An example of a prompt is, "The user has selected makeup for a remote meeting. Generate an ideal face based on the current facial image. Next, analyze the gap between the current face and the ideal face and suggest specific makeup steps to fill the gap (e.g., shaping the eyebrows, using a light beige foundation, etc.). Furthermore, analyze the user's emotions from their facial expressions and voice and consider how to optimize the makeup steps."

[0452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0453] Step 1:

[0454] The user launches the application using the smart glasses and selects the desired scene (e.g., remote meeting, date, party) on the scene selection screen.

[0455] Input: User scene selection (e.g. "Remote Meeting")

[0456] Output: Selected scene information

[0457] Step 2:

[0458] The terminal transmits the scene information selected by the user to the server, and then prompts the user to take a facial image using the camera built into the smart glasses.

[0459] Input: User operation (taking a face image after selecting a scene)

[0460] Output: Photographed face image

[0461] Step 3:

[0462] The facial image data captured by the terminal is encrypted and sent to the server.

[0463] Input: Photographed face image data

[0464] Output: Encrypted facial image data

[0465] Step 4:

[0466] The server analyzes the received facial image data and extracts the current facial features.

[0467] Input: Encrypted facial image data

[0468] Output: Current facial feature data

[0469] Step 5:

[0470] The server runs an AI algorithm to generate an ideal face for the selected scene.

[0471] Input: User's scene information and current facial feature data

[0472] Output: Ideal face data

[0473] Step 6:

[0474] The server compares the ideal face data with the current face data and analyzes the differences.

[0475] Input: Current face data and ideal face data

[0476] Output: Differential data

[0477] Step 7:

[0478] The server generates specific makeup procedures based on the differential data and recommends skin care products and cosmetics.

[0479] Input: differential data

[0480] Output: A list of specific makeup steps and recommended products

[0481] Step 8:

[0482] The device receives makeup instructions and recommended product information from the server and displays them on the smart glasses display in real time. The user applies makeup while viewing this information.

[0483] Input: List of makeup steps and recommended products

[0484] Output: Makeup instructions and recommended product information displayed on smart glasses

[0485] Step 9:

[0486] The device's front camera is used to display a real-time image of the user's face while checking the makeup procedure.

[0487] Input: Real-time facial video

[0488] Output: Real-time facial image displayed on the display

[0489] Step 10:

[0490] The server's emotion engine analyzes the user's facial expressions and voice, assesses the user's emotional state in real time, and adjusts recommendations as needed.

[0491] Input: Real-time facial video and audio data

[0492] Output: Tailored recommendations and feedback

[0493] Step 11:

[0494] After the makeup application is completed, the device displays a feedback questionnaire to the user to collect data on satisfaction and usability.

[0495] Input: Feedback data from users

[0496] Output: Feedback data stored on the device

[0497] Step 12:

[0498] The server improves and evolves the AI ​​model based on the feedback data it receives.

[0499] Input: Feedback data

[0500] Output: An improved AI model

[0501] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0502] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0503] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0504] [Second embodiment]

[0505] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0506] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0507] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0508] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0509] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0510] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0511] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0512] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0513] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0514] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0515] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0516] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0517] The system allows users to select a specific scene, such as a remote meeting or a daily outing, generate an ideal face for that scene, analyze the gap between that and their current face, and suggest specific makeup steps and appropriate skincare products and cosmetics. Using this system, users can learn makeup techniques that suit them, select skincare products according to their skin condition, and check the effects in real time.

[0518] The system consists of the following elements:

[0519] Scene selection method

[0520] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0521] Facial image capturing means

[0522] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[0523] Facial image receiving means

[0524] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[0525] Image analysis methods

[0526] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[0527] Differential analysis means

[0528] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0529] Makeup procedure generation means

[0530] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[0531] Skincare product recommendation tool and cosmetic recommendation tool

[0532] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[0533] Real-time mirror function

[0534] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[0535] Feedback receiving means and feedback analyzing means

[0536] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0537] By combining these elements, users can easily learn and apply the appropriate makeup and skincare products. Real-time confirmation is also possible, resulting in higher satisfaction. Specifically, the system can suggest makeup steps and skincare products to create a professional look suitable for remote meetings, and the suggestions can be confirmed in real time while being applied.

[0538] The processing flow will be explained below.

[0539] Step 1:

[0540] The user launches an application.

[0541] Action: The device launches the application and displays the scene selection screen.

[0542] Step 2:

[0543] The user selects a scene.

[0544] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[0545] Step 3:

[0546] The device activates the camera function.

[0547] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[0548] Step 4:

[0549] The user takes a picture of their face and saves the image.

[0550] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[0551] Step 5:

[0552] The terminal transmits the facial image data to the server.

[0553] Operation: The stored facial image data is encrypted and securely sent to the server.

[0554] Step 6:

[0555] The server receives the facial image data.

[0556] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[0557] Step 7:

[0558] The server analyzes the facial image and generates the ideal face.

[0559] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[0560] Step 8:

[0561] The server analyzes the differences between the ideal face and the current face.

[0562] How it works: Compare the generated ideal face with your current face and analyze the gap.

[0563] Step 9:

[0564] The server generates the make procedure.

[0565] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[0566] Step 10:

[0567] The server sends the make procedure to the terminal.

[0568] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[0569] Step 11:

[0570] The device will display the makeup instructions.

[0571] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[0572] Step 12:

[0573] The server evaluates the user's skin condition.

[0574] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[0575] Step 13:

[0576] Servers recommend skin care products and cosmetics.

[0577] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[0578] Step 14:

[0579] The device will activate the real-time mirror function.

[0580] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[0581] Step 15:

[0582] The user performs the make procedure.

[0583] Action: The user performs the suggested makeup steps while using a real-time mirror.

[0584] Step 16:

[0585] The device will display a feedback screen.

[0586] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[0587] Step 17:

[0588] The user submits feedback.

[0589] Action: The user enters feedback and presses the submit button.

[0590] Step 18:

[0591] The terminal transmits the feedback data to the server.

[0592] Action: Feedback data is encrypted and sent to the server.

[0593] Step 19:

[0594] The server analyzes the feedback data.

[0595] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[0596] The above are the detailed processing steps of the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skin care, allowing them to approach remote meetings and daily life with confidence.

[0597] Example 1

[0598] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0599] In modern society, there is a demand for instantly preparing the perfect look for various situations, such as remote meetings or outings. However, it is not easy for individual users to know what kind of makeup will bring them closer to their ideal face. It is also difficult to select the appropriate skincare products and cosmetics depending on their skin condition. Furthermore, there is no way to check the effects of makeup in real time as they are applied. A system that can solve these issues is needed.

[0600] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0601] In this invention, the server includes means for allowing a user to select a specific scene, means for capturing a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for executing the makeup routine, means for receiving and analyzing feedback from the user, and means for evolving an artificial intelligence model using the analyzed feedback data. This allows users to learn makeup techniques that suit them, select skin care products according to their skin condition, and proceed while checking the effects in real time.

[0602] "Specific scenes" refer to situations where users need to dress appropriately for the occasion, such as remote meetings or everyday outings.

[0603] "User's face image" refers to photographic data of the user's face taken using the camera of the terminal.

[0604] "Image data" refers to digital data that includes a facial image captured by a user.

[0605] An "ideal face" refers to a facial image that models the user's appearance that is considered optimal for a particular selected scene.

[0606] "Difference" refers to the result of calculating the difference between the current facial image and the ideal facial image.

[0607] "Makeup procedure" refers to the specific makeup method that a user should follow to get closer to their ideal face.

[0608] "Skin condition" refers to an evaluation including the dryness, oil content, health, etc. of the user's skin.

[0609] "Skin care products" refers to beauty products such as creams and lotions used to maintain and improve the health of the skin.

[0610] "Cosmetics" refers to all cosmetics used in makeup (e.g., eye shadow, foundation).

[0611] The "mirror function" refers to a function that uses the device's camera to display the user's face in real time, allowing them to check the progress of their makeup.

[0612] "Feedback" refers to information entered by users about their satisfaction and opinions after using the product.

[0613] An "artificial intelligence model" refers to an algorithm that evolves based on data about the user's makeup and skincare.

[0614] This system allows users to select a specific scene, generate an ideal face for that scene, analyze the gap between the ideal face and their current face, and suggest specific makeup procedures and appropriate skin care products and cosmetics. By using this system, users can learn makeup techniques that suit them, select skin care products according to their skin condition, and check the effects in real time.

[0615] The system consists of the following elements:

[0616] Scene selection method

[0617] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0618] Facial image capturing means

[0619] After the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved in the app. The captured facial image data is encrypted and sent from the device to the server.

[0620] Facial image receiving means

[0621] The server stores the received facial images and inputs them into the analysis engine.

[0622] Image analysis methods

[0623] The server uses OpenCV and deep learning models (e.g., TensorFlow) to analyze the user's current facial features based on the received facial image data. It also runs AI algorithms to generate an ideal face model for each scene, generating the ideal face. For example, if the user selects "Remote Meeting," an ideal face with a professional appearance will be generated.

[0624] Differential analysis means

[0625] The server compares the generated ideal face with the current face and analyzes the gap. An AI algorithm calculates the difference and generates specific makeup steps to close the gap, such as how to apply eyeshadow or choose foundation.

[0626] Makeup procedure generation means

[0627] The server generates specific makeup instructions based on the analysis results. The generated makeup instructions are sent to the device, where the user can review them. The specific makeup instructions are created using a generative AI model (e.g., GPT-4). An example of a prompt to input to the generative AI model is, "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products."

[0628] Skincare product recommendation tool and cosmetic recommendation tool

[0629] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. For example, information such as "Use this moisturizing cream for dry skin" is sent to the terminal, allowing the user to select the optimal skin care and cosmetics.

[0630] Real-time mirror function

[0631] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can perform the suggested makeup steps and check the results. By tapping the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed in real time.

[0632] Feedback receiving means and feedback analyzing means

[0633] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The server analyzes the received feedback data and uses it to evolve the AI ​​model. The user enters their satisfaction after applying the makeup, and the data is analyzed and used to improve future makeup suggestions.

[0634] By combining these elements, users can easily learn the proper makeup and skincare products to create a look that suits the real-life situation, and the ability to see the results in real time provides a high level of satisfaction.

[0635] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0636] Step 1:

[0637] The user launches the app and a scene selection screen appears. The user taps to select the desired scene from options such as remote meeting or everyday outing. The input includes the user's scene selection information. This information is sent from the device to the server. Specifically, the user taps the app icon to launch the app, and then taps the "Remote Meeting" button from the multiple scene selection buttons displayed on the home screen.

[0638] Input: User selected scene information

[0639] Output: Transfer of scene information from the device to the server

[0640] Step 2:

[0641] After the user selects a scene, the device launches the camera function and prompts the user to take a photo of their face. The user uses the camera to take a photo of their face and then reviews and saves the image within the app. The input is the user's face image. The captured face image data is encrypted and sent from the device to the server. Specifically, the device automatically launches the camera app, displays a message saying "Please take a photo of your face," and the user taps the shutter button to take a photo of their face. When the user presses the "Save" button, the image is encrypted and sent to the server.

[0642] Input: User's face image

[0643] Output: Sending encrypted facial image data to the server

[0644] Step 3:

[0645] The server stores the received facial image data and inputs it into the analysis engine. The input is encrypted facial image data. Based on this, the system analyzes the user's current facial features and generates an ideal face model for the scene. Specifically, it uses OpenCV and TensorFlow to analyze facial features and generates the ideal face using AI algorithms.

[0646] Input: Encrypted facial image data

[0647] Output: User's facial feature analysis results, ideal face model

[0648] Step 4:

[0649] The server compares the generated ideal face with the user's current face and analyzes the gap. The inputs are the user's current facial features and the ideal face model. Based on these differences, the server generates specific makeup steps to help the user achieve their ideal face. Specifically, the AI ​​algorithm calculates the optimal makeup steps (e.g., applying a thin layer of eyeshadow around the eyes) to close the gap.

[0650] Input: Current facial features, ideal face model

[0651] Output: Makeup steps to help the user achieve their ideal face

[0652] Step 5:

[0653] The generated makeup instructions are sent to the device, where the user can review them. The input is the makeup instructions. The generated makeup instructions are created using a generative AI model (e.g., GPT-4). Specifically, the server inputs the prompt "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products." into the generative AI model, and then sends the results to the device.

[0654] Input: Generated Make steps

[0655] Output: Display the make steps on the terminal for the user to review.

[0656] Step 6:

[0657] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. The input is the user's skin condition data. The server recommends specific skin care products and cosmetics based on the skin analysis results. For example, information such as "Use this moisturizing cream for dry skin" is sent to the device.

[0658] Input: User's skin condition data

[0659] Output: Recommended skincare products and cosmetic information

[0660] Step 7:

[0661] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can apply the suggested makeup steps and check the results. The input is a real-time image of the face. Specifically, when the user taps the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed.

[0662] Input: Real-time facial video

[0663] Output: Facial image projected on a real-time mirror

[0664] Step 8:

[0665] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The input is the user's feedback data. The server analyzes the received feedback data and uses it to evolve the AI ​​model. Specifically, the user enters their satisfaction after applying the makeup into the device, and the data is analyzed and used to improve future makeup suggestions.

[0666] Input: User feedback data

[0667] Output: Analyzed feedback data, evolved AI model

[0668] (Application example 1)

[0669] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] Compared to face-to-face services, makeup advice in virtual stores often falls short, making it difficult for users to achieve professional makeup in the comfort of their own homes. It is also often difficult for users to determine effective makeup techniques and appropriate skincare products on their own. Furthermore, existing systems lack the ability to check the effects of makeup in real time or to use feedback to improve AI models. Therefore, there is a need for a system that can provide highly accurate makeup advice in virtual stores and improve user satisfaction.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0672] In this invention, the server includes: a means for allowing a user to select a specific scene; a means for capturing a facial image of the user and receiving the image data; a means for analyzing the received facial image data and generating an ideal face; a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; a means for recommending skin care products and cosmetics based on the user's skin condition; a means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; a means for receiving and analyzing feedback from the user; a means for evolving an AI model using the analyzed feedback data; a means for providing makeup advice in a virtual store; and a means for confirming and executing the suggested makeup routine in real time. This allows users to receive professional makeup advice from the comfort of their own home and select effective makeup techniques and skin care products. Furthermore, the ability to check the makeup effects in real time significantly increases user satisfaction.

[0673] "User" refers to an individual who uses this system to receive makeup advice tailored to a specific scene.

[0674] A "specific scene" refers to a specific situation or occasion that the user desires, such as a remote meeting, a party event, or everyday outings.

[0675] "Facial image" refers to image data of a user's face that the user provides to the system.

[0676] "Means for receiving" refers to a method or device for the system to import facial image data and other information provided by the user into a server or the like.

[0677] "Means for analyzing" includes techniques and algorithms for analyzing a user's facial features based on facial image data received by the system.

[0678] An "ideal face" refers to a model that generates the optimal facial condition and appearance for a specific scene desired by the user.

[0679] "Difference" refers to the difference that exists between the current user's facial features and the generated ideal facial features.

[0680] The "makeup procedure" refers to the specific makeup method or procedure that the user follows to achieve an ideal face.

[0681] "Skin Care Products" refers to cosmetics and related products for skin care that are recommended based on the user's skin condition.

[0682] "Cosmetics" refers to all cosmetics that users use for makeup.

[0683] The "mirror function" refers to a function that allows users to project their own face on the screen in real time and check the makeup steps as they are performed.

[0684] "Feedback" refers to information about the user's experience and satisfaction with the system after using it.

[0685] "Means for checking in real time" refers to a method or device that allows a user to check the results of a proposed makeup routine in real time while performing the routine.

[0686] "Virtual Store" means an online makeup advice and skin care product recommendation service provided via the Internet.

[0687] An embodiment of the present invention is a system for a virtual store where users can receive professional makeup advice from the comfort of their own home. The system provides makeup procedures and appropriate skin care products tailored to specific scenes, and users can check and implement the effects in real time.

[0688] Hardware and software used

[0689] Hardware: Terminal devices such as smartphones, PCs, and cameras

[0690] Software: Python, Flask, OpenCV, Requests, Heroku, AWS (EC2, S3)

[0691] Overall system flow

[0692] 1. Scene selection:

[0693] The user launches the application and selects a specific scene, such as a remote meeting, a party event, or a daily outing, and this information is sent from the device to the server.

[0694] 2. Facial image capture:

[0695] When the user selects a scene, the device activates the camera function and prompts the user to take a face image. The user uses the camera to take a picture of their face, and the image is saved.

[0696] 3. Receipt and analysis of image data:

[0697] The captured facial image data is encrypted and sent from the device to a server. The server's analysis engine receives this data and analyzes the user's facial features. It also runs an AI algorithm to generate an ideal face model for each scene, generating the ideal face.

[0698] 4. Differential analysis:

[0699] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0700] 5. Makeup procedure and skin care product suggestions:

[0701] Based on the analysis results, the server generates specific makeup procedures for the eyes, skin, lips, etc. and sends them to the device. It also recommends appropriate skin care products and cosmetics based on the user's skin condition. This information can be viewed on the device.

[0702] 6. Real-time mirror function:

[0703] The device uses the front camera to display a real-time image of the user's face, allowing the user to see the results of the proposed makeup application in real time.

[0704] 7. Feedback Receipt and Analysis:

[0705] After the makeup application is complete, the device displays a questionnaire screen to the user about their experience and satisfaction, and collects their feedback. This data is sent to and stored on a server and used to improve the AI ​​model.

[0706] Examples and prompts

[0707] Specific examples

[0708] When a user wants to join a remote meeting, they select "Remote Meeting" from the scene selection menu and take a photo of their face. The server then suggests a professional look suitable for the remote meeting and applies makeup while checking it in real time. During this process, the user can purchase the suggested skin care products and provide feedback to improve the system.

[0709] Prompt Sentence Examples

[0710] "Analyze the facial features in this image and suggest the best makeup routine for a remote meeting."

[0711] As described above, this system enables users to receive highly accurate makeup advice and check and apply it in real time, significantly improving satisfaction in the virtual store.

[0712] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0713] Step 1:

[0714] The user launches the application and a scene selection screen appears. The user selects a specific scene, such as a remote meeting, a party event, or a daily outing. This scene information is sent from the device to the server, which then obtains input to determine the next step based on the user's selection.

[0715] Step 2:

[0716] When the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved within the app. Once the face image is saved, the device encrypts the data before sending it to the server.

[0717] Step 3:

[0718] The server receives the encrypted facial image data, decrypts it, and inputs it into the analysis engine. The analysis engine extracts features from the image and analyzes the user's current facial features. Specifically, it uses AI algorithms to analyze the facial contours, position of the eyes, nose, and mouth, as well as the condition of the skin.

[0719] Step 4:

[0720] The analysis engine runs an AI algorithm to generate an ideal face model for each scene. The algorithm uses the prompt "Please analyze the facial features in this image and suggest the best makeup routine for a remote meeting" as input. The server then obtains the ideal face model as output.

[0721] Step 5:

[0722] The server analyzes the differences between the user's current face and the generated ideal face. This process compares the facial contours, the relative positions of each feature, and color tones to identify any gaps. Based on the results of this analysis, the server generates specific makeup instructions to help the user achieve their ideal face.

[0723] Step 6:

[0724] The server then recommends appropriate skincare products and cosmetics based on the user's skin condition along with the generated makeup instructions. This information is then sent to the device, where the user can view the specific makeup instructions and recommended skincare products.

[0725] Step 7:

[0726] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the real-time mirror function to check the results of the proposed makeup steps as they are applied.

[0727] Step 8:

[0728] After the makeup application is complete, the device displays a survey screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0729] This allows makeup advice to be given in the virtual store with high accuracy, realizing a system that improves user satisfaction.

[0730] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0731] The present invention is a system that allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between that and the user's current face, and suggests specific makeup routines, skin care products, and cosmetics. The present invention also incorporates an emotion engine that recognizes the user's emotions, and has the function of analyzing the user's real-time emotions to adjust the makeup routine and skin care product recommendations, allowing users to learn and apply makeup routines that suit them and see the results in real time.

[0732] The system consists of the following elements:

[0733] Scene selection method

[0734] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0735] Facial image capturing means

[0736] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[0737] Facial image receiving means

[0738] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[0739] Image analysis methods

[0740] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[0741] Differential analysis means

[0742] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[0743] Makeup procedure generation means

[0744] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[0745] Skincare product recommendation tool and cosmetic recommendation tool

[0746] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[0747] Real-time mirror function

[0748] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[0749] Emotion Engine

[0750] The device uses an emotion engine to analyze the user's real-time emotions based on facial video and voice data, and the analysis results are sent to a server where they are used to adjust makeup routines and skin care product recommendations.

[0751] Feedback receiving means and feedback analyzing means

[0752] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[0753] Specific examples

[0754] A user wants to put on makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup instructions (for example, "shape your eyebrows" or "use a light beige foundation") are generated and sent to the device. The user can check the instructions on the device and use the real-time mirror function to apply their makeup.

[0755] Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides feedback and adjustments accordingly. After the makeup is complete, the user can send feedback, and the server will use that data to improve the AI ​​model and suggest more effective makeup procedures for the next time.

[0756] As described above, by providing detailed support that also takes the user's emotions into consideration, users can easily achieve their ideal appearance.

[0757] The processing flow will be explained below.

[0758] Step 1:

[0759] The user launches an application.

[0760] Action: The device launches the application and displays the scene selection screen.

[0761] Step 2:

[0762] The user selects a scene.

[0763] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[0764] Step 3:

[0765] The device activates the camera function.

[0766] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[0767] Step 4:

[0768] The user takes a picture of their face and saves the image.

[0769] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[0770] Step 5:

[0771] The terminal transmits the facial image data to the server.

[0772] Operation: The stored facial image data is encrypted and securely sent to the server.

[0773] Step 6:

[0774] The server receives the facial image data.

[0775] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[0776] Step 7:

[0777] The server analyzes the facial image and generates the ideal face.

[0778] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[0779] Step 8:

[0780] The server analyzes the differences between the ideal face and the current face.

[0781] How it works: Compare the generated ideal face with your current face and analyze the gap.

[0782] Step 9:

[0783] The server generates the make procedure.

[0784] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[0785] Step 10:

[0786] The server sends the make procedure to the terminal.

[0787] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[0788] Step 11:

[0789] The device will display the makeup instructions.

[0790] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[0791] Step 12:

[0792] The server evaluates the user's skin condition.

[0793] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[0794] Step 13:

[0795] Servers recommend skin care products and cosmetics.

[0796] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[0797] Step 14:

[0798] The device will activate the real-time mirror function.

[0799] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[0800] Step 15:

[0801] The user performs the make procedure.

[0802] Action: The user performs the suggested makeup steps while using a real-time mirror.

[0803] Step 16:

[0804] The emotion engine analyzes the user's emotions.

[0805] How it works: The device collects the user's facial expressions and voice data and uses an emotion engine to analyze emotions in real time.

[0806] Step 17:

[0807] The server receives and adjusts the emotion data.

[0808] How it works: The server adjusts makeup routines and skin care product recommendations based on the emotion data it receives.

[0809] Step 18:

[0810] The device will display a feedback screen.

[0811] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[0812] Step 19:

[0813] The user submits feedback.

[0814] Action: The user enters feedback and presses the submit button.

[0815] Step 20:

[0816] The terminal transmits the feedback data to the server.

[0817] Action: Feedback data is encrypted and sent to the server.

[0818] Step 21:

[0819] The server analyzes the feedback data.

[0820] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[0821] The above are the detailed processing steps for combining the emotion engine with the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skincare routine, allowing them to approach remote meetings and daily life with confidence.

[0822] Example 2

[0823] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0824] Conventional makeup instruction systems are unable to consider users' emotions or real-time reactions, and only provide general advice on specific makeup steps and skin care product recommendations. This makes it difficult for users to receive specific and effective instruction to achieve their ideal appearance.

[0825] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for the user to select a specific scene; means for capturing a facial image of the user and receiving the image data; means for analyzing the received facial image data to generate an ideal face; means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; means for recommending skin care products and cosmetics based on the user's skin condition; means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; means for analyzing real-time emotions based on the user's facial image and voice and adjusting the makeup routine and skin care product recommendations; means for receiving and analyzing feedback from the user; and means for evolving the generative AI model using the analyzed feedback data. This enables detailed makeup instruction tailored to the user's emotions and real-time situation, and recommendations of skin care products and cosmetics tailored to the user's individual needs.

[0826] "Scene selection means" refers to a means by which a user selects a particular scene.

[0827] The "face image capturing means" refers to a means for capturing a face image of the user.

[0828] The "image data receiving means" refers to a means for receiving face image data sent from a terminal on the server side.

[0829] The "ideal face generating means" refers to a means for generating an ideal face based on received face image data.

[0830] "Difference analysis means" refers to means for analyzing the difference between an ideal face and a current face.

[0831] The "makeup procedure generation means" refers to a means for generating a specific makeup procedure based on the difference analysis results.

[0832] "Recommendation means" refers to a means for recommending skin care products and cosmetics based on the user's skin condition.

[0833] The "mirror function providing means" refers to a means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures.

[0834] "Emotion analysis means" refers to a means for analyzing real-time emotions based on facial image and voice data of a user.

[0835] "Feedback receiving means" refers to a means for receiving feedback from a user.

[0836] "Feedback analysis means" refers to means for analyzing received feedback data.

[0837] "Generative AI model" refers to an artificial intelligence model that generates ideal faces and provides makeup instructions.

[0838] This system allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between the current face and the ideal face, and provides specific makeup procedures, skin care products, and cosmetics. Furthermore, this system has the function of analyzing the user's real-time emotions and adjusting the makeup procedures and skin care product recommendations.

[0839] The system consists of the following components:

[0840] Hardware and Software Configuration

[0841] User device: A smartphone or tablet with a built-in camera, display, and microphone.

[0842] Server: A high-performance computer server used to analyze facial image data and emotional data, and generate ideal facial and makeup procedures.

[0843] Software: A software stack including image processing libraries such as OpenCV and Dlib, emotion recognition APIs, and generative AI models.

[0844] Specific processing explanation

[0845] 1. Application launch and scene selection

[0846] On the device: When the user launches the application, a scene selection screen appears. The user selects a scene, such as a remote meeting, work, a date, or a casual outing. This information is sent to the server.

[0847] 2. Taking a facial image

[0848] Device: When the user selects a scene, the camera function is activated and the user is prompted to take a facial image. The user uses the camera to take a facial image, then reviews and saves the image.

[0849] 3. Sending and receiving facial image data

[0850] Terminal: The captured facial image data is encrypted for security purposes and sent to the server.

[0851] Server: Decodes the received facial image data and stores it in a database.

[0852] 4. Facial feature analysis and ideal face model generation

[0853] Server: Based on the received facial image data, it uses image processing libraries such as OpenCV and Dlib to analyze the user's current facial features, and then uses generative AI models to generate an ideal face model for the scene.

[0854] 5. Difference analysis and generation of specific make-up steps

[0855] Server: Compares the difference between the ideal face and the current face and analyzes the gap. Based on the analyzed data, it generates specific makeup instructions to help the user achieve their ideal face.

[0856] 6. Skin care and cosmetic product recommendations

[0857] Server: Based on facial image analysis, evaluates the user's skin condition and recommends the most suitable skincare products and cosmetics. This information is also sent to the device and displayed to the user.

[0858] 7. Executing the real-time mirror function

[0859] Device: Uses the front camera to display a real-time image of the user's face, allowing the user to see the effects of the suggested makeup steps as they are performed.

[0860] 8. Emotional analysis and feedback regulation

[0861] Device: Analyzes facial video and audio data using an emotion analysis API, and transmits the user's real-time emotions to the server.

[0862] Server: Based on the analysis results, it adjusts makeup routines and skin care product recommendations in real time.

[0863] 9. Feedback Collection and Analysis

[0864] Device: After the makeup application is completed, the user is asked to fill out a survey about their experience and satisfaction.

[0865] User: Enter and submit feedback.

[0866] Server: Stores and analyzes the received feedback data, evolves the generative AI model based on that data, and reflects it the next time it is used.

[0867] Specific examples

[0868] A user wants to apply makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for "Remote Meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "Shape your eyebrows" or "Use a light beige foundation") are generated and sent to the device. The user checks the steps on the device and uses the real-time mirror function to apply the makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides appropriate feedback and adjustments. After completing the makeup, the user submits feedback, and the server uses that data to evolve the generative AI model and suggest a more customized makeup step the next time they use it.

[0869] Prompt Sentence Examples

[0870] "I would like the AI ​​to automate the process of comparing my current face with the ideal face for remote meeting situations and suggesting specific makeup steps. I would also like the facial image data to be encrypted before transmission, and appropriate skin care products and cosmetics to be recommended based on the analysis results."

[0871] The above is a detailed description of the preferred embodiment of the invention, which allows users to easily achieve their ideal appearance.

[0872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0873] Step 1: Launch the application and select a scene

[0874] On the device: When the user launches the application, a scene selection screen appears, displaying multiple options (e.g., remote meeting, work, date, everyday outing, etc.) as input.

[0875] User: Selects the desired scene and enters the information into the device.

[0876] Terminal: Sends the user's selection to the server. The input data to the server is the scene information selected by the user. The output data is the scene information sent to the server.

[0877] Step 2: Capture a facial image

[0878] Terminal: When a scene is selected, the terminal activates the camera function and prompts the user to take a facial image. The input at this stage is the scene information that triggers the camera activation.

[0879] User: Take a picture of your face with the camera and view and save the image within the app.

[0880] Terminal: The captured face image data is saved. The output data is the captured face image data.

[0881] Step 3: Encrypt and transmit facial image data

[0882] Terminal: Encrypts the stored facial image data to ensure security. The input data is facial image data, which is then encrypted.

[0883] Terminal: Sends the encrypted facial image data to the server. The encrypted facial image data is sent to the server and becomes the output data.

[0884] Step 4: Receiving and storing facial image data

[0885] Server: Receives the encrypted facial image data and decrypts it. The input data is the encrypted facial image data.

[0886] Server: Saves the decoded facial image data in the database. The decoded facial image data is saved as output.

[0887] Step 5: Facial feature analysis and ideal face model generation

[0888] Server: Analyzes the user's current facial features based on the received facial image data. This process uses image processing libraries such as OpenCV and Dlib. The input data is the decoded facial image data.

[0889] Server: Uses a generative AI model to generate an ideal face model based on scene information. The input data is the decoded face image data and scene information, and the output data is the generated ideal face model.

[0890] Step 6: Difference analysis and generation of specific make procedures

[0891] Server: Compares and analyzes the differences between the ideal face and the current face. A facial landmark detection algorithm is used for the analysis. The input data is the current facial features and the ideal face model.

[0892] Server: Generates specific make procedures based on the difference analysis results. The generated make procedures become the output data.

[0893] Step 7: Skincare and cosmetic product recommendations

[0894] Server: Evaluates the user's skin condition based on the results of facial image analysis and recommends appropriate skin care products and cosmetics. The input data is the results of facial image analysis.

[0895] Server: Sends the recommendation information to the terminal and displays it to the user. The information on the recommended skin care products and cosmetics is the output data.

[0896] Step 8: Run the real-time mirror function

[0897] Terminal: Uses the front camera to display the user's face image in real time. The camera is activated when the user performs the makeup procedure. The input data is the trigger to start the makeup procedure.

[0898] User: Perform the proposed makeup steps and check their effectiveness.

[0899] Terminal: Real-time facial image is displayed to the user, which becomes the output data.

[0900] Step 9: Emotional analysis and feedback adjustment

[0901] Terminal: Analyzes real-time emotions using an emotion analysis API based on the user's facial video and audio data. The input data is real-time facial video and audio data.

[0902] Server: Receives the analysis results and adjusts makeup routines and skin care product recommendations in real time. The output data is the adjusted makeup routine and skin care product information.

[0903] Step 10: Collect and analyze feedback

[0904] Terminal: After the user has finished applying makeup, a questionnaire about the user's experience and satisfaction is displayed. The input data is the trigger for finishing the makeup.

[0905] User: Enter and submit feedback.

[0906] Server: Stores and analyzes the received feedback data. The analysis results are used to evolve the generative AI model and are reflected the next time it is used. The output data is the evolved generative AI model.

[0907] (Application example 2)

[0908] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0909] Conventional makeup and skincare recommendation systems are not optimized for the user's emotions or specific situations, limiting their ability to improve the user experience. Furthermore, users often lack real-time feedback and adjustments, preventing them from accurately executing the suggested steps. Furthermore, when using smart glasses or other devices, their interfaces are often not optimized, resulting in a lack of user convenience.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0911] In this invention, the server includes a means for capturing a user's facial image and receiving the image data, a means for analyzing the received facial image data and generating an ideal face, a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, a means for analyzing the user's emotions and adjusting the recommendations, an interface displayed on the smart glasses, a means for receiving and analyzing feedback from the user, and a means for evolving the AI ​​model using the analyzed feedback data. This enables recommendations of optimal makeup routines and products according to the user's specific emotions and situations. Furthermore, real-time adjustments and feedback improve the user experience and enable intuitive operation using the smart glasses.

[0912] The "means for the user to select a specific scene" is an operation interface that allows the user to select a desired scene according to a specific situation such as a remote meeting, a date, or a party.

[0913] "Means for capturing an image of a user's face and receiving the image data" refers to a device or system for capturing an image of a user's face using smart glasses or a camera device and receiving the data to an application or server.

[0914] The "means for analyzing the received facial image data and generating an ideal face" refers to an AI algorithm and a system for executing the algorithm, which uses the received facial image data to generate an ideal face suitable for a specific scene.

[0915] The "means for analyzing the difference between the ideal face and the current face and generating specific makeup procedures" is a system for comparing and analyzing the gap between the current face and the generated ideal face, and obtaining specific makeup procedures to fill the gap.

[0916] The "means for recommending skin care products and cosmetics based on the user's skin condition" is a system for analyzing the user's skin condition and suggesting optimal skin care products and cosmetics.

[0917] "Means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures" refers to a function that displays a real-time image of the user's face through the display of smart glasses or a device, allowing the user to check the procedure while applying makeup.

[0918] The "means for analyzing the user's emotions and adjusting the recommended content" is a function that analyzes the user's emotions from their facial expressions and voice, and dynamically adjusts the makeup steps and recommended content based on that.

[0919] The "interface means displayed on the smart glasses" refers to a user interface for displaying information such as makeup procedures, recommended products, and real-time feedback on the display of the smart glasses.

[0920] The "means for receiving and analyzing feedback from users" is a system for collecting feedback from users regarding their experience and satisfaction with the product after they have finished applying their makeup, and analyzing that data.

[0921] The "means for evolving the AI ​​model using the analyzed feedback data" refers to a system that improves and evolves the AI ​​model based on feedback data collected from users, enabling it to suggest more appropriate makeup procedures and products the next time the model is used.

[0922] This invention is a system that generates an ideal face for a specific scene, analyzes the gap between the ideal face and the current face, and suggests makeup routines and skin care products. It also improves the user experience by dynamically adjusting the recommendations using emotion analysis. A detailed description of a system for implementing this invention is provided below.

[0923] The system includes means for a user to select a specific scene, means for taking a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating specific makeup steps, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for performing makeup steps, means for analyzing the user's emotions and adjusting the recommended content, interface means displayed on the smart glasses, means for receiving and analyzing feedback from the user, and means for evolving the AI ​​model using the analyzed feedback data.

[0924] The server runs an AI algorithm to generate an ideal face based on the scene selected by the user. For example, for a remote meeting, an algorithm is used to generate a natural and professional appearance. The user's facial image is captured using the camera in the smart glasses, and the image data is encrypted and sent to the server. The server analyzes the received facial image data and extracts the current facial features.

[0925] The server then analyzes the differences between the ideal face and the current face and generates specific makeup steps to fill the gap. Based on the user's skin condition, skincare products and cosmetics are also suggested. This information is displayed in real time on the smart glasses' display, allowing the user to apply makeup while viewing it. Additionally, a real-time mirror function allows users to check their own face on the display as they apply their makeup.

[0926] An emotion engine using facial expression and voice recognition is used to analyze user emotions. This emotion engine analyzes the user's emotional state in real time and adjusts recommendations based on the results. For example, if the user appears dissatisfied, it will suggest alternative products or procedures to improve satisfaction.

[0927] After the makeup application is complete, the system receives feedback from the user and analyzes the data. This feedback data is used to improve the AI ​​model, allowing it to make more appropriate suggestions the next time the product is used.

[0928] As a specific example, a user wants to apply makeup for a remote meeting and launches the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face using the smart glasses' camera. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "shape your eyebrows" or "use a light beige foundation") are generated and displayed on the smart glasses' display. The user checks these as they apply their makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with their makeup, it provides appropriate feedback and adjustments. After completing their makeup, the user submits feedback, and the server uses that data to evolve the AI ​​model and suggest more effective makeup steps the next time they use the app.

[0929] An example of a prompt is, "The user has selected makeup for a remote meeting. Generate an ideal face based on the current facial image. Next, analyze the gap between the current face and the ideal face and suggest specific makeup steps to fill the gap (e.g., shaping the eyebrows, using a light beige foundation, etc.). Furthermore, analyze the user's emotions from their facial expressions and voice and consider how to optimize the makeup steps."

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The user launches the application using the smart glasses and selects the desired scene (e.g., remote meeting, date, party) on the scene selection screen.

[0933] Input: User scene selection (e.g. "Remote Meeting")

[0934] Output: Selected scene information

[0935] Step 2:

[0936] The terminal transmits the scene information selected by the user to the server, and then prompts the user to take a facial image using the camera built into the smart glasses.

[0937] Input: User operation (taking a face image after selecting a scene)

[0938] Output: Photographed face image

[0939] Step 3:

[0940] The facial image data captured by the terminal is encrypted and sent to the server.

[0941] Input: Photographed face image data

[0942] Output: Encrypted facial image data

[0943] Step 4:

[0944] The server analyzes the received facial image data and extracts the current facial features.

[0945] Input: Encrypted facial image data

[0946] Output: Current facial feature data

[0947] Step 5:

[0948] The server runs an AI algorithm to generate an ideal face for the selected scene.

[0949] Input: User's scene information and current facial feature data

[0950] Output: Ideal face data

[0951] Step 6:

[0952] The server compares the ideal face data with the current face data and analyzes the differences.

[0953] Input: Current face data and ideal face data

[0954] Output: Differential data

[0955] Step 7:

[0956] The server generates specific makeup procedures based on the differential data and recommends skin care products and cosmetics.

[0957] Input: differential data

[0958] Output: A list of specific makeup steps and recommended products

[0959] Step 8:

[0960] The device receives makeup instructions and recommended product information from the server and displays them on the smart glasses display in real time. The user applies makeup while viewing this information.

[0961] Input: List of makeup steps and recommended products

[0962] Output: Makeup instructions and recommended product information displayed on smart glasses

[0963] Step 9:

[0964] The device's front camera is used to display a real-time image of the user's face while checking the makeup procedure.

[0965] Input: Real-time facial video

[0966] Output: Real-time facial image displayed on the display

[0967] Step 10:

[0968] The server's emotion engine analyzes the user's facial expressions and voice, assesses the user's emotional state in real time, and adjusts recommendations as needed.

[0969] Input: Real-time facial video and audio data

[0970] Output: Tailored recommendations and feedback

[0971] Step 11:

[0972] After the makeup application is completed, the device displays a feedback questionnaire to the user to collect data on satisfaction and usability.

[0973] Input: Feedback data from users

[0974] Output: Feedback data stored on the device

[0975] Step 12:

[0976] The server improves and evolves the AI ​​model based on the feedback data it receives.

[0977] Input: Feedback data

[0978] Output: An improved AI model

[0979] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0980] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0981] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0982] [Third embodiment]

[0983] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0984] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0985] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0986] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0987] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0988] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0989] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0990] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0991] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0992] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0993] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0994] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0995] The system allows users to select a specific scene, such as a remote meeting or a daily outing, generate an ideal face for that scene, analyze the gap between that and their current face, and suggest specific makeup steps and appropriate skincare products and cosmetics. Using this system, users can learn makeup techniques that suit them, select skincare products according to their skin condition, and check the effects in real time.

[0996] The system consists of the following elements:

[0997] Scene selection method

[0998] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[0999] Facial image capturing means

[1000] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[1001] Facial image receiving means

[1002] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[1003] Image analysis methods

[1004] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[1005] Differential analysis means

[1006] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1007] Makeup procedure generation means

[1008] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[1009] Skincare product recommendation tool and cosmetic recommendation tool

[1010] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[1011] Real-time mirror function

[1012] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[1013] Feedback receiving means and feedback analyzing means

[1014] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1015] By combining these elements, users can easily learn and apply the appropriate makeup and skincare products. Real-time confirmation is also possible, resulting in higher satisfaction. Specifically, the system can suggest makeup steps and skincare products to create a professional look suitable for remote meetings, and the suggestions can be confirmed in real time while being applied.

[1016] The processing flow will be explained below.

[1017] Step 1:

[1018] The user launches an application.

[1019] Action: The device launches the application and displays the scene selection screen.

[1020] Step 2:

[1021] The user selects a scene.

[1022] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[1023] Step 3:

[1024] The device activates the camera function.

[1025] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[1026] Step 4:

[1027] The user takes a picture of their face and saves the image.

[1028] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[1029] Step 5:

[1030] The terminal transmits the facial image data to the server.

[1031] Operation: The stored facial image data is encrypted and securely sent to the server.

[1032] Step 6:

[1033] The server receives the facial image data.

[1034] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[1035] Step 7:

[1036] The server analyzes the facial image and generates the ideal face.

[1037] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[1038] Step 8:

[1039] The server analyzes the differences between the ideal face and the current face.

[1040] How it works: Compare the generated ideal face with your current face and analyze the gap.

[1041] Step 9:

[1042] The server generates the make procedure.

[1043] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[1044] Step 10:

[1045] The server sends the make procedure to the terminal.

[1046] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[1047] Step 11:

[1048] The device will display the makeup instructions.

[1049] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[1050] Step 12:

[1051] The server evaluates the user's skin condition.

[1052] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[1053] Step 13:

[1054] Servers recommend skin care products and cosmetics.

[1055] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[1056] Step 14:

[1057] The device will activate the real-time mirror function.

[1058] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[1059] Step 15:

[1060] The user performs the make procedure.

[1061] Action: The user performs the suggested makeup steps while using a real-time mirror.

[1062] Step 16:

[1063] The device will display a feedback screen.

[1064] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[1065] Step 17:

[1066] The user submits feedback.

[1067] Action: The user enters feedback and presses the submit button.

[1068] Step 18:

[1069] The terminal transmits the feedback data to the server.

[1070] Action: Feedback data is encrypted and sent to the server.

[1071] Step 19:

[1072] The server analyzes the feedback data.

[1073] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[1074] The above are the detailed processing steps of the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skin care, allowing them to approach remote meetings and daily life with confidence.

[1075] Example 1

[1076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] In modern society, there is a demand for instantly preparing the perfect look for various situations, such as remote meetings or outings. However, it is not easy for individual users to know what kind of makeup will bring them closer to their ideal face. It is also difficult to select the appropriate skincare products and cosmetics depending on their skin condition. Furthermore, there is no way to check the effects of makeup in real time as they are applied. A system that can solve these issues is needed.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1079] In this invention, the server includes means for allowing a user to select a specific scene, means for capturing a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for executing the makeup routine, means for receiving and analyzing feedback from the user, and means for evolving an artificial intelligence model using the analyzed feedback data. This allows users to learn makeup techniques that suit them, select skin care products according to their skin condition, and proceed while checking the effects in real time.

[1080] "Specific scenes" refer to situations where users need to dress appropriately for the occasion, such as remote meetings or everyday outings.

[1081] "User's face image" refers to photographic data of the user's face taken using the camera of the terminal.

[1082] "Image data" refers to digital data that includes a facial image captured by a user.

[1083] An "ideal face" refers to a facial image that models the user's appearance that is considered optimal for a particular selected scene.

[1084] "Difference" refers to the result of calculating the difference between the current facial image and the ideal facial image.

[1085] "Makeup procedure" refers to the specific makeup method that a user should follow to get closer to their ideal face.

[1086] "Skin condition" refers to an evaluation including the dryness, oil content, health, etc. of the user's skin.

[1087] "Skin care products" refers to beauty products such as creams and lotions used to maintain and improve the health of the skin.

[1088] "Cosmetics" refers to all cosmetics used in makeup (e.g., eye shadow, foundation).

[1089] The "mirror function" refers to a function that uses the device's camera to display the user's face in real time, allowing them to check the progress of their makeup.

[1090] "Feedback" refers to information entered by users about their satisfaction and opinions after using the product.

[1091] An "artificial intelligence model" refers to an algorithm that evolves based on data about the user's makeup and skincare.

[1092] This system allows users to select a specific scene, generate an ideal face for that scene, analyze the gap between the ideal face and their current face, and suggest specific makeup procedures and appropriate skin care products and cosmetics. By using this system, users can learn makeup techniques that suit them, select skin care products according to their skin condition, and check the effects in real time.

[1093] The system consists of the following elements:

[1094] Scene selection method

[1095] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[1096] Facial image capturing means

[1097] After the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved in the app. The captured facial image data is encrypted and sent from the device to the server.

[1098] Facial image receiving means

[1099] The server stores the received facial images and inputs them into the analysis engine.

[1100] Image analysis methods

[1101] The server uses OpenCV and deep learning models (e.g., TensorFlow) to analyze the user's current facial features based on the received facial image data. It also runs AI algorithms to generate an ideal face model for each scene, generating the ideal face. For example, if the user selects "Remote Meeting," an ideal face with a professional appearance will be generated.

[1102] Differential analysis means

[1103] The server compares the generated ideal face with the current face and analyzes the gap. An AI algorithm calculates the difference and generates specific makeup steps to close the gap, such as how to apply eyeshadow or choose foundation.

[1104] Makeup procedure generation means

[1105] The server generates specific makeup instructions based on the analysis results. The generated makeup instructions are sent to the device, where the user can review them. The specific makeup instructions are created using a generative AI model (e.g., GPT-4). An example of a prompt to input to the generative AI model is, "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products."

[1106] Skincare product recommendation tool and cosmetic recommendation tool

[1107] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. For example, information such as "Use this moisturizing cream for dry skin" is sent to the terminal, allowing the user to select the optimal skin care and cosmetics.

[1108] Real-time mirror function

[1109] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can perform the suggested makeup steps and check the results. By tapping the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed in real time.

[1110] Feedback receiving means and feedback analyzing means

[1111] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The server analyzes the received feedback data and uses it to evolve the AI ​​model. The user enters their satisfaction after applying the makeup, and the data is analyzed and used to improve future makeup suggestions.

[1112] By combining these elements, users can easily learn the proper makeup and skincare products to create a look that suits the real-life situation, and the ability to see the results in real time provides a high level of satisfaction.

[1113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1114] Step 1:

[1115] The user launches the app and a scene selection screen appears. The user taps to select the desired scene from options such as remote meeting or everyday outing. The input includes the user's scene selection information. This information is sent from the device to the server. Specifically, the user taps the app icon to launch the app, and then taps the "Remote Meeting" button from the multiple scene selection buttons displayed on the home screen.

[1116] Input: User selected scene information

[1117] Output: Transfer of scene information from the device to the server

[1118] Step 2:

[1119] After the user selects a scene, the device launches the camera function and prompts the user to take a photo of their face. The user uses the camera to take a photo of their face and then reviews and saves the image within the app. The input is the user's face image. The captured face image data is encrypted and sent from the device to the server. Specifically, the device automatically launches the camera app, displays a message saying "Please take a photo of your face," and the user taps the shutter button to take a photo of their face. When the user presses the "Save" button, the image is encrypted and sent to the server.

[1120] Input: User's face image

[1121] Output: Sending encrypted facial image data to the server

[1122] Step 3:

[1123] The server stores the received facial image data and inputs it into the analysis engine. The input is encrypted facial image data. Based on this, the system analyzes the user's current facial features and generates an ideal face model for the scene. Specifically, it uses OpenCV and TensorFlow to analyze facial features and generates the ideal face using AI algorithms.

[1124] Input: Encrypted facial image data

[1125] Output: User's facial feature analysis results, ideal face model

[1126] Step 4:

[1127] The server compares the generated ideal face with the user's current face and analyzes the gap. The inputs are the user's current facial features and the ideal face model. Based on these differences, the server generates specific makeup steps to help the user achieve their ideal face. Specifically, the AI ​​algorithm calculates the optimal makeup steps (e.g., applying a thin layer of eyeshadow around the eyes) to close the gap.

[1128] Input: Current facial features, ideal face model

[1129] Output: Makeup steps to help the user achieve their ideal face

[1130] Step 5:

[1131] The generated makeup instructions are sent to the device, where the user can review them. The input is the makeup instructions. The generated makeup instructions are created using a generative AI model (e.g., GPT-4). Specifically, the server inputs the prompt "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products." into the generative AI model, and then sends the results to the device.

[1132] Input: Generated Make steps

[1133] Output: Display the make steps on the terminal for the user to review.

[1134] Step 6:

[1135] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. The input is the user's skin condition data. The server recommends specific skin care products and cosmetics based on the skin analysis results. For example, information such as "Use this moisturizing cream for dry skin" is sent to the device.

[1136] Input: User's skin condition data

[1137] Output: Recommended skincare products and cosmetic information

[1138] Step 7:

[1139] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can apply the suggested makeup steps and check the results. The input is a real-time image of the face. Specifically, when the user taps the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed.

[1140] Input: Real-time facial video

[1141] Output: Facial image projected on a real-time mirror

[1142] Step 8:

[1143] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The input is the user's feedback data. The server analyzes the received feedback data and uses it to evolve the AI ​​model. Specifically, the user enters their satisfaction after applying the makeup into the device, and the data is analyzed and used to improve future makeup suggestions.

[1144] Input: User feedback data

[1145] Output: Analyzed feedback data, evolved AI model

[1146] (Application example 1)

[1147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] Compared to face-to-face services, makeup advice in virtual stores often falls short, making it difficult for users to achieve professional makeup in the comfort of their own homes. It is also often difficult for users to determine effective makeup techniques and appropriate skincare products on their own. Furthermore, existing systems lack the ability to check the effects of makeup in real time or to use feedback to improve AI models. Therefore, there is a need for a system that can provide highly accurate makeup advice in virtual stores and improve user satisfaction.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1150] In this invention, the server includes: a means for allowing a user to select a specific scene; a means for capturing a facial image of the user and receiving the image data; a means for analyzing the received facial image data and generating an ideal face; a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; a means for recommending skin care products and cosmetics based on the user's skin condition; a means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; a means for receiving and analyzing feedback from the user; a means for evolving an AI model using the analyzed feedback data; a means for providing makeup advice in a virtual store; and a means for confirming and executing the suggested makeup routine in real time. This allows users to receive professional makeup advice from the comfort of their own home and select effective makeup techniques and skin care products. Furthermore, the ability to check the makeup effects in real time significantly increases user satisfaction.

[1151] "User" refers to an individual who uses this system to receive makeup advice tailored to a specific scene.

[1152] A "specific scene" refers to a specific situation or occasion that the user desires, such as a remote meeting, a party event, or everyday outings.

[1153] "Facial image" refers to image data of a user's face that the user provides to the system.

[1154] "Means for receiving" refers to a method or device for the system to import facial image data and other information provided by the user into a server or the like.

[1155] "Means for analyzing" includes techniques and algorithms for analyzing a user's facial features based on facial image data received by the system.

[1156] An "ideal face" refers to a model that generates the optimal facial condition and appearance for a specific scene desired by the user.

[1157] "Difference" refers to the difference that exists between the current user's facial features and the generated ideal facial features.

[1158] The "makeup procedure" refers to the specific makeup method or procedure that the user follows to achieve an ideal face.

[1159] "Skin Care Products" refers to cosmetics and related products for skin care that are recommended based on the user's skin condition.

[1160] "Cosmetics" refers to all cosmetics that users use for makeup.

[1161] The "mirror function" refers to a function that allows users to project their own face on the screen in real time and check the makeup steps as they are performed.

[1162] "Feedback" refers to information about the user's experience and satisfaction with the system after using it.

[1163] "Means for checking in real time" refers to a method or device that allows a user to check the results of a proposed makeup routine in real time while performing the routine.

[1164] "Virtual Store" means an online makeup advice and skin care product recommendation service provided via the Internet.

[1165] An embodiment of the present invention is a system for a virtual store where users can receive professional makeup advice from the comfort of their own home. The system provides makeup procedures and appropriate skin care products tailored to specific scenes, and users can check and implement the effects in real time.

[1166] Hardware and software used

[1167] Hardware: Terminal devices such as smartphones, PCs, and cameras

[1168] Software: Python, Flask, OpenCV, Requests, Heroku, AWS (EC2, S3)

[1169] Overall system flow

[1170] 1. Scene selection:

[1171] The user launches the application and selects a specific scene, such as a remote meeting, a party event, or a daily outing, and this information is sent from the device to the server.

[1172] 2. Facial image capture:

[1173] When the user selects a scene, the device activates the camera function and prompts the user to take a face image. The user uses the camera to take a picture of their face, and the image is saved.

[1174] 3. Receipt and analysis of image data:

[1175] The captured facial image data is encrypted and sent from the device to a server. The server's analysis engine receives this data and analyzes the user's facial features. It also runs an AI algorithm to generate an ideal face model for each scene, generating the ideal face.

[1176] 4. Differential analysis:

[1177] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1178] 5. Makeup procedure and skin care product suggestions:

[1179] Based on the analysis results, the server generates specific makeup procedures for the eyes, skin, lips, etc. and sends them to the device. It also recommends appropriate skin care products and cosmetics based on the user's skin condition. This information can be viewed on the device.

[1180] 6. Real-time mirror function:

[1181] The device uses the front camera to display a real-time image of the user's face, allowing the user to see the results of the proposed makeup application in real time.

[1182] 7. Feedback Receipt and Analysis:

[1183] After the makeup application is complete, the device displays a questionnaire screen to the user about their experience and satisfaction, and collects their feedback. This data is sent to and stored on a server and used to improve the AI ​​model.

[1184] Examples and prompts

[1185] Specific examples

[1186] When a user wants to join a remote meeting, they select "Remote Meeting" from the scene selection menu and take a photo of their face. The server then suggests a professional look suitable for the remote meeting and applies makeup while checking it in real time. During this process, the user can purchase the suggested skin care products and provide feedback to improve the system.

[1187] Prompt Sentence Examples

[1188] "Analyze the facial features in this image and suggest the best makeup routine for a remote meeting."

[1189] As described above, this system enables users to receive highly accurate makeup advice and check and apply it in real time, significantly improving satisfaction in the virtual store.

[1190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1191] Step 1:

[1192] The user launches the application and a scene selection screen appears. The user selects a specific scene, such as a remote meeting, a party event, or a daily outing. This scene information is sent from the device to the server, which then obtains input to determine the next step based on the user's selection.

[1193] Step 2:

[1194] When the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved within the app. Once the face image is saved, the device encrypts the data before sending it to the server.

[1195] Step 3:

[1196] The server receives the encrypted facial image data, decrypts it, and inputs it into the analysis engine. The analysis engine extracts features from the image and analyzes the user's current facial features. Specifically, it uses AI algorithms to analyze the facial contours, position of the eyes, nose, and mouth, as well as the condition of the skin.

[1197] Step 4:

[1198] The analysis engine runs an AI algorithm to generate an ideal face model for each scene. The algorithm uses the prompt "Please analyze the facial features in this image and suggest the best makeup routine for a remote meeting" as input. The server then obtains the ideal face model as output.

[1199] Step 5:

[1200] The server analyzes the differences between the user's current face and the generated ideal face. This process compares the facial contours, the relative positions of each feature, and color tones to identify any gaps. Based on the results of this analysis, the server generates specific makeup instructions to help the user achieve their ideal face.

[1201] Step 6:

[1202] The server then recommends appropriate skincare products and cosmetics based on the user's skin condition along with the generated makeup instructions. This information is then sent to the device, where the user can view the specific makeup instructions and recommended skincare products.

[1203] Step 7:

[1204] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the real-time mirror function to check the results of the proposed makeup steps as they are applied.

[1205] Step 8:

[1206] After the makeup application is complete, the device displays a survey screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1207] This allows makeup advice to be given in the virtual store with high accuracy, realizing a system that improves user satisfaction.

[1208] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1209] The present invention is a system that allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between that and the user's current face, and suggests specific makeup routines, skin care products, and cosmetics. The present invention also incorporates an emotion engine that recognizes the user's emotions, and has the function of analyzing the user's real-time emotions to adjust the makeup routine and skin care product recommendations, allowing users to learn and apply makeup routines that suit them and see the results in real time.

[1210] The system consists of the following elements:

[1211] Scene selection method

[1212] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[1213] Facial image capturing means

[1214] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[1215] Facial image receiving means

[1216] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[1217] Image analysis methods

[1218] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[1219] Differential analysis means

[1220] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1221] Makeup procedure generation means

[1222] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[1223] Skincare product recommendation tool and cosmetic recommendation tool

[1224] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[1225] Real-time mirror function

[1226] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[1227] Emotion Engine

[1228] The device uses an emotion engine to analyze the user's real-time emotions based on facial video and voice data, and the analysis results are sent to a server where they are used to adjust makeup routines and skin care product recommendations.

[1229] Feedback receiving means and feedback analyzing means

[1230] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1231] Specific examples

[1232] A user wants to put on makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup instructions (for example, "shape your eyebrows" or "use a light beige foundation") are generated and sent to the device. The user can check the instructions on the device and use the real-time mirror function to apply their makeup.

[1233] Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides feedback and adjustments accordingly. After the makeup is complete, the user can send feedback, and the server will use that data to improve the AI ​​model and suggest more effective makeup procedures for the next time.

[1234] As described above, by providing detailed support that also takes the user's emotions into consideration, users can easily achieve their ideal appearance.

[1235] The processing flow will be explained below.

[1236] Step 1:

[1237] The user launches an application.

[1238] Action: The device launches the application and displays the scene selection screen.

[1239] Step 2:

[1240] The user selects a scene.

[1241] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[1242] Step 3:

[1243] The device activates the camera function.

[1244] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[1245] Step 4:

[1246] The user takes a picture of their face and saves the image.

[1247] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[1248] Step 5:

[1249] The terminal transmits the facial image data to the server.

[1250] Operation: The stored facial image data is encrypted and securely sent to the server.

[1251] Step 6:

[1252] The server receives the facial image data.

[1253] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[1254] Step 7:

[1255] The server analyzes the facial image and generates the ideal face.

[1256] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[1257] Step 8:

[1258] The server analyzes the differences between the ideal face and the current face.

[1259] How it works: Compare the generated ideal face with your current face and analyze the gap.

[1260] Step 9:

[1261] The server generates the make procedure.

[1262] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[1263] Step 10:

[1264] The server sends the make procedure to the terminal.

[1265] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[1266] Step 11:

[1267] The device will display the makeup instructions.

[1268] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[1269] Step 12:

[1270] The server evaluates the user's skin condition.

[1271] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[1272] Step 13:

[1273] Servers recommend skin care products and cosmetics.

[1274] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[1275] Step 14:

[1276] The device will activate the real-time mirror function.

[1277] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[1278] Step 15:

[1279] The user performs the make procedure.

[1280] Action: The user performs the suggested makeup steps while using a real-time mirror.

[1281] Step 16:

[1282] The emotion engine analyzes the user's emotions.

[1283] How it works: The device collects the user's facial expressions and voice data and uses an emotion engine to analyze emotions in real time.

[1284] Step 17:

[1285] The server receives and adjusts the emotion data.

[1286] How it works: The server adjusts makeup routines and skin care product recommendations based on the emotion data it receives.

[1287] Step 18:

[1288] The device will display a feedback screen.

[1289] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[1290] Step 19:

[1291] The user submits feedback.

[1292] Action: The user enters feedback and presses the submit button.

[1293] Step 20:

[1294] The terminal transmits the feedback data to the server.

[1295] Action: Feedback data is encrypted and sent to the server.

[1296] Step 21:

[1297] The server analyzes the feedback data.

[1298] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[1299] The above are the detailed processing steps for combining the emotion engine with the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skincare routine, allowing them to approach remote meetings and daily life with confidence.

[1300] Example 2

[1301] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1302] Conventional makeup instruction systems are unable to consider users' emotions or real-time reactions, and only provide general advice on specific makeup steps and skin care product recommendations. This makes it difficult for users to receive specific and effective instruction to achieve their ideal appearance.

[1303] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for the user to select a specific scene; means for capturing a facial image of the user and receiving the image data; means for analyzing the received facial image data to generate an ideal face; means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; means for recommending skin care products and cosmetics based on the user's skin condition; means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; means for analyzing real-time emotions based on the user's facial image and voice and adjusting the makeup routine and skin care product recommendations; means for receiving and analyzing feedback from the user; and means for evolving the generative AI model using the analyzed feedback data. This enables detailed makeup instruction tailored to the user's emotions and real-time situation, and recommendations of skin care products and cosmetics tailored to the user's individual needs.

[1304] "Scene selection means" refers to a means by which a user selects a particular scene.

[1305] The "face image capturing means" refers to a means for capturing a face image of the user.

[1306] The "image data receiving means" refers to a means for receiving face image data sent from a terminal on the server side.

[1307] The "ideal face generating means" refers to a means for generating an ideal face based on received face image data.

[1308] "Difference analysis means" refers to means for analyzing the difference between an ideal face and a current face.

[1309] The "makeup procedure generation means" refers to a means for generating a specific makeup procedure based on the difference analysis results.

[1310] "Recommendation means" refers to a means for recommending skin care products and cosmetics based on the user's skin condition.

[1311] The "mirror function providing means" refers to a means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures.

[1312] "Emotion analysis means" refers to a means for analyzing real-time emotions based on facial image and voice data of a user.

[1313] "Feedback receiving means" refers to a means for receiving feedback from a user.

[1314] "Feedback analysis means" refers to means for analyzing received feedback data.

[1315] "Generative AI model" refers to an artificial intelligence model that generates ideal faces and provides makeup instructions.

[1316] This system allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between the current face and the ideal face, and provides specific makeup procedures, skin care products, and cosmetics. Furthermore, this system has the function of analyzing the user's real-time emotions and adjusting the makeup procedures and skin care product recommendations.

[1317] The system consists of the following components:

[1318] Hardware and Software Configuration

[1319] User device: A smartphone or tablet with a built-in camera, display, and microphone.

[1320] Server: A high-performance computer server used to analyze facial image data and emotional data, and generate ideal facial and makeup procedures.

[1321] Software: A software stack including image processing libraries such as OpenCV and Dlib, emotion recognition APIs, and generative AI models.

[1322] Specific processing explanation

[1323] 1. Application launch and scene selection

[1324] On the device: When the user launches the application, a scene selection screen appears. The user selects a scene, such as a remote meeting, work, a date, or a casual outing. This information is sent to the server.

[1325] 2. Taking a facial image

[1326] Device: When the user selects a scene, the camera function is activated and the user is prompted to take a facial image. The user uses the camera to take a facial image, then reviews and saves the image.

[1327] 3. Sending and receiving facial image data

[1328] Terminal: The captured facial image data is encrypted for security purposes and sent to the server.

[1329] Server: Decodes the received facial image data and stores it in a database.

[1330] 4. Facial feature analysis and ideal face model generation

[1331] Server: Based on the received facial image data, it uses image processing libraries such as OpenCV and Dlib to analyze the user's current facial features, and then uses generative AI models to generate an ideal face model for the scene.

[1332] 5. Difference analysis and generation of specific make-up steps

[1333] Server: Compares the difference between the ideal face and the current face and analyzes the gap. Based on the analyzed data, it generates specific makeup instructions to help the user achieve their ideal face.

[1334] 6. Skin care and cosmetic product recommendations

[1335] Server: Based on facial image analysis, evaluates the user's skin condition and recommends the most suitable skincare products and cosmetics. This information is also sent to the device and displayed to the user.

[1336] 7. Executing the real-time mirror function

[1337] Device: Uses the front camera to display a real-time image of the user's face, allowing the user to see the effects of the suggested makeup steps as they are performed.

[1338] 8. Emotional analysis and feedback regulation

[1339] Device: Analyzes facial video and audio data using an emotion analysis API, and transmits the user's real-time emotions to the server.

[1340] Server: Based on the analysis results, it adjusts makeup routines and skin care product recommendations in real time.

[1341] 9. Feedback Collection and Analysis

[1342] Device: After the makeup application is completed, the user is asked to fill out a survey about their experience and satisfaction.

[1343] User: Enter and submit feedback.

[1344] Server: Stores and analyzes the received feedback data, evolves the generative AI model based on that data, and reflects it the next time it is used.

[1345] Specific examples

[1346] A user wants to apply makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for "Remote Meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "Shape your eyebrows" or "Use a light beige foundation") are generated and sent to the device. The user checks the steps on the device and uses the real-time mirror function to apply the makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides appropriate feedback and adjustments. After completing the makeup, the user submits feedback, and the server uses that data to evolve the generative AI model and suggest a more customized makeup step the next time they use it.

[1347] Prompt Sentence Examples

[1348] "I would like the AI ​​to automate the process of comparing my current face with the ideal face for remote meeting situations and suggesting specific makeup steps. I would also like the facial image data to be encrypted before transmission, and appropriate skin care products and cosmetics to be recommended based on the analysis results."

[1349] The above is a detailed description of the preferred embodiment of the invention, which allows users to easily achieve their ideal appearance.

[1350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1351] Step 1: Launch the application and select a scene

[1352] On the device: When the user launches the application, a scene selection screen appears, displaying multiple options (e.g., remote meeting, work, date, everyday outing, etc.) as input.

[1353] User: Selects the desired scene and enters the information into the device.

[1354] Terminal: Sends the user's selection to the server. The input data to the server is the scene information selected by the user. The output data is the scene information sent to the server.

[1355] Step 2: Capture a facial image

[1356] Terminal: When a scene is selected, the terminal activates the camera function and prompts the user to take a facial image. The input at this stage is the scene information that triggers the camera activation.

[1357] User: Take a picture of your face with the camera and view and save the image within the app.

[1358] Terminal: The captured face image data is saved. The output data is the captured face image data.

[1359] Step 3: Encrypt and transmit facial image data

[1360] Terminal: Encrypts the stored facial image data to ensure security. The input data is facial image data, which is then encrypted.

[1361] Terminal: Sends the encrypted facial image data to the server. The encrypted facial image data is sent to the server and becomes the output data.

[1362] Step 4: Receiving and storing facial image data

[1363] Server: Receives the encrypted facial image data and decrypts it. The input data is the encrypted facial image data.

[1364] Server: Saves the decoded facial image data in the database. The decoded facial image data is saved as output.

[1365] Step 5: Facial feature analysis and ideal face model generation

[1366] Server: Analyzes the user's current facial features based on the received facial image data. This process uses image processing libraries such as OpenCV and Dlib. The input data is the decoded facial image data.

[1367] Server: Uses a generative AI model to generate an ideal face model based on scene information. The input data is the decoded face image data and scene information, and the output data is the generated ideal face model.

[1368] Step 6: Difference analysis and generation of specific make procedures

[1369] Server: Compares and analyzes the differences between the ideal face and the current face. A facial landmark detection algorithm is used for the analysis. The input data is the current facial features and the ideal face model.

[1370] Server: Generates specific make procedures based on the difference analysis results. The generated make procedures become the output data.

[1371] Step 7: Skincare and cosmetic product recommendations

[1372] Server: Evaluates the user's skin condition based on the results of facial image analysis and recommends appropriate skin care products and cosmetics. The input data is the results of facial image analysis.

[1373] Server: Sends the recommendation information to the terminal and displays it to the user. The information on the recommended skin care products and cosmetics is the output data.

[1374] Step 8: Run the real-time mirror function

[1375] Terminal: Uses the front camera to display the user's face image in real time. The camera is activated when the user performs the makeup procedure. The input data is the trigger to start the makeup procedure.

[1376] User: Perform the proposed makeup steps and check their effectiveness.

[1377] Terminal: Real-time facial image is displayed to the user, which becomes the output data.

[1378] Step 9: Emotional analysis and feedback adjustment

[1379] Terminal: Analyzes real-time emotions using an emotion analysis API based on the user's facial video and audio data. The input data is real-time facial video and audio data.

[1380] Server: Receives the analysis results and adjusts makeup routines and skin care product recommendations in real time. The output data is the adjusted makeup routine and skin care product information.

[1381] Step 10: Collect and analyze feedback

[1382] Terminal: After the user has finished applying makeup, a questionnaire about the user's experience and satisfaction is displayed. The input data is the trigger for finishing the makeup.

[1383] User: Enter and submit feedback.

[1384] Server: Stores and analyzes the received feedback data. The analysis results are used to evolve the generative AI model and are reflected the next time it is used. The output data is the evolved generative AI model.

[1385] (Application example 2)

[1386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1387] Conventional makeup and skincare recommendation systems are not optimized for the user's emotions or specific situations, limiting their ability to improve the user experience. Furthermore, users often lack real-time feedback and adjustments, preventing them from accurately executing the suggested steps. Furthermore, when using smart glasses or other devices, their interfaces are often not optimized, resulting in a lack of user convenience.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1389] In this invention, the server includes a means for capturing a user's facial image and receiving the image data, a means for analyzing the received facial image data and generating an ideal face, a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, a means for analyzing the user's emotions and adjusting the recommendations, an interface displayed on the smart glasses, a means for receiving and analyzing feedback from the user, and a means for evolving the AI ​​model using the analyzed feedback data. This enables recommendations of optimal makeup routines and products according to the user's specific emotions and situations. Furthermore, real-time adjustments and feedback improve the user experience and enable intuitive operation using the smart glasses.

[1390] The "means for the user to select a specific scene" is an operation interface that allows the user to select a desired scene according to a specific situation such as a remote meeting, a date, or a party.

[1391] "Means for capturing an image of a user's face and receiving the image data" refers to a device or system for capturing an image of a user's face using smart glasses or a camera device and receiving the data to an application or server.

[1392] The "means for analyzing the received facial image data and generating an ideal face" refers to an AI algorithm and a system for executing the algorithm, which uses the received facial image data to generate an ideal face suitable for a specific scene.

[1393] The "means for analyzing the difference between the ideal face and the current face and generating specific makeup procedures" is a system for comparing and analyzing the gap between the current face and the generated ideal face, and obtaining specific makeup procedures to fill the gap.

[1394] The "means for recommending skin care products and cosmetics based on the user's skin condition" is a system for analyzing the user's skin condition and suggesting optimal skin care products and cosmetics.

[1395] "Means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures" refers to a function that displays a real-time image of the user's face through the display of smart glasses or a device, allowing the user to check the procedure while applying makeup.

[1396] The "means for analyzing the user's emotions and adjusting the recommended content" is a function that analyzes the user's emotions from their facial expressions and voice, and dynamically adjusts the makeup steps and recommended content based on that.

[1397] The "interface means displayed on the smart glasses" refers to a user interface for displaying information such as makeup procedures, recommended products, and real-time feedback on the display of the smart glasses.

[1398] The "means for receiving and analyzing feedback from users" is a system for collecting feedback from users regarding their experience and satisfaction with the product after they have finished applying their makeup, and analyzing that data.

[1399] The "means for evolving the AI ​​model using the analyzed feedback data" refers to a system that improves and evolves the AI ​​model based on feedback data collected from users, enabling it to suggest more appropriate makeup procedures and products the next time the model is used.

[1400] This invention is a system that generates an ideal face for a specific scene, analyzes the gap between the ideal face and the current face, and suggests makeup routines and skin care products. It also improves the user experience by dynamically adjusting the recommendations using emotion analysis. A detailed description of a system for implementing this invention is provided below.

[1401] The system includes means for a user to select a specific scene, means for taking a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating specific makeup steps, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for performing makeup steps, means for analyzing the user's emotions and adjusting the recommended content, interface means displayed on the smart glasses, means for receiving and analyzing feedback from the user, and means for evolving the AI ​​model using the analyzed feedback data.

[1402] The server runs an AI algorithm to generate an ideal face based on the scene selected by the user. For example, for a remote meeting, an algorithm is used to generate a natural and professional appearance. The user's facial image is captured using the camera in the smart glasses, and the image data is encrypted and sent to the server. The server analyzes the received facial image data and extracts the current facial features.

[1403] The server then analyzes the differences between the ideal face and the current face and generates specific makeup steps to fill the gap. Based on the user's skin condition, skincare products and cosmetics are also suggested. This information is displayed in real time on the smart glasses' display, allowing the user to apply makeup while viewing it. Additionally, a real-time mirror function allows users to check their own face on the display as they apply their makeup.

[1404] An emotion engine using facial expression and voice recognition is used to analyze user emotions. This emotion engine analyzes the user's emotional state in real time and adjusts recommendations based on the results. For example, if the user appears dissatisfied, it will suggest alternative products or procedures to improve satisfaction.

[1405] After the makeup application is complete, the system receives feedback from the user and analyzes the data. This feedback data is used to improve the AI ​​model, allowing it to make more appropriate suggestions the next time the product is used.

[1406] As a specific example, a user wants to apply makeup for a remote meeting and launches the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face using the smart glasses' camera. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "shape your eyebrows" or "use a light beige foundation") are generated and displayed on the smart glasses' display. The user checks these as they apply their makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with their makeup, it provides appropriate feedback and adjustments. After completing their makeup, the user submits feedback, and the server uses that data to evolve the AI ​​model and suggest more effective makeup steps the next time they use the app.

[1407] An example of a prompt is, "The user has selected makeup for a remote meeting. Generate an ideal face based on the current facial image. Next, analyze the gap between the current face and the ideal face and suggest specific makeup steps to fill the gap (e.g., shaping the eyebrows, using a light beige foundation, etc.). Furthermore, analyze the user's emotions from their facial expressions and voice and consider how to optimize the makeup steps."

[1408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1409] Step 1:

[1410] The user launches the application using the smart glasses and selects the desired scene (e.g., remote meeting, date, party) on the scene selection screen.

[1411] Input: User scene selection (e.g. "Remote Meeting")

[1412] Output: Selected scene information

[1413] Step 2:

[1414] The terminal transmits the scene information selected by the user to the server, and then prompts the user to take a facial image using the camera built into the smart glasses.

[1415] Input: User operation (taking a face image after selecting a scene)

[1416] Output: Photographed face image

[1417] Step 3:

[1418] The facial image data captured by the terminal is encrypted and sent to the server.

[1419] Input: Photographed face image data

[1420] Output: Encrypted facial image data

[1421] Step 4:

[1422] The server analyzes the received facial image data and extracts the current facial features.

[1423] Input: Encrypted facial image data

[1424] Output: Current facial feature data

[1425] Step 5:

[1426] The server runs an AI algorithm to generate an ideal face for the selected scene.

[1427] Input: User's scene information and current facial feature data

[1428] Output: Ideal face data

[1429] Step 6:

[1430] The server compares the ideal face data with the current face data and analyzes the differences.

[1431] Input: Current face data and ideal face data

[1432] Output: Differential data

[1433] Step 7:

[1434] The server generates specific makeup procedures based on the differential data and recommends skin care products and cosmetics.

[1435] Input: differential data

[1436] Output: A list of specific makeup steps and recommended products

[1437] Step 8:

[1438] The device receives makeup instructions and recommended product information from the server and displays them on the smart glasses display in real time. The user applies makeup while viewing this information.

[1439] Input: List of makeup steps and recommended products

[1440] Output: Makeup instructions and recommended product information displayed on smart glasses

[1441] Step 9:

[1442] The device's front camera is used to display a real-time image of the user's face while checking the makeup procedure.

[1443] Input: Real-time facial video

[1444] Output: Real-time facial image displayed on the display

[1445] Step 10:

[1446] The server's emotion engine analyzes the user's facial expressions and voice, assesses the user's emotional state in real time, and adjusts recommendations as needed.

[1447] Input: Real-time facial video and audio data

[1448] Output: Tailored recommendations and feedback

[1449] Step 11:

[1450] After the makeup application is completed, the device displays a feedback questionnaire to the user to collect data on satisfaction and usability.

[1451] Input: Feedback data from users

[1452] Output: Feedback data stored on the device

[1453] Step 12:

[1454] The server improves and evolves the AI ​​model based on the feedback data it receives.

[1455] Input: Feedback data

[1456] Output: An improved AI model

[1457] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1459] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1460] [Fourth embodiment]

[1461] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1462] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1463] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1464] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1465] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1466] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1467] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1468] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1469] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1470] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1471] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1472] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1473] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] The system allows users to select a specific scene, such as a remote meeting or a daily outing, generate an ideal face for that scene, analyze the gap between that and their current face, and suggest specific makeup steps and appropriate skincare products and cosmetics. Using this system, users can learn makeup techniques that suit them, select skincare products according to their skin condition, and check the effects in real time.

[1475] The system consists of the following elements:

[1476] Scene selection method

[1477] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[1478] Facial image capturing means

[1479] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[1480] Facial image receiving means

[1481] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[1482] Image analysis methods

[1483] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[1484] Differential analysis means

[1485] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1486] Makeup procedure generation means

[1487] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[1488] Skincare product recommendation tool and cosmetic recommendation tool

[1489] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[1490] Real-time mirror function

[1491] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[1492] Feedback receiving means and feedback analyzing means

[1493] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1494] By combining these elements, users can easily learn and apply the appropriate makeup and skincare products. Real-time confirmation is also possible, resulting in higher satisfaction. Specifically, the system can suggest makeup steps and skincare products to create a professional look suitable for remote meetings, and the suggestions can be confirmed in real time while being applied.

[1495] The processing flow will be explained below.

[1496] Step 1:

[1497] The user launches an application.

[1498] Action: The device launches the application and displays the scene selection screen.

[1499] Step 2:

[1500] The user selects a scene.

[1501] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[1502] Step 3:

[1503] The device activates the camera function.

[1504] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[1505] Step 4:

[1506] The user takes a picture of their face and saves the image.

[1507] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[1508] Step 5:

[1509] The terminal transmits the facial image data to the server.

[1510] Operation: The stored facial image data is encrypted and securely sent to the server.

[1511] Step 6:

[1512] The server receives the facial image data.

[1513] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[1514] Step 7:

[1515] The server analyzes the facial image and generates the ideal face.

[1516] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[1517] Step 8:

[1518] The server analyzes the differences between the ideal face and the current face.

[1519] How it works: Compare the generated ideal face with your current face and analyze the gap.

[1520] Step 9:

[1521] The server generates the make procedure.

[1522] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[1523] Step 10:

[1524] The server sends the make procedure to the terminal.

[1525] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[1526] Step 11:

[1527] The device will display the makeup instructions.

[1528] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[1529] Step 12:

[1530] The server evaluates the user's skin condition.

[1531] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[1532] Step 13:

[1533] Servers recommend skin care products and cosmetics.

[1534] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[1535] Step 14:

[1536] The device will activate the real-time mirror function.

[1537] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[1538] Step 15:

[1539] The user performs the make procedure.

[1540] Action: The user performs the suggested makeup steps while using a real-time mirror.

[1541] Step 16:

[1542] The device will display a feedback screen.

[1543] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[1544] Step 17:

[1545] The user submits feedback.

[1546] Action: The user enters feedback and presses the submit button.

[1547] Step 18:

[1548] The terminal transmits the feedback data to the server.

[1549] Action: Feedback data is encrypted and sent to the server.

[1550] Step 19:

[1551] The server analyzes the feedback data.

[1552] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[1553] The above are the detailed processing steps of the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skin care, allowing them to approach remote meetings and daily life with confidence.

[1554] Example 1

[1555] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1556] In modern society, there is a demand for instantly preparing the perfect look for various situations, such as remote meetings or outings. However, it is not easy for individual users to know what kind of makeup will bring them closer to their ideal face. It is also difficult to select the appropriate skincare products and cosmetics depending on their skin condition. Furthermore, there is no way to check the effects of makeup in real time as they are applied. A system that can solve these issues is needed.

[1557] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1558] In this invention, the server includes means for allowing a user to select a specific scene, means for capturing a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for executing the makeup routine, means for receiving and analyzing feedback from the user, and means for evolving an artificial intelligence model using the analyzed feedback data. This allows users to learn makeup techniques that suit them, select skin care products according to their skin condition, and proceed while checking the effects in real time.

[1559] "Specific scenes" refer to situations where users need to dress appropriately for the occasion, such as remote meetings or everyday outings.

[1560] "User's face image" refers to photographic data of the user's face taken using the camera of the terminal.

[1561] "Image data" refers to digital data that includes a facial image captured by a user.

[1562] An "ideal face" refers to a facial image that models the user's appearance that is considered optimal for a particular selected scene.

[1563] "Difference" refers to the result of calculating the difference between the current facial image and the ideal facial image.

[1564] "Makeup procedure" refers to the specific makeup method that a user should follow to get closer to their ideal face.

[1565] "Skin condition" refers to an evaluation including the dryness, oil content, health, etc. of the user's skin.

[1566] "Skin care products" refers to beauty products such as creams and lotions used to maintain and improve the health of the skin.

[1567] "Cosmetics" refers to all cosmetics used in makeup (e.g., eye shadow, foundation).

[1568] The "mirror function" refers to a function that uses the device's camera to display the user's face in real time, allowing them to check the progress of their makeup.

[1569] "Feedback" refers to information entered by users about their satisfaction and opinions after using the product.

[1570] An "artificial intelligence model" refers to an algorithm that evolves based on data about the user's makeup and skincare.

[1571] This system allows users to select a specific scene, generate an ideal face for that scene, analyze the gap between the ideal face and their current face, and suggest specific makeup procedures and appropriate skin care products and cosmetics. By using this system, users can learn makeup techniques that suit them, select skin care products according to their skin condition, and check the effects in real time.

[1572] The system consists of the following elements:

[1573] Scene selection method

[1574] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[1575] Facial image capturing means

[1576] After the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved in the app. The captured facial image data is encrypted and sent from the device to the server.

[1577] Facial image receiving means

[1578] The server stores the received facial images and inputs them into the analysis engine.

[1579] Image analysis methods

[1580] The server uses OpenCV and deep learning models (e.g., TensorFlow) to analyze the user's current facial features based on the received facial image data. It also runs AI algorithms to generate an ideal face model for each scene, generating the ideal face. For example, if the user selects "Remote Meeting," an ideal face with a professional appearance will be generated.

[1581] Differential analysis means

[1582] The server compares the generated ideal face with the current face and analyzes the gap. An AI algorithm calculates the difference and generates specific makeup steps to close the gap, such as how to apply eyeshadow or choose foundation.

[1583] Makeup procedure generation means

[1584] The server generates specific makeup instructions based on the analysis results. The generated makeup instructions are sent to the device, where the user can review them. The specific makeup instructions are created using a generative AI model (e.g., GPT-4). An example of a prompt to input to the generative AI model is, "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products."

[1585] Skincare product recommendation tool and cosmetic recommendation tool

[1586] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. For example, information such as "Use this moisturizing cream for dry skin" is sent to the terminal, allowing the user to select the optimal skin care and cosmetics.

[1587] Real-time mirror function

[1588] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can perform the suggested makeup steps and check the results. By tapping the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed in real time.

[1589] Feedback receiving means and feedback analyzing means

[1590] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The server analyzes the received feedback data and uses it to evolve the AI ​​model. The user enters their satisfaction after applying the makeup, and the data is analyzed and used to improve future makeup suggestions.

[1591] By combining these elements, users can easily learn the proper makeup and skincare products to create a look that suits the real-life situation, and the ability to see the results in real time provides a high level of satisfaction.

[1592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1593] Step 1:

[1594] The user launches the app and a scene selection screen appears. The user taps to select the desired scene from options such as remote meeting or everyday outing. The input includes the user's scene selection information. This information is sent from the device to the server. Specifically, the user taps the app icon to launch the app, and then taps the "Remote Meeting" button from the multiple scene selection buttons displayed on the home screen.

[1595] Input: User selected scene information

[1596] Output: Transfer of scene information from the device to the server

[1597] Step 2:

[1598] After the user selects a scene, the device launches the camera function and prompts the user to take a photo of their face. The user uses the camera to take a photo of their face and then reviews and saves the image within the app. The input is the user's face image. The captured face image data is encrypted and sent from the device to the server. Specifically, the device automatically launches the camera app, displays a message saying "Please take a photo of your face," and the user taps the shutter button to take a photo of their face. When the user presses the "Save" button, the image is encrypted and sent to the server.

[1599] Input: User's face image

[1600] Output: Sending encrypted facial image data to the server

[1601] Step 3:

[1602] The server stores the received facial image data and inputs it into the analysis engine. The input is encrypted facial image data. Based on this, the system analyzes the user's current facial features and generates an ideal face model for the scene. Specifically, it uses OpenCV and TensorFlow to analyze facial features and generates the ideal face using AI algorithms.

[1603] Input: Encrypted facial image data

[1604] Output: User's facial feature analysis results, ideal face model

[1605] Step 4:

[1606] The server compares the generated ideal face with the user's current face and analyzes the gap. The inputs are the user's current facial features and the ideal face model. Based on these differences, the server generates specific makeup steps to help the user achieve their ideal face. Specifically, the AI ​​algorithm calculates the optimal makeup steps (e.g., applying a thin layer of eyeshadow around the eyes) to close the gap.

[1607] Input: Current facial features, ideal face model

[1608] Output: Makeup steps to help the user achieve their ideal face

[1609] Step 5:

[1610] The generated makeup instructions are sent to the device, where the user can review them. The input is the makeup instructions. The generated makeup instructions are created using a generative AI model (e.g., GPT-4). Specifically, the server inputs the prompt "Please tell me how to apply makeup for a remote meeting. Please analyze the user's facial image and recommend specific makeup instructions and appropriate skincare products." into the generative AI model, and then sends the results to the device.

[1611] Input: Generated Make steps

[1612] Output: Display the make steps on the terminal for the user to review.

[1613] Step 6:

[1614] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. The input is the user's skin condition data. The server recommends specific skin care products and cosmetics based on the skin analysis results. For example, information such as "Use this moisturizing cream for dry skin" is sent to the device.

[1615] Input: User's skin condition data

[1616] Output: Recommended skincare products and cosmetic information

[1617] Step 7:

[1618] The device uses the front camera to display a real-time image of the user's face. Using this mirror function, the user can apply the suggested makeup steps and check the results. The input is a real-time image of the face. Specifically, when the user taps the "Real-time Mirror" button in the app, the camera is activated and the user's face is displayed.

[1619] Input: Real-time facial video

[1620] Output: Facial image projected on a real-time mirror

[1621] Step 8:

[1622] After completing the makeup application, the device displays a questionnaire screen to the user about the experience and satisfaction. When the user fills in and submits their feedback, the data is sent to and saved on the server. The input is the user's feedback data. The server analyzes the received feedback data and uses it to evolve the AI ​​model. Specifically, the user enters their satisfaction after applying the makeup into the device, and the data is analyzed and used to improve future makeup suggestions.

[1623] Input: User feedback data

[1624] Output: Analyzed feedback data, evolved AI model

[1625] (Application example 1)

[1626] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] Compared to face-to-face services, makeup advice in virtual stores often falls short, making it difficult for users to achieve professional makeup in the comfort of their own homes. It is also often difficult for users to determine effective makeup techniques and appropriate skincare products on their own. Furthermore, existing systems lack the ability to check the effects of makeup in real time or to use feedback to improve AI models. Therefore, there is a need for a system that can provide highly accurate makeup advice in virtual stores and improve user satisfaction.

[1628] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1629] In this invention, the server includes: a means for allowing a user to select a specific scene; a means for capturing a facial image of the user and receiving the image data; a means for analyzing the received facial image data and generating an ideal face; a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; a means for recommending skin care products and cosmetics based on the user's skin condition; a means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; a means for receiving and analyzing feedback from the user; a means for evolving an AI model using the analyzed feedback data; a means for providing makeup advice in a virtual store; and a means for confirming and executing the suggested makeup routine in real time. This allows users to receive professional makeup advice from the comfort of their own home and select effective makeup techniques and skin care products. Furthermore, the ability to check the makeup effects in real time significantly increases user satisfaction.

[1630] "User" refers to an individual who uses this system to receive makeup advice tailored to a specific scene.

[1631] A "specific scene" refers to a specific situation or occasion that the user desires, such as a remote meeting, a party event, or everyday outings.

[1632] "Facial image" refers to image data of a user's face that the user provides to the system.

[1633] "Means for receiving" refers to a method or device for the system to import facial image data and other information provided by the user into a server or the like.

[1634] "Means for analyzing" includes techniques and algorithms for analyzing a user's facial features based on facial image data received by the system.

[1635] An "ideal face" refers to a model that generates the optimal facial condition and appearance for a specific scene desired by the user.

[1636] "Difference" refers to the difference that exists between the current user's facial features and the generated ideal facial features.

[1637] The "makeup procedure" refers to the specific makeup method or procedure that the user follows to achieve an ideal face.

[1638] "Skin Care Products" refers to cosmetics and related products for skin care that are recommended based on the user's skin condition.

[1639] "Cosmetics" refers to all cosmetics that users use for makeup.

[1640] The "mirror function" refers to a function that allows users to project their own face on the screen in real time and check the makeup steps as they are performed.

[1641] "Feedback" refers to information about the user's experience and satisfaction with the system after using it.

[1642] "Means for checking in real time" refers to a method or device that allows a user to check the results of a proposed makeup routine in real time while performing the routine.

[1643] "Virtual Store" means an online makeup advice and skin care product recommendation service provided via the Internet.

[1644] An embodiment of the present invention is a system for a virtual store where users can receive professional makeup advice from the comfort of their own home. The system provides makeup procedures and appropriate skin care products tailored to specific scenes, and users can check and implement the effects in real time.

[1645] Hardware and software used

[1646] Hardware: Terminal devices such as smartphones, PCs, and cameras

[1647] Software: Python, Flask, OpenCV, Requests, Heroku, AWS (EC2, S3)

[1648] Overall system flow

[1649] 1. Scene selection:

[1650] The user launches the application and selects a specific scene, such as a remote meeting, a party event, or a daily outing, and this information is sent from the device to the server.

[1651] 2. Facial image capture:

[1652] When the user selects a scene, the device activates the camera function and prompts the user to take a face image. The user uses the camera to take a picture of their face, and the image is saved.

[1653] 3. Receipt and analysis of image data:

[1654] The captured facial image data is encrypted and sent from the device to a server. The server's analysis engine receives this data and analyzes the user's facial features. It also runs an AI algorithm to generate an ideal face model for each scene, generating the ideal face.

[1655] 4. Differential analysis:

[1656] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1657] 5. Makeup procedure and skin care product suggestions:

[1658] Based on the analysis results, the server generates specific makeup procedures for the eyes, skin, lips, etc. and sends them to the device. It also recommends appropriate skin care products and cosmetics based on the user's skin condition. This information can be viewed on the device.

[1659] 6. Real-time mirror function:

[1660] The device uses the front camera to display a real-time image of the user's face, allowing the user to see the results of the proposed makeup application in real time.

[1661] 7. Feedback Receipt and Analysis:

[1662] After the makeup application is complete, the device displays a questionnaire screen to the user about their experience and satisfaction, and collects their feedback. This data is sent to and stored on a server and used to improve the AI ​​model.

[1663] Examples and prompts

[1664] Specific examples

[1665] When a user wants to join a remote meeting, they select "Remote Meeting" from the scene selection menu and take a photo of their face. The server then suggests a professional look suitable for the remote meeting and applies makeup while checking it in real time. During this process, the user can purchase the suggested skin care products and provide feedback to improve the system.

[1666] Prompt Sentence Examples

[1667] "Analyze the facial features in this image and suggest the best makeup routine for a remote meeting."

[1668] As described above, this system enables users to receive highly accurate makeup advice and check and apply it in real time, significantly improving satisfaction in the virtual store.

[1669] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1670] Step 1:

[1671] The user launches the application and a scene selection screen appears. The user selects a specific scene, such as a remote meeting, a party event, or a daily outing. This scene information is sent from the device to the server, which then obtains input to determine the next step based on the user's selection.

[1672] Step 2:

[1673] When the user selects a scene, the device activates the camera function and prompts the user to take a picture of their face. The user then uses the camera to take a picture of their face, which can then be viewed and saved within the app. Once the face image is saved, the device encrypts the data before sending it to the server.

[1674] Step 3:

[1675] The server receives the encrypted facial image data, decrypts it, and inputs it into the analysis engine. The analysis engine extracts features from the image and analyzes the user's current facial features. Specifically, it uses AI algorithms to analyze the facial contours, position of the eyes, nose, and mouth, as well as the condition of the skin.

[1676] Step 4:

[1677] The analysis engine runs an AI algorithm to generate an ideal face model for each scene. The algorithm uses the prompt "Please analyze the facial features in this image and suggest the best makeup routine for a remote meeting" as input. The server then obtains the ideal face model as output.

[1678] Step 5:

[1679] The server analyzes the differences between the user's current face and the generated ideal face. This process compares the facial contours, the relative positions of each feature, and color tones to identify any gaps. Based on the results of this analysis, the server generates specific makeup instructions to help the user achieve their ideal face.

[1680] Step 6:

[1681] The server then recommends appropriate skincare products and cosmetics based on the user's skin condition along with the generated makeup instructions. This information is then sent to the device, where the user can view the specific makeup instructions and recommended skincare products.

[1682] Step 7:

[1683] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the real-time mirror function to check the results of the proposed makeup steps as they are applied.

[1684] Step 8:

[1685] After the makeup application is complete, the device displays a survey screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1686] This allows makeup advice to be given in the virtual store with high accuracy, realizing a system that improves user satisfaction.

[1687] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1688] The present invention is a system that allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between that and the user's current face, and suggests specific makeup routines, skin care products, and cosmetics. The present invention also incorporates an emotion engine that recognizes the user's emotions, and has the function of analyzing the user's real-time emotions to adjust the makeup routine and skin care product recommendations, allowing users to learn and apply makeup routines that suit them and see the results in real time.

[1689] The system consists of the following elements:

[1690] Scene selection method

[1691] When a user launches the application, a scene selection screen appears. The user taps to select the desired scene from options such as remote meetings or everyday outings. This information is sent from the device to the server.

[1692] Facial image capturing means

[1693] After the user selects a scene, the device will activate the camera function and prompt the user to take a picture of their face. The user can then use the camera to take a picture of their face and view and save the image in the app.

[1694] Facial image receiving means

[1695] The captured facial image data is encrypted and sent from the device to a server, which stores the received facial images and inputs them into an analysis engine.

[1696] Image analysis methods

[1697] The server analyzes the user's current facial features based on the received facial image data, and then runs an AI algorithm to generate an ideal face model for each scene.

[1698] Differential analysis means

[1699] The server compares the generated ideal face with the user's current face and analyzes the difference between them. Based on the results of this difference analysis, the server generates specific makeup steps to help the user achieve their ideal face.

[1700] Makeup procedure generation means

[1701] Based on the analysis results, the server generates specific makeup steps for the eyes, skin, lips, etc., and sends them to the device. The user can then check these makeup steps on the device.

[1702] Skincare product recommendation tool and cosmetic recommendation tool

[1703] The server evaluates the user's skin condition and recommends appropriate skin care products and cosmetics for the occasion. This information is also sent to the device, allowing the user to select the optimal skin care and cosmetics.

[1704] Real-time mirror function

[1705] The device uses the front camera to display a real-time image of the user's face, allowing the user to use the mirror function to carry out the suggested makeup steps and check the results.

[1706] Emotion Engine

[1707] The device uses an emotion engine to analyze the user's real-time emotions based on facial video and voice data, and the analysis results are sent to a server where they are used to adjust makeup routines and skin care product recommendations.

[1708] Feedback receiving means and feedback analyzing means

[1709] After the makeup application is complete, the device displays a questionnaire screen asking the user about their experience and satisfaction. Once the user fills out and submits their feedback, the data is sent to and stored on the server. The server analyzes the received feedback data and uses it to improve the AI ​​model.

[1710] Specific examples

[1711] A user wants to put on makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup instructions (for example, "shape your eyebrows" or "use a light beige foundation") are generated and sent to the device. The user can check the instructions on the device and use the real-time mirror function to apply their makeup.

[1712] Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides feedback and adjustments accordingly. After the makeup is complete, the user can send feedback, and the server will use that data to improve the AI ​​model and suggest more effective makeup procedures for the next time.

[1713] As described above, by providing detailed support that also takes the user's emotions into consideration, users can easily achieve their ideal appearance.

[1714] The processing flow will be explained below.

[1715] Step 1:

[1716] The user launches an application.

[1717] Action: The device launches the application and displays the scene selection screen.

[1718] Step 2:

[1719] The user selects a scene.

[1720] How it works: The user taps to select a scene, such as "Remote Meeting," and that information is sent from the device to the server.

[1721] Step 3:

[1722] The device activates the camera function.

[1723] How it works: Following scene selection, the device automatically opens the camera app and prompts the user to take a photo of their face.

[1724] Step 4:

[1725] The user takes a picture of their face and saves the image.

[1726] How it works: The user uses the camera to take a picture of their face, then the app views and saves the image.

[1727] Step 5:

[1728] The terminal transmits the facial image data to the server.

[1729] Operation: The stored facial image data is encrypted and securely sent to the server.

[1730] Step 6:

[1731] The server receives the facial image data.

[1732] Operation: The server stores the received image data and passes it to the facial feature analysis engine.

[1733] Step 7:

[1734] The server analyzes the facial image and generates the ideal face.

[1735] How it works: The server uses AI algorithms to analyze your current facial features and generate an ideal face based on the selected scene.

[1736] Step 8:

[1737] The server analyzes the differences between the ideal face and the current face.

[1738] How it works: Compare the generated ideal face with your current face and analyze the gap.

[1739] Step 9:

[1740] The server generates the make procedure.

[1741] Operation: Based on the analysis results, the system generates specific makeup steps to help the user achieve their ideal face.

[1742] Step 10:

[1743] The server sends the make procedure to the terminal.

[1744] Operation: The generated makeup procedure data is packaged and sent to the terminal.

[1745] Step 11:

[1746] The device will display the makeup instructions.

[1747] Operation: The device displays the received makeup instructions and provides the user with specific instructions on how to apply makeup.

[1748] Step 12:

[1749] The server evaluates the user's skin condition.

[1750] How it works: The server evaluates the user's skin type and condition and generates analytical data.

[1751] Step 13:

[1752] Servers recommend skin care products and cosmetics.

[1753] Operation: The server generates data recommending appropriate skin care products and cosmetics based on the analysis results and sends it to the device.

[1754] Step 14:

[1755] The device will activate the real-time mirror function.

[1756] How it works: The device uses the front camera to provide a mirror function that displays the user's face in real time.

[1757] Step 15:

[1758] The user performs the make procedure.

[1759] Action: The user performs the suggested makeup steps while using a real-time mirror.

[1760] Step 16:

[1761] The emotion engine analyzes the user's emotions.

[1762] How it works: The device collects the user's facial expressions and voice data and uses an emotion engine to analyze emotions in real time.

[1763] Step 17:

[1764] The server receives and adjusts the emotion data.

[1765] How it works: The server adjusts makeup routines and skin care product recommendations based on the emotion data it receives.

[1766] Step 18:

[1767] The device will display a feedback screen.

[1768] How it works: After completing the makeup steps, the device will display a feedback screen about your satisfaction and impressions.

[1769] Step 19:

[1770] The user submits feedback.

[1771] Action: The user enters feedback and presses the submit button.

[1772] Step 20:

[1773] The terminal transmits the feedback data to the server.

[1774] Action: Feedback data is encrypted and sent to the server.

[1775] Step 21:

[1776] The server analyzes the feedback data.

[1777] How it works: The server analyzes the feedback data it receives and uses the results to improve the AI ​​model.

[1778] The above are the detailed processing steps for combining the emotion engine with the makeup assist system of the present invention. By following these steps, users can achieve their ideal makeup and skincare routine, allowing them to approach remote meetings and daily life with confidence.

[1779] Example 2

[1780] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1781] Conventional makeup instruction systems are unable to consider users' emotions or real-time reactions, and only provide general advice on specific makeup steps and skin care product recommendations. This makes it difficult for users to receive specific and effective instruction to achieve their ideal appearance.

[1782] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for the user to select a specific scene; means for capturing a facial image of the user and receiving the image data; means for analyzing the received facial image data to generate an ideal face; means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine; means for recommending skin care products and cosmetics based on the user's skin condition; means for displaying a real-time image of the user's face and providing a mirror function for executing the makeup routine; means for analyzing real-time emotions based on the user's facial image and voice and adjusting the makeup routine and skin care product recommendations; means for receiving and analyzing feedback from the user; and means for evolving the generative AI model using the analyzed feedback data. This enables detailed makeup instruction tailored to the user's emotions and real-time situation, and recommendations of skin care products and cosmetics tailored to the user's individual needs.

[1783] "Scene selection means" refers to a means by which a user selects a particular scene.

[1784] The "face image capturing means" refers to a means for capturing a face image of the user.

[1785] The "image data receiving means" refers to a means for receiving face image data sent from a terminal on the server side.

[1786] The "ideal face generating means" refers to a means for generating an ideal face based on received face image data.

[1787] "Difference analysis means" refers to means for analyzing the difference between an ideal face and a current face.

[1788] The "makeup procedure generation means" refers to a means for generating a specific makeup procedure based on the difference analysis results.

[1789] "Recommendation means" refers to a means for recommending skin care products and cosmetics based on the user's skin condition.

[1790] The "mirror function providing means" refers to a means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures.

[1791] "Emotion analysis means" refers to a means for analyzing real-time emotions based on facial image and voice data of a user.

[1792] "Feedback receiving means" refers to a means for receiving feedback from a user.

[1793] "Feedback analysis means" refers to means for analyzing received feedback data.

[1794] "Generative AI model" refers to an artificial intelligence model that generates ideal faces and provides makeup instructions.

[1795] This system allows a user to select a specific scene, generates an ideal face for that scene, analyzes the gap between the current face and the ideal face, and provides specific makeup procedures, skin care products, and cosmetics. Furthermore, this system has the function of analyzing the user's real-time emotions and adjusting the makeup procedures and skin care product recommendations.

[1796] The system consists of the following components:

[1797] Hardware and Software Configuration

[1798] User device: A smartphone or tablet with a built-in camera, display, and microphone.

[1799] Server: A high-performance computer server used to analyze facial image data and emotional data, and generate ideal facial and makeup procedures.

[1800] Software: A software stack including image processing libraries such as OpenCV and Dlib, emotion recognition APIs, and generative AI models.

[1801] Specific processing explanation

[1802] 1. Application launch and scene selection

[1803] On the device: When the user launches the application, a scene selection screen appears. The user selects a scene, such as a remote meeting, work, a date, or a casual outing. This information is sent to the server.

[1804] 2. Taking a facial image

[1805] Device: When the user selects a scene, the camera function is activated and the user is prompted to take a facial image. The user uses the camera to take a facial image, then reviews and saves the image.

[1806] 3. Sending and receiving facial image data

[1807] Terminal: The captured facial image data is encrypted for security purposes and sent to the server.

[1808] Server: Decodes the received facial image data and stores it in a database.

[1809] 4. Facial feature analysis and ideal face model generation

[1810] Server: Based on the received facial image data, it uses image processing libraries such as OpenCV and Dlib to analyze the user's current facial features, and then uses generative AI models to generate an ideal face model for the scene.

[1811] 5. Difference analysis and generation of specific make-up steps

[1812] Server: Compares the difference between the ideal face and the current face and analyzes the gap. Based on the analyzed data, it generates specific makeup instructions to help the user achieve their ideal face.

[1813] 6. Skin care and cosmetic product recommendations

[1814] Server: Based on facial image analysis, evaluates the user's skin condition and recommends the most suitable skincare products and cosmetics. This information is also sent to the device and displayed to the user.

[1815] 7. Executing the real-time mirror function

[1816] Device: Uses the front camera to display a real-time image of the user's face, allowing the user to see the effects of the suggested makeup steps as they are performed.

[1817] 8. Emotional analysis and feedback regulation

[1818] Device: Analyzes facial video and audio data using an emotion analysis API, and transmits the user's real-time emotions to the server.

[1819] Server: Based on the analysis results, it adjusts makeup routines and skin care product recommendations in real time.

[1820] 9. Feedback Collection and Analysis

[1821] Device: After the makeup application is completed, the user is asked to fill out a survey about their experience and satisfaction.

[1822] User: Enter and submit feedback.

[1823] Server: Stores and analyzes the received feedback data, evolves the generative AI model based on that data, and reflects it the next time it is used.

[1824] Specific examples

[1825] A user wants to apply makeup for a remote meeting, so they launch the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for "Remote Meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "Shape your eyebrows" or "Use a light beige foundation") are generated and sent to the device. The user checks the steps on the device and uses the real-time mirror function to apply the makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with the makeup, it provides appropriate feedback and adjustments. After completing the makeup, the user submits feedback, and the server uses that data to evolve the generative AI model and suggest a more customized makeup step the next time they use it.

[1826] Prompt Sentence Examples

[1827] "I would like the AI ​​to automate the process of comparing my current face with the ideal face for remote meeting situations and suggesting specific makeup steps. I would also like the facial image data to be encrypted before transmission, and appropriate skin care products and cosmetics to be recommended based on the analysis results."

[1828] The above is a detailed description of the preferred embodiment of the invention, which allows users to easily achieve their ideal appearance.

[1829] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1830] Step 1: Launch the application and select a scene

[1831] On the device: When the user launches the application, a scene selection screen appears, displaying multiple options (e.g., remote meeting, work, date, everyday outing, etc.) as input.

[1832] User: Selects the desired scene and enters the information into the device.

[1833] Terminal: Sends the user's selection to the server. The input data to the server is the scene information selected by the user. The output data is the scene information sent to the server.

[1834] Step 2: Capture a facial image

[1835] Terminal: When a scene is selected, the terminal activates the camera function and prompts the user to take a facial image. The input at this stage is the scene information that triggers the camera activation.

[1836] User: Take a picture of your face with the camera and view and save the image within the app.

[1837] Terminal: The captured face image data is saved. The output data is the captured face image data.

[1838] Step 3: Encrypt and transmit facial image data

[1839] Terminal: Encrypts the stored facial image data to ensure security. The input data is facial image data, which is then encrypted.

[1840] Terminal: Sends the encrypted facial image data to the server. The encrypted facial image data is sent to the server and becomes the output data.

[1841] Step 4: Receiving and storing facial image data

[1842] Server: Receives the encrypted facial image data and decrypts it. The input data is the encrypted facial image data.

[1843] Server: Saves the decoded facial image data in the database. The decoded facial image data is saved as output.

[1844] Step 5: Facial feature analysis and ideal face model generation

[1845] Server: Analyzes the user's current facial features based on the received facial image data. This process uses image processing libraries such as OpenCV and Dlib. The input data is the decoded facial image data.

[1846] Server: Uses a generative AI model to generate an ideal face model based on scene information. The input data is the decoded face image data and scene information, and the output data is the generated ideal face model.

[1847] Step 6: Difference analysis and generation of specific make procedures

[1848] Server: Compares and analyzes the differences between the ideal face and the current face. A facial landmark detection algorithm is used for the analysis. The input data is the current facial features and the ideal face model.

[1849] Server: Generates specific make procedures based on the difference analysis results. The generated make procedures become the output data.

[1850] Step 7: Skincare and cosmetic product recommendations

[1851] Server: Evaluates the user's skin condition based on the results of facial image analysis and recommends appropriate skin care products and cosmetics. The input data is the results of facial image analysis.

[1852] Server: Sends the recommendation information to the terminal and displays it to the user. The information on the recommended skin care products and cosmetics is the output data.

[1853] Step 8: Run the real-time mirror function

[1854] Terminal: Uses the front camera to display the user's face image in real time. The camera is activated when the user performs the makeup procedure. The input data is the trigger to start the makeup procedure.

[1855] User: Perform the proposed makeup steps and check their effectiveness.

[1856] Terminal: Real-time facial image is displayed to the user, which becomes the output data.

[1857] Step 9: Emotional analysis and feedback adjustment

[1858] Terminal: Analyzes real-time emotions using an emotion analysis API based on the user's facial video and audio data. The input data is real-time facial video and audio data.

[1859] Server: Receives the analysis results and adjusts makeup routines and skin care product recommendations in real time. The output data is the adjusted makeup routine and skin care product information.

[1860] Step 10: Collect and analyze feedback

[1861] Terminal: After the user has finished applying makeup, a questionnaire about the user's experience and satisfaction is displayed. The input data is the trigger for finishing the makeup.

[1862] User: Enter and submit feedback.

[1863] Server: Stores and analyzes the received feedback data. The analysis results are used to evolve the generative AI model and are reflected the next time it is used. The output data is the evolved generative AI model.

[1864] (Application example 2)

[1865] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1866] Conventional makeup and skincare recommendation systems are not optimized for the user's emotions or specific situations, limiting their ability to improve the user experience. Furthermore, users often lack real-time feedback and adjustments, preventing them from accurately executing the suggested steps. Furthermore, when using smart glasses or other devices, their interfaces are often not optimized, resulting in a lack of user convenience.

[1867] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1868] In this invention, the server includes a means for capturing a user's facial image and receiving the image data, a means for analyzing the received facial image data and generating an ideal face, a means for analyzing the difference between the ideal face and the current face and generating a specific makeup routine, a means for analyzing the user's emotions and adjusting the recommendations, an interface displayed on the smart glasses, a means for receiving and analyzing feedback from the user, and a means for evolving the AI ​​model using the analyzed feedback data. This enables recommendations of optimal makeup routines and products according to the user's specific emotions and situations. Furthermore, real-time adjustments and feedback improve the user experience and enable intuitive operation using the smart glasses.

[1869] The "means for the user to select a specific scene" is an operation interface that allows the user to select a desired scene according to a specific situation such as a remote meeting, a date, or a party.

[1870] "Means for capturing an image of a user's face and receiving the image data" refers to a device or system for capturing an image of a user's face using smart glasses or a camera device and receiving the data to an application or server.

[1871] The "means for analyzing the received facial image data and generating an ideal face" refers to an AI algorithm and a system for executing the algorithm, which uses the received facial image data to generate an ideal face suitable for a specific scene.

[1872] The "means for analyzing the difference between the ideal face and the current face and generating specific makeup procedures" is a system for comparing and analyzing the gap between the current face and the generated ideal face, and obtaining specific makeup procedures to fill the gap.

[1873] The "means for recommending skin care products and cosmetics based on the user's skin condition" is a system for analyzing the user's skin condition and suggesting optimal skin care products and cosmetics.

[1874] "Means for displaying a real-time image of the user's face and providing a mirror function for performing makeup procedures" refers to a function that displays a real-time image of the user's face through the display of smart glasses or a device, allowing the user to check the procedure while applying makeup.

[1875] The "means for analyzing the user's emotions and adjusting the recommended content" is a function that analyzes the user's emotions from their facial expressions and voice, and dynamically adjusts the makeup steps and recommended content based on that.

[1876] The "interface means displayed on the smart glasses" refers to a user interface for displaying information such as makeup procedures, recommended products, and real-time feedback on the display of the smart glasses.

[1877] The "means for receiving and analyzing feedback from users" is a system for collecting feedback from users regarding their experience and satisfaction with the product after they have finished applying their makeup, and analyzing that data.

[1878] The "means for evolving the AI ​​model using the analyzed feedback data" refers to a system that improves and evolves the AI ​​model based on feedback data collected from users, enabling it to suggest more appropriate makeup procedures and products the next time the model is used.

[1879] This invention is a system that generates an ideal face for a specific scene, analyzes the gap between the ideal face and the current face, and suggests makeup routines and skin care products. It also improves the user experience by dynamically adjusting the recommendations using emotion analysis. A detailed description of a system for implementing this invention is provided below.

[1880] The system includes means for a user to select a specific scene, means for taking a facial image of the user and receiving the image data, means for analyzing the received facial image data and generating an ideal face, means for analyzing the difference between the ideal face and the current face and generating specific makeup steps, means for recommending skin care products and cosmetics based on the user's skin condition, means for displaying an image of the user's face in real time and providing a mirror function for performing makeup steps, means for analyzing the user's emotions and adjusting the recommended content, interface means displayed on the smart glasses, means for receiving and analyzing feedback from the user, and means for evolving the AI ​​model using the analyzed feedback data.

[1881] The server runs an AI algorithm to generate an ideal face based on the scene selected by the user. For example, for a remote meeting, an algorithm is used to generate a natural and professional appearance. The user's facial image is captured using the camera in the smart glasses, and the image data is encrypted and sent to the server. The server analyzes the received facial image data and extracts the current facial features.

[1882] The server then analyzes the differences between the ideal face and the current face and generates specific makeup steps to fill the gap. Based on the user's skin condition, skincare products and cosmetics are also suggested. This information is displayed in real time on the smart glasses' display, allowing the user to apply makeup while viewing it. Additionally, a real-time mirror function allows users to check their own face on the display as they apply their makeup.

[1883] An emotion engine using facial expression and voice recognition is used to analyze user emotions. This emotion engine analyzes the user's emotional state in real time and adjusts recommendations based on the results. For example, if the user appears dissatisfied, it will suggest alternative products or procedures to improve satisfaction.

[1884] After the makeup application is complete, the system receives feedback from the user and analyzes the data. This feedback data is used to improve the AI ​​model, allowing it to make more appropriate suggestions the next time the product is used.

[1885] As a specific example, a user wants to apply makeup for a remote meeting and launches the app. They select "Remote Meeting" on the scene selection screen and then take a photo of their face using the smart glasses' camera. The facial image data is encrypted and sent to the server. The server analyzes the image, generates an ideal face suitable for a "remote meeting," and analyzes any discrepancies with their current face. As a result, specific makeup steps (e.g., "shape your eyebrows" or "use a light beige foundation") are generated and displayed on the smart glasses' display. The user checks these as they apply their makeup. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and voice, and if the user is dissatisfied with their makeup, it provides appropriate feedback and adjustments. After completing their makeup, the user submits feedback, and the server uses that data to evolve the AI ​​model and suggest more effective makeup steps the next time they use the app.

[1886] An example of a prompt is, "The user has selected makeup for a remote meeting. Generate an ideal face based on the current facial image. Next, analyze the gap between the current face and the ideal face and suggest specific makeup steps to fill the gap (e.g., shaping the eyebrows, using a light beige foundation, etc.). Furthermore, analyze the user's emotions from their facial expressions and voice and consider how to optimize the makeup steps."

[1887] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1888] Step 1:

[1889] The user launches the application using the smart glasses and selects the desired scene (e.g., remote meeting, date, party) on the scene selection screen.

[1890] Input: User scene selection (e.g. "Remote Meeting")

[1891] Output: Selected scene information

[1892] Step 2:

[1893] The terminal transmits the scene information selected by the user to the server, and then prompts the user to take a facial image using the camera built into the smart glasses.

[1894] Input: User operation (taking a face image after selecting a scene)

[1895] Output: Photographed face image

[1896] Step 3:

[1897] The facial image data captured by the terminal is encrypted and sent to the server.

[1898] Input: Photographed face image data

[1899] Output: Encrypted facial image data

[1900] Step 4:

[1901] The server analyzes the received facial image data and extracts the current facial features.

[1902] Input: Encrypted facial image data

[1903] Output: Current facial feature data

[1904] Step 5:

[1905] The server runs an AI algorithm to generate an ideal face for the selected scene.

[1906] Input: User's scene information and current facial feature data

[1907] Output: Ideal face data

[1908] Step 6:

[1909] The server compares the ideal face data with the current face data and analyzes the differences.

[1910] Input: Current face data and ideal face data

[1911] Output: Differential data

[1912] Step 7:

[1913] The server generates specific makeup procedures based on the differential data and recommends skin care products and cosmetics.

[1914] Input: differential data

[1915] Output: A list of specific makeup steps and recommended products

[1916] Step 8:

[1917] The device receives makeup instructions and recommended product information from the server and displays them on the smart glasses display in real time. The user applies makeup while viewing this information.

[1918] Input: List of makeup steps and recommended products

[1919] Output: Makeup instructions and recommended product information displayed on smart glasses

[1920] Step 9:

[1921] The device's front camera is used to display a real-time image of the user's face while checking the makeup procedure.

[1922] Input: Real-time facial video

[1923] Output: Real-time facial image displayed on the display

[1924] Step 10:

[1925] The server's emotion engine analyzes the user's facial expressions and voice, assesses the user's emotional state in real time, and adjusts recommendations as needed.

[1926] Input: Real-time facial video and audio data

[1927] Output: Tailored recommendations and feedback

[1928] Step 11:

[1929] After the makeup application is completed, the device displays a feedback questionnaire to the user to collect data on satisfaction and usability.

[1930] Input: Feedback data from users

[1931] Output: Feedback data stored on the device

[1932] Step 12:

[1933] The server improves and evolves the AI ​​model based on the feedback data it receives.

[1934] Input: Feedback data

[1935] Output: An improved AI model

[1936] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1937] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1938] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1939] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1940] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structu...

Claims

1. a means for a user to select a particular scene; A means for capturing a face image of a user and receiving the image data; means for analyzing the received facial image data and generating an ideal face; a means for analyzing the difference between the ideal face and the current face and generating a specific makeup procedure; A means for recommending skin care products and cosmetics based on the user's skin condition; a means for displaying a user's face image in real time and providing a mirror function for performing a makeup procedure; means for receiving and analyzing feedback from users; means for evolving an AI model using the analyzed feedback data; A system including:

2. means for encrypting the captured facial image data and transmitting it to a server; means for evaluating a skin condition based on the facial image data; The system of claim 1 further comprising:

3. A means for implementing an AI algorithm to generate an ideal face; means for analyzing feedback data to improve said AI model; The system of claim 1 further comprising:

Images