system

Abstract

Provide a system. The system includes: a means for users to upload images of celebrities; A method for inputting uploaded images into an AI model for image recognition to detect celebrity makeup features; means for a user to upload an image of his or her face; A means for inputting the uploaded facial image of the user into an AI model for facial recognition and extracting the facial features of the user; A means for generating makeup techniques suited to a user by combining makeup features of celebrities with facial features of the user; The system includes a means for providing the generated makeup techniques to the user step by step.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a technology that allows users to recreate the makeup of celebrities they admire on their own faces. Its purpose is to enable makeup beginners and those who are unsure how to achieve the look they desire to easily receive professional makeup advice. Conventional makeup instruction methods do not adequately consider individual facial features, making it difficult for users to find makeup techniques that are suited to their face. Furthermore, the lack of real-time advice and the ability to share with other users means that users lack support when actually trying out makeup. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] The system includes a means for users to upload images of celebrities and a means for inputting the uploaded images into an AI model for image recognition and detecting the celebrity's makeup features. It also includes a means for users to upload their own facial images and a means for inputting the uploaded facial images of the user into an AI model for facial recognition and extracting the user's facial features. It also includes a means for combining the celebrity's makeup features with the user's facial features to generate makeup techniques suited to the user. It also includes a means for providing the generated makeup techniques to the user step by step.

[0007] The system also includes a means for providing additional makeup advice and video tutorials in real time based on the user's facial features, thereby enhancing support for users when trying out makeup. In addition, the system includes a means for users to upload photos of their makeup creations and share them with other users, thereby promoting the formation of communities and information sharing.

[0008] "User" refers to an individual who receives makeup instruction using this system.

[0009] "Celebrity images" refers to photographic data that shows a celebrity's face or makeup.

[0010] "Upload" refers to the act of sending image data from a local terminal to a server.

[0011] "Terminal" refers to an electronic device operated by a user, such as a smartphone, PC, or tablet.

[0012] "Server" refers to the central system that receives and processes data uploaded by users.

[0013] An "AI model for image recognition" refers to an artificial intelligence algorithm that detects specific features from an input image.

[0014] "Makeup features" refer to detailed makeup elements such as eyeshadow, lip color, and blush positioning obtained from images of celebrities' faces.

[0015] "Facial image" refers to photographic data that shows the user's own face.

[0016] An "AI model for facial recognition" refers to an artificial intelligence algorithm for extracting specific features from an input facial image.

[0017] "Facial features" refer to features such as eye shape, eyebrow shape, and facial contours obtained from a user's facial image.

[0018] "Makeup techniques" refer to methods and procedures for applying makeup that users can recreate on their own faces.

[0019] "Step by step" refers to a format that provides detailed step-by-step instructions on how to apply makeup.

[0020] "Real-time" refers to providing instant feedback and advice to users at the moment they apply their makeup.

[0021] "Video tutorials" refer to video-based teaching materials that allow you to visually learn how to apply makeup.

[0022] A "finished photo" refers to a photo of the user's face after applying makeup.

[0023] "Sharing" refers to the act of publishing and showing the results of one's makeup to other users on the Internet. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] The present invention relates to a system that allows users to recreate the makeup of their favorite celebrities on their own faces. This system analyzes images of celebrities and the user's facial images, and generates and provides makeup techniques that are suitable for the user based on the makeup characteristics. Specific embodiments of the present invention are described below.

[0046] System Overview

[0047] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on their own facial image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides users with step-by-step instructions on how to apply these techniques, and allows them to share the results with other users.

[0048] Program processing explanation

[0049] Celebrity image upload and analysis

[0050] User: The user first opens the application and uploads a picture of their favorite celebrity.

[0051] Terminal: The terminal sends image data to the server.

[0052] Makeup feature extraction

[0053] Server: The server inputs the received image into an AI model for image recognition and analyzes the celebrity's makeup features, thereby identifying detailed makeup elements such as eyeshadow, lip color, and blush placement.

[0054] Uploading and analyzing user facial images

[0055] User: Next, the user uploads a photo of their face to the application.

[0056] Terminal: The terminal sends this facial photo data to the server.

[0057] Facial feature extraction

[0058] Server: The server inputs the user's facial image into an AI model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[0059] Creating makeup techniques

[0060] Server: The server generates the best makeup tips for the user based on the makeup features of celebrities and the user's facial features. This makeup tip is constructed as a step-by-step guide that the user can follow.

[0061] Providing makeup techniques

[0062] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0063] Specific examples

[0064] The following explains this with specific examples.

[0065] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0066] Terminal: The terminal sends these images to the server.

[0067] Server: The server analyzes the makeup features of the model and the user's facial features. Based on this, it generates makeup techniques that are suitable for the user.

[0068] Specific makeup techniques are provided as follows:

[0069] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0070] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0071] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0072] Users: Users can follow the guide to apply makeup, take a photo of the results, and upload them to the application. They can also share their makeup results with other users.

[0073] This embodiment allows users to easily receive professional makeup advice suited to their facial features. In addition, the ability to provide additional advice in real time and share with other users can provide a more satisfying makeup experience.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0077] Step 2:

[0078] Device: The device sends the uploaded celebrity image to the server.

[0079] Step 3:

[0080] Server: The server inputs the received celebrity images into an AI model for image recognition.

[0081] Step 4:

[0082] Server: The server performs pre-processing on the images, including image resizing, normalization, etc.

[0083] Step 5:

[0084] Server: The server uses an AI model to extract the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[0085] Step 6:

[0086] Server: The server stores the extracted makeup features in a database.

[0087] Step 7:

[0088] User: The user then uploads a photo of their face to the application.

[0089] Step 8:

[0090] Terminal: The terminal sends the user's facial photo to the server.

[0091] Step 9:

[0092] Server: The server inputs the received user's facial photo into an AI model for facial recognition.

[0093] Step 10:

[0094] Server: The server performs preprocessing on the face photo, including cropping, resizing, and normalizing the face.

[0095] Step 11:

[0096] Server: The server uses an AI model to extract the user's facial features, such as eye shape, eyebrow shape, and facial contours.

[0097] Step 12:

[0098] Server: The server stores the extracted facial features in a database.

[0099] Step 13:

[0100] Server: The server combines the celebrity's makeup features with the user's facial features to generate makeup techniques suited to the user.

[0101] Step 14:

[0102] Sarver: Sarver creates detailed guides that organize makeup techniques into a step-by-step format.

[0103] Step 15:

[0104] Server: The server sends the generated makeup technique guide to the terminal.

[0105] Step 16:

[0106] Terminal: The terminal displays the makeup technique guide to the user.

[0107] Step 17:

[0108] User: The user follows the displayed guide and actually applies the makeup.

[0109] Step 18:

[0110] User: The user uploads a photo of the completed makeup look to the application.

[0111] Step 19:

[0112] Terminal: The terminal sends the finished photo to the server.

[0113] Step 20:

[0114] Server: The server receives the finished photos and stores them in a database so that they can be shared with other users if desired.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Conventional makeup instruction systems have struggled to provide detailed makeup guides tailored to each user's individual facial features. Furthermore, they lacked specific steps for faithfully recreating famous makeup styles on the user's face, making it difficult for users to find a makeup method that best suits their own features. Furthermore, they lacked real-time advice and the ability to share with other users, preventing them from providing a satisfying makeup experience.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for allowing a user to upload an image of a famous person, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the makeup features of the famous person, means for allowing the user to upload their own facial image, means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the makeup features of the famous person with the user's facial features to generate makeup techniques suitable for the user, means for inputting the generated makeup techniques as prompt sentences into the artificial intelligence model to generate a step-by-step guide, and means for providing the generated makeup guide to a user terminal. This allows users to receive professional makeup advice optimized for their facial features in real time. Furthermore, users can share the results of their makeup with other users, providing a fulfilling makeup experience.

[0120] A "famous person" is a widely known public figure or prominent individual.

[0121] An "artificial intelligence model for image recognition" is a learning algorithm used to analyze image data and detect specific features.

[0122] An "artificial intelligence model for facial recognition" is a learning algorithm that analyzes a user's facial image and identifies the shape and placement of specific parts of the face.

[0123] "Makeup features" refers to specific makeup elements and their placement, such as the color around the eyes, the color of the lips, and the position of the cheeks.

[0124] A "prompt" is text data that is input to a generative artificial intelligence model and is used to provide specific instructions or information to generate a specific output.

[0125] A "generative artificial intelligence model" is a learning algorithm used to generate new text or instructions based on prompts and other data.

[0126] A "step-by-step guide" is a sequence of detailed instructions that a user can follow in order.

[0127] "User terminal" refers to an electronic device used by a user to operate and view information, such as a smartphone or a personal computer.

[0128] The present invention relates to a system that allows a user to recreate makeup looks of famous people on their own face. This system analyzes makeup features using an image of a famous person selected by the user and an image of the user's own face, and generates and provides the user with makeup techniques that are optimal for the user. Specific embodiments for implementing the present invention are described below.

[0129] First, a user opens the application and uploads an image of a famous person. The device receives the image and sends it to the server. The server then inputs the received image into an artificial intelligence model for image recognition (e.g., OpenCV or TENSORFLOW®) to analyze and detect detailed makeup features such as eye shadow, lip color, and blush placement on the famous person.

[0130] Next, the user uploads a photo of their face to the application. The device sends this photo data to the server, which then inputs the user's face image into an artificial intelligence model for facial recognition (e.g., Dlib or FaceNet) to analyze facial features such as eye shape, eyebrow shape, and facial contours.

[0131] The server then generates the best makeup tips for the user based on the makeup features of famous people and the user's facial features. This makeup tip is input as a prompt into a generative artificial intelligence model (e.g., GPT-4 (registered trademark)) to generate a step-by-step guide. Specific examples of prompts are as follows:

[0132] "Generate the following step-by-step makeup guide for a facial image of user X, who has uploaded an image of famous person A. Include detailed instructions for each element of eyeshadow, lipstick, and blush."

[0133] The generated makeup guide is sent from the server to the user's device, which displays it to the user. The user can follow this guide to apply makeup step by step. After completing their makeup, the user can upload a photo of their finished makeup to the application and share it with other users. This function allows users to compare makeup looks and receive advice from each other, thereby enriching the makeup experience.

[0134] As described above, the present invention allows users to receive professional makeup advice tailored to their facial features in real time. Furthermore, by sharing the results of their makeup with other users, users can enjoy a more satisfying makeup experience.

[0135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0136] Step 1:

[0137] User: The user opens the application, selects an image of a famous person and presses the upload button.

[0138] Input: Image files (e.g. JPEG, PNG) of famous people.

[0139] Output: Image file temporarily saved on the device.

[0140] Specific operation: The user operates the application and selects an image of a famous person from the gallery or camera. This image is temporarily stored on the device.

[0141] Step 2:

[0142] Device: The device sends the image of the celebrity uploaded by the user to the server.

[0143] Input: Image file in the device.

[0144] Output: Image data sent to the server.

[0145] What happens: The device sends the saved image file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[0146] Step 3:

[0147] Server: The server inputs the received image of the famous person into an artificial intelligence model for image recognition and analyzes makeup features such as eye shadow, lip color, and blush position.

[0148] Input: Received image data.

[0149] Output: Analyzed makeup feature data (e.g. eyeshadow color, lip color, blush position, etc.).

[0150] Specific operation: The server analyzes the image using an artificial intelligence model for image recognition (e.g., TensorFlow, OpenCV). The makeup feature data extracted as a result of the analysis is stored in a database.

[0151] Step 4:

[0152] User: The user uploads a photo of their face to the application.

[0153] Input: User's face photo file (e.g. JPEG, PNG).

[0154] Output: A face photo file temporarily saved on the device.

[0155] Specific operation: The user operates the application and selects a photo of their face from the gallery or camera. This photo is temporarily stored on the device.

[0156] Step 5:

[0157] Terminal: The terminal sends the user's facial photo to the server.

[0158] Input: Face photo file in the device.

[0159] Output: Facial photo data sent to the server.

[0160] What it does: The device sends the stored face photo file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[0161] Step 6:

[0162] Server: The server inputs the received user's facial photo into an artificial intelligence model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[0163] Input: Received facial photo data.

[0164] Output: Analyzed facial feature data (e.g. eye shape, eyebrow shape, facial contour, etc.).

[0165] Specific operation: The server analyzes the facial image using an AI model for facial recognition (e.g., Dlib, FaceNet). The facial feature data extracted as the analysis result is stored in a database.

[0166] Step 7:

[0167] Server: The server generates makeup prompt sentences based on the makeup features of famous people and the user's facial features, and inputs them into the generative artificial intelligence model.

[0168] Input: Makeup feature data of famous people and facial feature data of the user.

[0169] Output: Step-by-step makeup guide.

[0170] Specific operation: The server combines both sets of data to generate a prompt. For example, "Generate the following step-by-step makeup guide for the face image of user X, who uploaded an image of famous person A. Please include detailed instructions for each element: eyeshadow, lipstick, and blush." ​​This prompt is then input into a generative AI model (e.g., GPT-4) to generate a specific makeup guide.

[0171] Step 8:

[0172] Terminal: The makeup guide generated by the server is provided to the terminal, which displays it to the user.

[0173] Input: The generated step-by-step makeup guide.

[0174] Output: The makeup guide displayed on the terminal by the user.

[0175] Specific operation: The terminal displays the makeup guide received from the server. The user follows this guide to perform specific makeup steps.

[0176] Step 9:

[0177] User: After applying makeup, the user uploads a photo of the finished look to the application and shares it with other users.

[0178] Input: Finished photo file (e.g. JPEG, PNG).

[0179] Output: Uploaded finished photo.

[0180] Specific operation: The user presses the upload button, selects a photo of the completed makeup look, and uploads it to the application. The application sends it to the server and shares it with other users.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] When users try to recreate celebrity makeup looks on their own faces, existing methods that suggest appropriate makeup techniques struggle to provide real-time advice or generate specific prompts. Furthermore, there are limited ways to share the makeup results with others or receive feedback. This makes it difficult to improve users' makeup skills and increase their satisfaction.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for users to upload images of celebrities, means for inputting the uploaded images into a generative AI model for image recognition and detecting the celebrity's makeup features, means for inputting the uploaded user's facial image into a generative AI model for face recognition and extracting the user's facial features, means for providing a user interface for displaying the generated makeup technique guide, means for generating prompt sentences for guiding users through the steps of applying the suggested makeup techniques, and means for sharing the results with other people. This allows users to receive appropriate makeup techniques along with specific steps in real time, and also allows them to share their makeup results with others to receive feedback.

[0186] 1. "User" refers to a person who uses this system to recreate celebrity makeup on their own face.

[0187] 2. "Celebrity images" refers to visual data, such as photographs or illustrations, that show the face of a celebrity.

[0188] 3. "Upload" refers to sending data from a particular device to a remote storage system, such as a server.

[0189] 4. "Generative AI model for image recognition" refers to an artificial intelligence model that processes image data to recognize and extract specific features.

[0190] 5. "Makeup features" refers to specific features of makeup applied to a celebrity's face, such as makeup patterns and colors.

[0191] 6. "Facial Image" refers to photographs or visual data that show a user's face.

[0192] 7. “Generative AI model for facial recognition” means an artificial intelligence model used to analyze facial features from facial images.

[0193] 8. "Facial Features" refers to specific external features of a user's face, such as the eyes, nose, and mouth.

[0194] 9. "Makeup Technique" refers to the specific makeup methods and procedures applied to the user's face.

[0195] 10. "Step-by-step means" refers to a method that provides users with specific, step-by-step instructions.

[0196] 11. "User interface" refers to the screens and methods of operation that allow a user to interact with a system.

[0197] 12. "Prompt sentences" refers to a series of sentences that provide the user with makeup application instructions or advice in text form.

[0198] 13. "Means for sharing results with others" refers to a function that allows users to show their makeup results to other users via the Internet or other means and receive their evaluations.

[0199] This invention relates to a system that allows users to recreate celebrity makeup on their own face. The system analyzes images of celebrities and the user's facial image, and uses a generative AI model to provide the user with makeup techniques that are suited to the user.

[0200] System Overview

[0201] This system is implemented with the following hardware and software configuration. The user's smartphone or tablet is used as the device, and the server hosts the AI ​​model as a cloud service. The main software includes the OpenCV library for image processing and the Keras framework for using deep learning models.

[0202] System Operation

[0203] Celebrity image upload and analysis

[0204] User: First, the user uploads an image of their favorite celebrity from their device in order to recreate that celebrity's makeup.

[0205] Device: The device sends uploaded images to a server, which then inputs them into a generative AI model for image recognition.

[0206] Server: The server uses a generative AI model to analyze the image and extract the celebrity's makeup features.

[0207] Uploading and analyzing user facial images

[0208] User: Next, the user uploads a photo of their face from their device.

[0209] Terminal: The terminal sends this facial photo to the server.

[0210] Server: The server analyzes the user's facial features using a generative AI model for facial recognition.

[0211] Creating and providing makeup techniques

[0212] Server: The server combines celebrity makeup features with the user's facial features to generate makeup techniques suited to the user. The generated makeup techniques are structured as a step-by-step guide and are presented through a user interface.

[0213] Terminal: The terminal displays the generated makeup techniques to the user and provides prompts on how to apply the proposed makeup techniques.

[0214] Usage example

[0215] Suppose a user wants to apply a makeup style to their face using a famous model. The user first uploads an image of the model to the application, then takes and uploads a photo of their own face. The server analyzes these images and suggests the best makeup techniques by combining the celebrity's makeup style with the user's facial features. The user can then apply the makeup step by step by following the prompts displayed on the smartphone screen.

[0216] Prompt Sentence Examples

[0217] The prompt is presented to the user in the following format:

[0218] "Upload an image of your favorite celebrity and take a photo of yourself. The system will analyze the image and suggest the best makeup techniques."

[0219] This allows users to easily receive professional makeup advice realized using advanced machine learning technology. Furthermore, the ability to provide additional advice in real time and share with others provides users with a more satisfying makeup experience.

[0220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0221] Step 1:

[0222] User: The user uploads an image of their favorite celebrity to the application using their smartphone or tablet. This input image data is the initial data for analyzing the celebrity's makeup features.

[0223] Step 2:

[0224] Device: The device sends the celebrity image uploaded by the user to the server, where it is converted into the appropriate format (e.g., JPEG or PNG).

[0225] Step 3:

[0226] Server: The server inputs the received images into a generative AI model for image recognition. The AI ​​model analyzes the makeup features in the image (e.g., eyeshadow color, lip color, blush position) and extracts specific features. The output is a dataset showing celebrity makeup features.

[0227] Step 4:

[0228] User: Next, the user uploads a photo of their face using the same application. This photo data becomes the base data for analyzing the user's facial features.

[0229] Step 5:

[0230] Terminal: The terminal converts the user's facial photo into a specified format and sends it to the server.

[0231] Step 6:

[0232] Server: The server inputs the received user's face photo into a generative AI model for facial recognition. The AI ​​model analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.) and generates facial feature data as output.

[0233] Step 7:

[0234] Server: The server combines and analyzes celebrity makeup feature data with the user's facial feature data to generate the optimal makeup technique for the user. This makeup technique is constructed as a step-by-step guide. The output is a list of specific makeup steps and cosmetic items to use.

[0235] Step 8:

[0236] Terminal: The terminal displays the step-by-step makeup guide received from the server to the user, who can then apply makeup by following the guide.

[0237] Step 9:

[0238] Server: Based on the generated makeup technique guide, the server generates a prompt sentence to guide the user through the makeup application procedure. This prompt sentence is displayed on the GUI screen.

[0239] Step 10:

[0240] User: The user applies makeup using the provided guide and prompts, then takes a photo of the completed makeup and uploads it to the application.

[0241] Step 11:

[0242] Device: The device sends the photos of the makeup results uploaded by the user to the server and stores them on the platform for sharing with other users.

[0243] Through these steps, users can recreate the makeup of their favorite celebrities on their own faces and share the results with others.

[0244] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0245] The present invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques suited to the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, the present invention can provide makeup advice and feedback according to the user's emotional state. Specific embodiments of the present invention are described below.

[0246] System Overview

[0247] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on the image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides the user with step-by-step instructions on how to apply the makeup, and allows the results to be shared with other users. It also uses an emotion engine to recognize the user's emotions and provides advice and feedback based on those emotions.

[0248] Program processing explanation

[0249] Celebrity image upload and analysis

[0250] User: The user launches the application, selects an image of their favorite celebrity, and uploads it.

[0251] Terminal: The terminal transmits the selected image data to the server.

[0252] Makeup feature extraction

[0253] Server: The server inputs the received celebrity image into an AI model for image recognition. As preprocessing, the image is resized and normalized. The AI ​​model then extracts the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[0254] Uploading and analyzing user facial images

[0255] User: Next, the user uploads a photo of their face to the application.

[0256] Terminal: The terminal sends the user's facial photo to the server.

[0257] Facial feature extraction

[0258] Server: The server inputs the user's facial image into the AI ​​model for facial recognition. After preprocessing such as cropping, resizing, and normalizing the face, the AI ​​model extracts facial features such as eye shape, eyebrow shape, and facial contours.

[0259] Creating makeup techniques

[0260] Server: The server generates makeup techniques suitable for the user based on the makeup features of celebrities and the user's facial features. The generated makeup techniques are constructed as a step-by-step guide that the user can follow.

[0261] Providing makeup techniques

[0262] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0263] Use of emotion engine

[0264] User: While applying makeup, the user's facial expressions are captured and analyzed by a camera to recognize emotions in real time.

[0265] Device: The device sends facial expression data captured by the camera to the emotion engine.

[0266] Server: The emotion engine analyzes facial expressions to determine the user's emotional state, for example, whether they are confused or happy.

[0267] Emotion-based advice and feedback

[0268] Server: Based on the emotional state recognized by the emotion engine, the server provides the user with appropriate advice and feedback. If the user is confused, the server provides additional support, such as re-explaining the steps, and if the user is satisfied, the server displays messages of praise and encouragement.

[0269] Device: The device displays real-time advice and feedback to the user, providing enhanced support during the makeup application process.

[0270] Specific examples

[0271] The following explains this with specific examples.

[0272] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0273] Terminal: The terminal sends these images to the server.

[0274] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[0275] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0276] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0277] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0278] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[0279] This embodiment allows users to easily receive professional makeup advice suited to their facial features. Furthermore, the function of providing additional advice in real time and feedback according to emotions can provide a more satisfying makeup experience.

[0280] The processing flow will be explained below.

[0281] Step 1:

[0282] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0283] Step 2:

[0284] Terminal: The terminal sends the image data of the selected celebrity to the server.

[0285] Step 3:

[0286] Server: The server preprocesses the received celebrity images, specifically resizing and normalizing the images.

[0287] Step 4:

[0288] Server: The server inputs preprocessed images into an AI model for image recognition and extracts makeup features such as eyeshadow, lip color, and blush position.

[0289] Step 5:

[0290] Server: The server stores the extracted makeup features in a database.

[0291] Step 6:

[0292] User: Next, the user uploads a photo of their face to the application.

[0293] Step 7:

[0294] Terminal: The terminal sends the user's facial photo data to the server.

[0295] Step 8:

[0296] Server: The server preprocesses the received user face photo, specifically cropping, resizing, and normalizing the face.

[0297] Step 9:

[0298] Server: The server inputs preprocessed facial photos into an AI model for facial recognition, extracting facial features such as eye shape, eyebrow shape, and facial contours.

[0299] Step 10:

[0300] Server: The server stores the extracted facial features in a database.

[0301] Step 11:

[0302] Server: The server generates makeup techniques suitable for the user based on the makeup characteristics of celebrities and the user's facial features.

[0303] Step 12:

[0304] Server: The server builds the generated makeup techniques as a step-by-step guide.

[0305] Step 13:

[0306] Server: The server sends the guide to the device.

[0307] Step 14:

[0308] Terminal: The terminal displays the makeup technique guide to the user.

[0309] Step 15:

[0310] User: The user follows the displayed guide and applies makeup in the correct order.

[0311] Step 16:

[0312] Terminal: The terminal captures the user's facial expressions while applying makeup with a camera and sends the facial expression data to the server.

[0313] Step 17:

[0314] Server: The server uses an emotion engine to analyze the captured facial expression data and recognize the user's emotions.

[0315] Step 18:

[0316] Server: Based on emotion recognition, adjust makeup steps and advice as needed. For example, if the user is confused, restate the steps or suggest an alternative approach.

[0317] Step 19:

[0318] Server: Generates positive feedback such as encouragement or praise according to the user's emotional state and sends it to the device.

[0319] Step 20:

[0320] Terminal: The terminal displays feedback and advice received from the server to the user in real time, supporting the makeup experience.

[0321] Step 21:

[0322] User: After the user has completed their makeup, they take a photo of the finished look and upload it to the application.

[0323] Step 22:

[0324] Terminal: The terminal sends the finished photo to the server.

[0325] Step 23:

[0326] Server: The server stores the finished photos in a database and makes them available for sharing with other users as needed.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Conventional makeup advice systems are required to not only detect celebrity makeup features, but also generate makeup techniques suitable for the user and provide them as a practical, concrete, step-by-step guide. However, these systems lack real-time advice and feedback based on the user's emotional state, which can lead to confusion and reduced user satisfaction. A new system that solves these problems is needed.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0331] In this invention, the server includes means for a user to upload an image of a celebrity, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features, means for a user to upload his or her own facial image, means for inputting the uploaded user's facial image into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user, means for providing the generated makeup technique to the user step by step, means for capturing and analyzing the user's facial expressions in real time, and means for recognizing the user's emotional state based on the results of the facial expression analysis and providing appropriate advice and feedback. This not only enables the user to more effectively recreate celebrity makeup, but also enables a satisfying makeup experience without confusion due to the real-time emotion recognition and feedback.

[0332] The "means for uploading images of celebrities" is a function that allows a user to select an image of a celebrity through an application and transmit the image data to a server via a network.

[0333] "Artificial intelligence model for image recognition" refers to an automated learning algorithm that analyzes received image data and extracts specific features from it, such as cosmetic features like eyeshadow, lip color, and blush placement.

[0334] The "means for detecting makeup features" is a function that uses an artificial intelligence model for image recognition to identify specific makeup elements from images of celebrities.

[0335] "Means for users to upload their own facial images" refers to a function that allows users to take or select a photo of their own face via the application and send it to the server.

[0336] "Artificial intelligence model for facial recognition" refers to an automated learning algorithm that analyzes a user's facial image and identifies specific facial features such as eye shape, eyebrow shape, and facial contours to extract features.

[0337] The "means for extracting a user's facial features" is a function that uses an artificial intelligence model for facial recognition to identify specific features from a user's facial image and extract them as data.

[0338] The "means for generating makeup techniques" is a function that integrates the makeup features of celebrities with the user's facial features and generates a step-by-step guide for a makeup method that is suitable for the user.

[0339] The "means for providing step-by-step instructions to the user" is a function of instructing the user in an easy-to-understand manner for each step of the generated makeup guide, and displaying or explaining the instructions so that the user can proceed as instructed.

[0340] "Means for capturing and analyzing a user's facial expressions in real time" refers to a function that uses a camera to capture a user's facial expressions in real time while the user is applying makeup, and analyzes the facial expression data.

[0341] "Means for recognizing emotional states and providing appropriate advice and feedback" refers to a function that analyzes captured facial expression data, determines the user's emotional state, and provides appropriate advice and supportive messages in real time.

[0342] This invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques that are suitable for the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide makeup advice and feedback according to the user's emotional state.

[0343] System Overview

[0344] This system consists of multiple hardware and software components. The main hardware components are the user's device, such as a smartphone or tablet, and a cloud server. The software components include an AI model for image recognition (e.g., DeepLabV3+), an AI model for face recognition (e.g., FaceNet), and an emotion engine (e.g., Emotion API).

[0345] Users upload images of celebrities and their own faces through the application. The device sends these images to a server, where image analysis is performed. The celebrity's makeup features and the user's facial features are analyzed and extracted, and the optimal makeup technique for the user is generated based on this. The generated makeup technique is then provided to the user step by step.

[0346] System Details

[0347] Celebrity image upload and analysis

[0348] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0349] Terminal: The terminal sends the image data selected by the user to the server using a secure protocol such as HTTPS.

[0350] Server: The server temporarily stores the received image data and inputs it into an AI model for image recognition. After preprocessing such as resizing and normalizing the image, the AI ​​model extracts the celebrity's makeup characteristics.

[0351] Uploading and analyzing user facial images

[0352] User: Next, the user uploads a photo of their face to the application.

[0353] Device: The device temporarily stores the user's facial photo locally and then sends it to the server, again using a secure protocol.

[0354] Server: The server temporarily stores the received user's face photo and inputs it into an AI model for facial recognition. After preprocessing such as cropping, resizing, and normalization, the AI ​​model extracts the user's facial features.

[0355] Creation and provision of cosmetic techniques

[0356] Server: The server generates a makeup technique suited to the user based on the makeup characteristics of celebrities and the user's facial features. The generated makeup technique is constructed as a step-by-step guide. This guide includes detailed information on the type and amount of makeup to use in each step, as well as the order and location of application.

[0357] Terminal: Receives the makeup guide generated by the server and displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0358] Use of emotion engine

[0359] User: While applying makeup, the device camera captures the user's facial expressions.

[0360] Device: The device transmits the facial expression data captured in real time to the emotion engine.

[0361] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state, for example, re-explaining the steps if the user is confused, or generating an encouraging message if the user is satisfied.

[0362] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[0363] Specific examples

[0364] Specific examples are shown below.

[0365] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0366] Terminal: The terminal sends these images to the server.

[0367] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[0368] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0369] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0370] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0371] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[0372] Prompt Sentence Examples

[0373] Below are some examples of prompt sentences.

[0374] Makeup feature extraction

[0375] "Identify makeup features from the following celebrity images: eyeshadow, lip color, blush placement, etc."

[0376] Facial feature extraction

[0377] "Analyze this user's facial image to extract eye shape, eyebrow shape, and facial contours."

[0378] Use of emotion engine

[0379] "Analyze this user's facial expression to determine their current emotional state (confusion, satisfaction, etc.)"

[0380] This allows users to easily and efficiently learn and practice professional makeup techniques, and the system also provides real-time support, improving user satisfaction.

[0381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0382] Step 1: Upload and submit a photo of your celebrity

[0383] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0384] Terminal: The terminal temporarily stores the image data selected by the user locally and then transmits it to the server using a secure protocol such as HTTPS.

[0385] Input: celebrity image file

[0386] Output: Image data sent to the server

[0387] Step 2: Extracting celebrity makeup features

[0388] Server: The server temporarily stores the received image data. Next, it resizes and normalizes the image. After preprocessing, the image is input into an artificial intelligence model for image recognition (e.g., DeepLabV3+) to extract makeup features such as the celebrity's eyeshadow, lip color, and blush position.

[0389] Input: Resized and normalized celebrity image data

[0390] Output: Extracted makeup feature data

[0391] Step 3: Upload and send the user's face image

[0392] User: Next, the user uploads a photo of their face to the application.

[0393] Device: The device temporarily stores the user's facial photo locally and then sends it to the server using a secure protocol such as HTTPS.

[0394] Input: User's face image file

[0395] Output: Facial image data sent to the server

[0396] Step 4: Extracting the user's facial features

[0397] Server: The server temporarily stores the received user's facial image data. Then, it performs preprocessing such as cropping, resizing, and normalization of the face. After preprocessing, the facial image is input into an artificial intelligence model for facial recognition (e.g., FaceNet) to extract facial features such as eye shape, eyebrow shape, and facial contours.

[0398] Input: Resized and normalized user face image data

[0399] Output: Extracted facial feature data

[0400] Step 5: Creating makeup techniques

[0401] Server: The server combines celebrity makeup feature data with the user's facial feature data to generate the best makeup techniques for the user. During the generation process, a specific algorithm is used to map makeup features and facial features, and a step-by-step guide for the best makeup techniques for the user is constructed.

[0402] Input: celebrity makeup feature data and user facial feature data

[0403] Output: Step-by-step makeup guide

[0404] Step 6: Providing makeup tips

[0405] Device: The device receives the step-by-step makeup tutorial generated by the server and displays it to the user via the application. UI design is important for the presentation, and each step is often supplemented with images or short video clips.

[0406] User: The user follows the guide to apply makeup in the correct order.

[0407] Enter: a step-by-step makeup tutorial

[0408] Output: User's Make execution results

[0409] Step 7: Use the Emotion Engine

[0410] User: While applying makeup, the device camera captures the user's facial expressions in real time.

[0411] Device: The device sends the facial expression data captured in real time to the emotion engine (e.g., Emotion API).

[0412] Input: Captured user facial expression data

[0413] Output: Facial expression data sent to the server

[0414] Step 8: Emotion-based advice and feedback

[0415] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state. Based on the determined emotional state, the server generates appropriate advice and feedback. This feedback can include re-explaining the steps if the user is confused, or providing an encouraging message if the user is satisfied.

[0416] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[0417] Input: Parsed user emotion data

[0418] Output: Real-time advice and feedback

[0419] (Application example 2)

[0420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] In the modern beauty industry, users are highly interested in replicating celebrity makeup looks on their own faces. However, there are few systems in brick-and-mortar stores that allow users to receive personalized makeup techniques in real time. Furthermore, mechanisms for providing feedback based on the user's emotional state during the makeup process are also lacking. The present invention aims to solve these issues by providing a system that provides real-time emotional advice while replicating celebrity makeup looks.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0423] In this invention, the server includes: means for a user to upload an image of a celebrity; means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features; means for a user to upload his or her own facial image; means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features; means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user; means for providing the generated makeup technique to the user step by step; and means for recognizing the user's emotional state in real time and providing advice and feedback according to the user's emotions. This enables users to recreate celebrity makeup looks even in physical stores, and further enables users to achieve higher satisfaction by receiving feedback according to their emotions during the makeup process.

[0424] "User" refers to someone who uses the system to recreate celebrity makeup looks.

[0425] A "celebrity image" is a photograph of a famous person's face that is used to extract the person's makeup features.

[0426] An "artificial intelligence model for image recognition" is an artificial intelligence program that detects and analyzes specific features from input images.

[0427] "Cosmetic features" refer to specific elements of makeup such as eyeshadow, lip color, and blush placement.

[0428] A "face image" is a photograph of the user's face that is uploaded to the system.

[0429] An "artificial intelligence model for facial recognition" is an artificial intelligence program that extracts facial features such as eye shape, eyebrow shape, and facial contours from an input facial image.

[0430] "Facial features" refer to the shape of the user's face and the characteristics of each part of the face.

[0431] "Makeup techniques" refer to makeup techniques that are suitable for a user and are generated by combining the makeup features of celebrities with the user's facial features.

[0432] "Providing step-by-step instructions" means providing guidance so that the user can perform the generated makeup techniques in a step-by-step manner.

[0433] "Emotional state" refers to the emotion expressed by the user while applying makeup, and indicates states such as satisfaction or confusion.

[0434] "Providing advice and feedback" means notifying the user of appropriate instructions or encouraging messages depending on the user's emotional state.

[0435] In this invention, we will build a system that provides users with a specific method for recreating celebrity makeup on their own face. The overall configuration of the system consists of a process in which users upload images of celebrities and are provided with makeup techniques that are suitable for the user based on the analysis results. A detailed explanation of the system is provided below.

[0436] Hardware and software used

[0437] Hardware:

[0438] Smart glasses or head-mounted displays (HMD)

[0439] Server (with high-performance processor and sufficient storage capacity)

[0440] Cameras (built into smart glasses or HMDs)

[0441] software:

[0442] Artificial intelligence models for image recognition: OpenCV, TensorFlow

[0443] Artificial intelligence models for face recognition: Dlib, FaceNet

[0444] Emotion engine: Microsoft® Azure® Emotion API

[0445] Process Overview

[0446] The system mainly includes the following processing steps:

[0447] 1. A user uploads an image of a celebrity.

[0448] 2. The server inputs this image into an artificial intelligence model to extract makeup features.

[0449] 3. The user uploads a photo of their face.

[0450] 4. The server inputs the user's facial image into an artificial intelligence model and extracts facial features.

[0451] 5. The server generates the optimal makeup techniques based on the celebrity's makeup characteristics and the user's facial features.

[0452] 6. Display a step-by-step makeup application guide on smart glasses or an HMD.

[0453] 7. Recognize the user's emotional state in real time and provide advice and feedback according to that state.

[0454] Program processing explanation

[0455] User face scan:

[0456] When a user wears the smart glasses, the camera scans the user's face in real time. OpenCV preprocesses the image (resizing, normalizing), and then Dlib and FaceNet are used to extract facial features. The extracted facial features are sent to the server and stored as facial feature data.

[0457] Celebrity makeup feature extraction:

[0458] Users upload images of celebrities, and the server inputs the images into a TensorFlow-based artificial intelligence model that analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[0459] Makeup Technique Creation:

[0460] The server generates optimal makeup techniques based on the makeup characteristics of celebrities and the user's facial features. The generated makeup techniques are then constructed as a step-by-step guide for the user to follow.

[0461] Emotional feedback:

[0462] The camera captures the user's facial expressions while applying makeup and inputs them into the emotion engine. Using the Microsoft Azure Emotion API, the system analyzes the user's emotional state in real time, providing re-explanations if the user is confused, and positive feedback if the user is satisfied.

[0463] Specific examples

[0464] 1. User flow:

[0465] A user visits a cosmetics store, puts on the smart glasses, selects a makeup image of a celebrity they like, and then captures and uploads a photo of their own face.

[0466] 2. Program processing example:

[0467] Celebrity image analysis prompt:

[0468] "Analyze and extract the location of eyeshadow, lip color, and blush in this image."

[0469] Sentiment Analysis Prompt:

[0470] "Please recognize the emotion from the facial expression in this captured image and provide appropriate advice."

[0471] In this way, the system can provide users with a highly satisfying and effective makeup experience by recreating celebrity makeup looks while providing real-time emotional feedback.

[0472] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0473] Step 1:

[0474] A user uploads an image of a celebrity.

[0475] Input: A user-selected celebrity face image.

[0476] How it works: The user launches the application and selects an image of a celebrity. The application then sends the selected image data from the device to the server.

[0477] Output: Celebrity face image data uploaded to the server.

[0478] Step 2:

[0479] The server inputs images of celebrities into an image recognition AI model and extracts the celebrities' makeup features.

[0480] Input: Uploaded celebrity face image.

[0481] How it works: The server preprocesses the received image data (resizing, normalizing) and inputs it into a TensorFlow-based artificial intelligence model for image recognition. The model analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[0482] Output: Extracted celebrity makeup feature data.

[0483] Step 3:

[0484] The user uploads a picture of their face.

[0485] Input: A photo of the user's face.

[0486] Operation: The user uploads a photo of their face to the application, and the device sends the photo to the server.

[0487] Output: User's facial image data uploaded to the server.

[0488] Step 4:

[0489] The server inputs the user's facial image into a facial recognition AI model and extracts the user's facial features.

[0490] Input: Uploaded user face image.

[0491] How it works: The server preprocesses (resizes, normalizes) the received facial images and feeds them into a Dlib or FaceNet-based AI model for facial recognition, which extracts facial features such as eye shape, eyebrow shape, and facial contours.

[0492] Output: Extracted user facial feature data.

[0493] Step 5:

[0494] The server generates the most suitable makeup technique based on the makeup features of celebrities and the user's facial features.

[0495] Input: celebrity makeup feature data and user facial feature data.

[0496] How it works: The server combines both sets of data and generates a makeup application tailored to the user, which is structured as a step-by-step guide.

[0497] Output: Generated step-by-step makeup tutorial.

[0498] Step 6:

[0499] The terminal provides the generated makeup technique guide to the user.

[0500] Enter: a step-by-step makeup artistry guide.

[0501] How it works: The server sends the generated guide to the device, which displays it to the user. The user can view the guide step by step through smart glasses or a head-mounted display.

[0502] Output: A step-by-step makeup tutorial provided to the user.

[0503] Step 7:

[0504] It recognizes the user's emotional state in real time and provides advice and feedback according to that emotion.

[0505] Input: User facial expression data captured by the camera.

[0506] How it works: The device captures the user's facial expressions in real time and sends the data to an emotion engine (Microsoft Azure Emotion API). The server analyzes the changes in facial expressions and determines the user's emotional state (confusion, satisfaction, etc.). It then generates appropriate feedback and advice and sends it to the device.

[0507] Output: Advice or feedback based on the user's emotional state.

[0508] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0510] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0511] [Second embodiment]

[0512] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0513] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0515] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0519] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0520] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0521] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0522] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0523] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0524] The present invention relates to a system that allows users to recreate the makeup of their favorite celebrities on their own faces. This system analyzes images of celebrities and the user's facial images, and generates and provides makeup techniques that are suitable for the user based on the makeup characteristics. Specific embodiments of the present invention are described below.

[0525] System Overview

[0526] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on their own facial image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides users with step-by-step instructions on how to apply these techniques, and allows them to share the results with other users.

[0527] Program processing explanation

[0528] Celebrity image upload and analysis

[0529] User: The user first opens the application and uploads a picture of their favorite celebrity.

[0530] Terminal: The terminal sends image data to the server.

[0531] Makeup feature extraction

[0532] Server: The server inputs the received image into an AI model for image recognition and analyzes the celebrity's makeup features, thereby identifying detailed makeup elements such as eyeshadow, lip color, and blush placement.

[0533] Uploading and analyzing user facial images

[0534] User: Next, the user uploads a photo of their face to the application.

[0535] Terminal: The terminal sends this facial photo data to the server.

[0536] Facial feature extraction

[0537] Server: The server inputs the user's facial image into an AI model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[0538] Creating makeup techniques

[0539] Server: The server generates the best makeup tips for the user based on the makeup features of celebrities and the user's facial features. This makeup tip is constructed as a step-by-step guide that the user can follow.

[0540] Providing makeup techniques

[0541] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0542] Specific examples

[0543] The following explains this with specific examples.

[0544] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0545] Terminal: The terminal sends these images to the server.

[0546] Server: The server analyzes the makeup features of the model and the user's facial features. Based on this, it generates makeup techniques that are suitable for the user.

[0547] Specific makeup techniques are provided as follows:

[0548] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0549] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0550] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0551] Users: Users can follow the guide to apply makeup, take a photo of the results, and upload them to the application. They can also share their makeup results with other users.

[0552] This embodiment allows users to easily receive professional makeup advice suited to their facial features. In addition, the ability to provide additional advice in real time and share with other users can provide a more satisfying makeup experience.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0556] Step 2:

[0557] Device: The device sends the uploaded celebrity image to the server.

[0558] Step 3:

[0559] Server: The server inputs the received celebrity images into an AI model for image recognition.

[0560] Step 4:

[0561] Server: The server performs pre-processing on the images, including image resizing, normalization, etc.

[0562] Step 5:

[0563] Server: The server uses an AI model to extract the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[0564] Step 6:

[0565] Server: The server stores the extracted makeup features in a database.

[0566] Step 7:

[0567] User: The user then uploads a photo of their face to the application.

[0568] Step 8:

[0569] Terminal: The terminal sends the user's facial photo to the server.

[0570] Step 9:

[0571] Server: The server inputs the received user's facial photo into an AI model for facial recognition.

[0572] Step 10:

[0573] Server: The server performs preprocessing on the face photo, including cropping, resizing, and normalizing the face.

[0574] Step 11:

[0575] Server: The server uses an AI model to extract the user's facial features, such as eye shape, eyebrow shape, and facial contours.

[0576] Step 12:

[0577] Server: The server stores the extracted facial features in a database.

[0578] Step 13:

[0579] Server: The server combines the celebrity's makeup features with the user's facial features to generate makeup techniques suited to the user.

[0580] Step 14:

[0581] Sarver: Sarver creates detailed guides that organize makeup techniques into a step-by-step format.

[0582] Step 15:

[0583] Server: The server sends the generated makeup technique guide to the terminal.

[0584] Step 16:

[0585] Terminal: The terminal displays the makeup technique guide to the user.

[0586] Step 17:

[0587] User: The user follows the displayed guide and actually applies the makeup.

[0588] Step 18:

[0589] User: The user uploads a photo of the completed makeup look to the application.

[0590] Step 19:

[0591] Terminal: The terminal sends the finished photo to the server.

[0592] Step 20:

[0593] Server: The server receives the finished photos and stores them in a database so that they can be shared with other users if desired.

[0594] Example 1

[0595] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0596] Conventional makeup instruction systems have struggled to provide detailed makeup guides tailored to each user's individual facial features. Furthermore, they lacked specific steps for faithfully recreating famous makeup styles on the user's face, making it difficult for users to find a makeup method that best suits their own features. Furthermore, they lacked real-time advice and the ability to share with other users, preventing them from providing a satisfying makeup experience.

[0597] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0598] In this invention, the server includes means for allowing a user to upload an image of a famous person, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the makeup features of the famous person, means for allowing the user to upload their own facial image, means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the makeup features of the famous person with the user's facial features to generate makeup techniques suitable for the user, means for inputting the generated makeup techniques as prompt sentences into the artificial intelligence model to generate a step-by-step guide, and means for providing the generated makeup guide to a user terminal. This allows users to receive professional makeup advice optimized for their facial features in real time. Furthermore, users can share the results of their makeup with other users, providing a fulfilling makeup experience.

[0599] A "famous person" is a widely known public figure or prominent individual.

[0600] An "artificial intelligence model for image recognition" is a learning algorithm used to analyze image data and detect specific features.

[0601] An "artificial intelligence model for facial recognition" is a learning algorithm that analyzes a user's facial image and identifies the shape and placement of specific parts of the face.

[0602] "Makeup features" refers to specific makeup elements and their placement, such as the color around the eyes, the color of the lips, and the position of the cheeks.

[0603] A "prompt" is text data that is input to a generative artificial intelligence model and is used to provide specific instructions or information to generate a specific output.

[0604] A "generative artificial intelligence model" is a learning algorithm used to generate new text or instructions based on prompts and other data.

[0605] A "step-by-step guide" is a sequence of detailed instructions that a user can follow in order.

[0606] "User terminal" refers to an electronic device used by a user to operate and view information, such as a smartphone or a personal computer.

[0607] The present invention relates to a system that allows a user to recreate makeup looks of famous people on their own face. This system analyzes makeup features using an image of a famous person selected by the user and an image of the user's own face, and generates and provides the user with makeup techniques that are optimal for the user. Specific embodiments for implementing the present invention are described below.

[0608] First, a user opens the application and uploads an image of a famous person. The device receives the image and sends it to the server. The server then inputs the received image into an artificial intelligence model for image recognition (such as OpenCV or TensorFlow) to analyze and detect detailed makeup features such as eyeshadow, lip color, and blush placement on the famous person.

[0609] Next, the user uploads a photo of their face to the application. The device sends this photo data to the server, which then inputs the user's face image into an artificial intelligence model for facial recognition (e.g., Dlib or FaceNet) to analyze facial features such as eye shape, eyebrow shape, and facial contours.

[0610] The server then generates the best makeup tips for the user based on the makeup features of famous people and the user's facial features. This makeup tip is input as a prompt into a generative AI model (e.g., GPT-4) to generate a step-by-step guide. Specific examples of prompts are as follows:

[0611] "Generate the following step-by-step makeup guide for a facial image of user X, who has uploaded an image of famous person A. Include detailed instructions for each element of eyeshadow, lipstick, and blush."

[0612] The generated makeup guide is sent from the server to the user's device, which displays it to the user. The user can follow this guide to apply makeup step by step. After completing their makeup, the user can upload a photo of their finished makeup to the application and share it with other users. This function allows users to compare makeup looks and receive advice from each other, thereby enriching the makeup experience.

[0613] As described above, the present invention allows users to receive professional makeup advice tailored to their facial features in real time. Furthermore, by sharing the results of their makeup with other users, users can enjoy a more satisfying makeup experience.

[0614] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0615] Step 1:

[0616] User: The user opens the application, selects an image of a famous person and presses the upload button.

[0617] Input: Image files (e.g. JPEG, PNG) of famous people.

[0618] Output: Image file temporarily saved on the device.

[0619] Specific operation: The user operates the application and selects an image of a famous person from the gallery or camera. This image is temporarily stored on the device.

[0620] Step 2:

[0621] Device: The device sends the image of the celebrity uploaded by the user to the server.

[0622] Input: Image file in the device.

[0623] Output: Image data sent to the server.

[0624] What happens: The device sends the saved image file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[0625] Step 3:

[0626] Server: The server inputs the received image of the famous person into an artificial intelligence model for image recognition and analyzes makeup features such as eye shadow, lip color, and blush position.

[0627] Input: Received image data.

[0628] Output: Analyzed makeup feature data (e.g. eyeshadow color, lip color, blush position, etc.).

[0629] Specific operation: The server analyzes the image using an artificial intelligence model for image recognition (e.g., TensorFlow, OpenCV). The makeup feature data extracted as a result of the analysis is stored in a database.

[0630] Step 4:

[0631] User: The user uploads a photo of their face to the application.

[0632] Input: User's face photo file (e.g. JPEG, PNG).

[0633] Output: A face photo file temporarily saved on the device.

[0634] Specific operation: The user operates the application and selects a photo of their face from the gallery or camera. This photo is temporarily stored on the device.

[0635] Step 5:

[0636] Terminal: The terminal sends the user's facial photo to the server.

[0637] Input: Face photo file in the device.

[0638] Output: Facial photo data sent to the server.

[0639] What it does: The device sends the stored face photo file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[0640] Step 6:

[0641] Server: The server inputs the received user's facial photo into an artificial intelligence model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[0642] Input: Received facial photo data.

[0643] Output: Analyzed facial feature data (e.g. eye shape, eyebrow shape, facial contour, etc.).

[0644] Specific operation: The server analyzes the facial image using an AI model for facial recognition (e.g., Dlib, FaceNet). The facial feature data extracted as the analysis result is stored in a database.

[0645] Step 7:

[0646] Server: The server generates makeup prompt sentences based on the makeup features of famous people and the user's facial features, and inputs them into the generative artificial intelligence model.

[0647] Input: Makeup feature data of famous people and facial feature data of the user.

[0648] Output: Step-by-step makeup guide.

[0649] Specific operation: The server combines both sets of data to generate a prompt. For example, "Generate the following step-by-step makeup guide for the face image of user X, who uploaded an image of famous person A. Please include detailed instructions for each element: eyeshadow, lipstick, and blush." ​​This prompt is then input into a generative AI model (e.g., GPT-4) to generate a specific makeup guide.

[0650] Step 8:

[0651] Terminal: The makeup guide generated by the server is provided to the terminal, which displays it to the user.

[0652] Input: The generated step-by-step makeup guide.

[0653] Output: The makeup guide displayed on the terminal by the user.

[0654] Specific operation: The terminal displays the makeup guide received from the server. The user follows this guide to perform specific makeup steps.

[0655] Step 9:

[0656] User: After applying makeup, the user uploads a photo of the finished look to the application and shares it with other users.

[0657] Input: Finished photo file (e.g. JPEG, PNG).

[0658] Output: Uploaded finished photo.

[0659] Specific operation: The user presses the upload button, selects a photo of the completed makeup look, and uploads it to the application. The application sends it to the server and shares it with other users.

[0660] (Application example 1)

[0661] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0662] When users try to recreate celebrity makeup looks on their own faces, existing methods that suggest appropriate makeup techniques struggle to provide real-time advice or generate specific prompts. Furthermore, there are limited ways to share the makeup results with others or receive feedback. This makes it difficult to improve users' makeup skills and increase their satisfaction.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0664] In this invention, the server includes means for users to upload images of celebrities, means for inputting the uploaded images into a generative AI model for image recognition and detecting the celebrity's makeup features, means for inputting the uploaded user's facial image into a generative AI model for face recognition and extracting the user's facial features, means for providing a user interface for displaying the generated makeup technique guide, means for generating prompt sentences for guiding users through the steps of applying the suggested makeup techniques, and means for sharing the results with other people. This allows users to receive appropriate makeup techniques along with specific steps in real time, and also allows them to share their makeup results with others to receive feedback.

[0665] 1. "User" refers to a person who uses this system to recreate celebrity makeup on their own face.

[0666] 2. "Celebrity images" refers to visual data, such as photographs or illustrations, that show the face of a celebrity.

[0667] 3. "Upload" refers to sending data from a particular device to a remote storage system, such as a server.

[0668] 4. "Generative AI model for image recognition" refers to an artificial intelligence model that processes image data to recognize and extract specific features.

[0669] 5. "Makeup features" refers to specific features of makeup applied to a celebrity's face, such as makeup patterns and colors.

[0670] 6. "Facial Image" refers to photographs or visual data that show a user's face.

[0671] 7. “Generative AI model for facial recognition” means an artificial intelligence model used to analyze facial features from facial images.

[0672] 8. "Facial Features" refers to specific external features of a user's face, such as the eyes, nose, and mouth.

[0673] 9. "Makeup Technique" refers to the specific makeup methods and procedures applied to the user's face.

[0674] 10. "Step-by-step means" refers to a method that provides users with specific, step-by-step instructions.

[0675] 11. "User interface" refers to the screens and methods of operation that allow a user to interact with a system.

[0676] 12. "Prompt sentences" refers to a series of sentences that provide the user with makeup application instructions or advice in text form.

[0677] 13. "Means for sharing results with others" refers to a function that allows users to show their makeup results to other users via the Internet or other means and receive their evaluations.

[0678] This invention relates to a system that allows users to recreate celebrity makeup on their own face. The system analyzes images of celebrities and the user's facial image, and uses a generative AI model to provide the user with makeup techniques that are suited to the user.

[0679] System Overview

[0680] This system is implemented with the following hardware and software configuration. The user's smartphone or tablet is used as the device, and the server hosts the AI ​​model as a cloud service. The main software includes the OpenCV library for image processing and the Keras framework for using deep learning models.

[0681] System Operation

[0682] Celebrity image upload and analysis

[0683] User: First, the user uploads an image of their favorite celebrity from their device in order to recreate that celebrity's makeup.

[0684] Device: The device sends uploaded images to a server, which then inputs them into a generative AI model for image recognition.

[0685] Server: The server uses a generative AI model to analyze the image and extract the celebrity's makeup features.

[0686] Uploading and analyzing user facial images

[0687] User: Next, the user uploads a photo of their face from their device.

[0688] Terminal: The terminal sends this facial photo to the server.

[0689] Server: The server analyzes the user's facial features using a generative AI model for facial recognition.

[0690] Creating and providing makeup techniques

[0691] Server: The server combines celebrity makeup features with the user's facial features to generate makeup techniques suited to the user. The generated makeup techniques are structured as a step-by-step guide and are presented through a user interface.

[0692] Terminal: The terminal displays the generated makeup techniques to the user and provides prompts on how to apply the proposed makeup techniques.

[0693] Usage example

[0694] Suppose a user wants to apply a makeup style to their face using a famous model. The user first uploads an image of the model to the application, then takes and uploads a photo of their own face. The server analyzes these images and suggests the best makeup techniques by combining the celebrity's makeup style with the user's facial features. The user can then apply the makeup step by step by following the prompts displayed on the smartphone screen.

[0695] Prompt Sentence Examples

[0696] The prompt is presented to the user in the following format:

[0697] "Upload an image of your favorite celebrity and take a photo of yourself. The system will analyze the image and suggest the best makeup techniques."

[0698] This allows users to easily receive professional makeup advice realized using advanced machine learning technology. Furthermore, the ability to provide additional advice in real time and share with others provides users with a more satisfying makeup experience.

[0699] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0700] Step 1:

[0701] User: The user uploads an image of their favorite celebrity to the application using their smartphone or tablet. This input image data is the initial data for analyzing the celebrity's makeup features.

[0702] Step 2:

[0703] Device: The device sends the celebrity image uploaded by the user to the server, where it is converted into the appropriate format (e.g., JPEG or PNG).

[0704] Step 3:

[0705] Server: The server inputs the received images into a generative AI model for image recognition. The AI ​​model analyzes the makeup features in the image (e.g., eyeshadow color, lip color, blush position) and extracts specific features. The output is a dataset showing celebrity makeup features.

[0706] Step 4:

[0707] User: Next, the user uploads a photo of their face using the same application. This photo data becomes the base data for analyzing the user's facial features.

[0708] Step 5:

[0709] Terminal: The terminal converts the user's facial photo into a specified format and sends it to the server.

[0710] Step 6:

[0711] Server: The server inputs the received user's face photo into a generative AI model for facial recognition. The AI ​​model analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.) and generates facial feature data as output.

[0712] Step 7:

[0713] Server: The server combines and analyzes celebrity makeup feature data with the user's facial feature data to generate the optimal makeup technique for the user. This makeup technique is constructed as a step-by-step guide. The output is a list of specific makeup steps and cosmetic items to use.

[0714] Step 8:

[0715] Terminal: The terminal displays the step-by-step makeup guide received from the server to the user, who can then apply makeup by following the guide.

[0716] Step 9:

[0717] Server: Based on the generated makeup technique guide, the server generates a prompt sentence to guide the user through the makeup application procedure. This prompt sentence is displayed on the GUI screen.

[0718] Step 10:

[0719] User: The user applies makeup using the provided guide and prompts, then takes a photo of the completed makeup and uploads it to the application.

[0720] Step 11:

[0721] Device: The device sends the photos of the makeup results uploaded by the user to the server and stores them on the platform for sharing with other users.

[0722] Through these steps, users can recreate the makeup of their favorite celebrities on their own faces and share the results with others.

[0723] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0724] The present invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques suited to the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, the present invention can provide makeup advice and feedback according to the user's emotional state. Specific embodiments of the present invention are described below.

[0725] System Overview

[0726] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on the image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides the user with step-by-step instructions on how to apply the makeup, and allows the results to be shared with other users. It also uses an emotion engine to recognize the user's emotions and provides advice and feedback based on those emotions.

[0727] Program processing explanation

[0728] Celebrity image upload and analysis

[0729] User: The user launches the application, selects an image of their favorite celebrity, and uploads it.

[0730] Terminal: The terminal transmits the selected image data to the server.

[0731] Makeup feature extraction

[0732] Server: The server inputs the received celebrity image into an AI model for image recognition. As preprocessing, the image is resized and normalized. The AI ​​model then extracts the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[0733] Uploading and analyzing user facial images

[0734] User: Next, the user uploads a photo of their face to the application.

[0735] Terminal: The terminal sends the user's facial photo to the server.

[0736] Facial feature extraction

[0737] Server: The server inputs the user's facial image into the AI ​​model for facial recognition. After preprocessing such as cropping, resizing, and normalizing the face, the AI ​​model extracts facial features such as eye shape, eyebrow shape, and facial contours.

[0738] Creating makeup techniques

[0739] Server: The server generates makeup techniques suitable for the user based on the makeup features of celebrities and the user's facial features. The generated makeup techniques are constructed as a step-by-step guide that the user can follow.

[0740] Providing makeup techniques

[0741] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0742] Use of emotion engine

[0743] User: While applying makeup, the user's facial expressions are captured and analyzed by a camera to recognize emotions in real time.

[0744] Device: The device sends facial expression data captured by the camera to the emotion engine.

[0745] Server: The emotion engine analyzes facial expressions to determine the user's emotional state, for example, whether they are confused or happy.

[0746] Emotion-based advice and feedback

[0747] Server: Based on the emotional state recognized by the emotion engine, the server provides the user with appropriate advice and feedback. If the user is confused, the server provides additional support, such as re-explaining the steps, and if the user is satisfied, the server displays messages of praise and encouragement.

[0748] Device: The device displays real-time advice and feedback to the user, providing enhanced support during the makeup application process.

[0749] Specific examples

[0750] The following explains this with specific examples.

[0751] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0752] Terminal: The terminal sends these images to the server.

[0753] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[0754] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0755] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0756] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0757] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[0758] This embodiment allows users to easily receive professional makeup advice suited to their facial features. Furthermore, the function of providing additional advice in real time and feedback according to emotions can provide a more satisfying makeup experience.

[0759] The processing flow will be explained below.

[0760] Step 1:

[0761] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0762] Step 2:

[0763] Terminal: The terminal sends the image data of the selected celebrity to the server.

[0764] Step 3:

[0765] Server: The server preprocesses the received celebrity images, specifically resizing and normalizing the images.

[0766] Step 4:

[0767] Server: The server inputs preprocessed images into an AI model for image recognition and extracts makeup features such as eyeshadow, lip color, and blush position.

[0768] Step 5:

[0769] Server: The server stores the extracted makeup features in a database.

[0770] Step 6:

[0771] User: Next, the user uploads a photo of their face to the application.

[0772] Step 7:

[0773] Terminal: The terminal sends the user's facial photo data to the server.

[0774] Step 8:

[0775] Server: The server preprocesses the received user face photo, specifically cropping, resizing, and normalizing the face.

[0776] Step 9:

[0777] Server: The server inputs preprocessed facial photos into an AI model for facial recognition, extracting facial features such as eye shape, eyebrow shape, and facial contours.

[0778] Step 10:

[0779] Server: The server stores the extracted facial features in a database.

[0780] Step 11:

[0781] Server: The server generates makeup techniques suitable for the user based on the makeup characteristics of celebrities and the user's facial features.

[0782] Step 12:

[0783] Server: The server builds the generated makeup techniques as a step-by-step guide.

[0784] Step 13:

[0785] Server: The server sends the guide to the device.

[0786] Step 14:

[0787] Terminal: The terminal displays the makeup technique guide to the user.

[0788] Step 15:

[0789] User: The user follows the displayed guide and applies makeup in the correct order.

[0790] Step 16:

[0791] Terminal: The terminal captures the user's facial expressions while applying makeup with a camera and sends the facial expression data to the server.

[0792] Step 17:

[0793] Server: The server uses an emotion engine to analyze the captured facial expression data and recognize the user's emotions.

[0794] Step 18:

[0795] Server: Based on emotion recognition, adjust makeup steps and advice as needed. For example, if the user is confused, restate the steps or suggest an alternative approach.

[0796] Step 19:

[0797] Server: Generates positive feedback such as encouragement or praise according to the user's emotional state and sends it to the device.

[0798] Step 20:

[0799] Terminal: The terminal displays feedback and advice received from the server to the user in real time, supporting the makeup experience.

[0800] Step 21:

[0801] User: After the user has completed their makeup, they take a photo of the finished look and upload it to the application.

[0802] Step 22:

[0803] Terminal: The terminal sends the finished photo to the server.

[0804] Step 23:

[0805] Server: The server stores the finished photos in a database and makes them available for sharing with other users as needed.

[0806] Example 2

[0807] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0808] Conventional makeup advice systems are required to not only detect celebrity makeup features, but also generate makeup techniques suitable for the user and provide them as a practical, concrete, step-by-step guide. However, these systems lack real-time advice and feedback based on the user's emotional state, which can lead to confusion and reduced user satisfaction. A new system that solves these problems is needed.

[0809] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0810] In this invention, the server includes means for a user to upload an image of a celebrity, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features, means for a user to upload his or her own facial image, means for inputting the uploaded user's facial image into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user, means for providing the generated makeup technique to the user step by step, means for capturing and analyzing the user's facial expressions in real time, and means for recognizing the user's emotional state based on the results of the facial expression analysis and providing appropriate advice and feedback. This not only enables the user to more effectively recreate celebrity makeup, but also enables a satisfying makeup experience without confusion due to the real-time emotion recognition and feedback.

[0811] The "means for uploading images of celebrities" is a function that allows a user to select an image of a celebrity through an application and transmit the image data to a server via a network.

[0812] "Artificial intelligence model for image recognition" refers to an automated learning algorithm that analyzes received image data and extracts specific features from it, such as cosmetic features like eyeshadow, lip color, and blush placement.

[0813] The "means for detecting makeup features" is a function that uses an artificial intelligence model for image recognition to identify specific makeup elements from images of celebrities.

[0814] "Means for users to upload their own facial images" refers to a function that allows users to take or select a photo of their own face via the application and send it to the server.

[0815] "Artificial intelligence model for facial recognition" refers to an automated learning algorithm that analyzes a user's facial image and identifies specific facial features such as eye shape, eyebrow shape, and facial contours to extract features.

[0816] The "means for extracting a user's facial features" is a function that uses an artificial intelligence model for facial recognition to identify specific features from a user's facial image and extract them as data.

[0817] The "means for generating makeup techniques" is a function that integrates the makeup features of celebrities with the user's facial features and generates a step-by-step guide for a makeup method that is suitable for the user.

[0818] The "means for providing step-by-step instructions to the user" is a function of instructing the user in an easy-to-understand manner for each step of the generated makeup guide, and displaying or explaining the instructions so that the user can proceed as instructed.

[0819] "Means for capturing and analyzing a user's facial expressions in real time" refers to a function that uses a camera to capture a user's facial expressions in real time while the user is applying makeup, and analyzes the facial expression data.

[0820] "Means for recognizing emotional states and providing appropriate advice and feedback" refers to a function that analyzes captured facial expression data, determines the user's emotional state, and provides appropriate advice and supportive messages in real time.

[0821] This invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques that are suitable for the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide makeup advice and feedback according to the user's emotional state.

[0822] System Overview

[0823] This system consists of multiple hardware and software components. The main hardware components are the user's device, such as a smartphone or tablet, and a cloud server. The software components include an AI model for image recognition (e.g., DeepLabV3+), an AI model for face recognition (e.g., FaceNet), and an emotion engine (e.g., Emotion API).

[0824] Users upload images of celebrities and their own faces through the application. The device sends these images to a server, where image analysis is performed. The celebrity's makeup features and the user's facial features are analyzed and extracted, and the optimal makeup technique for the user is generated based on this. The generated makeup technique is then provided to the user step by step.

[0825] System Details

[0826] Celebrity image upload and analysis

[0827] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0828] Terminal: The terminal sends the image data selected by the user to the server using a secure protocol such as HTTPS.

[0829] Server: The server temporarily stores the received image data and inputs it into an AI model for image recognition. After preprocessing such as resizing and normalizing the image, the AI ​​model extracts the celebrity's makeup characteristics.

[0830] Uploading and analyzing user facial images

[0831] User: Next, the user uploads a photo of their face to the application.

[0832] Device: The device temporarily stores the user's facial photo locally and then sends it to the server, again using a secure protocol.

[0833] Server: The server temporarily stores the received user's face photo and inputs it into an AI model for facial recognition. After preprocessing such as cropping, resizing, and normalization, the AI ​​model extracts the user's facial features.

[0834] Creation and provision of cosmetic techniques

[0835] Server: The server generates a makeup technique suited to the user based on the makeup characteristics of celebrities and the user's facial features. The generated makeup technique is constructed as a step-by-step guide. This guide includes detailed information on the type and amount of makeup to use in each step, as well as the order and location of application.

[0836] Terminal: Receives the makeup guide generated by the server and displays it to the user. The user can follow the guide to apply makeup in the correct order.

[0837] Use of emotion engine

[0838] User: While applying makeup, the device camera captures the user's facial expressions.

[0839] Device: The device transmits the facial expression data captured in real time to the emotion engine.

[0840] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state, for example, re-explaining the steps if the user is confused, or generating an encouraging message if the user is satisfied.

[0841] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[0842] Specific examples

[0843] Specific examples are shown below.

[0844] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[0845] Terminal: The terminal sends these images to the server.

[0846] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[0847] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[0848] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[0849] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[0850] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[0851] Prompt Sentence Examples

[0852] Below are some examples of prompt sentences.

[0853] Makeup feature extraction

[0854] "Identify makeup features from the following celebrity images: eyeshadow, lip color, blush placement, etc."

[0855] Facial feature extraction

[0856] "Analyze this user's facial image to extract eye shape, eyebrow shape, and facial contours."

[0857] Use of emotion engine

[0858] "Analyze this user's facial expression to determine their current emotional state (confusion, satisfaction, etc.)"

[0859] This allows users to easily and efficiently learn and practice professional makeup techniques, and the system also provides real-time support, improving user satisfaction.

[0860] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0861] Step 1: Upload and submit a photo of your celebrity

[0862] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[0863] Terminal: The terminal temporarily stores the image data selected by the user locally and then transmits it to the server using a secure protocol such as HTTPS.

[0864] Input: celebrity image file

[0865] Output: Image data sent to the server

[0866] Step 2: Extracting celebrity makeup features

[0867] Server: The server temporarily stores the received image data. Next, it resizes and normalizes the image. After preprocessing, the image is input into an artificial intelligence model for image recognition (e.g., DeepLabV3+) to extract makeup features such as the celebrity's eyeshadow, lip color, and blush position.

[0868] Input: Resized and normalized celebrity image data

[0869] Output: Extracted makeup feature data

[0870] Step 3: Upload and send the user's face image

[0871] User: Next, the user uploads a photo of their face to the application.

[0872] Device: The device temporarily stores the user's facial photo locally and then sends it to the server using a secure protocol such as HTTPS.

[0873] Input: User's face image file

[0874] Output: Facial image data sent to the server

[0875] Step 4: Extracting the user's facial features

[0876] Server: The server temporarily stores the received user's facial image data. Then, it performs preprocessing such as cropping, resizing, and normalization of the face. After preprocessing, the facial image is input into an artificial intelligence model for facial recognition (e.g., FaceNet) to extract facial features such as eye shape, eyebrow shape, and facial contours.

[0877] Input: Resized and normalized user face image data

[0878] Output: Extracted facial feature data

[0879] Step 5: Creating makeup techniques

[0880] Server: The server combines celebrity makeup feature data with the user's facial feature data to generate the best makeup techniques for the user. During the generation process, a specific algorithm is used to map makeup features and facial features, and a step-by-step guide for the best makeup techniques for the user is constructed.

[0881] Input: celebrity makeup feature data and user facial feature data

[0882] Output: Step-by-step makeup guide

[0883] Step 6: Providing makeup tips

[0884] Device: The device receives the step-by-step makeup tutorial generated by the server and displays it to the user via the application. UI design is important for the presentation, and each step is often supplemented with images or short video clips.

[0885] User: The user follows the guide to apply makeup in the correct order.

[0886] Enter: a step-by-step makeup tutorial

[0887] Output: User's Make execution results

[0888] Step 7: Use the Emotion Engine

[0889] User: While applying makeup, the device camera captures the user's facial expressions in real time.

[0890] Device: The device sends the facial expression data captured in real time to the emotion engine (e.g., Emotion API).

[0891] Input: Captured user facial expression data

[0892] Output: Facial expression data sent to the server

[0893] Step 8: Emotion-based advice and feedback

[0894] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state. Based on the determined emotional state, the server generates appropriate advice and feedback. This feedback can include re-explaining the steps if the user is confused, or providing an encouraging message if the user is satisfied.

[0895] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[0896] Input: Parsed user emotion data

[0897] Output: Real-time advice and feedback

[0898] (Application example 2)

[0899] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0900] In the modern beauty industry, users are highly interested in replicating celebrity makeup looks on their own faces. However, there are few systems in brick-and-mortar stores that allow users to receive personalized makeup techniques in real time. Furthermore, mechanisms for providing feedback based on the user's emotional state during the makeup process are also lacking. The present invention aims to solve these issues by providing a system that provides real-time emotional advice while replicating celebrity makeup looks.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0902] In this invention, the server includes: means for a user to upload an image of a celebrity; means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features; means for a user to upload his or her own facial image; means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features; means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user; means for providing the generated makeup technique to the user step by step; and means for recognizing the user's emotional state in real time and providing advice and feedback according to the user's emotions. This enables users to recreate celebrity makeup looks even in physical stores, and further enables users to achieve higher satisfaction by receiving feedback according to their emotions during the makeup process.

[0903] "User" refers to someone who uses the system to recreate celebrity makeup looks.

[0904] A "celebrity image" is a photograph of a famous person's face that is used to extract the person's makeup features.

[0905] An "artificial intelligence model for image recognition" is an artificial intelligence program that detects and analyzes specific features from input images.

[0906] "Cosmetic features" refer to specific elements of makeup such as eyeshadow, lip color, and blush placement.

[0907] A "face image" is a photograph of the user's face that is uploaded to the system.

[0908] An "artificial intelligence model for facial recognition" is an artificial intelligence program that extracts facial features such as eye shape, eyebrow shape, and facial contours from an input facial image.

[0909] "Facial features" refer to the shape of the user's face and the characteristics of each part of the face.

[0910] "Makeup techniques" refer to makeup techniques that are suitable for a user and are generated by combining the makeup features of celebrities with the user's facial features.

[0911] "Providing step-by-step instructions" means providing guidance so that the user can perform the generated makeup techniques in a step-by-step manner.

[0912] "Emotional state" refers to the emotion expressed by the user while applying makeup, and indicates states such as satisfaction or confusion.

[0913] "Providing advice and feedback" means notifying the user of appropriate instructions or encouraging messages depending on the user's emotional state.

[0914] In this invention, we will build a system that provides users with a specific method for recreating celebrity makeup on their own face. The overall configuration of the system consists of a process in which users upload images of celebrities and are provided with makeup techniques that are suitable for the user based on the analysis results. A detailed explanation of the system is provided below.

[0915] Hardware and software used

[0916] Hardware:

[0917] Smart glasses or head-mounted displays (HMD)

[0918] Server (with high-performance processor and sufficient storage capacity)

[0919] Cameras (built into smart glasses or HMDs)

[0920] software:

[0921] Artificial intelligence models for image recognition: OpenCV, TensorFlow

[0922] Artificial intelligence models for face recognition: Dlib, FaceNet

[0923] Emotion engine: Microsoft Azure Emotion API

[0924] Process Overview

[0925] The system mainly includes the following processing steps:

[0926] 1. A user uploads an image of a celebrity.

[0927] 2. The server inputs this image into an artificial intelligence model to extract makeup features.

[0928] 3. The user uploads a photo of their face.

[0929] 4. The server inputs the user's facial image into an artificial intelligence model and extracts facial features.

[0930] 5. The server generates the optimal makeup techniques based on the celebrity's makeup characteristics and the user's facial features.

[0931] 6. Display a step-by-step makeup application guide on smart glasses or an HMD.

[0932] 7. Recognize the user's emotional state in real time and provide advice and feedback according to that state.

[0933] Program processing explanation

[0934] User face scan:

[0935] When a user wears the smart glasses, the camera scans the user's face in real time. OpenCV preprocesses the image (resizing, normalizing), and then Dlib and FaceNet are used to extract facial features. The extracted facial features are sent to the server and stored as facial feature data.

[0936] Celebrity makeup feature extraction:

[0937] Users upload images of celebrities, and the server inputs the images into a TensorFlow-based artificial intelligence model that analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[0938] Makeup Technique Creation:

[0939] The server generates optimal makeup techniques based on the makeup characteristics of celebrities and the user's facial features. The generated makeup techniques are then constructed as a step-by-step guide for the user to follow.

[0940] Emotional feedback:

[0941] The camera captures the user's facial expressions while applying makeup and inputs them into the emotion engine. Using the Microsoft Azure Emotion API, the system analyzes the user's emotional state in real time, providing re-explanations if the user is confused, and positive feedback if the user is satisfied.

[0942] Specific examples

[0943] 1. User flow:

[0944] A user visits a cosmetics store, puts on the smart glasses, selects a makeup image of a celebrity they like, and then captures and uploads a photo of their own face.

[0945] 2. Program processing example:

[0946] Celebrity image analysis prompt:

[0947] "Analyze and extract the location of eyeshadow, lip color, and blush in this image."

[0948] Sentiment Analysis Prompt:

[0949] "Please recognize the emotion from the facial expression in this captured image and provide appropriate advice."

[0950] In this way, the system can provide users with a highly satisfying and effective makeup experience by recreating celebrity makeup looks while providing real-time emotional feedback.

[0951] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0952] Step 1:

[0953] A user uploads an image of a celebrity.

[0954] Input: A user-selected celebrity face image.

[0955] How it works: The user launches the application and selects an image of a celebrity. The application then sends the selected image data from the device to the server.

[0956] Output: Celebrity face image data uploaded to the server.

[0957] Step 2:

[0958] The server inputs images of celebrities into an image recognition AI model and extracts the celebrities' makeup features.

[0959] Input: Uploaded celebrity face image.

[0960] How it works: The server preprocesses the received image data (resizing, normalizing) and inputs it into a TensorFlow-based artificial intelligence model for image recognition. The model analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[0961] Output: Extracted celebrity makeup feature data.

[0962] Step 3:

[0963] The user uploads a picture of their face.

[0964] Input: A photo of the user's face.

[0965] Operation: The user uploads a photo of their face to the application, and the device sends the photo to the server.

[0966] Output: User's facial image data uploaded to the server.

[0967] Step 4:

[0968] The server inputs the user's facial image into a facial recognition AI model and extracts the user's facial features.

[0969] Input: Uploaded user face image.

[0970] How it works: The server preprocesses (resizes, normalizes) the received facial images and feeds them into a Dlib or FaceNet-based AI model for facial recognition, which extracts facial features such as eye shape, eyebrow shape, and facial contours.

[0971] Output: Extracted user facial feature data.

[0972] Step 5:

[0973] The server generates the most suitable makeup technique based on the makeup features of celebrities and the user's facial features.

[0974] Input: celebrity makeup feature data and user facial feature data.

[0975] How it works: The server combines both sets of data and generates a makeup application tailored to the user, which is structured as a step-by-step guide.

[0976] Output: Generated step-by-step makeup tutorial.

[0977] Step 6:

[0978] The terminal provides the generated makeup technique guide to the user.

[0979] Enter: a step-by-step makeup artistry guide.

[0980] How it works: The server sends the generated guide to the device, which displays it to the user. The user can view the guide step by step through smart glasses or a head-mounted display.

[0981] Output: A step-by-step makeup tutorial provided to the user.

[0982] Step 7:

[0983] It recognizes the user's emotional state in real time and provides advice and feedback according to that emotion.

[0984] Input: User facial expression data captured by the camera.

[0985] How it works: The device captures the user's facial expressions in real time and sends the data to an emotion engine (Microsoft Azure Emotion API). The server analyzes the changes in facial expressions and determines the user's emotional state (confusion, satisfaction, etc.). It then generates appropriate feedback and advice and sends it to the device.

[0986] Output: Advice or feedback based on the user's emotional state.

[0987] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0988] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0989] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0990] [Third embodiment]

[0991] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0992] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0993] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0994] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0995] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0996] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0997] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0998] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0999] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1000] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1001] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1002] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1003] The present invention relates to a system that allows users to recreate the makeup of their favorite celebrities on their own faces. This system analyzes images of celebrities and the user's facial images, and generates and provides makeup techniques that are suitable for the user based on the makeup characteristics. Specific embodiments of the present invention are described below.

[1004] System Overview

[1005] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on their own facial image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides users with step-by-step instructions on how to apply these techniques, and allows them to share the results with other users.

[1006] Program processing explanation

[1007] Celebrity image upload and analysis

[1008] User: The user first opens the application and uploads a picture of their favorite celebrity.

[1009] Terminal: The terminal sends image data to the server.

[1010] Makeup feature extraction

[1011] Server: The server inputs the received image into an AI model for image recognition and analyzes the celebrity's makeup features, thereby identifying detailed makeup elements such as eyeshadow, lip color, and blush placement.

[1012] Uploading and analyzing user facial images

[1013] User: Next, the user uploads a photo of their face to the application.

[1014] Terminal: The terminal sends this facial photo data to the server.

[1015] Facial feature extraction

[1016] Server: The server inputs the user's facial image into an AI model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[1017] Creating makeup techniques

[1018] Server: The server generates the best makeup tips for the user based on the makeup features of celebrities and the user's facial features. This makeup tip is constructed as a step-by-step guide that the user can follow.

[1019] Providing makeup techniques

[1020] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1021] Specific examples

[1022] The following explains this with specific examples.

[1023] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1024] Terminal: The terminal sends these images to the server.

[1025] Server: The server analyzes the makeup features of the model and the user's facial features. Based on this, it generates makeup techniques that are suitable for the user.

[1026] Specific makeup techniques are provided as follows:

[1027] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1028] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1029] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1030] Users: Users can follow the guide to apply makeup, take a photo of the results, and upload them to the application. They can also share their makeup results with other users.

[1031] This embodiment allows users to easily receive professional makeup advice suited to their facial features. In addition, the ability to provide additional advice in real time and share with other users can provide a more satisfying makeup experience.

[1032] The processing flow will be explained below.

[1033] Step 1:

[1034] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1035] Step 2:

[1036] Device: The device sends the uploaded celebrity image to the server.

[1037] Step 3:

[1038] Server: The server inputs the received celebrity images into an AI model for image recognition.

[1039] Step 4:

[1040] Server: The server performs pre-processing on the images, including image resizing, normalization, etc.

[1041] Step 5:

[1042] Server: The server uses an AI model to extract the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[1043] Step 6:

[1044] Server: The server stores the extracted makeup features in a database.

[1045] Step 7:

[1046] User: The user then uploads a photo of their face to the application.

[1047] Step 8:

[1048] Terminal: The terminal sends the user's facial photo to the server.

[1049] Step 9:

[1050] Server: The server inputs the received user's facial photo into an AI model for facial recognition.

[1051] Step 10:

[1052] Server: The server performs preprocessing on the face photo, including cropping, resizing, and normalizing the face.

[1053] Step 11:

[1054] Server: The server uses an AI model to extract the user's facial features, such as eye shape, eyebrow shape, and facial contours.

[1055] Step 12:

[1056] Server: The server stores the extracted facial features in a database.

[1057] Step 13:

[1058] Server: The server combines the celebrity's makeup features with the user's facial features to generate makeup techniques suited to the user.

[1059] Step 14:

[1060] Sarver: Sarver creates detailed guides that organize makeup techniques into a step-by-step format.

[1061] Step 15:

[1062] Server: The server sends the generated makeup technique guide to the terminal.

[1063] Step 16:

[1064] Terminal: The terminal displays the makeup technique guide to the user.

[1065] Step 17:

[1066] User: The user follows the displayed guide and actually applies the makeup.

[1067] Step 18:

[1068] User: The user uploads a photo of the completed makeup look to the application.

[1069] Step 19:

[1070] Terminal: The terminal sends the finished photo to the server.

[1071] Step 20:

[1072] Server: The server receives the finished photos and stores them in a database so that they can be shared with other users if desired.

[1073] Example 1

[1074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] Conventional makeup instruction systems have struggled to provide detailed makeup guides tailored to each user's individual facial features. Furthermore, they lacked specific steps for faithfully recreating famous makeup styles on the user's face, making it difficult for users to find a makeup method that best suits their own features. Furthermore, they lacked real-time advice and the ability to share with other users, preventing them from providing a satisfying makeup experience.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1077] In this invention, the server includes means for allowing a user to upload an image of a famous person, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the makeup features of the famous person, means for allowing the user to upload their own facial image, means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the makeup features of the famous person with the user's facial features to generate makeup techniques suitable for the user, means for inputting the generated makeup techniques as prompt sentences into the artificial intelligence model to generate a step-by-step guide, and means for providing the generated makeup guide to a user terminal. This allows users to receive professional makeup advice optimized for their facial features in real time. Furthermore, users can share the results of their makeup with other users, providing a fulfilling makeup experience.

[1078] A "famous person" is a widely known public figure or prominent individual.

[1079] An "artificial intelligence model for image recognition" is a learning algorithm used to analyze image data and detect specific features.

[1080] An "artificial intelligence model for facial recognition" is a learning algorithm that analyzes a user's facial image and identifies the shape and placement of specific parts of the face.

[1081] "Makeup features" refers to specific makeup elements and their placement, such as the color around the eyes, the color of the lips, and the position of the cheeks.

[1082] A "prompt" is text data that is input to a generative artificial intelligence model and is used to provide specific instructions or information to generate a specific output.

[1083] A "generative artificial intelligence model" is a learning algorithm used to generate new text or instructions based on prompts and other data.

[1084] A "step-by-step guide" is a sequence of detailed instructions that a user can follow in order.

[1085] "User terminal" refers to an electronic device used by a user to operate and view information, such as a smartphone or a personal computer.

[1086] The present invention relates to a system that allows a user to recreate makeup looks of famous people on their own face. This system analyzes makeup features using an image of a famous person selected by the user and an image of the user's own face, and generates and provides the user with makeup techniques that are optimal for the user. Specific embodiments for implementing the present invention are described below.

[1087] First, a user opens the application and uploads an image of a famous person. The device receives the image and sends it to the server. The server then inputs the received image into an artificial intelligence model for image recognition (such as OpenCV or TensorFlow) to analyze and detect detailed makeup features such as eyeshadow, lip color, and blush placement on the famous person.

[1088] Next, the user uploads a photo of their face to the application. The device sends this photo data to the server, which then inputs the user's face image into an artificial intelligence model for facial recognition (e.g., Dlib or FaceNet) to analyze facial features such as eye shape, eyebrow shape, and facial contours.

[1089] The server then generates the best makeup tips for the user based on the makeup features of famous people and the user's facial features. This makeup tip is input as a prompt into a generative AI model (e.g., GPT-4) to generate a step-by-step guide. Specific examples of prompts are as follows:

[1090] "Generate the following step-by-step makeup guide for a facial image of user X, who has uploaded an image of famous person A. Include detailed instructions for each element of eyeshadow, lipstick, and blush."

[1091] The generated makeup guide is sent from the server to the user's device, which displays it to the user. The user can follow this guide to apply makeup step by step. After completing their makeup, the user can upload a photo of their finished makeup to the application and share it with other users. This function allows users to compare makeup looks and receive advice from each other, thereby enriching the makeup experience.

[1092] As described above, the present invention allows users to receive professional makeup advice tailored to their facial features in real time. Furthermore, by sharing the results of their makeup with other users, users can enjoy a more satisfying makeup experience.

[1093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] User: The user opens the application, selects an image of a famous person and presses the upload button.

[1096] Input: Image files (e.g. JPEG, PNG) of famous people.

[1097] Output: Image file temporarily saved on the device.

[1098] Specific operation: The user operates the application and selects an image of a famous person from the gallery or camera. This image is temporarily stored on the device.

[1099] Step 2:

[1100] Device: The device sends the image of the celebrity uploaded by the user to the server.

[1101] Input: Image file in the device.

[1102] Output: Image data sent to the server.

[1103] What happens: The device sends the saved image file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[1104] Step 3:

[1105] Server: The server inputs the received image of the famous person into an artificial intelligence model for image recognition and analyzes makeup features such as eye shadow, lip color, and blush position.

[1106] Input: Received image data.

[1107] Output: Analyzed makeup feature data (e.g. eyeshadow color, lip color, blush position, etc.).

[1108] Specific operation: The server analyzes the image using an artificial intelligence model for image recognition (e.g., TensorFlow, OpenCV). The makeup feature data extracted as a result of the analysis is stored in a database.

[1109] Step 4:

[1110] User: The user uploads a photo of their face to the application.

[1111] Input: User's face photo file (e.g. JPEG, PNG).

[1112] Output: A face photo file temporarily saved on the device.

[1113] Specific operation: The user operates the application and selects a photo of their face from the gallery or camera. This photo is temporarily stored on the device.

[1114] Step 5:

[1115] Terminal: The terminal sends the user's facial photo to the server.

[1116] Input: Face photo file in the device.

[1117] Output: Facial photo data sent to the server.

[1118] What it does: The device sends the stored face photo file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[1119] Step 6:

[1120] Server: The server inputs the received user's facial photo into an artificial intelligence model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[1121] Input: Received facial photo data.

[1122] Output: Analyzed facial feature data (e.g. eye shape, eyebrow shape, facial contour, etc.).

[1123] Specific operation: The server analyzes the facial image using an AI model for facial recognition (e.g., Dlib, FaceNet). The facial feature data extracted as the analysis result is stored in a database.

[1124] Step 7:

[1125] Server: The server generates makeup prompt sentences based on the makeup features of famous people and the user's facial features, and inputs them into the generative artificial intelligence model.

[1126] Input: Makeup feature data of famous people and facial feature data of the user.

[1127] Output: Step-by-step makeup guide.

[1128] Specific operation: The server combines both sets of data to generate a prompt. For example, "Generate the following step-by-step makeup guide for the face image of user X, who uploaded an image of famous person A. Please include detailed instructions for each element: eyeshadow, lipstick, and blush." ​​This prompt is then input into a generative AI model (e.g., GPT-4) to generate a specific makeup guide.

[1129] Step 8:

[1130] Terminal: The makeup guide generated by the server is provided to the terminal, which displays it to the user.

[1131] Input: The generated step-by-step makeup guide.

[1132] Output: The makeup guide displayed on the terminal by the user.

[1133] Specific operation: The terminal displays the makeup guide received from the server. The user follows this guide to perform specific makeup steps.

[1134] Step 9:

[1135] User: After applying makeup, the user uploads a photo of the finished look to the application and shares it with other users.

[1136] Input: Finished photo file (e.g. JPEG, PNG).

[1137] Output: Uploaded finished photo.

[1138] Specific operation: The user presses the upload button, selects a photo of the completed makeup look, and uploads it to the application. The application sends it to the server and shares it with other users.

[1139] (Application example 1)

[1140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] When users try to recreate celebrity makeup looks on their own faces, existing methods that suggest appropriate makeup techniques struggle to provide real-time advice or generate specific prompts. Furthermore, there are limited ways to share the makeup results with others or receive feedback. This makes it difficult to improve users' makeup skills and increase their satisfaction.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1143] In this invention, the server includes means for users to upload images of celebrities, means for inputting the uploaded images into a generative AI model for image recognition and detecting the celebrity's makeup features, means for inputting the uploaded user's facial image into a generative AI model for face recognition and extracting the user's facial features, means for providing a user interface for displaying the generated makeup technique guide, means for generating prompt sentences for guiding users through the steps of applying the suggested makeup techniques, and means for sharing the results with other people. This allows users to receive appropriate makeup techniques along with specific steps in real time, and also allows them to share their makeup results with others to receive feedback.

[1144] 1. "User" refers to a person who uses this system to recreate celebrity makeup on their own face.

[1145] 2. "Celebrity images" refers to visual data, such as photographs or illustrations, that show the face of a celebrity.

[1146] 3. "Upload" refers to sending data from a particular device to a remote storage system, such as a server.

[1147] 4. "Generative AI model for image recognition" refers to an artificial intelligence model that processes image data to recognize and extract specific features.

[1148] 5. "Makeup features" refers to specific features of makeup applied to a celebrity's face, such as makeup patterns and colors.

[1149] 6. "Facial Image" refers to photographs or visual data that show a user's face.

[1150] 7. “Generative AI model for facial recognition” means an artificial intelligence model used to analyze facial features from facial images.

[1151] 8. "Facial Features" refers to specific external features of a user's face, such as the eyes, nose, and mouth.

[1152] 9. "Makeup Technique" refers to the specific makeup methods and procedures applied to the user's face.

[1153] 10. "Step-by-step means" refers to a method that provides users with specific, step-by-step instructions.

[1154] 11. "User interface" refers to the screens and methods of operation that allow a user to interact with a system.

[1155] 12. "Prompt sentences" refers to a series of sentences that provide the user with makeup application instructions or advice in text form.

[1156] 13. "Means for sharing results with others" refers to a function that allows users to show their makeup results to other users via the Internet or other means and receive their evaluations.

[1157] This invention relates to a system that allows users to recreate celebrity makeup on their own face. The system analyzes images of celebrities and the user's facial images, and uses a generative AI model to provide the user with makeup techniques suited to their needs.

[1158] System Overview

[1159] This system is implemented with the following hardware and software configuration. The user's smartphone or tablet is used as the device, and the server hosts the AI ​​model as a cloud service. The main software includes the OpenCV library for image processing and the Keras framework for using deep learning models.

[1160] System Operation

[1161] Celebrity image upload and analysis

[1162] User: First, the user uploads an image of their favorite celebrity from their device in order to recreate that celebrity's makeup.

[1163] Device: The device sends uploaded images to a server, which then inputs them into a generative AI model for image recognition.

[1164] Server: The server uses a generative AI model to analyze the image and extract the celebrity's makeup features.

[1165] Uploading and analyzing user facial images

[1166] User: Next, the user uploads a photo of their face from their device.

[1167] Terminal: The terminal sends this facial photo to the server.

[1168] Server: The server analyzes the user's facial features using a generative AI model for facial recognition.

[1169] Creating and providing makeup techniques

[1170] Server: The server combines celebrity makeup features with the user's facial features to generate makeup techniques suited to the user. The generated makeup techniques are structured as a step-by-step guide and are presented through a user interface.

[1171] Terminal: The terminal displays the generated makeup techniques to the user and provides prompts on how to apply the proposed makeup techniques.

[1172] Usage example

[1173] Suppose a user wants to apply a makeup style to their face using a famous model. The user first uploads an image of the model to the application, then takes and uploads a photo of their own face. The server analyzes these images and suggests the best makeup techniques by combining the celebrity's makeup style with the user's facial features. The user can then apply the makeup step by step by following the prompts displayed on the smartphone screen.

[1174] Prompt Sentence Examples

[1175] The prompt is presented to the user in the following format:

[1176] "Upload an image of your favorite celebrity and take a photo of yourself. The system will analyze the image and suggest the best makeup techniques."

[1177] This allows users to easily receive professional makeup advice realized using advanced machine learning technology. Furthermore, the ability to provide additional advice in real time and share with others provides users with a more satisfying makeup experience.

[1178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1179] Step 1:

[1180] User: The user uploads an image of their favorite celebrity to the application using their smartphone or tablet. This input image data is the initial data for analyzing the celebrity's makeup features.

[1181] Step 2:

[1182] Device: The device sends the celebrity image uploaded by the user to the server, where it is converted into the appropriate format (e.g., JPEG or PNG).

[1183] Step 3:

[1184] Server: The server inputs the received images into a generative AI model for image recognition. The AI ​​model analyzes the makeup features in the image (e.g., eyeshadow color, lip color, blush position) and extracts specific features. The output is a dataset showing celebrity makeup features.

[1185] Step 4:

[1186] User: Next, the user uploads a photo of their face using the same application. This photo data becomes the base data for analyzing the user's facial features.

[1187] Step 5:

[1188] Terminal: The terminal converts the user's facial photo into a specified format and sends it to the server.

[1189] Step 6:

[1190] Server: The server inputs the received user's face photo into a generative AI model for facial recognition. The AI ​​model analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.) and generates facial feature data as output.

[1191] Step 7:

[1192] Server: The server combines and analyzes celebrity makeup feature data with the user's facial feature data to generate the optimal makeup technique for the user. This makeup technique is constructed as a step-by-step guide. The output is a list of specific makeup steps and cosmetic items to use.

[1193] Step 8:

[1194] Terminal: The terminal displays the step-by-step makeup guide received from the server to the user, who can then apply makeup by following the guide.

[1195] Step 9:

[1196] Server: Based on the generated makeup technique guide, the server generates a prompt sentence to guide the user through the makeup application procedure. This prompt sentence is displayed on the GUI screen.

[1197] Step 10:

[1198] User: The user applies makeup using the provided guide and prompts, then takes a photo of the completed makeup and uploads it to the application.

[1199] Step 11:

[1200] Device: The device sends the photos of the makeup results uploaded by the user to the server and stores them on the platform for sharing with other users.

[1201] Through these steps, users can recreate the makeup of their favorite celebrities on their own faces and share the results with others.

[1202] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1203] The present invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques suited to the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, the present invention can provide makeup advice and feedback according to the user's emotional state. Specific embodiments of the present invention are described below.

[1204] System Overview

[1205] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on the image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides the user with step-by-step instructions on how to apply the makeup, and allows the results to be shared with other users. It also uses an emotion engine to recognize the user's emotions and provides advice and feedback based on those emotions.

[1206] Program processing explanation

[1207] Celebrity image upload and analysis

[1208] User: The user launches the application, selects an image of their favorite celebrity, and uploads it.

[1209] Terminal: The terminal transmits the selected image data to the server.

[1210] Makeup feature extraction

[1211] Server: The server inputs the received celebrity image into an AI model for image recognition. As preprocessing, the image is resized and normalized. The AI ​​model then extracts the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[1212] Uploading and analyzing user facial images

[1213] User: Next, the user uploads a photo of their face to the application.

[1214] Terminal: The terminal sends the user's facial photo to the server.

[1215] Facial feature extraction

[1216] Server: The server inputs the user's facial image into the AI ​​model for facial recognition. After preprocessing such as cropping, resizing, and normalizing the face, the AI ​​model extracts facial features such as eye shape, eyebrow shape, and facial contours.

[1217] Creating makeup techniques

[1218] Server: The server generates makeup techniques suitable for the user based on the makeup features of celebrities and the user's facial features. The generated makeup techniques are constructed as a step-by-step guide that the user can follow.

[1219] Providing makeup techniques

[1220] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1221] Use of emotion engine

[1222] User: While applying makeup, the user's facial expressions are captured and analyzed by a camera to recognize emotions in real time.

[1223] Device: The device sends facial expression data captured by the camera to the emotion engine.

[1224] Server: The emotion engine analyzes facial expressions to determine the user's emotional state, for example, whether they are confused or happy.

[1225] Emotion-based advice and feedback

[1226] Server: Based on the emotional state recognized by the emotion engine, the server provides the user with appropriate advice and feedback. If the user is confused, the server provides additional support, such as re-explaining the steps, and if the user is satisfied, the server displays messages of praise and encouragement.

[1227] Device: The device displays real-time advice and feedback to the user, providing enhanced support during the makeup application process.

[1228] Specific examples

[1229] The following explains this with specific examples.

[1230] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1231] Terminal: The terminal sends these images to the server.

[1232] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[1233] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1234] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1235] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1236] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[1237] This embodiment allows users to easily receive professional makeup advice suited to their facial features. Furthermore, the function of providing additional advice in real time and feedback according to emotions can provide a more satisfying makeup experience.

[1238] The processing flow will be explained below.

[1239] Step 1:

[1240] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1241] Step 2:

[1242] Terminal: The terminal sends the image data of the selected celebrity to the server.

[1243] Step 3:

[1244] Server: The server preprocesses the received celebrity images, specifically resizing and normalizing the images.

[1245] Step 4:

[1246] Server: The server inputs preprocessed images into an AI model for image recognition and extracts makeup features such as eyeshadow, lip color, and blush position.

[1247] Step 5:

[1248] Server: The server stores the extracted makeup features in a database.

[1249] Step 6:

[1250] User: Next, the user uploads a photo of their face to the application.

[1251] Step 7:

[1252] Terminal: The terminal sends the user's facial photo data to the server.

[1253] Step 8:

[1254] Server: The server preprocesses the received user face photo, specifically cropping, resizing, and normalizing the face.

[1255] Step 9:

[1256] Server: The server inputs preprocessed facial photos into an AI model for facial recognition, extracting facial features such as eye shape, eyebrow shape, and facial contours.

[1257] Step 10:

[1258] Server: The server stores the extracted facial features in a database.

[1259] Step 11:

[1260] Server: The server generates makeup techniques suitable for the user based on the makeup characteristics of celebrities and the user's facial features.

[1261] Step 12:

[1262] Server: The server builds the generated makeup techniques as a step-by-step guide.

[1263] Step 13:

[1264] Server: The server sends the guide to the device.

[1265] Step 14:

[1266] Terminal: The terminal displays the makeup technique guide to the user.

[1267] Step 15:

[1268] User: The user follows the displayed guide and applies makeup in the correct order.

[1269] Step 16:

[1270] Terminal: The terminal captures the user's facial expressions while applying makeup with a camera and sends the facial expression data to the server.

[1271] Step 17:

[1272] Server: The server uses an emotion engine to analyze the captured facial expression data and recognize the user's emotions.

[1273] Step 18:

[1274] Server: Based on emotion recognition, adjust makeup steps and advice as needed. For example, if the user is confused, restate the steps or suggest an alternative approach.

[1275] Step 19:

[1276] Server: Generates positive feedback such as encouragement or praise according to the user's emotional state and sends it to the device.

[1277] Step 20:

[1278] Terminal: The terminal displays feedback and advice received from the server to the user in real time, supporting the makeup experience.

[1279] Step 21:

[1280] User: After the user has completed their makeup, they take a photo of the finished look and upload it to the application.

[1281] Step 22:

[1282] Terminal: The terminal sends the finished photo to the server.

[1283] Step 23:

[1284] Server: The server stores the finished photos in a database and makes them available for sharing with other users as needed.

[1285] Example 2

[1286] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1287] Conventional makeup advice systems are required to not only detect celebrity makeup features, but also generate makeup techniques suitable for the user and provide them as a practical, concrete, step-by-step guide. However, these systems lack real-time advice and feedback based on the user's emotional state, which can lead to confusion and reduced user satisfaction. A new system that solves these problems is needed.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1289] In this invention, the server includes means for a user to upload an image of a celebrity, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features, means for a user to upload his or her own facial image, means for inputting the uploaded user's facial image into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user, means for providing the generated makeup technique to the user step by step, means for capturing and analyzing the user's facial expressions in real time, and means for recognizing the user's emotional state based on the results of the facial expression analysis and providing appropriate advice and feedback. This not only enables the user to more effectively recreate celebrity makeup, but also enables a satisfying makeup experience without confusion due to the real-time emotion recognition and feedback.

[1290] The "means for uploading images of celebrities" is a function that allows a user to select an image of a celebrity through an application and transmit the image data to a server via a network.

[1291] "Artificial intelligence model for image recognition" refers to an automated learning algorithm that analyzes received image data and extracts specific features from it, such as cosmetic features like eyeshadow, lip color, and blush placement.

[1292] The "means for detecting makeup features" is a function that uses an artificial intelligence model for image recognition to identify specific makeup elements from images of celebrities.

[1293] "Means for users to upload their own facial images" refers to a function that allows users to take or select a photo of their own face via the application and send it to the server.

[1294] "Artificial intelligence model for facial recognition" refers to an automated learning algorithm that analyzes a user's facial image and identifies specific facial features such as eye shape, eyebrow shape, and facial contours to extract features.

[1295] The "means for extracting a user's facial features" is a function that uses an artificial intelligence model for facial recognition to identify specific features from a user's facial image and extract them as data.

[1296] The "means for generating makeup techniques" is a function that integrates the makeup features of celebrities with the user's facial features and generates a step-by-step guide for a makeup method that is suitable for the user.

[1297] The "means for providing step-by-step instructions to the user" is a function of instructing the user in an easy-to-understand manner for each step of the generated makeup guide, and displaying or explaining the instructions so that the user can proceed as instructed.

[1298] "Means for capturing and analyzing a user's facial expressions in real time" refers to a function that uses a camera to capture a user's facial expressions in real time while the user is applying makeup, and analyzes the facial expression data.

[1299] "Means for recognizing emotional states and providing appropriate advice and feedback" refers to a function that analyzes captured facial expression data, determines the user's emotional state, and provides appropriate advice and supportive messages in real time.

[1300] This invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques that are suitable for the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide makeup advice and feedback according to the user's emotional state.

[1301] System Overview

[1302] This system consists of multiple hardware and software components. The main hardware components are the user's device, such as a smartphone or tablet, and a cloud server. The software components include an AI model for image recognition (e.g., DeepLabV3+), an AI model for face recognition (e.g., FaceNet), and an emotion engine (e.g., Emotion API).

[1303] Users upload images of celebrities and their own faces through the application. The device sends these images to a server, where image analysis is performed. The celebrity's makeup features and the user's facial features are analyzed and extracted, and the optimal makeup technique for the user is generated based on this. The generated makeup technique is then provided to the user step by step.

[1304] System Details

[1305] Celebrity image upload and analysis

[1306] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1307] Terminal: The terminal sends the image data selected by the user to the server using a secure protocol such as HTTPS.

[1308] Server: The server temporarily stores the received image data and inputs it into an AI model for image recognition. After preprocessing such as resizing and normalizing the image, the AI ​​model extracts the celebrity's makeup characteristics.

[1309] Uploading and analyzing user facial images

[1310] User: Next, the user uploads a photo of their face to the application.

[1311] Device: The device temporarily stores the user's facial photo locally and then sends it to the server, again using a secure protocol.

[1312] Server: The server temporarily stores the received user's face photo and inputs it into an AI model for facial recognition. After preprocessing such as cropping, resizing, and normalization, the AI ​​model extracts the user's facial features.

[1313] Creation and provision of cosmetic techniques

[1314] Server: The server generates a makeup technique suited to the user based on the makeup characteristics of celebrities and the user's facial features. The generated makeup technique is constructed as a step-by-step guide. This guide includes detailed information on the type and amount of makeup to use in each step, as well as the order and location of application.

[1315] Terminal: Receives the makeup guide generated by the server and displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1316] Use of emotion engine

[1317] User: While applying makeup, the device camera captures the user's facial expressions.

[1318] Device: The device transmits the facial expression data captured in real time to the emotion engine.

[1319] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state, for example, re-explaining the steps if the user is confused, or generating an encouraging message if the user is satisfied.

[1320] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[1321] Specific examples

[1322] Specific examples are shown below.

[1323] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1324] Terminal: The terminal sends these images to the server.

[1325] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[1326] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1327] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1328] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1329] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[1330] Prompt Sentence Examples

[1331] Below are some examples of prompt sentences.

[1332] Makeup feature extraction

[1333] "Identify makeup features from the following celebrity images: eyeshadow, lip color, blush placement, etc."

[1334] Facial feature extraction

[1335] "Analyze this user's facial image to extract eye shape, eyebrow shape, and facial contours."

[1336] Use of emotion engine

[1337] "Analyze this user's facial expression to determine their current emotional state (confusion, satisfaction, etc.)"

[1338] This allows users to easily and efficiently learn and practice professional makeup techniques, and the system also provides real-time support, improving user satisfaction.

[1339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1340] Step 1: Upload and submit a photo of your celebrity

[1341] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1342] Terminal: The terminal temporarily stores the image data selected by the user locally and then transmits it to the server using a secure protocol such as HTTPS.

[1343] Input: celebrity image file

[1344] Output: Image data sent to the server

[1345] Step 2: Extracting celebrity makeup features

[1346] Server: The server temporarily stores the received image data. Next, it resizes and normalizes the image. After preprocessing, the image is input into an artificial intelligence model for image recognition (e.g., DeepLabV3+) to extract makeup features such as the celebrity's eyeshadow, lip color, and blush position.

[1347] Input: Resized and normalized celebrity image data

[1348] Output: Extracted makeup feature data

[1349] Step 3: Upload and send the user's face image

[1350] User: Next, the user uploads a photo of their face to the application.

[1351] Device: The device temporarily stores the user's facial photo locally and then sends it to the server using a secure protocol such as HTTPS.

[1352] Input: User's face image file

[1353] Output: Facial image data sent to the server

[1354] Step 4: Extracting the user's facial features

[1355] Server: The server temporarily stores the received user's facial image data. Then, it performs preprocessing such as cropping, resizing, and normalization of the face. After preprocessing, the facial image is input into an artificial intelligence model for facial recognition (e.g., FaceNet) to extract facial features such as eye shape, eyebrow shape, and facial contours.

[1356] Input: Resized and normalized user face image data

[1357] Output: Extracted facial feature data

[1358] Step 5: Creating makeup techniques

[1359] Server: The server combines celebrity makeup feature data with the user's facial feature data to generate the best makeup techniques for the user. During the generation process, a specific algorithm is used to map makeup features and facial features, and a step-by-step guide for the best makeup techniques for the user is constructed.

[1360] Input: celebrity makeup feature data and user facial feature data

[1361] Output: Step-by-step makeup guide

[1362] Step 6: Providing makeup tips

[1363] Device: The device receives the step-by-step makeup tutorial generated by the server and displays it to the user via the application. UI design is important for the presentation, and each step is often supplemented with images or short video clips.

[1364] User: The user follows the guide to apply makeup in the correct order.

[1365] Enter: a step-by-step makeup tutorial

[1366] Output: User's Make execution results

[1367] Step 7: Use the Emotion Engine

[1368] User: While applying makeup, the device camera captures the user's facial expressions in real time.

[1369] Device: The device sends the facial expression data captured in real time to the emotion engine (e.g., Emotion API).

[1370] Input: Captured user facial expression data

[1371] Output: Facial expression data sent to the server

[1372] Step 8: Emotion-based advice and feedback

[1373] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state. Based on the determined emotional state, the server generates appropriate advice and feedback. This feedback can include re-explaining the steps if the user is confused, or providing an encouraging message if the user is satisfied.

[1374] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[1375] Input: Parsed user emotion data

[1376] Output: Real-time advice and feedback

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1379] In the modern beauty industry, users are highly interested in replicating celebrity makeup looks on their own faces. However, there are few systems in brick-and-mortar stores that allow users to receive personalized makeup techniques in real time. Furthermore, mechanisms for providing feedback based on the user's emotional state during the makeup process are also lacking. The present invention aims to solve these issues by providing a system that provides real-time emotional advice while replicating celebrity makeup looks.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1381] In this invention, the server includes: means for a user to upload an image of a celebrity; means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features; means for a user to upload his or her own facial image; means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features; means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user; means for providing the generated makeup technique to the user step by step; and means for recognizing the user's emotional state in real time and providing advice and feedback according to the user's emotions. This enables users to recreate celebrity makeup looks even in physical stores, and further enables users to achieve higher satisfaction by receiving feedback according to their emotions during the makeup process.

[1382] "User" refers to someone who uses the system to recreate celebrity makeup looks.

[1383] A "celebrity image" is a photograph of a famous person's face that is used to extract the person's makeup features.

[1384] An "artificial intelligence model for image recognition" is an artificial intelligence program that detects and analyzes specific features from input images.

[1385] "Cosmetic features" refer to specific elements of makeup such as eyeshadow, lip color, and blush placement.

[1386] A "face image" is a photograph of the user's face that is uploaded to the system.

[1387] An "artificial intelligence model for facial recognition" is an artificial intelligence program that extracts facial features such as eye shape, eyebrow shape, and facial contours from an input facial image.

[1388] "Facial features" refer to the shape of the user's face and the characteristics of each part of the face.

[1389] "Makeup techniques" refer to makeup techniques that are suitable for a user and are generated by combining the makeup features of celebrities with the user's facial features.

[1390] "Providing step-by-step instructions" means providing guidance so that the user can perform the generated makeup techniques in a step-by-step manner.

[1391] "Emotional state" refers to the emotion expressed by the user while applying makeup, and indicates states such as satisfaction or confusion.

[1392] "Providing advice and feedback" means notifying the user of appropriate instructions or encouraging messages depending on the user's emotional state.

[1393] In this invention, we will build a system that provides users with a specific method for recreating celebrity makeup on their own face. The overall configuration of the system consists of a process in which users upload images of celebrities and are provided with makeup techniques that are suitable for the user based on the analysis results. A detailed explanation of the system is provided below.

[1394] Hardware and software used

[1395] Hardware:

[1396] Smart glasses or head-mounted displays (HMD)

[1397] Server (with high-performance processor and sufficient storage capacity)

[1398] Cameras (built into smart glasses or HMDs)

[1399] software:

[1400] Artificial intelligence models for image recognition: OpenCV, TensorFlow

[1401] Artificial intelligence models for face recognition: Dlib, FaceNet

[1402] Emotion engine: Microsoft Azure Emotion API

[1403] Process Overview

[1404] The system mainly includes the following processing steps:

[1405] 1. A user uploads an image of a celebrity.

[1406] 2. The server inputs this image into an artificial intelligence model to extract makeup features.

[1407] 3. The user uploads a photo of their face.

[1408] 4. The server inputs the user's facial image into an artificial intelligence model and extracts facial features.

[1409] 5. The server generates the optimal makeup techniques based on the celebrity's makeup characteristics and the user's facial features.

[1410] 6. Display a step-by-step makeup application guide on smart glasses or an HMD.

[1411] 7. Recognize the user's emotional state in real time and provide advice and feedback according to that state.

[1412] Program processing explanation

[1413] User face scan:

[1414] When a user wears the smart glasses, the camera scans the user's face in real time. OpenCV preprocesses the image (resizing, normalizing), and then Dlib and FaceNet are used to extract facial features. The extracted facial features are sent to the server and stored as facial feature data.

[1415] Celebrity makeup feature extraction:

[1416] Users upload images of celebrities, and the server inputs the images into a TensorFlow-based artificial intelligence model that analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[1417] Makeup Technique Creation:

[1418] The server generates optimal makeup techniques based on the makeup characteristics of celebrities and the user's facial features. The generated makeup techniques are then constructed as a step-by-step guide for the user to follow.

[1419] Emotional feedback:

[1420] The camera captures the user's facial expressions while applying makeup and inputs them into the emotion engine. Using the Microsoft Azure Emotion API, the system analyzes the user's emotional state in real time, providing re-explanations if the user is confused, and positive feedback if the user is satisfied.

[1421] Specific examples

[1422] 1. User flow:

[1423] A user visits a cosmetics store, puts on the smart glasses, selects a makeup image of a celebrity they like, and then captures and uploads a photo of their own face.

[1424] 2. Program processing example:

[1425] Celebrity image analysis prompt:

[1426] "Analyze and extract the location of eyeshadow, lip color, and blush in this image."

[1427] Sentiment Analysis Prompt:

[1428] "Please recognize the emotion from the facial expression in this captured image and provide appropriate advice."

[1429] In this way, the system can provide users with a highly satisfying and effective makeup experience by recreating celebrity makeup looks while providing real-time emotional feedback.

[1430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1431] Step 1:

[1432] A user uploads an image of a celebrity.

[1433] Input: A user-selected celebrity face image.

[1434] How it works: The user launches the application and selects an image of a celebrity. The application then sends the selected image data from the device to the server.

[1435] Output: Celebrity face image data uploaded to the server.

[1436] Step 2:

[1437] The server inputs images of celebrities into an image recognition AI model and extracts the celebrities' makeup features.

[1438] Input: Uploaded celebrity face image.

[1439] How it works: The server preprocesses the received image data (resizing, normalizing) and inputs it into a TensorFlow-based artificial intelligence model for image recognition. The model analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[1440] Output: Extracted celebrity makeup feature data.

[1441] Step 3:

[1442] The user uploads a picture of their face.

[1443] Input: A photo of the user's face.

[1444] Operation: The user uploads a photo of their face to the application, and the device sends the photo to the server.

[1445] Output: User's facial image data uploaded to the server.

[1446] Step 4:

[1447] The server inputs the user's facial image into a facial recognition AI model and extracts the user's facial features.

[1448] Input: Uploaded user face image.

[1449] How it works: The server preprocesses (resizes, normalizes) the received facial images and feeds them into a Dlib or FaceNet-based AI model for facial recognition, which extracts facial features such as eye shape, eyebrow shape, and facial contours.

[1450] Output: Extracted user facial feature data.

[1451] Step 5:

[1452] The server generates the most suitable makeup technique based on the makeup features of celebrities and the user's facial features.

[1453] Input: celebrity makeup feature data and user facial feature data.

[1454] How it works: The server combines both sets of data and generates a makeup application tailored to the user, which is structured as a step-by-step guide.

[1455] Output: Generated step-by-step makeup tutorial.

[1456] Step 6:

[1457] The terminal provides the generated makeup technique guide to the user.

[1458] Enter: a step-by-step makeup artistry guide.

[1459] How it works: The server sends the generated guide to the device, which displays it to the user. The user can view the guide step by step through smart glasses or a head-mounted display.

[1460] Output: A step-by-step makeup tutorial provided to the user.

[1461] Step 7:

[1462] It recognizes the user's emotional state in real time and provides advice and feedback according to that emotion.

[1463] Input: User facial expression data captured by the camera.

[1464] How it works: The device captures the user's facial expressions in real time and sends the data to an emotion engine (Microsoft Azure Emotion API). The server analyzes the changes in facial expressions and determines the user's emotional state (confusion, satisfaction, etc.). It then generates appropriate feedback and advice and sends it to the device.

[1465] Output: Advice or feedback based on the user's emotional state.

[1466] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1468] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1469] [Fourth embodiment]

[1470] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1471] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1473] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1477] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1478] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1479] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1480] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1481] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1482] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1483] The present invention relates to a system that allows users to recreate the makeup of their favorite celebrities on their own faces. This system analyzes images of celebrities and the user's facial images, and generates and provides makeup techniques that are suitable for the user based on the makeup characteristics. Specific embodiments of the present invention are described below.

[1484] System Overview

[1485] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on their own facial image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides users with step-by-step instructions on how to apply these techniques, and allows them to share the results with other users.

[1486] Program processing explanation

[1487] Celebrity image upload and analysis

[1488] User: The user first opens the application and uploads a picture of their favorite celebrity.

[1489] Terminal: The terminal sends image data to the server.

[1490] Makeup feature extraction

[1491] Server: The server inputs the received image into an AI model for image recognition and analyzes the celebrity's makeup features, thereby identifying detailed makeup elements such as eyeshadow, lip color, and blush placement.

[1492] Uploading and analyzing user facial images

[1493] User: Next, the user uploads a photo of their face to the application.

[1494] Terminal: The terminal sends this facial photo data to the server.

[1495] Facial feature extraction

[1496] Server: The server inputs the user's facial image into an AI model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[1497] Creating makeup techniques

[1498] Server: The server generates the best makeup tips for the user based on the makeup features of celebrities and the user's facial features. This makeup tip is constructed as a step-by-step guide that the user can follow.

[1499] Providing makeup techniques

[1500] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1501] Specific examples

[1502] The following explains this with specific examples.

[1503] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1504] Terminal: The terminal sends these images to the server.

[1505] Server: The server analyzes the makeup features of the model and the user's facial features. Based on this, it generates makeup techniques that are suitable for the user.

[1506] Specific makeup techniques are provided as follows:

[1507] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1508] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1509] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1510] Users: Users can follow the guide to apply makeup, take a photo of the results, and upload them to the application. They can also share their makeup results with other users.

[1511] This embodiment allows users to easily receive professional makeup advice suited to their facial features. In addition, the ability to provide additional advice in real time and share with other users can provide a more satisfying makeup experience.

[1512] The processing flow will be explained below.

[1513] Step 1:

[1514] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1515] Step 2:

[1516] Device: The device sends the uploaded celebrity image to the server.

[1517] Step 3:

[1518] Server: The server inputs the received celebrity images into an AI model for image recognition.

[1519] Step 4:

[1520] Server: The server performs pre-processing on the images, including image resizing, normalization, etc.

[1521] Step 5:

[1522] Server: The server uses an AI model to extract the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[1523] Step 6:

[1524] Server: The server stores the extracted makeup features in a database.

[1525] Step 7:

[1526] User: The user then uploads a photo of their face to the application.

[1527] Step 8:

[1528] Terminal: The terminal sends the user's facial photo to the server.

[1529] Step 9:

[1530] Server: The server inputs the received user's facial photo into an AI model for facial recognition.

[1531] Step 10:

[1532] Server: The server performs preprocessing on the face photo, including cropping, resizing, and normalizing the face.

[1533] Step 11:

[1534] Server: The server uses an AI model to extract the user's facial features, such as eye shape, eyebrow shape, and facial contours.

[1535] Step 12:

[1536] Server: The server stores the extracted facial features in a database.

[1537] Step 13:

[1538] Server: The server combines the celebrity's makeup features with the user's facial features to generate makeup techniques suited to the user.

[1539] Step 14:

[1540] Sarver: Sarver creates detailed guides that organize makeup techniques into a step-by-step format.

[1541] Step 15:

[1542] Server: The server sends the generated makeup technique guide to the terminal.

[1543] Step 16:

[1544] Terminal: The terminal displays the makeup technique guide to the user.

[1545] Step 17:

[1546] User: The user follows the displayed guide and actually applies the makeup.

[1547] Step 18:

[1548] User: The user uploads a photo of the completed makeup look to the application.

[1549] Step 19:

[1550] Terminal: The terminal sends the finished photo to the server.

[1551] Step 20:

[1552] Server: The server receives the finished photos and stores them in a database so that they can be shared with other users if desired.

[1553] Example 1

[1554] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] Conventional makeup instruction systems have struggled to provide detailed makeup guides tailored to each user's individual facial features. Furthermore, they lacked specific steps for faithfully recreating famous makeup styles on the user's face, making it difficult for users to find a makeup method that best suits their own features. Furthermore, they lacked real-time advice and the ability to share with other users, preventing them from providing a satisfying makeup experience.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1557] In this invention, the server includes means for allowing a user to upload an image of a famous person, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the makeup features of the famous person, means for allowing the user to upload their own facial image, means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the makeup features of the famous person with the user's facial features to generate makeup techniques suitable for the user, means for inputting the generated makeup techniques as prompt sentences into the artificial intelligence model to generate a step-by-step guide, and means for providing the generated makeup guide to a user terminal. This allows users to receive professional makeup advice optimized for their facial features in real time. Furthermore, users can share the results of their makeup with other users, providing a fulfilling makeup experience.

[1558] A "famous person" is a widely known public figure or prominent individual.

[1559] An "artificial intelligence model for image recognition" is a learning algorithm used to analyze image data and detect specific features.

[1560] An "artificial intelligence model for facial recognition" is a learning algorithm that analyzes a user's facial image and identifies the shape and placement of specific parts of the face.

[1561] "Makeup features" refers to specific makeup elements and their placement, such as the color around the eyes, the color of the lips, and the position of the cheeks.

[1562] A "prompt" is text data that is input to a generative artificial intelligence model and is used to provide specific instructions or information to generate a specific output.

[1563] A "generative artificial intelligence model" is a learning algorithm used to generate new text or instructions based on prompts and other data.

[1564] A "step-by-step guide" is a sequence of detailed instructions that a user can follow in order.

[1565] "User terminal" refers to an electronic device used by a user to operate and view information, such as a smartphone or a personal computer.

[1566] The present invention relates to a system that allows a user to recreate makeup looks of famous people on their own face. This system analyzes makeup features using an image of a famous person selected by the user and an image of the user's own face, and generates and provides the user with makeup techniques that are optimal for the user. Specific embodiments for implementing the present invention are described below.

[1567] First, a user opens the application and uploads an image of a famous person. The device receives the image and sends it to the server. The server then inputs the received image into an artificial intelligence model for image recognition (such as OpenCV or TensorFlow) to analyze and detect detailed makeup features such as eyeshadow, lip color, and blush placement on the famous person.

[1568] Next, the user uploads a photo of their face to the application. The device sends this photo data to the server, which then inputs the user's face image into an artificial intelligence model for facial recognition (e.g., Dlib or FaceNet) to analyze facial features such as eye shape, eyebrow shape, and facial contours.

[1569] The server then generates the best makeup tips for the user based on the makeup features of famous people and the user's facial features. This makeup tip is input as a prompt into a generative AI model (e.g., GPT-4) to generate a step-by-step guide. Specific examples of prompts are as follows:

[1570] "Generate the following step-by-step makeup guide for a facial image of user X, who has uploaded an image of famous person A. Include detailed instructions for each element of eyeshadow, lipstick, and blush."

[1571] The generated makeup guide is sent from the server to the user's device, which displays it to the user. The user can follow this guide to apply makeup step by step. After completing their makeup, the user can upload a photo of their finished makeup to the application and share it with other users. This function allows users to compare makeup looks and receive advice from each other, thereby enriching the makeup experience.

[1572] As described above, the present invention allows users to receive professional makeup advice tailored to their facial features in real time. Furthermore, by sharing the results of their makeup with other users, users can enjoy a more satisfying makeup experience.

[1573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1574] Step 1:

[1575] User: The user opens the application, selects an image of a famous person and presses the upload button.

[1576] Input: Image files (e.g. JPEG, PNG) of famous people.

[1577] Output: Image file temporarily saved on the device.

[1578] Specific operation: The user operates the application and selects an image of a famous person from the gallery or camera. This image is temporarily stored on the device.

[1579] Step 2:

[1580] Device: The device sends the image of the celebrity uploaded by the user to the server.

[1581] Input: Image file in the device.

[1582] Output: Image data sent to the server.

[1583] What happens: The device sends the saved image file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[1584] Step 3:

[1585] Server: The server inputs the received image of the famous person into an artificial intelligence model for image recognition and analyzes makeup features such as eye shadow, lip color, and blush position.

[1586] Input: Received image data.

[1587] Output: Analyzed makeup feature data (e.g. eyeshadow color, lip color, blush position, etc.).

[1588] Specific operation: The server analyzes the image using an artificial intelligence model for image recognition (e.g., TensorFlow, OpenCV). The makeup feature data extracted as a result of the analysis is stored in a database.

[1589] Step 4:

[1590] User: The user uploads a photo of their face to the application.

[1591] Input: User's face photo file (e.g. JPEG, PNG).

[1592] Output: A face photo file temporarily saved on the device.

[1593] Specific operation: The user operates the application and selects a photo of their face from the gallery or camera. This photo is temporarily stored on the device.

[1594] Step 5:

[1595] Terminal: The terminal sends the user's facial photo to the server.

[1596] Input: Face photo file in the device.

[1597] Output: Facial photo data sent to the server.

[1598] What it does: The device sends the stored face photo file to the backend server in an HTTP request, which may also include metadata such as the user ID and upload date and time.

[1599] Step 6:

[1600] Server: The server inputs the received user's facial photo into an artificial intelligence model for facial recognition and analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.).

[1601] Input: Received facial photo data.

[1602] Output: Analyzed facial feature data (e.g. eye shape, eyebrow shape, facial contour, etc.).

[1603] Specific operation: The server analyzes the facial image using an AI model for facial recognition (e.g., Dlib, FaceNet). The facial feature data extracted as the analysis result is stored in a database.

[1604] Step 7:

[1605] Server: The server generates makeup prompt sentences based on the makeup features of famous people and the user's facial features, and inputs them into the generative artificial intelligence model.

[1606] Input: Makeup feature data of famous people and facial feature data of the user.

[1607] Output: Step-by-step makeup guide.

[1608] Specific operation: The server combines both sets of data to generate a prompt. For example, "Generate the following step-by-step makeup guide for the face image of user X, who uploaded an image of famous person A. Please include detailed instructions for each element: eyeshadow, lipstick, and blush." ​​This prompt is then input into a generative AI model (e.g., GPT-4) to generate a specific makeup guide.

[1609] Step 8:

[1610] Terminal: The makeup guide generated by the server is provided to the terminal, which displays it to the user.

[1611] Input: The generated step-by-step makeup guide.

[1612] Output: The makeup guide displayed on the terminal by the user.

[1613] Specific operation: The terminal displays the makeup guide received from the server. The user follows this guide to perform specific makeup steps.

[1614] Step 9:

[1615] User: After applying makeup, the user uploads a photo of the finished look to the application and shares it with other users.

[1616] Input: Finished photo file (e.g. JPEG, PNG).

[1617] Output: Uploaded finished photo.

[1618] Specific operation: The user presses the upload button, selects a photo of the completed makeup look, and uploads it to the application. The application sends it to the server and shares it with other users.

[1619] (Application example 1)

[1620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1621] When users try to recreate celebrity makeup looks on their own faces, existing methods that suggest appropriate makeup techniques struggle to provide real-time advice or generate specific prompts. Furthermore, there are limited ways to share the makeup results with others or receive feedback. This makes it difficult to improve users' makeup skills and increase their satisfaction.

[1622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1623] In this invention, the server includes means for users to upload images of celebrities, means for inputting the uploaded images into a generative AI model for image recognition and detecting the celebrity's makeup features, means for inputting the uploaded user's facial image into a generative AI model for face recognition and extracting the user's facial features, means for providing a user interface for displaying the generated makeup technique guide, means for generating prompt sentences for guiding users through the steps of applying the suggested makeup techniques, and means for sharing the results with other people. This allows users to receive appropriate makeup techniques along with specific steps in real time, and also allows them to share their makeup results with others to receive feedback.

[1624] 1. "User" refers to a person who uses this system to recreate celebrity makeup on their own face.

[1625] 2. "Celebrity images" refers to visual data, such as photographs or illustrations, that show the face of a celebrity.

[1626] 3. "Upload" refers to sending data from a particular device to a remote storage system, such as a server.

[1627] 4. "Generative AI model for image recognition" refers to an artificial intelligence model that processes image data to recognize and extract specific features.

[1628] 5. "Makeup features" refers to specific features of makeup applied to a celebrity's face, such as makeup patterns and colors.

[1629] 6. "Facial Image" refers to photographs or visual data that show a user's face.

[1630] 7. “Generative AI model for facial recognition” means an artificial intelligence model used to analyze facial features from facial images.

[1631] 8. "Facial Features" refers to specific external features of a user's face, such as the eyes, nose, and mouth.

[1632] 9. "Makeup Technique" refers to the specific makeup methods and procedures applied to the user's face.

[1633] 10. "Step-by-step means" refers to a method that provides users with specific, step-by-step instructions.

[1634] 11. "User interface" refers to the screens and methods of operation that allow a user to interact with a system.

[1635] 12. "Prompt sentences" refers to a series of sentences that provide the user with makeup application instructions or advice in text form.

[1636] 13. "Means for sharing results with others" refers to a function that allows users to show their makeup results to other users via the Internet or other means and receive their evaluations.

[1637] This invention relates to a system that allows users to recreate celebrity makeup on their own face. The system analyzes images of celebrities and the user's facial image, and uses a generative AI model to provide the user with makeup techniques that are suited to the user.

[1638] System Overview

[1639] This system is implemented with the following hardware and software configuration. The user's smartphone or tablet is used as the device, and the server hosts the AI ​​model as a cloud service. The main software includes the OpenCV library for image processing and the Keras framework for using deep learning models.

[1640] System Operation

[1641] Celebrity image upload and analysis

[1642] User: First, the user uploads an image of their favorite celebrity from their device in order to recreate that celebrity's makeup.

[1643] Device: The device sends uploaded images to a server, which then inputs them into a generative AI model for image recognition.

[1644] Server: The server uses a generative AI model to analyze the image and extract the celebrity's makeup features.

[1645] Uploading and analyzing user facial images

[1646] User: Next, the user uploads a photo of their face from their device.

[1647] Terminal: The terminal sends this facial photo to the server.

[1648] Server: The server analyzes the user's facial features using a generative AI model for facial recognition.

[1649] Creating and providing makeup techniques

[1650] Server: The server combines celebrity makeup features with the user's facial features to generate makeup techniques suited to the user. The generated makeup techniques are structured as a step-by-step guide and are presented through a user interface.

[1651] Terminal: The terminal displays the generated makeup techniques to the user and provides prompts on how to apply the proposed makeup techniques.

[1652] Usage example

[1653] Suppose a user wants to apply a makeup style to their face using a famous model. The user first uploads an image of the model to the application, then takes and uploads a photo of their own face. The server analyzes these images and suggests the best makeup techniques by combining the celebrity's makeup style with the user's facial features. The user can then apply the makeup step by step by following the prompts displayed on the smartphone screen.

[1654] Prompt Sentence Examples

[1655] The prompt is presented to the user in the following format:

[1656] "Upload an image of your favorite celebrity and take a photo of yourself. The system will analyze the image and suggest the best makeup techniques."

[1657] This allows users to easily receive professional makeup advice realized using advanced machine learning technology. Furthermore, the ability to provide additional advice in real time and share with others provides users with a more satisfying makeup experience.

[1658] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1659] Step 1:

[1660] User: The user uploads an image of their favorite celebrity to the application using their smartphone or tablet. This input image data is the initial data for analyzing the celebrity's makeup features.

[1661] Step 2:

[1662] Device: The device sends the celebrity image uploaded by the user to the server, where it is converted into the appropriate format (e.g., JPEG or PNG).

[1663] Step 3:

[1664] Server: The server inputs the received images into a generative AI model for image recognition. The AI ​​model analyzes the makeup features in the image (e.g., eyeshadow color, lip color, blush position) and extracts specific features. The output is a dataset showing celebrity makeup features.

[1665] Step 4:

[1666] User: Next, the user uploads a photo of their face using the same application. This photo data becomes the base data for analyzing the user's facial features.

[1667] Step 5:

[1668] Terminal: The terminal converts the user's facial photo into a specified format and sends it to the server.

[1669] Step 6:

[1670] Server: The server inputs the received user's face photo into a generative AI model for facial recognition. The AI ​​model analyzes the user's facial features (eye shape, eyebrow shape, facial contours, etc.) and generates facial feature data as output.

[1671] Step 7:

[1672] Server: The server combines and analyzes celebrity makeup feature data with the user's facial feature data to generate the optimal makeup technique for the user. This makeup technique is constructed as a step-by-step guide. The output is a list of specific makeup steps and cosmetic items to use.

[1673] Step 8:

[1674] Terminal: The terminal displays the step-by-step makeup guide received from the server to the user, who can then apply makeup by following the guide.

[1675] Step 9:

[1676] Server: Based on the generated makeup technique guide, the server generates a prompt sentence to guide the user through the makeup application procedure. This prompt sentence is displayed on the GUI screen.

[1677] Step 10:

[1678] User: The user applies makeup using the provided guide and prompts, then takes a photo of the completed makeup and uploads it to the application.

[1679] Step 11:

[1680] Device: The device sends the photos of the makeup results uploaded by the user to the server and stores them on the platform for sharing with other users.

[1681] Through these steps, users can recreate the makeup of their favorite celebrities on their own faces and share the results with others.

[1682] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1683] The present invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques suited to the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, the present invention can provide makeup advice and feedback according to the user's emotional state. Specific embodiments of the present invention are described below.

[1684] System Overview

[1685] This system allows users to upload images of celebrities and extract makeup features from those images. It then analyzes the user's facial features based on the image and combines them with the celebrity's makeup features to generate makeup techniques that are optimal for the user. Furthermore, it provides the user with step-by-step instructions on how to apply the makeup, and allows the results to be shared with other users. It also uses an emotion engine to recognize the user's emotions and provides advice and feedback based on those emotions.

[1686] Program processing explanation

[1687] Celebrity image upload and analysis

[1688] User: The user launches the application, selects an image of their favorite celebrity, and uploads it.

[1689] Terminal: The terminal transmits the selected image data to the server.

[1690] Makeup feature extraction

[1691] Server: The server inputs the received celebrity image into an AI model for image recognition. As preprocessing, the image is resized and normalized. The AI ​​model then extracts the celebrity's makeup features, such as eyeshadow, lip color, and blush placement.

[1692] Uploading and analyzing user facial images

[1693] User: Next, the user uploads a photo of their face to the application.

[1694] Terminal: The terminal sends the user's facial photo to the server.

[1695] Facial feature extraction

[1696] Server: The server inputs the user's facial image into the AI ​​model for facial recognition. After preprocessing such as cropping, resizing, and normalizing the face, the AI ​​model extracts facial features such as eye shape, eyebrow shape, and facial contours.

[1697] Creating makeup techniques

[1698] Server: The server generates makeup techniques suitable for the user based on the makeup features of celebrities and the user's facial features. The generated makeup techniques are constructed as a step-by-step guide that the user can follow.

[1699] Providing makeup techniques

[1700] Terminal: The makeup guide generated by the server is sent to the terminal, which displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1701] Use of emotion engine

[1702] User: While applying makeup, the user's facial expressions are captured and analyzed by a camera to recognize emotions in real time.

[1703] Device: The device sends facial expression data captured by the camera to the emotion engine.

[1704] Server: The emotion engine analyzes facial expressions to determine the user's emotional state, for example, whether they are confused or happy.

[1705] Emotion-based advice and feedback

[1706] Server: Based on the emotional state recognized by the emotion engine, the server provides the user with appropriate advice and feedback. If the user is confused, the server provides additional support, such as re-explaining the steps, and if the user is satisfied, the server displays messages of praise and encouragement.

[1707] Device: The device displays real-time advice and feedback to the user, providing enhanced support during the makeup application process.

[1708] Specific examples

[1709] The following explains this with specific examples.

[1710] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1711] Terminal: The terminal sends these images to the server.

[1712] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[1713] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1714] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1715] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1716] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[1717] This embodiment allows users to easily receive professional makeup advice suited to their facial features. Furthermore, the function of providing additional advice in real time and feedback according to emotions can provide a more satisfying makeup experience.

[1718] The processing flow will be explained below.

[1719] Step 1:

[1720] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1721] Step 2:

[1722] Terminal: The terminal sends the image data of the selected celebrity to the server.

[1723] Step 3:

[1724] Server: The server preprocesses the received celebrity images, specifically resizing and normalizing the images.

[1725] Step 4:

[1726] Server: The server inputs preprocessed images into an AI model for image recognition and extracts makeup features such as eyeshadow, lip color, and blush position.

[1727] Step 5:

[1728] Server: The server stores the extracted makeup features in a database.

[1729] Step 6:

[1730] User: Next, the user uploads a photo of their face to the application.

[1731] Step 7:

[1732] Terminal: The terminal sends the user's facial photo data to the server.

[1733] Step 8:

[1734] Server: The server preprocesses the received user face photo, specifically cropping, resizing, and normalizing the face.

[1735] Step 9:

[1736] Server: The server inputs preprocessed facial photos into an AI model for facial recognition, extracting facial features such as eye shape, eyebrow shape, and facial contours.

[1737] Step 10:

[1738] Server: The server stores the extracted facial features in a database.

[1739] Step 11:

[1740] Server: The server generates makeup techniques suitable for the user based on the makeup characteristics of celebrities and the user's facial features.

[1741] Step 12:

[1742] Server: The server builds the generated makeup techniques as a step-by-step guide.

[1743] Step 13:

[1744] Server: The server sends the guide to the device.

[1745] Step 14:

[1746] Terminal: The terminal displays the makeup technique guide to the user.

[1747] Step 15:

[1748] User: The user follows the displayed guide and applies makeup in the correct order.

[1749] Step 16:

[1750] Terminal: The terminal captures the user's facial expressions while applying makeup with a camera and sends the facial expression data to the server.

[1751] Step 17:

[1752] Server: The server uses an emotion engine to analyze the captured facial expression data and recognize the user's emotions.

[1753] Step 18:

[1754] Server: Based on emotion recognition, adjust makeup steps and advice as needed. For example, if the user is confused, restate the steps or suggest an alternative approach.

[1755] Step 19:

[1756] Server: Generates positive feedback such as encouragement or praise according to the user's emotional state and sends it to the device.

[1757] Step 20:

[1758] Terminal: The terminal displays feedback and advice received from the server to the user in real time, supporting the makeup experience.

[1759] Step 21:

[1760] User: After the user has completed their makeup, they take a photo of the finished look and upload it to the application.

[1761] Step 22:

[1762] Terminal: The terminal sends the finished photo to the server.

[1763] Step 23:

[1764] Server: The server stores the finished photos in a database and makes them available for sharing with other users as needed.

[1765] Example 2

[1766] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1767] Conventional makeup advice systems are required to not only detect celebrity makeup features, but also generate makeup techniques suitable for the user and provide them as a practical, concrete, step-by-step guide. However, these systems lack real-time advice and feedback based on the user's emotional state, which can lead to confusion and reduced user satisfaction. A new system that solves these problems is needed.

[1768] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1769] In this invention, the server includes means for a user to upload an image of a celebrity, means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features, means for a user to upload his or her own facial image, means for inputting the uploaded user's facial image into an artificial intelligence model for face recognition and extracting the user's facial features, means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user, means for providing the generated makeup technique to the user step by step, means for capturing and analyzing the user's facial expressions in real time, and means for recognizing the user's emotional state based on the results of the facial expression analysis and providing appropriate advice and feedback. This not only enables the user to more effectively recreate celebrity makeup, but also enables a satisfying makeup experience without confusion due to the real-time emotion recognition and feedback.

[1770] The "means for uploading images of celebrities" is a function that allows a user to select an image of a celebrity through an application and transmit the image data to a server via a network.

[1771] "Artificial intelligence model for image recognition" refers to an automated learning algorithm that analyzes received image data and extracts specific features from it, such as cosmetic features like eyeshadow, lip color, and blush placement.

[1772] The "means for detecting makeup features" is a function that uses an artificial intelligence model for image recognition to identify specific makeup elements from images of celebrities.

[1773] "Means for users to upload their own facial images" refers to a function that allows users to take or select a photo of their own face via the application and send it to the server.

[1774] "Artificial intelligence model for facial recognition" refers to an automated learning algorithm that analyzes a user's facial image and identifies specific facial features such as eye shape, eyebrow shape, and facial contours to extract features.

[1775] The "means for extracting a user's facial features" is a function that uses an artificial intelligence model for facial recognition to identify specific features from a user's facial image and extract them as data.

[1776] The "means for generating makeup techniques" is a function that integrates the makeup features of celebrities with the user's facial features and generates a step-by-step guide for a makeup method that is suitable for the user.

[1777] The "means for providing step-by-step instructions to the user" is a function of instructing the user in an easy-to-understand manner for each step of the generated makeup guide, and displaying or explaining the instructions so that the user can proceed as instructed.

[1778] "Means for capturing and analyzing a user's facial expressions in real time" refers to a function that uses a camera to capture a user's facial expressions in real time while the user is applying makeup, and analyzes the facial expression data.

[1779] "Means for recognizing emotional states and providing appropriate advice and feedback" refers to a function that analyzes captured facial expression data, determines the user's emotional state, and provides appropriate advice and supportive messages in real time.

[1780] This invention relates to a system that allows users to recreate celebrity makeup on their own face. This system analyzes images of celebrities and the user's face, and generates and provides makeup techniques that are suitable for the user based on the characteristics of the makeup. In addition, by combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide makeup advice and feedback according to the user's emotional state.

[1781] System Overview

[1782] This system consists of multiple hardware and software components. The main hardware components are the user's device, such as a smartphone or tablet, and a cloud server. The software components include an AI model for image recognition (e.g., DeepLabV3+), an AI model for face recognition (e.g., FaceNet), and an emotion engine (e.g., Emotion API).

[1783] Users upload images of celebrities and their own faces through the application. The device sends these images to a server, where image analysis is performed. The celebrity's makeup features and the user's facial features are analyzed and extracted, and the optimal makeup technique for the user is generated based on this. The generated makeup technique is then provided to the user step by step.

[1784] System Details

[1785] Celebrity image upload and analysis

[1786] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1787] Terminal: The terminal sends the image data selected by the user to the server using a secure protocol such as HTTPS.

[1788] Server: The server temporarily stores the received image data and inputs it into an AI model for image recognition. After preprocessing such as resizing and normalizing the image, the AI ​​model extracts the celebrity's makeup characteristics.

[1789] Uploading and analyzing user facial images

[1790] User: Next, the user uploads a photo of their face to the application.

[1791] Device: The device temporarily stores the user's facial photo locally and then sends it to the server, again using a secure protocol.

[1792] Server: The server temporarily stores the received user's face photo and inputs it into an AI model for facial recognition. After preprocessing such as cropping, resizing, and normalization, the AI ​​model extracts the user's facial features.

[1793] Creation and provision of cosmetic techniques

[1794] Server: The server generates a makeup technique suited to the user based on the makeup characteristics of celebrities and the user's facial features. The generated makeup technique is constructed as a step-by-step guide. This guide includes detailed information on the type and amount of makeup to use in each step, as well as the order and location of application.

[1795] Terminal: Receives the makeup guide generated by the server and displays it to the user. The user can follow the guide to apply makeup in the correct order.

[1796] Use of emotion engine

[1797] User: While applying makeup, the device camera captures the user's facial expressions.

[1798] Device: The device transmits the facial expression data captured in real time to the emotion engine.

[1799] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state, for example, re-explaining the steps if the user is confused, or generating an encouraging message if the user is satisfied.

[1800] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[1801] Specific examples

[1802] Specific examples are shown below.

[1803] User: For example, if a user wants to learn makeup from a famous model, they can upload an image of that model to the application, and then upload a photo of their own face.

[1804] Terminal: The terminal sends these images to the server.

[1805] Server: The server analyzes the makeup features of the model and also analyzes the user's facial features. Based on this, it generates makeup techniques suitable for the user. Specific makeup techniques are provided as follows:

[1806] 1. Eyeshadow: To create a smoky eye, blend a light to dark grey shade towards the outer edges of your eyes.

[1807] 2. Lips: Use a red matte lipstick and apply evenly from the center of your lips outwards.

[1808] 3. Blush: Lightly apply pink blush along your cheekbones while smiling.

[1809] User: The user follows the guide to apply makeup, takes a photo of the finished look, and uploads it to the application. At this time, the emotion engine recognizes the user's emotions from their facial expressions and provides appropriate feedback in real time.

[1810] Prompt Sentence Examples

[1811] Below are some examples of prompt sentences.

[1812] Makeup feature extraction

[1813] "Identify makeup features from the following celebrity images: eyeshadow, lip color, blush placement, etc."

[1814] Facial feature extraction

[1815] "Analyze this user's facial image to extract eye shape, eyebrow shape, and facial contours."

[1816] Use of emotion engine

[1817] "Analyze this user's facial expression to determine their current emotional state (confusion, satisfaction, etc.)"

[1818] This allows users to easily and efficiently learn and practice professional makeup techniques, and the system also provides real-time support, improving user satisfaction.

[1819] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1820] Step 1: Upload and submit a photo of your celebrity

[1821] User: The user launches the application and selects and uploads an image of their favorite celebrity.

[1822] Terminal: The terminal temporarily stores the image data selected by the user locally and then transmits it to the server using a secure protocol such as HTTPS.

[1823] Input: celebrity image file

[1824] Output: Image data sent to the server

[1825] Step 2: Extracting celebrity makeup features

[1826] Server: The server temporarily stores the received image data. Next, it resizes and normalizes the image. After preprocessing, the image is input into an artificial intelligence model for image recognition (e.g., DeepLabV3+) to extract makeup features such as the celebrity's eyeshadow, lip color, and blush position.

[1827] Input: Resized and normalized celebrity image data

[1828] Output: Extracted makeup feature data

[1829] Step 3: Upload and send the user's face image

[1830] User: Next, the user uploads a photo of their face to the application.

[1831] Device: The device temporarily stores the user's facial photo locally and then sends it to the server using a secure protocol such as HTTPS.

[1832] Input: User's face image file

[1833] Output: Facial image data sent to the server

[1834] Step 4: Extracting the user's facial features

[1835] Server: The server temporarily stores the received user's facial image data. Then, it performs preprocessing such as cropping, resizing, and normalization of the face. After preprocessing, the facial image is input into an artificial intelligence model for facial recognition (e.g., FaceNet) to extract facial features such as eye shape, eyebrow shape, and facial contours.

[1836] Input: Resized and normalized user face image data

[1837] Output: Extracted facial feature data

[1838] Step 5: Creating makeup techniques

[1839] Server: The server combines celebrity makeup feature data with the user's facial feature data to generate the best makeup techniques for the user. During the generation process, a specific algorithm is used to map makeup features and facial features, and a step-by-step guide for the best makeup techniques for the user is constructed.

[1840] Input: celebrity makeup feature data and user facial feature data

[1841] Output: Step-by-step makeup guide

[1842] Step 6: Providing makeup tips

[1843] Device: The device receives the step-by-step makeup tutorial generated by the server and displays it to the user via the application. UI design is important for the presentation, and each step is often supplemented with images or short video clips.

[1844] User: The user follows the guide to apply makeup in the correct order.

[1845] Enter: a step-by-step makeup tutorial

[1846] Output: User's Make execution results

[1847] Step 7: Use the Emotion Engine

[1848] User: While applying makeup, the device camera captures the user's facial expressions in real time.

[1849] Device: The device sends the facial expression data captured in real time to the emotion engine (e.g., Emotion API).

[1850] Input: Captured user facial expression data

[1851] Output: Facial expression data sent to the server

[1852] Step 8: Emotion-based advice and feedback

[1853] Server: The emotion engine analyzes the captured facial expression data and determines the user's emotional state. Based on the determined emotional state, the server generates appropriate advice and feedback. This feedback can include re-explaining the steps if the user is confused, or providing an encouraging message if the user is satisfied.

[1854] Device: The device generates real-time feedback and displays it to the user, possibly via text message or voice assistant.

[1855] Input: Parsed user emotion data

[1856] Output: Real-time advice and feedback

[1857] (Application example 2)

[1858] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1859] In the modern beauty industry, users are highly interested in replicating celebrity makeup looks on their own faces. However, there are few systems in brick-and-mortar stores that allow users to receive personalized makeup techniques in real time. Furthermore, mechanisms for providing feedback based on the user's emotional state during the makeup process are also lacking. The present invention aims to solve these issues by providing a system that provides real-time emotional advice while replicating celebrity makeup looks.

[1860] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1861] In this invention, the server includes: means for a user to upload an image of a celebrity; means for inputting the uploaded image into an artificial intelligence model for image recognition and detecting the celebrity's makeup features; means for a user to upload his or her own facial image; means for inputting the uploaded facial image of the user into an artificial intelligence model for face recognition and extracting the user's facial features; means for combining the celebrity's makeup features with the user's facial features to generate a makeup technique suitable for the user; means for providing the generated makeup technique to the user step by step; and means for recognizing the user's emotional state in real time and providing advice and feedback according to the user's emotions. This enables users to recreate celebrity makeup looks even in physical stores, and further enables users to achieve higher satisfaction by receiving feedback according to their emotions during the makeup process.

[1862] "User" refers to someone who uses the system to recreate celebrity makeup looks.

[1863] A "celebrity image" is a photograph of a famous person's face that is used to extract the person's makeup features.

[1864] An "artificial intelligence model for image recognition" is an artificial intelligence program that detects and analyzes specific features from input images.

[1865] "Cosmetic features" refer to specific elements of makeup such as eyeshadow, lip color, and blush placement.

[1866] A "face image" is a photograph of the user's face that is uploaded to the system.

[1867] An "artificial intelligence model for facial recognition" is an artificial intelligence program that extracts facial features such as eye shape, eyebrow shape, and facial contours from an input facial image.

[1868] "Facial features" refer to the shape of the user's face and the characteristics of each part of the face.

[1869] "Makeup techniques" refer to makeup techniques that are suitable for a user and are generated by combining the makeup features of celebrities with the user's facial features.

[1870] "Providing step-by-step instructions" means providing guidance so that the user can perform the generated makeup techniques in a step-by-step manner.

[1871] "Emotional state" refers to the emotion expressed by the user while applying makeup, and indicates states such as satisfaction or confusion.

[1872] "Providing advice and feedback" means notifying the user of appropriate instructions or encouraging messages depending on the user's emotional state.

[1873] In this invention, we will build a system that provides users with a specific method for recreating celebrity makeup on their own face. The overall configuration of the system consists of a process in which users upload images of celebrities and are provided with makeup techniques that are suitable for the user based on the analysis results. A detailed explanation of the system is provided below.

[1874] Hardware and software used

[1875] Hardware:

[1876] Smart glasses or head-mounted displays (HMD)

[1877] Server (with high-performance processor and sufficient storage capacity)

[1878] Cameras (built into smart glasses or HMDs)

[1879] software:

[1880] Artificial intelligence models for image recognition: OpenCV, TensorFlow

[1881] Artificial intelligence models for face recognition: Dlib, FaceNet

[1882] Emotion engine: Microsoft Azure Emotion API

[1883] Process Overview

[1884] The system mainly includes the following processing steps:

[1885] 1. A user uploads an image of a celebrity.

[1886] 2. The server inputs this image into an artificial intelligence model to extract makeup features.

[1887] 3. The user uploads a photo of their face.

[1888] 4. The server inputs the user's facial image into an artificial intelligence model and extracts facial features.

[1889] 5. The server generates the optimal makeup techniques based on the celebrity's makeup characteristics and the user's facial features.

[1890] 6. Display a step-by-step makeup application guide on smart glasses or an HMD.

[1891] 7. Recognize the user's emotional state in real time and provide advice and feedback according to that state.

[1892] Program processing explanation

[1893] User face scan:

[1894] When a user wears the smart glasses, the camera scans the user's face in real time. OpenCV preprocesses the image (resizing, normalizing), and then Dlib and FaceNet are used to extract facial features. The extracted facial features are sent to the server and stored as facial feature data.

[1895] Celebrity makeup feature extraction:

[1896] Users upload images of celebrities, and the server inputs the images into a TensorFlow-based artificial intelligence model that analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[1897] Makeup Technique Creation:

[1898] The server generates optimal makeup techniques based on the makeup characteristics of celebrities and the user's facial features. The generated makeup techniques are then constructed as a step-by-step guide for the user to follow.

[1899] Emotional feedback:

[1900] The camera captures the user's facial expressions while applying makeup and inputs them into the emotion engine. Using the Microsoft Azure Emotion API, the system analyzes the user's emotional state in real time, providing re-explanations if the user is confused, and positive feedback if the user is satisfied.

[1901] Specific examples

[1902] 1. User flow:

[1903] A user visits a cosmetics store, puts on the smart glasses, selects a makeup image of a celebrity they like, and then captures and uploads a photo of their own face.

[1904] 2. Program processing example:

[1905] Celebrity image analysis prompt:

[1906] "Analyze and extract the location of eyeshadow, lip color, and blush in this image."

[1907] Sentiment Analysis Prompt:

[1908] "Please recognize the emotion from the facial expression in this captured image and provide appropriate advice."

[1909] In this way, the system can provide users with a highly satisfying and effective makeup experience by recreating celebrity makeup looks while providing real-time emotional feedback.

[1910] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1911] Step 1:

[1912] A user uploads an image of a celebrity.

[1913] Input: A user-selected celebrity face image.

[1914] How it works: The user launches the application and selects an image of a celebrity. The application then sends the selected image data from the device to the server.

[1915] Output: Celebrity face image data uploaded to the server.

[1916] Step 2:

[1917] The server inputs images of celebrities into an image recognition AI model and extracts the celebrities' makeup features.

[1918] Input: Uploaded celebrity face image.

[1919] How it works: The server preprocesses the received image data (resizing, normalizing) and inputs it into a TensorFlow-based artificial intelligence model for image recognition. The model analyzes and extracts makeup features such as eyeshadow, lip color, and blush.

[1920] Output: Extracted celebrity makeup feature data.

[1921] Step 3:

[1922] The user uploads a picture of their face.

[1923] Input: A photo of the user's face.

[1924] Operation: The user uploads a photo of their face to the application, and the device sends the photo to the server.

[1925] Output: User's facial image data uploaded to the server.

[1926] Step 4:

[1927] The server inputs the user's facial image into a facial recognition AI model and extracts the user's facial features.

[1928] Input: Uploaded user face image.

[1929] How it works: The server preprocesses (resizes, normalizes) the received facial images and feeds them into a Dlib or FaceNet-based AI model for facial recognition, which extracts facial features such as eye shape, eyebrow shape, and facial contours.

[1930] Output: Extracted user facial feature data.

[1931] Step 5:

[1932] The server generates the most suitable makeup technique based on the makeup features of celebrities and the user's facial features.

[1933] Input: celebrity makeup feature data and user facial feature data.

[1934] How it works: The server combines both sets of data and generates a makeup application tailored to the user, which is structured as a step-by-step guide.

[1935] Output: Generated step-by-step makeup tutorial.

[1936] Step 6:

[1937] The terminal provides the generated makeup technique guide to the user.

[1938] Enter: a step-by-step makeup artistry guide.

[1939] How it works: The server sends the generated guide to the device, which displays it to the user. The user can view the guide step by step through smart glasses or a head-mounted display.

[1940] Output: A step-by-step makeup tutorial provided to the user.

[1941] Step 7:

[1942] It recognizes the user's emotional state in real time and provides advice and feedback according to that emotion.

[1943] Input: User facial expression data captured by the camera.

[1944] How it works: The device captures the user's facial expressions in real time and sends the data to an emotion engine (Microsoft Azure Emotion API). The server analyzes the changes in facial expressions and determines the user's emotional state (confusion, satisfaction, etc.). It then generates appropriate feedback and advice and sends it to the device.

[1945] Output: Advice or feedback based on the user's emotional state.

[1946] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1947] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1948] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1949] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1950] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1951] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1952] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1953] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1954] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1955] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1956] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1957] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1958] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1959] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1960] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1961] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1962] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1963] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1964] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1965] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1966] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1967] The following is further disclosed regarding the above embodiment.

[1968] (Claim 1)

[1969] means for users to upload images of celebrities;

[1970] A method for inputting uploaded images into an AI model for image recognition to detect celebrity makeup features;

[1971] means for a user to upload an image of his or her face;

[1972] A means for inputting the uploaded facial image of the user into an AI model for facial recognition and extracting the facial features of the user;

[1973] A means for generating makeup techniques suited to a user by combining makeup features of celebrities with facial features of the user;

[1974] The system includes a means for providing the generated makeup techniques to the user step by step.

[1975] (Claim 2)

[1976] 10. The system of claim 1, further comprising means for providing additional makeup advice and video tutorials in real time based on the user's facial features.

[1977] (Claim 3)

[1978] 2. The system according to claim 1, further comprising means for uploading a photo of the makeup applied by the user so that the photo can be shared with other users.

[1979] "Example 1"

[1980] (Claim 1)

[1981] means for users to upload images of famous people;

[1982] A means for inputting the uploaded image into an artificial intelligence model for image recognition to detect makeup features of famous people;

[1983] means for a user to upload an image of his or her face;

[1984] A means for inputting the uploaded facial image of the user into an artificial intelligence model for facial recognition and extracting facial features of the user;

[1985] A means for generating makeup techniques suited to a user by combining makeup features of famous people with facial features of the user;

[1986] A means for inputting the generated makeup techniques as prompt sentences into a generative artificial intelligence model to generate a step-by-step guide;

[1987] The system includes a means for providing the generated makeup guide to a user terminal.

[1988] (Claim 2)

[1989] 10. The system of claim 1, further comprising means for providing additional makeup advice and video tutorials in real time based on the user's facial features.

[1990] (Claim 3)

[1991] 2. The system according to claim 1, further comprising means for uploading a photo of the makeup applied by the user so that the photo can be shared with other users.

[1992] "Application Example 1"

[1993] (Claim 1)

[1994] means for users to upload images of celebrities;

[1995] A means of inputting uploaded images into a generative AI model for image recognition to detect celebrity makeup features;

[1996] means for a user to upload an image of his or her face;

[1997] A means for inputting the uploaded user's facial image into a generative AI model for facial recognition and extracting the user's facial features;

[1998] A means for generating makeup techniques suited to a user by combining makeup features of celebrities with facial features of the user;

[1999] A means for providing the generated makeup techniques to a user step by step;

[2000] means for providing a user interface for displaying the generated makeup technique guide;

[2001] means for generating prompt sentences to guide the user through the steps of applying the proposed makeup technique;

[2002] A way to share your results with others

[2003] A system including:

[2004] (Claim 2)

[2005] 10. The system of claim 1, further comprising means for providing additional makeup advice and video tutorials in real time based on the user's facial features.

[2006] (Claim 3)

[2007] 10. The system of claim 1, further comprising means for enabling a user to upload and share photos of makeup results with other people.

[2008] "Example 2: Combining Emotion Engines"

[2009] (Claim 1)

[2010] means for users to upload images of celebrities;

[2011] A means for inputting the uploaded image into an artificial intelligence model for image recognition to detect makeup features of celebrities;

[2012] means for a user to upload an image of his or her face;

[2013] A means for inputting the uploaded facial image of the user into an artificial intelligence model for facial recognition and extracting facial features of the user;

[2014] A means for generating a makeup technique suitable for a user by combining makeup features of celebrities with facial features of the user;

[2015] A means for providing the generated makeup technique to a user step by step;

[2016] means for capturing and analyzing a user's facial expression in real time;

[2017] A system including a means for recognizing emotional states based on facial expression analysis results and providing appropriate advice and feedback.

[2018] (Claim 2)

[2019] 10. The system of claim 1, further comprising: providing additional makeup advice and video tutorials in real time based on the user's facial features.

[2020] (Claim 3)

[2021] 2. The system according to claim 1, further comprising means for uploading a photo of the makeup application result of the user so that the photo can be shared with other users.

[2022] "Application example 2 when combining emotion engines"

[2023] (Claim 1)

[2024] means for users to upload images of celebrities;

[2025] A means for inputting the uploaded image into an artificial intelligence model for image recognition to detect makeup features of celebrities;

[2026] means for a user to upload an image of his or her face;

[2027] A means for inputting the uploaded facial image of the user into an artificial intelligence model for facial recognition and extracting facial features of the user;

[2028] A means for generating a makeup technique suitable for a user by combining makeup features of celebrities with facial features of the user;

[2029] A means for providing the generated makeup technique to a user step by step;

[2030] A system that includes a means for recognizing a user's emotional state in real time and providing advice or feedback according to that emotion.

[2031] (Claim 2)

[2032] 10. The system of claim 1, further comprising: means for providing additional makeup advice or video tutorials in real time based on the user's facial features; and means for providing feedback based on emotion recognition.

[2033] (Claim 3)

[2034] 2. The system according to claim 1, further comprising means for uploading a photo of the makeup applied by the user so that the photo can be shared with other users. [Explanation of symbols]

[2035] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for users to upload images of celebrities; A method for inputting uploaded images into an AI model for image recognition to detect celebrity makeup features; means for a user to upload an image of his or her face; A means for inputting the uploaded facial image of the user into an AI model for facial recognition and extracting the facial features of the user; A means for generating makeup techniques suited to a user by combining makeup features of celebrities with facial features of the user; The system includes a means for providing the generated makeup techniques to the user step by step.

2. 10. The system of claim 1, further comprising means for providing additional makeup advice and video tutorials in real time based on the user's facial features.

3. 2. The system according to claim 1, further comprising means for uploading a photo of the makeup applied by the user so that the photo can be shared with other users.

Images