System

Abstract

A system is provided.SOLUTION: A system including means for providing a smartphone app for a user to capture and upload an image of a bare face, means for causing the user to input trouble or a desire related to makeup, means for using an interactive AI that provides appropriate advice on the basis of the input trouble or desire, means for using an image generation AI that generates a face image after makeup on the basis of the image uploaded by the user and the advice of the interactive AI, means for displaying the generated face image after makeup to the user, and means for providing the user with information on an optimal cosmetic or brand and a purchase link.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For users interested in beauty, trial and error to find the best makeup style or selecting the right product from the many available can be difficult. This process is time-consuming and expensive, and users often cannot see how the results of their trial and error will actually look on them. Furthermore, a lack of knowledge about makeup often leads to users choosing the wrong product and failing to achieve the results they expected. Therefore, there is a need for a system that provides users with personalized makeup information and allows them to visually check the results. [Means for solving the problem]

[0005] To solve the above problems, the present invention proposes a system that provides a smartphone app that allows users to take and upload images of their bare faces. This system includes a means for using an interactive AI that allows users to input their makeup concerns and preferences and provides appropriate advice based on the input content. The system also employs an image generation AI that generates a made-up facial image based on the input content and the uploaded image. The system also includes a means for displaying the generated made-up facial image to the user and providing information on the most suitable cosmetics and brands, as well as a purchase link. This allows users to easily identify makeup styles that suit them and select the most suitable products.

[0006] "User" refers to an individual who uses this system to obtain makeup information.

[0007] A "bare face image" refers to a photo of the user's face without makeup.

[0008] "Smartphone app" refers to software that runs on a smartphone and interacts with the user and processes images.

[0009] "Concerns and hopes regarding makeup" refers to the problems the user feels about makeup and specific requests regarding the appearance they want to achieve.

[0010] "Input means" refers to an interface that allows a user to provide information about their own makeup to the system.

[0011] "Conversational AI" refers to an artificial intelligence system that asks questions and gives advice in natural language based on user input.

[0012] "Image generation AI" refers to an artificial intelligence system that generates a face image with makeup applied based on an input image of a bare face and information about the makeup.

[0013] "Facial image generation means" refers to the process and technology for generating a facial image after makeup using image generation AI.

[0014] "Cosmetics and brand information" refers to detailed information about makeup products and their brands that are recommended to the user.

[0015] "Purchase Link" refers to a web link that allows a user to purchase the recommended cosmetic product online. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a smartphone app, a server, and multiple artificial intelligence technologies.

[0038] Program processing and specific examples

[0039] Account creation and login

[0040] First, a user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0041] Makeup counseling

[0042] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and wishes, and the device sends this information to the server. The server receives this information, analyzes the counseling content using conversational AI (e.g., artificial intelligence that processes natural language), generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0043] Upload a photo of your bare face

[0044] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0045] Generating face images after makeup

[0046] Based on the bare face image sent to the server and the counseling details, the image generation AI analyzes the user's face shape and skin tone and generates a face image with makeup applied. For example, if the user requests to "cover up redness," the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup applied is sent via the server to the device, which then displays it to the user.

[0047] Makeup product recommendations

[0048] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0049] Providing you with special offers and discounts

[0050] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0051] The system of the present invention provides an environment where users can easily find the makeup style that suits them and select the most suitable products, thereby significantly improving the user's beauty experience.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0055] Terminal: Sends the entered information to the server.

[0056] Step 2:

[0057] Server: Stores the received user information in a database and generates an authentication token.

[0058] Server: Returns the authentication token to the device.

[0059] Terminal: Receives the authentication token and displays the login status to the user.

[0060] Step 3:

[0061] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0062] User: Enter their concerns and wishes in response to the questions.

[0063] Terminal: Sends the entered information to the server.

[0064] Step 4:

[0065] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0066] Server: Sends advice to the device.

[0067] Terminal: Display advice to the user.

[0068] Step 5:

[0069] On the device: Prompt the user to take and upload a photo of their face.

[0070] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0071] Device: Sends the captured image to the server.

[0072] Step 6:

[0073] Server: Processes the received face images and counseling content.

[0074] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0075] Server: Sends the generated face image after makeup to the device.

[0076] Terminal: Displays the face image after makeup to the user.

[0077] Step 7:

[0078] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0079] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0080] Device: Display recommendation information and purchase links to the user.

[0081] Step 8:

[0082] Server: Collects special offers and discounts from cosmetic manufacturers.

[0083] Server: Sends special offers and discounts to your device.

[0084] On the device: Show users special offers and discounts.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional makeup advisor systems have difficulty providing appropriate advice based on users' concerns and desires, and often take time to select the most suitable makeup style and cosmetics for the user. Furthermore, it is difficult to realistically reproduce facial images after makeup application, which can result in low user satisfaction. Furthermore, the provision of special offers and discount information is not centralized, making it difficult for users to receive this information at the appropriate time.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using an interactive artificial intelligence that provides appropriate advice based on the input concerns and wishes, means for using an image generation artificial intelligence that generates a made-up facial image based on the image uploaded by the user and the advice from the interactive artificial intelligence, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on beauty products and brands that are most suitable for the user, and means for collecting and providing special offer and discount information. This allows users to easily select makeup styles and cosmetics that suit them, check realistic made-up facial images, and also receive special offer and discount information in a timely manner.

[0090] A "mobile device application" is software that is installed on a mobile phone or tablet device and provides users with the ability to take and upload images of their natural face.

[0091] "Conversational artificial intelligence" is an artificial intelligence technology that has the ability to analyze text data entered by the user and provide appropriate responses and advice in natural language.

[0092] "Image generation artificial intelligence" is an artificial intelligence technology that has the ability to generate new images based on image data and related information uploaded by users.

[0093] "Beauty products" is a general term for products, such as cosmetics and skin care products, used to assist or improve a user's appearance or beauty.

[0094] "Purchase Link" means a hyperlink that provides direct access to a web page for purchasing a particular product or service.

[0095] A "special offer" is a temporary promotional activity that offers a lower price or special benefits than the regular selling price.

[0096] "Discount information" is information that notifies users that the price of a product will be cheaper than usual for a certain period of time.

[0097] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0098] The mobile application is software that users install on their smartphones or tablets and provides the functionality to take and upload images of their faces, allowing users to provide images of their own faces to the system.

[0099] The server provides the following main functions: First, it receives information such as the user's name, email address, and password entered into the mobile device application and creates a user account in the database. Next, the interactive AI generates appropriate advice based on the user's makeup concerns and preferences and provides it to the user. Furthermore, it processes the bare-face image uploaded by the user and generates a face image with makeup using the image generation AI. Finally, based on the generated face image with makeup, it provides the user with information on the most suitable beauty products and brands, purchase links, special offers, and discount information.

[0100] Conversational AI analyzes text data entered by the user (e.g., "What are your skin concerns?", "What kind of makeup style are you aiming for?") and provides appropriate responses and advice in natural language. For example, if a user enters, "It's dry today, so I want to wear makeup that emphasizes moisturizing," the conversational AI will generate a response such as, "Moisturizing is important. Next, please upload an image of your bare face."

[0101] Image generation AI is a technology that generates a new make-up image of a face based on an image of the user's natural face and advice from an interactive AI. For example, if a user requests to "cover up redness," the image generation AI generates a face image with natural make-up that covers up the redness and sends it to the terminal via the server. The terminal then displays this make-up image to the user.

[0102] The server then provides the user with information on the most suitable beauty products and brands, along with a link to purchase them, based on the generated face image after makeup application. For example, specific recommendation information such as "This makeup style is best suited to a specific brand of foundation" is included. This allows the user to easily access the purchasing site.

[0103] In addition, the server collects special offers and discount information from cosmetics manufacturers and provides this information to users. The terminal supports users' purchasing behavior by displaying information such as "Currently, products from a specific brand are 10% off."

[0104] In this way, the system of the present invention provides an environment in which users can easily find a makeup style that suits them and select the most suitable products, thereby significantly improving their beauty experience.

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] The user installs and launches the mobile application. The application prompts the user to enter information such as their name, email address, and password. After the user enters the required information and taps the "Submit" button, the device sends this information to the server.

[0108] Input: User information such as name, email address, and password

[0109] Data processing / calculation: The server registers the received information in the database and creates a user account. Once registration is complete, an authentication token is generated.

[0110] Output: Sends the authentication token to the device.

[0111] Specific operation: The server adds a new record to the database based on the user information, generates an authentication token, and returns it. The terminal receives the authentication token and displays a success message to the user.

[0112] Step 2:

[0113] The user logs in using an authentication token. The device displays makeup-related questions to the user. For example, questions like, "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The user enters their concerns and preferences and taps "Send." The device then sends this information to the server.

[0114] Input: User's concerns and desires regarding makeup

[0115] Data processing / calculation: The server uses conversational AI to analyze the user's input, which generates appropriate advice.

[0116] Output: Sends generated advice to the terminal

[0117] Specific operation: The server passes the user's input text to the conversational AI and receives the analysis results. Advice is generated as text and sent back to the terminal. The terminal displays the received advice to the user.

[0118] Step 3:

[0119] The device displays a message to the user urging them to take a picture of their face and upload it. The user takes a picture of their face using the smartphone camera and taps the "Upload" button to send it to the device. The device then sends this image data to the server.

[0120] Input: Image file of a natural face

[0121] Data processing / calculation: The server sends the received image file to the image generation AI, which analyzes the image and extracts the user's facial features.

[0122] Output: Analyzed facial feature data

[0123] How it works: The server passes the received image to a dedicated image analysis module to detect facial feature points and generate corresponding data, which will be used in the next step.

[0124] Step 4:

[0125] The server uses image generation AI to generate a make-up image of the user's face based on an image of their natural face and advice provided by the interactive AI.

[0126] Input: natural face image, conversational AI advice

[0127] Data processing / calculation: Image generation AI analyzes face shape and skin tone and generates a face image after makeup based on advice.

[0128] Output: Face image after makeup

[0129] Specific operation: The server uses image generation AI to apply makeup effects to a bare-face image and generate a new image. The generated facial image is then sent from the server to the device.

[0130] Step 5:

[0131] Based on the generated post-makeup facial image, the server selects information on beauty products and brands that are most suitable for the user and generates a corresponding purchase link.

[0132] Input: Face image after makeup

[0133] Data processing / calculation: The server uses a dedicated recommendation system to select appropriate beauty products to match the face image after makeup application.

[0134] Output: Beauty product information and purchase links

[0135] Specific operation: The server analyzes the facial image and selects the most suitable beauty products based on the database. Information about the selected beauty products and a link to purchase them are generated and sent to the terminal, which then displays them to the user.

[0136] Step 6:

[0137] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, information such as "Currently, products from a specific brand are 10% off."

[0138] Enter: Special offers and discounts

[0139] Data processing / calculation: The server periodically collects discount information from affiliated cosmetic manufacturers and selects special offers tailored to the user.

[0140] Output: Special offers and discounts

[0141] Specific operation: The server analyzes discount information received from manufacturers, extracts relevant information based on the user profile, and sends it to the device, which then displays it to the user.

[0142] (Application example 1)

[0143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] Conventional makeup support systems make it difficult for users to actually try on makeup products, making it difficult to find a makeup style that suits them before purchasing. Furthermore, because special offers and discount information are not provided, users may miss the optimal timing for purchasing. Furthermore, there is a lack of virtual makeup experience, making it impossible to visually confirm the realistic makeup look.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0146] In this invention, the server includes means for providing a mobile information terminal app that allows a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using a natural language processing AI to provide appropriate advice based on the input concerns and wishes, means for using an image generation AI to generate a made-up facial image based on the image uploaded by the user and the advice from the natural language processing AI, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on cosmetics and brands that are most suitable for the user, means for providing special offer and discount information, and means for using augmented reality technology to try on makeup in a virtual space. This allows the user to try on realistic makeup in a virtual space, making it easier to find the makeup style and cosmetics that are most suitable for them, and enabling them to receive special offer and discount information at the appropriate time.

[0147] A "mobile information terminal app" is application software that is installed on mobile information terminals such as smartphones and tablets.

[0148] "User" means a person who uses the services of this system.

[0149] "Natural language processing AI" is an artificial intelligence technology that deciphers, understands, and generates human language.

[0150] "Image generation AI" refers to artificial intelligence technology that uses algorithms to generate, transform, or modify images.

[0151] "Augmented reality technology" is a technology that overlays digital information onto real-world images and environments.

[0152] "Customized makeup advice" refers to advice on specific makeup methods and products provided based on a user's individual concerns and desires.

[0153] A "special offer" is a product or service offered at a more favorable price than is normally offered.

[0154] "Discount information" refers to information that allows you to purchase products or services at a price lower than the regular price.

[0155] A "virtual space" is a space that uses computer graphics and display technology to create a visual environment that differs from reality.

[0156] A "face image after makeup" is an image of the user's bare face after makeup effects have been applied.

[0157] "Information about cosmetics and brands" refers to information such as descriptions, features, and uses of specific cosmetics and their manufacturers.

[0158] A "purchase link" is a link to a web page where a user can purchase related products online.

[0159] The present invention is a support system that helps users find the makeup style that best suits them. This system is composed of a mobile information terminal app, a server, natural language processing AI, image generation AI, and augmented reality technology. The detailed configuration and operation procedure of the present invention are as follows.

[0160] Hardware and Software Configuration

[0161] Hardware used

[0162] Mobile devices: smartphones and tablets

[0163] Server: Server for main processing

[0164] Software used

[0165] Mobile device app: Developed with React Native

[0166] Server-side framework: Django

[0167] Natural Language Processing AI: Dialogflow

[0168] Image generation AI: TensorFlow or PyTorch

[0169] Augmented reality technology: ARKit or ARCore

[0170] System Operation

[0171] Step 1: Create an account and log in

[0172] The user enters the required information, such as name, email address, and password, into the device and sends it to the server. The server stores this information in a database, generates an authentication token, and returns it to the device. This logs the user in.

[0173] Step 2: Virtual Counseling

[0174] After logging in, the device asks the user a series of questions about makeup. The user's answers are sent from the device to the server, which uses natural language processing AI to analyze the answers, generate appropriate counseling results and advice, and return them to the device.

[0175] Step 3: Upload a photo of your bare face

[0176] Next, the device prompts the user to take a picture of their natural face and upload it. The user takes the picture and sends it from the device to the server. The server receives the image and sends it to the image generation AI.

[0177] Step 4: Generate a face image after makeup

[0178] Based on the bare face image sent to the server and the counseling content, the image generation AI generates a face image with makeup. For example, it generates a makeup image that covers redness based on the user's wishes. This generated face image with makeup is sent from the server to the device.

[0179] Step 5: Makeup product recommendations

[0180] Based on the generated makeup-applied facial image, the server generates information on the most suitable cosmetics and brands for the user, along with a link to purchase them. This information is sent to the terminal and displayed to the user.

[0181] Step 6: Provide special offers and discounts

[0182] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal, which displays this information to the user and assists them in their purchasing decisions.

[0183] Virtual makeup experience

[0184] Using the device, users can try on makeup in real time in a virtual space using augmented reality technology, allowing them to visually check which makeup looks best for them.

[0185] Prompt Sentence Examples

[0186] For example, if the user enters "I would like natural makeup that covers redness," the following prompt sentence is used to begin the consultation:

[0187] text

[0188] Input: I'd like to cover redness and have a natural looking makeup look. Can you give me some advice and product recommendations for this?

[0189] The system allows users to try on realistic makeup in a virtual world, helping them find the makeup style and cosmetics that best suit them, while also allowing them to receive special offers and discount information at the right time.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The terminal prompts the user to enter necessary information such as name, email address, and password. The entered information is sent to the server. The server stores this information in a database and creates a new account. The server also generates an authentication token and sends it back to the terminal. The terminal receives the authentication token and logs the user in. The input is the user's personal information, and the output is the authentication token.

[0193] Step 2:

[0194] The terminal displays a series of makeup-related questions to the logged-in user. The user answers the questions and inputs their wishes and concerns. The terminal sends this input to the server. The server uses natural language processing AI to analyze the answers and returns the generated counseling results and advice to the terminal. The input is the user's answers, and the output is the counseling results and advice.

[0195] Step 3:

[0196] The device then prompts the user to take an image of their natural face and upload it. The user takes an image using the camera on their mobile device and sends it to the server. The server receives this image and sends it to the image generation AI. The input is the image of the user's natural face, and the output is the server's receipt of the image.

[0197] Step 4:

[0198] The server uses image generation AI to generate a face image with makeup based on the received bare face image and the counseling content obtained in step 2. This image generation process simulates a specific makeup effect based on the image data and counseling content. The generated face image with makeup is sent from the server to the device. The input is the bare face image and counseling data, and the output is the face image with makeup.

[0199] Step 5:

[0200] The device displays the generated image of the user's face after makeup application to the user. The user can review this image and request changes. The server also generates information on the cosmetics and brands that are most suitable for the user based on the results of the image generation AI. It also generates a corresponding purchase link and sends it to the device. The input is the image of the face after makeup application, and the output is information on the cosmetics and brands and a purchase link.

[0201] Step 6:

[0202] The server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays this information to the user, allowing the user to avoid missing the optimal purchasing timing. The input is the special offers and discount information, and the output is the display of that information.

[0203] Step 7:

[0204] Using a device, users can try on makeup in real time in a virtual space using augmented reality technology. The device uses a camera to recognize the user's face and overlay makeup effects in real time, allowing users to visually check which makeup looks best on them. The input is a real-time image of the user's face, and the output is the makeup look in the virtual space.

[0205] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0206] The present invention relates to a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. The present invention is implemented by combining a smartphone app, a server, multiple artificial intelligence technologies, and an emotion engine.

[0207] Program processing and specific examples

[0208] Account creation and login

[0209] First, the user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0210] Makeup counseling

[0211] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and preferences, which the device then sends to the server. The server receives this information, analyzes the counseling content using conversational AI, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0212] emotion recognition

[0213] When a user enters counseling details or uploads images, the emotion engine analyzes the user's facial expressions and text data to recognize the user's emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion engine will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling details.

[0214] Upload a photo of your bare face

[0215] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0216] Generating face images after makeup

[0217] Based on the bare face image and counseling details sent to the server, as well as the user's emotional information recognized by the emotion engine, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. For example, if the user wishes to "cover up redness" and the emotion engine strongly senses this desire, the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup is sent to the device via the server, and the device displays it to the user.

[0218] Makeup product recommendations

[0219] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0220] Providing you with special offers and discounts

[0221] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0222] Specific examples

[0223] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0224] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0225] The processing flow will be explained below.

[0226] Step 1:

[0227] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0228] Terminal: Sends the entered information to the server.

[0229] Step 2:

[0230] Server: Stores the received user information in a database and generates an authentication token.

[0231] Server: Returns the authentication token to the device.

[0232] Terminal: Receives the authentication token and displays the login status to the user.

[0233] Step 3:

[0234] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0235] User: Enter their concerns and wishes in response to the questions.

[0236] Terminal: Sends the entered information to the server.

[0237] Step 4:

[0238] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0239] Server: Sends advice to the device.

[0240] Terminal: Display advice to the user.

[0241] Step 5:

[0242] Terminal: Displays a message instructing the user to take a picture of their input and their real face.

[0243] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0244] Device: Sends the captured image to the server.

[0245] Step 6:

[0246] Server: Processes the received face images and counseling content.

[0247] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0248] Server: Sends the generated face image after makeup to the device.

[0249] Terminal: Displays the face image after makeup to the user.

[0250] Step 7:

[0251] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0252] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0253] Device: Display recommendation information and purchase links to the user.

[0254] Step 8:

[0255] Server: Collects special offers and discounts from cosmetic manufacturers.

[0256] Server: Sends special offers and discounts to your device.

[0257] On the device: Show users special offers and discounts.

[0258] Step 9:

[0259] Device: When the user enters counseling content or uploads images, the emotion engine is activated and analyzes the user's facial expressions and text data.

[0260] Emotion engine (server): Recognizes the user's emotions and sends the analysis results to the server.

[0261] Server: Based on the analysis results from the emotion engine, the results are reflected in the analysis of conversational AI and image generation AI.

[0262] Specific examples

[0263] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0264] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0265] Example 2

[0266] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0267] Conventional makeup systems have difficulty fully reflecting the user's individual concerns and desires, and are unable to provide optimal suggestions that take the user's emotions into account. As a result, it has been difficult to improve user satisfaction. In addition, the information provided to users is general, limiting the ability to provide a personalized experience.

[0268] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for providing a mobile terminal application that allows a user to take and upload an image of their bare face; means for allowing the user to input their makeup concerns and wishes; means for using an interactive AI that provides appropriate advice based on the input concerns and wishes; means for using an image generation AI that generates a made-up face image based on the image uploaded by the user, the advice from the interactive AI, and the user's emotional information; means for displaying the generated made-up face image to the user; means for providing information on optimal cosmetics and brands and a purchase link for the user; means for using an emotion recognition engine that recognizes the user's emotions and suggests optimal makeup; and means for providing special offer and discount information. This enables optimal makeup suggestions that reflect the user's individual concerns, wishes, and emotions, thereby providing a personalized and highly satisfying experience.

[0269] A "mobile device application" is software that a user operates using a mobile device, and is a means for inputting information about makeup, taking and uploading images, and displaying the results.

[0270] "Conversational AI" is an AI system that analyzes input from users and generates appropriate advice and answers using natural language processing technology.

[0271] "Image generation artificial intelligence" is an artificial intelligence system that generates a face image with makeup applied based on a bare face image uploaded by the user, advice provided by an interactive artificial intelligence, and the user's emotional information.

[0272] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and text data to recognize their emotions and then makes appropriate makeup suggestions based on that information.

[0273] A "server" is a computer system that receives data sent by a user, executes various artificial intelligence processes, and returns the results to the user's mobile device.

[0274] The "means for providing information on cosmetics and brands and purchase links" is a function for presenting information on the cosmetics and brands that are most suitable for the user, as well as purchase links, based on the generated face image after makeup.

[0275] The "means for providing special offers and discount information" is a function for providing users with special offers and discount information obtained from various cosmetic manufacturers, thereby promoting purchasing behavior.

[0276] The present invention relates to a system that generates a personalized makeup-applied facial image based on a user's natural facial image and provides information on the most suitable cosmetics and brands. This system is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0277] A user first installs the mobile device application, and when they launch the app for the first time, an account creation screen appears. When the user enters their name, email address, and password and presses the submit button, the device sends this data to the server. The server creates a new user account in the database based on the received data, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0278] Next, the user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0279] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are asked. The user enters their answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0280] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. This engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0281] Next, the device displays a message prompting the user to take and upload an image of their bare face. The user takes an image of their bare face using the smartphone camera and uploads it to the server via the device. The server receives this image and sends it to the image generation AI. The image generation AI generates a face image with makeup applied based on the received bare face image, the advice provided by the conversational AI, and the emotional information. For example, if the user wishes to "cover up redness," and the emotion engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover up the user's redness. This generated image is sent back to the device via the server, and the device displays it to the user.

[0282] Furthermore, the server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the information includes recommendation information such as "A specific brand of foundation is suitable for this makeup style" and a link to the purchasing site. The server sends this information to the terminal, which displays it to the user.

[0283] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[0284] Here are some examples of prompts for generative AI models:

[0285] "Based on an image of the user's bare face, please generate an optimal makeup-applied image, taking into account the content of this consultation and the user's emotions. Also, please provide recommendations for makeup products that will be suitable."

[0286] As a result, the system of the present invention can provide optimal makeup suggestions and product recommendations that reflect the user's individual concerns, desires, and emotions, thereby achieving a high level of satisfaction.

[0287] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0288] Step 1:

[0289] The user installs the mobile device application and launches the account creation screen. The user provides input data such as name, email address, and password. The device sends this input data to the server. The server uses the received data to create a new user account in the database, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0290] Input: Name, Email Address, Password

[0291] Output: New user account created, authentication token

[0292] Step 2:

[0293] The user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0294] Input: Email address, password

[0295] Output: Authentication token, login status

[0296] Step 3:

[0297] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are displayed. The user enters the answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0298] Input: Answer to makeup question

[0299] Output: Analysis results and advice by interactive AI

[0300] Step 4:

[0301] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. The emotion recognition engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0302] Input: facial expression data, text data

[0303] Output: Recognized emotion information

[0304] Step 5:

[0305] The device then displays a message prompting the user to take and upload a picture of their face without makeup. The user then takes a picture of their face with their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0306] Input: natural face image

[0307] Output: The image of the face sent to the server

[0308] Step 6:

[0309] The server uses the image generation AI to generate a face image with makeup based on the received bare face image, advice provided by the conversational AI, and emotional information. For example, if the user wishes to "cover redness," and the emotion recognition engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover the user's redness. This generated image is sent back to the terminal via the server, and the terminal displays it to the user.

[0310] Input: Face image, advice, emotional information

[0311] Output: Face image after makeup

[0312] Step 7:

[0313] The server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the server may recommend a specific brand of foundation that is suitable for this makeup style, along with a link to a purchasing site. The server sends this information to the device, which then displays it to the user.

[0314] Input: Face image after makeup

[0315] Output: Cosmetics and brand information, purchase link

[0316] Step 8:

[0317] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[0318] Enter: Special offers and discounts

[0319] Output: Display of special offers and discounts

[0320] (Application example 2)

[0321] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0322] Conventional makeup experience systems have the drawback of requiring users to purchase numerous actual products to try out different makeup looks, or to seek expert advice, which is time-consuming and expensive. Furthermore, the lack of emotion recognition and real-time makeup simulation makes it difficult to provide optimal advice tailored to the user's individual emotions and preferences. A new system that solves these problems is needed.

[0323] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0324] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their concerns and desires regarding makeup, and means for using interactive artificial intelligence to provide appropriate advice based on the input concerns and desires.

[0325] The system includes a means for using an image generation AI to generate a made-up facial image based on an image uploaded by a user and advice from an interactive AI, a means for displaying the generated made-up facial image to the user, a means for providing information on cosmetics and brands that are most suitable for the user and a link to purchase them, a means for recognizing emotions by analyzing the user's facial expressions and text data, and a means for suggesting a makeup style based on the emotional information. This system enables users to experience makeup styles in real time that correspond to their concerns, wishes, and emotions.

[0326] The "mobile terminal application" is an application that allows a user to take an image of their bare face and input their concerns and desires regarding makeup.

[0327] "Conversational AI" is an AI that provides appropriate advice based on the user's concerns and desires regarding makeup.

[0328] "Image generation artificial intelligence" is an artificial intelligence that generates a face image after makeup based on an image uploaded by the user and advice from an interactive artificial intelligence.

[0329] "Emotion recognition" is a technology that recognizes a user's emotions by analyzing their facial expressions and text data.

[0330] The "means for proposing a makeup style" is a means for proposing an optimal makeup style based on emotional information and the user's worries and desires.

[0331] The "means for displaying the generated face image after makeup to the user" refers to a means for displaying the face image after makeup generated by the image generation artificial intelligence to the user.

[0332] The "means for providing information on the most suitable cosmetics and brands to the user and a purchase link" is a means for providing information on the most suitable cosmetics and brands to the user and a corresponding purchase link based on the generated face image after makeup application.

[0333] The present invention is a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. This system is implemented by combining mobile terminal applications, conversational artificial intelligence, image generation artificial intelligence, and emotion recognition technology.

[0334] The user installs the application for the first time and creates an account. The application sends necessary information such as name, email address, and password to the server. The server creates a user account in the database based on the received information, generates an authentication token, and returns it to the device. This logs the user in.

[0335] After logging in, the server presents the user with a series of makeup-related questions, prompting them to input their concerns and preferences. For example, questions such as "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The input information is sent to the server, where it is analyzed by an interactive AI. Advice generated as a result of the analysis is returned to the device and displayed to the user.

[0336] When a user uploads an image of their bare face, emotion recognition technology analyzes the user's facial expressions and text data to recognize their emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion recognition technology will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling content.

[0337] Next, the user takes a picture of their bare face using their smartphone camera and uploads it. This image is sent to a server where it is processed by the image generation AI. Based on the bare face image sent to the server, the consultation details, and the user's emotional information recognized by emotion recognition technology, the image generation AI analyzes the face shape and skin tone to generate an image of the face with makeup applied.

[0338] The generated image of the face with makeup applied is sent to the terminal via the server and displayed to the user. A specific example is when a user inputs a concern such as "I'm bothered by the redness of my face" and uploads an image of their bare face. Emotion recognition technology recognizes the emotion of "confusion," and the conversational AI and image generation AI perform analysis taking this emotion into account. The generated image of the face with makeup is displayed in a style that effectively covers the redness.

[0339] Based on the generated image, the server also provides users with information on the best cosmetics and brands. For example, specific information such as "This makeup style is best suited to a specific brand of foundation" is included. This information also includes a purchase link, allowing users to easily access the purchasing site. In addition, if there are any special offers or discount information, these will also be provided.

[0340] The hardware used includes smartphones and head-mounted displays (HMDs), and the software used includes mobile terminal applications, server-side conversational artificial intelligence (e.g., OpenAI GPT-4), emotion recognition technology (e.g., Emotion API), and image generation artificial intelligence (e.g., GANs).

[0341] An example prompt might look like this:

[0342] User input: "I'm concerned about redness on my face, so I'd like makeup that makes my skin look even."

[0343] Prompt: "User desires: Concerned about redness on face, Emotion: Confused. Please suggest an appropriate makeup style."

[0344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0345] Step 1:

[0346] A user installs a smartphone app and creates an account when they launch it for the first time. The user enters their name, email address, and password. The device sends this information to the server, and the server uses the received information to create a user account in a database, generates an authentication token, and returns it to the device. The input is user information, and the output is an authentication token. This operation puts the user in a logged-in state.

[0347] Step 2:

[0348] After the user logs in, the device displays a series of makeup-related questions to the user. The user inputs their concerns and desires in response to questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The input is the user's concerns and desires, and the output is text data. This data is sent from the device to the server.

[0349] Step 3:

[0350] The server passes the received text data to the conversational AI, which generates appropriate advice. The input is the text data of the user's concerns and wishes, and the output is the generated advice. The conversational AI analyzes this and generates advice customized for each user. The generated advice is sent from the server to the terminal, which displays it to the user.

[0351] Step 4:

[0352] When a user inputs counseling content, emotion recognition technology analyzes the user's facial expressions and text data to recognize the user's emotions. The input is the user's facial expression image and text data, and the output is recognized emotional information. For example, if a user inputs a concern that "my usual makeup doesn't work," the server will determine that the user is feeling dissatisfied or confused. This emotional information is also sent to the server and used to analyze the counseling content.

[0353] Step 5:

[0354] A user takes a picture of their face using a smartphone camera and uploads it to the server via their device. The input is the face image, and the output is the image data sent to the server. The server then sends the received face image to the image generation AI.

[0355] Step 6:

[0356] Based on the bare face image and counseling details sent to the server, as well as the emotional information recognized by emotion recognition technology, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. The input is the bare face image, the user's concerns and wishes, and emotional information, and the output is the face image with makeup applied.

[0357] Step 7:

[0358] The generated face image after makeup is sent from the server to the terminal and displayed to the user. The input is the generated face image, and the output is the face image displayed on the terminal. As a specific example, if a user inputs "I'm troubled by the redness of my face" and the emotion recognition technology recognizes "confused," the image generation technology will suggest a makeup style that covers the redness and display that image to the user.

[0359] Step 8:

[0360] The server provides the user with information on the most suitable cosmetics and brands based on the generated face image after makeup. The input is the face image after makeup, and the output is recommendation information. For example, it includes specific information such as "A specific brand of foundation is suitable for this makeup style." This information also includes a purchase link, allowing the user to easily access the purchasing site.

[0361] Step 9:

[0362] If there is special offer or discount information, the server collects it and sends it to the terminal. The input is the special offer or discount information, and the output is the displayed special offer or discount information. The user can use this information to purchase cosmetics at a discount.

[0363] This series of processing steps allows the user to experience the most suitable makeup style in real time according to their desires and emotions.

[0364] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0365] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0366] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0367] [Second embodiment]

[0368] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0369] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0370] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0371] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0372] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0373] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0374] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0375] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0376] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0377] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0378] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0379] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0380] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a smartphone app, a server, and multiple artificial intelligence technologies.

[0381] Program processing and specific examples

[0382] Account creation and login

[0383] First, a user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0384] Makeup counseling

[0385] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and wishes, and the device sends this information to the server. The server receives this information, analyzes the counseling content using conversational AI (e.g., artificial intelligence that processes natural language), generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0386] Upload a photo of your bare face

[0387] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0388] Generating face images after makeup

[0389] Based on the bare face image sent to the server and the counseling details, the image generation AI analyzes the user's face shape and skin tone and generates a face image with makeup applied. For example, if the user requests to "cover up redness," the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup applied is sent via the server to the device, which then displays it to the user.

[0390] Makeup product recommendations

[0391] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0392] Providing you with special offers and discounts

[0393] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0394] The system of the present invention provides an environment where users can easily find the makeup style that suits them and select the most suitable products, thereby significantly improving the user's beauty experience.

[0395] The processing flow will be explained below.

[0396] Step 1:

[0397] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0398] Terminal: Sends the entered information to the server.

[0399] Step 2:

[0400] Server: Stores the received user information in a database and generates an authentication token.

[0401] Server: Returns the authentication token to the device.

[0402] Terminal: Receives the authentication token and displays the login status to the user.

[0403] Step 3:

[0404] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0405] User: Enter their concerns and wishes in response to the questions.

[0406] Terminal: Sends the entered information to the server.

[0407] Step 4:

[0408] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0409] Server: Sends advice to the device.

[0410] Terminal: Display advice to the user.

[0411] Step 5:

[0412] On the device: Prompt the user to take and upload a photo of their face.

[0413] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0414] Device: Sends the captured image to the server.

[0415] Step 6:

[0416] Server: Processes the received face images and counseling content.

[0417] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0418] Server: Sends the generated face image after makeup to the device.

[0419] Terminal: Displays the face image after makeup to the user.

[0420] Step 7:

[0421] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0422] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0423] Device: Display recommendation information and purchase links to the user.

[0424] Step 8:

[0425] Server: Collects special offers and discounts from cosmetic manufacturers.

[0426] Server: Sends special offers and discounts to your device.

[0427] On the device: Show users special offers and discounts.

[0428] Example 1

[0429] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0430] Conventional makeup advisor systems have difficulty providing appropriate advice based on users' concerns and desires, and often take time to select the most suitable makeup style and cosmetics for the user. Furthermore, it is difficult to realistically reproduce facial images after makeup application, which can result in low user satisfaction. Furthermore, the provision of special offers and discount information is not centralized, making it difficult for users to receive this information at the appropriate time.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using an interactive artificial intelligence that provides appropriate advice based on the input concerns and wishes, means for using an image generation artificial intelligence that generates a made-up facial image based on the image uploaded by the user and the advice from the interactive artificial intelligence, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on beauty products and brands that are most suitable for the user, and means for collecting and providing special offer and discount information. This allows users to easily select makeup styles and cosmetics that suit them, check realistic made-up facial images, and also receive special offer and discount information in a timely manner.

[0433] A "mobile device application" is software that is installed on a mobile phone or tablet device and provides users with the ability to take and upload images of their natural face.

[0434] "Conversational artificial intelligence" is an artificial intelligence technology that has the ability to analyze text data entered by the user and provide appropriate responses and advice in natural language.

[0435] "Image generation artificial intelligence" is an artificial intelligence technology that has the ability to generate new images based on image data and related information uploaded by users.

[0436] "Beauty products" is a general term for products, such as cosmetics and skin care products, used to assist or improve a user's appearance or beauty.

[0437] "Purchase Link" means a hyperlink that provides direct access to a web page for purchasing a particular product or service.

[0438] A "special offer" is a temporary promotional activity that offers a lower price or special benefits than the regular selling price.

[0439] "Discount information" is information that notifies users that the price of a product will be cheaper than usual for a certain period of time.

[0440] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0441] The mobile application is software that users install on their smartphones or tablets and provides the functionality to take and upload images of their faces, allowing users to provide images of their own faces to the system.

[0442] The server provides the following main functions: First, it receives information such as the user's name, email address, and password entered into the mobile device application and creates a user account in the database. Next, the interactive AI generates appropriate advice based on the user's makeup concerns and preferences and provides it to the user. Furthermore, it processes the bare-face image uploaded by the user and generates a face image with makeup using the image generation AI. Finally, based on the generated face image with makeup, it provides the user with information on the most suitable beauty products and brands, purchase links, special offers, and discount information.

[0443] Conversational AI analyzes text data entered by the user (e.g., "What are your skin concerns?", "What kind of makeup style are you aiming for?") and provides appropriate responses and advice in natural language. For example, if a user enters, "It's dry today, so I want to wear makeup that emphasizes moisturizing," the conversational AI will generate a response such as, "Moisturizing is important. Next, please upload an image of your bare face."

[0444] Image generation AI is a technology that generates a new make-up image of a face based on an image of the user's natural face and advice from an interactive AI. For example, if a user requests to "cover up redness," the image generation AI generates a face image with natural make-up that covers up the redness and sends it to the terminal via the server. The terminal then displays this make-up image to the user.

[0445] The server then provides the user with information on the most suitable beauty products and brands, along with a link to purchase them, based on the generated face image after makeup application. For example, specific recommendation information such as "This makeup style is best suited to a specific brand of foundation" is included. This allows the user to easily access the purchasing site.

[0446] In addition, the server collects special offers and discount information from cosmetics manufacturers and provides this information to users. The terminal supports users' purchasing behavior by displaying information such as "Currently, products from a specific brand are 10% off."

[0447] In this way, the system of the present invention provides an environment in which users can easily find a makeup style that suits them and select the most suitable products, thereby significantly improving their beauty experience.

[0448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0449] Step 1:

[0450] The user installs and launches the mobile application. The application prompts the user to enter information such as their name, email address, and password. After the user enters the required information and taps the "Submit" button, the device sends this information to the server.

[0451] Input: User information such as name, email address, and password

[0452] Data processing / calculation: The server registers the received information in the database and creates a user account. Once registration is complete, an authentication token is generated.

[0453] Output: Sends the authentication token to the device.

[0454] Specific operation: The server adds a new record to the database based on the user information, generates an authentication token, and returns it. The terminal receives the authentication token and displays a success message to the user.

[0455] Step 2:

[0456] The user logs in using an authentication token. The device displays makeup-related questions to the user. For example, questions like, "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The user enters their concerns and preferences and taps "Send." The device then sends this information to the server.

[0457] Input: User's concerns and desires regarding makeup

[0458] Data processing / calculation: The server uses conversational AI to analyze the user's input, which generates appropriate advice.

[0459] Output: Sends generated advice to the terminal

[0460] Specific operation: The server passes the user's input text to the conversational AI and receives the analysis results. Advice is generated as text and sent back to the terminal. The terminal displays the received advice to the user.

[0461] Step 3:

[0462] The device displays a message to the user urging them to take a picture of their face and upload it. The user takes a picture of their face using the smartphone camera and taps the "Upload" button to send it to the device. The device then sends this image data to the server.

[0463] Input: Image file of a natural face

[0464] Data processing / calculation: The server sends the received image file to the image generation AI, which analyzes the image and extracts the user's facial features.

[0465] Output: Analyzed facial feature data

[0466] How it works: The server passes the received image to a dedicated image analysis module to detect facial feature points and generate corresponding data, which will be used in the next step.

[0467] Step 4:

[0468] The server uses image generation AI to generate a make-up image of the user's face based on an image of their natural face and advice provided by the interactive AI.

[0469] Input: natural face image, conversational AI advice

[0470] Data processing / calculation: Image generation AI analyzes face shape and skin tone and generates a face image after makeup based on advice.

[0471] Output: Face image after makeup

[0472] Specific operation: The server uses image generation AI to apply makeup effects to a bare-face image and generate a new image. The generated facial image is then sent from the server to the device.

[0473] Step 5:

[0474] Based on the generated post-makeup facial image, the server selects information on beauty products and brands that are most suitable for the user and generates a corresponding purchase link.

[0475] Input: Face image after makeup

[0476] Data processing / calculation: The server uses a dedicated recommendation system to select appropriate beauty products to match the face image after makeup application.

[0477] Output: Beauty product information and purchase links

[0478] Specific operation: The server analyzes the facial image and selects the most suitable beauty products based on the database. Information about the selected beauty products and a link to purchase them are generated and sent to the terminal, which then displays them to the user.

[0479] Step 6:

[0480] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, information such as "Currently, products from a specific brand are 10% off."

[0481] Enter: Special offers and discounts

[0482] Data processing / calculation: The server periodically collects discount information from affiliated cosmetic manufacturers and selects special offers tailored to the user.

[0483] Output: Special offers and discounts

[0484] Specific operation: The server analyzes discount information received from manufacturers, extracts relevant information based on the user profile, and sends it to the device, which then displays it to the user.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] Conventional makeup support systems make it difficult for users to actually try on makeup products, making it difficult to find a makeup style that suits them before purchasing. Furthermore, because special offers and discount information are not provided, users may miss the optimal timing for purchasing. Furthermore, there is a lack of virtual makeup experience, making it impossible to visually confirm the realistic makeup look.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes means for providing a mobile information terminal app that allows a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using a natural language processing AI to provide appropriate advice based on the input concerns and wishes, means for using an image generation AI to generate a made-up facial image based on the image uploaded by the user and the advice from the natural language processing AI, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on cosmetics and brands that are most suitable for the user, means for providing special offer and discount information, and means for using augmented reality technology to try on makeup in a virtual space. This allows the user to try on realistic makeup in a virtual space, making it easier to find the makeup style and cosmetics that are most suitable for them, and enabling them to receive special offer and discount information at the appropriate time.

[0490] A "mobile information terminal app" is application software that is installed on mobile information terminals such as smartphones and tablets.

[0491] "User" means a person who uses the services of this system.

[0492] "Natural language processing AI" is an artificial intelligence technology that deciphers, understands, and generates human language.

[0493] "Image generation AI" refers to artificial intelligence technology that uses algorithms to generate, transform, or modify images.

[0494] "Augmented reality technology" is a technology that overlays digital information onto real-world images and environments.

[0495] "Customized makeup advice" refers to advice on specific makeup methods and products provided based on a user's individual concerns and desires.

[0496] A "special offer" is a product or service offered at a more favorable price than is normally offered.

[0497] "Discount information" refers to information that allows you to purchase products or services at a price lower than the regular price.

[0498] A "virtual space" is a space that uses computer graphics and display technology to create a visual environment that differs from reality.

[0499] A "face image after makeup" is an image of the user's bare face after makeup effects have been applied.

[0500] "Information about cosmetics and brands" refers to information such as descriptions, features, and uses of specific cosmetics and their manufacturers.

[0501] A "purchase link" is a link to a web page where a user can purchase related products online.

[0502] The present invention is a support system that helps users find the makeup style that best suits them. This system is composed of a mobile information terminal app, a server, natural language processing AI, image generation AI, and augmented reality technology. The detailed configuration and operation procedure of the present invention are as follows.

[0503] Hardware and Software Configuration

[0504] Hardware used

[0505] Mobile devices: smartphones and tablets

[0506] Server: Server for main processing

[0507] Software used

[0508] Mobile device app: Developed with React Native

[0509] Server-side framework: Django

[0510] Natural Language Processing AI: Dialogflow

[0511] Image generation AI: TensorFlow or PyTorch

[0512] Augmented reality technology: ARKit or ARCore

[0513] System Operation

[0514] Step 1: Create an account and log in

[0515] The user enters the required information, such as name, email address, and password, into the device and sends it to the server. The server stores this information in a database, generates an authentication token, and returns it to the device. This logs the user in.

[0516] Step 2: Virtual Counseling

[0517] After logging in, the device asks the user a series of questions about makeup. The user's answers are sent from the device to the server, which uses natural language processing AI to analyze the answers, generate appropriate counseling results and advice, and return them to the device.

[0518] Step 3: Upload a photo of your bare face

[0519] Next, the device prompts the user to take a picture of their natural face and upload it. The user takes the picture and sends it from the device to the server. The server receives the image and sends it to the image generation AI.

[0520] Step 4: Generate a face image after makeup

[0521] Based on the bare face image sent to the server and the counseling content, the image generation AI generates a face image with makeup. For example, it generates a makeup image that covers redness based on the user's wishes. This generated face image with makeup is sent from the server to the device.

[0522] Step 5: Makeup product recommendations

[0523] Based on the generated makeup-applied facial image, the server generates information on the most suitable cosmetics and brands for the user, along with a link to purchase them. This information is sent to the terminal and displayed to the user.

[0524] Step 6: Provide special offers and discounts

[0525] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal, which displays this information to the user and assists them in their purchasing decisions.

[0526] Virtual makeup experience

[0527] Using the device, users can try on makeup in real time in a virtual space using augmented reality technology, allowing them to visually check which makeup looks best for them.

[0528] Prompt Sentence Examples

[0529] For example, if the user enters "I would like natural makeup that covers redness," the following prompt sentence is used to begin the consultation:

[0530] text

[0531] Input: I'd like to cover redness and have a natural looking makeup look. Can you give me some advice and product recommendations for this?

[0532] The system allows users to try on realistic makeup in a virtual world, helping them find the makeup style and cosmetics that best suit them, while also allowing them to receive special offers and discount information at the right time.

[0533] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0534] Step 1:

[0535] The terminal prompts the user to enter necessary information such as name, email address, and password. The entered information is sent to the server. The server stores this information in a database and creates a new account. The server also generates an authentication token and sends it back to the terminal. The terminal receives the authentication token and logs the user in. The input is the user's personal information, and the output is the authentication token.

[0536] Step 2:

[0537] The terminal displays a series of makeup-related questions to the logged-in user. The user answers the questions and inputs their wishes and concerns. The terminal sends this input to the server. The server uses natural language processing AI to analyze the answers and returns the generated counseling results and advice to the terminal. The input is the user's answers, and the output is the counseling results and advice.

[0538] Step 3:

[0539] The device then prompts the user to take an image of their natural face and upload it. The user takes an image using the camera on their mobile device and sends it to the server. The server receives this image and sends it to the image generation AI. The input is the image of the user's natural face, and the output is the server's receipt of the image.

[0540] Step 4:

[0541] The server uses image generation AI to generate a face image with makeup based on the received bare face image and the counseling content obtained in step 2. This image generation process simulates a specific makeup effect based on the image data and counseling content. The generated face image with makeup is sent from the server to the device. The input is the bare face image and counseling data, and the output is the face image with makeup.

[0542] Step 5:

[0543] The device displays the generated image of the user's face after makeup application to the user. The user can review this image and request changes. The server also generates information on the cosmetics and brands that are most suitable for the user based on the results of the image generation AI. It also generates a corresponding purchase link and sends it to the device. The input is the image of the face after makeup application, and the output is information on the cosmetics and brands and a purchase link.

[0544] Step 6:

[0545] The server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays this information to the user, allowing the user to avoid missing the optimal purchasing timing. The input is the special offers and discount information, and the output is the display of that information.

[0546] Step 7:

[0547] Using a device, users can try on makeup in real time in a virtual space using augmented reality technology. The device uses a camera to recognize the user's face and overlay makeup effects in real time, allowing users to visually check which makeup looks best on them. The input is a real-time image of the user's face, and the output is the makeup look in the virtual space.

[0548] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0549] The present invention relates to a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. The present invention is implemented by combining a smartphone app, a server, multiple artificial intelligence technologies, and an emotion engine.

[0550] Program processing and specific examples

[0551] Account creation and login

[0552] First, the user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0553] Makeup counseling

[0554] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and preferences, which the device then sends to the server. The server receives this information, analyzes the counseling content using conversational AI, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0555] emotion recognition

[0556] When a user enters counseling details or uploads images, the emotion engine analyzes the user's facial expressions and text data to recognize the user's emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion engine will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling details.

[0557] Upload a photo of your bare face

[0558] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0559] Generating face images after makeup

[0560] Based on the bare face image and counseling details sent to the server, as well as the user's emotional information recognized by the emotion engine, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. For example, if the user wishes to "cover up redness" and the emotion engine strongly senses this desire, the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup is sent to the device via the server, and the device displays it to the user.

[0561] Makeup product recommendations

[0562] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0563] Providing you with special offers and discounts

[0564] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0565] Specific examples

[0566] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0567] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0568] The processing flow will be explained below.

[0569] Step 1:

[0570] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0571] Terminal: Sends the entered information to the server.

[0572] Step 2:

[0573] Server: Stores the received user information in a database and generates an authentication token.

[0574] Server: Returns the authentication token to the device.

[0575] Terminal: Receives the authentication token and displays the login status to the user.

[0576] Step 3:

[0577] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0578] User: Enter their concerns and wishes in response to the questions.

[0579] Terminal: Sends the entered information to the server.

[0580] Step 4:

[0581] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0582] Server: Sends advice to the device.

[0583] Terminal: Display advice to the user.

[0584] Step 5:

[0585] Terminal: Displays a message instructing the user to take a picture of their input and their real face.

[0586] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0587] Device: Sends the captured image to the server.

[0588] Step 6:

[0589] Server: Processes the received face images and counseling content.

[0590] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0591] Server: Sends the generated face image after makeup to the device.

[0592] Terminal: Displays the face image after makeup to the user.

[0593] Step 7:

[0594] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0595] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0596] Device: Display recommendation information and purchase links to the user.

[0597] Step 8:

[0598] Server: Collects special offers and discounts from cosmetic manufacturers.

[0599] Server: Sends special offers and discounts to your device.

[0600] On the device: Show users special offers and discounts.

[0601] Step 9:

[0602] Device: When the user enters counseling content or uploads images, the emotion engine is activated and analyzes the user's facial expressions and text data.

[0603] Emotion engine (server): Recognizes the user's emotions and sends the analysis results to the server.

[0604] Server: Based on the analysis results from the emotion engine, the results are reflected in the analysis of conversational AI and image generation AI.

[0605] Specific examples

[0606] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0607] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0608] Example 2

[0609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] Conventional makeup systems have difficulty fully reflecting the user's individual concerns and desires, and are unable to provide optimal suggestions that take the user's emotions into account. As a result, it has been difficult to improve user satisfaction. In addition, the information provided to users is general, limiting the ability to provide a personalized experience.

[0611] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for providing a mobile terminal application that allows a user to take and upload an image of their bare face; means for allowing the user to input their makeup concerns and wishes; means for using an interactive AI that provides appropriate advice based on the input concerns and wishes; means for using an image generation AI that generates a made-up face image based on the image uploaded by the user, the advice from the interactive AI, and the user's emotional information; means for displaying the generated made-up face image to the user; means for providing information on optimal cosmetics and brands and a purchase link for the user; means for using an emotion recognition engine that recognizes the user's emotions and suggests optimal makeup; and means for providing special offer and discount information. This enables optimal makeup suggestions that reflect the user's individual concerns, wishes, and emotions, thereby providing a personalized and highly satisfying experience.

[0612] A "mobile device application" is software that a user operates using a mobile device, and is a means for inputting information about makeup, taking and uploading images, and displaying the results.

[0613] "Conversational AI" is an AI system that analyzes input from users and generates appropriate advice and answers using natural language processing technology.

[0614] "Image generation artificial intelligence" is an artificial intelligence system that generates a face image with makeup applied based on a bare face image uploaded by the user, advice provided by an interactive artificial intelligence, and the user's emotional information.

[0615] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and text data to recognize their emotions and then makes appropriate makeup suggestions based on that information.

[0616] A "server" is a computer system that receives data sent by a user, executes various artificial intelligence processes, and returns the results to the user's mobile device.

[0617] The "means for providing information on cosmetics and brands and purchase links" is a function for presenting information on the cosmetics and brands that are most suitable for the user, as well as purchase links, based on the generated face image after makeup.

[0618] The "means for providing special offers and discount information" is a function for providing users with special offers and discount information obtained from various cosmetic manufacturers, thereby promoting purchasing behavior.

[0619] The present invention relates to a system that generates a personalized makeup-applied facial image based on a user's natural facial image and provides information on the most suitable cosmetics and brands. This system is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0620] A user first installs the mobile device application, and when they launch the app for the first time, an account creation screen appears. When the user enters their name, email address, and password and presses the submit button, the device sends this data to the server. The server creates a new user account in the database based on the received data, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0621] Next, the user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0622] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are asked. The user enters their answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0623] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. This engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0624] Next, the device displays a message prompting the user to take and upload an image of their bare face. The user takes an image of their bare face using the smartphone camera and uploads it to the server via the device. The server receives this image and sends it to the image generation AI. The image generation AI generates a face image with makeup applied based on the received bare face image, the advice provided by the conversational AI, and the emotional information. For example, if the user wishes to "cover up redness," and the emotion engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover up the user's redness. This generated image is sent back to the device via the server, and the device displays it to the user.

[0625] Furthermore, the server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the information includes recommendation information such as "A specific brand of foundation is suitable for this makeup style" and a link to the purchasing site. The server sends this information to the terminal, which displays it to the user.

[0626] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[0627] Here are some examples of prompts for generative AI models:

[0628] "Based on an image of the user's bare face, please generate an optimal makeup-applied image, taking into account the content of this consultation and the user's emotions. Also, please provide recommendations for makeup products that will be suitable."

[0629] As a result, the system of the present invention can provide optimal makeup suggestions and product recommendations that reflect the user's individual concerns, desires, and emotions, thereby achieving a high level of satisfaction.

[0630] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0631] Step 1:

[0632] The user installs the mobile device application and launches the account creation screen. The user provides input data such as name, email address, and password. The device sends this input data to the server. The server uses the received data to create a new user account in the database, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0633] Input: Name, Email Address, Password

[0634] Output: New user account created, authentication token

[0635] Step 2:

[0636] The user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0637] Input: Email address, password

[0638] Output: Authentication token, login status

[0639] Step 3:

[0640] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are displayed. The user enters the answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0641] Input: Answer to makeup question

[0642] Output: Analysis results and advice by interactive AI

[0643] Step 4:

[0644] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. The emotion recognition engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0645] Input: facial expression data, text data

[0646] Output: Recognized emotion information

[0647] Step 5:

[0648] The device then displays a message prompting the user to take and upload a picture of their face without makeup. The user then takes a picture of their face with their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0649] Input: natural face image

[0650] Output: The image of the face sent to the server

[0651] Step 6:

[0652] The server uses the image generation AI to generate a face image with makeup based on the received bare face image, advice provided by the conversational AI, and emotional information. For example, if the user wishes to "cover redness," and the emotion recognition engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover the user's redness. This generated image is sent back to the terminal via the server, and the terminal displays it to the user.

[0653] Input: Face image, advice, emotional information

[0654] Output: Face image after makeup

[0655] Step 7:

[0656] The server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the server may recommend a specific brand of foundation that is suitable for this makeup style, along with a link to a purchasing site. The server sends this information to the device, which then displays it to the user.

[0657] Input: Face image after makeup

[0658] Output: Cosmetics and brand information, purchase link

[0659] Step 8:

[0660] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[0661] Enter: Special offers and discounts

[0662] Output: Display of special offers and discounts

[0663] (Application example 2)

[0664] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Conventional makeup experience systems have the drawback of requiring users to purchase numerous actual products to try out different makeup looks, or to seek expert advice, which is time-consuming and expensive. Furthermore, the lack of emotion recognition and real-time makeup simulation makes it difficult to provide optimal advice tailored to the user's individual emotions and preferences. A new system that solves these problems is needed.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0667] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their concerns and desires regarding makeup, and means for using interactive artificial intelligence to provide appropriate advice based on the input concerns and desires.

[0668] The system includes a means for using an image generation AI to generate a made-up facial image based on an image uploaded by a user and advice from an interactive AI, a means for displaying the generated made-up facial image to the user, a means for providing information on cosmetics and brands that are most suitable for the user and a link to purchase them, a means for recognizing emotions by analyzing the user's facial expressions and text data, and a means for suggesting a makeup style based on the emotional information. This system enables users to experience makeup styles in real time that correspond to their concerns, wishes, and emotions.

[0669] The "mobile terminal application" is an application that allows a user to take an image of their bare face and input their concerns and desires regarding makeup.

[0670] "Conversational AI" is an AI that provides appropriate advice based on the user's concerns and desires regarding makeup.

[0671] "Image generation artificial intelligence" is an artificial intelligence that generates a face image after makeup based on an image uploaded by the user and advice from an interactive artificial intelligence.

[0672] "Emotion recognition" is a technology that recognizes a user's emotions by analyzing their facial expressions and text data.

[0673] The "means for proposing a makeup style" is a means for proposing an optimal makeup style based on emotional information and the user's worries and desires.

[0674] The "means for displaying the generated face image after makeup to the user" refers to a means for displaying the face image after makeup generated by the image generation artificial intelligence to the user.

[0675] The "means for providing information on the most suitable cosmetics and brands to the user and a purchase link" is a means for providing information on the most suitable cosmetics and brands to the user and a corresponding purchase link based on the generated face image after makeup application.

[0676] The present invention is a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. This system is implemented by combining mobile terminal applications, conversational artificial intelligence, image generation artificial intelligence, and emotion recognition technology.

[0677] The user installs the application for the first time and creates an account. The application sends necessary information such as name, email address, and password to the server. The server creates a user account in the database based on the received information, generates an authentication token, and returns it to the device. This logs the user in.

[0678] After logging in, the server presents the user with a series of makeup-related questions, prompting them to input their concerns and preferences. For example, questions such as "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The input information is sent to the server, where it is analyzed by an interactive AI. Advice generated as a result of the analysis is returned to the device and displayed to the user.

[0679] When a user uploads an image of their bare face, emotion recognition technology analyzes the user's facial expressions and text data to recognize their emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion recognition technology will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling content.

[0680] Next, the user takes a picture of their bare face using their smartphone camera and uploads it. This image is sent to a server where it is processed by the image generation AI. Based on the bare face image sent to the server, the consultation details, and the user's emotional information recognized by emotion recognition technology, the image generation AI analyzes the face shape and skin tone to generate an image of the face with makeup applied.

[0681] The generated image of the face with makeup applied is sent to the terminal via the server and displayed to the user. A specific example is when a user inputs a concern such as "I'm bothered by the redness of my face" and uploads an image of their bare face. Emotion recognition technology recognizes the emotion of "confusion," and the conversational AI and image generation AI perform analysis taking this emotion into account. The generated image of the face with makeup is displayed in a style that effectively covers the redness.

[0682] Based on the generated image, the server also provides users with information on the best cosmetics and brands. For example, specific information such as "This makeup style is best suited to a specific brand of foundation" is included. This information also includes a purchase link, allowing users to easily access the purchasing site. In addition, if there are any special offers or discount information, these will also be provided.

[0683] The hardware used includes smartphones and head-mounted displays (HMDs), and the software used includes mobile terminal applications, server-side conversational artificial intelligence (e.g., OpenAI GPT-4), emotion recognition technology (e.g., Emotion API), and image generation artificial intelligence (e.g., GANs).

[0684] An example prompt might look like this:

[0685] User input: "I'm concerned about redness on my face, so I'd like makeup that makes my skin look even."

[0686] Prompt: "User desires: Concerned about redness on face, Emotion: Confused. Please suggest an appropriate makeup style."

[0687] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0688] Step 1:

[0689] A user installs a smartphone app and creates an account when they launch it for the first time. The user enters their name, email address, and password. The device sends this information to the server, and the server uses the received information to create a user account in a database, generates an authentication token, and returns it to the device. The input is user information, and the output is an authentication token. This operation puts the user in a logged-in state.

[0690] Step 2:

[0691] After the user logs in, the device displays a series of makeup-related questions to the user. The user inputs their concerns and desires in response to questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The input is the user's concerns and desires, and the output is text data. This data is sent from the device to the server.

[0692] Step 3:

[0693] The server passes the received text data to the conversational AI, which generates appropriate advice. The input is the text data of the user's concerns and wishes, and the output is the generated advice. The conversational AI analyzes this and generates advice customized for each user. The generated advice is sent from the server to the terminal, which displays it to the user.

[0694] Step 4:

[0695] When a user inputs counseling content, emotion recognition technology analyzes the user's facial expressions and text data to recognize the user's emotions. The input is the user's facial expression image and text data, and the output is recognized emotional information. For example, if a user inputs a concern that "my usual makeup doesn't work," the server will determine that the user is feeling dissatisfied or confused. This emotional information is also sent to the server and used to analyze the counseling content.

[0696] Step 5:

[0697] A user takes a picture of their face using a smartphone camera and uploads it to the server via their device. The input is the face image, and the output is the image data sent to the server. The server then sends the received face image to the image generation AI.

[0698] Step 6:

[0699] Based on the bare face image and counseling details sent to the server, as well as the emotional information recognized by emotion recognition technology, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. The input is the bare face image, the user's concerns and wishes, and emotional information, and the output is the face image with makeup applied.

[0700] Step 7:

[0701] The generated face image after makeup is sent from the server to the terminal and displayed to the user. The input is the generated face image, and the output is the face image displayed on the terminal. As a specific example, if a user inputs "I'm troubled by the redness of my face" and the emotion recognition technology recognizes "confused," the image generation technology will suggest a makeup style that covers the redness and display that image to the user.

[0702] Step 8:

[0703] The server provides the user with information on the most suitable cosmetics and brands based on the generated face image after makeup. The input is the face image after makeup, and the output is recommendation information. For example, it includes specific information such as "A specific brand of foundation is suitable for this makeup style." This information also includes a purchase link, allowing the user to easily access the purchasing site.

[0704] Step 9:

[0705] If there is special offer or discount information, the server collects it and sends it to the terminal. The input is the special offer or discount information, and the output is the displayed special offer or discount information. The user can use this information to purchase cosmetics at a discount.

[0706] This series of processing steps allows the user to experience the most suitable makeup style in real time according to their desires and emotions.

[0707] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0708] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0709] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0710] [Third embodiment]

[0711] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0712] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0713] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0714] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0715] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0716] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0717] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0718] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0719] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0720] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0721] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0722] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0723] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a smartphone app, a server, and multiple artificial intelligence technologies.

[0724] Program processing and specific examples

[0725] Account creation and login

[0726] First, a user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0727] Makeup counseling

[0728] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and wishes, and the device sends this information to the server. The server receives this information, analyzes the counseling content using conversational AI (e.g., artificial intelligence that processes natural language), generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0729] Upload a photo of your bare face

[0730] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0731] Generating face images after makeup

[0732] Based on the bare face image sent to the server and the counseling details, the image generation AI analyzes the user's face shape and skin tone and generates a face image with makeup applied. For example, if the user requests to "cover up redness," the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup applied is sent via the server to the device, which then displays it to the user.

[0733] Makeup product recommendations

[0734] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0735] Providing you with special offers and discounts

[0736] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0737] The system of the present invention provides an environment where users can easily find the makeup style that suits them and select the most suitable products, thereby significantly improving the user's beauty experience.

[0738] The processing flow will be explained below.

[0739] Step 1:

[0740] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0741] Terminal: Sends the entered information to the server.

[0742] Step 2:

[0743] Server: Stores the received user information in a database and generates an authentication token.

[0744] Server: Returns the authentication token to the device.

[0745] Terminal: Receives the authentication token and displays the login status to the user.

[0746] Step 3:

[0747] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0748] User: Enter their concerns and wishes in response to the questions.

[0749] Terminal: Sends the entered information to the server.

[0750] Step 4:

[0751] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0752] Server: Sends advice to the device.

[0753] Terminal: Display advice to the user.

[0754] Step 5:

[0755] On the device: Prompt the user to take and upload a photo of their face.

[0756] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0757] Device: Sends the captured image to the server.

[0758] Step 6:

[0759] Server: Processes the received face images and counseling content.

[0760] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0761] Server: Sends the generated face image after makeup to the device.

[0762] Terminal: Displays the face image after makeup to the user.

[0763] Step 7:

[0764] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0765] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0766] Device: Display recommendation information and purchase links to the user.

[0767] Step 8:

[0768] Server: Collects special offers and discounts from cosmetic manufacturers.

[0769] Server: Sends special offers and discounts to your device.

[0770] On the device: Show users special offers and discounts.

[0771] Example 1

[0772] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0773] Conventional makeup advisor systems have difficulty providing appropriate advice based on users' concerns and desires, and often take time to select the most suitable makeup style and cosmetics for the user. Furthermore, it is difficult to realistically reproduce facial images after makeup application, which can result in low user satisfaction. Furthermore, the provision of special offers and discount information is not centralized, making it difficult for users to receive this information at the appropriate time.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0775] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using an interactive artificial intelligence that provides appropriate advice based on the input concerns and wishes, means for using an image generation artificial intelligence that generates a made-up facial image based on the image uploaded by the user and the advice from the interactive artificial intelligence, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on beauty products and brands that are most suitable for the user, and means for collecting and providing special offer and discount information. This allows users to easily select makeup styles and cosmetics that suit them, check realistic made-up facial images, and also receive special offer and discount information in a timely manner.

[0776] A "mobile device application" is software that is installed on a mobile phone or tablet device and provides users with the ability to take and upload images of their natural face.

[0777] "Conversational artificial intelligence" is an artificial intelligence technology that has the ability to analyze text data entered by the user and provide appropriate responses and advice in natural language.

[0778] "Image generation artificial intelligence" is an artificial intelligence technology that has the ability to generate new images based on image data and related information uploaded by users.

[0779] "Beauty products" is a general term for products, such as cosmetics and skin care products, used to assist or improve a user's appearance or beauty.

[0780] "Purchase Link" means a hyperlink that provides direct access to a web page for purchasing a particular product or service.

[0781] A "special offer" is a temporary promotional activity that offers a lower price or special benefits than the regular selling price.

[0782] "Discount information" is information that notifies users that the price of a product will be cheaper than usual for a certain period of time.

[0783] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0784] The mobile application is software that users install on their smartphones or tablets and provides the functionality to take and upload images of their faces, allowing users to provide images of their own faces to the system.

[0785] The server provides the following main functions: First, it receives information such as the user's name, email address, and password entered into the mobile device application and creates a user account in the database. Next, the interactive AI generates appropriate advice based on the user's makeup concerns and preferences and provides it to the user. Furthermore, it processes the bare-face image uploaded by the user and generates a face image with makeup using the image generation AI. Finally, based on the generated face image with makeup, it provides the user with information on the most suitable beauty products and brands, purchase links, special offers, and discount information.

[0786] Conversational AI analyzes text data entered by the user (e.g., "What are your skin concerns?", "What kind of makeup style are you aiming for?") and provides appropriate responses and advice in natural language. For example, if a user enters, "It's dry today, so I want to wear makeup that emphasizes moisturizing," the conversational AI will generate a response such as, "Moisturizing is important. Next, please upload an image of your bare face."

[0787] Image generation AI is a technology that generates a new make-up image of a face based on an image of the user's natural face and advice from an interactive AI. For example, if a user requests to "cover up redness," the image generation AI generates a face image with natural make-up that covers up the redness and sends it to the terminal via the server. The terminal then displays this make-up image to the user.

[0788] The server then provides the user with information on the most suitable beauty products and brands, along with a link to purchase them, based on the generated face image after makeup application. For example, specific recommendation information such as "This makeup style is best suited to a specific brand of foundation" is included. This allows the user to easily access the purchasing site.

[0789] In addition, the server collects special offers and discount information from cosmetics manufacturers and provides this information to users. The terminal supports users' purchasing behavior by displaying information such as "Currently, products from a specific brand are 10% off."

[0790] In this way, the system of the present invention provides an environment in which users can easily find a makeup style that suits them and select the most suitable products, thereby significantly improving their beauty experience.

[0791] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0792] Step 1:

[0793] The user installs and launches the mobile application. The application prompts the user to enter information such as their name, email address, and password. After the user enters the required information and taps the "Submit" button, the device sends this information to the server.

[0794] Input: User information such as name, email address, and password

[0795] Data processing / calculation: The server registers the received information in the database and creates a user account. Once registration is complete, an authentication token is generated.

[0796] Output: Sends the authentication token to the device.

[0797] Specific operation: The server adds a new record to the database based on the user information, generates an authentication token, and returns it. The terminal receives the authentication token and displays a success message to the user.

[0798] Step 2:

[0799] The user logs in using an authentication token. The device displays makeup-related questions to the user. For example, questions like, "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The user enters their concerns and preferences and taps "Send." The device then sends this information to the server.

[0800] Input: User's concerns and desires regarding makeup

[0801] Data processing / calculation: The server uses conversational AI to analyze the user's input, which generates appropriate advice.

[0802] Output: Sends generated advice to the terminal

[0803] Specific operation: The server passes the user's input text to the conversational AI and receives the analysis results. Advice is generated as text and sent back to the terminal. The terminal displays the received advice to the user.

[0804] Step 3:

[0805] The device displays a message to the user urging them to take a picture of their face and upload it. The user takes a picture of their face using the smartphone camera and taps the "Upload" button to send it to the device. The device then sends this image data to the server.

[0806] Input: Image file of a natural face

[0807] Data processing / calculation: The server sends the received image file to the image generation AI, which analyzes the image and extracts the user's facial features.

[0808] Output: Analyzed facial feature data

[0809] How it works: The server passes the received image to a dedicated image analysis module to detect facial feature points and generate corresponding data, which will be used in the next step.

[0810] Step 4:

[0811] The server uses image generation AI to generate a make-up image of the user's face based on an image of their natural face and advice provided by the interactive AI.

[0812] Input: natural face image, conversational AI advice

[0813] Data processing / calculation: Image generation AI analyzes face shape and skin tone and generates a face image after makeup based on advice.

[0814] Output: Face image after makeup

[0815] Specific operation: The server uses image generation AI to apply makeup effects to a bare-face image and generate a new image. The generated facial image is then sent from the server to the device.

[0816] Step 5:

[0817] Based on the generated post-makeup facial image, the server selects information on beauty products and brands that are most suitable for the user and generates a corresponding purchase link.

[0818] Input: Face image after makeup

[0819] Data processing / calculation: The server uses a dedicated recommendation system to select appropriate beauty products to match the face image after makeup application.

[0820] Output: Beauty product information and purchase links

[0821] Specific operation: The server analyzes the facial image and selects the most suitable beauty products based on the database. Information about the selected beauty products and a link to purchase them are generated and sent to the terminal, which then displays them to the user.

[0822] Step 6:

[0823] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, information such as "Currently, products from a specific brand are 10% off."

[0824] Enter: Special offers and discounts

[0825] Data processing / calculation: The server periodically collects discount information from affiliated cosmetic manufacturers and selects special offers tailored to the user.

[0826] Output: Special offers and discounts

[0827] Specific operation: The server analyzes discount information received from manufacturers, extracts relevant information based on the user profile, and sends it to the device, which then displays it to the user.

[0828] (Application example 1)

[0829] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0830] Conventional makeup support systems make it difficult for users to actually try on makeup products, making it difficult to find a makeup style that suits them before purchasing. Furthermore, because special offers and discount information are not provided, users may miss the optimal timing for purchasing. Furthermore, there is a lack of virtual makeup experience, making it impossible to visually confirm the realistic makeup look.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0832] In this invention, the server includes means for providing a mobile information terminal app that allows a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using a natural language processing AI to provide appropriate advice based on the input concerns and wishes, means for using an image generation AI to generate a made-up facial image based on the image uploaded by the user and the advice from the natural language processing AI, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on cosmetics and brands that are most suitable for the user, means for providing special offer and discount information, and means for using augmented reality technology to try on makeup in a virtual space. This allows the user to try on realistic makeup in a virtual space, making it easier to find the makeup style and cosmetics that are most suitable for them, and enabling them to receive special offer and discount information at the appropriate time.

[0833] A "mobile information terminal app" is application software that is installed on mobile information terminals such as smartphones and tablets.

[0834] "User" means a person who uses the services of this system.

[0835] "Natural language processing AI" is an artificial intelligence technology that deciphers, understands, and generates human language.

[0836] "Image generation AI" refers to artificial intelligence technology that uses algorithms to generate, transform, or modify images.

[0837] "Augmented reality technology" is a technology that overlays digital information onto real-world images and environments.

[0838] "Customized makeup advice" refers to advice on specific makeup methods and products provided based on a user's individual concerns and desires.

[0839] A "special offer" is a product or service offered at a more favorable price than is normally offered.

[0840] "Discount information" refers to information that allows you to purchase products or services at a price lower than the regular price.

[0841] A "virtual space" is a space that uses computer graphics and display technology to create a visual environment that differs from reality.

[0842] A "face image after makeup" is an image of the user's bare face after makeup effects have been applied.

[0843] "Information about cosmetics and brands" refers to information such as descriptions, features, and uses of specific cosmetics and their manufacturers.

[0844] A "purchase link" is a link to a web page where a user can purchase related products online.

[0845] The present invention is a support system that helps users find the makeup style that best suits them. This system is composed of a mobile information terminal app, a server, natural language processing AI, image generation AI, and augmented reality technology. The detailed configuration and operation procedure of the present invention are as follows.

[0846] Hardware and Software Configuration

[0847] Hardware used

[0848] Mobile devices: smartphones and tablets

[0849] Server: Server for main processing

[0850] Software used

[0851] Mobile device app: Developed with React Native

[0852] Server-side framework: Django

[0853] Natural Language Processing AI: Dialogflow

[0854] Image generation AI: TensorFlow or PyTorch

[0855] Augmented reality technology: ARKit or ARCore

[0856] System Operation

[0857] Step 1: Create an account and log in

[0858] The user enters the required information, such as name, email address, and password, into the device and sends it to the server. The server stores this information in a database, generates an authentication token, and returns it to the device. This logs the user in.

[0859] Step 2: Virtual Counseling

[0860] After logging in, the device asks the user a series of questions about makeup. The user's answers are sent from the device to the server, which uses natural language processing AI to analyze the answers, generate appropriate counseling results and advice, and return them to the device.

[0861] Step 3: Upload a photo of your bare face

[0862] Next, the device prompts the user to take a picture of their natural face and upload it. The user takes the picture and sends it from the device to the server. The server receives the image and sends it to the image generation AI.

[0863] Step 4: Generate a face image after makeup

[0864] Based on the bare face image sent to the server and the counseling content, the image generation AI generates a face image with makeup. For example, it generates a makeup image that covers redness based on the user's wishes. This generated face image with makeup is sent from the server to the device.

[0865] Step 5: Makeup product recommendations

[0866] Based on the generated makeup-applied facial image, the server generates information on the most suitable cosmetics and brands for the user, along with a link to purchase them. This information is sent to the terminal and displayed to the user.

[0867] Step 6: Provide special offers and discounts

[0868] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal, which displays this information to the user and assists them in their purchasing decisions.

[0869] Virtual makeup experience

[0870] Using the device, users can try on makeup in real time in a virtual space using augmented reality technology, allowing them to visually check which makeup looks best for them.

[0871] Prompt Sentence Examples

[0872] For example, if the user enters "I would like natural makeup that covers redness," the following prompt sentence is used to begin the consultation:

[0873] text

[0874] Input: I'd like to cover redness and have a natural looking makeup look. Can you give me some advice and product recommendations for this?

[0875] The system allows users to try on realistic makeup in a virtual world, helping them find the makeup style and cosmetics that best suit them, while also allowing them to receive special offers and discount information at the right time.

[0876] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0877] Step 1:

[0878] The terminal prompts the user to enter necessary information such as name, email address, and password. The entered information is sent to the server. The server stores this information in a database and creates a new account. The server also generates an authentication token and sends it back to the terminal. The terminal receives the authentication token and logs the user in. The input is the user's personal information, and the output is the authentication token.

[0879] Step 2:

[0880] The terminal displays a series of makeup-related questions to the logged-in user. The user answers the questions and inputs their wishes and concerns. The terminal sends this input to the server. The server uses natural language processing AI to analyze the answers and returns the generated counseling results and advice to the terminal. The input is the user's answers, and the output is the counseling results and advice.

[0881] Step 3:

[0882] The device then prompts the user to take an image of their natural face and upload it. The user takes an image using the camera on their mobile device and sends it to the server. The server receives this image and sends it to the image generation AI. The input is the image of the user's natural face, and the output is the server's receipt of the image.

[0883] Step 4:

[0884] The server uses image generation AI to generate a face image with makeup based on the received bare face image and the counseling content obtained in step 2. This image generation process simulates a specific makeup effect based on the image data and counseling content. The generated face image with makeup is sent from the server to the device. The input is the bare face image and counseling data, and the output is the face image with makeup.

[0885] Step 5:

[0886] The device displays the generated image of the user's face after makeup application to the user. The user can review this image and request changes. The server also generates information on the cosmetics and brands that are most suitable for the user based on the results of the image generation AI. It also generates a corresponding purchase link and sends it to the device. The input is the image of the face after makeup application, and the output is information on the cosmetics and brands and a purchase link.

[0887] Step 6:

[0888] The server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays this information to the user, allowing the user to avoid missing the optimal purchasing timing. The input is the special offers and discount information, and the output is the display of that information.

[0889] Step 7:

[0890] Using a device, users can try on makeup in real time in a virtual space using augmented reality technology. The device uses a camera to recognize the user's face and overlay makeup effects in real time, allowing users to visually check which makeup looks best on them. The input is a real-time image of the user's face, and the output is the makeup look in the virtual space.

[0891] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0892] The present invention relates to a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. The present invention is implemented by combining a smartphone app, a server, multiple artificial intelligence technologies, and an emotion engine.

[0893] Program processing and specific examples

[0894] Account creation and login

[0895] First, the user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[0896] Makeup counseling

[0897] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and preferences, which the device then sends to the server. The server receives this information, analyzes the counseling content using conversational AI, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0898] emotion recognition

[0899] When a user enters counseling details or uploads images, the emotion engine analyzes the user's facial expressions and text data to recognize the user's emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion engine will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling details.

[0900] Upload a photo of your bare face

[0901] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0902] Generating face images after makeup

[0903] Based on the bare face image and counseling details sent to the server, as well as the user's emotional information recognized by the emotion engine, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. For example, if the user wishes to "cover up redness" and the emotion engine strongly senses this desire, the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup is sent to the device via the server, and the device displays it to the user.

[0904] Makeup product recommendations

[0905] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[0906] Providing you with special offers and discounts

[0907] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[0908] Specific examples

[0909] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0910] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0911] The processing flow will be explained below.

[0912] Step 1:

[0913] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[0914] Terminal: Sends the entered information to the server.

[0915] Step 2:

[0916] Server: Stores the received user information in a database and generates an authentication token.

[0917] Server: Returns the authentication token to the device.

[0918] Terminal: Receives the authentication token and displays the login status to the user.

[0919] Step 3:

[0920] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[0921] User: Enter their concerns and wishes in response to the questions.

[0922] Terminal: Sends the entered information to the server.

[0923] Step 4:

[0924] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[0925] Server: Sends advice to the device.

[0926] Terminal: Display advice to the user.

[0927] Step 5:

[0928] Terminal: Displays a message instructing the user to take a picture of their input and their real face.

[0929] User: Take a picture of their face using their smartphone camera and upload it to the app.

[0930] Device: Sends the captured image to the server.

[0931] Step 6:

[0932] Server: Processes the received face images and counseling content.

[0933] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[0934] Server: Sends the generated face image after makeup to the device.

[0935] Terminal: Displays the face image after makeup to the user.

[0936] Step 7:

[0937] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[0938] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[0939] Device: Display recommendation information and purchase links to the user.

[0940] Step 8:

[0941] Server: Collects special offers and discounts from cosmetic manufacturers.

[0942] Server: Sends special offers and discounts to your device.

[0943] On the device: Show users special offers and discounts.

[0944] Step 9:

[0945] Device: When the user enters counseling content or uploads images, the emotion engine is activated and analyzes the user's facial expressions and text data.

[0946] Emotion engine (server): Recognizes the user's emotions and sends the analysis results to the server.

[0947] Server: Based on the analysis results from the emotion engine, the results are reflected in the analysis of conversational AI and image generation AI.

[0948] Specific examples

[0949] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[0950] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[0951] Example 2

[0952] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] Conventional makeup systems have difficulty fully reflecting the user's individual concerns and desires, and are unable to provide optimal suggestions that take the user's emotions into account. As a result, it has been difficult to improve user satisfaction. In addition, the information provided to users is general, limiting the ability to provide a personalized experience.

[0954] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for providing a mobile terminal application that allows a user to take and upload an image of their bare face; means for allowing the user to input their makeup concerns and wishes; means for using an interactive AI that provides appropriate advice based on the input concerns and wishes; means for using an image generation AI that generates a made-up face image based on the image uploaded by the user, the advice from the interactive AI, and the user's emotional information; means for displaying the generated made-up face image to the user; means for providing information on optimal cosmetics and brands and a purchase link for the user; means for using an emotion recognition engine that recognizes the user's emotions and suggests optimal makeup; and means for providing special offer and discount information. This enables optimal makeup suggestions that reflect the user's individual concerns, wishes, and emotions, thereby providing a personalized and highly satisfying experience.

[0955] A "mobile device application" is software that a user operates using a mobile device, and is a means for inputting information about makeup, taking and uploading images, and displaying the results.

[0956] "Conversational AI" is an AI system that analyzes input from users and generates appropriate advice and answers using natural language processing technology.

[0957] "Image generation artificial intelligence" is an artificial intelligence system that generates a face image with makeup applied based on a bare face image uploaded by the user, advice provided by an interactive artificial intelligence, and the user's emotional information.

[0958] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and text data to recognize their emotions and then makes appropriate makeup suggestions based on that information.

[0959] A "server" is a computer system that receives data sent by a user, executes various artificial intelligence processes, and returns the results to the user's mobile device.

[0960] The "means for providing information on cosmetics and brands and purchase links" is a function for presenting information on the cosmetics and brands that are most suitable for the user, as well as purchase links, based on the generated face image after makeup.

[0961] The "means for providing special offers and discount information" is a function for providing users with special offers and discount information obtained from various cosmetic manufacturers, thereby promoting purchasing behavior.

[0962] The present invention relates to a system that generates a personalized makeup-applied facial image based on a user's natural facial image and provides information on the most suitable cosmetics and brands. This system is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[0963] A user first installs the mobile device application, and when they launch the app for the first time, an account creation screen appears. When the user enters their name, email address, and password and presses the submit button, the device sends this data to the server. The server creates a new user account in the database based on the received data, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0964] Next, the user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0965] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are asked. The user enters their answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0966] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. This engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0967] Next, the device displays a message prompting the user to take and upload an image of their bare face. The user takes an image of their bare face using the smartphone camera and uploads it to the server via the device. The server receives this image and sends it to the image generation AI. The image generation AI generates a face image with makeup applied based on the received bare face image, the advice provided by the conversational AI, and the emotional information. For example, if the user wishes to "cover up redness," and the emotion engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover up the user's redness. This generated image is sent back to the device via the server, and the device displays it to the user.

[0968] Furthermore, the server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the information includes recommendation information such as "A specific brand of foundation is suitable for this makeup style" and a link to the purchasing site. The server sends this information to the terminal, which displays it to the user.

[0969] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[0970] Here are some examples of prompts for generative AI models:

[0971] "Based on an image of the user's bare face, please generate an optimal makeup-applied image, taking into account the content of this consultation and the user's emotions. Also, please provide recommendations for makeup products that will be suitable."

[0972] As a result, the system of the present invention can provide optimal makeup suggestions and product recommendations that reflect the user's individual concerns, desires, and emotions, thereby achieving a high level of satisfaction.

[0973] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0974] Step 1:

[0975] The user installs the mobile device application and launches the account creation screen. The user provides input data such as name, email address, and password. The device sends this input data to the server. The server uses the received data to create a new user account in the database, generates an authentication token, and returns it to the device. This completes the user's account creation.

[0976] Input: Name, Email Address, Password

[0977] Output: New user account created, authentication token

[0978] Step 2:

[0979] The user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[0980] Input: Email address, password

[0981] Output: Authentication token, login status

[0982] Step 3:

[0983] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are displayed. The user enters the answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[0984] Input: Answer to makeup question

[0985] Output: Analysis results and advice by interactive AI

[0986] Step 4:

[0987] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. The emotion recognition engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[0988] Input: facial expression data, text data

[0989] Output: Recognized emotion information

[0990] Step 5:

[0991] The device then displays a message prompting the user to take and upload a picture of their face without makeup. The user then takes a picture of their face with their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[0992] Input: natural face image

[0993] Output: The image of the face sent to the server

[0994] Step 6:

[0995] The server uses the image generation AI to generate a face image with makeup based on the received bare face image, advice provided by the conversational AI, and emotional information. For example, if the user wishes to "cover redness," and the emotion recognition engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover the user's redness. This generated image is sent back to the terminal via the server, and the terminal displays it to the user.

[0996] Input: Face image, advice, emotional information

[0997] Output: Face image after makeup

[0998] Step 7:

[0999] The server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the server may recommend a specific brand of foundation that is suitable for this makeup style, along with a link to a purchasing site. The server sends this information to the device, which then displays it to the user.

[1000] Input: Face image after makeup

[1001] Output: Cosmetics and brand information, purchase link

[1002] Step 8:

[1003] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[1004] Enter: Special offers and discounts

[1005] Output: Display of special offers and discounts

[1006] (Application example 2)

[1007] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1008] Conventional makeup experience systems have the drawback of requiring users to purchase numerous actual products to try out different makeup looks, or to seek expert advice, which is time-consuming and expensive. Furthermore, the lack of emotion recognition and real-time makeup simulation makes it difficult to provide optimal advice tailored to the user's individual emotions and preferences. A new system that solves these problems is needed.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1010] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their concerns and desires regarding makeup, and means for using interactive artificial intelligence to provide appropriate advice based on the input concerns and desires.

[1011] The system includes a means for using an image generation AI to generate a made-up facial image based on an image uploaded by a user and advice from an interactive AI, a means for displaying the generated made-up facial image to the user, a means for providing information on cosmetics and brands that are most suitable for the user and a link to purchase them, a means for recognizing emotions by analyzing the user's facial expressions and text data, and a means for suggesting a makeup style based on the emotional information. This system enables users to experience makeup styles in real time that correspond to their concerns, wishes, and emotions.

[1012] The "mobile terminal application" is an application that allows a user to take an image of their bare face and input their concerns and desires regarding makeup.

[1013] "Conversational AI" is an AI that provides appropriate advice based on the user's concerns and desires regarding makeup.

[1014] "Image generation artificial intelligence" is an artificial intelligence that generates a face image after makeup based on an image uploaded by the user and advice from an interactive artificial intelligence.

[1015] "Emotion recognition" is a technology that recognizes a user's emotions by analyzing their facial expressions and text data.

[1016] The "means for proposing a makeup style" is a means for proposing an optimal makeup style based on emotional information and the user's worries and desires.

[1017] The "means for displaying the generated face image after makeup to the user" refers to a means for displaying the face image after makeup generated by the image generation artificial intelligence to the user.

[1018] The "means for providing information on the most suitable cosmetics and brands to the user and a purchase link" is a means for providing information on the most suitable cosmetics and brands to the user and a corresponding purchase link based on the generated face image after makeup application.

[1019] The present invention is a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. This system is implemented by combining mobile terminal applications, conversational artificial intelligence, image generation artificial intelligence, and emotion recognition technology.

[1020] The user installs the application for the first time and creates an account. The application sends necessary information such as name, email address, and password to the server. The server creates a user account in the database based on the received information, generates an authentication token, and returns it to the device. This logs the user in.

[1021] After logging in, the server presents the user with a series of makeup-related questions, prompting them to input their concerns and preferences. For example, questions such as "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The input information is sent to the server, where it is analyzed by an interactive AI. Advice generated as a result of the analysis is returned to the device and displayed to the user.

[1022] When a user uploads an image of their bare face, emotion recognition technology analyzes the user's facial expressions and text data to recognize their emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion recognition technology will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling content.

[1023] Next, the user takes a picture of their bare face using their smartphone camera and uploads it. This image is sent to a server where it is processed by the image generation AI. Based on the bare face image sent to the server, the consultation details, and the user's emotional information recognized by emotion recognition technology, the image generation AI analyzes the face shape and skin tone to generate an image of the face with makeup applied.

[1024] The generated image of the face with makeup applied is sent to the terminal via the server and displayed to the user. A specific example is when a user inputs a concern such as "I'm bothered by the redness of my face" and uploads an image of their bare face. Emotion recognition technology recognizes the emotion of "confusion," and the conversational AI and image generation AI perform analysis taking this emotion into account. The generated image of the face with makeup is displayed in a style that effectively covers the redness.

[1025] Based on the generated image, the server also provides users with information on the best cosmetics and brands. For example, specific information such as "This makeup style is best suited to a specific brand of foundation" is included. This information also includes a purchase link, allowing users to easily access the purchasing site. In addition, if there are any special offers or discount information, these will also be provided.

[1026] The hardware used includes smartphones and head-mounted displays (HMDs), and the software used includes mobile terminal applications, server-side conversational artificial intelligence (e.g., OpenAI GPT-4), emotion recognition technology (e.g., Emotion API), and image generation artificial intelligence (e.g., GANs).

[1027] An example prompt might look like this:

[1028] User input: "I'm concerned about redness on my face, so I'd like makeup that makes my skin look even."

[1029] Prompt: "User desires: Concerned about redness on face, Emotion: Confused. Please suggest an appropriate makeup style."

[1030] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1031] Step 1:

[1032] A user installs a smartphone app and creates an account when they launch it for the first time. The user enters their name, email address, and password. The device sends this information to the server, and the server uses the received information to create a user account in a database, generates an authentication token, and returns it to the device. The input is user information, and the output is an authentication token. This operation puts the user in a logged-in state.

[1033] Step 2:

[1034] After the user logs in, the device displays a series of makeup-related questions to the user. The user inputs their concerns and desires in response to questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The input is the user's concerns and desires, and the output is text data. This data is sent from the device to the server.

[1035] Step 3:

[1036] The server passes the received text data to the conversational AI, which generates appropriate advice. The input is the text data of the user's concerns and wishes, and the output is the generated advice. The conversational AI analyzes this and generates advice customized for each user. The generated advice is sent from the server to the terminal, which displays it to the user.

[1037] Step 4:

[1038] When a user inputs counseling content, emotion recognition technology analyzes the user's facial expressions and text data to recognize the user's emotions. The input is the user's facial expression image and text data, and the output is recognized emotional information. For example, if a user inputs a concern that "my usual makeup doesn't work," the server will determine that the user is feeling dissatisfied or confused. This emotional information is also sent to the server and used to analyze the counseling content.

[1039] Step 5:

[1040] A user takes a picture of their face using a smartphone camera and uploads it to the server via their device. The input is the face image, and the output is the image data sent to the server. The server then sends the received face image to the image generation AI.

[1041] Step 6:

[1042] Based on the bare face image and counseling details sent to the server, as well as the emotional information recognized by emotion recognition technology, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. The input is the bare face image, the user's concerns and wishes, and emotional information, and the output is the face image with makeup applied.

[1043] Step 7:

[1044] The generated face image after makeup is sent from the server to the terminal and displayed to the user. The input is the generated face image, and the output is the face image displayed on the terminal. As a specific example, if a user inputs "I'm troubled by the redness of my face" and the emotion recognition technology recognizes "confused," the image generation technology will suggest a makeup style that covers the redness and display that image to the user.

[1045] Step 8:

[1046] The server provides the user with information on the most suitable cosmetics and brands based on the generated face image after makeup. The input is the face image after makeup, and the output is recommendation information. For example, it includes specific information such as "A specific brand of foundation is suitable for this makeup style." This information also includes a purchase link, allowing the user to easily access the purchasing site.

[1047] Step 9:

[1048] If there is special offer or discount information, the server collects it and sends it to the terminal. The input is the special offer or discount information, and the output is the displayed special offer or discount information. The user can use this information to purchase cosmetics at a discount.

[1049] This series of processing steps allows the user to experience the most suitable makeup style in real time according to their desires and emotions.

[1050] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1051] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1052] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1053] [Fourth embodiment]

[1054] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1055] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1056] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1057] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1058] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1059] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1060] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1061] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1062] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1063] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1064] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1065] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1066] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1067] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a smartphone app, a server, and multiple artificial intelligence technologies.

[1068] Program processing and specific examples

[1069] Account creation and login

[1070] First, a user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[1071] Makeup counseling

[1072] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and wishes, and the device sends this information to the server. The server receives this information, analyzes the counseling content using conversational AI (e.g., artificial intelligence that processes natural language), generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[1073] Upload a photo of your bare face

[1074] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[1075] Generating face images after makeup

[1076] Based on the bare face image sent to the server and the counseling details, the image generation AI analyzes the user's face shape and skin tone and generates a face image with makeup applied. For example, if the user requests to "cover up redness," the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup applied is sent via the server to the device, which then displays it to the user.

[1077] Makeup product recommendations

[1078] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[1079] Providing you with special offers and discounts

[1080] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[1081] The system of the present invention provides an environment where users can easily find the makeup style that suits them and select the most suitable products, thereby significantly improving the user's beauty experience.

[1082] The processing flow will be explained below.

[1083] Step 1:

[1084] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[1085] Terminal: Sends the entered information to the server.

[1086] Step 2:

[1087] Server: Stores the received user information in a database and generates an authentication token.

[1088] Server: Returns the authentication token to the device.

[1089] Terminal: Receives the authentication token and displays the login status to the user.

[1090] Step 3:

[1091] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[1092] User: Enter their concerns and wishes in response to the questions.

[1093] Terminal: Sends the entered information to the server.

[1094] Step 4:

[1095] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[1096] Server: Sends advice to the device.

[1097] Terminal: Display advice to the user.

[1098] Step 5:

[1099] On the device: Prompt the user to take and upload a photo of their face.

[1100] User: Take a picture of their face using their smartphone camera and upload it to the app.

[1101] Device: Sends the captured image to the server.

[1102] Step 6:

[1103] Server: Processes the received face images and counseling content.

[1104] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[1105] Server: Sends the generated face image after makeup to the device.

[1106] Terminal: Displays the face image after makeup to the user.

[1107] Step 7:

[1108] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[1109] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[1110] Device: Display recommendation information and purchase links to the user.

[1111] Step 8:

[1112] Server: Collects special offers and discounts from cosmetic manufacturers.

[1113] Server: Sends special offers and discounts to your device.

[1114] On the device: Show users special offers and discounts.

[1115] Example 1

[1116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1117] Conventional makeup advisor systems have difficulty providing appropriate advice based on users' concerns and desires, and often take time to select the most suitable makeup style and cosmetics for the user. Furthermore, it is difficult to realistically reproduce facial images after makeup application, which can result in low user satisfaction. Furthermore, the provision of special offers and discount information is not centralized, making it difficult for users to receive this information at the appropriate time.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1119] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using an interactive artificial intelligence that provides appropriate advice based on the input concerns and wishes, means for using an image generation artificial intelligence that generates a made-up facial image based on the image uploaded by the user and the advice from the interactive artificial intelligence, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on beauty products and brands that are most suitable for the user, and means for collecting and providing special offer and discount information. This allows users to easily select makeup styles and cosmetics that suit them, check realistic made-up facial images, and also receive special offer and discount information in a timely manner.

[1120] A "mobile device application" is software that is installed on a mobile phone or tablet device and provides users with the ability to take and upload images of their natural face.

[1121] "Conversational artificial intelligence" is an artificial intelligence technology that has the ability to analyze text data entered by the user and provide appropriate responses and advice in natural language.

[1122] "Image generation artificial intelligence" is an artificial intelligence technology that has the ability to generate new images based on image data and related information uploaded by users.

[1123] "Beauty products" is a general term for products, such as cosmetics and skin care products, used to assist or improve a user's appearance or beauty.

[1124] "Purchase Link" means a hyperlink that provides direct access to a web page for purchasing a particular product or service.

[1125] A "special offer" is a temporary promotional activity that offers a lower price or special benefits than the regular selling price.

[1126] "Discount information" is information that notifies users that the price of a product will be cheaper than usual for a certain period of time.

[1127] The present invention relates to a system that allows users to upload a bare-face image and receive makeup consultations, then generates a realistic makeup-applied image and provides personalized makeup information. The present invention is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[1128] The mobile application is software that users install on their smartphones or tablets and provides the functionality to take and upload images of their faces, allowing users to provide images of their own faces to the system.

[1129] The server provides the following main functions: First, it receives information such as the user's name, email address, and password entered into the mobile device application and creates a user account in the database. Next, the interactive AI generates appropriate advice based on the user's makeup concerns and preferences and provides it to the user. Furthermore, it processes the bare-face image uploaded by the user and generates a face image with makeup using the image generation AI. Finally, based on the generated face image with makeup, it provides the user with information on the most suitable beauty products and brands, purchase links, special offers, and discount information.

[1130] Conversational AI analyzes text data entered by the user (e.g., "What are your skin concerns?", "What kind of makeup style are you aiming for?") and provides appropriate responses and advice in natural language. For example, if a user enters, "It's dry today, so I want to wear makeup that emphasizes moisturizing," the conversational AI will generate a response such as, "Moisturizing is important. Next, please upload an image of your bare face."

[1131] Image generation AI is a technology that generates a new make-up image of a face based on an image of the user's natural face and advice from an interactive AI. For example, if a user requests to "cover up redness," the image generation AI generates a face image with natural make-up that covers up the redness and sends it to the terminal via the server. The terminal then displays this make-up image to the user.

[1132] The server then provides the user with information on the most suitable beauty products and brands, along with a link to purchase them, based on the generated face image after makeup application. For example, specific recommendation information such as "This makeup style is best suited to a specific brand of foundation" is included. This allows the user to easily access the purchasing site.

[1133] In addition, the server collects special offers and discount information from cosmetics manufacturers and provides this information to users. The terminal supports users' purchasing behavior by displaying information such as "Currently, products from a specific brand are 10% off."

[1134] In this way, the system of the present invention provides an environment in which users can easily find a makeup style that suits them and select the most suitable products, thereby significantly improving their beauty experience.

[1135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1136] Step 1:

[1137] The user installs and launches the mobile application. The application prompts the user to enter information such as their name, email address, and password. After the user enters the required information and taps the "Submit" button, the device sends this information to the server.

[1138] Input: User information such as name, email address, and password

[1139] Data processing / calculation: The server registers the received information in the database and creates a user account. Once registration is complete, an authentication token is generated.

[1140] Output: Sends the authentication token to the device.

[1141] Specific operation: The server adds a new record to the database based on the user information, generates an authentication token, and returns it. The terminal receives the authentication token and displays a success message to the user.

[1142] Step 2:

[1143] The user logs in using an authentication token. The device displays makeup-related questions to the user. For example, questions like, "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The user enters their concerns and preferences and taps "Send." The device then sends this information to the server.

[1144] Input: User's concerns and desires regarding makeup

[1145] Data processing / calculation: The server uses conversational AI to analyze the user's input, which generates appropriate advice.

[1146] Output: Sends generated advice to the terminal

[1147] Specific operation: The server passes the user's input text to the conversational AI and receives the analysis results. Advice is generated as text and sent back to the terminal. The terminal displays the received advice to the user.

[1148] Step 3:

[1149] The device displays a message to the user urging them to take a picture of their face and upload it. The user takes a picture of their face using the smartphone camera and taps the "Upload" button to send it to the device. The device then sends this image data to the server.

[1150] Input: Image file of a natural face

[1151] Data processing / calculation: The server sends the received image file to the image generation AI, which analyzes the image and extracts the user's facial features.

[1152] Output: Analyzed facial feature data

[1153] How it works: The server passes the received image to a dedicated image analysis module to detect facial feature points and generate corresponding data, which will be used in the next step.

[1154] Step 4:

[1155] The server uses image generation AI to generate a make-up image of the user's face based on an image of their natural face and advice provided by the interactive AI.

[1156] Input: natural face image, conversational AI advice

[1157] Data processing / calculation: Image generation AI analyzes face shape and skin tone and generates a face image after makeup based on advice.

[1158] Output: Face image after makeup

[1159] Specific operation: The server uses image generation AI to apply makeup effects to a bare-face image and generate a new image. The generated facial image is then sent from the server to the device.

[1160] Step 5:

[1161] Based on the generated post-makeup facial image, the server selects information on beauty products and brands that are most suitable for the user and generates a corresponding purchase link.

[1162] Input: Face image after makeup

[1163] Data processing / calculation: The server uses a dedicated recommendation system to select appropriate beauty products to match the face image after makeup application.

[1164] Output: Beauty product information and purchase links

[1165] Specific operation: The server analyzes the facial image and selects the most suitable beauty products based on the database. Information about the selected beauty products and a link to purchase them are generated and sent to the terminal, which then displays them to the user.

[1166] Step 6:

[1167] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, information such as "Currently, products from a specific brand are 10% off."

[1168] Enter: Special offers and discounts

[1169] Data processing / calculation: The server periodically collects discount information from affiliated cosmetic manufacturers and selects special offers tailored to the user.

[1170] Output: Special offers and discounts

[1171] Specific operation: The server analyzes discount information received from manufacturers, extracts relevant information based on the user profile, and sends it to the device, which then displays it to the user.

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1174] Conventional makeup support systems make it difficult for users to actually try on makeup products, making it difficult to find a makeup style that suits them before purchasing. Furthermore, because special offers and discount information are not provided, users may miss the optimal timing for purchasing. Furthermore, there is a lack of virtual makeup experience, making it impossible to visually confirm the realistic makeup look.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes means for providing a mobile information terminal app that allows a user to take and upload an image of their bare face, means for allowing the user to input their makeup concerns and wishes, means for using a natural language processing AI to provide appropriate advice based on the input concerns and wishes, means for using an image generation AI to generate a made-up facial image based on the image uploaded by the user and the advice from the natural language processing AI, means for displaying the generated made-up facial image to the user, means for providing information and purchase links on cosmetics and brands that are most suitable for the user, means for providing special offer and discount information, and means for using augmented reality technology to try on makeup in a virtual space. This allows the user to try on realistic makeup in a virtual space, making it easier to find the makeup style and cosmetics that are most suitable for them, and enabling them to receive special offer and discount information at the appropriate time.

[1177] A "mobile information terminal app" is application software that is installed on mobile information terminals such as smartphones and tablets.

[1178] "User" means a person who uses the services of this system.

[1179] "Natural language processing AI" is an artificial intelligence technology that deciphers, understands, and generates human language.

[1180] "Image generation AI" refers to artificial intelligence technology that uses algorithms to generate, transform, or modify images.

[1181] "Augmented reality technology" is a technology that overlays digital information onto real-world images and environments.

[1182] "Customized makeup advice" refers to advice on specific makeup methods and products provided based on a user's individual concerns and desires.

[1183] A "special offer" is a product or service offered at a more favorable price than is normally offered.

[1184] "Discount information" refers to information that allows you to purchase products or services at a price lower than the regular price.

[1185] A "virtual space" is a space that uses computer graphics and display technology to create a visual environment that differs from reality.

[1186] A "face image after makeup" is an image of the user's bare face after makeup effects have been applied.

[1187] "Information about cosmetics and brands" refers to information such as descriptions, features, and uses of specific cosmetics and their manufacturers.

[1188] A "purchase link" is a link to a web page where a user can purchase related products online.

[1189] The present invention is a support system that helps users find the makeup style that best suits them. This system is composed of a mobile information terminal app, a server, natural language processing AI, image generation AI, and augmented reality technology. The detailed configuration and operation procedure of the present invention are as follows.

[1190] Hardware and Software Configuration

[1191] Hardware used

[1192] Mobile devices: smartphones and tablets

[1193] Server: Server for main processing

[1194] Software used

[1195] Mobile device app: Developed with React Native

[1196] Server-side framework: Django

[1197] Natural Language Processing AI: Dialogflow

[1198] Image generation AI: TensorFlow or PyTorch

[1199] Augmented reality technology: ARKit or ARCore

[1200] System Operation

[1201] Step 1: Create an account and log in

[1202] The user enters the required information, such as name, email address, and password, into the device and sends it to the server. The server stores this information in a database, generates an authentication token, and returns it to the device. This logs the user in.

[1203] Step 2: Virtual Counseling

[1204] After logging in, the device asks the user a series of questions about makeup. The user's answers are sent from the device to the server, which uses natural language processing AI to analyze the answers, generate appropriate counseling results and advice, and return them to the device.

[1205] Step 3: Upload a photo of your bare face

[1206] Next, the device prompts the user to take a picture of their natural face and upload it. The user takes the picture and sends it from the device to the server. The server receives the image and sends it to the image generation AI.

[1207] Step 4: Generate a face image after makeup

[1208] Based on the bare face image sent to the server and the counseling content, the image generation AI generates a face image with makeup. For example, it generates a makeup image that covers redness based on the user's wishes. This generated face image with makeup is sent from the server to the device.

[1209] Step 5: Makeup product recommendations

[1210] Based on the generated makeup-applied facial image, the server generates information on the most suitable cosmetics and brands for the user, along with a link to purchase them. This information is sent to the terminal and displayed to the user.

[1211] Step 6: Provide special offers and discounts

[1212] The server collects special offers and discount information from cosmetics manufacturers and sends it to the terminal, which displays this information to the user and assists them in their purchasing decisions.

[1213] Virtual makeup experience

[1214] Using the device, users can try on makeup in real time in a virtual space using augmented reality technology, allowing them to visually check which makeup looks best for them.

[1215] Prompt Sentence Examples

[1216] For example, if the user enters "I would like natural makeup that covers redness," the following prompt sentence is used to begin the consultation:

[1217] text

[1218] Input: I'd like to cover redness and have a natural looking makeup look. Can you give me some advice and product recommendations for this?

[1219] The system allows users to try on realistic makeup in a virtual world, helping them find the makeup style and cosmetics that best suit them, while also allowing them to receive special offers and discount information at the right time.

[1220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1221] Step 1:

[1222] The terminal prompts the user to enter necessary information such as name, email address, and password. The entered information is sent to the server. The server stores this information in a database and creates a new account. The server also generates an authentication token and sends it back to the terminal. The terminal receives the authentication token and logs the user in. The input is the user's personal information, and the output is the authentication token.

[1223] Step 2:

[1224] The terminal displays a series of makeup-related questions to the logged-in user. The user answers the questions and inputs their wishes and concerns. The terminal sends this input to the server. The server uses natural language processing AI to analyze the answers and returns the generated counseling results and advice to the terminal. The input is the user's answers, and the output is the counseling results and advice.

[1225] Step 3:

[1226] The device then prompts the user to take an image of their natural face and upload it. The user takes an image using the camera on their mobile device and sends it to the server. The server receives this image and sends it to the image generation AI. The input is the image of the user's natural face, and the output is the server's receipt of the image.

[1227] Step 4:

[1228] The server uses image generation AI to generate a face image with makeup based on the received bare face image and the counseling content obtained in step 2. This image generation process simulates a specific makeup effect based on the image data and counseling content. The generated face image with makeup is sent from the server to the device. The input is the bare face image and counseling data, and the output is the face image with makeup.

[1229] Step 5:

[1230] The device displays the generated image of the user's face after makeup application to the user. The user can review this image and request changes. The server also generates information on the cosmetics and brands that are most suitable for the user based on the results of the image generation AI. It also generates a corresponding purchase link and sends it to the device. The input is the image of the face after makeup application, and the output is information on the cosmetics and brands and a purchase link.

[1231] Step 6:

[1232] The server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays this information to the user, allowing the user to avoid missing the optimal purchasing timing. The input is the special offers and discount information, and the output is the display of that information.

[1233] Step 7:

[1234] Using a device, users can try on makeup in real time in a virtual space using augmented reality technology. The device uses a camera to recognize the user's face and overlay makeup effects in real time, allowing users to visually check which makeup looks best on them. The input is a real-time image of the user's face, and the output is the makeup look in the virtual space.

[1235] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1236] The present invention relates to a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. The present invention is implemented by combining a smartphone app, a server, multiple artificial intelligence technologies, and an emotion engine.

[1237] Program processing and specific examples

[1238] Account creation and login

[1239] First, the user installs the smartphone app and creates an account when they launch it for the first time. When the user enters the required information, such as their name, email address, and password, the device sends this information to the server. The server uses the received information to create a user account in the database, generates an authentication token, and returns it to the device. This logs the user in.

[1240] Makeup counseling

[1241] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The user then inputs their concerns and preferences, which the device then sends to the server. The server receives this information, analyzes the counseling content using conversational AI, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[1242] emotion recognition

[1243] When a user enters counseling details or uploads images, the emotion engine analyzes the user's facial expressions and text data to recognize the user's emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion engine will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling details.

[1244] Upload a photo of your bare face

[1245] The device then displays a message prompting the user to take a picture of their natural face and upload it. The user then takes a picture of their natural face using their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[1246] Generating face images after makeup

[1247] Based on the bare face image and counseling details sent to the server, as well as the user's emotional information recognized by the emotion engine, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. For example, if the user wishes to "cover up redness" and the emotion engine strongly senses this desire, the image generation AI will generate an image with makeup applied based on the user's preferences while covering up the redness. This generated face image with makeup is sent to the device via the server, and the device displays it to the user.

[1248] Makeup product recommendations

[1249] Based on the generated face image after makeup, the server selects information about the most suitable makeup products and brands for the user. For example, specific recommendation information such as "XYZ brand foundation is suitable for this makeup style" is included. This information also includes a corresponding purchase link, allowing the user to easily access the purchasing site. The server sends this information to the device, which then displays it to the user.

[1250] Providing you with special offers and discounts

[1251] In addition, the server collects special offers and discount information from cosmetic manufacturers and sends it to the terminal. The terminal displays the special offers and discount information to the user to assist in purchasing. For example, "XYZ brand products are currently 10% off."

[1252] Specific examples

[1253] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[1254] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[1255] The processing flow will be explained below.

[1256] Step 1:

[1257] User: After installing the smartphone app and launching it for the first time, the user creates an account by entering their name, email address, and password.

[1258] Terminal: Sends the entered information to the server.

[1259] Step 2:

[1260] Server: Stores the received user information in a database and generates an authentication token.

[1261] Server: Returns the authentication token to the device.

[1262] Terminal: Receives the authentication token and displays the login status to the user.

[1263] Step 3:

[1264] Device: Prompts the user with a series of makeup-related questions, such as "What are your skin concerns?" and "What makeup style are you aiming for?"

[1265] User: Enter their concerns and wishes in response to the questions.

[1266] Terminal: Sends the entered information to the server.

[1267] Step 4:

[1268] Server: Using conversational AI such as ChatGPT, analyzes the counseling content entered by the user and generates appropriate advice.

[1269] Server: Sends advice to the device.

[1270] Terminal: Display advice to the user.

[1271] Step 5:

[1272] Terminal: Displays a message instructing the user to take a picture of their input and their real face.

[1273] User: Take a picture of their face using their smartphone camera and upload it to the app.

[1274] Device: Sends the captured image to the server.

[1275] Step 6:

[1276] Server: Processes the received face images and counseling content.

[1277] Server: Using image generation AI, it generates a make-up image of the user's face based on their face shape, skin tone, and consultation details.

[1278] Server: Sends the generated face image after makeup to the device.

[1279] Terminal: Displays the face image after makeup to the user.

[1280] Step 7:

[1281] Server: Based on the facial image after makeup application and the contents of the consultation, the server selects the most suitable cosmetics and brand information for the user.

[1282] Server: Sends information about recommended makeup products and brands, as well as a link to purchase, to the device.

[1283] Device: Display recommendation information and purchase links to the user.

[1284] Step 8:

[1285] Server: Collects special offers and discounts from cosmetic manufacturers.

[1286] Server: Sends special offers and discounts to your device.

[1287] On the device: Show users special offers and discounts.

[1288] Step 9:

[1289] Device: When the user enters counseling content or uploads images, the emotion engine is activated and analyzes the user's facial expressions and text data.

[1290] Emotion engine (server): Recognizes the user's emotions and sends the analysis results to the server.

[1291] Server: Based on the analysis results from the emotion engine, the results are reflected in the analysis of conversational AI and image generation AI.

[1292] Specific examples

[1293] For example, consider a case where a user inputs a concern such as "I'm troubled by the redness of my face" and uploads an image of their bare face. The emotion engine recognizes the emotion "confused" from the user's input, and the conversational AI and image generation AI analyze the image taking this emotion into account. Based on the advice from the emotion engine and conversational AI, the server suggests a makeup style that effectively covers the redness and generates an image based on that recommendation. At the same time, it recommends brands of concealer and foundation to cover the redness, and also provides special offers if any are available.

[1294] In this way, the system of the present invention can significantly improve the user's beauty experience by recognizing the user's emotions and suggesting optimal makeup looks.

[1295] Example 2

[1296] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1297] Conventional makeup systems have difficulty fully reflecting the user's individual concerns and desires, and are unable to provide optimal suggestions that take the user's emotions into account. As a result, it has been difficult to improve user satisfaction. In addition, the information provided to users is general, limiting the ability to provide a personalized experience.

[1298] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for providing a mobile terminal application that allows a user to take and upload an image of their bare face; means for allowing the user to input their makeup concerns and wishes; means for using an interactive AI that provides appropriate advice based on the input concerns and wishes; means for using an image generation AI that generates a made-up face image based on the image uploaded by the user, the advice from the interactive AI, and the user's emotional information; means for displaying the generated made-up face image to the user; means for providing information on optimal cosmetics and brands and a purchase link for the user; means for using an emotion recognition engine that recognizes the user's emotions and suggests optimal makeup; and means for providing special offer and discount information. This enables optimal makeup suggestions that reflect the user's individual concerns, wishes, and emotions, thereby providing a personalized and highly satisfying experience.

[1299] A "mobile device application" is software that a user operates using a mobile device, and is a means for inputting information about makeup, taking and uploading images, and displaying the results.

[1300] "Conversational AI" is an AI system that analyzes input from users and generates appropriate advice and answers using natural language processing technology.

[1301] "Image generation artificial intelligence" is an artificial intelligence system that generates a face image with makeup applied based on a bare face image uploaded by the user, advice provided by an interactive artificial intelligence, and the user's emotional information.

[1302] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and text data to recognize their emotions and then makes appropriate makeup suggestions based on that information.

[1303] A "server" is a computer system that receives data sent by a user, executes various artificial intelligence processes, and returns the results to the user's mobile device.

[1304] The "means for providing information on cosmetics and brands and purchase links" is a function for presenting information on the cosmetics and brands that are most suitable for the user, as well as purchase links, based on the generated face image after makeup.

[1305] The "means for providing special offers and discount information" is a function for providing users with special offers and discount information obtained from various cosmetic manufacturers, thereby promoting purchasing behavior.

[1306] The present invention relates to a system that generates a personalized makeup-applied facial image based on a user's natural facial image and provides information on the most suitable cosmetics and brands. This system is implemented by combining a mobile terminal application, a server, and multiple artificial intelligence technologies.

[1307] A user first installs the mobile device application, and when they launch the app for the first time, an account creation screen appears. When the user enters their name, email address, and password and presses the submit button, the device sends this data to the server. The server creates a new user account in the database based on the received data, generates an authentication token, and returns it to the device. This completes the user's account creation.

[1308] Next, the user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[1309] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are asked. The user enters their answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[1310] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. This engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[1311] Next, the device displays a message prompting the user to take and upload an image of their bare face. The user takes an image of their bare face using the smartphone camera and uploads it to the server via the device. The server receives this image and sends it to the image generation AI. The image generation AI generates a face image with makeup applied based on the received bare face image, the advice provided by the conversational AI, and the emotional information. For example, if the user wishes to "cover up redness," and the emotion engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover up the user's redness. This generated image is sent back to the device via the server, and the device displays it to the user.

[1312] Furthermore, the server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the information includes recommendation information such as "A specific brand of foundation is suitable for this makeup style" and a link to the purchasing site. The server sends this information to the terminal, which displays it to the user.

[1313] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[1314] Here are some examples of prompts for generative AI models:

[1315] "Based on an image of the user's bare face, please generate an optimal makeup-applied image, taking into account the content of this consultation and the user's emotions. Also, please provide recommendations for makeup products that will be suitable."

[1316] As a result, the system of the present invention can provide optimal makeup suggestions and product recommendations that reflect the user's individual concerns, desires, and emotions, thereby achieving a high level of satisfaction.

[1317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1318] Step 1:

[1319] The user installs the mobile device application and launches the account creation screen. The user provides input data such as name, email address, and password. The device sends this input data to the server. The server uses the received data to create a new user account in the database, generates an authentication token, and returns it to the device. This completes the user's account creation.

[1320] Input: Name, Email Address, Password

[1321] Output: New user account created, authentication token

[1322] Step 2:

[1323] The user enters their email address and password on the app's login screen and presses the login button. The device sends this authentication information to the server. The server checks the user information in the database, and if authentication is successful, generates an authentication token and returns it to the device. This puts the user into a logged-in state.

[1324] Input: Email address, password

[1325] Output: Authentication token, login status

[1326] Step 3:

[1327] After logging in, the device displays a series of makeup-related questions to the user. For example, questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" are displayed. The user enters the answers and presses the send button. The device then sends the user's answers to the server. The server uses conversational artificial intelligence to analyze the answers, generates appropriate advice, and returns it to the device. The device then displays this advice to the user.

[1328] Input: Answer to makeup question

[1329] Output: Analysis results and advice by interactive AI

[1330] Step 4:

[1331] When a user enters counseling details or uploads images, the device captures facial expressions and text data and sends it to an emotion recognition engine. The emotion recognition engine analyzes the user's words and facial expressions to recognize emotional information. This information is sent back to the server and reflected in the analysis of counseling details. For example, if a user writes that "my usual makeup routine doesn't work," the emotion engine will recognize emotions such as dissatisfaction or confusion and adjust its advice accordingly.

[1332] Input: facial expression data, text data

[1333] Output: Recognized emotion information

[1334] Step 5:

[1335] The device then displays a message prompting the user to take and upload a picture of their face without makeup. The user then takes a picture of their face with their smartphone camera and uploads it to the server via the device. The server receives the image and sends it to the image generation AI.

[1336] Input: natural face image

[1337] Output: The image of the face sent to the server

[1338] Step 6:

[1339] The server uses the image generation AI to generate a face image with makeup based on the received bare face image, advice provided by the conversational AI, and emotional information. For example, if the user wishes to "cover redness," and the emotion recognition engine strongly senses this, the image generation AI will generate a face image with makeup applied to cover the user's redness. This generated image is sent back to the terminal via the server, and the terminal displays it to the user.

[1340] Input: Face image, advice, emotional information

[1341] Output: Face image after makeup

[1342] Step 7:

[1343] The server selects the most suitable cosmetics and brand information for the user based on the generated face image after makeup. For example, the server may recommend a specific brand of foundation that is suitable for this makeup style, along with a link to a purchasing site. The server sends this information to the device, which then displays it to the user.

[1344] Input: Face image after makeup

[1345] Output: Cosmetics and brand information, purchase link

[1346] Step 8:

[1347] Finally, the server collects special offers and discount information from cosmetics manufacturers and sends it to the device. The device then displays this information to the user. For example, the device may display information such as "Currently, a specific brand of product is 10% off," to assist the user in making a purchase.

[1348] Enter: Special offers and discounts

[1349] Output: Display of special offers and discounts

[1350] (Application example 2)

[1351] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1352] Conventional makeup experience systems have the drawback of requiring users to purchase numerous actual products to try out different makeup looks, or to seek expert advice, which is time-consuming and expensive. Furthermore, the lack of emotion recognition and real-time makeup simulation makes it difficult to provide optimal advice tailored to the user's individual emotions and preferences. A new system that solves these problems is needed.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1354] In this invention, the server includes means for providing a mobile terminal application that enables a user to take and upload an image of their bare face, means for allowing the user to input their concerns and desires regarding makeup, and means for using interactive artificial intelligence to provide appropriate advice based on the input concerns and desires.

[1355] The system includes a means for using an image generation AI to generate a made-up facial image based on an image uploaded by a user and advice from an interactive AI, a means for displaying the generated made-up facial image to the user, a means for providing information on cosmetics and brands that are most suitable for the user and a link to purchase them, a means for recognizing emotions by analyzing the user's facial expressions and text data, and a means for suggesting a makeup style based on the emotional information. This system enables users to experience makeup styles in real time that correspond to their concerns, wishes, and emotions.

[1356] The "mobile terminal application" is an application that allows a user to take an image of their bare face and input their concerns and desires regarding makeup.

[1357] "Conversational AI" is an AI that provides appropriate advice based on the user's concerns and desires regarding makeup.

[1358] "Image generation artificial intelligence" is an artificial intelligence that generates a face image after makeup based on an image uploaded by the user and advice from an interactive artificial intelligence.

[1359] "Emotion recognition" is a technology that recognizes a user's emotions by analyzing their facial expressions and text data.

[1360] The "means for proposing a makeup style" is a means for proposing an optimal makeup style based on emotional information and the user's worries and desires.

[1361] The "means for displaying the generated face image after makeup to the user" refers to a means for displaying the face image after makeup generated by the image generation artificial intelligence to the user.

[1362] The "means for providing information on the most suitable cosmetics and brands to the user and a purchase link" is a means for providing information on the most suitable cosmetics and brands to the user and a corresponding purchase link based on the generated face image after makeup application.

[1363] The present invention is a system that allows users to upload a bare-face image and consult with them about makeup, then generates a realistic makeup-applied image, provides personalized makeup information, and suggests optimal makeup by recognizing the user's emotions. This system is implemented by combining mobile terminal applications, conversational artificial intelligence, image generation artificial intelligence, and emotion recognition technology.

[1364] The user installs the application for the first time and creates an account. The application sends necessary information such as name, email address, and password to the server. The server creates a user account in the database based on the received information, generates an authentication token, and returns it to the device. This logs the user in.

[1365] After logging in, the server presents the user with a series of makeup-related questions, prompting them to input their concerns and preferences. For example, questions such as "What are your skin concerns?" or "What kind of makeup style are you aiming for?" The input information is sent to the server, where it is analyzed by an interactive AI. Advice generated as a result of the analysis is returned to the device and displayed to the user.

[1366] When a user uploads an image of their bare face, emotion recognition technology analyzes the user's facial expressions and text data to recognize their emotions. For example, if a user writes about a concern such as "I can't do my usual makeup well," the emotion recognition technology will determine that the user is feeling dissatisfied or confused. This emotional information is sent to the server and reflected in the analysis of the counseling content.

[1367] Next, the user takes a picture of their bare face using their smartphone camera and uploads it. This image is sent to a server where it is processed by the image generation AI. Based on the bare face image sent to the server, the consultation details, and the user's emotional information recognized by emotion recognition technology, the image generation AI analyzes the face shape and skin tone to generate an image of the face with makeup applied.

[1368] The generated image of the face with makeup applied is sent to the terminal via the server and displayed to the user. A specific example is when a user inputs a concern such as "I'm bothered by the redness of my face" and uploads an image of their bare face. Emotion recognition technology recognizes the emotion of "confusion," and the conversational AI and image generation AI perform analysis taking this emotion into account. The generated image of the face with makeup is displayed in a style that effectively covers the redness.

[1369] Based on the generated image, the server also provides users with information on the best cosmetics and brands. For example, specific information such as "This makeup style is best suited to a specific brand of foundation" is included. This information also includes a purchase link, allowing users to easily access the purchasing site. In addition, if there are any special offers or discount information, these will also be provided.

[1370] The hardware used includes smartphones and head-mounted displays (HMDs), and the software used includes mobile terminal applications, server-side conversational artificial intelligence (e.g., OpenAI GPT-4), emotion recognition technology (e.g., Emotion API), and image generation artificial intelligence (e.g., GANs).

[1371] An example prompt might look like this:

[1372] User input: "I'm concerned about redness on my face, so I'd like makeup that makes my skin look even."

[1373] Prompt: "User desires: Concerned about redness on face, Emotion: Confused. Please suggest an appropriate makeup style."

[1374] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1375] Step 1:

[1376] A user installs a smartphone app and creates an account when they launch it for the first time. The user enters their name, email address, and password. The device sends this information to the server, and the server uses the received information to create a user account in a database, generates an authentication token, and returns it to the device. The input is user information, and the output is an authentication token. This operation puts the user in a logged-in state.

[1377] Step 2:

[1378] After the user logs in, the device displays a series of makeup-related questions to the user. The user inputs their concerns and desires in response to questions such as "What are your skin concerns?" and "What kind of makeup style are you aiming for?" The input is the user's concerns and desires, and the output is text data. This data is sent from the device to the server.

[1379] Step 3:

[1380] The server passes the received text data to the conversational AI, which generates appropriate advice. The input is the text data of the user's concerns and wishes, and the output is the generated advice. The conversational AI analyzes this and generates advice customized for each user. The generated advice is sent from the server to the terminal, which displays it to the user.

[1381] Step 4:

[1382] When a user inputs counseling content, emotion recognition technology analyzes the user's facial expressions and text data to recognize the user's emotions. The input is the user's facial expression image and text data, and the output is recognized emotional information. For example, if a user inputs a concern that "my usual makeup doesn't work," the server will determine that the user is feeling dissatisfied or confused. This emotional information is also sent to the server and used to analyze the counseling content.

[1383] Step 5:

[1384] A user takes a picture of their face using a smartphone camera and uploads it to the server via their device. The input is the face image, and the output is the image data sent to the server. The server then sends the received face image to the image generation AI.

[1385] Step 6:

[1386] Based on the bare face image and counseling details sent to the server, as well as the emotional information recognized by emotion recognition technology, the image generation AI analyzes the user's facial shape and skin tone and generates a face image with makeup applied. The input is the bare face image, the user's concerns and wishes, and emotional information, and the output is the face image with makeup applied.

[1387] Step 7:

[1388] The generated face image after makeup is sent from the server to the terminal and displayed to the user. The input is the generated face image, and the output is the face image displayed on the terminal. As a specific example, if a user inputs "I'm troubled by the redness of my face" and the emotion recognition technology recognizes "confused," the image generation technology will suggest a makeup style that covers the redness and display that image to the user.

[1389] Step 8:

[1390] The server provides the user with information on the most suitable cosmetics and brands based on the generated face image after makeup. The input is the face image after makeup, and the output is recommendation information. For example, it includes specific information such as "A specific brand of foundation is suitable for this makeup style." This information also includes a purchase link, allowing the user to easily access the purchasing site.

[1391] Step 9:

[1392] If there is special offer or discount information, the server collects it and sends it to the terminal. The input is the special offer or discount information, and the output is the displayed special offer or discount information. The user can use this information to purchase cosmetics at a discount.

[1393] This series of processing steps allows the user to experience the most suitable makeup style in real time according to their desires and emotions.

[1394] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1395] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1396] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1397] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1398] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1399] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1400] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1401] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1402] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1403] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1404] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1405] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1406] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1407] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1408] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1409] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1410] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1411] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1412] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1413] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1414] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1415] The following is further disclosed regarding the above embodiment.

[1416] (Claim 1)

[1417] providing a smartphone application for users to take and upload images of their natural faces;

[1418] A means for allowing a user to input worries and desires regarding makeup;

[1419] A means of using conversational AI to provide appropriate advice based on input concerns and wishes, and

[1420] A means for using an image generation AI that generates a face image after makeup based on an image uploaded by a user and advice from an interactive AI;

[1421] means for displaying the generated face image after makeup to a user;

[1422] A means of providing users with information and purchase links on the most suitable cosmetics and brands;

[1423] A system including:

[1424] (Claim 2)

[1425] The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation AI on the server.

[1426] (Claim 3)

[1427] The system of claim 1, wherein the interactive AI analyzes input from the user and provides individually customized makeup advice.

[1428] "Example 1"

[1429] (Claim 1)

[1430] means for providing a mobile device application for allowing a user to take and upload an image of their natural face;

[1431] A means for allowing a user to input worries and desires regarding makeup;

[1432] A means for using conversational artificial intelligence to provide appropriate advice based on input worries and wishes;

[1433] A means for using an image generating artificial intelligence to generate a face image after makeup based on an image uploaded by a user and advice from an interactive artificial intelligence;

[1434] means for displaying the generated face image after makeup to a user;

[1435] A means to provide users with information and purchasing links for the best beauty products and brands;

[1436] Collecting and providing special offers and discounts;

[1437] A system including:

[1438] (Claim 2)

[1439] The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation artificial intelligence on the server.

[1440] (Claim 3)

[1441] 2. The system of claim 1, wherein the interactive artificial intelligence analyzes input from the user and provides individually customized makeup advice.

[1442] "Application Example 1"

[1443] (Claim 1)

[1444] means for providing a mobile information terminal application for allowing a user to take and upload an image of their natural face;

[1445] A means for allowing a user to input worries and desires regarding makeup;

[1446] A method using natural language processing AI to provide appropriate advice based on input concerns and wishes,

[1447] A method using image generation AI that generates a face image after makeup based on an image uploaded by the user and advice from natural language processing AI;

[1448] means for displaying the generated face image after makeup to a user;

[1449] A means of providing users with information and purchase links on the most suitable cosmetics and brands;

[1450] a means to provide you with special offers and discounts;

[1451] A means of using augmented reality technology to try on makeup in a virtual space;

[1452] A system including:

[1453] (Claim 2)

[1454] The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation AI on the server.

[1455] (Claim 3)

[1456] The system of claim 1, wherein a natural language processing AI analyzes input from the user and provides individually customized makeup advice.

[1457] "Example 2: Combining Emotion Engines"

[1458] (Claim 1)

[1459] means for providing a mobile device application for a user to take and upload an image of their natural face;

[1460] A means for allowing a user to input worries and desires regarding makeup;

[1461] A means for using conversational artificial intelligence to provide appropriate advice based on input worries and wishes;

[1462] A means for using an image generating artificial intelligence to generate a face image after makeup based on an image uploaded by a user, advice from the interactive artificial intelligence, and emotional information of the user;

[1463] means for displaying the generated face image after makeup to a user;

[1464] A means of providing users with information and purchase links on the most suitable cosmetics and brands;

[1465] means for using an emotion recognition engine to recognize the user's emotions and suggest the most suitable makeup;

[1466] a means to provide you with special offers and discounts;

[1467] A system including:

[1468] (Claim 2)

[1469] The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation artificial intelligence on the server.

[1470] (Claim 3)

[1471] 2. The system of claim 1, wherein the interactive artificial intelligence analyzes input from the user and provides individually customized makeup advice.

[1472] "Application example 2 when combining emotion engines"

[1473] (Claim 1)

[1474] means for providing a mobile terminal application for a user to take and upload an image of their natural face;

[1475] A means for allowing a user to input worries and desires regarding makeup;

[1476] A means for using conversational artificial intelligence to provide appropriate advice based on input worries and wishes;

[1477] A means for using an image generating artificial intelligence to generate a face image after makeup based on an image uploaded by a user and advice from an interactive artificial intelligence;

[1478] means for displaying the generated face image after makeup to a user;

[1479] A means of providing users with information and purchase links on the most suitable cosmetics and brands;

[1480] A means for recognizing emotions by analyzing the user's facial expressions and text data;

[1481] A method for suggesting makeup styles based on emotional information,

[1482] A system including:

[1483] (Claim 2)

[1484] The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation artificial intelligence on the server.

[1485] (Claim 3)

[1486] 2. The system of claim 1, wherein the interactive artificial intelligence analyzes input from the user and provides individually customized makeup advice. [Explanation of symbols]

[1487] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. providing a smartphone application for users to take and upload images of their natural faces; A means for allowing a user to input worries and desires regarding makeup; A means of using conversational AI to provide appropriate advice based on input concerns and wishes, and A means for using an image generation AI that generates a face image after makeup based on an image uploaded by a user and advice from an interactive AI; means for displaying the generated face image after makeup to a user; A means of providing users with information and purchase links on the most suitable cosmetics and brands; A system including:

2. The system according to claim 1, wherein the captured image of the face is sent to a server and processed by an image generation AI on the server.

3. The system according to claim 1, wherein the interactive AI analyzes input from the user and provides individually customized makeup advice.

Images