System

Abstract

A system is provided.SOLUTION: A system including image recognition means, means for analyzing a skin color and a skin condition of a user, means for proposing a makeup product on the basis of an analysis result, means for proposing a skin care product on the basis of the analysis result, means for proposing beauty treatment on the basis of the analysis result, means for comparing a past image with a current image, and means for making a proposal for approximating the past state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, interest in beauty is on the rise, and a wide variety of makeup, skincare, and cosmetic medical treatments are available. However, selecting each product and procedure requires specialized knowledge. However, it is not easy for average consumers to accurately understand their own skin condition and past skin conditions and choose the most appropriate beauty products and medical services. It is also difficult to find specific measures to restore previous healthy skin conditions. Therefore, there is a need for a system that can efficiently provide optimal makeup, skincare, and cosmetic medical treatment suggestions based on each user's individual skin condition. [Means for solving the problem]

[0005] The present invention provides a system including an image recognition unit, a unit for analyzing a user's skin color and skin condition, a unit for suggesting makeup products based on the analysis results, a unit for suggesting skin care products based on the analysis results, a unit for suggesting cosmetic treatments based on the analysis results, a unit for comparing past images with current images, and a unit for making suggestions to bring the skin closer to the past condition. Furthermore, the system includes a unit for receiving images uploaded by the user, a unit for analyzing the received images, a unit for searching a database using the analysis results, and a unit for sending the search results to a terminal, making it possible to efficiently provide optimal beauty suggestions based on the user's skin condition. Furthermore, by suggesting skin care products and cosmetic treatments based on changes in the user's skin condition and displaying these suggestions on the user's terminal, the user can easily make their own beauty choices.

[0006] "Image recognition means" is a function that analyzes images uploaded by users and identifies facial and body information.

[0007] The "means for analyzing the user's skin color and skin condition" is a function for determining the user's skin characteristics such as skin color, dryness, oiliness, blemishes, wrinkles, etc. from information acquired by the image recognition means.

[0008] The "means of suggesting makeup products" is a function that lists and suggests cosmetics such as foundation, eye shadow, blush, and lipstick that suit the user based on the analysis results.

[0009] "Means for suggesting skin care products" refers to a function that selects and suggests the optimal skin care products (cleansers, toners, serums, creams, etc.) to improve the user's skin condition.

[0010] "Means for suggesting cosmetic medical treatments" is a function that suggests cosmetic medical treatments such as laser treatments and hyaluronic acid injections for skin problems that cannot be solved by skin care.

[0011] "Means for comparing past and current images" refers to a function that analyzes past and current images uploaded by users to identify changes in skin condition.

[0012] The "means of making suggestions to bring the skin closer to its previous state" refers to a function that uses the patient's past healthy skin condition as a reference to suggest specific skin care products and cosmetic medical treatments to improve the patient's current skin condition.

[0013] "Means for searching the database" refers to a function that searches and obtains information on optimal makeup products, skin care products, and cosmetic medical treatments from the database based on the analysis results.

[0014] "Means for displaying proposals to users" refers to a function that displays analysis results and information obtained from a database in a form that is easy for users to understand. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in skin condition, and provides specific beauty recommendations based on the results.

[0037] Program processing overview

[0038] This system operates roughly as follows:

[0039] 1. User Actions

[0040] Users install a dedicated app on their device, create an account and log in. They upload a current photo of their face and a previous photo of their healthy face.

[0041] 2. Terminal Processing

[0042] The device sends the current and past facial photos uploaded by the user to the server. The device then sends the image data to the server in an appropriate format and waits for a response.

[0043] 3. Server Processing

[0044] The server analyzes the received images to identify the user's skin tone and condition, then searches a database to generate optimal recommendations for makeup, skincare products, and cosmetic treatments based on the analysis results.

[0045] Learn more about natural language processing

[0046] User Actions

[0047] 1. The user launches the app, creates an account, and logs in.

[0048] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0049] 3. Next, select the "Upload a Previous Photo" option and upload a photo of your face from a time when you were healthy to the app.

[0050] Terminal handling

[0051] 1. The device sends the uploaded current and past facial photos to a server via the Internet.

[0052] 2. The device receives a response from the server and waits for the analysis results and proposals.

[0053] Server Processing

[0054] 1. The server analyzes the received image and identifies the user's skin color, dryness or oiliness, distribution of blemishes and wrinkles, etc.

[0055] 2. Based on the analysis results, the server selects makeup products (foundation, eye shadow, blush, lipstick, etc.) from a database.

[0056] 3. The server then searches for skin care products to improve the user's current skin condition and generates recommendations.

[0057] 4. The server compares the past and current images and suggests cosmetic medical treatments the user needs to restore their previous healthy skin condition.

[0058] Specific examples

[0059] 1. The user uploads a photo of their current face and also uploads a photo of their face from a few years ago with healthy skin.

[0060] 2. The device sends these photos to the server.

[0061] 3. The server analyzes the received image and determines that "the skin is dry overall, but oily in the T-zone. The skin color is light ochre."

[0062] 4. Depending on your skin condition, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0063] 5. Select skin care products such as whitening toner and moisturizing cream, and provide instructions for their use.

[0064] 6. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing blemishes and hyaluronic acid injections for skin lifting.

[0065] In this way, users can easily obtain specific measures to improve their skin condition.

[0066] The processing flow will be explained below.

[0067] Program processing flow

[0068] User Actions

[0069] Step 1:

[0070] Users install the dedicated app on their device, create an account, and log in.

[0071] Step 2:

[0072] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[0073] Step 3:

[0074] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[0075] Terminal handling

[0076] Step 1:

[0077] The device sends the uploaded current and past facial photos to the server.

[0078] Step 2:

[0079] The device continues to maintain the network connection while waiting for a response from the server.

[0080] Server Processing

[0081] Step 1:

[0082] The server inputs the current and past facial photos received from the device into an algorithm that analyzes them.

[0083] Step 2:

[0084] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0085] Step 3:

[0086] Based on the analysis results, databases of makeup products, skin care products, and cosmetic medical treatments are queried.

[0087] Step 4:

[0088] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition.

[0089] Step 5:

[0090] The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous condition, and also provides instructions for use.

[0091] Step 6:

[0092] The server will suggest the most appropriate cosmetic medical treatment (e.g., laser treatment for removing spots, hyaluronic acid injections) for areas of the skin that need improvement.

[0093] Step 7:

[0094] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[0095] Terminal handling (again)

[0096] Step 1:

[0097] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0098] Step 2:

[0099] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0100] User Action (again)

[0101] Step 1:

[0102] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0103] Step 2:

[0104] Users consider whether the proposed products and treatments are suitable for them.

[0105] Step 3:

[0106] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[0107] In this way, specific actions are performed for each processing step, allowing the user to easily make optimal choices regarding their own beauty.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional beauty recommendation systems lack detailed suggestions based on the user's specific skin condition or past skin condition, making it difficult for users to efficiently find products and cosmetic treatments that are suitable for their skin condition. In addition, since they do not provide specific advice on how to return to a previous healthy skin condition, user satisfaction can be low.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes an image recognition unit, a unit for receiving current and past images uploaded by the user, and a unit for analyzing the user's skin color and condition. This allows for detailed recommendations for makeup products, skin care products, and cosmetic treatments based on the analysis results. Furthermore, by comparing past and current images and making specific recommendations to bring the user's skin closer to its previous condition, user satisfaction is improved.

[0113] "Image recognition means" refers to a device or software that has the function of analyzing input image data and identifying and extracting specific features or patterns.

[0114] "Means for receiving current and past images uploaded by users" refers to devices or software that have the function of receiving image data sent by users via the Internet or an application.

[0115] "Means for analyzing the user's skin color and skin condition" refers to a device or software that has the function of identifying the user's skin condition, such as skin color, dryness, oiliness, and distribution of spots and wrinkles, based on the received image data.

[0116] The "means for suggesting makeup products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable makeup products to the user based on the analysis results of the skin condition.

[0117] The "means for suggesting skin care products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable skin care products to the user based on the analysis results of the skin condition.

[0118] "Means for proposing cosmetic medical treatments based on analysis results" refers to devices or software that have the function of selecting and proposing cosmetic medical treatments or procedures suitable for a user based on the analysis results of the user's skin condition.

[0119] The "means for comparing a past image with a current image" refers to a device or software that has the function of comparing a past image provided by a user with a current image and identifying points of change.

[0120] The "means for making suggestions to bring the skin closer to its previous state" refers to a device or software that has the function of generating suggestions for makeup products, skin care products, and cosmetic medical treatments necessary to restore the skin to its previous healthy state.

[0121] The "means for transmitting the analysis results and proposal contents to the user's terminal" refers to a device or software that has the function of transmitting the analysis results and proposal contents to the user's terminal via the Internet and displaying them.

[0122] "Skin care products" refers to a broad range of products including cosmetics, moisturizing creams, whitening toners, etc. that are used for the purpose of caring for and maintaining healthy skin.

[0123] "Cosmetic medicine" refers to treatments and procedures that use medical technology to pursue and improve beauty, such as laser treatments, hyaluronic acid injections, and skin lifting.

[0124] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in the skin condition, and provides specific beauty suggestions based on the results.

[0125] Hardware and software used

[0126] To implement this system, a user device, a server, and image recognition software are required. Specifically, it uses a user device such as a smartphone or tablet, a cloud server, and image analysis software (e.g., OpenCV, TensorFlow).

[0127] User Actions

[0128] Users install a dedicated app on their device. This app has functions such as uploading images, displaying analysis results, and receiving suggestions. Users launch the app, create an account, and log in. After logging in, users select the "upload photos" option and upload a current photo of their face and a previous photo of their healthy face to the app.

[0129] Terminal handling

[0130] The device temporarily stores the uploaded current and past facial photos in its memory. The device then converts these image data into an appropriate format (e.g., JPEG, PNG) and sends them to a server via the Internet. The device then waits for a response from the server. Through this series of operations, users can easily analyze their skin condition and receive recommendations.

[0131] Server Processing

[0132] The server stores the received image data in a dedicated directory. It then uses image recognition software to analyze the image data and identify the user's skin condition (skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.). The analysis results are stored in a database. Based on the analysis results, the server then searches the database to generate recommendations for makeup products, skin care products, and cosmetic treatments that are optimal for the user. For example, if the server determines that the user's skin condition is "dry overall, but oily in the T-zone, and light ochre skin color," it can recommend products accordingly.

[0133] Specific examples

[0134] 1. Users upload a current photo of their face, as well as a past photo of their face when they were healthy.

[0135] 2. The device sends these photos to the server.

[0136] 3. Based on the analysis results, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0137] 4. As skin care products, select toners and moisturizing creams with whitening effects and provide instructions for their use.

[0138] 5. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing age spots and hyaluronic acid injections for skin lifting.

[0139] Prompt Sentence Examples

[0140] "Please upload a current photo of your face and a photo from a previous time. We will analyze your skin condition and suggest the best makeup products, skincare products, and cosmetic treatments for you."

[0141] In this way, users can easily obtain specific measures to improve their skin condition, and this system effectively resolves users' beauty concerns and problems.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1: User Actions

[0144] Input: The user installs the app, creates an account, and logs in. Within the app, they select "Upload Photos" and upload their current and past facial photos.

[0145] Data processing: The app temporarily stores photos uploaded by users in the device's memory.

[0146] Output: Saved current and past face photos.

[0147] Users install the app on their smartphone or tablet, create an account, and log in. They select the "upload photo" option within the app, take or select a photo of their current face, and then upload a photo of their face from a previous period when they looked healthy.

[0148] Step 2: Send an image from your device

[0149] Input: Your saved current and past face photos.

[0150] Data processing: The device converts image data into an appropriate format (e.g., JPEG, PNG) and optimizes the data size.

[0151] Output: Image data for transmission.

[0152] The device temporarily stores the current and past facial photos uploaded by the user in memory, then converts the image data into an appropriate format and sends it to a server over the Internet.

[0153] Step 3: Server Image Analysis

[0154] Input: Current and past face photos submitted.

[0155] Data processing: The server uses image recognition software (e.g., OpenCV, TensorFlow) to analyze the user's skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.

[0156] Output: Analysis results (e.g. skin tone is light ochre, dryness is high, oiliness is medium, number of blemishes is 12, etc.).

[0157] The server stores the received image data in a dedicated directory and uses image recognition software to analyze the user's skin condition. The analysis results are stored in a database as numerical and text data.

[0158] Step 4: Generate a Server Proposal

[0159] Input: Analysis result data.

[0160] Data processing: The server searches the database and generates optimal makeup, skincare, and cosmetic medical treatment recommendations based on the analysis results.

[0161] Output: Suggestions (e.g. moisturizing foundation, matte finish eyeshadow, whitening toner, hyaluronic acid injections, etc.).

[0162] Based on the analysis results, the server searches the database and makes recommendations for the most suitable makeup products, skin care products, and cosmetic medical treatments for the user. For example, appropriate recommendations are generated based on analysis results such as "overall dry skin, but oily T-zone. Skin tone is light ochre."

[0163] Step 5: Handling the Server Response

[0164] Input: Proposal data.

[0165] Data processing: The server formats the proposal content into a format that can be sent to the user's device.

[0166] Output: Formatted proposal.

[0167] The server then sends the generated recommendations to the user's device, which then displays them on the app's interface, providing the user with easy-to-read information. Users can then review the suggested makeup, skincare, and cosmetic treatments and try them out as needed.

[0168] (Application example 1)

[0169] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0170] In today's world, many people worry about their skin condition in their daily lives, but often lack the knowledge and information to select appropriate skincare products and cosmetic medical treatments. Time and location constraints also make it difficult to consult with a beauty professional. Conventional methods require in-store consultations, so a quick and easy method is needed.

[0171] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0172] In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic medical treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the skin closer to the past condition, and means for receiving images uploaded by the user and displaying the analysis results and suggestions, which is an application installed on a smartphone. This allows the user to easily analyze their own skin condition from the comfort of their own home and receive suggestions for appropriate beauty products and services.

[0173] "Image recognition means" is a technology for analyzing images uploaded by users and identifying their contents.

[0174] "Skin color" indicates the color tone of the user's skin, and is information obtained by image analysis.

[0175] "Skin condition" refers to the health and appearance of the user's skin, including dryness, oiliness, blemishes, wrinkles, and the like.

[0176] "User" means a person who uses the System and uploads images to analyze their skin condition and receive suggestions for beauty products and services.

[0177] "Makeup products" refers to cosmetics intended to enhance the user's appearance, including foundation, eye shadow, lip gloss, etc.

[0178] "Skin care products" refers to products that are intended to maintain and improve the health of a user's skin, including toners and moisturizing creams.

[0179] "Cosmetic medicine" refers to medical procedures aimed at improving a user's skin condition, including, for example, laser treatments and hyaluronic acid injections.

[0180] A "past image" is a photograph of the user's face taken in the past, showing a past healthy skin condition.

[0181] The "current image" is the most recent photograph of the user's face taken to show the user's current facial condition.

[0182] The "comparison means" is a technology for analyzing past and current images to identify changes in skin condition.

[0183] The "means of suggestion" refers to technology that provides users with appropriate makeup products, skin care products, and cosmetic medical treatments based on the analysis results.

[0184] A "smartphone" is a multi-functional mobile phone that allows users to install and use applications.

[0185] "Application" refers to a software program that is installed on a smartphone and provides the functionality of the present invention.

[0186] "Means for receiving" refers to the technology used to send and receive images uploaded by users to the server.

[0187] "Display means" refers to technology for visually presenting the analysis results and proposals to users.

[0188] A "generative AI model" is a model that uses artificial intelligence technology to generate prompts and suggestions based on the analysis results.

[0189] A "prompt" is an instruction entered when utilizing a generative AI model, and is a guideline for obtaining specific output results.

[0190] This invention is a system that analyzes a user's skin color and condition and, based on that, suggests optimal makeup products, skin care products, and cosmetic medical treatments. This system includes an application and a server installed on a smartphone, and operates according to the following specific steps.

[0191] First, users log in to the application installed on their smartphone and upload their current and past facial photos. The smartphone then sends the uploaded images to a server via the Internet.

[0192] The server analyzes the received images using image recognition means to identify the user's skin tone and condition (dryness, oiliness, blemishes, wrinkles, etc.) Based on the analysis results, the server searches a database to generate optimal recommendations for makeup products, skin care products, and cosmetic medical treatments.

[0193] The recommendations, along with the analysis results, are sent to the user's smartphone and displayed, allowing them to easily find beauty products and services that are suitable for their skin condition from the comfort of their own home.

[0194] The server then uses a generative AI model based on the analysis results to create a prompt. The prompt is generated as a guideline for obtaining specific output results and is provided to the user. Utilizing the generative AI model enables more accurate suggestions.

[0195] For example, a user can upload a photo of their current face and a photo of their face in a previous healthy state. The server compares these photos and generates an analysis result that indicates "dry skin overall, but oily T-zone." It then makes specific recommendations, such as moisturizing foundation, whitening toner, and hyaluronic acid injections.

[0196] Example prompt sentence:

[0197] Analyze a user-uploaded facial photo. Identify skin tone, dryness, oiliness, blemishes, and wrinkles, and provide specific recommendations for skin care products, makeup products, and cosmetic treatments based on the results. Generate specific output such as "Skin condition analysis: Light ochre, overall dry, T-zone oily, blemish distribution: High."

[0198] In this way, users can easily select the best beauty products and services at home without the need for expert guidance.

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] The user launches the application installed on their smartphone, creates an account and logs in.

[0202] Input: User account information (ID, password, etc.).

[0203] Data processing / data calculation: Account information is sent to the server for login authentication and verification.

[0204] Output: The home screen is displayed after successful login.

[0205] Step 2:

[0206] The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0207] Input: A current photo of the user.

[0208] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[0209] Output: Current face photo data is stored within the app.

[0210] Step 3:

[0211] Users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[0212] Input: A previous photo of the user's face.

[0213] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[0214] Output: Past facial photo data is stored within the app.

[0215] Step 4:

[0216] The app sends the current and past facial photos uploaded by the user to the server.

[0217] Input: Current face photo, past face photo.

[0218] Data processing / data calculation: Photo data is sent to a server via the Internet.

[0219] Output: The photo data is saved on the server.

[0220] Step 5:

[0221] The server analyzes the received images and identifies the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0222] Input: Current face photo, past face photo.

[0223] Data processing / data calculation: Apply image recognition algorithms to analyze skin condition (e.g., using Python's OpenCV library).

[0224] Output: Analysis results for skin color, dryness, oiliness, spots, wrinkles, etc.

[0225] Step 6:

[0226] Based on the analysis results, the server searches its database to generate optimal suggestions for makeup products, skin care products, and cosmetic medical treatments.

[0227] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[0228] Data Processing / Data Calculation: Performing database queries to find suitable products and services.

[0229] Output: A list of suggested beauty products and services.

[0230] Step 7:

[0231] The suggestions, along with the analysis results, are sent to the smartphone and displayed to the user through the application.

[0232] Input: A list of beauty products and services you would like to suggest.

[0233] Data processing / data calculation: The proposal is formatted and sent to the application.

[0234] Output: The analysis results and suggestions are displayed on the smartphone screen.

[0235] Step 8:

[0236] Based on the analysis results, the server uses a generative AI model to form a prompt sentence.

[0237] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[0238] Data processing / data calculation: Input the analysis results into a generative AI model to generate a prompt sentence (e.g., an AI model such as GPT-3).

[0239] Output: The generated prompt statement.

[0240] Step 9:

[0241] The generated prompt text is presented to the user.

[0242] Input: The generated prompt statement.

[0243] Data processing / data calculation: Formats the prompt statement and sends it to the application.

[0244] Output: The generated prompt text is displayed on the smartphone screen.

[0245] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0246] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition and, based on that, recommend optimal makeup products, skin care products, and cosmetic medical treatments. The system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[0247] Program processing overview

[0248] This system operates roughly as follows:

[0249] User Actions

[0250] Users install the app on their device, create an account, and log in. They upload a current photo of their face and a previous photo of their healthy face. The system can also use the camera feed to recognize the user's real-time emotional state.

[0251] Terminal handling

[0252] The device sends the current and past facial photos uploaded by the user, as well as the real-time camera feed, to the server, which then sends this data in the appropriate format and waits for a response.

[0253] Server Processing

[0254] The server analyzes the received images and real-time camera feed to identify the user's skin tone, skin condition and emotional state, then searches a database to generate optimal recommendations for makeup, skincare products and cosmetic treatments based on the analysis.

[0255] Learn more about natural language processing

[0256] User Actions

[0257] 1. The user launches the app, creates an account, and logs in.

[0258] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0259] 3. Next, users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[0260] 4. The system uses the camera feed to capture the user's face to recognize their real-time emotional state.

[0261] Terminal handling

[0262] 1. The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[0263] 2. The device maintains the network connection while waiting for a response from the server.

[0264] Server Processing

[0265] 1. The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[0266] 2. The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0267] 3. Furthermore, the server uses emotion recognition algorithms to analyze the user's current emotional state.

[0268] 4. Based on the analysis results, query databases of makeup products, skin care products, and cosmetic medical treatments.

[0269] 5. The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit their skin condition and emotional state.

[0270] 6. The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the previous skin condition, and also provides instructions for using them.

[0271] 7. When it comes to cosmetic medicine, the app will suggest the most appropriate treatment (e.g., laser treatment for removing dark spots, hyaluronic acid injections) based on the user's current emotional state.

[0272] 8. The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device.

[0273] Terminal handling (again)

[0274] 1. The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0275] 2. The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0276] User Action (again)

[0277] 1. The user checks the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0278] 2. The user considers whether the proposed products or treatments are suitable for them.

[0279] 3. The user uses the suggestions to purchase a specific product or make an appointment at a beauty salon or clinic.

[0280] Specific examples

[0281] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[0282] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[0283] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[0284] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[0285] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[0286] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0287] The processing flow will be explained below.

[0288] MODE FOR CARRYING OUT THE INVENTION

[0289] User Actions

[0290] Step 1:

[0291] Users install a dedicated app on their device, create an account and log in.

[0292] Step 2:

[0293] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[0294] Step 3:

[0295] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[0296] Step 4:

[0297] The app uses real-time emotion recognition to capture the user's current emotions from the camera feed.

[0298] Terminal handling

[0299] Step 1:

[0300] The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[0301] Step 2:

[0302] The device maintains a network connection while waiting for analysis results and recommendations from the server.

[0303] Server Processing

[0304] Step 1:

[0305] The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[0306] Step 2:

[0307] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0308] Step 3:

[0309] The server uses emotion recognition algorithms to analyze the user's current emotional state (e.g., stress, relaxation, joy, sadness, etc.).

[0310] Step 4:

[0311] Based on the analysis results (skin condition and emotional state), the server queries databases of makeup products, skin care products, and cosmetic medical treatments.

[0312] Step 5:

[0313] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition and emotional state.

[0314] Step 6:

[0315] The server compares the past and current images and selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous healthy state.

[0316] Step 7:

[0317] Taking into account your emotional state, the server will suggest relaxing skin care products (e.g., moisturizing cream containing lavender oil) or cosmetic treatments that promote psychological stress reduction (e.g., relaxation massage).

[0318] Step 8:

[0319] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[0320] Terminal handling (again)

[0321] Step 1:

[0322] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0323] Step 2:

[0324] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0325] User Action (again)

[0326] Step 1:

[0327] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0328] Step 2:

[0329] Users consider whether the proposed products and treatments are suitable for them.

[0330] Step 3:

[0331] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[0332] Specific examples

[0333] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[0334] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[0335] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[0336] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[0337] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[0338] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0339] Example 2

[0340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] Conventional beauty recommendation systems can only make recommendations based on the user's current skin condition, making it difficult to provide comprehensive beauty recommendations that take into account the user's past healthy skin condition and emotional state. Furthermore, since recommendations for skin care products and cosmetic medical treatments do not reflect the user's real-time emotional state, it is not possible to provide optimal recommendations for each individual user.

[0342] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image recognition means, a means for analyzing the user's skin color and skin condition, a means for suggesting makeup products based on the analysis results, a means for suggesting skin care products based on the analysis results, a means for suggesting cosmetic medical treatments based on the analysis results, a means for comparing a past image with a current image, a means for making suggestions to bring the image closer to the past state, a means for recognizing the user's real-time emotional state, and a means for making optimal suggestions according to the user's emotional state based on the analysis results. This makes it possible to provide beauty suggestions that comprehensively consider the user's skin condition and emotional state.

[0343] "Image recognition means" refers to technology or equipment for detecting and analyzing specific features (e.g., face, skin condition, etc.) from an image.

[0344] "Means for analyzing a user's skin color and skin condition" refers to technology or equipment that analyzes a user's skin color, texture, dryness / oiliness, blemishes, wrinkles, etc.

[0345] "Means for suggesting makeup products based on analysis results" refers to technology or equipment that selects appropriate makeup products based on the results of skin analysis and suggests them to users.

[0346] The "means for suggesting skin care products based on the analysis results" refers to a technology or device that uses the skin analysis results to select appropriate skin care products and suggest them to the user.

[0347] "Means for proposing cosmetic medical treatment based on analysis results" refers to technology or devices that suggest cosmetic medical treatment (e.g., laser treatment, injections, etc.) based on the results of skin analysis.

[0348] The "means for comparing a past image with a current image" refers to a technique or device that compares a past image provided by the user with a current image to evaluate changes in skin condition.

[0349] The "means for making suggestions to bring the skin closer to its previous state" refers to a technology or device that suggests skin care methods or cosmetic medical treatments to bring the skin back to its previous healthy state.

[0350] "Means for recognizing a user's real-time emotional state" refers to a technology or device that analyzes and recognizes a user's current emotional state (e.g., stress, relaxation, etc.) in real time.

[0351] "Means for making optimal suggestions based on the user's emotional state based on the analysis results" refers to technology or equipment that selects and suggests appropriate skin care products, cosmetic medical treatments, makeup products, etc. based on the user's emotional state.

[0352] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on that analysis. Specifically, the system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[0353] overview

[0354] This system roughly consists of operations by users who have installed a dedicated app, processing by the device, and processing by the server. The user uploads a photo of their face and performs real-time emotion recognition, and the device sends this data to the server. The server performs image and emotion recognition, generates beauty suggestions based on the analysis results, sends them to the device, and finally makes them available to the user.

[0355] Technology and software used

[0356] Image recognition techniques: Use convolutional neural network (CNN) models such as ResNet and VGG.

[0357] Emotion recognition technology: Use emotion recognition models such as FER+.

[0358] Databases: Use databases related to beauty products and aesthetic medicine (e.g., general beauty product databases).

[0359] Communication technology: Uses network communication protocols such as HTTP and HTTPS.

[0360] The process begins when a user launches the app, creates an account, and logs in. The user uploads a current photo of their face and a previous photo of their healthy face, and the app also uses their camera feed to recognize the user's emotional state in real time.

[0361] The device sends this data (current facial photo, past facial photos, and real-time camera feed) to a server, which then analyzes the data using image and emotion recognition algorithms to identify skin condition and emotional state.

[0362] The server then searches its database to suggest makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss), skin care products (e.g., whitening toner, moisturizing cream), and cosmetic medical treatments (e.g., laser treatment for dark spots, hyaluronic acid injections) based on the analysis results.

[0363] The server converts the generated proposals into JSON format and sends them to the device, which then visually displays the proposals and allows the user to consider specific products and treatments.

[0364] Specific examples

[0365] For example, a user can upload a current photo of their face and a photo of their face from a previous period when they were healthy. At the same time, the real-time camera feed is used to analyze their emotional state. The server determines that their skin is dry overall, but oily in the T-zone, and their skin tone is light ochre, and detects that the user's emotional state is stressed.

[0366] Based on these conditions, the server makes the following suggestions:

[0367] Moisturizing foundation (brand name is common)

[0368] Matte Finish Eyeshadow (same as above)

[0369] Natural lip gloss (same as above)

[0370] Relaxing skin care products (for example, moisturizing cream with lavender oil)

[0371] How to use whitening toner and moisturizing cream

[0372] Laser treatment for removing spots and hyaluronic acid injections for lifting

[0373] Prompt Sentence Examples

[0374] "Design a system that analyzes current and past facial photos and suggests optimal makeup and skin care."

[0375] "Please explain in detail the system that compares your current skin condition with your past skin condition and suggests cosmetic medical treatments."

[0376] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0378] Step 1: User installs and launches the app

[0379] The user installs a dedicated app from the App Store or Google Play on their smartphone. Once the installation is complete, they tap the app icon to launch the app. This is the input. The output is the display of the app's launch screen.

[0380] Step 2: User creates account and logs in

[0381] When a user launches the app for the first time, they create an account using their email address or social media account. They are prompted to enter their ID and password. Existing users log in by entering their ID and password. The input is account information and login information, and the output is obtaining access rights to the user's personal page.

[0382] Step 3: User uploads a current photo of their face

[0383] The user selects the "upload photo" option in the app, selects a current face photo, and uploads it. The app uses the smartphone's camera roll and interface. The input is the current face photo, and the output is sending the photo to the server.

[0384] Step 4: User uploads a photo of their face

[0385] The user selects the "Upload a Previous Photo" option and selects and uploads a previous photo of their face with healthy skin. The smartphone's camera roll and interface are used again. The input is the previous photo of the face, and the output is the transmission of the photo to the server.

[0386] Step 5: The device sends the real-time camera feed to the server

[0387] The device activates the device's camera to recognize the user's real-time emotional state, captures the user's face, and sends the camera feed to the server. The input is the real-time camera feed, and the output is the feed data sent to the server.

[0388] Step 6: Server parses images and camera feed

[0389] The server analyzes the received current face photo, past face photos, and real-time camera feed. It processes the data using image recognition algorithms (e.g., ResNet, VGG) and emotion recognition algorithms (e.g., FER+). The input is the received image and camera feed, and the output is the analysis results such as the user's skin color, skin condition, and emotional state.

[0390] Step 7: The server generates beauty suggestions based on the analysis results.

[0391] The server then searches a database based on the analysis results to make beauty recommendations. For example, it selects makeup products, skin care products, and cosmetic treatments based on skin condition. The input is the analysis results of skin color, skin condition, and emotional state, and the output is a set of recommended beauty products and cosmetic treatments.

[0392] Step 8: Send the server-generated proposal to the device

[0393] The server converts the generated beauty suggestions into an appropriate format (e.g., JSON) and sends them to the device via a network communication protocol (e.g., HTTP or HTTPS). The input is a set of beauty suggestions, and the output is JSON-formatted data.

[0394] Step 9: The device displays a visual suggestion

[0395] The device analyzes the beauty suggestions received from the server and displays them visually in the app for the user. The input is JSON format data, and the output is a visual display within the app.

[0396] Step 10: User confirms and uses the proposal

[0397] The user checks the suggestions displayed on the device and clicks on a link to purchase the suggested product or make an appointment at a beauty salon or clinic. The input is the visually displayed suggestion, and the output is the actual purchase or appointment.

[0398] (Application example 2)

[0399] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] Conventional beauty recommendation systems suggest makeup and skincare products based on an analysis of the user's skin tone and condition, but they lacked the ability to consider the user's real-time emotional state. Additionally, while some systems had a function to compare a user's current and past images, they did not suggest cosmetic treatments based on the user's emotional state, making it difficult to provide more personalized recommendations.

[0401] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the image closer to the past state, means for recognizing emotional state in real time, means for suggesting skin care products based on the emotional state, and means for suggesting cosmetic treatments based on the emotional state. This makes it possible to comprehensively analyze the user's skin condition and emotional state and make optimal beauty suggestions based on the current state and emotion.

[0402] "Image recognition means" is a general term for software and hardware that receives images uploaded by users, analyzes them, and recognizes the user's facial features and skin condition.

[0403] The "means for analyzing skin color and skin condition" is a system for analyzing a user's skin color, dryness, oiliness, blemishes, wrinkles, etc. using image recognition technology.

[0404] The "means for suggesting makeup products" refers to algorithms and software that suggest optimal cosmetics to users based on the analysis results.

[0405] The "means for suggesting skin care products" is a system that selects and suggests skin care items suitable for the user's skin condition based on the analysis results.

[0406] The "means for proposing cosmetic medical treatment" is a system for proposing optimal cosmetic medical treatments and procedures based on the user's skin condition and emotional state.

[0407] The "means for comparing images" refers to software and hardware that uses an algorithm to compare past and current images and analyze changes and areas for improvement in skin condition.

[0408] The "means for making suggestions" is a system that suggests to users the makeup products, skin care products, and cosmetic medical treatments they need to bring their appearance closer to their previous state.

[0409] "Means for recognizing emotional state" refers to technology for analyzing a user's emotional state (e.g., stress, relaxation) using real-time camera feeds, etc.

[0410] The "means for suggesting skin care products based on emotional state" is a system for suggesting skin care products suitable for a user in consideration of the analyzed emotional state.

[0411] The "means for suggesting cosmetic medical treatments based on emotional state" is a system for suggesting cosmetic medical treatments or procedures according to the user's emotional state.

[0412] This invention is a system that analyzes a user's skin color and condition and suggests optimal makeup products, skin care products, and cosmetic medical treatments. The system also includes a function to recognize the user's emotional state in real time and make suggestions based on the emotional state.

[0413] The overall structure of the system is as follows: Users install a dedicated application on their smartphone, create an account, and log in. They then take a photo of their current face and upload it through the application. By uploading past facial photos as well, the server can compare and analyze past and current images. The application also uses the smartphone's camera to recognize the user's real-time emotional state.

[0414] The process on the device (user's smartphone) is as follows: The current face photo uploaded by the user, past face photos, and real-time camera feed are sent to the server, and this data is sent to the server in an appropriate format.

[0415] The server includes the following means for analyzing the received data:

[0416] 1. Image recognition means: Analyzes the received current and past facial photos to recognize the user's skin color and condition (dry, oily, spots, wrinkles, etc.).

[0417] 2. Skin color and condition analysis: Analyze the user's skin characteristics based on the results of image recognition. This analysis uses image processing libraries such as OpenCV.

[0418] 3. Real-time emotional state recognition: Using the camera feed, the user's emotional state is analyzed in real time using emotion recognition algorithms.

[0419] 4. Means for making suggestions based on analysis results: Based on the analysis results and emotional state, the most suitable makeup products, skin care products, and cosmetic medical treatments are searched from a database and suggestions are generated.

[0420] 5. Image comparison: Compare past and current images to analyze changes in skin condition and generate suggestions to restore a previous healthy state.

[0421] 6. Means for sending proposals: The generated proposals are converted into an appropriate format such as JSON and sent to the device.

[0422] For example, the server first analyzes the user's current and past facial photos and determines that "the skin is dry overall, but oily in the T-zone, and light ochre in color." At the same time, it detects from the real-time camera feed that the user's emotional state is "stressed." Based on this analysis, the server suggests moisturizing foundation from a department store brand and skin care products with relaxing effects. It also suggests whitening toners and usage procedures to restore the user's previous healthy skin state. It also selects products such as refreshing eye care, taking the user's emotional state into consideration.

[0423] An example of a prompt to input to a generative AI model is as follows:

[0424] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[0425] In this way, users can receive optimal beauty recommendations based on their skin condition and emotional state.

[0426] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0427] Step 1:

[0428] A user installs a dedicated application on their smartphone, creates an account, and logs in. This registers the user information in the application and completes the initial settings for subsequent processing. The input is the user's basic information, and the output is a message that the account was successfully created.

[0429] Step 2:

[0430] The user takes a photo of their current face and uploads a previous healthy face photo to the application. This prepares the necessary image data. The input is the current face photo and the previous face photo, and the output is a message confirming the upload of these image data. The process is completed when the user selects the previous face photo.

[0431] Step 3:

[0432] The device (smartphone) sends the current facial photo, past facial photos, and real-time camera feed to the server. This allows the server to receive the data necessary for analysis. The input is the current facial photo, past facial photos, and camera feed, and the output is a confirmation notification that the data has been sent to the server. The device converts the image data into the required format and sends it.

[0433] Step 4:

[0434] The server analyzes the received current and past facial photos. First, it uses image recognition to extract the user's facial features, and then it determines their skin color and condition. The input is the current and past facial photos, and the output is the analysis results of their skin color and condition. Specifically, it uses libraries such as OpenCV to analyze the pixel information of the images.

[0435] Step 5:

[0436] The server analyzes the real-time camera feed to recognize the user's emotional state. This analysis uses an emotion recognition algorithm to identify emotional states such as stress or relaxation. The input is the real-time camera feed, and the output is the recognition result of the user's emotional state. This is completed by running the emotion recognition algorithm to analyze facial expressions.

[0437] Step 6:

[0438] The server searches a database for optimal makeup products, skin care products, and cosmetic treatments based on the analysis results of skin color and skin condition, as well as the recognition results of emotional state. Based on the search results, it generates recommendations. The input is the analysis results and recognition results, and the output is the recommendations. The server completes the process by executing a database query to select the appropriate products and treatments.

[0439] Step 7:

[0440] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device. This delivers the proposal to the user. The input is the proposal, and the output is the result of sending the formatted proposal. The server converts the data format and completes the process by sending a response to the device.

[0441] Step 8:

[0442] The terminal analyzes the received proposal content and displays it to the user in a visually easy-to-understand format, allowing the user to confirm the proposal content. The input is the proposal content sent from the server, and the output is the proposal content displayed to the user. The terminal analyzes the received data and completes the process by displaying it on the screen.

[0443] Step 9:

[0444] The user reviews the makeup, skincare, and cosmetic treatment suggestions displayed in the application and then purchases the product or makes an appointment at a beauty salon or clinic based on the suggestions. This results in optimal beauty recommendations. The input is the displayed suggestions, and the output is the user's action (purchase or appointment). The process is completed when the user makes a selection and performs the specific action.

[0445] In this way, users can receive optimal beauty suggestions based on their skin condition and emotional state. An example of a prompt is as follows:

[0446] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0450] [Second embodiment]

[0451] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0452] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0453] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0455] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0457] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0458] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0459] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0462] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0463] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in skin condition, and provides specific beauty recommendations based on the results.

[0464] Program processing overview

[0465] This system operates roughly as follows:

[0466] 1. User Actions

[0467] Users install a dedicated app on their device, create an account and log in. They upload a current photo of their face and a previous photo of their healthy face.

[0468] 2. Terminal Processing

[0469] The device sends the current and past facial photos uploaded by the user to the server. The device then sends the image data to the server in an appropriate format and waits for a response.

[0470] 3. Server Processing

[0471] The server analyzes the received images to identify the user's skin tone and condition, then searches a database to generate optimal recommendations for makeup, skincare products, and cosmetic treatments based on the analysis results.

[0472] Learn more about natural language processing

[0473] User Actions

[0474] 1. The user launches the app, creates an account, and logs in.

[0475] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0476] 3. Next, select the "Upload a Previous Photo" option and upload a photo of your face from a time when you were healthy to the app.

[0477] Terminal handling

[0478] 1. The device sends the uploaded current and past facial photos to a server via the Internet.

[0479] 2. The device receives a response from the server and waits for the analysis results and proposals.

[0480] Server Processing

[0481] 1. The server analyzes the received image and identifies the user's skin color, dryness or oiliness, distribution of blemishes and wrinkles, etc.

[0482] 2. Based on the analysis results, the server selects makeup products (foundation, eye shadow, blush, lipstick, etc.) from a database.

[0483] 3. The server then searches for skin care products to improve the user's current skin condition and generates recommendations.

[0484] 4. The server compares the past and current images and suggests cosmetic medical treatments the user needs to restore their previous healthy skin condition.

[0485] Specific examples

[0486] 1. The user uploads a photo of their current face and also uploads a photo of their face from a few years ago with healthy skin.

[0487] 2. The device sends these photos to the server.

[0488] 3. The server analyzes the received image and determines that "the skin is dry overall, but oily in the T-zone. The skin color is light ochre."

[0489] 4. Depending on your skin condition, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0490] 5. Select skin care products such as whitening toner and moisturizing cream, and provide instructions for their use.

[0491] 6. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing blemishes and hyaluronic acid injections for skin lifting.

[0492] In this way, users can easily obtain specific measures to improve their skin condition.

[0493] The processing flow will be explained below.

[0494] Program processing flow

[0495] User Actions

[0496] Step 1:

[0497] Users install the dedicated app on their device, create an account, and log in.

[0498] Step 2:

[0499] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[0500] Step 3:

[0501] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[0502] Terminal handling

[0503] Step 1:

[0504] The device sends the uploaded current and past facial photos to the server.

[0505] Step 2:

[0506] The device continues to maintain the network connection while waiting for a response from the server.

[0507] Server Processing

[0508] Step 1:

[0509] The server inputs the current and past facial photos received from the device into an algorithm that analyzes them.

[0510] Step 2:

[0511] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0512] Step 3:

[0513] Based on the analysis results, databases of makeup products, skin care products, and cosmetic medical treatments are queried.

[0514] Step 4:

[0515] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition.

[0516] Step 5:

[0517] The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous condition, and also provides instructions for use.

[0518] Step 6:

[0519] The server will suggest the most appropriate cosmetic medical treatment (e.g., laser treatment for removing spots, hyaluronic acid injections) for areas of the skin that need improvement.

[0520] Step 7:

[0521] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[0522] Terminal handling (again)

[0523] Step 1:

[0524] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0525] Step 2:

[0526] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0527] User Action (again)

[0528] Step 1:

[0529] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0530] Step 2:

[0531] Users consider whether the proposed products and treatments are suitable for them.

[0532] Step 3:

[0533] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[0534] In this way, specific actions are performed for each processing step, allowing the user to easily make optimal choices regarding their own beauty.

[0535] Example 1

[0536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0537] Conventional beauty recommendation systems lack detailed suggestions based on the user's specific skin condition or past skin condition, making it difficult for users to efficiently find products and cosmetic treatments that are suitable for their skin condition. In addition, since they do not provide specific advice on how to return to a previous healthy skin condition, user satisfaction can be low.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0539] In this invention, the server includes an image recognition unit, a unit for receiving current and past images uploaded by the user, and a unit for analyzing the user's skin color and condition. This allows for detailed recommendations for makeup products, skin care products, and cosmetic treatments based on the analysis results. Furthermore, by comparing past and current images and making specific recommendations to bring the user's skin closer to its previous condition, user satisfaction is improved.

[0540] "Image recognition means" refers to a device or software that has the function of analyzing input image data and identifying and extracting specific features or patterns.

[0541] "Means for receiving current and past images uploaded by users" refers to devices or software that have the function of receiving image data sent by users via the Internet or an application.

[0542] "Means for analyzing the user's skin color and skin condition" refers to a device or software that has the function of identifying the user's skin condition, such as skin color, dryness, oiliness, and distribution of spots and wrinkles, based on the received image data.

[0543] The "means for suggesting makeup products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable makeup products to the user based on the analysis results of the skin condition.

[0544] The "means for suggesting skin care products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable skin care products to the user based on the analysis results of the skin condition.

[0545] "Means for proposing cosmetic medical treatments based on analysis results" refers to devices or software that have the function of selecting and proposing cosmetic medical treatments or procedures suitable for a user based on the analysis results of the user's skin condition.

[0546] The "means for comparing a past image with a current image" refers to a device or software that has the function of comparing a past image provided by a user with a current image and identifying points of change.

[0547] The "means for making suggestions to bring the skin closer to its previous state" refers to a device or software that has the function of generating suggestions for makeup products, skin care products, and cosmetic medical treatments necessary to restore the skin to its previous healthy state.

[0548] The "means for transmitting the analysis results and proposal contents to the user's terminal" refers to a device or software that has the function of transmitting the analysis results and proposal contents to the user's terminal via the Internet and displaying them.

[0549] "Skin care products" refers to a broad range of products including cosmetics, moisturizing creams, whitening toners, etc. that are used for the purpose of caring for and maintaining healthy skin.

[0550] "Cosmetic medicine" refers to treatments and procedures that use medical technology to pursue and improve beauty, such as laser treatments, hyaluronic acid injections, and skin lifting.

[0551] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in the skin condition, and provides specific beauty suggestions based on the results.

[0552] Hardware and software used

[0553] To implement this system, a user device, a server, and image recognition software are required. Specifically, it uses a user device such as a smartphone or tablet, a cloud server, and image analysis software (e.g., OpenCV, TensorFlow).

[0554] User Actions

[0555] Users install a dedicated app on their device. This app has functions such as uploading images, displaying analysis results, and receiving suggestions. Users launch the app, create an account, and log in. After logging in, users select the "upload photos" option and upload a current photo of their face and a previous photo of their healthy face to the app.

[0556] Terminal handling

[0557] The device temporarily stores the uploaded current and past facial photos in its memory. The device then converts these image data into an appropriate format (e.g., JPEG, PNG) and sends them to a server via the Internet. The device then waits for a response from the server. Through this series of operations, users can easily analyze their skin condition and receive recommendations.

[0558] Server Processing

[0559] The server stores the received image data in a dedicated directory. It then uses image recognition software to analyze the image data and identify the user's skin condition (skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.). The analysis results are stored in a database. Based on the analysis results, the server then searches the database to generate recommendations for makeup products, skin care products, and cosmetic treatments that are optimal for the user. For example, if the server determines that the user's skin condition is "dry overall, but oily in the T-zone, and light ochre skin color," it can recommend products accordingly.

[0560] Specific examples

[0561] 1. Users upload a current photo of their face, as well as a past photo of their face when they were healthy.

[0562] 2. The device sends these photos to the server.

[0563] 3. Based on the analysis results, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0564] 4. As skin care products, select toners and moisturizing creams with whitening effects and provide instructions for their use.

[0565] 5. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing age spots and hyaluronic acid injections for skin lifting.

[0566] Prompt Sentence Examples

[0567] "Please upload a current photo of your face and a photo from a previous time. We will analyze your skin condition and suggest the best makeup products, skincare products, and cosmetic treatments for you."

[0568] In this way, users can easily obtain specific measures to improve their skin condition, and this system effectively resolves users' beauty concerns and problems.

[0569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0570] Step 1: User Actions

[0571] Input: The user installs the app, creates an account, and logs in. Within the app, they select "Upload Photos" and upload their current and past facial photos.

[0572] Data processing: The app temporarily stores photos uploaded by users in the device's memory.

[0573] Output: Saved current and past face photos.

[0574] Users install the app on their smartphone or tablet, create an account, and log in. They select the "upload photo" option within the app, take or select a photo of their current face, and then upload a photo of their face from a previous period when they looked healthy.

[0575] Step 2: Send an image from your device

[0576] Input: Your saved current and past face photos.

[0577] Data processing: The device converts image data into an appropriate format (e.g., JPEG, PNG) and optimizes the data size.

[0578] Output: Image data for transmission.

[0579] The device temporarily stores the current and past facial photos uploaded by the user in memory, then converts the image data into an appropriate format and sends it to a server over the Internet.

[0580] Step 3: Server Image Analysis

[0581] Input: Current and past face photos submitted.

[0582] Data processing: The server uses image recognition software (e.g., OpenCV, TensorFlow) to analyze the user's skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.

[0583] Output: Analysis results (e.g. skin tone is light ochre, dryness is high, oiliness is medium, number of blemishes is 12, etc.).

[0584] The server stores the received image data in a dedicated directory and uses image recognition software to analyze the user's skin condition. The analysis results are stored in a database as numerical and text data.

[0585] Step 4: Generate a Server Proposal

[0586] Input: Analysis result data.

[0587] Data processing: The server searches the database and generates optimal makeup, skincare, and cosmetic medical treatment recommendations based on the analysis results.

[0588] Output: Suggestions (e.g. moisturizing foundation, matte finish eyeshadow, whitening toner, hyaluronic acid injections, etc.).

[0589] Based on the analysis results, the server searches the database and makes recommendations for the most suitable makeup products, skin care products, and cosmetic medical treatments for the user. For example, appropriate recommendations are generated based on analysis results such as "overall dry skin, but oily T-zone. Skin tone is light ochre."

[0590] Step 5: Handling the Server Response

[0591] Input: Proposal data.

[0592] Data processing: The server formats the proposal content into a format that can be sent to the user's device.

[0593] Output: Formatted proposal.

[0594] The server then sends the generated recommendations to the user's device, which then displays them on the app's interface, providing the user with easy-to-read information. Users can then review the suggested makeup, skincare, and cosmetic treatments and try them out as needed.

[0595] (Application example 1)

[0596] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] In today's world, many people worry about their skin condition in their daily lives, but often lack the knowledge and information to select appropriate skincare products and cosmetic medical treatments. Time and location constraints also make it difficult to consult with a beauty professional. Conventional methods require in-store consultations, so a quick and easy method is needed.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0599] In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic medical treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the skin closer to the past condition, and means for receiving images uploaded by the user and displaying the analysis results and suggestions, which is an application installed on a smartphone. This allows the user to easily analyze their own skin condition from the comfort of their own home and receive suggestions for appropriate beauty products and services.

[0600] "Image recognition means" is a technology for analyzing images uploaded by users and identifying their contents.

[0601] "Skin color" indicates the color tone of the user's skin, and is information obtained by image analysis.

[0602] "Skin condition" refers to the health and appearance of the user's skin, including dryness, oiliness, blemishes, wrinkles, and the like.

[0603] "User" means a person who uses the System and uploads images to analyze their skin condition and receive suggestions for beauty products and services.

[0604] "Makeup products" refers to cosmetics intended to enhance the user's appearance, including foundation, eye shadow, lip gloss, etc.

[0605] "Skin care products" refers to products that are intended to maintain and improve the health of a user's skin, including toners and moisturizing creams.

[0606] "Cosmetic medicine" refers to medical procedures aimed at improving a user's skin condition, including, for example, laser treatments and hyaluronic acid injections.

[0607] A "past image" is a photograph of the user's face taken in the past, showing a past healthy skin condition.

[0608] The "current image" is the most recent photograph of the user's face taken to show the user's current facial condition.

[0609] The "comparison means" is a technology for analyzing past and current images to identify changes in skin condition.

[0610] The "means of suggestion" refers to technology that provides users with appropriate makeup products, skin care products, and cosmetic medical treatments based on the analysis results.

[0611] A "smartphone" is a multi-functional mobile phone that allows users to install and use applications.

[0612] "Application" refers to a software program that is installed on a smartphone and provides the functionality of the present invention.

[0613] "Means for receiving" refers to the technology used to send and receive images uploaded by users to the server.

[0614] "Display means" refers to technology for visually presenting the analysis results and proposals to users.

[0615] A "generative AI model" is a model that uses artificial intelligence technology to generate prompts and suggestions based on the analysis results.

[0616] A "prompt" is an instruction entered when utilizing a generative AI model, and is a guideline for obtaining specific output results.

[0617] This invention is a system that analyzes a user's skin color and condition and, based on that, suggests optimal makeup products, skin care products, and cosmetic medical treatments. This system includes an application and a server installed on a smartphone, and operates according to the following specific steps.

[0618] First, users log in to the application installed on their smartphone and upload their current and past facial photos. The smartphone then sends the uploaded images to a server via the Internet.

[0619] The server analyzes the received images using image recognition means to identify the user's skin tone and condition (dryness, oiliness, blemishes, wrinkles, etc.) Based on the analysis results, the server searches a database to generate optimal recommendations for makeup products, skin care products, and cosmetic medical treatments.

[0620] The recommendations, along with the analysis results, are sent to the user's smartphone and displayed, allowing them to easily find beauty products and services that are suitable for their skin condition from the comfort of their own home.

[0621] The server then uses a generative AI model based on the analysis results to create a prompt. The prompt is generated as a guideline for obtaining specific output results and is provided to the user. Utilizing the generative AI model enables more accurate suggestions.

[0622] For example, a user can upload a photo of their current face and a photo of their face in a previous healthy state. The server compares these photos and generates an analysis result that indicates "dry skin overall, but oily T-zone." It then makes specific recommendations, such as moisturizing foundation, whitening toner, and hyaluronic acid injections.

[0623] Example prompt sentence:

[0624] Analyze a user-uploaded facial photo. Identify skin tone, dryness, oiliness, blemishes, and wrinkles, and provide specific recommendations for skin care products, makeup products, and cosmetic treatments based on the results. Generate specific output such as "Skin condition analysis: Light ochre, overall dry, T-zone oily, blemish distribution: High."

[0625] In this way, users can easily select the best beauty products and services at home without the need for expert guidance.

[0626] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0627] Step 1:

[0628] The user launches the application installed on their smartphone, creates an account and logs in.

[0629] Input: User account information (ID, password, etc.).

[0630] Data processing / data calculation: Account information is sent to the server for login authentication and verification.

[0631] Output: The home screen is displayed after successful login.

[0632] Step 2:

[0633] The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0634] Input: A current photo of the user.

[0635] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[0636] Output: Current face photo data is stored within the app.

[0637] Step 3:

[0638] Users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[0639] Input: A previous photo of the user's face.

[0640] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[0641] Output: Past facial photo data is stored within the app.

[0642] Step 4:

[0643] The app sends the current and past facial photos uploaded by the user to the server.

[0644] Input: Current face photo, past face photo.

[0645] Data processing / data calculation: Photo data is sent to a server via the Internet.

[0646] Output: The photo data is saved on the server.

[0647] Step 5:

[0648] The server analyzes the received images and identifies the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0649] Input: Current face photo, past face photo.

[0650] Data processing / data calculation: Apply image recognition algorithms to analyze skin condition (e.g., using Python's OpenCV library).

[0651] Output: Analysis results for skin color, dryness, oiliness, spots, wrinkles, etc.

[0652] Step 6:

[0653] Based on the analysis results, the server searches its database to generate optimal suggestions for makeup products, skin care products, and cosmetic medical treatments.

[0654] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[0655] Data Processing / Data Calculation: Performing database queries to find suitable products and services.

[0656] Output: A list of suggested beauty products and services.

[0657] Step 7:

[0658] The suggestions, along with the analysis results, are sent to the smartphone and displayed to the user through the application.

[0659] Input: A list of beauty products and services you would like to suggest.

[0660] Data processing / data calculation: The proposal is formatted and sent to the application.

[0661] Output: The analysis results and suggestions are displayed on the smartphone screen.

[0662] Step 8:

[0663] Based on the analysis results, the server uses a generative AI model to form a prompt sentence.

[0664] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[0665] Data processing / data calculation: Input the analysis results into a generative AI model to generate a prompt sentence (e.g., an AI model such as GPT-3).

[0666] Output: The generated prompt statement.

[0667] Step 9:

[0668] The generated prompt text is presented to the user.

[0669] Input: The generated prompt statement.

[0670] Data processing / data calculation: Formats the prompt statement and sends it to the application.

[0671] Output: The generated prompt text is displayed on the smartphone screen.

[0672] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0673] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition and, based on that, recommend optimal makeup products, skin care products, and cosmetic medical treatments. The system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[0674] Program processing overview

[0675] This system operates roughly as follows:

[0676] User Actions

[0677] Users install the app on their device, create an account, and log in. They upload a current photo of their face and a previous photo of their healthy face. The system can also use the camera feed to recognize the user's real-time emotional state.

[0678] Terminal handling

[0679] The device sends the current and past facial photos uploaded by the user, as well as the real-time camera feed, to the server, which then sends this data in the appropriate format and waits for a response.

[0680] Server Processing

[0681] The server analyzes the received images and real-time camera feed to identify the user's skin tone, skin condition and emotional state, then searches a database to generate optimal recommendations for makeup, skincare products and cosmetic treatments based on the analysis.

[0682] Learn more about natural language processing

[0683] User Actions

[0684] 1. The user launches the app, creates an account, and logs in.

[0685] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0686] 3. Next, users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[0687] 4. The system uses the camera feed to capture the user's face to recognize their real-time emotional state.

[0688] Terminal handling

[0689] 1. The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[0690] 2. The device maintains the network connection while waiting for a response from the server.

[0691] Server Processing

[0692] 1. The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[0693] 2. The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0694] 3. Furthermore, the server uses emotion recognition algorithms to analyze the user's current emotional state.

[0695] 4. Based on the analysis results, query databases of makeup products, skin care products, and cosmetic medical treatments.

[0696] 5. The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit their skin condition and emotional state.

[0697] 6. The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the previous skin condition, and also provides instructions for using them.

[0698] 7. When it comes to cosmetic medicine, the app will suggest the most appropriate treatment (e.g., laser treatment for removing dark spots, hyaluronic acid injections) based on the user's current emotional state.

[0699] 8. The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device.

[0700] Terminal handling (again)

[0701] 1. The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0702] 2. The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0703] User Action (again)

[0704] 1. The user checks the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0705] 2. The user considers whether the proposed products or treatments are suitable for them.

[0706] 3. The user uses the suggestions to purchase a specific product or make an appointment at a beauty salon or clinic.

[0707] Specific examples

[0708] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[0709] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[0710] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[0711] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[0712] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[0713] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0714] The processing flow will be explained below.

[0715] MODE FOR CARRYING OUT THE INVENTION

[0716] User Actions

[0717] Step 1:

[0718] Users install a dedicated app on their device, create an account and log in.

[0719] Step 2:

[0720] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[0721] Step 3:

[0722] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[0723] Step 4:

[0724] The app uses real-time emotion recognition to capture the user's current emotions from the camera feed.

[0725] Terminal handling

[0726] Step 1:

[0727] The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[0728] Step 2:

[0729] The device maintains a network connection while waiting for analysis results and recommendations from the server.

[0730] Server Processing

[0731] Step 1:

[0732] The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[0733] Step 2:

[0734] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0735] Step 3:

[0736] The server uses emotion recognition algorithms to analyze the user's current emotional state (e.g., stress, relaxation, joy, sadness, etc.).

[0737] Step 4:

[0738] Based on the analysis results (skin condition and emotional state), the server queries databases of makeup products, skin care products, and cosmetic medical treatments.

[0739] Step 5:

[0740] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition and emotional state.

[0741] Step 6:

[0742] The server compares the past and current images and selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous healthy state.

[0743] Step 7:

[0744] Taking into account your emotional state, the server will suggest relaxing skin care products (e.g., moisturizing cream containing lavender oil) or cosmetic treatments that promote psychological stress reduction (e.g., relaxation massage).

[0745] Step 8:

[0746] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[0747] Terminal handling (again)

[0748] Step 1:

[0749] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0750] Step 2:

[0751] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0752] User Action (again)

[0753] Step 1:

[0754] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0755] Step 2:

[0756] Users consider whether the proposed products and treatments are suitable for them.

[0757] Step 3:

[0758] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[0759] Specific examples

[0760] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[0761] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[0762] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[0763] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[0764] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[0765] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0766] Example 2

[0767] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Conventional beauty recommendation systems can only make recommendations based on the user's current skin condition, making it difficult to provide comprehensive beauty recommendations that take into account the user's past healthy skin condition and emotional state. Furthermore, since recommendations for skin care products and cosmetic medical treatments do not reflect the user's real-time emotional state, it is not possible to provide optimal recommendations for each individual user.

[0769] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image recognition means, a means for analyzing the user's skin color and skin condition, a means for suggesting makeup products based on the analysis results, a means for suggesting skin care products based on the analysis results, a means for suggesting cosmetic medical treatments based on the analysis results, a means for comparing a past image with a current image, a means for making suggestions to bring the image closer to the past state, a means for recognizing the user's real-time emotional state, and a means for making optimal suggestions according to the user's emotional state based on the analysis results. This makes it possible to provide beauty suggestions that comprehensively consider the user's skin condition and emotional state.

[0770] "Image recognition means" refers to technology or equipment for detecting and analyzing specific features (e.g., face, skin condition, etc.) from an image.

[0771] "Means for analyzing a user's skin color and skin condition" refers to technology or equipment that analyzes a user's skin color, texture, dryness / oiliness, blemishes, wrinkles, etc.

[0772] "Means for suggesting makeup products based on analysis results" refers to technology or equipment that selects appropriate makeup products based on the results of skin analysis and suggests them to users.

[0773] The "means for suggesting skin care products based on the analysis results" refers to a technology or device that uses the skin analysis results to select appropriate skin care products and suggest them to the user.

[0774] "Means for proposing cosmetic medical treatment based on analysis results" refers to technology or devices that suggest cosmetic medical treatment (e.g., laser treatment, injections, etc.) based on the results of skin analysis.

[0775] The "means for comparing a past image with a current image" refers to a technique or device that compares a past image provided by the user with a current image to evaluate changes in skin condition.

[0776] The "means for making suggestions to bring the skin closer to its previous state" refers to a technology or device that suggests skin care methods or cosmetic medical treatments to bring the skin back to its previous healthy state.

[0777] "Means for recognizing a user's real-time emotional state" refers to a technology or device that analyzes and recognizes a user's current emotional state (e.g., stress, relaxation, etc.) in real time.

[0778] "Means for making optimal suggestions based on the user's emotional state based on the analysis results" refers to technology or equipment that selects and suggests appropriate skin care products, cosmetic medical treatments, makeup products, etc. based on the user's emotional state.

[0779] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on that analysis. Specifically, the system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[0780] overview

[0781] This system roughly consists of operations by users who have installed a dedicated app, processing by the device, and processing by the server. The user uploads a photo of their face and performs real-time emotion recognition, and the device sends this data to the server. The server performs image and emotion recognition, generates beauty suggestions based on the analysis results, sends them to the device, and finally makes them available to the user.

[0782] Technology and software used

[0783] Image recognition techniques: Use convolutional neural network (CNN) models such as ResNet and VGG.

[0784] Emotion recognition technology: Use emotion recognition models such as FER+.

[0785] Databases: Use databases related to beauty products and aesthetic medicine (e.g., general beauty product databases).

[0786] Communication technology: Uses network communication protocols such as HTTP and HTTPS.

[0787] The process begins when a user launches the app, creates an account, and logs in. The user uploads a current photo of their face and a previous photo of their healthy face, and the app also uses their camera feed to recognize the user's emotional state in real time.

[0788] The device sends this data (current facial photo, past facial photos, and real-time camera feed) to a server, which then analyzes the data using image and emotion recognition algorithms to identify skin condition and emotional state.

[0789] The server then searches its database to suggest makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss), skin care products (e.g., whitening toner, moisturizing cream), and cosmetic medical treatments (e.g., laser treatment for dark spots, hyaluronic acid injections) based on the analysis results.

[0790] The server converts the generated proposals into JSON format and sends them to the device, which then visually displays the proposals and allows the user to consider specific products and treatments.

[0791] Specific examples

[0792] For example, a user can upload a current photo of their face and a photo of their face from a previous period when they were healthy. At the same time, the real-time camera feed is used to analyze their emotional state. The server determines that their skin is dry overall, but oily in the T-zone, and their skin tone is light ochre, and detects that the user's emotional state is stressed.

[0793] Based on these conditions, the server makes the following suggestions:

[0794] Moisturizing foundation (brand name is common)

[0795] Matte Finish Eyeshadow (same as above)

[0796] Natural lip gloss (same as above)

[0797] Relaxing skin care products (for example, moisturizing cream with lavender oil)

[0798] How to use whitening toner and moisturizing cream

[0799] Laser treatment for removing spots and hyaluronic acid injections for lifting

[0800] Prompt Sentence Examples

[0801] "Design a system that analyzes current and past facial photos and suggests optimal makeup and skin care."

[0802] "Please explain in detail the system that compares your current skin condition with your past skin condition and suggests cosmetic medical treatments."

[0803] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[0804] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0805] Step 1: User installs and launches the app

[0806] The user installs a dedicated app from the App Store or Google Play on their smartphone. Once the installation is complete, they tap the app icon to launch the app. This is the input. The output is the display of the app's launch screen.

[0807] Step 2: User creates account and logs in

[0808] When a user launches the app for the first time, they create an account using their email address or social media account. They are prompted to enter their ID and password. Existing users log in by entering their ID and password. The input is account information and login information, and the output is obtaining access rights to the user's personal page.

[0809] Step 3: User uploads a current photo of their face

[0810] The user selects the "upload photo" option in the app, selects a current face photo, and uploads it. The app uses the smartphone's camera roll and interface. The input is the current face photo, and the output is sending the photo to the server.

[0811] Step 4: User uploads a photo of their face

[0812] The user selects the "Upload a Previous Photo" option and selects and uploads a previous photo of their face with healthy skin. The smartphone's camera roll and interface are used again. The input is the previous photo of the face, and the output is the transmission of the photo to the server.

[0813] Step 5: The device sends the real-time camera feed to the server

[0814] The device activates the device's camera to recognize the user's real-time emotional state, captures the user's face, and sends the camera feed to the server. The input is the real-time camera feed, and the output is the feed data sent to the server.

[0815] Step 6: Server parses images and camera feed

[0816] The server analyzes the received current face photo, past face photos, and real-time camera feed. It processes the data using image recognition algorithms (e.g., ResNet, VGG) and emotion recognition algorithms (e.g., FER+). The input is the received image and camera feed, and the output is the analysis results such as the user's skin color, skin condition, and emotional state.

[0817] Step 7: The server generates beauty suggestions based on the analysis results.

[0818] The server then searches a database based on the analysis results to make beauty recommendations. For example, it selects makeup products, skin care products, and cosmetic treatments based on skin condition. The input is the analysis results of skin color, skin condition, and emotional state, and the output is a set of recommended beauty products and cosmetic treatments.

[0819] Step 8: Send the server-generated proposal to the device

[0820] The server converts the generated beauty suggestions into an appropriate format (e.g., JSON) and sends them to the device via a network communication protocol (e.g., HTTP or HTTPS). The input is a set of beauty suggestions, and the output is JSON-formatted data.

[0821] Step 9: The device displays a visual suggestion

[0822] The device analyzes the beauty suggestions received from the server and displays them visually in the app for the user. The input is JSON format data, and the output is a visual display within the app.

[0823] Step 10: User confirms and uses the proposal

[0824] The user checks the suggestions displayed on the device and clicks on a link to purchase the suggested product or make an appointment at a beauty salon or clinic. The input is the visually displayed suggestion, and the output is the actual purchase or appointment.

[0825] (Application example 2)

[0826] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] Conventional beauty recommendation systems suggest makeup and skincare products based on an analysis of the user's skin tone and condition, but they lacked the ability to consider the user's real-time emotional state. Additionally, while some systems had a function to compare a user's current and past images, they did not suggest cosmetic treatments based on the user's emotional state, making it difficult to provide more personalized recommendations.

[0828] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the image closer to the past state, means for recognizing emotional state in real time, means for suggesting skin care products based on the emotional state, and means for suggesting cosmetic treatments based on the emotional state. This makes it possible to comprehensively analyze the user's skin condition and emotional state and make optimal beauty suggestions based on the current state and emotion.

[0829] "Image recognition means" is a general term for software and hardware that receives images uploaded by users, analyzes them, and recognizes the user's facial features and skin condition.

[0830] The "means for analyzing skin color and skin condition" is a system for analyzing a user's skin color, dryness, oiliness, blemishes, wrinkles, etc. using image recognition technology.

[0831] The "means for suggesting makeup products" refers to algorithms and software that suggest optimal cosmetics to users based on the analysis results.

[0832] The "means for suggesting skin care products" is a system that selects and suggests skin care items suitable for the user's skin condition based on the analysis results.

[0833] The "means for proposing cosmetic medical treatment" is a system for proposing optimal cosmetic medical treatments and procedures based on the user's skin condition and emotional state.

[0834] The "means for comparing images" refers to software and hardware that uses an algorithm to compare past and current images and analyze changes and areas for improvement in skin condition.

[0835] The "means for making suggestions" is a system that suggests to users the makeup products, skin care products, and cosmetic medical treatments they need to bring their appearance closer to their previous state.

[0836] "Means for recognizing emotional state" refers to technology for analyzing a user's emotional state (e.g., stress, relaxation) using real-time camera feeds, etc.

[0837] The "means for suggesting skin care products based on emotional state" is a system for suggesting skin care products suitable for a user in consideration of the analyzed emotional state.

[0838] The "means for suggesting cosmetic medical treatments based on emotional state" is a system for suggesting cosmetic medical treatments or procedures according to the user's emotional state.

[0839] This invention is a system that analyzes a user's skin color and condition and suggests optimal makeup products, skin care products, and cosmetic medical treatments. The system also includes a function to recognize the user's emotional state in real time and make suggestions based on the emotional state.

[0840] The overall structure of the system is as follows: Users install a dedicated application on their smartphone, create an account, and log in. They then take a photo of their current face and upload it through the application. By uploading past facial photos as well, the server can compare and analyze past and current images. The application also uses the smartphone's camera to recognize the user's real-time emotional state.

[0841] The process on the device (user's smartphone) is as follows: The current face photo uploaded by the user, past face photos, and real-time camera feed are sent to the server, and this data is sent to the server in an appropriate format.

[0842] The server includes the following means for analyzing the received data:

[0843] 1. Image recognition means: Analyzes the received current and past facial photos to recognize the user's skin color and condition (dry, oily, spots, wrinkles, etc.).

[0844] 2. Skin color and condition analysis: Analyze the user's skin characteristics based on the results of image recognition. This analysis uses image processing libraries such as OpenCV.

[0845] 3. Real-time emotional state recognition: Using the camera feed, the user's emotional state is analyzed in real time using emotion recognition algorithms.

[0846] 4. Means for making suggestions based on analysis results: Based on the analysis results and emotional state, the most suitable makeup products, skin care products, and cosmetic medical treatments are searched from a database and suggestions are generated.

[0847] 5. Image comparison: Compare past and current images to analyze changes in skin condition and generate suggestions to restore a previous healthy state.

[0848] 6. Means for sending proposals: The generated proposals are converted into an appropriate format such as JSON and sent to the device.

[0849] For example, the server first analyzes the user's current and past facial photos and determines that "the skin is dry overall, but oily in the T-zone, and light ochre in color." At the same time, it detects from the real-time camera feed that the user's emotional state is "stressed." Based on this analysis, the server suggests moisturizing foundation from a department store brand and skin care products with relaxing effects. It also suggests whitening toners and usage procedures to restore the user's previous healthy skin state. It also selects products such as refreshing eye care, taking the user's emotional state into consideration.

[0850] An example of a prompt to input to a generative AI model is as follows:

[0851] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[0852] In this way, users can receive optimal beauty recommendations based on their skin condition and emotional state.

[0853] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0854] Step 1:

[0855] A user installs a dedicated application on their smartphone, creates an account, and logs in. This registers the user information in the application and completes the initial settings for subsequent processing. The input is the user's basic information, and the output is a message that the account was successfully created.

[0856] Step 2:

[0857] The user takes a photo of their current face and uploads a previous healthy face photo to the application. This prepares the necessary image data. The input is the current face photo and the previous face photo, and the output is a message confirming the upload of these image data. The process is completed when the user selects the previous face photo.

[0858] Step 3:

[0859] The device (smartphone) sends the current facial photo, past facial photos, and real-time camera feed to the server. This allows the server to receive the data necessary for analysis. The input is the current facial photo, past facial photos, and camera feed, and the output is a confirmation notification that the data has been sent to the server. The device converts the image data into the required format and sends it.

[0860] Step 4:

[0861] The server analyzes the received current and past facial photos. First, it uses image recognition to extract the user's facial features, and then it determines their skin color and condition. The input is the current and past facial photos, and the output is the analysis results of their skin color and condition. Specifically, it uses libraries such as OpenCV to analyze the pixel information of the images.

[0862] Step 5:

[0863] The server analyzes the real-time camera feed to recognize the user's emotional state. This analysis uses an emotion recognition algorithm to identify emotional states such as stress or relaxation. The input is the real-time camera feed, and the output is the recognition result of the user's emotional state. This is completed by running the emotion recognition algorithm to analyze facial expressions.

[0864] Step 6:

[0865] The server searches a database for optimal makeup products, skin care products, and cosmetic treatments based on the analysis results of skin color and skin condition, as well as the recognition results of emotional state. Based on the search results, it generates recommendations. The input is the analysis results and recognition results, and the output is the recommendations. The server completes the process by executing a database query to select the appropriate products and treatments.

[0866] Step 7:

[0867] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device. This delivers the proposal to the user. The input is the proposal, and the output is the result of sending the formatted proposal. The server converts the data format and completes the process by sending a response to the device.

[0868] Step 8:

[0869] The terminal analyzes the received proposal content and displays it to the user in a visually easy-to-understand format, allowing the user to confirm the proposal content. The input is the proposal content sent from the server, and the output is the proposal content displayed to the user. The terminal analyzes the received data and completes the process by displaying it on the screen.

[0870] Step 9:

[0871] The user reviews the makeup, skincare, and cosmetic treatment suggestions displayed in the application and then purchases the product or makes an appointment at a beauty salon or clinic based on the suggestions. This results in optimal beauty recommendations. The input is the displayed suggestions, and the output is the user's action (purchase or appointment). The process is completed when the user makes a selection and performs the specific action.

[0872] In this way, users can receive optimal beauty suggestions based on their skin condition and emotional state. An example of a prompt is as follows:

[0873] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[0874] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0875] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0876] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0877] [Third embodiment]

[0878] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0879] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0880] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0881] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0882] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0883] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0884] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0885] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0886] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0887] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0888] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0889] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0890] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in skin condition, and provides specific beauty recommendations based on the results.

[0891] Program processing overview

[0892] This system operates roughly as follows:

[0893] 1. User Actions

[0894] Users install a dedicated app on their device, create an account and log in. They upload a current photo of their face and a previous photo of their healthy face.

[0895] 2. Terminal Processing

[0896] The device sends the current and past facial photos uploaded by the user to the server. The device then sends the image data to the server in an appropriate format and waits for a response.

[0897] 3. Server Processing

[0898] The server analyzes the received images to identify the user's skin tone and condition, then searches a database to generate optimal recommendations for makeup, skincare products, and cosmetic treatments based on the analysis results.

[0899] Learn more about natural language processing

[0900] User Actions

[0901] 1. The user launches the app, creates an account, and logs in.

[0902] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[0903] 3. Next, select the "Upload a Previous Photo" option and upload a photo of your face from a time when you were healthy to the app.

[0904] Terminal handling

[0905] 1. The device sends the uploaded current and past facial photos to a server via the Internet.

[0906] 2. The device receives a response from the server and waits for the analysis results and proposals.

[0907] Server Processing

[0908] 1. The server analyzes the received image and identifies the user's skin color, dryness or oiliness, distribution of blemishes and wrinkles, etc.

[0909] 2. Based on the analysis results, the server selects makeup products (foundation, eye shadow, blush, lipstick, etc.) from a database.

[0910] 3. The server then searches for skin care products to improve the user's current skin condition and generates recommendations.

[0911] 4. The server compares the past and current images and suggests cosmetic medical treatments the user needs to restore their previous healthy skin condition.

[0912] Specific examples

[0913] 1. The user uploads a photo of their current face and also uploads a photo of their face from a few years ago with healthy skin.

[0914] 2. The device sends these photos to the server.

[0915] 3. The server analyzes the received image and determines that "the skin is dry overall, but oily in the T-zone. The skin color is light ochre."

[0916] 4. Depending on your skin condition, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0917] 5. Select skin care products such as whitening toner and moisturizing cream, and provide instructions for their use.

[0918] 6. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing blemishes and hyaluronic acid injections for skin lifting.

[0919] In this way, users can easily obtain specific measures to improve their skin condition.

[0920] The processing flow will be explained below.

[0921] Program processing flow

[0922] User Actions

[0923] Step 1:

[0924] Users install the dedicated app on their device, create an account, and log in.

[0925] Step 2:

[0926] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[0927] Step 3:

[0928] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[0929] Terminal handling

[0930] Step 1:

[0931] The device sends the uploaded current and past facial photos to the server.

[0932] Step 2:

[0933] The device continues to maintain the network connection while waiting for a response from the server.

[0934] Server Processing

[0935] Step 1:

[0936] The server inputs the current and past facial photos received from the device into an algorithm that analyzes them.

[0937] Step 2:

[0938] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[0939] Step 3:

[0940] Based on the analysis results, databases of makeup products, skin care products, and cosmetic medical treatments are queried.

[0941] Step 4:

[0942] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition.

[0943] Step 5:

[0944] The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous condition, and also provides instructions for use.

[0945] Step 6:

[0946] The server will suggest the most appropriate cosmetic medical treatment (e.g., laser treatment for removing spots, hyaluronic acid injections) for areas of the skin that need improvement.

[0947] Step 7:

[0948] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[0949] Terminal handling (again)

[0950] Step 1:

[0951] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[0952] Step 2:

[0953] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[0954] User Action (again)

[0955] Step 1:

[0956] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[0957] Step 2:

[0958] Users consider whether the proposed products and treatments are suitable for them.

[0959] Step 3:

[0960] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[0961] In this way, specific actions are performed for each processing step, allowing the user to easily make optimal choices regarding their own beauty.

[0962] Example 1

[0963] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Conventional beauty recommendation systems lack detailed suggestions based on the user's specific skin condition or past skin condition, making it difficult for users to efficiently find products and cosmetic treatments that are suitable for their skin condition. In addition, since they do not provide specific advice on how to return to a previous healthy skin condition, user satisfaction can be low.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0966] In this invention, the server includes an image recognition unit, a unit for receiving current and past images uploaded by the user, and a unit for analyzing the user's skin color and condition. This allows for detailed recommendations for makeup products, skin care products, and cosmetic treatments based on the analysis results. Furthermore, by comparing past and current images and making specific recommendations to bring the user's skin closer to its previous condition, user satisfaction is improved.

[0967] "Image recognition means" refers to a device or software that has the function of analyzing input image data and identifying and extracting specific features or patterns.

[0968] "Means for receiving current and past images uploaded by users" refers to devices or software that have the function of receiving image data sent by users via the Internet or an application.

[0969] "Means for analyzing the user's skin color and skin condition" refers to a device or software that has the function of identifying the user's skin condition, such as skin color, dryness, oiliness, and distribution of spots and wrinkles, based on the received image data.

[0970] The "means for suggesting makeup products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable makeup products to the user based on the analysis results of the skin condition.

[0971] The "means for suggesting skin care products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable skin care products to the user based on the analysis results of the skin condition.

[0972] "Means for proposing cosmetic medical treatments based on analysis results" refers to devices or software that have the function of selecting and proposing cosmetic medical treatments or procedures suitable for a user based on the analysis results of the user's skin condition.

[0973] The "means for comparing a past image with a current image" refers to a device or software that has the function of comparing a past image provided by a user with a current image and identifying points of change.

[0974] The "means for making suggestions to bring the skin closer to its previous state" refers to a device or software that has the function of generating suggestions for makeup products, skin care products, and cosmetic medical treatments necessary to restore the skin to its previous healthy state.

[0975] The "means for transmitting the analysis results and proposal contents to the user's terminal" refers to a device or software that has the function of transmitting the analysis results and proposal contents to the user's terminal via the Internet and displaying them.

[0976] "Skin care products" refers to a broad range of products including cosmetics, moisturizing creams, whitening toners, etc. that are used for the purpose of caring for and maintaining healthy skin.

[0977] "Cosmetic medicine" refers to treatments and procedures that use medical technology to pursue and improve beauty, such as laser treatments, hyaluronic acid injections, and skin lifting.

[0978] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in the skin condition, and provides specific beauty suggestions based on the results.

[0979] Hardware and software used

[0980] To implement this system, a user device, a server, and image recognition software are required. Specifically, it uses a user device such as a smartphone or tablet, a cloud server, and image analysis software (e.g., OpenCV, TensorFlow).

[0981] User Actions

[0982] Users install a dedicated app on their device. This app has functions such as uploading images, displaying analysis results, and receiving suggestions. Users launch the app, create an account, and log in. After logging in, users select the "upload photos" option and upload a current photo of their face and a previous photo of their healthy face to the app.

[0983] Terminal handling

[0984] The device temporarily stores the uploaded current and past facial photos in its memory. The device then converts these image data into an appropriate format (e.g., JPEG, PNG) and sends them to a server via the Internet. The device then waits for a response from the server. Through this series of operations, users can easily analyze their skin condition and receive recommendations.

[0985] Server Processing

[0986] The server stores the received image data in a dedicated directory. It then uses image recognition software to analyze the image data and identify the user's skin condition (skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.). The analysis results are stored in a database. Based on the analysis results, the server then searches the database to generate recommendations for makeup products, skin care products, and cosmetic treatments that are optimal for the user. For example, if the server determines that the user's skin condition is "dry overall, but oily in the T-zone, and light ochre skin color," it can recommend products accordingly.

[0987] Specific examples

[0988] 1. Users upload a current photo of their face, as well as a past photo of their face when they were healthy.

[0989] 2. The device sends these photos to the server.

[0990] 3. Based on the analysis results, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[0991] 4. As skin care products, select toners and moisturizing creams with whitening effects and provide instructions for their use.

[0992] 5. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing age spots and hyaluronic acid injections for skin lifting.

[0993] Prompt Sentence Examples

[0994] "Please upload a current photo of your face and a photo from a previous time. We will analyze your skin condition and suggest the best makeup products, skincare products, and cosmetic treatments for you."

[0995] In this way, users can easily obtain specific measures to improve their skin condition, and this system effectively resolves users' beauty concerns and problems.

[0996] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0997] Step 1: User Actions

[0998] Input: The user installs the app, creates an account, and logs in. Within the app, they select "Upload Photos" and upload their current and past facial photos.

[0999] Data processing: The app temporarily stores photos uploaded by users in the device's memory.

[1000] Output: Saved current and past face photos.

[1001] Users install the app on their smartphone or tablet, create an account, and log in. They select the "upload photo" option within the app, take or select a photo of their current face, and then upload a photo of their face from a previous period when they looked healthy.

[1002] Step 2: Send an image from your device

[1003] Input: Your saved current and past face photos.

[1004] Data processing: The device converts image data into an appropriate format (e.g., JPEG, PNG) and optimizes the data size.

[1005] Output: Image data for transmission.

[1006] The device temporarily stores the current and past facial photos uploaded by the user in memory, then converts the image data into an appropriate format and sends it to a server over the Internet.

[1007] Step 3: Server Image Analysis

[1008] Input: Current and past face photos submitted.

[1009] Data processing: The server uses image recognition software (e.g., OpenCV, TensorFlow) to analyze the user's skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.

[1010] Output: Analysis results (e.g. skin tone is light ochre, dryness is high, oiliness is medium, number of blemishes is 12, etc.).

[1011] The server stores the received image data in a dedicated directory and uses image recognition software to analyze the user's skin condition. The analysis results are stored in a database as numerical and text data.

[1012] Step 4: Generate a Server Proposal

[1013] Input: Analysis result data.

[1014] Data processing: The server searches the database and generates optimal makeup, skincare, and cosmetic medical treatment recommendations based on the analysis results.

[1015] Output: Suggestions (e.g. moisturizing foundation, matte finish eyeshadow, whitening toner, hyaluronic acid injections, etc.).

[1016] Based on the analysis results, the server searches the database and makes recommendations for the most suitable makeup products, skin care products, and cosmetic medical treatments for the user. For example, appropriate recommendations are generated based on analysis results such as "overall dry skin, but oily T-zone. Skin tone is light ochre."

[1017] Step 5: Handling the Server Response

[1018] Input: Proposal data.

[1019] Data processing: The server formats the proposal content into a format that can be sent to the user's device.

[1020] Output: Formatted proposal.

[1021] The server then sends the generated recommendations to the user's device, which then displays them on the app's interface, providing the user with easy-to-read information. Users can then review the suggested makeup, skincare, and cosmetic treatments and try them out as needed.

[1022] (Application example 1)

[1023] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] In today's world, many people worry about their skin condition in their daily lives, but often lack the knowledge and information to select appropriate skincare products and cosmetic medical treatments. Time and location constraints also make it difficult to consult with a beauty professional. Conventional methods require in-store consultations, so a quick and easy method is needed.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1026] In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic medical treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the skin closer to the past condition, and means for receiving images uploaded by the user and displaying the analysis results and suggestions, which is an application installed on a smartphone. This allows the user to easily analyze their own skin condition from the comfort of their own home and receive suggestions for appropriate beauty products and services.

[1027] "Image recognition means" is a technology for analyzing images uploaded by users and identifying their contents.

[1028] "Skin color" indicates the color tone of the user's skin, and is information obtained by image analysis.

[1029] "Skin condition" refers to the health and appearance of the user's skin, including dryness, oiliness, blemishes, wrinkles, and the like.

[1030] "User" means a person who uses the System and uploads images to analyze their skin condition and receive suggestions for beauty products and services.

[1031] "Makeup products" refers to cosmetics intended to enhance the user's appearance, including foundation, eye shadow, lip gloss, etc.

[1032] "Skin care products" refers to products that are intended to maintain and improve the health of a user's skin, including toners and moisturizing creams.

[1033] "Cosmetic medicine" refers to medical procedures aimed at improving a user's skin condition, including, for example, laser treatments and hyaluronic acid injections.

[1034] A "past image" is a photograph of the user's face taken in the past, showing a past healthy skin condition.

[1035] The "current image" is the most recent photograph of the user's face taken to show the user's current facial condition.

[1036] The "comparison means" is a technology for analyzing past and current images to identify changes in skin condition.

[1037] The "means of suggestion" refers to technology that provides users with appropriate makeup products, skin care products, and cosmetic medical treatments based on the analysis results.

[1038] A "smartphone" is a multi-functional mobile phone that allows users to install and use applications.

[1039] "Application" refers to a software program that is installed on a smartphone and provides the functionality of the present invention.

[1040] "Means for receiving" refers to the technology used to send and receive images uploaded by users to the server.

[1041] "Display means" refers to technology for visually presenting the analysis results and proposals to users.

[1042] A "generative AI model" is a model that uses artificial intelligence technology to generate prompts and suggestions based on the analysis results.

[1043] A "prompt" is an instruction entered when utilizing a generative AI model, and is a guideline for obtaining specific output results.

[1044] This invention is a system that analyzes a user's skin color and condition and, based on that, suggests optimal makeup products, skin care products, and cosmetic medical treatments. This system includes an application and a server installed on a smartphone, and operates according to the following specific steps.

[1045] First, users log in to the application installed on their smartphone and upload their current and past facial photos. The smartphone then sends the uploaded images to a server via the Internet.

[1046] The server analyzes the received images using image recognition means to identify the user's skin tone and condition (dryness, oiliness, blemishes, wrinkles, etc.) Based on the analysis results, the server searches a database to generate optimal recommendations for makeup products, skin care products, and cosmetic medical treatments.

[1047] The recommendations, along with the analysis results, are sent to the user's smartphone and displayed, allowing them to easily find beauty products and services that are suitable for their skin condition from the comfort of their own home.

[1048] The server then uses a generative AI model based on the analysis results to create a prompt. The prompt is generated as a guideline for obtaining specific output results and is provided to the user. Utilizing the generative AI model enables more accurate suggestions.

[1049] For example, a user can upload a photo of their current face and a photo of their face in a previous healthy state. The server compares these photos and generates an analysis result that indicates "dry skin overall, but oily T-zone." It then makes specific recommendations, such as moisturizing foundation, whitening toner, and hyaluronic acid injections.

[1050] Example prompt sentence:

[1051] Analyze a user-uploaded facial photo. Identify skin tone, dryness, oiliness, blemishes, and wrinkles, and provide specific recommendations for skin care products, makeup products, and cosmetic treatments based on the results. Generate specific output such as "Skin condition analysis: Light ochre, overall dry, T-zone oily, blemish distribution: High."

[1052] In this way, users can easily select the best beauty products and services at home without the need for expert guidance.

[1053] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1054] Step 1:

[1055] The user launches the application installed on their smartphone, creates an account and logs in.

[1056] Input: User account information (ID, password, etc.).

[1057] Data processing / data calculation: Account information is sent to the server for login authentication and verification.

[1058] Output: The home screen is displayed after successful login.

[1059] Step 2:

[1060] The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[1061] Input: A current photo of the user.

[1062] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[1063] Output: Current face photo data is stored within the app.

[1064] Step 3:

[1065] Users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[1066] Input: A previous photo of the user's face.

[1067] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[1068] Output: Past facial photo data is stored within the app.

[1069] Step 4:

[1070] The app sends the current and past facial photos uploaded by the user to the server.

[1071] Input: Current face photo, past face photo.

[1072] Data processing / data calculation: Photo data is sent to a server via the Internet.

[1073] Output: The photo data is saved on the server.

[1074] Step 5:

[1075] The server analyzes the received images and identifies the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1076] Input: Current face photo, past face photo.

[1077] Data processing / data calculation: Apply image recognition algorithms to analyze skin condition (e.g., using Python's OpenCV library).

[1078] Output: Analysis results for skin color, dryness, oiliness, spots, wrinkles, etc.

[1079] Step 6:

[1080] Based on the analysis results, the server searches its database to generate optimal suggestions for makeup products, skin care products, and cosmetic medical treatments.

[1081] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[1082] Data Processing / Data Calculation: Performing database queries to find suitable products and services.

[1083] Output: A list of suggested beauty products and services.

[1084] Step 7:

[1085] The suggestions, along with the analysis results, are sent to the smartphone and displayed to the user through the application.

[1086] Input: A list of beauty products and services you would like to suggest.

[1087] Data processing / data calculation: The proposal is formatted and sent to the application.

[1088] Output: The analysis results and suggestions are displayed on the smartphone screen.

[1089] Step 8:

[1090] Based on the analysis results, the server uses a generative AI model to form a prompt sentence.

[1091] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[1092] Data processing / data calculation: Input the analysis results into a generative AI model to generate a prompt sentence (e.g., an AI model such as GPT-3).

[1093] Output: The generated prompt statement.

[1094] Step 9:

[1095] The generated prompt text is presented to the user.

[1096] Input: The generated prompt statement.

[1097] Data processing / data calculation: Formats the prompt statement and sends it to the application.

[1098] Output: The generated prompt text is displayed on the smartphone screen.

[1099] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1100] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition and, based on that, recommend optimal makeup products, skin care products, and cosmetic medical treatments. The system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[1101] Program processing overview

[1102] This system operates roughly as follows:

[1103] User Actions

[1104] Users install the app on their device, create an account, and log in. They upload a current photo of their face and a previous photo of their healthy face. The system can also use the camera feed to recognize the user's real-time emotional state.

[1105] Terminal handling

[1106] The device sends the current and past facial photos uploaded by the user, as well as the real-time camera feed, to the server, which then sends this data in the appropriate format and waits for a response.

[1107] Server Processing

[1108] The server analyzes the received images and real-time camera feed to identify the user's skin tone, skin condition and emotional state, then searches a database to generate optimal recommendations for makeup, skincare products and cosmetic treatments based on the analysis.

[1109] Learn more about natural language processing

[1110] User Actions

[1111] 1. The user launches the app, creates an account, and logs in.

[1112] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[1113] 3. Next, users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[1114] 4. The system uses the camera feed to capture the user's face to recognize their real-time emotional state.

[1115] Terminal handling

[1116] 1. The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[1117] 2. The device maintains the network connection while waiting for a response from the server.

[1118] Server Processing

[1119] 1. The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[1120] 2. The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1121] 3. Furthermore, the server uses emotion recognition algorithms to analyze the user's current emotional state.

[1122] 4. Based on the analysis results, query databases of makeup products, skin care products, and cosmetic medical treatments.

[1123] 5. The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit their skin condition and emotional state.

[1124] 6. The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the previous skin condition, and also provides instructions for using them.

[1125] 7. When it comes to cosmetic medicine, the app will suggest the most appropriate treatment (e.g., laser treatment for removing dark spots, hyaluronic acid injections) based on the user's current emotional state.

[1126] 8. The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device.

[1127] Terminal handling (again)

[1128] 1. The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[1129] 2. The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[1130] User Action (again)

[1131] 1. The user checks the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[1132] 2. The user considers whether the proposed products or treatments are suitable for them.

[1133] 3. The user uses the suggestions to purchase a specific product or make an appointment at a beauty salon or clinic.

[1134] Specific examples

[1135] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[1136] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[1137] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[1138] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[1139] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[1140] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1141] The processing flow will be explained below.

[1142] MODE FOR CARRYING OUT THE INVENTION

[1143] User Actions

[1144] Step 1:

[1145] Users install a dedicated app on their device, create an account and log in.

[1146] Step 2:

[1147] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[1148] Step 3:

[1149] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[1150] Step 4:

[1151] The app uses real-time emotion recognition to capture the user's current emotions from the camera feed.

[1152] Terminal handling

[1153] Step 1:

[1154] The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[1155] Step 2:

[1156] The device maintains a network connection while waiting for analysis results and recommendations from the server.

[1157] Server Processing

[1158] Step 1:

[1159] The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[1160] Step 2:

[1161] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1162] Step 3:

[1163] The server uses emotion recognition algorithms to analyze the user's current emotional state (e.g., stress, relaxation, joy, sadness, etc.).

[1164] Step 4:

[1165] Based on the analysis results (skin condition and emotional state), the server queries databases of makeup products, skin care products, and cosmetic medical treatments.

[1166] Step 5:

[1167] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition and emotional state.

[1168] Step 6:

[1169] The server compares the past and current images and selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous healthy state.

[1170] Step 7:

[1171] Taking into account your emotional state, the server will suggest relaxing skin care products (e.g., moisturizing cream containing lavender oil) or cosmetic treatments that promote psychological stress reduction (e.g., relaxation massage).

[1172] Step 8:

[1173] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[1174] Terminal handling (again)

[1175] Step 1:

[1176] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[1177] Step 2:

[1178] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[1179] User Action (again)

[1180] Step 1:

[1181] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[1182] Step 2:

[1183] Users consider whether the proposed products and treatments are suitable for them.

[1184] Step 3:

[1185] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[1186] Specific examples

[1187] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[1188] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[1189] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[1190] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[1191] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[1192] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1193] Example 2

[1194] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] Conventional beauty recommendation systems can only make recommendations based on the user's current skin condition, making it difficult to provide comprehensive beauty recommendations that take into account the user's past healthy skin condition and emotional state. Furthermore, since recommendations for skin care products and cosmetic medical treatments do not reflect the user's real-time emotional state, it is not possible to provide optimal recommendations for each individual user.

[1196] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image recognition means, a means for analyzing the user's skin color and skin condition, a means for suggesting makeup products based on the analysis results, a means for suggesting skin care products based on the analysis results, a means for suggesting cosmetic medical treatments based on the analysis results, a means for comparing a past image with a current image, a means for making suggestions to bring the image closer to the past state, a means for recognizing the user's real-time emotional state, and a means for making optimal suggestions according to the user's emotional state based on the analysis results. This makes it possible to provide beauty suggestions that comprehensively consider the user's skin condition and emotional state.

[1197] "Image recognition means" refers to technology or equipment for detecting and analyzing specific features (e.g., face, skin condition, etc.) from an image.

[1198] "Means for analyzing a user's skin color and skin condition" refers to technology or equipment that analyzes a user's skin color, texture, dryness / oiliness, blemishes, wrinkles, etc.

[1199] "Means for suggesting makeup products based on analysis results" refers to technology or equipment that selects appropriate makeup products based on the results of skin analysis and suggests them to users.

[1200] The "means for suggesting skin care products based on the analysis results" refers to a technology or device that uses the skin analysis results to select appropriate skin care products and suggest them to the user.

[1201] "Means for proposing cosmetic medical treatment based on analysis results" refers to technology or devices that suggest cosmetic medical treatment (e.g., laser treatment, injections, etc.) based on the results of skin analysis.

[1202] The "means for comparing a past image with a current image" refers to a technique or device that compares a past image provided by the user with a current image to evaluate changes in skin condition.

[1203] The "means for making suggestions to bring the skin closer to its previous state" refers to a technology or device that suggests skin care methods or cosmetic medical treatments to bring the skin back to its previous healthy state.

[1204] "Means for recognizing a user's real-time emotional state" refers to a technology or device that analyzes and recognizes a user's current emotional state (e.g., stress, relaxation, etc.) in real time.

[1205] "Means for making optimal suggestions based on the user's emotional state based on the analysis results" refers to technology or equipment that selects and suggests appropriate skin care products, cosmetic medical treatments, makeup products, etc. based on the user's emotional state.

[1206] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on that analysis. Specifically, the system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[1207] overview

[1208] This system roughly consists of operations by users who have installed a dedicated app, processing by the device, and processing by the server. The user uploads a photo of their face and performs real-time emotion recognition, and the device sends this data to the server. The server performs image and emotion recognition, generates beauty suggestions based on the analysis results, sends them to the device, and finally makes them available to the user.

[1209] Technology and software used

[1210] Image recognition techniques: Use convolutional neural network (CNN) models such as ResNet and VGG.

[1211] Emotion recognition technology: Use emotion recognition models such as FER+.

[1212] Databases: Use databases related to beauty products and aesthetic medicine (e.g., general beauty product databases).

[1213] Communication technology: Uses network communication protocols such as HTTP and HTTPS.

[1214] The process begins when a user launches the app, creates an account, and logs in. The user uploads a current photo of their face and a previous photo of their healthy face, and the app also uses their camera feed to recognize the user's emotional state in real time.

[1215] The device sends this data (current facial photo, past facial photos, and real-time camera feed) to a server, which then analyzes the data using image and emotion recognition algorithms to identify skin condition and emotional state.

[1216] The server then searches its database to suggest makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss), skin care products (e.g., whitening toner, moisturizing cream), and cosmetic medical treatments (e.g., laser treatment for dark spots, hyaluronic acid injections) based on the analysis results.

[1217] The server converts the generated proposals into JSON format and sends them to the device, which then visually displays the proposals and allows the user to consider specific products and treatments.

[1218] Specific examples

[1219] For example, a user can upload a current photo of their face and a photo of their face from a previous period when they were healthy. At the same time, the real-time camera feed is used to analyze their emotional state. The server determines that their skin is dry overall, but oily in the T-zone, and their skin tone is light ochre, and detects that the user's emotional state is stressed.

[1220] Based on these conditions, the server makes the following suggestions:

[1221] Moisturizing foundation (brand name is common)

[1222] Matte Finish Eyeshadow (same as above)

[1223] Natural lip gloss (same as above)

[1224] Relaxing skin care products (for example, moisturizing cream with lavender oil)

[1225] How to use whitening toner and moisturizing cream

[1226] Laser treatment for removing spots and hyaluronic acid injections for lifting

[1227] Prompt Sentence Examples

[1228] "Design a system that analyzes current and past facial photos and suggests optimal makeup and skin care."

[1229] "Please explain in detail the system that compares your current skin condition with your past skin condition and suggests cosmetic medical treatments."

[1230] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1231] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1232] Step 1: User installs and launches the app

[1233] The user installs a dedicated app from the App Store or Google Play on their smartphone. Once the installation is complete, they tap the app icon to launch the app. This is the input. The output is the display of the app's launch screen.

[1234] Step 2: User creates account and logs in

[1235] When a user launches the app for the first time, they create an account using their email address or social media account. They are prompted to enter their ID and password. Existing users log in by entering their ID and password. The input is account information and login information, and the output is obtaining access rights to the user's personal page.

[1236] Step 3: User uploads a current photo of their face

[1237] The user selects the "upload photo" option in the app, selects a current face photo, and uploads it. The app uses the smartphone's camera roll and interface. The input is the current face photo, and the output is sending the photo to the server.

[1238] Step 4: User uploads a photo of their face

[1239] The user selects the "Upload a Previous Photo" option and selects and uploads a previous photo of their face with healthy skin. The smartphone's camera roll and interface are used again. The input is the previous photo of the face, and the output is the transmission of the photo to the server.

[1240] Step 5: The device sends the real-time camera feed to the server

[1241] The device activates the device's camera to recognize the user's real-time emotional state, captures the user's face, and sends the camera feed to the server. The input is the real-time camera feed, and the output is the feed data sent to the server.

[1242] Step 6: Server parses images and camera feed

[1243] The server analyzes the received current face photo, past face photos, and real-time camera feed. It processes the data using image recognition algorithms (e.g., ResNet, VGG) and emotion recognition algorithms (e.g., FER+). The input is the received image and camera feed, and the output is the analysis results such as the user's skin color, skin condition, and emotional state.

[1244] Step 7: The server generates beauty suggestions based on the analysis results.

[1245] The server then searches a database based on the analysis results to make beauty recommendations. For example, it selects makeup products, skin care products, and cosmetic treatments based on skin condition. The input is the analysis results of skin color, skin condition, and emotional state, and the output is a set of recommended beauty products and cosmetic treatments.

[1246] Step 8: Send the server-generated proposal to the device

[1247] The server converts the generated beauty suggestions into an appropriate format (e.g., JSON) and sends them to the device via a network communication protocol (e.g., HTTP or HTTPS). The input is a set of beauty suggestions, and the output is JSON-formatted data.

[1248] Step 9: The device displays a visual suggestion

[1249] The device analyzes the beauty suggestions received from the server and displays them visually in the app for the user. The input is JSON format data, and the output is a visual display within the app.

[1250] Step 10: User confirms and uses the proposal

[1251] The user checks the suggestions displayed on the device and clicks on a link to purchase the suggested product or make an appointment at a beauty salon or clinic. The input is the visually displayed suggestion, and the output is the actual purchase or appointment.

[1252] (Application example 2)

[1253] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1254] Conventional beauty recommendation systems suggest makeup and skincare products based on an analysis of the user's skin tone and condition, but they lacked the ability to consider the user's real-time emotional state. Additionally, while some systems had a function to compare a user's current and past images, they did not suggest cosmetic treatments based on the user's emotional state, making it difficult to provide more personalized recommendations.

[1255] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the image closer to the past state, means for recognizing emotional state in real time, means for suggesting skin care products based on the emotional state, and means for suggesting cosmetic treatments based on the emotional state. This makes it possible to comprehensively analyze the user's skin condition and emotional state and make optimal beauty suggestions based on the current state and emotion.

[1256] "Image recognition means" is a general term for software and hardware that receives images uploaded by users, analyzes them, and recognizes the user's facial features and skin condition.

[1257] The "means for analyzing skin color and skin condition" is a system for analyzing a user's skin color, dryness, oiliness, blemishes, wrinkles, etc. using image recognition technology.

[1258] The "means for suggesting makeup products" refers to algorithms and software that suggest optimal cosmetics to users based on the analysis results.

[1259] The "means for suggesting skin care products" is a system that selects and suggests skin care items suitable for the user's skin condition based on the analysis results.

[1260] The "means for proposing cosmetic medical treatment" is a system for proposing optimal cosmetic medical treatments and procedures based on the user's skin condition and emotional state.

[1261] The "means for comparing images" refers to software and hardware that uses an algorithm to compare past and current images and analyze changes and areas for improvement in skin condition.

[1262] The "means for making suggestions" is a system that suggests to users the makeup products, skin care products, and cosmetic medical treatments they need to bring their appearance closer to their previous state.

[1263] "Means for recognizing emotional state" refers to technology for analyzing a user's emotional state (e.g., stress, relaxation) using real-time camera feeds, etc.

[1264] The "means for suggesting skin care products based on emotional state" is a system for suggesting skin care products suitable for a user in consideration of the analyzed emotional state.

[1265] The "means for suggesting cosmetic medical treatments based on emotional state" is a system for suggesting cosmetic medical treatments or procedures according to the user's emotional state.

[1266] This invention is a system that analyzes a user's skin color and condition and suggests optimal makeup products, skin care products, and cosmetic medical treatments. The system also includes a function to recognize the user's emotional state in real time and make suggestions based on the emotional state.

[1267] The overall structure of the system is as follows: Users install a dedicated application on their smartphone, create an account, and log in. They then take a photo of their current face and upload it through the application. By uploading past facial photos as well, the server can compare and analyze past and current images. The application also uses the smartphone's camera to recognize the user's real-time emotional state.

[1268] The process on the device (user's smartphone) is as follows: The current face photo uploaded by the user, past face photos, and real-time camera feed are sent to the server, and this data is sent to the server in an appropriate format.

[1269] The server includes the following means for analyzing the received data:

[1270] 1. Image recognition means: Analyzes the received current and past facial photos to recognize the user's skin color and condition (dry, oily, spots, wrinkles, etc.).

[1271] 2. Skin color and condition analysis: Analyze the user's skin characteristics based on the results of image recognition. This analysis uses image processing libraries such as OpenCV.

[1272] 3. Real-time emotional state recognition: Using the camera feed, the user's emotional state is analyzed in real time using emotion recognition algorithms.

[1273] 4. Means for making suggestions based on analysis results: Based on the analysis results and emotional state, the most suitable makeup products, skin care products, and cosmetic medical treatments are searched from a database and suggestions are generated.

[1274] 5. Image comparison: Compare past and current images to analyze changes in skin condition and generate suggestions to restore a previous healthy state.

[1275] 6. Means for sending proposals: The generated proposals are converted into an appropriate format such as JSON and sent to the device.

[1276] For example, the server first analyzes the user's current and past facial photos and determines that "the skin is dry overall, but oily in the T-zone, and light ochre in color." At the same time, it detects from the real-time camera feed that the user's emotional state is "stressed." Based on this analysis, the server suggests moisturizing foundation from a department store brand and skin care products with relaxing effects. It also suggests whitening toners and usage procedures to restore the user's previous healthy skin state. It also selects products such as refreshing eye care, taking the user's emotional state into consideration.

[1277] An example of a prompt to input to a generative AI model is as follows:

[1278] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[1279] In this way, users can receive optimal beauty recommendations based on their skin condition and emotional state.

[1280] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1281] Step 1:

[1282] A user installs a dedicated application on their smartphone, creates an account, and logs in. This registers the user information in the application and completes the initial settings for subsequent processing. The input is the user's basic information, and the output is a message that the account was successfully created.

[1283] Step 2:

[1284] The user takes a photo of their current face and uploads a previous healthy face photo to the application. This prepares the necessary image data. The input is the current face photo and the previous face photo, and the output is a message confirming the upload of these image data. The process is completed when the user selects the previous face photo.

[1285] Step 3:

[1286] The device (smartphone) sends the current facial photo, past facial photos, and real-time camera feed to the server. This allows the server to receive the data necessary for analysis. The input is the current facial photo, past facial photos, and camera feed, and the output is a confirmation notification that the data has been sent to the server. The device converts the image data into the required format and sends it.

[1287] Step 4:

[1288] The server analyzes the received current and past facial photos. First, it uses image recognition to extract the user's facial features, and then it determines their skin color and condition. The input is the current and past facial photos, and the output is the analysis results of their skin color and condition. Specifically, it uses libraries such as OpenCV to analyze the pixel information of the images.

[1289] Step 5:

[1290] The server analyzes the real-time camera feed to recognize the user's emotional state. This analysis uses an emotion recognition algorithm to identify emotional states such as stress or relaxation. The input is the real-time camera feed, and the output is the recognition result of the user's emotional state. This is completed by running the emotion recognition algorithm to analyze facial expressions.

[1291] Step 6:

[1292] The server searches a database for optimal makeup products, skin care products, and cosmetic treatments based on the analysis results of skin color and skin condition, as well as the recognition results of emotional state. Based on the search results, it generates recommendations. The input is the analysis results and recognition results, and the output is the recommendations. The server completes the process by executing a database query to select the appropriate products and treatments.

[1293] Step 7:

[1294] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device. This delivers the proposal to the user. The input is the proposal, and the output is the result of sending the formatted proposal. The server converts the data format and completes the process by sending a response to the device.

[1295] Step 8:

[1296] The terminal analyzes the received proposal content and displays it to the user in a visually easy-to-understand format, allowing the user to confirm the proposal content. The input is the proposal content sent from the server, and the output is the proposal content displayed to the user. The terminal analyzes the received data and completes the process by displaying it on the screen.

[1297] Step 9:

[1298] The user reviews the makeup, skincare, and cosmetic treatment suggestions displayed in the application and then purchases the product or makes an appointment at a beauty salon or clinic based on the suggestions. This results in optimal beauty recommendations. The input is the displayed suggestions, and the output is the user's action (purchase or appointment). The process is completed when the user makes a selection and performs the specific action.

[1299] In this way, users can receive optimal beauty suggestions based on their skin condition and emotional state. An example of a prompt is as follows:

[1300] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[1301] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1302] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1303] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1304] [Fourth embodiment]

[1305] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1306] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1307] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1308] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1309] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1310] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1311] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1312] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1313] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1314] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1315] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1316] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1317] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1318] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in skin condition, and provides specific beauty recommendations based on the results.

[1319] Program processing overview

[1320] This system operates roughly as follows:

[1321] 1. User Actions

[1322] Users install a dedicated app on their device, create an account and log in. They upload a current photo of their face and a previous photo of their healthy face.

[1323] 2. Terminal Processing

[1324] The device sends the current and past facial photos uploaded by the user to the server. The device then sends the image data to the server in an appropriate format and waits for a response.

[1325] 3. Server Processing

[1326] The server analyzes the received images to identify the user's skin tone and condition, then searches a database to generate optimal recommendations for makeup, skincare products, and cosmetic treatments based on the analysis results.

[1327] Learn more about natural language processing

[1328] User Actions

[1329] 1. The user launches the app, creates an account, and logs in.

[1330] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[1331] 3. Next, select the "Upload a Previous Photo" option and upload a photo of your face from a time when you were healthy to the app.

[1332] Terminal handling

[1333] 1. The device sends the uploaded current and past facial photos to a server via the Internet.

[1334] 2. The device receives a response from the server and waits for the analysis results and proposals.

[1335] Server Processing

[1336] 1. The server analyzes the received image and identifies the user's skin color, dryness or oiliness, distribution of blemishes and wrinkles, etc.

[1337] 2. Based on the analysis results, the server selects makeup products (foundation, eye shadow, blush, lipstick, etc.) from a database.

[1338] 3. The server then searches for skin care products to improve the user's current skin condition and generates recommendations.

[1339] 4. The server compares the past and current images and suggests cosmetic medical treatments the user needs to restore their previous healthy skin condition.

[1340] Specific examples

[1341] 1. The user uploads a photo of their current face and also uploads a photo of their face from a few years ago with healthy skin.

[1342] 2. The device sends these photos to the server.

[1343] 3. The server analyzes the received image and determines that "the skin is dry overall, but oily in the T-zone. The skin color is light ochre."

[1344] 4. Depending on your skin condition, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[1345] 5. Select skin care products such as whitening toner and moisturizing cream, and provide instructions for their use.

[1346] 6. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing blemishes and hyaluronic acid injections for skin lifting.

[1347] In this way, users can easily obtain specific measures to improve their skin condition.

[1348] The processing flow will be explained below.

[1349] Program processing flow

[1350] User Actions

[1351] Step 1:

[1352] Users install the dedicated app on their device, create an account, and log in.

[1353] Step 2:

[1354] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[1355] Step 3:

[1356] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[1357] Terminal handling

[1358] Step 1:

[1359] The device sends the uploaded current and past facial photos to the server.

[1360] Step 2:

[1361] The device continues to maintain the network connection while waiting for a response from the server.

[1362] Server Processing

[1363] Step 1:

[1364] The server inputs the current and past facial photos received from the device into an algorithm that analyzes them.

[1365] Step 2:

[1366] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1367] Step 3:

[1368] Based on the analysis results, databases of makeup products, skin care products, and cosmetic medical treatments are queried.

[1369] Step 4:

[1370] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition.

[1371] Step 5:

[1372] The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous condition, and also provides instructions for use.

[1373] Step 6:

[1374] The server will suggest the most appropriate cosmetic medical treatment (e.g., laser treatment for removing spots, hyaluronic acid injections) for areas of the skin that need improvement.

[1375] Step 7:

[1376] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[1377] Terminal handling (again)

[1378] Step 1:

[1379] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[1380] Step 2:

[1381] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[1382] User Action (again)

[1383] Step 1:

[1384] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[1385] Step 2:

[1386] Users consider whether the proposed products and treatments are suitable for them.

[1387] Step 3:

[1388] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[1389] In this way, specific actions are performed for each processing step, allowing the user to easily make optimal choices regarding their own beauty.

[1390] Example 1

[1391] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1392] Conventional beauty recommendation systems lack detailed suggestions based on the user's specific skin condition or past skin condition, making it difficult for users to efficiently find products and cosmetic treatments that are suitable for their skin condition. In addition, since they do not provide specific advice on how to return to a previous healthy skin condition, user satisfaction can be low.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1394] In this invention, the server includes an image recognition unit, a unit for receiving current and past images uploaded by the user, and a unit for analyzing the user's skin color and condition. This allows for detailed recommendations for makeup products, skin care products, and cosmetic treatments based on the analysis results. Furthermore, by comparing past and current images and making specific recommendations to bring the user's skin closer to its previous condition, user satisfaction is improved.

[1395] "Image recognition means" refers to a device or software that has the function of analyzing input image data and identifying and extracting specific features or patterns.

[1396] "Means for receiving current and past images uploaded by users" refers to devices or software that have the function of receiving image data sent by users via the Internet or an application.

[1397] "Means for analyzing the user's skin color and skin condition" refers to a device or software that has the function of identifying the user's skin condition, such as skin color, dryness, oiliness, and distribution of spots and wrinkles, based on the received image data.

[1398] The "means for suggesting makeup products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable makeup products to the user based on the analysis results of the skin condition.

[1399] The "means for suggesting skin care products based on the analysis results" refers to a device or software that has the function of selecting and suggesting the most suitable skin care products to the user based on the analysis results of the skin condition.

[1400] "Means for proposing cosmetic medical treatments based on analysis results" refers to devices or software that have the function of selecting and proposing cosmetic medical treatments or procedures suitable for a user based on the analysis results of the user's skin condition.

[1401] The "means for comparing a past image with a current image" refers to a device or software that has the function of comparing a past image provided by a user with a current image and identifying points of change.

[1402] The "means for making suggestions to bring the skin closer to its previous state" refers to a device or software that has the function of generating suggestions for makeup products, skin care products, and cosmetic medical treatments necessary to restore the skin to its previous healthy state.

[1403] The "means for transmitting the analysis results and proposal contents to the user's terminal" refers to a device or software that has the function of transmitting the analysis results and proposal contents to the user's terminal via the Internet and displaying them.

[1404] "Skin care products" refers to a broad range of products including cosmetics, moisturizing creams, whitening toners, etc. that are used for the purpose of caring for and maintaining healthy skin.

[1405] "Cosmetic medicine" refers to treatments and procedures that use medical technology to pursue and improve beauty, such as laser treatments, hyaluronic acid injections, and skin lifting.

[1406] This invention relates to a system that uses image recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on the results. The system analyzes current and past images uploaded by the user, identifies changes in the skin condition, and provides specific beauty suggestions based on the results.

[1407] Hardware and software used

[1408] To implement this system, a user device, a server, and image recognition software are required. Specifically, it uses a user device such as a smartphone or tablet, a cloud server, and image analysis software (e.g., OpenCV, TensorFlow).

[1409] User Actions

[1410] Users install a dedicated app on their device. This app has functions such as uploading images, displaying analysis results, and receiving suggestions. Users launch the app, create an account, and log in. After logging in, users select the "upload photos" option and upload a current photo of their face and a previous photo of their healthy face to the app.

[1411] Terminal handling

[1412] The device temporarily stores the uploaded current and past facial photos in its memory. The device then converts these image data into an appropriate format (e.g., JPEG, PNG) and sends them to a server via the Internet. The device then waits for a response from the server. Through this series of operations, users can easily analyze their skin condition and receive recommendations.

[1413] Server Processing

[1414] The server stores the received image data in a dedicated directory. It then uses image recognition software to analyze the image data and identify the user's skin condition (skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.). The analysis results are stored in a database. Based on the analysis results, the server then searches the database to generate recommendations for makeup products, skin care products, and cosmetic treatments that are optimal for the user. For example, if the server determines that the user's skin condition is "dry overall, but oily in the T-zone, and light ochre skin color," it can recommend products accordingly.

[1415] Specific examples

[1416] 1. Users upload a current photo of their face, as well as a past photo of their face when they were healthy.

[1417] 2. The device sends these photos to the server.

[1418] 3. Based on the analysis results, the server will suggest a moisturizing foundation from a department store brand, a matte finish eyeshadow from a drugstore brand, and a natural lip gloss from a Korean brand.

[1419] 4. As skin care products, select toners and moisturizing creams with whitening effects and provide instructions for their use.

[1420] 5. As a cosmetic medical treatment suggestion, we will propose specific treatments such as laser treatment for removing age spots and hyaluronic acid injections for skin lifting.

[1421] Prompt Sentence Examples

[1422] "Please upload a current photo of your face and a photo from a previous time. We will analyze your skin condition and suggest the best makeup products, skincare products, and cosmetic treatments for you."

[1423] In this way, users can easily obtain specific measures to improve their skin condition, and this system effectively resolves users' beauty concerns and problems.

[1424] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1425] Step 1: User Actions

[1426] Input: The user installs the app, creates an account, and logs in. Within the app, they select "Upload Photos" and upload their current and past facial photos.

[1427] Data processing: The app temporarily stores photos uploaded by users in the device's memory.

[1428] Output: Saved current and past face photos.

[1429] Users install the app on their smartphone or tablet, create an account, and log in. They select the "upload photo" option within the app, take or select a photo of their current face, and then upload a photo of their face from a previous period when they looked healthy.

[1430] Step 2: Send an image from your device

[1431] Input: Your saved current and past face photos.

[1432] Data processing: The device converts image data into an appropriate format (e.g., JPEG, PNG) and optimizes the data size.

[1433] Output: Image data for transmission.

[1434] The device temporarily stores the current and past facial photos uploaded by the user in memory, then converts the image data into an appropriate format and sends it to a server over the Internet.

[1435] Step 3: Server Image Analysis

[1436] Input: Current and past face photos submitted.

[1437] Data processing: The server uses image recognition software (e.g., OpenCV, TensorFlow) to analyze the user's skin color, dryness, oiliness, distribution of blemishes and wrinkles, etc.

[1438] Output: Analysis results (e.g. skin tone is light ochre, dryness is high, oiliness is medium, number of blemishes is 12, etc.).

[1439] The server stores the received image data in a dedicated directory and uses image recognition software to analyze the user's skin condition. The analysis results are stored in a database as numerical and text data.

[1440] Step 4: Generate a Server Proposal

[1441] Input: Analysis result data.

[1442] Data processing: The server searches the database and generates optimal makeup, skincare, and cosmetic medical treatment recommendations based on the analysis results.

[1443] Output: Suggestions (e.g. moisturizing foundation, matte finish eyeshadow, whitening toner, hyaluronic acid injections, etc.).

[1444] Based on the analysis results, the server searches the database and makes recommendations for the most suitable makeup products, skin care products, and cosmetic medical treatments for the user. For example, appropriate recommendations are generated based on analysis results such as "overall dry skin, but oily T-zone. Skin tone is light ochre."

[1445] Step 5: Handling the Server Response

[1446] Input: Proposal data.

[1447] Data processing: The server formats the proposal content into a format that can be sent to the user's device.

[1448] Output: Formatted proposal.

[1449] The server then sends the generated recommendations to the user's device, which then displays them on the app's interface, providing the user with easy-to-read information. Users can then review the suggested makeup, skincare, and cosmetic treatments and try them out as needed.

[1450] (Application example 1)

[1451] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] In today's world, many people worry about their skin condition in their daily lives, but often lack the knowledge and information to select appropriate skincare products and cosmetic medical treatments. Time and location constraints also make it difficult to consult with a beauty professional. Conventional methods require in-store consultations, so a quick and easy method is needed.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1454] In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic medical treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the skin closer to the past condition, and means for receiving images uploaded by the user and displaying the analysis results and suggestions, which is an application installed on a smartphone. This allows the user to easily analyze their own skin condition from the comfort of their own home and receive suggestions for appropriate beauty products and services.

[1455] "Image recognition means" is a technology for analyzing images uploaded by users and identifying their contents.

[1456] "Skin color" indicates the color tone of the user's skin, and is information obtained by image analysis.

[1457] "Skin condition" refers to the health and appearance of the user's skin, including dryness, oiliness, blemishes, wrinkles, and the like.

[1458] "User" means a person who uses the System and uploads images to analyze their skin condition and receive suggestions for beauty products and services.

[1459] "Makeup products" refers to cosmetics intended to enhance the user's appearance, including foundation, eye shadow, lip gloss, etc.

[1460] "Skin care products" refers to products that are intended to maintain and improve the health of a user's skin, including toners and moisturizing creams.

[1461] "Cosmetic medicine" refers to medical procedures aimed at improving a user's skin condition, including, for example, laser treatments and hyaluronic acid injections.

[1462] A "past image" is a photograph of the user's face taken in the past, showing a past healthy skin condition.

[1463] The "current image" is the most recent photograph of the user's face taken to show the user's current facial condition.

[1464] The "comparison means" is a technology for analyzing past and current images to identify changes in skin condition.

[1465] The "means of suggestion" refers to technology that provides users with appropriate makeup products, skin care products, and cosmetic medical treatments based on the analysis results.

[1466] A "smartphone" is a multi-functional mobile phone that allows users to install and use applications.

[1467] "Application" refers to a software program that is installed on a smartphone and provides the functionality of the present invention.

[1468] "Means for receiving" refers to the technology used to send and receive images uploaded by users to the server.

[1469] "Display means" refers to technology for visually presenting the analysis results and proposals to users.

[1470] A "generative AI model" is a model that uses artificial intelligence technology to generate prompts and suggestions based on the analysis results.

[1471] A "prompt" is an instruction entered when utilizing a generative AI model, and is a guideline for obtaining specific output results.

[1472] This invention is a system that analyzes a user's skin color and condition and, based on that, suggests optimal makeup products, skin care products, and cosmetic medical treatments. This system includes an application and a server installed on a smartphone, and operates according to the following specific steps.

[1473] First, users log in to the application installed on their smartphone and upload their current and past facial photos. The smartphone then sends the uploaded images to a server via the Internet.

[1474] The server analyzes the received images using image recognition means to identify the user's skin tone and condition (dryness, oiliness, blemishes, wrinkles, etc.) Based on the analysis results, the server searches a database to generate optimal recommendations for makeup products, skin care products, and cosmetic medical treatments.

[1475] The recommendations, along with the analysis results, are sent to the user's smartphone and displayed, allowing them to easily find beauty products and services that are suitable for their skin condition from the comfort of their own home.

[1476] The server then uses a generative AI model based on the analysis results to create a prompt. The prompt is generated as a guideline for obtaining specific output results and is provided to the user. Utilizing the generative AI model enables more accurate suggestions.

[1477] For example, a user can upload a photo of their current face and a photo of their face in a previous healthy state. The server compares these photos and generates an analysis result that indicates "dry skin overall, but oily T-zone." It then makes specific recommendations, such as moisturizing foundation, whitening toner, and hyaluronic acid injections.

[1478] Example prompt sentence:

[1479] Analyze a user-uploaded facial photo. Identify skin tone, dryness, oiliness, blemishes, and wrinkles, and provide specific recommendations for skin care products, makeup products, and cosmetic treatments based on the results. Generate specific output such as "Skin condition analysis: Light ochre, overall dry, T-zone oily, blemish distribution: High."

[1480] In this way, users can easily select the best beauty products and services at home without the need for expert guidance.

[1481] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1482] Step 1:

[1483] The user launches the application installed on their smartphone, creates an account and logs in.

[1484] Input: User account information (ID, password, etc.).

[1485] Data processing / data calculation: Account information is sent to the server for login authentication and verification.

[1486] Output: The home screen is displayed after successful login.

[1487] Step 2:

[1488] The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[1489] Input: A current photo of the user.

[1490] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[1491] Output: Current face photo data is stored within the app.

[1492] Step 3:

[1493] Users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[1494] Input: A previous photo of the user's face.

[1495] Data processing / data calculation: Changing the format of photos (compression, resizing, etc.).

[1496] Output: Past facial photo data is stored within the app.

[1497] Step 4:

[1498] The app sends the current and past facial photos uploaded by the user to the server.

[1499] Input: Current face photo, past face photo.

[1500] Data processing / data calculation: Photo data is sent to a server via the Internet.

[1501] Output: The photo data is saved on the server.

[1502] Step 5:

[1503] The server analyzes the received images and identifies the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1504] Input: Current face photo, past face photo.

[1505] Data processing / data calculation: Apply image recognition algorithms to analyze skin condition (e.g., using Python's OpenCV library).

[1506] Output: Analysis results for skin color, dryness, oiliness, spots, wrinkles, etc.

[1507] Step 6:

[1508] Based on the analysis results, the server searches its database to generate optimal suggestions for makeup products, skin care products, and cosmetic medical treatments.

[1509] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[1510] Data Processing / Data Calculation: Performing database queries to find suitable products and services.

[1511] Output: A list of suggested beauty products and services.

[1512] Step 7:

[1513] The suggestions, along with the analysis results, are sent to the smartphone and displayed to the user through the application.

[1514] Input: A list of beauty products and services you would like to suggest.

[1515] Data processing / data calculation: The proposal is formatted and sent to the application.

[1516] Output: The analysis results and suggestions are displayed on the smartphone screen.

[1517] Step 8:

[1518] Based on the analysis results, the server uses a generative AI model to form a prompt sentence.

[1519] Input: Analysis results (skin color, dryness, oiliness, spots, wrinkles, etc.).

[1520] Data processing / data calculation: Input the analysis results into a generative AI model to generate a prompt sentence (e.g., an AI model such as GPT-3).

[1521] Output: The generated prompt statement.

[1522] Step 9:

[1523] The generated prompt text is presented to the user.

[1524] Input: The generated prompt statement.

[1525] Data processing / data calculation: Formats the prompt statement and sends it to the application.

[1526] Output: The generated prompt text is displayed on the smartphone screen.

[1527] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1528] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition and, based on that, recommend optimal makeup products, skin care products, and cosmetic medical treatments. The system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[1529] Program processing overview

[1530] This system operates roughly as follows:

[1531] User Actions

[1532] Users install the app on their device, create an account, and log in. They upload a current photo of their face and a previous photo of their healthy face. The system can also use the camera feed to recognize the user's real-time emotional state.

[1533] Terminal handling

[1534] The device sends the current and past facial photos uploaded by the user, as well as the real-time camera feed, to the server, which then sends this data in the appropriate format and waits for a response.

[1535] Server Processing

[1536] The server analyzes the received images and real-time camera feed to identify the user's skin tone, skin condition and emotional state, then searches a database to generate optimal recommendations for makeup, skincare products and cosmetic treatments based on the analysis.

[1537] Learn more about natural language processing

[1538] User Actions

[1539] 1. The user launches the app, creates an account, and logs in.

[1540] 2. The user selects the "Upload Photo" option and uploads a photo of their current face to the app.

[1541] 3. Next, users select the "Upload a Previous Photo" option and upload a photo of their face from a time when they were healthy to the app.

[1542] 4. The system uses the camera feed to capture the user's face to recognize their real-time emotional state.

[1543] Terminal handling

[1544] 1. The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[1545] 2. The device maintains the network connection while waiting for a response from the server.

[1546] Server Processing

[1547] 1. The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[1548] 2. The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1549] 3. Furthermore, the server uses emotion recognition algorithms to analyze the user's current emotional state.

[1550] 4. Based on the analysis results, query databases of makeup products, skin care products, and cosmetic medical treatments.

[1551] 5. The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit their skin condition and emotional state.

[1552] 6. The server compares the analysis results with past images, selects skin care products (e.g., whitening toner, moisturizing cream) to restore the previous skin condition, and also provides instructions for using them.

[1553] 7. When it comes to cosmetic medicine, the app will suggest the most appropriate treatment (e.g., laser treatment for removing dark spots, hyaluronic acid injections) based on the user's current emotional state.

[1554] 8. The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device.

[1555] Terminal handling (again)

[1556] 1. The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[1557] 2. The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[1558] User Action (again)

[1559] 1. The user checks the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[1560] 2. The user considers whether the proposed products or treatments are suitable for them.

[1561] 3. The user uses the suggestions to purchase a specific product or make an appointment at a beauty salon or clinic.

[1562] Specific examples

[1563] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[1564] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[1565] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[1566] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[1567] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[1568] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1569] The processing flow will be explained below.

[1570] MODE FOR CARRYING OUT THE INVENTION

[1571] User Actions

[1572] Step 1:

[1573] Users install a dedicated app on their device, create an account and log in.

[1574] Step 2:

[1575] Users can select the "Upload Photo" option and upload a photo of their current face to the app.

[1576] Step 3:

[1577] Next, users select the "Upload a Previous Photo" option and upload a previous photo of their healthy face to the app.

[1578] Step 4:

[1579] The app uses real-time emotion recognition to capture the user's current emotions from the camera feed.

[1580] Terminal handling

[1581] Step 1:

[1582] The device sends the uploaded current face photo, past face photos, and real-time camera feed to the server.

[1583] Step 2:

[1584] The device maintains a network connection while waiting for analysis results and recommendations from the server.

[1585] Server Processing

[1586] Step 1:

[1587] The server inputs the current and past facial photos received from the device, as well as real-time camera feeds, into an algorithm that analyzes them.

[1588] Step 2:

[1589] The server uses an image analysis algorithm to analyze the user's skin color and condition (dryness, oiliness, blemishes, wrinkles, etc.).

[1590] Step 3:

[1591] The server uses emotion recognition algorithms to analyze the user's current emotional state (e.g., stress, relaxation, joy, sadness, etc.).

[1592] Step 4:

[1593] Based on the analysis results (skin condition and emotional state), the server queries databases of makeup products, skin care products, and cosmetic medical treatments.

[1594] Step 5:

[1595] The server selects makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss) that suit the skin condition and emotional state.

[1596] Step 6:

[1597] The server compares the past and current images and selects skin care products (e.g., whitening toner, moisturizing cream) to restore the skin to its previous healthy state.

[1598] Step 7:

[1599] Taking into account your emotional state, the server will suggest relaxing skin care products (e.g., moisturizing cream containing lavender oil) or cosmetic treatments that promote psychological stress reduction (e.g., relaxation massage).

[1600] Step 8:

[1601] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the terminal.

[1602] Terminal handling (again)

[1603] Step 1:

[1604] The device receives makeup, skin care, and cosmetic medical treatment suggestions sent from the server.

[1605] Step 2:

[1606] The device analyzes the received suggestions and displays them to the user in a visually easy-to-understand format.

[1607] User Action (again)

[1608] Step 1:

[1609] Users can check the makeup, skincare, and cosmetic medical treatment suggestions displayed on the device.

[1610] Step 2:

[1611] Users consider whether the proposed products and treatments are suitable for them.

[1612] Step 3:

[1613] Users can use the suggestions to purchase specific products or make appointments at beauty salons or clinics.

[1614] Specific examples

[1615] For example, a user can upload a current photo of their face and a photo from a few years ago with healthy skin, while the system simultaneously analyzes the user's emotional state (e.g., stressed or relaxed) based on the real-time camera feed.

[1616] The server analyzes the received data and determines that the user's skin is dry overall, but oily in the T-zone, and that their skin tone is light ochre. At the same time, it detects that the user's emotional state is stressed.

[1617] The server will suggest moisturizing foundation from a department store brand, matte finish eyeshadow from a drugstore brand, and natural lip gloss from a Korean brand based on the user's skin condition and emotional state. It will also suggest skin care products with relaxing effects (such as moisturizing cream containing lavender oil) based on the user's stress level.

[1618] In addition, it will also select skin care products such as a whitening toner to restore your skin to its previous healthy state and a refreshing eye care to improve your emotional state, and provide instructions on how to use them.

[1619] Cosmetic medical treatments such as laser treatments for removing blemishes and hyaluronic acid injections for lifting are suggested based on the user's emotional state.

[1620] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1621] Example 2

[1622] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] Conventional beauty recommendation systems can only make recommendations based on the user's current skin condition, making it difficult to provide comprehensive beauty recommendations that take into account the user's past healthy skin condition and emotional state. Furthermore, since recommendations for skin care products and cosmetic medical treatments do not reflect the user's real-time emotional state, it is not possible to provide optimal recommendations for each individual user.

[1624] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an image recognition means, a means for analyzing the user's skin color and skin condition, a means for suggesting makeup products based on the analysis results, a means for suggesting skin care products based on the analysis results, a means for suggesting cosmetic medical treatments based on the analysis results, a means for comparing a past image with a current image, a means for making suggestions to bring the image closer to the past state, a means for recognizing the user's real-time emotional state, and a means for making optimal suggestions according to the user's emotional state based on the analysis results. This makes it possible to provide beauty suggestions that comprehensively consider the user's skin condition and emotional state.

[1625] "Image recognition means" refers to technology or equipment for detecting and analyzing specific features (e.g., face, skin condition, etc.) from an image.

[1626] "Means for analyzing a user's skin color and skin condition" refers to technology or equipment that analyzes a user's skin color, texture, dryness / oiliness, blemishes, wrinkles, etc.

[1627] "Means for suggesting makeup products based on analysis results" refers to technology or equipment that selects appropriate makeup products based on the results of skin analysis and suggests them to users.

[1628] The "means for suggesting skin care products based on the analysis results" refers to a technology or device that uses the skin analysis results to select appropriate skin care products and suggest them to the user.

[1629] "Means for proposing cosmetic medical treatment based on analysis results" refers to technology or devices that suggest cosmetic medical treatment (e.g., laser treatment, injections, etc.) based on the results of skin analysis.

[1630] The "means for comparing a past image with a current image" refers to a technique or device that compares a past image provided by the user with a current image to evaluate changes in skin condition.

[1631] The "means for making suggestions to bring the skin closer to its previous state" refers to a technology or device that suggests skin care methods or cosmetic medical treatments to bring the skin back to its previous healthy state.

[1632] "Means for recognizing a user's real-time emotional state" refers to a technology or device that analyzes and recognizes a user's current emotional state (e.g., stress, relaxation, etc.) in real time.

[1633] "Means for making optimal suggestions based on the user's emotional state based on the analysis results" refers to technology or equipment that selects and suggests appropriate skin care products, cosmetic medical treatments, makeup products, etc. based on the user's emotional state.

[1634] This invention relates to a system that combines image recognition technology and emotion recognition technology to analyze a user's skin color and condition, and then suggests optimal makeup products, skin care products, and cosmetic medical treatments based on that analysis. Specifically, the system aims to provide optimal beauty recommendations for individuals by analyzing current and past images uploaded by the user and also recognizing the user's emotional state.

[1635] overview

[1636] This system roughly consists of operations by users who have installed a dedicated app, processing by the device, and processing by the server. The user uploads a photo of their face and performs real-time emotion recognition, and the device sends this data to the server. The server performs image and emotion recognition, generates beauty suggestions based on the analysis results, sends them to the device, and finally makes them available to the user.

[1637] Technology and software used

[1638] Image recognition techniques: Use convolutional neural network (CNN) models such as ResNet and VGG.

[1639] Emotion recognition technology: Use emotion recognition models such as FER+.

[1640] Databases: Use databases related to beauty products and aesthetic medicine (e.g., general beauty product databases).

[1641] Communication technology: Uses network communication protocols such as HTTP and HTTPS.

[1642] The process begins when a user launches the app, creates an account, and logs in. The user uploads a current photo of their face and a previous photo of their healthy face, and the app also uses their camera feed to recognize the user's emotional state in real time.

[1643] The device sends this data (current facial photo, past facial photos, and real-time camera feed) to a server, which then analyzes the data using image and emotion recognition algorithms to identify skin condition and emotional state.

[1644] The server then searches its database to suggest makeup products (e.g., moisturizing foundation, matte finish eye shadow, natural lip gloss), skin care products (e.g., whitening toner, moisturizing cream), and cosmetic medical treatments (e.g., laser treatment for dark spots, hyaluronic acid injections) based on the analysis results.

[1645] The server converts the generated proposals into JSON format and sends them to the device, which then visually displays the proposals and allows the user to consider specific products and treatments.

[1646] Specific examples

[1647] For example, a user can upload a current photo of their face and a photo of their face from a previous period when they were healthy. At the same time, the real-time camera feed is used to analyze their emotional state. The server determines that their skin is dry overall, but oily in the T-zone, and their skin tone is light ochre, and detects that the user's emotional state is stressed.

[1648] Based on these conditions, the server makes the following suggestions:

[1649] Moisturizing foundation (brand name is common)

[1650] Matte Finish Eyeshadow (same as above)

[1651] Natural lip gloss (same as above)

[1652] Relaxing skin care products (for example, moisturizing cream with lavender oil)

[1653] How to use whitening toner and moisturizing cream

[1654] Laser treatment for removing spots and hyaluronic acid injections for lifting

[1655] Prompt Sentence Examples

[1656] "Design a system that analyzes current and past facial photos and suggests optimal makeup and skin care."

[1657] "Please explain in detail the system that compares your current skin condition with your past skin condition and suggests cosmetic medical treatments."

[1658] In this way, users can get optimal beauty suggestions based on their skin condition and emotional state.

[1659] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1660] Step 1: User installs and launches the app

[1661] The user installs a dedicated app from the App Store or Google Play on their smartphone. Once the installation is complete, they tap the app icon to launch the app. This is the input. The output is the display of the app's launch screen.

[1662] Step 2: User creates account and logs in

[1663] When a user launches the app for the first time, they create an account using their email address or social media account. They are prompted to enter their ID and password. Existing users log in by entering their ID and password. The input is account information and login information, and the output is obtaining access rights to the user's personal page.

[1664] Step 3: User uploads a current photo of their face

[1665] The user selects the "upload photo" option in the app, selects a current face photo, and uploads it. The app uses the smartphone's camera roll and interface. The input is the current face photo, and the output is sending the photo to the server.

[1666] Step 4: User uploads a photo of their face

[1667] The user selects the "Upload a Previous Photo" option and selects and uploads a previous photo of their face with healthy skin. The smartphone's camera roll and interface are used again. The input is the previous photo of the face, and the output is the transmission of the photo to the server.

[1668] Step 5: The device sends the real-time camera feed to the server

[1669] The device activates the device's camera to recognize the user's real-time emotional state, captures the user's face, and sends the camera feed to the server. The input is the real-time camera feed, and the output is the feed data sent to the server.

[1670] Step 6: Server parses images and camera feed

[1671] The server analyzes the received current face photo, past face photos, and real-time camera feed. It processes the data using image recognition algorithms (e.g., ResNet, VGG) and emotion recognition algorithms (e.g., FER+). The input is the received image and camera feed, and the output is the analysis results such as the user's skin color, skin condition, and emotional state.

[1672] Step 7: The server generates beauty suggestions based on the analysis results.

[1673] The server then searches a database based on the analysis results to make beauty recommendations. For example, it selects makeup products, skin care products, and cosmetic treatments based on skin condition. The input is the analysis results of skin color, skin condition, and emotional state, and the output is a set of recommended beauty products and cosmetic treatments.

[1674] Step 8: Send the server-generated proposal to the device

[1675] The server converts the generated beauty suggestions into an appropriate format (e.g., JSON) and sends them to the device via a network communication protocol (e.g., HTTP or HTTPS). The input is a set of beauty suggestions, and the output is JSON-formatted data.

[1676] Step 9: The device displays a visual suggestion

[1677] The device analyzes the beauty suggestions received from the server and displays them visually in the app for the user. The input is JSON format data, and the output is a visual display within the app.

[1678] Step 10: User confirms and uses the proposal

[1679] The user checks the suggestions displayed on the device and clicks on a link to purchase the suggested product or make an appointment at a beauty salon or clinic. The input is the visually displayed suggestion, and the output is the actual purchase or appointment.

[1680] (Application example 2)

[1681] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1682] Conventional beauty recommendation systems suggest makeup and skincare products based on an analysis of the user's skin tone and condition, but they lacked the ability to consider the user's real-time emotional state. Additionally, while some systems had a function to compare a user's current and past images, they did not suggest cosmetic treatments based on the user's emotional state, making it difficult to provide more personalized recommendations.

[1683] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image recognition means, means for analyzing the user's skin color and skin condition, means for suggesting makeup products based on the analysis results, means for suggesting skin care products based on the analysis results, means for suggesting cosmetic treatments based on the analysis results, means for comparing past images with current images, means for making suggestions to bring the image closer to the past state, means for recognizing emotional state in real time, means for suggesting skin care products based on the emotional state, and means for suggesting cosmetic treatments based on the emotional state. This makes it possible to comprehensively analyze the user's skin condition and emotional state and make optimal beauty suggestions based on the current state and emotion.

[1684] "Image recognition means" is a general term for software and hardware that receives images uploaded by users, analyzes them, and recognizes the user's facial features and skin condition.

[1685] The "means for analyzing skin color and skin condition" is a system for analyzing a user's skin color, dryness, oiliness, blemishes, wrinkles, etc. using image recognition technology.

[1686] The "means for suggesting makeup products" refers to algorithms and software that suggest optimal cosmetics to users based on the analysis results.

[1687] The "means for suggesting skin care products" is a system that selects and suggests skin care items suitable for the user's skin condition based on the analysis results.

[1688] The "means for proposing cosmetic medical treatment" is a system for proposing optimal cosmetic medical treatments and procedures based on the user's skin condition and emotional state.

[1689] The "means for comparing images" refers to software and hardware that uses an algorithm to compare past and current images and analyze changes and areas for improvement in skin condition.

[1690] The "means for making suggestions" is a system that suggests to users the makeup products, skin care products, and cosmetic medical treatments they need to bring their appearance closer to their previous state.

[1691] "Means for recognizing emotional state" refers to technology for analyzing a user's emotional state (e.g., stress, relaxation) using real-time camera feeds, etc.

[1692] The "means for suggesting skin care products based on emotional state" is a system for suggesting skin care products suitable for a user in consideration of the analyzed emotional state.

[1693] The "means for suggesting cosmetic medical treatments based on emotional state" is a system for suggesting cosmetic medical treatments or procedures according to the user's emotional state.

[1694] This invention is a system that analyzes a user's skin color and condition and suggests optimal makeup products, skin care products, and cosmetic medical treatments. The system also includes a function to recognize the user's emotional state in real time and make suggestions based on the emotional state.

[1695] The overall structure of the system is as follows: Users install a dedicated application on their smartphone, create an account, and log in. They then take a photo of their current face and upload it through the application. By uploading past facial photos as well, the server can compare and analyze past and current images. The application also uses the smartphone's camera to recognize the user's real-time emotional state.

[1696] The process on the device (user's smartphone) is as follows: The current face photo uploaded by the user, past face photos, and real-time camera feed are sent to the server, and this data is sent to the server in an appropriate format.

[1697] The server includes the following means for analyzing the received data:

[1698] 1. Image recognition means: Analyzes the received current and past facial photos to recognize the user's skin color and condition (dry, oily, spots, wrinkles, etc.).

[1699] 2. Skin color and condition analysis: Analyze the user's skin characteristics based on the results of image recognition. This analysis uses image processing libraries such as OpenCV.

[1700] 3. Real-time emotional state recognition: Using the camera feed, the user's emotional state is analyzed in real time using emotion recognition algorithms.

[1701] 4. Means for making suggestions based on analysis results: Based on the analysis results and emotional state, the most suitable makeup products, skin care products, and cosmetic medical treatments are searched from a database and suggestions are generated.

[1702] 5. Image comparison: Compare past and current images to analyze changes in skin condition and generate suggestions to restore a previous healthy state.

[1703] 6. Means for sending proposals: The generated proposals are converted into an appropriate format such as JSON and sent to the device.

[1704] For example, the server first analyzes the user's current and past facial photos and determines that "the skin is dry overall, but oily in the T-zone, and light ochre in color." At the same time, it detects from the real-time camera feed that the user's emotional state is "stressed." Based on this analysis, the server suggests moisturizing foundation from a department store brand and skin care products with relaxing effects. It also suggests whitening toners and usage procedures to restore the user's previous healthy skin state. It also selects products such as refreshing eye care, taking the user's emotional state into consideration.

[1705] An example of a prompt to input to a generative AI model is as follows:

[1706] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[1707] In this way, users can receive optimal beauty recommendations based on their skin condition and emotional state.

[1708] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1709] Step 1:

[1710] A user installs a dedicated application on their smartphone, creates an account, and logs in. This registers the user information in the application and completes the initial settings for subsequent processing. The input is the user's basic information, and the output is a message that the account was successfully created.

[1711] Step 2:

[1712] The user takes a photo of their current face and uploads a previous healthy face photo to the application. This prepares the necessary image data. The input is the current face photo and the previous face photo, and the output is a message confirming the upload of these image data. The process is completed when the user selects the previous face photo.

[1713] Step 3:

[1714] The device (smartphone) sends the current facial photo, past facial photos, and real-time camera feed to the server. This allows the server to receive the data necessary for analysis. The input is the current facial photo, past facial photos, and camera feed, and the output is a confirmation notification that the data has been sent to the server. The device converts the image data into the required format and sends it.

[1715] Step 4:

[1716] The server analyzes the received current and past facial photos. First, it uses image recognition to extract the user's facial features, and then it determines their skin color and condition. The input is the current and past facial photos, and the output is the analysis results of their skin color and condition. Specifically, it uses libraries such as OpenCV to analyze the pixel information of the images.

[1717] Step 5:

[1718] The server analyzes the real-time camera feed to recognize the user's emotional state. This analysis uses an emotion recognition algorithm to identify emotional states such as stress or relaxation. The input is the real-time camera feed, and the output is the recognition result of the user's emotional state. This is completed by running the emotion recognition algorithm to analyze facial expressions.

[1719] Step 6:

[1720] The server searches a database for optimal makeup products, skin care products, and cosmetic treatments based on the analysis results of skin color and skin condition, as well as the recognition results of emotional state. Based on the search results, it generates recommendations. The input is the analysis results and recognition results, and the output is the recommendations. The server completes the process by executing a database query to select the appropriate products and treatments.

[1721] Step 7:

[1722] The server converts the generated proposal into an appropriate format, such as JSON, and sends it to the device. This delivers the proposal to the user. The input is the proposal, and the output is the result of sending the formatted proposal. The server converts the data format and completes the process by sending a response to the device.

[1723] Step 8:

[1724] The terminal analyzes the received proposal content and displays it to the user in a visually easy-to-understand format, allowing the user to confirm the proposal content. The input is the proposal content sent from the server, and the output is the proposal content displayed to the user. The terminal analyzes the received data and completes the process by displaying it on the screen.

[1725] Step 9:

[1726] The user reviews the makeup, skincare, and cosmetic treatment suggestions displayed in the application and then purchases the product or makes an appointment at a beauty salon or clinic based on the suggestions. This results in optimal beauty recommendations. The input is the displayed suggestions, and the output is the user's action (purchase or appointment). The process is completed when the user makes a selection and performs the specific action.

[1727] In this way, users can receive optimal beauty suggestions based on their skin condition and emotional state. An example of a prompt is as follows:

[1728] Describe a system that analyzes a user's current and past healthy facial photos and suggests the best makeup, skincare, and cosmetic treatments based on the user's skin tone, skin condition, and emotional state. Explain how a smartphone app works that uses current and past facial photos and real-time camera feeds to provide optimal beauty recommendations on the spot.

[1729] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1730] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1731] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1732] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1733] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1734] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1735] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1736] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1737] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1738] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1739] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1740] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1741] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1742] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1743] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1744] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1745] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1746] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1747] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1748] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1749] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1750] The following is further disclosed regarding the above embodiment.

[1751] (Claim 1)

[1752] Image recognition means;

[1753] means for analyzing the user's skin color and skin condition;

[1754] A means for suggesting makeup products based on the analysis results;

[1755] A means for suggesting skin care products based on the analysis results;

[1756] A means for proposing cosmetic medical treatments based on the analysis results;

[1757] a means for comparing past and present images;

[1758] A means of making suggestions to bring it closer to its past state, and

[1759] A system including:

[1760] (Claim 2)

[1761] a means for receiving user-uploaded images;

[1762] means for analyzing the received image;

[1763] a means for searching a database using the analysis results;

[1764] means for transmitting search results to a terminal;

[1765] The system of claim 1 further comprising:

[1766] (Claim 3)

[1767] A means for suggesting skin care products according to changes in the user's skin condition;

[1768] A means for suggesting cosmetic medical treatments according to changes in the user's skin condition;

[1769] means for displaying the content of the proposal to a user;

[1770] The system of claim 1 further comprising:

[1771] "Example 1"

[1772] (Claim 1)

[1773] Image recognition means;

[1774] means for receiving current and past images uploaded by a user;

[1775] means for analyzing the user's skin color and skin condition;

[1776] A means for suggesting makeup products based on the analysis results;

[1777] A means for suggesting skin care products based on the analysis results;

[1778] A means for proposing cosmetic medical treatments based on the analysis results;

[1779] a means for comparing past and present images;

[1780] A means of making suggestions to bring it closer to its past state, and

[1781] A means for transmitting the analysis results and the proposal contents to the user terminal;

[1782] A system including:

[1783] (Claim 2)

[1784] means for analyzing received current and past images;

[1785] a means for searching a database using the analysis results;

[1786] a means for utilizing the search results to generate suggestions;

[1787] The system of claim 1 further comprising:

[1788] (Claim 3)

[1789] A means for suggesting skin care products according to changes in the user's skin condition;

[1790] A means for suggesting cosmetic medical treatments according to changes in the user's skin condition;

[1791] means for displaying the proposal content on a user terminal;

[1792] The system of claim 1 further comprising:

[1793] "Application Example 1"

[1794] (Claim 1)

[1795] Image recognition means;

[1796] means for analyzing the user's skin color and skin condition;

[1797] A means for suggesting makeup products based on the analysis results;

[1798] A means for suggesting skin care products based on the analysis results;

[1799] A means for proposing cosmetic medical treatments based on the analysis results;

[1800] a means for comparing past and present images;

[1801] A means of making suggestions to bring it closer to its past state, and

[1802] An application installed on a smartphone that receives images uploaded by users and displays the analysis results and proposals;

[1803] A system including:

[1804] (Claim 2)

[1805] a means for receiving user-uploaded images;

[1806] means for analyzing the received image;

[1807] a means for searching a database using the analysis results;

[1808] means for transmitting search results to a terminal;

[1809] The system of claim 1, further comprising means for displaying beauty products and services to the user based on the analysis results.

[1810] (Claim 3)

[1811] A means for suggesting skin care products according to changes in the user's skin condition;

[1812] A means for suggesting cosmetic medical treatments according to changes in the user's skin condition;

[1813] A means for forming a prompt sentence to be input to the generative AI model based on the analysis result;

[1814] means for displaying said suggestions and prompts to a user;

[1815] The system of claim 1 further comprising:

[1816] "Example 2: Combining Emotion Engines"

[1817] (Claim 1)

[1818] Image recognition means;

[1819] means for analyzing the user's skin color and skin condition;

[1820] A means for suggesting makeup products based on the analysis results;

[1821] A means for suggesting skin care products based on the analysis results;

[1822] A means for proposing cosmetic medical treatments based on the analysis results;

[1823] a means for comparing past and present images;

[1824] A means of making suggestions to bring it closer to its past state, and

[1825] a means for recognizing a user's real-time emotional state;

[1826] A means for making optimal suggestions based on the user's emotional state based on the analysis results;

[1827] A system including:

[1828] (Claim 2)

[1829] a means for receiving user uploaded images and real-time camera feeds;

[1830] means for analyzing received images and camera feeds;

[1831] a means for searching a database using the analysis results;

[1832] means for transmitting search results to a terminal;

[1833] The system of claim 1 further comprising:

[1834] (Claim 3)

[1835] A means for suggesting skin care products according to changes in the user's skin condition and emotional state;

[1836] A means for suggesting cosmetic medical treatments according to changes in the user's skin condition and emotional state;

[1837] means for displaying the content of the proposal to a user;

[1838] The system of claim 1 further comprising:

[1839] "Application example 2 when combining emotion engines"

[1840] (Claim 1)

[1841] Image recognition means;

[1842] means for analyzing the user's skin color and skin condition;

[1843] A means for suggesting makeup products based on the analysis results;

[1844] A means for suggesting skin care products based on the analysis results;

[1845] A means for proposing cosmetic medical treatments based on the analysis results;

[1846] a means for comparing past and present images;

[1847] A means of making suggestions to bring it closer to its past state, and

[1848] a means for recognizing real-time emotional states;

[1849] a means for suggesting skin care products based on emotional state;

[1850] A means for suggesting cosmetic medical treatments based on emotional state;

[1851] A system including:

[1852] (Claim 2)

[1853] a means for receiving user-uploaded images;

[1854] means for analyzing the received image;

[1855] a means for searching for information using the analysis results;

[1856] means for transmitting search results to a terminal;

[1857] The system of claim 1 further comprising:

[1858] (Claim 3)

[1859] A means for suggesting skin care products according to changes in the user's skin condition;

[1860] A ...

Claims

1. Image recognition means; means for analyzing the user's skin color and skin condition; A means for suggesting makeup products based on the analysis results; A means for suggesting skin care products based on the analysis results; A means for proposing cosmetic medical treatments based on the analysis results; a means for comparing past and present images; A means of making suggestions to bring it closer to its past state, and A system including:

2. a means for receiving user-uploaded images; means for analyzing the received image; a means for searching a database using the analysis results; means for transmitting search results to a terminal; The system of claim 1 further comprising:

3. A means for suggesting skin care products according to changes in the user's skin condition; A means for suggesting cosmetic medical treatments according to changes in the user's skin condition; means for displaying the content of the proposal to a user; The system of claim 1 further comprising:

Images