system

Abstract

Provide a system. The system includes an input means for a user to input skin concerns and upload an image showing the current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means that allows a user to select an ideal skin condition; A gap analysis means for the server to analyze a gap between a current skin condition and an ideal skin condition; a database search means for the server to search a database for an optimal skin care item to fill the gap; a display means for displaying search results to a user on the terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, many people have a variety of skin concerns. These concerns vary not only depending on the individual, but also on seasonal and environmental changes. Traditional methods for selecting skin care products generally rely on personal experience or marketing materials, making it difficult to accurately select products that suit individual skin conditions or ideal skin conditions. As a result, users often end up using products that are inappropriate for their skin condition and fail to experience any noticeable effects. Therefore, there is a need for a system that can precisely analyze individual skin conditions and recommend appropriate skin care items based on the analysis results. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The system includes an input means for users to input their skin concerns and upload an image showing their current skin condition. The server then uses an image analysis means to analyze the image and evaluate the current skin condition. It also includes a selection means that allows users to select their ideal skin condition. The server then uses a gap analysis means to analyze the gap between the current skin condition and the ideal skin condition, and provides a database search means to search a database for optimal skin care items based on the results. The terminal then implements a display means to display the search results to the user, thereby realizing a system that recommends optimal skin care items for each user. Furthermore, the server can perform image analysis using a machine learning model or deep learning model, generate a list of skin care items based on the evaluation of the skin condition, and the terminal displays the list to the user. In this way, it is possible to select the skin care products optimal for each individual skin condition.

[0007] "User" refers to a person who uses the system to input skin concerns, upload images, and select their ideal skin condition.

[0008] "Input means" refers to an interface or device that allows users to input their skin concerns and upload images showing their current skin condition.

[0009] "Server" refers to the central processing unit of the system, a computer device that performs image analysis, gap analysis, and database search.

[0010] "Image analysis means" refers to algorithms or software that analyze uploaded skin images and evaluate skin condition.

[0011] "Selection means" refers to the user interface or prompts that allow the user to select their ideal skin condition.

[0012] "Gap analysis means" refers to an algorithm or function that analyzes the difference between the current skin condition and the ideal skin condition.

[0013] "Database search means" refers to a function for searching a database for skin care items needed to fill the gaps.

[0014] "Display means" refers to a display or user interface for presenting search results to users.

[0015] A "machine learning model" refers to an algorithm that learns patterns from data and makes predictions and classifications to assess skin condition.

[0016] A "deep learning model" refers to an algorithm that uses neural networks to automatically learn and analyze complex patterns and features. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user's skin closer to ideal. Specific embodiments of this system are described below.

[0039] Input Method

[0040] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0041] Image analysis methods

[0042] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models. For example, the depth of wrinkles, degree of dryness, and distribution of redness in the image are evaluated.

[0043] Selection method

[0044] The server provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can select their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0045] Gap Analysis Tools

[0046] The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent.

[0047] Database search methods

[0048] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each item, and based on this information, the server generates search results. For example, it may select a cream rich in moisturizing ingredients or a serum with anti-aging effects.

[0049] Display means

[0050] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0051] Specific examples

[0052] As a specific example, the following processing is performed.

[0053] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0054] 2. The device sends this information to the server.

[0055] 3. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0056] 4. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0057] 5. The server compares the current state with the ideal state and analyzes the gap.

[0058] 6. The server searches the database for the most suitable skin care items and generates a list.

[0059] 7. The device displays a list of recommended items to the user.

[0060] In this way, users can compare their current skin condition with their ideal skin condition and obtain the skin care items that are best suited to them. This system allows users to easily find the skin care products that are best suited to them, thereby efficiently improving their skin condition.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0064] Step 2:

[0065] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0066] Step 3:

[0067] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0068] Step 4:

[0069] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0070] Step 5:

[0071] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0072] Step 6:

[0073] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0074] Step 7:

[0075] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. For example, if the current wrinkle depth is deeper than the ideal, the difference in depth is calculated.

[0076] Step 8:

[0077] The server uses the results of the gap analysis to search the database for the most suitable skin care items. The search criteria include information on ingredients and effects.

[0078] Step 9:

[0079] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0080] Step 10:

[0081] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0082] In this way, the specific actions performed in each step allow the user to effectively and efficiently find the skin care items that are best suited to their current skin condition.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] With conventional methods for selecting skin care items, it is often difficult for users to find the items that are best suited to their skin condition, resulting in ineffective skin care. Furthermore, because users lack specialized knowledge, it is difficult for them to accurately evaluate their own skin condition, and they lack the information to select appropriate products. This has led to users generally purchasing skin care products on their own, which has created the problem of making it difficult to achieve effective skin care.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes input means for allowing a user to input skin problems and upload an image showing their current skin condition, image analysis means for the server to analyze the image and evaluate the current skin condition, selection means for allowing the user to select an ideal skin condition, gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition, database search means for the server to search a database for cosmetics optimal for filling the gap, display means for the terminal to display the search results to the user, evaluation means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user, and recommendation means for selecting optimal cosmetics based on the ideal skin condition selected by the user. This enables users to accurately evaluate their current skin condition and easily find optimal skin care items.

[0088] The "input means" is a means by which a user inputs skin problems and uploads an image showing the current skin condition.

[0089] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0090] The "selection means" is a means that allows the user to select an ideal skin condition.

[0091] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition.

[0092] The "database search means" is a means by which the server searches the database for the most suitable cosmetics to fill the gap.

[0093] The "display means" is a means by which the terminal displays the search results to the user.

[0094] The "evaluation means" is a means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user.

[0095] The "recommendation means" is a means for selecting the most suitable cosmetics based on the ideal skin condition selected by the user.

[0096] The present invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal cosmetics to bring the user closer to ideal skin. This system is configured as follows.

[0097] First, users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, through which users can select multiple options such as "wrinkles," "dryness," and "redness."

[0098] Next, the device sends the image and input data uploaded by the user to a server. The server receives the image and uses an image analysis algorithm to evaluate the current skin condition. This analysis uses deep learning models such as TENSORFLOW (registered trademark) and PyTorch. The server quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image, and stores the evaluation results in a database.

[0099] The server then provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles." The device then sends this selection information to the server.

[0100] The server then analyzes the gap between the current skin condition and the ideal skin condition. This gap analysis identifies skin problems that need improvement and quantifies the factors for improving each problem. The server then searches a database for the most suitable cosmetics based on the results of the gap analysis. The database stores detailed information on the ingredients and effects of each cosmetic product. The server selects the most suitable cosmetics based on the required ingredients and effects and retrieves the results from the database.

[0101] Finally, the server sends the selected cosmetics list to the terminal. The terminal displays the list to the user and provides detailed information about the recommended cosmetics. For example, it may be displayed in the form of "High Moisturizing Cream 'Brand A'" or "Anti-Aging Serum 'Brand B'." The user can refer to the list and purchase the cosmetics that suit them best.

[0102] Specific examples

[0103] For example, a user can input that they are concerned about "wrinkles" and "dryness" and upload an image of their current skin. The device sends this information to a server, which then performs image analysis to numerically evaluate the depth of wrinkles and dryness. The user can then select "moisturized, wrinkle-free skin" as their ideal skin condition, and the server will analyze the gap between their current condition and their ideal condition and search a database for the most suitable cosmetics. For example, it will select products such as "cream containing hyaluronic acid" or "serum containing retinol" and display them to the user as a list.

[0104] Prompt Sentence Examples

[0105] "Please suggest the best skin care items to make my dry and wrinkled skin moisturized and less wrinkled. I will also send you a photo of my current skin, so please analyze it."

[0106] This prompt provides specific instructions to the generative AI model, allowing the system to recommend the most suitable skin care items for the user.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Program processing flow

[0109] Step 1: User Input and Data Submission

[0110] The user launches the dedicated application and selects a skin problem such as "wrinkles" or "dryness" on the problem selection screen (interface). When the user taps the "Next" button, the camera starts up and the user takes an image showing their current skin condition. After that, the user taps the "Upload" button, and the entered problem data and skin image are sent to the server by the device.

[0111] Input: Skin problem selection data and skin images

[0112] Output: Image data and user data sent to the server

[0113] Specifically, the user follows the application's instructions, selects a skin concern, and then takes a skin image using the camera. The device then sends the image to the server, which also sends the data using an HTTP request.

[0114] Step 2: Image analysis by the server

[0115] The server inputs the received image data into an image analysis algorithm (e.g., TensorFlow or PyTorch), which quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, etc., and stores the evaluation results in a database.

[0116] Input: Skin images and problem data sent from the device

[0117] Output: Skin condition evaluation results (e.g., numerical data on wrinkle depth, dryness, and redness distribution)

[0118] Specifically, the server launches an image analysis model, identifies wrinkle depth and dry areas from the image, and evaluates them as numerical data. The analysis results are then recorded in a database.

[0119] Step 3: Select your ideal skin condition

[0120] The server uses the analysis results to provide an interface that asks the user about their ideal skin condition. The user selects their ideal condition, such as "moisturized skin" or "few wrinkles," and taps the "Done" button. The device then sends this selection information to the server.

[0121] Input: Analysis results and options to ask about ideal skin condition

[0122] Output: Data of ideal skin condition selected by the user

[0123] Specifically, the server generates a user interface and provides options to the user. The user selects an ideal state and transmits the data from the terminal to the server.

[0124] Step 4: Gap analysis by server

[0125] The server compares the current skin condition (assessment result) with the ideal skin condition (user selection). The gap is quantified as the amount of wrinkle depth or dryness that needs to be reduced.

[0126] Input: Current skin condition evaluation results and ideal skin condition selected by the user

[0127] Output: Gap analysis results (points of improvement and their numerical data)

[0128] Specifically, the server compares the current state with the desired state, identifies and quantifies areas of improvement that need to be made, and these results are used in the next step.

[0129] Step 5: Database lookup by server

[0130] Based on the results of the gap analysis, the server searches the database for the most suitable cosmetics, such as a cream with high moisturizing properties or a serum with anti-aging properties.

[0131] Input: Gap analysis results

[0132] Output: A list of the best cosmetics

[0133] Specifically, the server queries the database for cosmetic information and identifies suitable items to fill the gaps. The search results are used in the next step.

[0134] Step 6: View recommended items

[0135] The server sends the search results to the terminal, which then displays a list of recommended cosmetics for the user. The list includes product name, effects, and ingredient information, and the user can refer to the list for more information.

[0136] Enter: the best makeup list

[0137] Output: Cosmetics list displayed on the terminal

[0138] Specifically, the server sends the generated list to the device, which then displays the list on the screen, allowing the user to review the list and select appropriate skin care items.

[0139] This series of processes allows users to accurately understand their current skin condition and easily find the skin care items that are best suited to them.

[0140] (Application example 1)

[0141] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] Conventional skin care product recommendation methods make it difficult for users to find the product that best suits their skin condition. Furthermore, product recommendations by sales staff in physical stores are often limited by time and information, leaving users unsatisfied. The present invention aims to improve a system that accurately evaluates a user's skin condition and recommends the most suitable skin care product, thereby improving the user experience in physical stores.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0144] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means for allowing the user to select an ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition; a database search means for the server to search a database for optimal skin care items to fill the gap; a display means for a terminal to display the search results to the user; a means including an application to be installed on a terminal held by a sales staff member in a physical store; and a means including a process for allowing customers to evaluate their skin condition on the spot and recommend optimal skin care products. This improves the user experience in physical stores and enables users to easily find skin care products optimal for their skin condition.

[0145] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[0146] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0147] The "selection means" is a means for enabling the user to select an ideal skin condition.

[0148] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[0149] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0150] The "display means" is a means by which the terminal displays the search results to the user.

[0151] "Means including applications" refers to means using applications installed on terminals held by sales staff in physical stores.

[0152] The "means including a recommendation process" is a means for executing a process in which a customer evaluates their skin condition on the spot and recommends the most suitable skin care product.

[0153] The present invention is a system that evaluates skin condition and smoothly recommends optimal skin care products to improve the user experience in physical stores. This system is mainly composed of a server and a terminal.

[0154] Hardware and Software

[0155] Hardware: Smartphones and servers owned by store staff.

[0156] Software: Applications, image analysis algorithms, machine learning models, deep learning models, databases.

[0157] Data processing and calculation

[0158] 1. Input method: Using the store staff's smartphone, the customer enters their skin concerns (e.g., wrinkles, dryness) in text and takes and uploads an image showing their current skin condition.

[0159] 2. Image analysis: The server analyzes the uploaded images. Specifically, it uses image analysis algorithms to evaluate the current skin condition, such as the depth of wrinkles and dryness. This analysis utilizes machine learning and deep learning models.

[0160] 3. Selection method: Based on the analysis results, the server provides the user with an interface for selecting their ideal skin condition. The user can select their ideal condition on the spot, such as "moisturized, wrinkle-free skin."

[0161] 4. Gap analysis method: The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. By using this gap analysis method, it is possible to identify which skin problems need to be improved and to what extent.

[0162] 5. Database search method: Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each product.

[0163] 6. Display: A list of the most suitable skincare items obtained as a result of the search is displayed on the staff member's smartphone. This list includes detailed information about each product, which the staff member can use to make recommendations to the customer.

[0164] Specific examples

[0165] For example, a customer can enter "wrinkles" and "dryness" as concerns at a store and upload an image of their current skin. The server receives this information and analyzes their skin condition using machine learning and deep learning models. As a result, the customer is asked to select their ideal state, such as "moisturized, wrinkle-free skin," and the gap between their current state and that state is analyzed. Finally, the system searches for the optimal skin care items (e.g., highly moisturizing cream, anti-aging serum) to fill this gap and displays them on the staff member's smartphone.

[0166] Prompt Sentence Examples

[0167] "Explain how the example Python code works in a scenario where a user inputs "wrinkles" and "dryness" as concerns and uploads an image showing their skin condition."

[0168] Through the above process, it is possible to significantly improve the user experience in physical stores.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] The device prompts the user to enter their skin concerns and take and upload an image showing their current skin condition. At this stage, text (e.g., wrinkles, dryness) and image data are obtained as input data. This data is then sent from the device to the server.

[0172] Step 2:

[0173] The server receives the transmitted image and text data. It analyzes the features in the image (e.g., depth of wrinkles, degree of dryness) based on the image analysis algorithm and generative AI model. As a result of the analysis, detailed evaluation information on the skin condition is obtained. The server generates this evaluation information.

[0174] Step 3:

[0175] The server provides an interface that allows the user to select their ideal skin condition based on the evaluation results. The user selects their ideal skin condition (e.g., moisturized and wrinkle-free skin) through their device. This selection information is also sent to the server.

[0176] Step 4:

[0177] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent. Specifically, the degree of wrinkle reduction and the level of relief from dryness are quantified.

[0178] Step 5:

[0179] The server searches the database for optimal skin care items based on the results of the gap analysis. This search takes into account information about the effects and ingredients of each item. As a result, a list of recommended skin care items is generated.

[0180] Step 6:

[0181] The server sends the list of skin care items it has created to the terminal. The terminal displays this list to the user and provides detailed product information (e.g., effects, usage instructions), allowing sales staff to recommend the most suitable products to the user.

[0182] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0183] The present invention is a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, recommendations can be made that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[0184] Input Method

[0185] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0186] Image analysis methods

[0187] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models to evaluate, for example, the depth of wrinkles, the degree of dryness, and the distribution of redness in the image.

[0188] Emotion Engine

[0189] The server is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. The emotion engine works in conjunction with image analysis to evaluate the user's facial expressions and uses voice recognition technology to detect emotions from the user's voice. This allows the system to reflect information such as if the user is feeling stressed in the selection of skin care items.

[0190] Selection method

[0191] The server provides an interface that allows users to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0192] Gap Analysis Tools

[0193] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user, taking into account the emotion evaluation results from the emotion engine. For example, if stress is likely causing dry skin, that information is reflected in the gap analysis.

[0194] Database search methods

[0195] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. This search includes ingredient information, efficacy, and emotional evaluation results from an emotion engine. For example, items containing stress-relieving ingredients may be selected.

[0196] Display means

[0197] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0198] Specific examples

[0199] As a specific example, the following processing is performed.

[0200] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0201] 2. The device sends this information to the server.

[0202] 3. The emotion engine evaluates the user's stress level based on their facial expressions and voice.

[0203] 4. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0204] 5. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0205] 6. The server compares the current state with the ideal state and analyzes the gap, taking into account the stress state evaluated by the emotion engine.

[0206] 7. The server generates a list containing skin care items with stress-relieving effects from the database.

[0207] 8. The device displays a list of recommended items to the user.

[0208] In this way, users can obtain optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. This system allows users to improve their skin condition more effectively and efficiently.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0212] Step 2:

[0213] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0214] Step 3:

[0215] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0216] Step 4:

[0217] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0218] Step 5:

[0219] At the same time, an emotion engine in the server analyzes the user's facial expressions and voice to assess their emotional state (e.g., stress or satisfaction), using facial recognition algorithms and voice analysis technology.

[0220] Step 6:

[0221] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0222] Step 7:

[0223] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0224] Step 8:

[0225] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. The emotion evaluation results from the emotion engine are also reflected in this gap analysis. For example, if stress is likely to be causing dry skin, that information will be taken into consideration.

[0226] Step 9:

[0227] The server uses the gap analysis results to search the database for optimal skin care items. The search criteria include ingredient information, efficacy, and emotional evaluation results. For example, items containing stress-relieving ingredients are selected.

[0228] Step 10:

[0229] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0230] Step 11:

[0231] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0232] In this way, the system can recommend optimal skin care items by comprehensively considering the user's skin concerns, current skin condition, ideal skin condition, and emotional state, providing highly personalized skin care solutions for each individual user.

[0233] Example 2

[0234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] While conventional skincare item recommendation systems can take into account a user's current skin condition and skin concerns, it has been difficult to make recommendations that take into account the user's emotional state. As a result, when emotional factors such as stress or discomfort affect skin problems, it is difficult to select the optimal skincare items.

[0236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0237] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; a transmission means for the terminal to transmit the input data and image data to the server; an image analysis means for the server to analyze the image using a machine learning model or a deep learning model and evaluate the current skin condition; an emotion evaluation means for the server to analyze the user's facial expressions and voice and evaluate their emotional state; a selection means for allowing the user to select their ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition and take the emotion evaluation result into consideration; a database search means for the server to search a database for skin care items optimal for filling the gap; and a display means for the terminal to display the search results to the user. This enables more accurate skin care item recommendations while also taking the user's emotional state into consideration.

[0238] The "input means" is a means by which a user inputs skin concerns and uploads an image showing the current skin condition.

[0239] The "transmission means" is a means by which the terminal transmits input data and image data to the server.

[0240] The "image analysis means" is a means by which the server analyzes an image using a machine learning model or a deep learning model to evaluate the current skin condition.

[0241] The "emotion evaluation means" is a means by which the server analyzes the user's facial expression and voice to evaluate the user's emotional state.

[0242] The "selection means" is a means that allows the user to select an ideal skin condition.

[0243] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition, and also takes into account the emotion evaluation results.

[0244] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0245] The "display means" is a means by which the terminal displays the search results to the user.

[0246] This invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, it is possible to realize recommendations that take the user's emotional state into consideration.

[0247] Hardware and software used

[0248] Hardware

[0249] User device: User device such as a smartphone or tablet

[0250] Server: Cloud server or on-premise server

[0251] software

[0252] Dedicated application: Application installed by the user

[0253] Image analysis algorithms: Machine learning models (TensorFlow, PyTorch, etc.)

[0254] Emotion engine: Emotion recognition software (Amazon Rekognition, IBM Watson®, etc.)

[0255] Database: A database containing information on ingredients and effects of skin care items

[0256] System processing flow

[0257] 1. User input and image upload

[0258] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application offers options such as "wrinkles," "dryness," and "redness," allowing users to select the appropriate concern.

[0259] 2. Data transmission

[0260] The device sends the skin concerns entered by the user and the uploaded images to the server. The data is sent as an HTTP request and is encrypted with SSL / TLS for security.

[0261] 3. Image analysis and emotion evaluation

[0262] The server passes the received image data to an image analysis algorithm. A machine learning model (e.g., TensorFlow or PyTorch) is used to quantify the depth of wrinkles and dryness. At the same time, an emotion engine analyzes the user's facial and voice data to evaluate their emotional state, such as stress or joy. Services such as Amazon Rekognition and IBM Watson can be used for emotion evaluation.

[0263] 4. Providing an interface for selecting ideal skin condition

[0264] Based on the image analysis results and emotion evaluation results, the server generates an interface that allows the user to select their ideal skin condition, and sends it to the device. The user selects their ideal skin condition from options such as "moisturized skin" or "few wrinkles."

[0265] 5. Gap analysis

[0266] The server analyzes the gap between the user's current skin condition and their ideal skin condition. For example, if the user's current skin condition is "dry" and their ideal skin condition is "moisturized," the server calculates the difference in specific numerical values ​​and also takes into account whether stress is a factor.

[0267] 6. Search for skin care items

[0268] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. In addition to ingredient information and efficacy, the server also takes into account the emotional evaluation results. For example, items containing stress-relieving ingredients are picked out.

[0269] 7. Generate and display a list of recommended items

[0270] The server generates a list of selected items and sends it to the device. The device then displays a list of recommended skin care items to the user. The list is displayed in formats such as "highly moisturizing cream," "anti-aging serum," and "moisturizing serum," and also provides detailed information on the effects and ingredients of each item.

[0271] Examples of concrete examples and prompts

[0272] Specific examples

[0273] The user inputs that they are concerned about "wrinkles" and "dryness" and uploads a current image of their skin.

[0274] The device sends this information to the server.

[0275] The emotion engine assesses the user's stress level based on their facial expressions and voice.

[0276] The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0277] The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0278] The server compares the current state with the ideal state and analyzes the gap, taking into account the stress level assessed by the emotion engine.

[0279] The server generates a list containing skin care items with stress-relieving effects from the database.

[0280] The device displays a list of recommended items to the user.

[0281] Prompt Sentence Examples

[0282] "When developing a system to recommend the best skin care items for women in their 30s, we will use an emotion engine to take into account the user's stress level. What technology should we use to recognize emotions from facial images and voice?"

[0283] This system allows users to select the optimal skin care items based on their current skin condition, ideal skin condition, and even their emotional state, allowing them to improve their skin condition more effectively and efficiently.

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Step 1:

[0286] Input: The user inputs their skin concerns and uploads an image showing their current skin condition.

[0287] Processing: The user opens the application, selects a concern from options such as "wrinkles," "dryness," or "redness" on the "Enter Skin Concerns" screen, and uploads an image taken with the camera.

[0288] Output: Skin concerns (text data) and image data of skin condition.

[0289] Specific operation: The user enters their concerns into a form within the application, takes a photo of their current skin condition using their smartphone camera, and uploads the image.

[0290] Step 2:

[0291] Input: Text data of entered skin concerns and uploaded image data.

[0292] Processing: The device sends the entered skin concerns and uploaded image data to the server. The data is sent as an HTTP request and encrypted with SSL / TLS.

[0293] Output: Text data and image data of skin concerns sent to the server.

[0294] Specific operation: The smartphone sends the entered data to a cloud server via an internet connection.

[0295] Step 3:

[0296] Input: Text data and image data of skin concerns sent to the server.

[0297] Processing: The server passes the received image data to an image analysis algorithm, which uses a machine learning model (e.g., TensorFlow or PyTorch) to evaluate the skin condition. Specifically, it quantifies the depth of wrinkles and the degree of dryness.

[0298] Output: Numerical data on skin condition (wrinkle depth, dryness, etc.).

[0299] How it works: A machine learning model running on the server analyzes the images and extracts parameters such as wrinkles and dryness.

[0300] Step 4:

[0301] Input: Text data and image data of skin concerns sent to the server, and numerical data of skin condition.

[0302] Processing: The emotion engine analyzes the user's facial and voice data to assess their emotional state, such as stress or joy, using services such as Amazon Rekognition and IBM Watson.

[0303] Output: Emotional assessment (e.g., quantified stress level).

[0304] Specific operation: The server analyzes facial expressions and voice data through the emotion recognition API and quantifies the emotional state.

[0305] Step 5:

[0306] Input: Numerical data of skin condition and emotion evaluation results.

[0307] Processing: Based on this data, the server generates an interface that allows the user to select their ideal skin condition and sends it to the terminal.

[0308] Output: Interface for selecting ideal skin condition.

[0309] Specific operation: The server generates an interface using HTML / CSS, allowing users to select their ideal skin condition on their smartphone.

[0310] Step 6:

[0311] Input: User-selected ideal skin condition.

[0312] Processing: The server analyzes the gap between the current skin condition and the ideal skin condition, and considers whether stress is a contributing factor.

[0313] Output: Gap analysis results (a numerical value indicating how much skin improvement is needed).

[0314] How it works: The server compares your current skin parameters with your ideal parameters and identifies areas that need improvement.

[0315] Step 7:

[0316] Input: Gap analysis results.

[0317] Processing: The server searches the database for the most suitable skin care items based on the results of the gap analysis, taking into account ingredient information, efficacy, and emotional evaluation results.

[0318] Output: A list of recommended skin care items.

[0319] Specific operation: The server uses SQL to search the database and extract the skin care items that are best suited to the user.

[0320] Step 8:

[0321] Enter: a list of recommended skin care items.

[0322] Processing: The server generates a list of the selected items and sends it to the terminal. The terminal displays the list to the user.

[0323] Output: A list of skin care items displayed on the device.

[0324] Specific operation: The smartphone application converts the data received from the server into a display format and presents it visually to the user.

[0325] In this way, the system allows users to select the optimal skin care items taking into account their current skin condition, ideal skin condition, and emotional state.

[0326] (Application example 2)

[0327] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] Conventional skincare item recommendation systems focus on assessing the user's skin condition, but rarely consider the user's emotional state. This makes it difficult to recommend optimal skincare items because it ignores factors such as stress that affect skin condition. Furthermore, when dealing with customers in physical stores, it is difficult to instantly recommend optimal products to users, limiting the provision of effective services.

[0329] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means through which the user inputs their skin concerns and uploads an image showing their current skin condition, an image analysis means through which the server analyzes the image and evaluates the current skin condition, and an emotion analysis means through which the server evaluates the user's emotional state using an emotion engine. This makes it possible to comprehensively evaluate the user's skin condition and emotional state and recommend optimal skin care items. Furthermore, instantaneous personalized product recommendations can be realized in customer service in physical stores.

[0330] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[0331] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0332] The "selection means" is a means that allows the user to select an ideal skin condition.

[0333] "Emotion analysis means" refers to means for assessing the user's emotional state using an emotion engine.

[0334] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[0335] The "database search means" is a means by which the server searches the database for the most suitable skin care item, taking into account the gap and emotional state.

[0336] The "display means" is a means by which the terminal displays the search results to the user.

[0337] A "machine learning model" is a learning algorithm used to perform image analysis and sentiment analysis.

[0338] "Deep learning models" are advanced neural network models used to perform image and sentiment analysis.

[0339] This invention relates to a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. In particular, it uses an emotion engine to realize recommendations that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[0340] First, the user uses a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. Input methods include a smartphone, tablet, or PC. Skin concerns include options such as "wrinkles," "dryness," and "redness," and the user can select multiple options.

[0341] The device then sends the uploaded image and input data to a server, which is equipped with image analysis tools and uses machine learning and deep learning models to evaluate the current skin condition. Specifically, the server analyzes the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image.

[0342] Furthermore, the server is equipped with an emotion analysis means, which uses an emotion engine to analyze the user's emotional state from their facial expressions and voice. For example, if the user is feeling stressed, this information is reflected in the selection of skin care items. This emotion analysis means generally uses voice recognition technology and facial expression recognition technology.

[0343] Next, the server uses a gap analysis means to analyze the gap between the ideal skin condition selected by the user and the current skin condition, taking into account the results of the emotion analysis. For example, if stress may be causing dry skin, that information will be reflected in the gap analysis.

[0344] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database search tool stores information on ingredients and efficacy of skin care items, and the most suitable items are selected based on a comprehensive evaluation that includes the results of sentiment analysis.

[0345] Finally, the server sends the selected list of skin care items to the terminal, and the list is displayed to the user via the display means. The user can check detailed information about specific skin care items, such as "highly moisturizing cream" and "anti-aging serum."

[0346] A specific example is given below. The user inputs that they are concerned about "wrinkles" and "dryness" and uploads an image of their current skin. The device sends this information to the server, which uses an emotion engine to evaluate the user's stress level based on their facial expressions and voice. Image analysis is used to numerically evaluate the depth of wrinkles and dryness. The user then selects "moisturized, wrinkle-free skin" as their ideal skin condition. The server compares the current condition with the ideal condition and analyzes the gap, taking into account the stress level evaluated by the emotion engine. The server then generates a list from its database that includes skin care items that have a stress-relieving effect, and the device displays this list to the user.

[0347] Here are some example prompts for a generative AI model:

[0348] User's skin care concern: Wrinkles

[0349] User's emotional state: Stress

[0350] Current skin condition:

[0351] Wrinkle depth: Medium

[0352] Dryness: High

[0353] Ideal skin condition: Moisturized skin with few wrinkles

[0354] This system allows users to select the optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. Even when used in physical stores, this system enables instant responses to customers, resulting in more effective presentations.

[0355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0356] Step 1:

[0357] The user inputs their skin concerns using a device with a dedicated application installed. For example, they select the concerns "wrinkles" and "dryness" and upload an image showing their current skin condition. The input data includes information on skin concerns (multiple selections possible) and image data showing the skin condition. Based on this, the device acquires the image data and input data.

[0358] Step 2:

[0359] The acquired data is sent from the device to a server. The server first analyzes the received image data using image analysis means. As a specific example, a machine learning model or deep learning model is used to quantify wrinkle depth, dryness, redness distribution, etc. The input is the uploaded image data, and the output is evaluation data of the skin condition as the analysis result.

[0360] Step 3:

[0361] At the same time, the server uses emotion analysis means to evaluate the user's emotional state from their facial expressions and voice. For example, it determines whether the user is feeling stressed. The technologies used include voice recognition and facial expression recognition. The input is the user's facial expression and voice data, and the output is the evaluation result of the user's emotional state.

[0362] Step 4:

[0363] The user selects their ideal skin condition through the application. Specific options such as "moisturized and wrinkle-free skin" are displayed, and the user selects the desired condition. The input is the user's selection, and the output is the ideal skin condition data.

[0364] Step 5:

[0365] The server compares the current skin condition with the ideal skin condition and analyzes the gap between them, taking into account the evaluation results of the emotional state. The inputs are the skin condition evaluation data, the ideal skin condition data, and the evaluation results of the emotional state, and the output is the gap analysis results.

[0366] Step 6:

[0367] Based on the gap analysis results, the server uses a database search means to search for the most suitable skin care items. The database contains information on the ingredients and effects of each item, and the most suitable items are selected taking into consideration the emotional state. The input is the gap analysis results, and the output is a list of the most suitable skin care items.

[0368] Step 7:

[0369] The list of skin care items generated by the server is sent to the terminal, which displays the list to the user and provides detailed information about the recommended skin care items. The input is the list of skin care items received from the server, and the output is the information displayed to the user.

[0370] Step 8:

[0371] Users can select and purchase skin care items from a displayed list. This includes in-store purchases, allowing users to instantly get the perfect product. The input is the user's selection, and the output is purchase procedure information.

[0372] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0373] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0374] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0375] [Second embodiment]

[0376] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0377] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0378] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0379] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0380] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0381] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0382] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0383] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0384] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0385] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0386] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0387] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0388] The present invention provides a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user's skin closer to ideal. Specific embodiments of this system are described below.

[0389] Input Method

[0390] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0391] Image analysis methods

[0392] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models. For example, the depth of wrinkles, degree of dryness, and distribution of redness in the image are evaluated.

[0393] Selection method

[0394] The server provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can select their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0395] Gap Analysis Tools

[0396] The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent.

[0397] Database search methods

[0398] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each item, and based on this information, the server generates search results. For example, it may select a cream rich in moisturizing ingredients or a serum with anti-aging effects.

[0399] Display means

[0400] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0401] Specific examples

[0402] As a specific example, the following processing is performed.

[0403] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0404] 2. The device sends this information to the server.

[0405] 3. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0406] 4. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0407] 5. The server compares the current state with the ideal state and analyzes the gap.

[0408] 6. The server searches the database for the most suitable skin care items and generates a list.

[0409] 7. The device displays a list of recommended items to the user.

[0410] In this way, users can compare their current skin condition with their ideal skin condition and obtain the skin care items that are best suited to them. This system allows users to easily find the skin care products that are best suited to them, thereby efficiently improving their skin condition.

[0411] The processing flow will be explained below.

[0412] Step 1:

[0413] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0414] Step 2:

[0415] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0416] Step 3:

[0417] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0418] Step 4:

[0419] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0420] Step 5:

[0421] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0422] Step 6:

[0423] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0424] Step 7:

[0425] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. For example, if the current wrinkle depth is deeper than the ideal, the difference in depth is calculated.

[0426] Step 8:

[0427] The server uses the results of the gap analysis to search the database for the most suitable skin care items. The search criteria include information on ingredients and effects.

[0428] Step 9:

[0429] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0430] Step 10:

[0431] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0432] In this way, the specific actions performed in each step allow the user to effectively and efficiently find the skin care items that are best suited to their current skin condition.

[0433] Example 1

[0434] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0435] With conventional methods for selecting skin care items, it is often difficult for users to find the items that are best suited to their skin condition, resulting in ineffective skin care. Furthermore, because users lack specialized knowledge, it is difficult for them to accurately evaluate their own skin condition, and they lack the information to select appropriate products. This has led to users generally purchasing skin care products on their own, which has created the problem of making it difficult to achieve effective skin care.

[0436] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0437] In this invention, the server includes input means for allowing a user to input skin problems and upload an image showing their current skin condition, image analysis means for the server to analyze the image and evaluate the current skin condition, selection means for allowing the user to select an ideal skin condition, gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition, database search means for the server to search a database for cosmetics optimal for filling the gap, display means for the terminal to display the search results to the user, evaluation means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user, and recommendation means for selecting optimal cosmetics based on the ideal skin condition selected by the user. This enables users to accurately evaluate their current skin condition and easily find optimal skin care items.

[0438] The "input means" is a means by which a user inputs skin problems and uploads an image showing the current skin condition.

[0439] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0440] The "selection means" is a means that allows the user to select an ideal skin condition.

[0441] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition.

[0442] The "database search means" is a means by which the server searches the database for the most suitable cosmetics to fill the gap.

[0443] The "display means" is a means by which the terminal displays the search results to the user.

[0444] The "evaluation means" is a means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user.

[0445] The "recommendation means" is a means for selecting the most suitable cosmetics based on the ideal skin condition selected by the user.

[0446] The present invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal cosmetics to bring the user closer to ideal skin. This system is configured as follows.

[0447] First, users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, through which users can select multiple options such as "wrinkles," "dryness," and "redness."

[0448] Next, the device sends the image and input data uploaded by the user to a server. The server receives the image and uses an image analysis algorithm to evaluate the current skin condition. This analysis uses deep learning models such as TensorFlow and PyTorch. The server quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image, and stores the evaluation results in a database.

[0449] The server then provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles." The device then sends this selection information to the server.

[0450] The server then analyzes the gap between the current skin condition and the ideal skin condition. This gap analysis identifies skin problems that need improvement and quantifies the factors for improving each problem. The server then searches a database for the most suitable cosmetics based on the results of the gap analysis. The database stores detailed information on the ingredients and effects of each cosmetic product. The server selects the most suitable cosmetics based on the required ingredients and effects and retrieves the results from the database.

[0451] Finally, the server sends the selected cosmetics list to the terminal. The terminal displays the list to the user and provides detailed information about the recommended cosmetics. For example, it may be displayed in the form of "High Moisturizing Cream 'Brand A'" or "Anti-Aging Serum 'Brand B'." The user can refer to the list and purchase the cosmetics that suit them best.

[0452] Specific examples

[0453] For example, a user can input that they are concerned about "wrinkles" and "dryness" and upload an image of their current skin. The device sends this information to a server, which then performs image analysis to numerically evaluate the depth of wrinkles and dryness. The user can then select "moisturized, wrinkle-free skin" as their ideal skin condition, and the server will analyze the gap between their current condition and their ideal condition and search a database for the most suitable cosmetics. For example, it will select products such as "cream containing hyaluronic acid" or "serum containing retinol" and display them to the user as a list.

[0454] Prompt Sentence Examples

[0455] "Please suggest the best skin care items to make my dry and wrinkled skin moisturized and less wrinkled. I will also send you a photo of my current skin, so please analyze it."

[0456] This prompt provides specific instructions to the generative AI model, allowing the system to recommend the most suitable skin care items for the user.

[0457] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0458] Program processing flow

[0459] Step 1: User Input and Data Submission

[0460] The user launches the dedicated application and selects a skin problem such as "wrinkles" or "dryness" on the problem selection screen (interface). When the user taps the "Next" button, the camera starts up and the user takes an image showing their current skin condition. After that, the user taps the "Upload" button, and the entered problem data and skin image are sent to the server by the device.

[0461] Input: Skin problem selection data and skin images

[0462] Output: Image data and user data sent to the server

[0463] Specifically, the user follows the application's instructions, selects a skin concern, and then takes a skin image using the camera. The device then sends the image to the server, which also sends the data using an HTTP request.

[0464] Step 2: Image analysis by the server

[0465] The server inputs the received image data into an image analysis algorithm (e.g., TensorFlow or PyTorch), which quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, etc., and stores the evaluation results in a database.

[0466] Input: Skin images and problem data sent from the device

[0467] Output: Skin condition evaluation results (e.g., numerical data on wrinkle depth, dryness, and redness distribution)

[0468] Specifically, the server launches an image analysis model, identifies wrinkle depth and dry areas from the image, and evaluates them as numerical data. The analysis results are then recorded in a database.

[0469] Step 3: Select your ideal skin condition

[0470] The server uses the analysis results to provide an interface that asks the user about their ideal skin condition. The user selects their ideal condition, such as "moisturized skin" or "few wrinkles," and taps the "Done" button. The device then sends this selection information to the server.

[0471] Input: Analysis results and options to ask about ideal skin condition

[0472] Output: Data of ideal skin condition selected by the user

[0473] Specifically, the server generates a user interface and provides options to the user. The user selects an ideal state and transmits the data from the terminal to the server.

[0474] Step 4: Gap analysis by server

[0475] The server compares the current skin condition (assessment result) with the ideal skin condition (user selection). The gap is quantified as the amount of wrinkle depth or dryness that needs to be reduced.

[0476] Input: Current skin condition evaluation results and ideal skin condition selected by the user

[0477] Output: Gap analysis results (points of improvement and their numerical data)

[0478] Specifically, the server compares the current state with the desired state, identifies and quantifies areas of improvement that need to be made, and these results are used in the next step.

[0479] Step 5: Database lookup by server

[0480] Based on the results of the gap analysis, the server searches the database for the most suitable cosmetics, such as a cream with high moisturizing properties or a serum with anti-aging properties.

[0481] Input: Gap analysis results

[0482] Output: A list of the best cosmetics

[0483] Specifically, the server queries the database for cosmetic information and identifies suitable items to fill the gaps. The search results are used in the next step.

[0484] Step 6: View recommended items

[0485] The server sends the search results to the terminal, which then displays a list of recommended cosmetics for the user. The list includes product name, effects, and ingredient information, and the user can refer to the list for more information.

[0486] Enter: the best makeup list

[0487] Output: Cosmetics list displayed on the terminal

[0488] Specifically, the server sends the generated list to the device, which then displays the list on the screen, allowing the user to review the list and select appropriate skin care items.

[0489] This series of processes allows users to accurately understand their current skin condition and easily find the skin care items that are best suited to them.

[0490] (Application example 1)

[0491] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0492] Conventional skin care product recommendation methods make it difficult for users to find the product that best suits their skin condition. Furthermore, product recommendations by sales staff in physical stores are often limited by time and information, leaving users unsatisfied. The present invention aims to improve a system that accurately evaluates a user's skin condition and recommends the most suitable skin care product, thereby improving the user experience in physical stores.

[0493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0494] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means for allowing the user to select an ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition; a database search means for the server to search a database for optimal skin care items to fill the gap; a display means for a terminal to display the search results to the user; a means including an application to be installed on a terminal held by a sales staff member in a physical store; and a means including a process for allowing customers to evaluate their skin condition on the spot and recommend optimal skin care products. This improves the user experience in physical stores and enables users to easily find skin care products optimal for their skin condition.

[0495] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[0496] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0497] The "selection means" is a means for enabling the user to select an ideal skin condition.

[0498] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[0499] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0500] The "display means" is a means by which the terminal displays the search results to the user.

[0501] "Means including applications" refers to means using applications installed on terminals held by sales staff in physical stores.

[0502] The "means including a recommendation process" is a means for executing a process in which a customer evaluates their skin condition on the spot and recommends the most suitable skin care product.

[0503] The present invention is a system that evaluates skin condition and smoothly recommends optimal skin care products to improve the user experience in physical stores. This system is mainly composed of a server and a terminal.

[0504] Hardware and Software

[0505] Hardware: Smartphones and servers owned by store staff.

[0506] Software: Applications, image analysis algorithms, machine learning models, deep learning models, databases.

[0507] Data processing and calculation

[0508] 1. Input method: Using the store staff's smartphone, the customer enters their skin concerns (e.g., wrinkles, dryness) in text and takes and uploads an image showing their current skin condition.

[0509] 2. Image analysis: The server analyzes the uploaded images. Specifically, it uses image analysis algorithms to evaluate the current skin condition, such as the depth of wrinkles and dryness. This analysis utilizes machine learning and deep learning models.

[0510] 3. Selection method: Based on the analysis results, the server provides the user with an interface for selecting their ideal skin condition. The user can select their ideal condition on the spot, such as "moisturized, wrinkle-free skin."

[0511] 4. Gap analysis method: The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. By using this gap analysis method, it is possible to identify which skin problems need to be improved and to what extent.

[0512] 5. Database search method: Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each product.

[0513] 6. Display: A list of the most suitable skincare items obtained as a result of the search is displayed on the staff member's smartphone. This list includes detailed information about each product, which the staff member can use to make recommendations to the customer.

[0514] Specific examples

[0515] For example, a customer can enter "wrinkles" and "dryness" as concerns at a store and upload an image of their current skin. The server receives this information and analyzes their skin condition using machine learning and deep learning models. As a result, the customer is asked to select their ideal state, such as "moisturized, wrinkle-free skin," and the gap between their current state and that state is analyzed. Finally, the system searches for the optimal skin care items (e.g., highly moisturizing cream, anti-aging serum) to fill this gap and displays them on the staff member's smartphone.

[0516] Prompt Sentence Examples

[0517] "Explain how the example Python code works in a scenario where a user inputs "wrinkles" and "dryness" as concerns and uploads an image showing their skin condition."

[0518] Through the above process, it is possible to significantly improve the user experience in physical stores.

[0519] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0520] Step 1:

[0521] The device prompts the user to enter their skin concerns and take and upload an image showing their current skin condition. At this stage, text (e.g., wrinkles, dryness) and image data are obtained as input data. This data is then sent from the device to the server.

[0522] Step 2:

[0523] The server receives the transmitted image and text data. It analyzes the features in the image (e.g., depth of wrinkles, degree of dryness) based on the image analysis algorithm and generative AI model. As a result of the analysis, detailed evaluation information on the skin condition is obtained. The server generates this evaluation information.

[0524] Step 3:

[0525] The server provides an interface that allows the user to select their ideal skin condition based on the evaluation results. The user selects their ideal skin condition (e.g., moisturized and wrinkle-free skin) through their device. This selection information is also sent to the server.

[0526] Step 4:

[0527] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent. Specifically, the degree of wrinkle reduction and the level of relief from dryness are quantified.

[0528] Step 5:

[0529] The server searches the database for optimal skin care items based on the results of the gap analysis. This search takes into account information about the effects and ingredients of each item. As a result, a list of recommended skin care items is generated.

[0530] Step 6:

[0531] The server sends the list of skin care items it has created to the terminal. The terminal displays this list to the user and provides detailed product information (e.g., effects, usage instructions), allowing sales staff to recommend the most suitable products to the user.

[0532] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0533] The present invention is a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, recommendations can be made that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[0534] Input Method

[0535] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0536] Image analysis methods

[0537] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models to evaluate, for example, the depth of wrinkles, the degree of dryness, and the distribution of redness in the image.

[0538] Emotion Engine

[0539] The server is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. The emotion engine works in conjunction with image analysis to evaluate the user's facial expressions and uses voice recognition technology to detect emotions from the user's voice. This allows the system to reflect information such as if the user is feeling stressed in the selection of skin care items.

[0540] Selection method

[0541] The server provides an interface that allows users to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0542] Gap Analysis Tools

[0543] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user, taking into account the emotion evaluation results from the emotion engine. For example, if stress is likely causing dry skin, that information is reflected in the gap analysis.

[0544] Database search methods

[0545] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. This search includes ingredient information, efficacy, and emotional evaluation results from an emotion engine. For example, items containing stress-relieving ingredients may be selected.

[0546] Display means

[0547] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0548] Specific examples

[0549] As a specific example, the following processing is performed.

[0550] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0551] 2. The device sends this information to the server.

[0552] 3. The emotion engine evaluates the user's stress level based on their facial expressions and voice.

[0553] 4. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0554] 5. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0555] 6. The server compares the current state with the ideal state and analyzes the gap, taking into account the stress state evaluated by the emotion engine.

[0556] 7. The server generates a list containing skin care items with stress-relieving effects from the database.

[0557] 8. The device displays a list of recommended items to the user.

[0558] In this way, users can obtain optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. This system allows users to improve their skin condition more effectively and efficiently.

[0559] The processing flow will be explained below.

[0560] Step 1:

[0561] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0562] Step 2:

[0563] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0564] Step 3:

[0565] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0566] Step 4:

[0567] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0568] Step 5:

[0569] At the same time, an emotion engine in the server analyzes the user's facial expressions and voice to assess their emotional state (e.g., stress or satisfaction), using facial recognition algorithms and voice analysis technology.

[0570] Step 6:

[0571] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0572] Step 7:

[0573] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0574] Step 8:

[0575] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. The emotion evaluation results from the emotion engine are also reflected in this gap analysis. For example, if stress is likely to be causing dry skin, that information will be taken into consideration.

[0576] Step 9:

[0577] The server uses the gap analysis results to search the database for optimal skin care items. The search criteria include ingredient information, efficacy, and emotional evaluation results. For example, items containing stress-relieving ingredients are selected.

[0578] Step 10:

[0579] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0580] Step 11:

[0581] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0582] In this way, the system can recommend optimal skin care items by comprehensively considering the user's skin concerns, current skin condition, ideal skin condition, and emotional state, providing highly personalized skin care solutions for each individual user.

[0583] Example 2

[0584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] While conventional skincare item recommendation systems can take into account a user's current skin condition and skin concerns, it has been difficult to make recommendations that take into account the user's emotional state. As a result, when emotional factors such as stress or discomfort affect skin problems, it is difficult to select the optimal skincare items.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0587] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; a transmission means for the terminal to transmit the input data and image data to the server; an image analysis means for the server to analyze the image using a machine learning model or a deep learning model and evaluate the current skin condition; an emotion evaluation means for the server to analyze the user's facial expressions and voice and evaluate their emotional state; a selection means for allowing the user to select their ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition and take the emotion evaluation result into consideration; a database search means for the server to search a database for skin care items optimal for filling the gap; and a display means for the terminal to display the search results to the user. This enables more accurate skin care item recommendations while also taking the user's emotional state into consideration.

[0588] The "input means" is a means by which a user inputs skin concerns and uploads an image showing the current skin condition.

[0589] The "transmission means" is a means by which the terminal transmits input data and image data to the server.

[0590] The "image analysis means" is a means by which the server analyzes an image using a machine learning model or a deep learning model to evaluate the current skin condition.

[0591] The "emotion evaluation means" is a means by which the server analyzes the user's facial expression and voice to evaluate the user's emotional state.

[0592] The "selection means" is a means that allows the user to select an ideal skin condition.

[0593] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition, and also takes into account the emotion evaluation results.

[0594] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0595] The "display means" is a means by which the terminal displays the search results to the user.

[0596] This invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, it is possible to realize recommendations that take the user's emotional state into consideration.

[0597] Hardware and software used

[0598] Hardware

[0599] User device: User device such as a smartphone or tablet

[0600] Server: Cloud server or on-premise server

[0601] software

[0602] Dedicated application: Application installed by the user

[0603] Image analysis algorithms: Machine learning models (TensorFlow, PyTorch, etc.)

[0604] Emotion engine: Emotion recognition software (Amazon Rekognition, IBM Watson, etc.)

[0605] Database: A database containing information on ingredients and effects of skin care items

[0606] System processing flow

[0607] 1. User input and image upload

[0608] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application offers options such as "wrinkles," "dryness," and "redness," allowing users to select the appropriate concern.

[0609] 2. Data transmission

[0610] The device sends the skin concerns entered by the user and the uploaded images to the server. The data is sent as an HTTP request and is encrypted with SSL / TLS for security.

[0611] 3. Image analysis and emotion evaluation

[0612] The server passes the received image data to an image analysis algorithm. A machine learning model (e.g., TensorFlow or PyTorch) is used to quantify the depth of wrinkles and dryness. At the same time, an emotion engine analyzes the user's facial and voice data to evaluate their emotional state, such as stress or joy. Services such as Amazon Rekognition and IBM Watson can be used for emotion evaluation.

[0613] 4. Providing an interface for selecting ideal skin condition

[0614] Based on the image analysis results and emotion evaluation results, the server generates an interface that allows the user to select their ideal skin condition, and sends it to the device. The user selects their ideal skin condition from options such as "moisturized skin" or "few wrinkles."

[0615] 5. Gap analysis

[0616] The server analyzes the gap between the user's current skin condition and their ideal skin condition. For example, if the user's current skin condition is "dry" and their ideal skin condition is "moisturized," the server calculates the difference in specific numerical values ​​and also takes into account whether stress is a factor.

[0617] 6. Search for skin care items

[0618] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. In addition to ingredient information and efficacy, the server also takes into account the emotional evaluation results. For example, items containing stress-relieving ingredients are picked out.

[0619] 7. Generate and display a list of recommended items

[0620] The server generates a list of selected items and sends it to the device. The device then displays a list of recommended skin care items to the user. The list is displayed in formats such as "highly moisturizing cream," "anti-aging serum," and "moisturizing serum," and also provides detailed information on the effects and ingredients of each item.

[0621] Examples of concrete examples and prompts

[0622] Specific examples

[0623] The user inputs that they are concerned about "wrinkles" and "dryness" and uploads a current image of their skin.

[0624] The device sends this information to the server.

[0625] The emotion engine assesses the user's stress level based on their facial expressions and voice.

[0626] The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0627] The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0628] The server compares the current state with the ideal state and analyzes the gap, taking into account the stress level assessed by the emotion engine.

[0629] The server generates a list containing skin care items with stress-relieving effects from the database.

[0630] The device displays a list of recommended items to the user.

[0631] Prompt Sentence Examples

[0632] "When developing a system to recommend the best skin care items for women in their 30s, we will use an emotion engine to take into account the user's stress level. What technology should we use to recognize emotions from facial images and voice?"

[0633] This system allows users to select the optimal skin care items based on their current skin condition, ideal skin condition, and even their emotional state, allowing them to improve their skin condition more effectively and efficiently.

[0634] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0635] Step 1:

[0636] Input: The user inputs their skin concerns and uploads an image showing their current skin condition.

[0637] Processing: The user opens the application, selects a concern from options such as "wrinkles," "dryness," or "redness" on the "Enter Skin Concerns" screen, and uploads an image taken with the camera.

[0638] Output: Skin concerns (text data) and image data of skin condition.

[0639] Specific operation: The user enters their concerns into a form within the application, takes a photo of their current skin condition using their smartphone camera, and uploads the image.

[0640] Step 2:

[0641] Input: Text data of entered skin concerns and uploaded image data.

[0642] Processing: The device sends the entered skin concerns and uploaded image data to the server. The data is sent as an HTTP request and encrypted with SSL / TLS.

[0643] Output: Text data and image data of skin concerns sent to the server.

[0644] Specific operation: The smartphone sends the entered data to a cloud server via an internet connection.

[0645] Step 3:

[0646] Input: Text data and image data of skin concerns sent to the server.

[0647] Processing: The server passes the received image data to an image analysis algorithm, which uses a machine learning model (e.g., TensorFlow or PyTorch) to evaluate the skin condition. Specifically, it quantifies the depth of wrinkles and the degree of dryness.

[0648] Output: Numerical data on skin condition (wrinkle depth, dryness, etc.).

[0649] How it works: A machine learning model running on the server analyzes the images and extracts parameters such as wrinkles and dryness.

[0650] Step 4:

[0651] Input: Text data and image data of skin concerns sent to the server, and numerical data of skin condition.

[0652] Processing: The emotion engine analyzes the user's facial and voice data to assess their emotional state, such as stress or joy, using services such as Amazon Rekognition and IBM Watson.

[0653] Output: Emotional assessment (e.g., quantified stress level).

[0654] Specific operation: The server analyzes facial expressions and voice data through the emotion recognition API and quantifies the emotional state.

[0655] Step 5:

[0656] Input: Numerical data of skin condition and emotion evaluation results.

[0657] Processing: Based on this data, the server generates an interface that allows the user to select their ideal skin condition and sends it to the terminal.

[0658] Output: Interface for selecting ideal skin condition.

[0659] Specific operation: The server generates an interface using HTML / CSS, allowing users to select their ideal skin condition on their smartphone.

[0660] Step 6:

[0661] Input: User-selected ideal skin condition.

[0662] Processing: The server analyzes the gap between the current skin condition and the ideal skin condition, and considers whether stress is a contributing factor.

[0663] Output: Gap analysis results (a numerical value indicating how much skin improvement is needed).

[0664] How it works: The server compares your current skin parameters with your ideal parameters and identifies areas that need improvement.

[0665] Step 7:

[0666] Input: Gap analysis results.

[0667] Processing: The server searches the database for the most suitable skin care items based on the results of the gap analysis, taking into account ingredient information, efficacy, and emotional evaluation results.

[0668] Output: A list of recommended skin care items.

[0669] Specific operation: The server uses SQL to search the database and extract the skin care items that are best suited to the user.

[0670] Step 8:

[0671] Enter: a list of recommended skin care items.

[0672] Processing: The server generates a list of the selected items and sends it to the terminal. The terminal displays the list to the user.

[0673] Output: A list of skin care items displayed on the device.

[0674] Specific operation: The smartphone application converts the data received from the server into a display format and presents it visually to the user.

[0675] In this way, the system allows users to select the optimal skin care items taking into account their current skin condition, ideal skin condition, and emotional state.

[0676] (Application example 2)

[0677] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] Conventional skincare item recommendation systems focus on assessing the user's skin condition, but rarely consider the user's emotional state. This makes it difficult to recommend optimal skincare items because it ignores factors such as stress that affect skin condition. Furthermore, when dealing with customers in physical stores, it is difficult to instantly recommend optimal products to users, limiting the provision of effective services.

[0679] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means through which the user inputs their skin concerns and uploads an image showing their current skin condition, an image analysis means through which the server analyzes the image and evaluates the current skin condition, and an emotion analysis means through which the server evaluates the user's emotional state using an emotion engine. This makes it possible to comprehensively evaluate the user's skin condition and emotional state and recommend optimal skin care items. Furthermore, instantaneous personalized product recommendations can be realized in customer service in physical stores.

[0680] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[0681] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0682] The "selection means" is a means that allows the user to select an ideal skin condition.

[0683] "Emotion analysis means" refers to means for assessing the user's emotional state using an emotion engine.

[0684] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[0685] The "database search means" is a means by which the server searches the database for the most suitable skin care item, taking into account the gap and emotional state.

[0686] The "display means" is a means by which the terminal displays the search results to the user.

[0687] A "machine learning model" is a learning algorithm used to perform image analysis and sentiment analysis.

[0688] "Deep learning models" are advanced neural network models used to perform image and sentiment analysis.

[0689] This invention relates to a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. In particular, it uses an emotion engine to realize recommendations that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[0690] First, the user uses a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. Input methods include a smartphone, tablet, or PC. Skin concerns include options such as "wrinkles," "dryness," and "redness," and the user can select multiple options.

[0691] The device then sends the uploaded image and input data to a server, which is equipped with image analysis tools and uses machine learning and deep learning models to evaluate the current skin condition. Specifically, the server analyzes the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image.

[0692] Furthermore, the server is equipped with an emotion analysis means, which uses an emotion engine to analyze the user's emotional state from their facial expressions and voice. For example, if the user is feeling stressed, this information is reflected in the selection of skin care items. This emotion analysis means generally uses voice recognition technology and facial expression recognition technology.

[0693] Next, the server uses a gap analysis means to analyze the gap between the ideal skin condition selected by the user and the current skin condition, taking into account the results of the emotion analysis. For example, if stress may be causing dry skin, that information will be reflected in the gap analysis.

[0694] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database search tool stores information on ingredients and efficacy of skin care items, and the most suitable items are selected based on a comprehensive evaluation that includes the results of sentiment analysis.

[0695] Finally, the server sends the selected list of skin care items to the terminal, and the list is displayed to the user via the display means. The user can check detailed information about specific skin care items, such as "highly moisturizing cream" and "anti-aging serum."

[0696] A specific example is given below. The user inputs that they are concerned about "wrinkles" and "dryness" and uploads an image of their current skin. The device sends this information to the server, which uses an emotion engine to evaluate the user's stress level based on their facial expressions and voice. Image analysis is used to numerically evaluate the depth of wrinkles and dryness. The user then selects "moisturized, wrinkle-free skin" as their ideal skin condition. The server compares the current condition with the ideal condition and analyzes the gap, taking into account the stress level evaluated by the emotion engine. The server then generates a list from its database that includes skin care items that have a stress-relieving effect, and the device displays this list to the user.

[0697] Here are some example prompts for a generative AI model:

[0698] User's skin care concern: Wrinkles

[0699] User's emotional state: Stress

[0700] Current skin condition:

[0701] Wrinkle depth: Medium

[0702] Dryness: High

[0703] Ideal skin condition: Moisturized skin with few wrinkles

[0704] This system allows users to select the optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. Even when used in physical stores, this system enables instant responses to customers, resulting in more effective presentations.

[0705] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0706] Step 1:

[0707] The user inputs their skin concerns using a device with a dedicated application installed. For example, they select the concerns "wrinkles" and "dryness" and upload an image showing their current skin condition. The input data includes information on skin concerns (multiple selections possible) and image data showing the skin condition. Based on this, the device acquires the image data and input data.

[0708] Step 2:

[0709] The acquired data is sent from the device to a server. The server first analyzes the received image data using image analysis means. As a specific example, a machine learning model or deep learning model is used to quantify wrinkle depth, dryness, redness distribution, etc. The input is the uploaded image data, and the output is evaluation data of the skin condition as the analysis result.

[0710] Step 3:

[0711] At the same time, the server uses emotion analysis means to evaluate the user's emotional state from their facial expressions and voice. For example, it determines whether the user is feeling stressed. The technologies used include voice recognition and facial expression recognition. The input is the user's facial expression and voice data, and the output is the evaluation result of the user's emotional state.

[0712] Step 4:

[0713] The user selects their ideal skin condition through the application. Specific options such as "moisturized and wrinkle-free skin" are displayed, and the user selects the desired condition. The input is the user's selection, and the output is the ideal skin condition data.

[0714] Step 5:

[0715] The server compares the current skin condition with the ideal skin condition and analyzes the gap between them, taking into account the evaluation results of the emotional state. The inputs are the skin condition evaluation data, the ideal skin condition data, and the evaluation results of the emotional state, and the output is the gap analysis results.

[0716] Step 6:

[0717] Based on the gap analysis results, the server uses a database search means to search for the most suitable skin care items. The database contains information on the ingredients and effects of each item, and the most suitable items are selected taking into consideration the emotional state. The input is the gap analysis results, and the output is a list of the most suitable skin care items.

[0718] Step 7:

[0719] The list of skin care items generated by the server is sent to the terminal, which displays the list to the user and provides detailed information about the recommended skin care items. The input is the list of skin care items received from the server, and the output is the information displayed to the user.

[0720] Step 8:

[0721] Users can select and purchase skin care items from a displayed list. This includes in-store purchases, allowing users to instantly get the perfect product. The input is the user's selection, and the output is purchase procedure information.

[0722] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0723] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0724] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0725] [Third embodiment]

[0726] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0727] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0728] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0729] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0730] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0731] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0732] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0733] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0734] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0735] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0736] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0737] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0738] The present invention provides a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user's skin closer to ideal. Specific embodiments of this system are described below.

[0739] Input Method

[0740] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0741] Image analysis methods

[0742] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models. For example, the depth of wrinkles, degree of dryness, and distribution of redness in the image are evaluated.

[0743] Selection method

[0744] The server provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can select their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0745] Gap Analysis Tools

[0746] The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent.

[0747] Database search methods

[0748] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each item, and based on this information, the server generates search results. For example, it may select a cream rich in moisturizing ingredients or a serum with anti-aging effects.

[0749] Display means

[0750] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0751] Specific examples

[0752] As a specific example, the following processing is performed.

[0753] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0754] 2. The device sends this information to the server.

[0755] 3. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0756] 4. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0757] 5. The server compares the current state with the ideal state and analyzes the gap.

[0758] 6. The server searches the database for the most suitable skin care items and generates a list.

[0759] 7. The device displays a list of recommended items to the user.

[0760] In this way, users can compare their current skin condition with their ideal skin condition and obtain the skin care items that are best suited to them. This system allows users to easily find the skin care products that are best suited to them, thereby efficiently improving their skin condition.

[0761] The processing flow will be explained below.

[0762] Step 1:

[0763] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0764] Step 2:

[0765] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0766] Step 3:

[0767] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0768] Step 4:

[0769] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0770] Step 5:

[0771] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0772] Step 6:

[0773] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0774] Step 7:

[0775] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. For example, if the current wrinkle depth is deeper than the ideal, the difference in depth is calculated.

[0776] Step 8:

[0777] The server uses the results of the gap analysis to search the database for the most suitable skin care items. The search criteria include information on ingredients and effects.

[0778] Step 9:

[0779] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0780] Step 10:

[0781] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0782] In this way, the specific actions performed in each step allow the user to effectively and efficiently find the skin care items that are best suited to their current skin condition.

[0783] Example 1

[0784] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0785] With conventional methods for selecting skin care items, it is often difficult for users to find the items that are best suited to their skin condition, resulting in ineffective skin care. Furthermore, because users lack specialized knowledge, it is difficult for them to accurately evaluate their own skin condition, and they lack the information to select appropriate products. This has led to users generally purchasing skin care products on their own, which has created the problem of making it difficult to achieve effective skin care.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0787] In this invention, the server includes input means for allowing a user to input skin problems and upload an image showing their current skin condition, image analysis means for the server to analyze the image and evaluate the current skin condition, selection means for allowing the user to select an ideal skin condition, gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition, database search means for the server to search a database for cosmetics optimal for filling the gap, display means for the terminal to display the search results to the user, evaluation means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user, and recommendation means for selecting optimal cosmetics based on the ideal skin condition selected by the user. This enables users to accurately evaluate their current skin condition and easily find optimal skin care items.

[0788] The "input means" is a means by which a user inputs skin problems and uploads an image showing the current skin condition.

[0789] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0790] The "selection means" is a means that allows the user to select an ideal skin condition.

[0791] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition.

[0792] The "database search means" is a means by which the server searches the database for the most suitable cosmetics to fill the gap.

[0793] The "display means" is a means by which the terminal displays the search results to the user.

[0794] The "evaluation means" is a means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user.

[0795] The "recommendation means" is a means for selecting the most suitable cosmetics based on the ideal skin condition selected by the user.

[0796] The present invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal cosmetics to bring the user closer to ideal skin. This system is configured as follows.

[0797] First, users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, through which users can select multiple options such as "wrinkles," "dryness," and "redness."

[0798] Next, the device sends the image and input data uploaded by the user to a server. The server receives the image and uses an image analysis algorithm to evaluate the current skin condition. This analysis uses deep learning models such as TensorFlow and PyTorch. The server quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image, and stores the evaluation results in a database.

[0799] The server then provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles." The device then sends this selection information to the server.

[0800] The server then analyzes the gap between the current skin condition and the ideal skin condition. This gap analysis identifies skin problems that need improvement and quantifies the factors for improving each problem. The server then searches a database for the most suitable cosmetics based on the results of the gap analysis. The database stores detailed information on the ingredients and effects of each cosmetic product. The server selects the most suitable cosmetics based on the required ingredients and effects and retrieves the results from the database.

[0801] Finally, the server sends the selected cosmetics list to the terminal. The terminal displays the list to the user and provides detailed information about the recommended cosmetics. For example, it may be displayed in the form of "High Moisturizing Cream 'Brand A'" or "Anti-Aging Serum 'Brand B'." The user can refer to the list and purchase the cosmetics that suit them best.

[0802] Specific examples

[0803] For example, a user can input that they are concerned about "wrinkles" and "dryness" and upload an image of their current skin. The device sends this information to a server, which then performs image analysis to numerically evaluate the depth of wrinkles and dryness. The user can then select "moisturized, wrinkle-free skin" as their ideal skin condition, and the server will analyze the gap between their current condition and their ideal condition and search a database for the most suitable cosmetics. For example, it will select products such as "cream containing hyaluronic acid" or "serum containing retinol" and display them to the user as a list.

[0804] Prompt Sentence Examples

[0805] "Please suggest the best skin care items to make my dry and wrinkled skin moisturized and less wrinkled. I will also send you a photo of my current skin, so please analyze it."

[0806] This prompt provides specific instructions to the generative AI model, allowing the system to recommend the most suitable skin care items for the user.

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Program processing flow

[0809] Step 1: User Input and Data Submission

[0810] The user launches the dedicated application and selects a skin problem such as "wrinkles" or "dryness" on the problem selection screen (interface). When the user taps the "Next" button, the camera starts up and the user takes an image showing their current skin condition. After that, the user taps the "Upload" button, and the entered problem data and skin image are sent to the server by the device.

[0811] Input: Skin problem selection data and skin images

[0812] Output: Image data and user data sent to the server

[0813] Specifically, the user follows the application's instructions, selects a skin concern, and then takes a skin image using the camera. The device then sends the image to the server, which also sends the data using an HTTP request.

[0814] Step 2: Image analysis by the server

[0815] The server inputs the received image data into an image analysis algorithm (e.g., TensorFlow or PyTorch), which quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, etc., and stores the evaluation results in a database.

[0816] Input: Skin images and problem data sent from the device

[0817] Output: Skin condition evaluation results (e.g., numerical data on wrinkle depth, dryness, and redness distribution)

[0818] Specifically, the server launches an image analysis model, identifies wrinkle depth and dry areas from the image, and evaluates them as numerical data. The analysis results are then recorded in a database.

[0819] Step 3: Select your ideal skin condition

[0820] The server uses the analysis results to provide an interface that asks the user about their ideal skin condition. The user selects their ideal condition, such as "moisturized skin" or "few wrinkles," and taps the "Done" button. The device then sends this selection information to the server.

[0821] Input: Analysis results and options to ask about ideal skin condition

[0822] Output: Data of ideal skin condition selected by the user

[0823] Specifically, the server generates a user interface and provides options to the user. The user selects an ideal state and transmits the data from the terminal to the server.

[0824] Step 4: Gap analysis by server

[0825] The server compares the current skin condition (assessment result) with the ideal skin condition (user selection). The gap is quantified as the amount of wrinkle depth or dryness that needs to be reduced.

[0826] Input: Current skin condition evaluation results and ideal skin condition selected by the user

[0827] Output: Gap analysis results (points of improvement and their numerical data)

[0828] Specifically, the server compares the current state with the desired state, identifies and quantifies areas of improvement that need to be made, and these results are used in the next step.

[0829] Step 5: Database lookup by server

[0830] Based on the results of the gap analysis, the server searches the database for the most suitable cosmetics, such as a cream with high moisturizing properties or a serum with anti-aging properties.

[0831] Input: Gap analysis results

[0832] Output: A list of the best cosmetics

[0833] Specifically, the server queries the database for cosmetic information and identifies suitable items to fill the gaps. The search results are used in the next step.

[0834] Step 6: View recommended items

[0835] The server sends the search results to the terminal, which then displays a list of recommended cosmetics for the user. The list includes product name, effects, and ingredient information, and the user can refer to the list for more information.

[0836] Enter: the best makeup list

[0837] Output: Cosmetics list displayed on the terminal

[0838] Specifically, the server sends the generated list to the device, which then displays the list on the screen, allowing the user to review the list and select appropriate skin care items.

[0839] This series of processes allows users to accurately understand their current skin condition and easily find the skin care items that are best suited to them.

[0840] (Application example 1)

[0841] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0842] Conventional skin care product recommendation methods make it difficult for users to find the product that best suits their skin condition. Furthermore, product recommendations by sales staff in physical stores are often limited by time and information, leaving users unsatisfied. The present invention aims to improve a system that accurately evaluates a user's skin condition and recommends the most suitable skin care product, thereby improving the user experience in physical stores.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0844] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means for allowing the user to select an ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition; a database search means for the server to search a database for optimal skin care items to fill the gap; a display means for a terminal to display the search results to the user; a means including an application to be installed on a terminal held by a sales staff member in a physical store; and a means including a process for allowing customers to evaluate their skin condition on the spot and recommend optimal skin care products. This improves the user experience in physical stores and enables users to easily find skin care products optimal for their skin condition.

[0845] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[0846] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[0847] The "selection means" is a means for enabling the user to select an ideal skin condition.

[0848] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[0849] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0850] The "display means" is a means by which the terminal displays the search results to the user.

[0851] "Means including applications" refers to means using applications installed on terminals held by sales staff in physical stores.

[0852] The "means including a recommendation process" is a means for executing a process in which a customer evaluates their skin condition on the spot and recommends the most suitable skin care product.

[0853] The present invention is a system that evaluates skin condition and smoothly recommends optimal skin care products to improve the user experience in physical stores. This system is mainly composed of a server and a terminal.

[0854] Hardware and Software

[0855] Hardware: Smartphones and servers owned by store staff.

[0856] Software: Applications, image analysis algorithms, machine learning models, deep learning models, databases.

[0857] Data processing and calculation

[0858] 1. Input method: Using the store staff's smartphone, the customer enters their skin concerns (e.g., wrinkles, dryness) in text and takes and uploads an image showing their current skin condition.

[0859] 2. Image analysis: The server analyzes the uploaded images. Specifically, it uses image analysis algorithms to evaluate the current skin condition, such as the depth of wrinkles and dryness. This analysis utilizes machine learning and deep learning models.

[0860] 3. Selection method: Based on the analysis results, the server provides the user with an interface for selecting their ideal skin condition. The user can select their ideal condition on the spot, such as "moisturized, wrinkle-free skin."

[0861] 4. Gap analysis method: The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. By using this gap analysis method, it is possible to identify which skin problems need to be improved and to what extent.

[0862] 5. Database search method: Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each product.

[0863] 6. Display: A list of the most suitable skincare items obtained as a result of the search is displayed on the staff member's smartphone. This list includes detailed information about each product, which the staff member can use to make recommendations to the customer.

[0864] Specific examples

[0865] For example, a customer can enter "wrinkles" and "dryness" as concerns at a store and upload an image of their current skin. The server receives this information and analyzes their skin condition using machine learning and deep learning models. As a result, the customer is asked to select their ideal state, such as "moisturized, wrinkle-free skin," and the gap between their current state and that state is analyzed. Finally, the system searches for the optimal skin care items (e.g., highly moisturizing cream, anti-aging serum) to fill this gap and displays them on the staff member's smartphone.

[0866] Prompt Sentence Examples

[0867] "Explain how the example Python code works in a scenario where a user inputs "wrinkles" and "dryness" as concerns and uploads an image showing their skin condition."

[0868] Through the above process, it is possible to significantly improve the user experience in physical stores.

[0869] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0870] Step 1:

[0871] The device prompts the user to enter their skin concerns and take and upload an image showing their current skin condition. At this stage, text (e.g., wrinkles, dryness) and image data are obtained as input data. This data is then sent from the device to the server.

[0872] Step 2:

[0873] The server receives the transmitted image and text data. It analyzes the features in the image (e.g., depth of wrinkles, degree of dryness) based on the image analysis algorithm and generative AI model. As a result of the analysis, detailed evaluation information on the skin condition is obtained. The server generates this evaluation information.

[0874] Step 3:

[0875] The server provides an interface that allows the user to select their ideal skin condition based on the evaluation results. The user selects their ideal skin condition (e.g., moisturized and wrinkle-free skin) through their device. This selection information is also sent to the server.

[0876] Step 4:

[0877] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent. Specifically, the degree of wrinkle reduction and the level of relief from dryness are quantified.

[0878] Step 5:

[0879] The server searches the database for optimal skin care items based on the results of the gap analysis. This search takes into account information about the effects and ingredients of each item. As a result, a list of recommended skin care items is generated.

[0880] Step 6:

[0881] The server sends the list of skin care items it has created to the terminal. The terminal displays this list to the user and provides detailed product information (e.g., effects, usage instructions), allowing sales staff to recommend the most suitable products to the user.

[0882] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0883] The present invention is a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, recommendations can be made that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[0884] Input Method

[0885] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[0886] Image analysis methods

[0887] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models to evaluate, for example, the depth of wrinkles, the degree of dryness, and the distribution of redness in the image.

[0888] Emotion Engine

[0889] The server is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. The emotion engine works in conjunction with image analysis to evaluate the user's facial expressions and uses voice recognition technology to detect emotions from the user's voice. This allows the system to reflect information such as if the user is feeling stressed in the selection of skin care items.

[0890] Selection method

[0891] The server provides an interface that allows users to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[0892] Gap Analysis Tools

[0893] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user, taking into account the emotion evaluation results from the emotion engine. For example, if stress is likely causing dry skin, that information is reflected in the gap analysis.

[0894] Database search methods

[0895] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. This search includes ingredient information, efficacy, and emotional evaluation results from an emotion engine. For example, items containing stress-relieving ingredients may be selected.

[0896] Display means

[0897] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[0898] Specific examples

[0899] As a specific example, the following processing is performed.

[0900] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[0901] 2. The device sends this information to the server.

[0902] 3. The emotion engine evaluates the user's stress level based on their facial expressions and voice.

[0903] 4. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0904] 5. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0905] 6. The server compares the current state with the ideal state and analyzes the gap, taking into account the stress state evaluated by the emotion engine.

[0906] 7. The server generates a list containing skin care items with stress-relieving effects from the database.

[0907] 8. The device displays a list of recommended items to the user.

[0908] In this way, users can obtain optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. This system allows users to improve their skin condition more effectively and efficiently.

[0909] The processing flow will be explained below.

[0910] Step 1:

[0911] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[0912] Step 2:

[0913] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[0914] Step 3:

[0915] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[0916] Step 4:

[0917] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[0918] Step 5:

[0919] At the same time, an emotion engine in the server analyzes the user's facial expressions and voice to assess their emotional state (e.g., stress or satisfaction), using facial recognition algorithms and voice analysis technology.

[0920] Step 6:

[0921] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[0922] Step 7:

[0923] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[0924] Step 8:

[0925] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. The emotion evaluation results from the emotion engine are also reflected in this gap analysis. For example, if stress is likely to be causing dry skin, that information will be taken into consideration.

[0926] Step 9:

[0927] The server uses the gap analysis results to search the database for optimal skin care items. The search criteria include ingredient information, efficacy, and emotional evaluation results. For example, items containing stress-relieving ingredients are selected.

[0928] Step 10:

[0929] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[0930] Step 11:

[0931] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[0932] In this way, the system can recommend optimal skin care items by comprehensively considering the user's skin concerns, current skin condition, ideal skin condition, and emotional state, providing highly personalized skin care solutions for each individual user.

[0933] Example 2

[0934] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0935] While conventional skincare item recommendation systems can take into account a user's current skin condition and skin concerns, it has been difficult to make recommendations that take into account the user's emotional state. As a result, when emotional factors such as stress or discomfort affect skin problems, it is difficult to select the optimal skincare items.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0937] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; a transmission means for the terminal to transmit the input data and image data to the server; an image analysis means for the server to analyze the image using a machine learning model or a deep learning model and evaluate the current skin condition; an emotion evaluation means for the server to analyze the user's facial expressions and voice and evaluate their emotional state; a selection means for allowing the user to select their ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition and take the emotion evaluation result into consideration; a database search means for the server to search a database for skin care items optimal for filling the gap; and a display means for the terminal to display the search results to the user. This enables more accurate skin care item recommendations while also taking the user's emotional state into consideration.

[0938] The "input means" is a means by which a user inputs skin concerns and uploads an image showing the current skin condition.

[0939] The "transmission means" is a means by which the terminal transmits input data and image data to the server.

[0940] The "image analysis means" is a means by which the server analyzes an image using a machine learning model or a deep learning model to evaluate the current skin condition.

[0941] The "emotion evaluation means" is a means by which the server analyzes the user's facial expression and voice to evaluate the user's emotional state.

[0942] The "selection means" is a means that allows the user to select an ideal skin condition.

[0943] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition, and also takes into account the emotion evaluation results.

[0944] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[0945] The "display means" is a means by which the terminal displays the search results to the user.

[0946] This invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, it is possible to realize recommendations that take the user's emotional state into consideration.

[0947] Hardware and software used

[0948] Hardware

[0949] User device: User device such as a smartphone or tablet

[0950] Server: Cloud server or on-premise server

[0951] software

[0952] Dedicated application: Application installed by the user

[0953] Image analysis algorithms: Machine learning models (TensorFlow, PyTorch, etc.)

[0954] Emotion engine: Emotion recognition software (Amazon Rekognition, IBM Watson, etc.)

[0955] Database: A database containing information on ingredients and effects of skin care items

[0956] System processing flow

[0957] 1. User input and image upload

[0958] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application offers options such as "wrinkles," "dryness," and "redness," allowing users to select the appropriate concern.

[0959] 2. Data transmission

[0960] The device sends the skin concerns entered by the user and the uploaded images to the server. The data is sent as an HTTP request and is encrypted with SSL / TLS for security.

[0961] 3. Image analysis and emotion evaluation

[0962] The server passes the received image data to an image analysis algorithm. A machine learning model (e.g., TensorFlow or PyTorch) is used to quantify the depth of wrinkles and dryness. At the same time, an emotion engine analyzes the user's facial and voice data to evaluate their emotional state, such as stress or joy. Services such as Amazon Rekognition and IBM Watson can be used for emotion evaluation.

[0963] 4. Providing an interface for selecting ideal skin condition

[0964] Based on the image analysis results and emotion evaluation results, the server generates an interface that allows the user to select their ideal skin condition, and sends it to the device. The user selects their ideal skin condition from options such as "moisturized skin" or "few wrinkles."

[0965] 5. Gap analysis

[0966] The server analyzes the gap between the user's current skin condition and their ideal skin condition. For example, if the user's current skin condition is "dry" and their ideal skin condition is "moisturized," the server calculates the difference in specific numerical values ​​and also takes into account whether stress is a factor.

[0967] 6. Search for skin care items

[0968] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. In addition to ingredient information and efficacy, the server also takes into account the emotional evaluation results. For example, items containing stress-relieving ingredients are picked out.

[0969] 7. Generate and display a list of recommended items

[0970] The server generates a list of selected items and sends it to the device. The device then displays a list of recommended skin care items to the user. The list is displayed in formats such as "highly moisturizing cream," "anti-aging serum," and "moisturizing serum," and also provides detailed information on the effects and ingredients of each item.

[0971] Examples of concrete examples and prompts

[0972] Specific examples

[0973] The user inputs that they are concerned about "wrinkles" and "dryness" and uploads a current image of their skin.

[0974] The device sends this information to the server.

[0975] The emotion engine assesses the user's stress level based on their facial expressions and voice.

[0976] The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[0977] The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[0978] The server compares the current state with the ideal state and analyzes the gap, taking into account the stress level assessed by the emotion engine.

[0979] The server generates a list containing skin care items with stress-relieving effects from the database.

[0980] The device displays a list of recommended items to the user.

[0981] Prompt Sentence Examples

[0982] "When developing a system to recommend the best skin care items for women in their 30s, we will use an emotion engine to take into account the user's stress level. What technology should we use to recognize emotions from facial images and voice?"

[0983] This system allows users to select the optimal skin care items based on their current skin condition, ideal skin condition, and even their emotional state, allowing them to improve their skin condition more effectively and efficiently.

[0984] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0985] Step 1:

[0986] Input: The user inputs their skin concerns and uploads an image showing their current skin condition.

[0987] Processing: The user opens the application, selects a concern from options such as "wrinkles," "dryness," or "redness" on the "Enter Skin Concerns" screen, and uploads an image taken with the camera.

[0988] Output: Skin concerns (text data) and image data of skin condition.

[0989] Specific operation: The user enters their concerns into a form within the application, takes a photo of their current skin condition using their smartphone camera, and uploads the image.

[0990] Step 2:

[0991] Input: Text data of entered skin concerns and uploaded image data.

[0992] Processing: The device sends the entered skin concerns and uploaded image data to the server. The data is sent as an HTTP request and encrypted with SSL / TLS.

[0993] Output: Text data and image data of skin concerns sent to the server.

[0994] Specific operation: The smartphone sends the entered data to a cloud server via an internet connection.

[0995] Step 3:

[0996] Input: Text data and image data of skin concerns sent to the server.

[0997] Processing: The server passes the received image data to an image analysis algorithm, which uses a machine learning model (e.g., TensorFlow or PyTorch) to evaluate the skin condition. Specifically, it quantifies the depth of wrinkles and the degree of dryness.

[0998] Output: Numerical data on skin condition (wrinkle depth, dryness, etc.).

[0999] How it works: A machine learning model running on the server analyzes the images and extracts parameters such as wrinkles and dryness.

[1000] Step 4:

[1001] Input: Text data and image data of skin concerns sent to the server, and numerical data of skin condition.

[1002] Processing: The emotion engine analyzes the user's facial and voice data to assess their emotional state, such as stress or joy, using services such as Amazon Rekognition and IBM Watson.

[1003] Output: Emotional assessment (e.g., quantified stress level).

[1004] Specific operation: The server analyzes facial expressions and voice data through the emotion recognition API and quantifies the emotional state.

[1005] Step 5:

[1006] Input: Numerical data of skin condition and emotion evaluation results.

[1007] Processing: Based on this data, the server generates an interface that allows the user to select their ideal skin condition and sends it to the terminal.

[1008] Output: Interface for selecting ideal skin condition.

[1009] Specific operation: The server generates an interface using HTML / CSS, allowing users to select their ideal skin condition on their smartphone.

[1010] Step 6:

[1011] Input: User-selected ideal skin condition.

[1012] Processing: The server analyzes the gap between the current skin condition and the ideal skin condition, and considers whether stress is a contributing factor.

[1013] Output: Gap analysis results (a numerical value indicating how much skin improvement is needed).

[1014] How it works: The server compares your current skin parameters with your ideal parameters and identifies areas that need improvement.

[1015] Step 7:

[1016] Input: Gap analysis results.

[1017] Processing: The server searches the database for the most suitable skin care items based on the results of the gap analysis, taking into account ingredient information, efficacy, and emotional evaluation results.

[1018] Output: A list of recommended skin care items.

[1019] Specific operation: The server uses SQL to search the database and extract the skin care items that are best suited to the user.

[1020] Step 8:

[1021] Enter: a list of recommended skin care items.

[1022] Processing: The server generates a list of the selected items and sends it to the terminal. The terminal displays the list to the user.

[1023] Output: A list of skin care items displayed on the device.

[1024] Specific operation: The smartphone application converts the data received from the server into a display format and presents it visually to the user.

[1025] In this way, the system allows users to select the optimal skin care items taking into account their current skin condition, ideal skin condition, and emotional state.

[1026] (Application example 2)

[1027] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1028] Conventional skincare item recommendation systems focus on assessing the user's skin condition, but rarely consider the user's emotional state. This makes it difficult to recommend optimal skincare items because it ignores factors such as stress that affect skin condition. Furthermore, when dealing with customers in physical stores, it is difficult to instantly recommend optimal products to users, limiting the provision of effective services.

[1029] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means through which the user inputs their skin concerns and uploads an image showing their current skin condition, an image analysis means through which the server analyzes the image and evaluates the current skin condition, and an emotion analysis means through which the server evaluates the user's emotional state using an emotion engine. This makes it possible to comprehensively evaluate the user's skin condition and emotional state and recommend optimal skin care items. Furthermore, instantaneous personalized product recommendations can be realized in customer service in physical stores.

[1030] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[1031] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[1032] The "selection means" is a means that allows the user to select an ideal skin condition.

[1033] "Emotion analysis means" refers to means for assessing the user's emotional state using an emotion engine.

[1034] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[1035] The "database search means" is a means by which the server searches the database for the most suitable skin care item, taking into account the gap and emotional state.

[1036] The "display means" is a means by which the terminal displays the search results to the user.

[1037] A "machine learning model" is a learning algorithm used to perform image analysis and sentiment analysis.

[1038] "Deep learning models" are advanced neural network models used to perform image and sentiment analysis.

[1039] This invention relates to a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. In particular, it uses an emotion engine to realize recommendations that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[1040] First, the user uses a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. Input methods include a smartphone, tablet, or PC. Skin concerns include options such as "wrinkles," "dryness," and "redness," and the user can select multiple options.

[1041] The device then sends the uploaded image and input data to a server, which is equipped with image analysis tools and uses machine learning and deep learning models to evaluate the current skin condition. Specifically, the server analyzes the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image.

[1042] Furthermore, the server is equipped with an emotion analysis means, which uses an emotion engine to analyze the user's emotional state from their facial expressions and voice. For example, if the user is feeling stressed, this information is reflected in the selection of skin care items. This emotion analysis means generally uses voice recognition technology and facial expression recognition technology.

[1043] Next, the server uses a gap analysis means to analyze the gap between the ideal skin condition selected by the user and the current skin condition, taking into account the results of the emotion analysis. For example, if stress may be causing dry skin, that information will be reflected in the gap analysis.

[1044] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database search tool stores information on ingredients and efficacy of skin care items, and the most suitable items are selected based on a comprehensive evaluation that includes the results of sentiment analysis.

[1045] Finally, the server sends the selected list of skin care items to the terminal, and the list is displayed to the user via the display means. The user can check detailed information about specific skin care items, such as "highly moisturizing cream" and "anti-aging serum."

[1046] A specific example is given below. The user inputs that they are concerned about "wrinkles" and "dryness" and uploads an image of their current skin. The device sends this information to the server, which uses an emotion engine to evaluate the user's stress level based on their facial expressions and voice. Image analysis is used to numerically evaluate the depth of wrinkles and dryness. The user then selects "moisturized, wrinkle-free skin" as their ideal skin condition. The server compares the current condition with the ideal condition and analyzes the gap, taking into account the stress level evaluated by the emotion engine. The server then generates a list from its database that includes skin care items that have a stress-relieving effect, and the device displays this list to the user.

[1047] Here are some example prompts for a generative AI model:

[1048] User's skin care concern: Wrinkles

[1049] User's emotional state: Stress

[1050] Current skin condition:

[1051] Wrinkle depth: Medium

[1052] Dryness: High

[1053] Ideal skin condition: Moisturized skin with few wrinkles

[1054] This system allows users to select the optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. Even when used in physical stores, this system enables instant responses to customers, resulting in more effective presentations.

[1055] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1056] Step 1:

[1057] The user inputs their skin concerns using a device with a dedicated application installed. For example, they select the concerns "wrinkles" and "dryness" and upload an image showing their current skin condition. The input data includes information on skin concerns (multiple selections possible) and image data showing the skin condition. Based on this, the device acquires the image data and input data.

[1058] Step 2:

[1059] The acquired data is sent from the device to a server. The server first analyzes the received image data using image analysis means. As a specific example, a machine learning model or deep learning model is used to quantify wrinkle depth, dryness, redness distribution, etc. The input is the uploaded image data, and the output is evaluation data of the skin condition as the analysis result.

[1060] Step 3:

[1061] At the same time, the server uses emotion analysis means to evaluate the user's emotional state from their facial expressions and voice. For example, it determines whether the user is feeling stressed. The technologies used include voice recognition and facial expression recognition. The input is the user's facial expression and voice data, and the output is the evaluation result of the user's emotional state.

[1062] Step 4:

[1063] The user selects their ideal skin condition through the application. Specific options such as "moisturized and wrinkle-free skin" are displayed, and the user selects the desired condition. The input is the user's selection, and the output is the ideal skin condition data.

[1064] Step 5:

[1065] The server compares the current skin condition with the ideal skin condition and analyzes the gap between them, taking into account the evaluation results of the emotional state. The inputs are the skin condition evaluation data, the ideal skin condition data, and the evaluation results of the emotional state, and the output is the gap analysis results.

[1066] Step 6:

[1067] Based on the gap analysis results, the server uses a database search means to search for the most suitable skin care items. The database contains information on the ingredients and effects of each item, and the most suitable items are selected taking into consideration the emotional state. The input is the gap analysis results, and the output is a list of the most suitable skin care items.

[1068] Step 7:

[1069] The list of skin care items generated by the server is sent to the terminal, which displays the list to the user and provides detailed information about the recommended skin care items. The input is the list of skin care items received from the server, and the output is the information displayed to the user.

[1070] Step 8:

[1071] Users can select and purchase skin care items from a displayed list. This includes in-store purchases, allowing users to instantly get the perfect product. The input is the user's selection, and the output is purchase procedure information.

[1072] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1073] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1074] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1075] [Fourth embodiment]

[1076] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1077] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1078] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1079] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1080] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1081] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1082] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1083] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1084] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1085] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1086] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1087] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1088] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1089] The present invention provides a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user's skin closer to ideal. Specific embodiments of this system are described below.

[1090] Input Method

[1091] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[1092] Image analysis methods

[1093] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models. For example, the depth of wrinkles, degree of dryness, and distribution of redness in the image are evaluated.

[1094] Selection method

[1095] The server provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can select their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[1096] Gap Analysis Tools

[1097] The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent.

[1098] Database search methods

[1099] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each item, and based on this information, the server generates search results. For example, it may select a cream rich in moisturizing ingredients or a serum with anti-aging effects.

[1100] Display means

[1101] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[1102] Specific examples

[1103] As a specific example, the following processing is performed.

[1104] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[1105] 2. The device sends this information to the server.

[1106] 3. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[1107] 4. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[1108] 5. The server compares the current state with the ideal state and analyzes the gap.

[1109] 6. The server searches the database for the most suitable skin care items and generates a list.

[1110] 7. The device displays a list of recommended items to the user.

[1111] In this way, users can compare their current skin condition with their ideal skin condition and obtain the skin care items that are best suited to them. This system allows users to easily find the skin care products that are best suited to them, thereby efficiently improving their skin condition.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[1115] Step 2:

[1116] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[1117] Step 3:

[1118] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[1119] Step 4:

[1120] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[1121] Step 5:

[1122] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[1123] Step 6:

[1124] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[1125] Step 7:

[1126] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. For example, if the current wrinkle depth is deeper than the ideal, the difference in depth is calculated.

[1127] Step 8:

[1128] The server uses the results of the gap analysis to search the database for the most suitable skin care items. The search criteria include information on ingredients and effects.

[1129] Step 9:

[1130] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[1131] Step 10:

[1132] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[1133] In this way, the specific actions performed in each step allow the user to effectively and efficiently find the skin care items that are best suited to their current skin condition.

[1134] Example 1

[1135] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1136] With conventional methods for selecting skin care items, it is often difficult for users to find the items that are best suited to their skin condition, resulting in ineffective skin care. Furthermore, because users lack specialized knowledge, it is difficult for them to accurately evaluate their own skin condition, and they lack the information to select appropriate products. This has led to users generally purchasing skin care products on their own, which has created the problem of making it difficult to achieve effective skin care.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1138] In this invention, the server includes input means for allowing a user to input skin problems and upload an image showing their current skin condition, image analysis means for the server to analyze the image and evaluate the current skin condition, selection means for allowing the user to select an ideal skin condition, gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition, database search means for the server to search a database for cosmetics optimal for filling the gap, display means for the terminal to display the search results to the user, evaluation means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user, and recommendation means for selecting optimal cosmetics based on the ideal skin condition selected by the user. This enables users to accurately evaluate their current skin condition and easily find optimal skin care items.

[1139] The "input means" is a means by which a user inputs skin problems and uploads an image showing the current skin condition.

[1140] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[1141] The "selection means" is a means that allows the user to select an ideal skin condition.

[1142] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition.

[1143] The "database search means" is a means by which the server searches the database for the most suitable cosmetics to fill the gap.

[1144] The "display means" is a means by which the terminal displays the search results to the user.

[1145] The "evaluation means" is a means for obtaining an evaluation result by comparing the analysis result with the ideal skin condition input by the user.

[1146] The "recommendation means" is a means for selecting the most suitable cosmetics based on the ideal skin condition selected by the user.

[1147] The present invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal cosmetics to bring the user closer to ideal skin. This system is configured as follows.

[1148] First, users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, through which users can select multiple options such as "wrinkles," "dryness," and "redness."

[1149] Next, the device sends the image and input data uploaded by the user to a server. The server receives the image and uses an image analysis algorithm to evaluate the current skin condition. This analysis uses deep learning models such as TensorFlow and PyTorch. The server quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image, and stores the evaluation results in a database.

[1150] The server then provides the user with an interface to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles." The device then sends this selection information to the server.

[1151] The server then analyzes the gap between the current skin condition and the ideal skin condition. This gap analysis identifies skin problems that need improvement and quantifies the factors for improving each problem. The server then searches a database for the most suitable cosmetics based on the results of the gap analysis. The database stores detailed information on the ingredients and effects of each cosmetic product. The server selects the most suitable cosmetics based on the required ingredients and effects and retrieves the results from the database.

[1152] Finally, the server sends the selected cosmetics list to the terminal. The terminal displays the list to the user and provides detailed information about the recommended cosmetics. For example, it may be displayed in the form of "High Moisturizing Cream 'Brand A'" or "Anti-Aging Serum 'Brand B'." The user can refer to the list and purchase the cosmetics that suit them best.

[1153] Specific examples

[1154] For example, a user can input that they are concerned about "wrinkles" and "dryness" and upload an image of their current skin. The device sends this information to a server, which then performs image analysis to numerically evaluate the depth of wrinkles and dryness. The user can then select "moisturized, wrinkle-free skin" as their ideal skin condition, and the server will analyze the gap between their current condition and their ideal condition and search a database for the most suitable cosmetics. For example, it will select products such as "cream containing hyaluronic acid" or "serum containing retinol" and display them to the user as a list.

[1155] Prompt Sentence Examples

[1156] "Please suggest the best skin care items to make my dry and wrinkled skin moisturized and less wrinkled. I will also send you a photo of my current skin, so please analyze it."

[1157] This prompt provides specific instructions to the generative AI model, allowing the system to recommend the most suitable skin care items for the user.

[1158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1159] Program processing flow

[1160] Step 1: User Input and Data Submission

[1161] The user launches the dedicated application and selects a skin problem such as "wrinkles" or "dryness" on the problem selection screen (interface). When the user taps the "Next" button, the camera starts up and the user takes an image showing their current skin condition. After that, the user taps the "Upload" button, and the entered problem data and skin image are sent to the server by the device.

[1162] Input: Skin problem selection data and skin images

[1163] Output: Image data and user data sent to the server

[1164] Specifically, the user follows the application's instructions, selects a skin concern, and then takes a skin image using the camera. The device then sends the image to the server, which also sends the data using an HTTP request.

[1165] Step 2: Image analysis by the server

[1166] The server inputs the received image data into an image analysis algorithm (e.g., TensorFlow or PyTorch), which quantifies the depth of wrinkles, the degree of dryness, the distribution of redness, etc., and stores the evaluation results in a database.

[1167] Input: Skin images and problem data sent from the device

[1168] Output: Skin condition evaluation results (e.g., numerical data on wrinkle depth, dryness, and redness distribution)

[1169] Specifically, the server launches an image analysis model, identifies wrinkle depth and dry areas from the image, and evaluates them as numerical data. The analysis results are then recorded in a database.

[1170] Step 3: Select your ideal skin condition

[1171] The server uses the analysis results to provide an interface that asks the user about their ideal skin condition. The user selects their ideal condition, such as "moisturized skin" or "few wrinkles," and taps the "Done" button. The device then sends this selection information to the server.

[1172] Input: Analysis results and options to ask about ideal skin condition

[1173] Output: Data of ideal skin condition selected by the user

[1174] Specifically, the server generates a user interface and provides options to the user. The user selects an ideal state and transmits the data from the terminal to the server.

[1175] Step 4: Gap analysis by server

[1176] The server compares the current skin condition (assessment result) with the ideal skin condition (user selection). The gap is quantified as the amount of wrinkle depth or dryness that needs to be reduced.

[1177] Input: Current skin condition evaluation results and ideal skin condition selected by the user

[1178] Output: Gap analysis results (points of improvement and their numerical data)

[1179] Specifically, the server compares the current state with the desired state, identifies and quantifies areas of improvement that need to be made, and these results are used in the next step.

[1180] Step 5: Database lookup by server

[1181] Based on the results of the gap analysis, the server searches the database for the most suitable cosmetics, such as a cream with high moisturizing properties or a serum with anti-aging properties.

[1182] Input: Gap analysis results

[1183] Output: A list of the best cosmetics

[1184] Specifically, the server queries the database for cosmetic information and identifies suitable items to fill the gaps. The search results are used in the next step.

[1185] Step 6: View recommended items

[1186] The server sends the search results to the terminal, which then displays a list of recommended cosmetics for the user. The list includes product name, effects, and ingredient information, and the user can refer to the list for more information.

[1187] Enter: the best makeup list

[1188] Output: Cosmetics list displayed on the terminal

[1189] Specifically, the server sends the generated list to the device, which then displays the list on the screen, allowing the user to review the list and select appropriate skin care items.

[1190] This series of processes allows users to accurately understand their current skin condition and easily find the skin care items that are best suited to them.

[1191] (Application example 1)

[1192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1193] Conventional skin care product recommendation methods make it difficult for users to find the product that best suits their skin condition. Furthermore, product recommendations by sales staff in physical stores are often limited by time and information, leaving users unsatisfied. The present invention aims to improve a system that accurately evaluates a user's skin condition and recommends the most suitable skin care product, thereby improving the user experience in physical stores.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1195] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means for allowing the user to select an ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition; a database search means for the server to search a database for optimal skin care items to fill the gap; a display means for a terminal to display the search results to the user; a means including an application to be installed on a terminal held by a sales staff member in a physical store; and a means including a process for allowing customers to evaluate their skin condition on the spot and recommend optimal skin care products. This improves the user experience in physical stores and enables users to easily find skin care products optimal for their skin condition.

[1196] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[1197] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[1198] The "selection means" is a means for enabling the user to select an ideal skin condition.

[1199] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[1200] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[1201] The "display means" is a means by which the terminal displays the search results to the user.

[1202] "Means including applications" refers to means using applications installed on terminals held by sales staff in physical stores.

[1203] The "means including a recommendation process" is a means for executing a process in which a customer evaluates their skin condition on the spot and recommends the most suitable skin care product.

[1204] The present invention is a system that evaluates skin condition and smoothly recommends optimal skin care products to improve the user experience in physical stores. This system is mainly composed of a server and a terminal.

[1205] Hardware and Software

[1206] Hardware: Smartphones and servers owned by store staff.

[1207] Software: Applications, image analysis algorithms, machine learning models, deep learning models, databases.

[1208] Data processing and calculation

[1209] 1. Input method: Using the store staff's smartphone, the customer enters their skin concerns (e.g., wrinkles, dryness) in text and takes and uploads an image showing their current skin condition.

[1210] 2. Image analysis: The server analyzes the uploaded images. Specifically, it uses image analysis algorithms to evaluate the current skin condition, such as the depth of wrinkles and dryness. This analysis utilizes machine learning and deep learning models.

[1211] 3. Selection method: Based on the analysis results, the server provides the user with an interface for selecting their ideal skin condition. The user can select their ideal condition on the spot, such as "moisturized, wrinkle-free skin."

[1212] 4. Gap analysis method: The server analyzes the gap between the current skin condition and the ideal skin condition selected by the user. By using this gap analysis method, it is possible to identify which skin problems need to be improved and to what extent.

[1213] 5. Database search method: Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. The database contains detailed information about the effects and ingredients of each product.

[1214] 6. Display: A list of the most suitable skincare items obtained as a result of the search is displayed on the staff member's smartphone. This list includes detailed information about each product, which the staff member can use to make recommendations to the customer.

[1215] Specific examples

[1216] For example, a customer can enter "wrinkles" and "dryness" as concerns at a store and upload an image of their current skin. The server receives this information and analyzes their skin condition using machine learning and deep learning models. As a result, the customer is asked to select their ideal state, such as "moisturized, wrinkle-free skin," and the gap between their current state and that state is analyzed. Finally, the system searches for the optimal skin care items (e.g., highly moisturizing cream, anti-aging serum) to fill this gap and displays them on the staff member's smartphone.

[1217] Prompt Sentence Examples

[1218] "Explain how the example Python code works in a scenario where a user inputs "wrinkles" and "dryness" as concerns and uploads an image showing their skin condition."

[1219] Through the above process, it is possible to significantly improve the user experience in physical stores.

[1220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1221] Step 1:

[1222] The device prompts the user to enter their skin concerns and take and upload an image showing their current skin condition. At this stage, text (e.g., wrinkles, dryness) and image data are obtained as input data. This data is then sent from the device to the server.

[1223] Step 2:

[1224] The server receives the transmitted image and text data. It analyzes the features in the image (e.g., depth of wrinkles, degree of dryness) based on the image analysis algorithm and generative AI model. As a result of the analysis, detailed evaluation information on the skin condition is obtained. The server generates this evaluation information.

[1225] Step 3:

[1226] The server provides an interface that allows the user to select their ideal skin condition based on the evaluation results. The user selects their ideal skin condition (e.g., moisturized and wrinkle-free skin) through their device. This selection information is also sent to the server.

[1227] Step 4:

[1228] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user. This gap analysis identifies which skin concerns need to be improved and to what extent. Specifically, the degree of wrinkle reduction and the level of relief from dryness are quantified.

[1229] Step 5:

[1230] The server searches the database for optimal skin care items based on the results of the gap analysis. This search takes into account information about the effects and ingredients of each item. As a result, a list of recommended skin care items is generated.

[1231] Step 6:

[1232] The server sends the list of skin care items it has created to the terminal. The terminal displays this list to the user and provides detailed product information (e.g., effects, usage instructions), allowing sales staff to recommend the most suitable products to the user.

[1233] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1234] The present invention is a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, recommendations can be made that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[1235] Input Method

[1236] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application has an interface for selecting skin concerns, where multiple options can be selected, such as "wrinkles," "dryness," and "redness."

[1237] Image analysis methods

[1238] The device sends the skin image and input data uploaded by the user to a server. The server receives the image and uses image analysis algorithms to evaluate the current skin condition. This analysis uses machine learning and deep learning models to evaluate, for example, the depth of wrinkles, the degree of dryness, and the distribution of redness in the image.

[1239] Emotion Engine

[1240] The server is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice. The emotion engine works in conjunction with image analysis to evaluate the user's facial expressions and uses voice recognition technology to detect emotions from the user's voice. This allows the system to reflect information such as if the user is feeling stressed in the selection of skin care items.

[1241] Selection method

[1242] The server provides an interface that allows users to select their ideal skin condition based on the evaluation results. Users can choose their ideal skin condition, such as "moisturized skin" or "few wrinkles."

[1243] Gap Analysis Tools

[1244] The server analyzes the gap between the user's current skin condition and the ideal skin condition selected by the user, taking into account the emotion evaluation results from the emotion engine. For example, if stress is likely causing dry skin, that information is reflected in the gap analysis.

[1245] Database search methods

[1246] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. This search includes ingredient information, efficacy, and emotional evaluation results from an emotion engine. For example, items containing stress-relieving ingredients may be selected.

[1247] Display means

[1248] The server sends the list of skin care items selected to the device, which then displays the list to the user and provides detailed information about the recommended products, such as "High Moisturizing Cream 'Brand A'," "Anti-Aging Serum 'Brand B'," and "Moisturizing Serum 'Brand C'."

[1249] Specific examples

[1250] As a specific example, the following processing is performed.

[1251] 1. The user enters that they are concerned about "wrinkles" and "dryness" and uploads a photo of their current skin.

[1252] 2. The device sends this information to the server.

[1253] 3. The emotion engine evaluates the user's stress level based on their facial expressions and voice.

[1254] 4. The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[1255] 5. The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[1256] 6. The server compares the current state with the ideal state and analyzes the gap, taking into account the stress state evaluated by the emotion engine.

[1257] 7. The server generates a list containing skin care items with stress-relieving effects from the database.

[1258] 8. The device displays a list of recommended items to the user.

[1259] In this way, users can obtain optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. This system allows users to improve their skin condition more effectively and efficiently.

[1260] The processing flow will be explained below.

[1261] Step 1:

[1262] The user launches the application and inputs their skin concerns. For example, they are provided with a function to select concerns such as "wrinkles" or "dryness" from a list of options.

[1263] Step 2:

[1264] Users take an image showing their current skin condition and upload it to the application, which may provide guidelines on image quality and lighting conditions.

[1265] Step 3:

[1266] The device sends the user's input data (skin concerns) and uploaded images to the server. A security protocol (e.g., HTTPS) is used for data transmission.

[1267] Step 4:

[1268] The server analyzes the received images and uses machine learning and deep learning models to numerically evaluate the skin condition (wrinkle depth, dryness, redness distribution, etc.).

[1269] Step 5:

[1270] At the same time, an emotion engine in the server analyzes the user's facial expressions and voice to assess their emotional state (e.g., stress or satisfaction), using facial recognition algorithms and voice analysis technology.

[1271] Step 6:

[1272] Based on the analysis results, the server generates a user interface that allows the user to select their ideal skin condition. This interface is displayed on the device.

[1273] Step 7:

[1274] The user selects their ideal skin condition. For example, they select "moisturized, wrinkle-free skin." This information is sent from the device to the server.

[1275] Step 8:

[1276] The server compares the current skin condition with the ideal skin condition selected by the user and analyzes the gap. The emotion evaluation results from the emotion engine are also reflected in this gap analysis. For example, if stress is likely to be causing dry skin, that information will be taken into consideration.

[1277] Step 9:

[1278] The server uses the gap analysis results to search the database for optimal skin care items. The search criteria include ingredient information, efficacy, and emotional evaluation results. For example, items containing stress-relieving ingredients are selected.

[1279] Step 10:

[1280] The server generates a list of search results and sends it to the device, including details such as product names, ingredients, and effects.

[1281] Step 11:

[1282] The device will display the search results to the user, such as a list of "Highly Moisturizing Cream (Brand A)," "Anti-Aging Serum (Brand B)," and "Moisturizing Serum (Brand C)."

[1283] In this way, the system can recommend optimal skin care items by comprehensively considering the user's skin concerns, current skin condition, ideal skin condition, and emotional state, providing highly personalized skin care solutions for each individual user.

[1284] Example 2

[1285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] While conventional skincare item recommendation systems can take into account a user's current skin condition and skin concerns, it has been difficult to make recommendations that take into account the user's emotional state. As a result, when emotional factors such as stress or discomfort affect skin problems, it is difficult to select the optimal skincare items.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1288] In this invention, the server includes: an input means for allowing a user to input skin concerns and upload an image showing their current skin condition; a transmission means for the terminal to transmit the input data and image data to the server; an image analysis means for the server to analyze the image using a machine learning model or a deep learning model and evaluate the current skin condition; an emotion evaluation means for the server to analyze the user's facial expressions and voice and evaluate their emotional state; a selection means for allowing the user to select their ideal skin condition; a gap analysis means for the server to analyze the gap between the current skin condition and the ideal skin condition and take the emotion evaluation result into consideration; a database search means for the server to search a database for skin care items optimal for filling the gap; and a display means for the terminal to display the search results to the user. This enables more accurate skin care item recommendations while also taking the user's emotional state into consideration.

[1289] The "input means" is a means by which a user inputs skin concerns and uploads an image showing the current skin condition.

[1290] The "transmission means" is a means by which the terminal transmits input data and image data to the server.

[1291] The "image analysis means" is a means by which the server analyzes an image using a machine learning model or a deep learning model to evaluate the current skin condition.

[1292] The "emotion evaluation means" is a means by which the server analyzes the user's facial expression and voice to evaluate the user's emotional state.

[1293] The "selection means" is a means that allows the user to select an ideal skin condition.

[1294] The "gap analysis means" is a means by which the server analyzes the gap between the current skin condition and the ideal skin condition, and also takes into account the emotion evaluation results.

[1295] The "database search means" is a means by which the server searches the database for the most suitable skin care items to fill the gaps.

[1296] The "display means" is a means by which the terminal displays the search results to the user.

[1297] This invention is a system that evaluates a user's skin concerns and current skin condition, and recommends optimal skin care items to bring the user closer to their ideal skin. By combining this system with an emotion engine, it is possible to realize recommendations that take the user's emotional state into consideration.

[1298] Hardware and software used

[1299] Hardware

[1300] User device: User device such as a smartphone or tablet

[1301] Server: Cloud server or on-premise server

[1302] software

[1303] Dedicated application: Application installed by the user

[1304] Image analysis algorithms: Machine learning models (TensorFlow, PyTorch, etc.)

[1305] Emotion engine: Emotion recognition software (Amazon Rekognition, IBM Watson, etc.)

[1306] Database: A database containing information on ingredients and effects of skin care items

[1307] System processing flow

[1308] 1. User input and image upload

[1309] Users use a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. The application offers options such as "wrinkles," "dryness," and "redness," allowing users to select the appropriate concern.

[1310] 2. Data transmission

[1311] The device sends the skin concerns entered by the user and the uploaded images to the server. The data is sent as an HTTP request and is encrypted with SSL / TLS for security.

[1312] 3. Image analysis and emotion evaluation

[1313] The server passes the received image data to an image analysis algorithm. A machine learning model (e.g., TensorFlow or PyTorch) is used to quantify the depth of wrinkles and dryness. At the same time, an emotion engine analyzes the user's facial and voice data to evaluate their emotional state, such as stress or joy. Services such as Amazon Rekognition and IBM Watson can be used for emotion evaluation.

[1314] 4. Providing an interface for selecting ideal skin condition

[1315] Based on the image analysis results and emotion evaluation results, the server generates an interface that allows the user to select their ideal skin condition, and sends it to the device. The user selects their ideal skin condition from options such as "moisturized skin" or "few wrinkles."

[1316] 5. Gap analysis

[1317] The server analyzes the gap between the user's current skin condition and their ideal skin condition. For example, if the user's current skin condition is "dry" and their ideal skin condition is "moisturized," the server calculates the difference in specific numerical values ​​and also takes into account whether stress is a factor.

[1318] 6. Search for skin care items

[1319] Based on the results of the gap analysis, the server searches the database for the most suitable skin care items. In addition to ingredient information and efficacy, the server also takes into account the emotional evaluation results. For example, items containing stress-relieving ingredients are picked out.

[1320] 7. Generate and display a list of recommended items

[1321] The server generates a list of selected items and sends it to the device. The device then displays a list of recommended skin care items to the user. The list is displayed in formats such as "highly moisturizing cream," "anti-aging serum," and "moisturizing serum," and also provides detailed information on the effects and ingredients of each item.

[1322] Examples of concrete examples and prompts

[1323] Specific examples

[1324] The user inputs that they are concerned about "wrinkles" and "dryness" and uploads a current image of their skin.

[1325] The device sends this information to the server.

[1326] The emotion engine assesses the user's stress level based on their facial expressions and voice.

[1327] The server performs image analysis and numerically evaluates the depth of wrinkles and dryness.

[1328] The user selects "moisturized, wrinkle-free skin" as their ideal skin condition.

[1329] The server compares the current state with the ideal state and analyzes the gap, taking into account the stress level assessed by the emotion engine.

[1330] The server generates a list containing skin care items with stress-relieving effects from the database.

[1331] The device displays a list of recommended items to the user.

[1332] Prompt Sentence Examples

[1333] "When developing a system to recommend the best skin care items for women in their 30s, we will use an emotion engine to take into account the user's stress level. What technology should we use to recognize emotions from facial images and voice?"

[1334] This system allows users to select the optimal skin care items based on their current skin condition, ideal skin condition, and even their emotional state, allowing them to improve their skin condition more effectively and efficiently.

[1335] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1336] Step 1:

[1337] Input: The user inputs their skin concerns and uploads an image showing their current skin condition.

[1338] Processing: The user opens the application, selects a concern from options such as "wrinkles," "dryness," or "redness" on the "Enter Skin Concerns" screen, and uploads an image taken with the camera.

[1339] Output: Skin concerns (text data) and image data of skin condition.

[1340] Specific operation: The user enters their concerns into a form within the application, takes a photo of their current skin condition using their smartphone camera, and uploads the image.

[1341] Step 2:

[1342] Input: Text data of entered skin concerns and uploaded image data.

[1343] Processing: The device sends the entered skin concerns and uploaded image data to the server. The data is sent as an HTTP request and encrypted with SSL / TLS.

[1344] Output: Text data and image data of skin concerns sent to the server.

[1345] Specific operation: The smartphone sends the entered data to a cloud server via an internet connection.

[1346] Step 3:

[1347] Input: Text data and image data of skin concerns sent to the server.

[1348] Processing: The server passes the received image data to an image analysis algorithm, which uses a machine learning model (e.g., TensorFlow or PyTorch) to evaluate the skin condition. Specifically, it quantifies the depth of wrinkles and the degree of dryness.

[1349] Output: Numerical data on skin condition (wrinkle depth, dryness, etc.).

[1350] How it works: A machine learning model running on the server analyzes the images and extracts parameters such as wrinkles and dryness.

[1351] Step 4:

[1352] Input: Text data and image data of skin concerns sent to the server, and numerical data of skin condition.

[1353] Processing: The emotion engine analyzes the user's facial and voice data to assess their emotional state, such as stress or joy, using services such as Amazon Rekognition and IBM Watson.

[1354] Output: Emotional assessment (e.g., quantified stress level).

[1355] Specific operation: The server analyzes facial expressions and voice data through the emotion recognition API and quantifies the emotional state.

[1356] Step 5:

[1357] Input: Numerical data of skin condition and emotion evaluation results.

[1358] Processing: Based on this data, the server generates an interface that allows the user to select their ideal skin condition and sends it to the terminal.

[1359] Output: Interface for selecting ideal skin condition.

[1360] Specific operation: The server generates an interface using HTML / CSS, allowing users to select their ideal skin condition on their smartphone.

[1361] Step 6:

[1362] Input: User-selected ideal skin condition.

[1363] Processing: The server analyzes the gap between the current skin condition and the ideal skin condition, and considers whether stress is a contributing factor.

[1364] Output: Gap analysis results (a numerical value indicating how much skin improvement is needed).

[1365] How it works: The server compares your current skin parameters with your ideal parameters and identifies areas that need improvement.

[1366] Step 7:

[1367] Input: Gap analysis results.

[1368] Processing: The server searches the database for the most suitable skin care items based on the results of the gap analysis, taking into account ingredient information, efficacy, and emotional evaluation results.

[1369] Output: A list of recommended skin care items.

[1370] Specific operation: The server uses SQL to search the database and extract the skin care items that are best suited to the user.

[1371] Step 8:

[1372] Enter: a list of recommended skin care items.

[1373] Processing: The server generates a list of the selected items and sends it to the terminal. The terminal displays the list to the user.

[1374] Output: A list of skin care items displayed on the device.

[1375] Specific operation: The smartphone application converts the data received from the server into a display format and presents it visually to the user.

[1376] In this way, the system allows users to select the optimal skin care items taking into account their current skin condition, ideal skin condition, and emotional state.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] Conventional skincare item recommendation systems focus on assessing the user's skin condition, but rarely consider the user's emotional state. This makes it difficult to recommend optimal skincare items because it ignores factors such as stress that affect skin condition. Furthermore, when dealing with customers in physical stores, it is difficult to instantly recommend optimal products to users, limiting the provision of effective services.

[1380] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means through which the user inputs their skin concerns and uploads an image showing their current skin condition, an image analysis means through which the server analyzes the image and evaluates the current skin condition, and an emotion analysis means through which the server evaluates the user's emotional state using an emotion engine. This makes it possible to comprehensively evaluate the user's skin condition and emotional state and recommend optimal skin care items. Furthermore, instantaneous personalized product recommendations can be realized in customer service in physical stores.

[1381] The "input means" is a means for a user to input skin concerns and upload an image showing the current skin condition.

[1382] The "image analysis means" is a means by which the server analyzes the uploaded image and evaluates the current skin condition.

[1383] The "selection means" is a means that allows the user to select an ideal skin condition.

[1384] "Emotion analysis means" refers to means for assessing the user's emotional state using an emotion engine.

[1385] The "gap analysis means" is a means for the server to analyze the gap between the current skin condition and the ideal skin condition.

[1386] The "database search means" is a means by which the server searches the database for the most suitable skin care item, taking into account the gap and emotional state.

[1387] The "display means" is a means by which the terminal displays the search results to the user.

[1388] A "machine learning model" is a learning algorithm used to perform image analysis and sentiment analysis.

[1389] "Deep learning models" are advanced neural network models used to perform image and sentiment analysis.

[1390] This invention relates to a system that evaluates a user's skin concerns and current skin condition and recommends optimal skin care items to bring the user closer to their ideal skin. In particular, it uses an emotion engine to realize recommendations that take the user's emotional state into consideration. Specific embodiments of the present invention are described below.

[1391] First, the user uses a device with a dedicated application installed to input their skin concerns and upload an image showing their current skin condition. Input methods include a smartphone, tablet, or PC. Skin concerns include options such as "wrinkles," "dryness," and "redness," and the user can select multiple options.

[1392] The device then sends the uploaded image and input data to a server, which is equipped with image analysis tools and uses machine learning and deep learning models to evaluate the current skin condition. Specifically, the server analyzes the depth of wrinkles, the degree of dryness, the distribution of redness, and other factors in the image.

[1393] Furthermore, the server is equipped with an emotion analysis means, which uses an emotion engine to analyze the user's emotional state from their facial expressions and voice. For example, if the user is feeling stressed, this information is reflected in the selection of skin care items. This emotion analysis means generally uses voice recognition technology and facial expression recognition technology.

[1394] Next, the server uses a gap analysis means to analyze the gap between the ideal skin condition selected by the user and the current skin condition, taking into account the results of the emotion analysis. For example, if stress may be causing dry skin, that information will be reflected in the gap analysis.

[1395] Based on the results of the gap analysis, the server searches a database for the most suitable skin care items. The database search tool stores information on ingredients and efficacy of skin care items, and the most suitable items are selected based on a comprehensive evaluation that includes the results of sentiment analysis.

[1396] Finally, the server sends the selected list of skin care items to the terminal, and the list is displayed to the user via the display means. The user can check detailed information about specific skin care items, such as "highly moisturizing cream" and "anti-aging serum."

[1397] A specific example is given below. The user inputs that they are concerned about "wrinkles" and "dryness" and uploads an image of their current skin. The device sends this information to the server, which uses an emotion engine to evaluate the user's stress level based on their facial expressions and voice. Image analysis is used to numerically evaluate the depth of wrinkles and dryness. The user then selects "moisturized, wrinkle-free skin" as their ideal skin condition. The server compares the current condition with the ideal condition and analyzes the gap, taking into account the stress level evaluated by the emotion engine. The server then generates a list from its database that includes skin care items that have a stress-relieving effect, and the device displays this list to the user.

[1398] Here are some example prompts for a generative AI model:

[1399] User's skin care concern: Wrinkles

[1400] User's emotional state: Stress

[1401] Current skin condition:

[1402] Wrinkle depth: Medium

[1403] Dryness: High

[1404] Ideal skin condition: Moisturized skin with few wrinkles

[1405] This system allows users to select the optimal skin care items that take into account not only their current and ideal skin conditions, but also their emotional state. Even when used in physical stores, this system enables instant responses to customers, resulting in more effective presentations.

[1406] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1407] Step 1:

[1408] The user inputs their skin concerns using a device with a dedicated application installed. For example, they select the concerns "wrinkles" and "dryness" and upload an image showing their current skin condition. The input data includes information on skin concerns (multiple selections possible) and image data showing the skin condition. Based on this, the device acquires the image data and input data.

[1409] Step 2:

[1410] The acquired data is sent from the device to a server. The server first analyzes the received image data using image analysis means. As a specific example, a machine learning model or deep learning model is used to quantify wrinkle depth, dryness, redness distribution, etc. The input is the uploaded image data, and the output is evaluation data of the skin condition as the analysis result.

[1411] Step 3:

[1412] At the same time, the server uses emotion analysis means to evaluate the user's emotional state from their facial expressions and voice. For example, it determines whether the user is feeling stressed. The technologies used include voice recognition and facial expression recognition. The input is the user's facial expression and voice data, and the output is the evaluation result of the user's emotional state.

[1413] Step 4:

[1414] The user selects their ideal skin condition through the application. Specific options such as "moisturized and wrinkle-free skin" are displayed, and the user selects the desired condition. The input is the user's selection, and the output is the ideal skin condition data.

[1415] Step 5:

[1416] The server compares the current skin condition with the ideal skin condition and analyzes the gap between them, taking into account the evaluation results of the emotional state. The inputs are the skin condition evaluation data, the ideal skin condition data, and the evaluation results of the emotional state, and the output is the gap analysis results.

[1417] Step 6:

[1418] Based on the gap analysis results, the server uses a database search means to search for the most suitable skin care items. The database contains information on the ingredients and effects of each item, and the most suitable items are selected taking into consideration the emotional state. The input is the gap analysis results, and the output is a list of the most suitable skin care items.

[1419] Step 7:

[1420] The list of skin care items generated by the server is sent to the terminal, which displays the list to the user and provides detailed information about the recommended skin care items. The input is the list of skin care items received from the server, and the output is the information displayed to the user.

[1421] Step 8:

[1422] Users can select and purchase skin care items from a displayed list. This includes in-store purchases, allowing users to instantly get the perfect product. The input is the user's selection, and the output is purchase procedure information.

[1423] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1425] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1426] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1427] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1428] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1429] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1430] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1431] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1432] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1433] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1434] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1435] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1436] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1437] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1438] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1439] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1440] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1441] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1442] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1443] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1444] The following is further disclosed regarding the above embodiment.

[1445] (Claim 1)

[1446] an input means for a user to input skin concerns and upload an image showing the current skin condition;

[1447] an image analysis means for the server to analyze the image and evaluate the current skin condition;

[1448] a selection means that allows a user to select an ideal skin condition;

[1449] A gap analysis means for the server to analyze a gap between a current skin condition and an ideal skin condition;

[1450] a database search means for the server to search a database for an optimal skin care item to fill the gap;

[1451] a display means for displaying search results to a user on the terminal;

[1452] A system including:

[1453] (Claim 2)

[1454] 2. The system of claim 1, wherein the server performs image analysis using a machine learning model or a deep learning model.

[1455] (Claim 3)

[1456] 2. The system according to claim 1, wherein the server generates a list of skin care items based on the evaluation of the skin condition, and the terminal displays the list to the user.

[1457] "Example 1"

[1458] (Claim 1)

[1459] an input means for a user to input skin problems and upload an image showing the current skin condition;

[1460] an image analysis means for the server to analyze the image and evaluate the current skin condition;

[1461] a selection means that allows a user to select an ideal skin condition;

[1462] a gap analysis means for analyzing a gap between a current skin condition and an ideal skin condition by the server;

[1463] a database search means for the server to search the database for the most suitable cosmetic product to fill the gap;

[1464] a display means for displaying search results to a user on the terminal;

[1465] an evaluation means for obtaining an evaluation result by comparing the analysis result with an ideal skin condition input by a user;

[1466] A recommendation means for selecting the most suitable cosmetics based on the ideal skin condition selected by the user;

[1467] A system including:

[1468] (Claim 2)

[1469] 2. The system of claim 1, wherein the server performs image analysis using a machine learning model or a deep learning model.

[1470] (Claim 3)

[1471] 2. The system according to claim 1, wherein the server generates a list of cosmetics based on the evaluation of the skin condition, and the terminal displays the list to the user.

[1472] "Application Example 1"

[1473] (Claim 1)

[1474] an input means for a user to input skin concerns and upload an image showing the current skin condition;

[1475] an image analysis means for the server to analyze the image and evaluate the current skin condition;

[1476] a selection means that allows a user to select an ideal skin condition;

[1477] A gap analysis means for the server to analyze a gap between a current skin condition and an ideal skin condition;

[1478] a database search means for the server to search a database for an optimal skin care item to fill the gap;

[1479] a display means for displaying search results to a user on the terminal;

[1480] A means including an application installed on a terminal carried by a sales staff member in a physical store;

[1481] A method including a process for customers to evaluate their skin condition on the spot and recommend the most suitable skin care products;

[1482] A system including:

[1483] (Claim 2)

[1484] 2. The system of claim 1, wherein the server performs image analysis using a machine learning model or a deep learning model.

[1485] (Claim 3)

[1486] 2. The system according to claim 1, wherein the server generates a list of skin care items based on the evaluation of the skin condition, and the terminal displays the list to the user.

[1487] "Example 2: Combining Emotion Engines"

[1488] (Claim 1)

[1489] an input means for a user to input skin concerns and upload an image showing the current skin condition;

[1490] a transmitting means for transmitting input data and image data from the terminal to the server;

[1491] an image analysis means for the server to analyze the image using a machine learning model or a deep learning model to evaluate the current skin condition;

[1492] emotion evaluation means for the server to analyze the facial expression and voice of the user and evaluate the emotional state of the user;

[1493] a selection means that allows a user to select an ideal skin condition;

[1494] A gap analysis means in which the server analyzes the gap between the current skin condition and the ideal skin condition and takes into account the emotion evaluation result;

[1495] a database search means for the server to search a database for an optimal skin care item to fill the gap;

[1496] a display means for displaying search results to a user on the terminal;

[1497] A system including:

[1498] (Claim 2)

[1499] 10. The system of claim 1, wherein the server performs image analysis and emotion assessment using a machine learning model or a deep learning model.

[1500] (Claim 3)

[1501] 2. The system according to claim 1, wherein the server generates a list of skin care items based on the skin condition evaluation and the emotion evaluation, and the terminal displays the list to the user.

[1502] "Application example 2 when combining emotion engines"

[1503] (Claim 1)

[1504] an input means for a user to input skin concerns and upload an image showing the current skin condition;

[1505] an image analysis means for the server to analyze the image and evaluate the current skin condition;

[1506] a selection means that allows a user to select an ideal skin condition;

[1507] emotion analysis means for assessing the user's emotional state using an emotion engine;

[1508] A gap analysis means for the server to analyze a gap between a current skin condition and an ideal skin condition;

[1509] a database search means for the server to search a database for the most suitable skin care item taking into consideration the gap and the emotional state;

[1510] a display means for displaying search results to a user on the terminal;

[1511] A system including:

[1512] (Claim 2)

[1513] 2. The system of claim 1, wherein the server performs image analysis and sentiment analysis using a machine learning model or a deep learning model.

[1514] (Claim 3)

[1515] 2. The system according to claim 1, wherein the server generates a list of skin care items based on the skin condition evaluation and the emotional state evaluation, and the terminal displays the list to the user. [Explanation of symbols]

[1516] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an input means for a user to input skin concerns and upload an image showing the current skin condition; an image analysis means for the server to analyze the image and evaluate the current skin condition; a selection means that allows a user to select an ideal skin condition; A gap analysis means for the server to analyze a gap between a current skin condition and an ideal skin condition; a database search means for the server to search a database for an optimal skin care item to fill the gap; a display means for displaying search results to a user on the terminal; A system including:

2. The system of claim 1 , wherein the server performs image analysis using a machine learning model or a deep learning model.

3. 2. The system according to claim 1, wherein the server generates a list of skin care items based on the evaluation of the skin condition, and the terminal displays the list to the user.

Images