System

Abstract

A system is provided.SOLUTION: A system comprising: means for analyzing image information received from a user device to evaluate a hair condition; means for recommending hair care products suitable for the user based on the hair condition evaluation using a generated AI; and means for transmitting the recommended hair care products to the user device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] This invention targets users who are concerned about their hair condition, particularly women in their 20s to 40s. With conventional systems, choosing the right shampoo and treatment requires a visit to a beauty salon, which is time-consuming and laborious. Furthermore, home care product recommendations at beauty salons are subjective and can lack consistency. Furthermore, there are issues with the burden of carrying heavy products home and the difficulty of replenishing them at the right time. Given these circumstances, there is a need for a system that can easily and accurately diagnose hair condition at home and recommend and purchase hair care products that meet your needs. [Means for solving the problem]

[0005] The present invention provides a system including the following means: a means for analyzing image information received from a user device to evaluate hair condition; a means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation; and a means for transmitting the recommended hair care products to the user device. The system also includes a means for storing the image information received from the user device in a database, a means for analyzing hair condition using the stored image information, and a means for the generation AI to generate counseling results based on the analysis results. The user device also includes a means for inputting information about the user's hair concerns and needs along with transmitting the image information, and the AI ​​counseling engine generates counseling results taking this information into consideration. This improves the convenience of users by allowing them to easily select and purchase appropriate hair care products from home.

[0006] A "user terminal" is an electronic device used by a user to input, send, and receive information, and includes smartphones, tablets, personal computers, etc.

[0007] "Image information" refers to photo data of hair taken by a user terminal and sent to a server.

[0008] "Analysis" refers to the process of evaluating and determining the condition of hair from received image information, and includes image processing technology using AI.

[0009] "Hair condition" refers to the evaluation results that show hair dryness, damage, color, texture, etc.

[0010] "Generative AI" refers to a system that uses artificial intelligence technology to provide appropriate counseling and product recommendations based on the results of an analysis of the user's hair condition.

[0011] "Hair care products" refers to products used for hair care, such as shampoo, treatment, and conditioner.

[0012] "Recommendation" refers to the act of suggesting optimal hair care products and care methods to users.

[0013] A "database" refers to a storage device for organizing and storing image information, analysis results, etc.

[0014] "Counseling results" refers to the advice and care plan provided by the generative AI based on the results of the hair condition analysis and the user's needs.

[0015] "EC Site" refers to an e-commerce platform where users can browse and purchase hair care products online. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that analyzes image information sent from a user device and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, user device, and generative AI counseling engine.

[0038] User terminal operation

[0039] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0040] Server Operation

[0041] The server receives image information and needs information sent from the user's device and stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0042] How the Generative AI Counseling Engine Works

[0043] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms and natural language processing technology. The generated counseling results and recommended product list are then sent back to the server.

[0044] Server Operation (cont.)

[0045] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easily understandable to the user, and sends it to the user's device.

[0046] User terminal operation (continued)

[0047] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0048] Specific examples

[0049] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0050] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The user launches the app, selects the hair diagnosis option, and then uses the app's camera to take front, back, left, and right images of their hair.

[0054] Step 2:

[0055] Users input their hair concerns and needs (for example, "I'm concerned about dryness" or "My hair is very damaged"), then press the send button to upload the captured image and the input information to the server.

[0056] Step 3:

[0057] The server receives the image information and input information sent from the user terminal, and the received data is stored in a database.

[0058] Step 4:

[0059] The server then passes the stored image information to an analysis module, which uses AI image processing technology to analyze the hair condition, assessing dryness, damage, color, and other factors.

[0060] Step 5:

[0061] The server sends the analysis results to the AI ​​counseling engine, which includes the hair condition evaluation results and information on the user's concerns and needs.

[0062] Step 6:

[0063] The generative AI counseling engine analyzes the data received from the server and generates counseling results and a list of recommended hair care products tailored to the user's hair condition and needs.

[0064] Step 7:

[0065] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine.

[0066] Step 8:

[0067] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and transmits this data to the user's terminal.

[0068] Step 9:

[0069] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0070] Step 10:

[0071] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0072] Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0074] It is not easy for users to choose the right hair care products for their hair health and concerns. Conventional methods require a visit to a beauty salon or specialty store, which is time-consuming and labor-intensive. It is also difficult to receive advice at home to find the right products for a user's hair condition. Furthermore, there is a lack of an objective evaluation method for selecting products that suit individual hair conditions and concerns. This makes it difficult for users to care for their hair efficiently and effectively.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, and means for transmitting the recommended hair care products to the user terminal, thereby enabling the user to easily select and purchase products suitable for their hair condition from the comfort of their own home.

[0077] A "user terminal" is an electronic device that a user uses to take pictures of their hair condition and input information about their hair concerns and needs.

[0078] "Image information" refers to data of photographs or images showing the condition of hair taken by a user terminal.

[0079] "Hair condition" refers to information that indicates the quality and health of hair, such as dryness, damage, and color.

[0080] "Generative AI" refers to an artificial intelligence model that generates appropriate hair care products based on the user's hair condition and concerns.

[0081] "Hair care products" refers to products used for hair care, such as shampoo, treatment, and conditioner.

[0082] "Server" refers to a computer system that receives, stores, analyzes, and transmits information sent from a user terminal to the generative AI counseling engine.

[0083] "Counseling results" refers to the hair care product recommendations generated by the AI ​​based on the user's hair condition and concerns.

[0084] "Database" refers to a data storage system for storing image information and hair information received by the server.

[0085] "Image Analysis Module" refers to software or libraries for assessing hair condition using received image information.

[0086] This invention is a system that analyzes image information sent from a user terminal and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, a user terminal, and a generative AI counseling engine.

[0087] User terminal operation

[0088] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0089] Server Operation

[0090] The server receives image information and hair needs information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module (such as OpenCV) to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0091] How the Generative AI Counseling Engine Works

[0092] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms (e.g., TensorFlow and PyTorch) and natural language processing technologies (e.g., GPT-3). The generated counseling results and recommended product list are then sent back to the server.

[0093] Server Operation (cont.)

[0094] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine. This data is converted into a format that is easily understandable by the user (e.g., HTML or JSON format) and sent to the user's device.

[0095] User terminal operation (continued)

[0096] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0097] Specific examples

[0098] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0099] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0100] Prompt Sentence Examples

[0101] You are a user who is concerned about dry and damaged hair. You launch the dedicated app, take photos of your hair from three angles (front, back, left, and right), and input your concerns such as "I'm worried about dryness" or "I'm very damaged." The server analyzes this information, and the AI ​​generator recommends the most suitable hair care products. What products will be recommended?

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] The user device begins operation when the user accesses a dedicated app or website. The user activates the camera function within the app and takes pictures of the condition of their hair from three directions: front, back, left, and right. They also input text about their hair concerns and needs (e.g., "I'm concerned about dryness" or "My hair is very damaged"). This input information (image and text) becomes the data required for subsequent processing. Before being sent to the server, the image and text data are converted into an appropriate format (e.g., JPEG image, JSON text).

[0105] input:

[0106] Hair images taken by users, hair concerns and needs entered by users

[0107] output:

[0108] Formatted image data and text data sent to the server

[0109] Specific behavior:

[0110] The user launches the app and taps the "Launch Camera" button.

[0111] Take three photos of your hair from the front, back, left and right and save them in the app.

[0112] Enter "I'm concerned about dryness" or "My hair is very damaged" in the text box on the screen and tap the send button.

[0113] Step 2:

[0114] The user device sends the captured image information and the entered text information about the user's worries to the server using an HTTP request. At this time, the format of the request header, etc. is checked to ensure that the image data and text data are sent correctly.

[0115] input:

[0116] Formatted image data and text data

[0117] output:

[0118] HTTP request sent to the server

[0119] Specific behavior:

[0120] The app generates an HTTP request and sends it to the server.

[0121] Notify the user that the request has been sent.

[0122] Step 3:

[0123] The server receives image and text information sent from the user's device and stores it in a database (e.g., MySQL, MongoDB). It then uses an image analysis module (e.g., OpenCV library) to evaluate the hair condition. This evaluation includes algorithms for analyzing features such as dryness, damage, and color.

[0124] input:

[0125] Image data and text data sent from the user's device

[0126] output:

[0127] Data stored in the database, analysis results (hair condition evaluation)

[0128] Specific behavior:

[0129] The server receives the HTTP request and stores the image and text data in a database.

[0130] Image analysis is performed using the OpenCV library to evaluate hair condition (e.g., "dryness" or "damage").

[0131] Step 4:

[0132] The server sends the analysis results and text information about the user's concerns to a generative AI counseling engine. The generative AI counseling engine uses machine learning algorithms (e.g., TensorFlow, PyTorch) and natural language processing techniques (e.g., GPT-3) to recommend the most suitable hair care products for the user. This process generates counseling results based on the evaluation results and the user's concerns.

[0133] input:

[0134] Analysis results (hair condition evaluation), text information (hair concerns and needs)

[0135] output:

[0136] Generated counseling results and recommended product list

[0137] Specific behavior:

[0138] The server sends the analysis results and text information to the generation AI.

[0139] The generative AI analyzes this data and generates a list of recommended products such as "shampoos with high moisturizing effects" and "treatments with damage repair effects."

[0140] Step 5:

[0141] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easy for the user to understand (e.g., HTML or JSON format), and sends it to the user's device.

[0142] input:

[0143] Counseling results and recommended product list returned by the AI

[0144] output:

[0145] Formatted data sent to the user device

[0146] Specific behavior:

[0147] The server receives the recommended product list and converts it into an appropriate format.

[0148] The converted data is sent to the user terminal as an HTTP response.

[0149] Step 6:

[0150] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the appropriate hair care products on the e-commerce site. If necessary, the user can enter their shipping address and payment method to complete the purchase process.

[0151] input:

[0152] Counseling results and recommended product list sent from the server

[0153] output:

[0154] Counseling results, recommended product list, and purchase procedure information displayed to users

[0155] Specific behavior:

[0156] The user terminal receives the response from the server and displays the recommended product list.

[0157] The user selects "Moisturizing Shampoo" and taps the purchase button.

[0158] Enter your shipping address and payment method to complete the purchase.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Conventional hair care product recommendation systems are limited to online use and do not support use in physical stores. As a result, it is not possible for customers to evaluate and analyze their hair condition in real time in physical stores and recommend the most suitable hair care products. Another problem is that it is difficult for customers to easily input their hair condition and concerns and receive on-the-spot counseling.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes means for analyzing image information received from a user terminal and evaluating the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, means for transmitting the recommended hair care products to the user terminal, means for photographing and analyzing the user's hair condition in real time using a smart mirror installed in a physical store, and means for displaying recommended product information to the user through the smart mirror. This makes it possible to evaluate a customer's hair condition even in a physical store and recommend optimal hair care products on the spot using a generation AI.

[0164] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0165] "Image information" is data of photographs or videos of the user's hair condition.

[0166] The "means for assessing hair condition" is a function that analyzes the received image information to determine hair dryness, damage, color, etc.

[0167] "Generative AI" is artificial intelligence that uses machine learning algorithms and natural language processing technology to recommend optimal hair care products.

[0168] "Hair care products" are products for caring for hair, such as shampoo, treatment, and conditioner.

[0169] A "smart mirror" is an electronic device in the shape of a mirror that is installed in a physical store and can photograph and analyze the condition of a customer's hair in real time and display the results.

[0170] The "means for displaying recommended product information to the user" is a function for visually presenting recommended hair care products to the user.

[0171] A "database" is a digital storage system for storing and managing image information, analysis results, and user information.

[0172] The "counseling results" are hair care advice and a list of recommended products tailored to the user, analyzed and generated by AI.

[0173] The "analysis results" are the results of evaluating the condition of the hair based on image information.

[0174] "Information about concerns and needs" is data entered by the user regarding hair-related problems and desired hair care effects.

[0175] This invention provides a hair care product recommendation system suitable for use in brick-and-mortar stores. Specifically, it consists of a server, a user terminal, a generative AI counseling engine, and a smart mirror installed in the brick-and-mortar store.

[0176] The server analyzes image information received from the user's device to evaluate the condition of the hair. For example, a user stands in front of a smart mirror in a physical store, and the smart mirror uses a camera to capture the condition of the hair in real time. This image information is sent to the server, which then evaluates the hair condition using an image processing library such as OpenCV. The evaluation results are expressed as parameters such as dryness, damage, and color.

[0177] The server then sends the evaluation results to a generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the most suitable hair care products for the user. The generated list of recommended products is then sent back to the server, which then displays the list on the smart mirror, allowing the user to check the appropriate products on the spot.

[0178] Furthermore, image information received from the user's device and information about the user's hair concerns and needs are stored in a database. This stored data can be used in subsequent consultations, enabling more personalized suggestions.

[0179] As a concrete example, when a user visits a brick-and-mortar hair salon, they stand in front of a smart mirror. The smart mirror takes a photo and analyzes the user's hair condition, and based on the results, the AI ​​generator recommends "shampoos with high moisturizing effects" and "treatments with damage repair effects." This list of recommended products is displayed on the smart mirror, allowing the user to check the products on the spot. It is also possible for users to directly input their concerns and needs using the smart mirror.

[0180] Example prompt sentence:

[0181] "Recommend an appropriate hair care product based on image data showing the dryness of hair. The image file name is 'captured_image.jpg'. In your recommendation, please include the specific product name and the reason for your recommendation."

[0182] As described above, the present invention provides a system that can evaluate hair condition in real time even in a physical store and recommend optimal hair care products.

[0183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0184] Step 1:

[0185] A user stands in front of a smart mirror in a physical store. The camera built into the smart mirror takes a picture of the user's hair condition. The captured image information is sent from the smart mirror to the server.

[0186] Input: Image of the user's hair condition

[0187] Output: Image data sent to the server

[0188] Step 2:

[0189] The server inputs the received image data into the image analysis module, which uses the OpenCV library to evaluate hair conditions such as dryness, damage, and color. The analysis results are extracted as parameters.

[0190] Input: Photographed image data

[0191] Output: Evaluated hair condition parameters (dryness, damage, color, etc.)

[0192] Step 3:

[0193] The server sends the evaluation results to the generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the best hair care products for the user. A list of recommended products is then sent back to the server.

[0194] Input: Hair condition parameters

[0195] Output: A list of recommended hair care products

[0196] Step 4:

[0197] The server sends the recommended product list returned by the generative AI counseling engine to the smart mirror, which then displays the information to the user.

[0198] Input: List of recommended hair care products

[0199] Output: Recommended product information displayed to the user

[0200] Step 5:

[0201] The smart mirror allows users to input information about their hair concerns and needs, which is then sent to a server.

[0202] Input: Information about hair concerns and needs entered by the user

[0203] Output: User concerns and needs sent to the server

[0204] Step 6:

[0205] The server stores the received information on the user's worries and needs in a database, which can be used for future counseling.

[0206] Input: Information about the user's concerns and needs

[0207] Output: Data on worries and needs recorded in a database

[0208] Step 7:

[0209] Furthermore, the server will re-launch the generative AI counseling engine based on the stored data to generate more personalized counseling results, which will also be displayed on the smart mirror.

[0210] Input: Information about user concerns and needs stored in a database

[0211] Output: More personalized counseling results

[0212] Through these steps, users can receive a real-time evaluation of their hair condition even in a physical store, and be recommended the most suitable hair care products.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0215] User terminal operation

[0216] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and simultaneously input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information are sent from the user device to the server.

[0217] Server Operation

[0218] The server receives image and emotion information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and the results are also stored in the database. This evaluation result is sent to the generative AI counseling engine.

[0219] How the Generative AI Counseling Engine Works

[0220] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0221] Server Operation (cont.)

[0222] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and sends it to the user's device. The user's device displays the counseling results and recommended product list, allowing the user to easily select products on the EC site and complete the purchase procedure.

[0223] User terminal operation (continued)

[0224] The user can then review the displayed counseling results and recommended product list, select and purchase hair care products that suit their preferences, and the system will provide the best options for the user, taking into consideration the user's emotional state, such as stress and satisfaction.

[0225] Specific examples

[0226] For example, User B is troubled by dry and faded hair, and has recently been under a lot of stress. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate the user's emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B immediately purchases the recommended products on an e-commerce site.

[0227] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, resulting in more personalized care and increased user satisfaction.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] The user launches the app and selects the hair diagnosis option. They then use the app's camera to take front, back, left, and right images of their hair. At this time, the app also captures an image of the user's face using facial recognition.

[0231] Step 2:

[0232] The user inputs emotional information. Emotional information is collected through simple questions within the app or through voice input via a microphone. For example, emotional data is collected by answering questions such as, "What is your stress level these days?"

[0233] Step 3:

[0234] The user presses a send button to send the captured hair image and emotion information to the server. The user terminal uploads the hair image data, emotion information, and face image data to the server.

[0235] Step 4:

[0236] The server receives the image information and emotion information sent from the user terminal, and the received data is stored in a database.

[0237] Step 5:

[0238] The server then passes the stored image information to an image analysis module, which analyzes the hair condition. This analysis uses AI image processing technology to evaluate hair dryness, damage, color, etc. At the same time, facial recognition technology is used to analyze emotional data.

[0239] Step 6:

[0240] The emotion engine uses facial recognition and voice analysis technologies to assess the user's emotions. The results of facial recognition and voice analysis are integrated to assess the user's stress level, satisfaction, and emotional state. These results are also stored in a database.

[0241] Step 7:

[0242] The server sends the hair condition evaluation results and emotion evaluation results to the AI ​​counseling engine. The data sent includes the hair condition evaluation results and the user's emotional state.

[0243] Step 8:

[0244] The generative AI counseling engine analyzes the data received from the server and generates counseling results and recommended hair care products suitable for the user. This generation process uses natural language processing technology and machine learning algorithms.

[0245] Step 9:

[0246] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine. The AI ​​also takes into account the user's emotional state, and adjusts the recommendation accordingly, for example, recommending products with a relaxing effect if the user is feeling stressed.

[0247] Step 10:

[0248] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and sends this data to the user's terminal. The user can then review it and select a product.

[0249] Step 11:

[0250] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0251] Step 12:

[0252] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0253] Example 2

[0254] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] Conventional hair care product recommendation systems recommend products based solely on the user's hair condition, without taking into account psychological factors such as the user's emotional state or stress level. This makes it difficult to recommend products that perfectly match the user's preferences and needs, resulting in a lack of user satisfaction. Furthermore, although recommendation systems that utilize user emotional information exist, these systems still have room for improvement in terms of accuracy and response speed.

[0256] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0257] In this invention, the server includes a means for analyzing image information received from a user terminal to evaluate the hair condition, a means for analyzing emotional information received from the user terminal to evaluate the user's emotional state, and a means for using a generative AI to recommend hair care products suitable for the user based on the hair condition evaluation and the emotional state evaluation. This enables more personalized hair care product recommendations that take into account not only the user's hair condition but also their emotional state. This also increases user satisfaction and enables the rapid provision of products that meet the user's needs.

[0258] A "user terminal" is a device that allows a user to input information and acquire and transmit image information and emotional information. This includes smartphones and PCs.

[0259] "Image information" is visual data such as photos or videos showing the condition of the user's hair.

[0260] "Emotional information" is data that indicates the user's psychological state and emotions, obtained based on the user's facial expressions and voice analysis.

[0261] A "server" is a computing device that receives data sent from a user terminal, stores it in a database, and analyzes it.

[0262] A "database" is a structured information storage system for storing information received by a server.

[0263] The "image analysis module" is a software and hardware component for assessing hair condition, which can use machine learning algorithms.

[0264] An "emotion engine" is a system for analyzing emotional information and assessing a user's emotional state, including facial recognition and voice analysis technologies.

[0265] The "generative AI counseling engine" is a system that uses a generative AI model to recommend the most suitable hair care products to users based on the results of hair condition assessment and emotional assessment.

[0266] The "counseling results" are advice and a list of recommended hair care products based on the user's hair condition and emotional state.

[0267] The "recommended product list" is a list of hair care products suitable for the user, generated by the generative AI counseling engine.

[0268] "Natural language processing technology" is a technical field in computer science for understanding and generating text data.

[0269] A "machine learning algorithm" is a computational method for analyzing data, learning patterns, and making predictions and classifications.

[0270] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0271] User terminal operation

[0272] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and also input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information is sent from the user device to the server.

[0273] Server Operation

[0274] The server receives image and emotion information sent from the user's device and first stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and stores the results in the database.

[0275] How the Generative AI Counseling Engine Works

[0276] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0277] Server Operation (cont.)

[0278] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and transmits it to the user's terminal.

[0279] User terminal operation (continued)

[0280] The user terminal displays the counseling results and a list of recommended products, and the user can check the displayed information, select and purchase hair care products that suit their preferences. At this time, the user's emotional state is taken into consideration, and the best options are provided for the user.

[0281] Specific examples

[0282] For example, User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate their emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B can then purchase the recommended products on an e-commerce site.

[0283] Prompt Sentence Examples

[0284] Below are some example prompts for a generative AI model to recommend appropriate hair care products based on the user's hair condition and emotional information:

[0285] The user's hair is dry and faded. They've also been under a lot of stress lately. Recommend them the best hair care products. For example, you could recommend a moisturizing shampoo, a color-preserving treatment, or even a relaxing scented hairspray.

[0286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0287] Step 1: The user accesses a dedicated app or website

[0288] The user accesses a dedicated app or website using a smartphone or PC. The app or website displays an interface for understanding the condition of the hair and instructs the user to take a photo of the condition of the hair. The input is the user's action (launching the app), and the output is a form that displays instructions for taking a photo.

[0289] Step 2: The user takes a photo of their hair condition and inputs emotional information.

[0290] The user follows the app's instructions to take a photo of the condition of their hair. The app also recognizes facial expressions through the camera and obtains emotional information by analyzing audio through the microphone. The input is the photo and audio data taken by the user, and the output is image information and emotional information. This information is sent from the user's device to the server.

[0291] Step 3: The server receives the image information and emotion information and stores them in the database.

[0292] The server receives image information and emotion information sent from the user terminal. It stores this data in a database. The input is the image information and emotion information received from the user terminal, and the output is the records stored in the database.

[0293] Step 4: The server uses the image analysis module to evaluate the hair condition

[0294] The server uses an image analysis module to analyze the received image information and evaluate the hair condition (dryness, damage, color, etc.). Specifically, it analyzes the image data using, for example, TensorFlow, a machine learning library, and outputs the hair condition as quantified data. The input is the image information stored in the database, and the output is the hair condition evaluation result.

[0295] Step 5: The server evaluates the emotional state using the emotion engine

[0296] The server uses an emotion engine to analyze the received emotion information and evaluate the user's emotional state (stress, relaxation, etc.). Specifically, it analyzes the emotion data using face recognition algorithms and voice analysis technology, and outputs the emotional state as numerically quantified and classified data. The input is the emotion information stored in the database, and the output is the emotional state evaluation result.

[0297] Step 6: The server sends the evaluation results to the generative AI counseling engine

[0298] The server sends the hair condition evaluation results and emotion evaluation results to the generative AI counseling engine. The inputs are the hair condition evaluation results and emotion state evaluation results, and the generative AI receives these data as outputs.

[0299] Step 7: The generative AI counseling engine generates appropriate counseling results and products.

[0300] The generative AI counseling engine generates counseling results and recommended hair care products suitable for the user based on the received evaluation results. This process uses natural language processing technology and machine learning algorithms. Specifically, the evaluation results are analyzed as input data, and the generative AI outputs counseling results and a product list based on the prompt text. The inputs are the hair condition evaluation results and emotional state evaluation results, and the output is counseling results and a list of recommended products.

[0301] Step 8: Send the results from the generative AI counseling engine back to the server

[0302] The generative AI counseling engine sends the generated counseling results and recommended product list to the server. The input is the counseling results and recommended product list generated by the generative AI, and the server receives these results as output.

[0303] Step 9: The server formats the consultation results and product list

[0304] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand. Specifically, it converts them into HTML or JSON format. The input is the counseling results and recommended product list received from the generation AI, and the output is the formatted data.

[0305] Step 10: The server sends the formatted results to the user's device.

[0306] The server sends the formatted counseling results and the product list to the user terminal. The formatted counseling results and the product list are input, and the output is sent to the user terminal.

[0307] Step 11: User reviews results and selects product

[0308] The user terminal displays the received counseling results and the recommended product list. The user checks the displayed information and selects hair care products that suit their preferences. The input is the counseling results and product list sent from the server, and the output is a product selection based on the user's preferences.

[0309] Step 12: The user purchases the product on the e-commerce site.

[0310] The user purchases the recommended product on an e-commerce website. Apps and websites often provide a purchase link for the recommended product, which the user can click to proceed directly with the purchase process. The input is product information selected by the user, and the output is information indicating that the purchase has been completed.

[0311] (Application example 2)

[0312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0313] Conventional hair care product recommendation systems only evaluate the user's hair condition, making it difficult to provide personalized recommendations that reflect the user's emotional state. As a result, while they can recommend appropriate hair care products based on the user's hair condition, they cannot recommend products that match the user's emotional state at the time, which prevents them from fully improving user satisfaction.

[0314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0315] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for analyzing the user's emotional information, means for recommending hair care products suitable for the user using a generative AI model based on the hair condition evaluation and the emotional information analysis, and means for transmitting the recommended hair care products to the user terminal, thereby making it possible to recommend optimal hair care products that take into account the user's emotional state as well as their hair condition.

[0316] A "user terminal" is a device used by a user, including a smartphone, tablet, personal computer, smart glasses, etc.

[0317] "Image information" is visual data captured or acquired using a user terminal, and includes hair condition, facial expression, and the like.

[0318] "Hair condition" is information indicating attributes such as the user's hair type, dryness, whether or not it is damaged, and color tone.

[0319] "Emotion information" is information indicating the emotional state of the user obtained from the user's facial expression and voice, and includes, for example, joy, anger, sadness, surprise, fear, and the like.

[0320] A "generative AI model" is a technology that uses pre-trained AI algorithms to generate optimal outputs for specific inputs, and utilizes natural language processing technology and machine learning algorithms.

[0321] "Hair care products" are products used to maintain the health and beauty of hair, including shampoos, treatments, hairsprays, etc.

[0322] "Recommend" means presenting the best option based on specific criteria.

[0323] A "server" is a computer system that processes, stores, and distributes data, analyzes information received from user devices, and executes generative AI models.

[0324] A "database" is a system for organizing and storing information in digital form.

[0325] The "counseling result" is information that indicates advice or recommendations for the user, generated based on the analysis results of image information and emotion information.

[0326] The system for implementing this invention analyzes the user's hair condition and emotional information, and uses a generative AI model to recommend optimal hair care products. The specific configuration and operation procedure are shown below.

[0327] composition

[0328] User terminal

[0329] The user terminals are devices such as smartphones, tablets, personal computers, and smart glasses. These terminals are equipped with cameras and microphones to capture image information (hair condition and facial expression) and audio. Through a dedicated application or website, users can take photos of their hair condition and input emotional information.

[0330] server

[0331] The server is a computer system that processes and analyzes image and emotion information received from user devices. The server has the following functions:

[0332] 1. Database: Stores image information and emotion information received from users.

[0333] 2. Image analysis module: Evaluates hair condition and recognizes facial expressions.

[0334] 3. Sentiment Analysis Module: Evaluates the user's emotions using voice analysis.

[0335] 4. Generative AI Counseling Engine: Integrates the user's hair condition and emotional information to generate optimal counseling results and hair care products. Generative AI models used include GPT-3.

[0336] Operating Procedure

[0337] User terminal operation

[0338] Using a dedicated application, users can take photos of their hair condition and simultaneously input facial expressions and voice, and this information is sent from the user's device to the server.

[0339] Server Operation

[0340] The server stores the received image and emotion information in a database. It then uses an image analysis module to break down and evaluate the hair condition into factors such as dryness, damage, and color. It also uses an emotion analysis module to classify the user's emotional state into categories such as joy, anger, and sadness.

[0341] Generative AI Counseling Engine

[0342] The generative AI counseling engine operates based on the results of image analysis and emotion analysis. For example, if a user feels "stressed" and their hair condition is "dry" and "faded," it will generate recommended products such as "a highly moisturizing shampoo" or "a hairspray with a relaxing scent." Prompt text is used for this generation.

[0343] Example prompt sentence:

[0344] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[0345] Sending and viewing recommendations

[0346] The generated counseling results and recommended product list are sent from the server to the user's terminal and visually displayed on the user's terminal, allowing the user to easily purchase hair care products based on the recommendations.

[0347] Specific examples

[0348] Let's say User B is stressed about dry, faded hair. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, facial expression recognition and voice analysis are performed, and this information is sent to the server. The server evaluates the hair condition, and the emotion analysis module recognizes User B's stress level. Based on this information, the generative AI counseling engine recommends a "highly moisturizing shampoo" and a "hairspray with a relaxing scent," and sends the results to the user's device. User B can then purchase the products on the e-commerce site based on the recommendations.

[0349] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, providing more personalized care.

[0350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0351] Step 1:

[0352] The user launches a dedicated application and takes a photo of the condition of their hair. At the same time, they input their facial expression and voice. This data (image information and voice information) is captured by the user's device. The input data is an image showing the condition of the hair, and facial expression and voice that capture the user's emotional state. This data is sent from the user's device to the server.

[0353] Step 2:

[0354] The server stores the image and audio information received from the user terminal in a database. The input data is the user's image and audio information, and each data is stored in the database. This saves the data for subsequent analysis processing.

[0355] Step 3:

[0356] The server's image analysis module evaluates hair condition based on image information stored in a database. The input data is image information, and the output data is the evaluation results for hair dryness, damage, color, etc. Specifically, the image is analyzed, pixel-based features are extracted, and the hair condition is classified.

[0357] Step 4:

[0358] At the same time, the server's emotion analysis module analyzes the voice information to assess the user's emotional state. The input data is the voice information, and the output data is the emotion assessment results, such as joy, anger, sadness, and stress. The voice analysis calculates emotion vectors of pitch, tone, and vocabulary to classify emotions.

[0359] Step 5:

[0360] The server sends the hair condition assessment results and emotion assessment results to the generative AI counseling engine. The generative AI counseling engine recommends optimal hair care products to the user based on this data. The input data are the hair condition assessment results and emotion assessment results, and the output data is a list of recommended products. For example, based on data such as "the user's emotional state is stressed and their hair is dry," the generative AI uses prompt sentences to select appropriate hair care products.

[0361] Step 6:

[0362] Example prompt sentence:

[0363] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[0364] The server's generative AI counseling engine uses the prompt sentences shown above to request the generative AI model to generate recommended products. This process allows the AI ​​model to use natural language processing technology to create a list of appropriate products.

[0365] Step 7:

[0366] The server sends the generated counseling results and recommended product list to the user's device. The input data are the recommendations made by the generative AI counseling engine, and the output data are the counseling results and product list in a format that is visually displayed to the user.

[0367] Step 8:

[0368] The user's device displays the received counseling results and a list of recommended products. The user can check this display and select hair care products that suit their preferences. This allows the user to purchase the recommended products directly on the e-commerce site. As a result, the user is provided with hair care products that are optimal for their hair condition and emotional state.

[0369] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0370] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0371] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0372] [Second embodiment]

[0373] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0374] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0375] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0376] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0377] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0378] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0379] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0380] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0381] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0382] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0383] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0384] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0385] This invention is a system that analyzes image information sent from a user device and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, user device, and generative AI counseling engine.

[0386] User terminal operation

[0387] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0388] Server Operation

[0389] The server receives image information and needs information sent from the user's device and stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0390] How the Generative AI Counseling Engine Works

[0391] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms and natural language processing technology. The generated counseling results and recommended product list are then sent back to the server.

[0392] Server Operation (cont.)

[0393] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easily understandable to the user, and sends it to the user's device.

[0394] User terminal operation (continued)

[0395] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0396] Specific examples

[0397] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0398] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0399] The processing flow will be explained below.

[0400] Step 1:

[0401] The user launches the app, selects the hair diagnosis option, and then uses the app's camera to take front, back, left, and right images of their hair.

[0402] Step 2:

[0403] Users input their hair concerns and needs (for example, "I'm concerned about dryness" or "My hair is very damaged"), then press the send button to upload the captured image and the input information to the server.

[0404] Step 3:

[0405] The server receives the image information and input information sent from the user terminal, and the received data is stored in a database.

[0406] Step 4:

[0407] The server then passes the stored image information to an analysis module, which uses AI image processing technology to analyze the hair condition, assessing dryness, damage, color, and other factors.

[0408] Step 5:

[0409] The server sends the analysis results to the AI ​​counseling engine, which includes the hair condition evaluation results and information on the user's concerns and needs.

[0410] Step 6:

[0411] The generative AI counseling engine analyzes the data received from the server and generates counseling results and a list of recommended hair care products tailored to the user's hair condition and needs.

[0412] Step 7:

[0413] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine.

[0414] Step 8:

[0415] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and transmits this data to the user's terminal.

[0416] Step 9:

[0417] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0418] Step 10:

[0419] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0420] Example 1

[0421] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0422] It is not easy for users to choose the right hair care products for their hair health and concerns. Conventional methods require a visit to a beauty salon or specialty store, which is time-consuming and labor-intensive. It is also difficult to receive advice at home to find the right products for a user's hair condition. Furthermore, there is a lack of an objective evaluation method for selecting products that suit individual hair conditions and concerns. This makes it difficult for users to care for their hair efficiently and effectively.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0424] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, and means for transmitting the recommended hair care products to the user terminal, thereby enabling the user to easily select and purchase products suitable for their hair condition from the comfort of their own home.

[0425] A "user terminal" is an electronic device that a user uses to take pictures of their hair condition and input information about their hair concerns and needs.

[0426] "Image information" refers to data of photographs or images showing the condition of hair taken by a user terminal.

[0427] "Hair condition" refers to information that indicates the quality and health of hair, such as dryness, damage, and color.

[0428] "Generative AI" refers to an artificial intelligence model that generates appropriate hair care products based on the user's hair condition and concerns.

[0429] "Hair care products" refers to products used for hair care, such as shampoo, treatment, and conditioner.

[0430] "Server" refers to a computer system that receives, stores, analyzes, and transmits information sent from a user terminal to the generative AI counseling engine.

[0431] "Counseling results" refers to the hair care product recommendations generated by the AI ​​based on the user's hair condition and concerns.

[0432] "Database" refers to a data storage system for storing image information and hair information received by the server.

[0433] "Image Analysis Module" refers to software or libraries for assessing hair condition using received image information.

[0434] This invention is a system that analyzes image information sent from a user terminal and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, a user terminal, and a generative AI counseling engine.

[0435] User terminal operation

[0436] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0437] Server Operation

[0438] The server receives image information and hair needs information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module (such as OpenCV) to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0439] How the Generative AI Counseling Engine Works

[0440] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms (e.g., TensorFlow and PyTorch) and natural language processing technologies (e.g., GPT-3). The generated counseling results and recommended product list are then sent back to the server.

[0441] Server Operation (cont.)

[0442] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine. This data is converted into a format that is easily understandable by the user (e.g., HTML or JSON format) and sent to the user's device.

[0443] User terminal operation (continued)

[0444] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0445] Specific examples

[0446] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0447] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0448] Prompt Sentence Examples

[0449] You are a user who is concerned about dry and damaged hair. You launch the dedicated app, take photos of your hair from three angles (front, back, left, and right), and input your concerns such as "I'm worried about dryness" or "I'm very damaged." The server analyzes this information, and the AI ​​generator recommends the most suitable hair care products. What products will be recommended?

[0450] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0451] Step 1:

[0452] The user device begins operation when the user accesses a dedicated app or website. The user activates the camera function within the app and takes pictures of the condition of their hair from three directions: front, back, left, and right. They also input text about their hair concerns and needs (e.g., "I'm concerned about dryness" or "My hair is very damaged"). This input information (image and text) becomes the data required for subsequent processing. Before being sent to the server, the image and text data are converted into an appropriate format (e.g., JPEG image, JSON text).

[0453] input:

[0454] Hair images taken by users, hair concerns and needs entered by users

[0455] output:

[0456] Formatted image data and text data sent to the server

[0457] Specific behavior:

[0458] The user launches the app and taps the "Launch Camera" button.

[0459] Take three photos of your hair from the front, back, left and right and save them in the app.

[0460] Enter "I'm concerned about dryness" or "My hair is very damaged" in the text box on the screen and tap the send button.

[0461] Step 2:

[0462] The user device sends the captured image information and the entered text information about the user's worries to the server using an HTTP request. At this time, the format of the request header, etc. is checked to ensure that the image data and text data are sent correctly.

[0463] input:

[0464] Formatted image data and text data

[0465] output:

[0466] HTTP request sent to the server

[0467] Specific behavior:

[0468] The app generates an HTTP request and sends it to the server.

[0469] Notify the user that the request has been sent.

[0470] Step 3:

[0471] The server receives image and text information sent from the user's device and stores it in a database (e.g., MySQL, MongoDB). It then uses an image analysis module (e.g., OpenCV library) to evaluate the hair condition. This evaluation includes algorithms for analyzing features such as dryness, damage, and color.

[0472] input:

[0473] Image data and text data sent from the user's device

[0474] output:

[0475] Data stored in the database, analysis results (hair condition evaluation)

[0476] Specific behavior:

[0477] The server receives the HTTP request and stores the image and text data in a database.

[0478] Image analysis is performed using the OpenCV library to evaluate hair condition (e.g., "dryness" or "damage").

[0479] Step 4:

[0480] The server sends the analysis results and text information about the user's concerns to a generative AI counseling engine. The generative AI counseling engine uses machine learning algorithms (e.g., TensorFlow, PyTorch) and natural language processing techniques (e.g., GPT-3) to recommend the most suitable hair care products for the user. This process generates counseling results based on the evaluation results and the user's concerns.

[0481] input:

[0482] Analysis results (hair condition evaluation), text information (hair concerns and needs)

[0483] output:

[0484] Generated counseling results and recommended product list

[0485] Specific behavior:

[0486] The server sends the analysis results and text information to the generation AI.

[0487] The generative AI analyzes this data and generates a list of recommended products such as "shampoos with high moisturizing effects" and "treatments with damage repair effects."

[0488] Step 5:

[0489] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easy for the user to understand (e.g., HTML or JSON format), and sends it to the user's device.

[0490] input:

[0491] Counseling results and recommended product list returned by the AI

[0492] output:

[0493] Formatted data sent to the user device

[0494] Specific behavior:

[0495] The server receives the recommended product list and converts it into an appropriate format.

[0496] The converted data is sent to the user terminal as an HTTP response.

[0497] Step 6:

[0498] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the appropriate hair care products on the e-commerce site. If necessary, the user can enter their shipping address and payment method to complete the purchase process.

[0499] input:

[0500] Counseling results and recommended product list sent from the server

[0501] output:

[0502] Counseling results, recommended product list, and purchase procedure information displayed to users

[0503] Specific behavior:

[0504] The user terminal receives the response from the server and displays the recommended product list.

[0505] The user selects "Moisturizing Shampoo" and taps the purchase button.

[0506] Enter your shipping address and payment method to complete the purchase.

[0507] (Application example 1)

[0508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] Conventional hair care product recommendation systems are limited to online use and do not support use in physical stores. As a result, it is not possible for customers to evaluate and analyze their hair condition in real time in physical stores and recommend the most suitable hair care products. Another problem is that it is difficult for customers to easily input their hair condition and concerns and receive on-the-spot counseling.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0511] In this invention, the server includes means for analyzing image information received from a user terminal and evaluating the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, means for transmitting the recommended hair care products to the user terminal, means for photographing and analyzing the user's hair condition in real time using a smart mirror installed in a physical store, and means for displaying recommended product information to the user through the smart mirror. This makes it possible to evaluate a customer's hair condition even in a physical store and recommend optimal hair care products on the spot using a generation AI.

[0512] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0513] "Image information" is data of photographs or videos of the user's hair condition.

[0514] The "means for assessing hair condition" is a function that analyzes the received image information to determine hair dryness, damage, color, etc.

[0515] "Generative AI" is artificial intelligence that uses machine learning algorithms and natural language processing technology to recommend optimal hair care products.

[0516] "Hair care products" are products for caring for hair, such as shampoo, treatment, and conditioner.

[0517] A "smart mirror" is an electronic device in the shape of a mirror that is installed in a physical store and can photograph and analyze the condition of a customer's hair in real time and display the results.

[0518] The "means for displaying recommended product information to the user" is a function for visually presenting recommended hair care products to the user.

[0519] A "database" is a digital storage system for storing and managing image information, analysis results, and user information.

[0520] The "counseling results" are hair care advice and a list of recommended products tailored to the user, analyzed and generated by AI.

[0521] The "analysis results" are the results of evaluating the condition of the hair based on image information.

[0522] "Information about concerns and needs" is data entered by the user regarding hair-related problems and desired hair care effects.

[0523] This invention provides a hair care product recommendation system suitable for use in brick-and-mortar stores. Specifically, it consists of a server, a user terminal, a generative AI counseling engine, and a smart mirror installed in the brick-and-mortar store.

[0524] The server analyzes image information received from the user's device to evaluate the condition of the hair. For example, a user stands in front of a smart mirror in a physical store, and the smart mirror uses a camera to capture the condition of the hair in real time. This image information is sent to the server, which then evaluates the hair condition using an image processing library such as OpenCV. The evaluation results are expressed as parameters such as dryness, damage, and color.

[0525] The server then sends the evaluation results to a generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the most suitable hair care products for the user. The generated list of recommended products is then sent back to the server, which then displays the list on the smart mirror, allowing the user to check the appropriate products on the spot.

[0526] Furthermore, image information received from the user's device and information about the user's hair concerns and needs are stored in a database. This stored data can be used in subsequent consultations, enabling more personalized suggestions.

[0527] As a concrete example, when a user visits a brick-and-mortar hair salon, they stand in front of a smart mirror. The smart mirror takes a photo and analyzes the user's hair condition, and based on the results, the AI ​​generator recommends "shampoos with high moisturizing effects" and "treatments with damage repair effects." This list of recommended products is displayed on the smart mirror, allowing the user to check the products on the spot. It is also possible for users to directly input their concerns and needs using the smart mirror.

[0528] Example prompt sentence:

[0529] "Recommend an appropriate hair care product based on image data showing the dryness of hair. The image file name is 'captured_image.jpg'. In your recommendation, please include the specific product name and the reason for your recommendation."

[0530] As described above, the present invention provides a system that can evaluate hair condition in real time even in a physical store and recommend optimal hair care products.

[0531] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0532] Step 1:

[0533] A user stands in front of a smart mirror in a physical store. The camera built into the smart mirror takes a picture of the user's hair condition. The captured image information is sent from the smart mirror to the server.

[0534] Input: Image of the user's hair condition

[0535] Output: Image data sent to the server

[0536] Step 2:

[0537] The server inputs the received image data into the image analysis module, which uses the OpenCV library to evaluate hair conditions such as dryness, damage, and color. The analysis results are extracted as parameters.

[0538] Input: Photographed image data

[0539] Output: Evaluated hair condition parameters (dryness, damage, color, etc.)

[0540] Step 3:

[0541] The server sends the evaluation results to the generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the best hair care products for the user. A list of recommended products is then sent back to the server.

[0542] Input: Hair condition parameters

[0543] Output: A list of recommended hair care products

[0544] Step 4:

[0545] The server sends the recommended product list returned by the generative AI counseling engine to the smart mirror, which then displays the information to the user.

[0546] Input: List of recommended hair care products

[0547] Output: Recommended product information displayed to the user

[0548] Step 5:

[0549] The smart mirror allows users to input information about their hair concerns and needs, which is then sent to a server.

[0550] Input: Information about hair concerns and needs entered by the user

[0551] Output: User concerns and needs sent to the server

[0552] Step 6:

[0553] The server stores the received information on the user's worries and needs in a database, which can be used for future counseling.

[0554] Input: Information about the user's concerns and needs

[0555] Output: Data on worries and needs recorded in a database

[0556] Step 7:

[0557] Furthermore, the server will re-launch the generative AI counseling engine based on the stored data to generate more personalized counseling results, which will also be displayed on the smart mirror.

[0558] Input: Information about user concerns and needs stored in a database

[0559] Output: More personalized counseling results

[0560] Through these steps, users can receive a real-time evaluation of their hair condition even in a physical store, and be recommended the most suitable hair care products.

[0561] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0562] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0563] User terminal operation

[0564] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and simultaneously input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information are sent from the user device to the server.

[0565] Server Operation

[0566] The server receives image and emotion information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and the results are also stored in the database. This evaluation result is sent to the generative AI counseling engine.

[0567] How the Generative AI Counseling Engine Works

[0568] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0569] Server Operation (cont.)

[0570] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and sends it to the user's device. The user's device displays the counseling results and recommended product list, allowing the user to easily select products on the EC site and complete the purchase procedure.

[0571] User terminal operation (continued)

[0572] The user can then review the displayed counseling results and recommended product list, select and purchase hair care products that suit their preferences, and the system will provide the best options for the user, taking into consideration the user's emotional state, such as stress and satisfaction.

[0573] Specific examples

[0574] For example, User B is troubled by dry and faded hair, and has recently been under a lot of stress. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate the user's emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B immediately purchases the recommended products on an e-commerce site.

[0575] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, resulting in more personalized care and increased user satisfaction.

[0576] The processing flow will be explained below.

[0577] Step 1:

[0578] The user launches the app and selects the hair diagnosis option. They then use the app's camera to take front, back, left, and right images of their hair. At this time, the app also captures an image of the user's face using facial recognition.

[0579] Step 2:

[0580] The user inputs emotional information. Emotional information is collected through simple questions within the app or through voice input via a microphone. For example, emotional data is collected by answering questions such as, "What is your stress level these days?"

[0581] Step 3:

[0582] The user presses a send button to send the captured hair image and emotion information to the server. The user terminal uploads the hair image data, emotion information, and face image data to the server.

[0583] Step 4:

[0584] The server receives the image information and emotion information sent from the user terminal, and the received data is stored in a database.

[0585] Step 5:

[0586] The server then passes the stored image information to an image analysis module, which analyzes the hair condition. This analysis uses AI image processing technology to evaluate hair dryness, damage, color, etc. At the same time, facial recognition technology is used to analyze emotional data.

[0587] Step 6:

[0588] The emotion engine uses facial recognition and voice analysis technologies to assess the user's emotions. The results of facial recognition and voice analysis are integrated to assess the user's stress level, satisfaction, and emotional state. These results are also stored in a database.

[0589] Step 7:

[0590] The server sends the hair condition evaluation results and emotion evaluation results to the AI ​​counseling engine. The data sent includes the hair condition evaluation results and the user's emotional state.

[0591] Step 8:

[0592] The generative AI counseling engine analyzes the data received from the server and generates counseling results and recommended hair care products suitable for the user. This generation process uses natural language processing technology and machine learning algorithms.

[0593] Step 9:

[0594] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine. The AI ​​also takes into account the user's emotional state, and adjusts the recommendation accordingly, for example, recommending products with a relaxing effect if the user is feeling stressed.

[0595] Step 10:

[0596] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and sends this data to the user's terminal. The user can then review it and select a product.

[0597] Step 11:

[0598] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0599] Step 12:

[0600] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0601] Example 2

[0602] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] Conventional hair care product recommendation systems recommend products based solely on the user's hair condition, without taking into account psychological factors such as the user's emotional state or stress level. This makes it difficult to recommend products that perfectly match the user's preferences and needs, resulting in a lack of user satisfaction. Furthermore, although recommendation systems that utilize user emotional information exist, these systems still have room for improvement in terms of accuracy and response speed.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0605] In this invention, the server includes a means for analyzing image information received from a user terminal to evaluate the hair condition, a means for analyzing emotional information received from the user terminal to evaluate the user's emotional state, and a means for using a generative AI to recommend hair care products suitable for the user based on the hair condition evaluation and the emotional state evaluation. This enables more personalized hair care product recommendations that take into account not only the user's hair condition but also their emotional state. This also increases user satisfaction and enables the rapid provision of products that meet the user's needs.

[0606] A "user terminal" is a device that allows a user to input information and acquire and transmit image information and emotional information. This includes smartphones and PCs.

[0607] "Image information" is visual data such as photos or videos showing the condition of the user's hair.

[0608] "Emotional information" is data that indicates the user's psychological state and emotions, obtained based on the user's facial expressions and voice analysis.

[0609] A "server" is a computing device that receives data sent from a user terminal, stores it in a database, and analyzes it.

[0610] A "database" is a structured information storage system for storing information received by a server.

[0611] The "image analysis module" is a software and hardware component for assessing hair condition, which can use machine learning algorithms.

[0612] An "emotion engine" is a system for analyzing emotional information and assessing a user's emotional state, including facial recognition and voice analysis technologies.

[0613] The "generative AI counseling engine" is a system that uses a generative AI model to recommend the most suitable hair care products to users based on the results of hair condition assessment and emotional assessment.

[0614] The "counseling results" are advice and a list of recommended hair care products based on the user's hair condition and emotional state.

[0615] The "recommended product list" is a list of hair care products suitable for the user, generated by the generative AI counseling engine.

[0616] "Natural language processing technology" is a technical field in computer science for understanding and generating text data.

[0617] A "machine learning algorithm" is a computational method for analyzing data, learning patterns, and making predictions and classifications.

[0618] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0619] User terminal operation

[0620] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and also input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information is sent from the user device to the server.

[0621] Server Operation

[0622] The server receives image and emotion information sent from the user's device and first stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and stores the results in the database.

[0623] How the Generative AI Counseling Engine Works

[0624] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0625] Server Operation (cont.)

[0626] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and transmits it to the user's terminal.

[0627] User terminal operation (continued)

[0628] The user terminal displays the counseling results and a list of recommended products, and the user can check the displayed information, select and purchase hair care products that suit their preferences. At this time, the user's emotional state is taken into consideration, and the best options are provided for the user.

[0629] Specific examples

[0630] For example, User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate their emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B can then purchase the recommended products on an e-commerce site.

[0631] Prompt Sentence Examples

[0632] Below are some example prompts for a generative AI model to recommend appropriate hair care products based on the user's hair condition and emotional information:

[0633] The user's hair is dry and faded. They've also been under a lot of stress lately. Recommend them the best hair care products. For example, you could recommend a moisturizing shampoo, a color-preserving treatment, or even a relaxing scented hairspray.

[0634] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0635] Step 1: The user accesses a dedicated app or website

[0636] The user accesses a dedicated app or website using a smartphone or PC. The app or website displays an interface for understanding the condition of the hair and instructs the user to take a photo of the condition of the hair. The input is the user's action (launching the app), and the output is a form that displays instructions for taking a photo.

[0637] Step 2: The user takes a photo of their hair condition and inputs emotional information.

[0638] The user follows the app's instructions to take a photo of the condition of their hair. The app also recognizes facial expressions through the camera and obtains emotional information by analyzing audio through the microphone. The input is the photo and audio data taken by the user, and the output is image information and emotional information. This information is sent from the user's device to the server.

[0639] Step 3: The server receives the image information and emotion information and stores them in the database.

[0640] The server receives image information and emotion information sent from the user terminal. It stores this data in a database. The input is the image information and emotion information received from the user terminal, and the output is the records stored in the database.

[0641] Step 4: The server uses the image analysis module to evaluate the hair condition

[0642] The server uses an image analysis module to analyze the received image information and evaluate the hair condition (dryness, damage, color, etc.). Specifically, it analyzes the image data using, for example, TensorFlow, a machine learning library, and outputs the hair condition as quantified data. The input is the image information stored in the database, and the output is the hair condition evaluation result.

[0643] Step 5: The server evaluates the emotional state using the emotion engine

[0644] The server uses an emotion engine to analyze the received emotion information and evaluate the user's emotional state (stress, relaxation, etc.). Specifically, it analyzes the emotion data using face recognition algorithms and voice analysis technology, and outputs the emotional state as numerically quantified and classified data. The input is the emotion information stored in the database, and the output is the emotional state evaluation result.

[0645] Step 6: The server sends the evaluation results to the generative AI counseling engine

[0646] The server sends the hair condition evaluation results and emotion evaluation results to the generative AI counseling engine. The inputs are the hair condition evaluation results and emotion state evaluation results, and the generative AI receives these data as outputs.

[0647] Step 7: The generative AI counseling engine generates appropriate counseling results and products.

[0648] The generative AI counseling engine generates counseling results and recommended hair care products suitable for the user based on the received evaluation results. This process uses natural language processing technology and machine learning algorithms. Specifically, the evaluation results are analyzed as input data, and the generative AI outputs counseling results and a product list based on the prompt text. The inputs are the hair condition evaluation results and emotional state evaluation results, and the output is counseling results and a list of recommended products.

[0649] Step 8: Send the results from the generative AI counseling engine back to the server

[0650] The generative AI counseling engine sends the generated counseling results and recommended product list to the server. The input is the counseling results and recommended product list generated by the generative AI, and the server receives these results as output.

[0651] Step 9: The server formats the consultation results and product list

[0652] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand. Specifically, it converts them into HTML or JSON format. The input is the counseling results and recommended product list received from the generation AI, and the output is the formatted data.

[0653] Step 10: The server sends the formatted results to the user's device.

[0654] The server sends the formatted counseling results and the product list to the user terminal. The formatted counseling results and the product list are input, and the output is sent to the user terminal.

[0655] Step 11: User reviews results and selects product

[0656] The user terminal displays the received counseling results and the recommended product list. The user checks the displayed information and selects hair care products that suit their preferences. The input is the counseling results and product list sent from the server, and the output is a product selection based on the user's preferences.

[0657] Step 12: The user purchases the product on the e-commerce site.

[0658] The user purchases the recommended product on an e-commerce website. Apps and websites often provide a purchase link for the recommended product, which the user can click to proceed directly with the purchase process. The input is product information selected by the user, and the output is information indicating that the purchase has been completed.

[0659] (Application example 2)

[0660] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0661] Conventional hair care product recommendation systems only evaluate the user's hair condition, making it difficult to provide personalized recommendations that reflect the user's emotional state. As a result, while they can recommend appropriate hair care products based on the user's hair condition, they cannot recommend products that match the user's emotional state at the time, which prevents them from fully improving user satisfaction.

[0662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0663] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for analyzing the user's emotional information, means for recommending hair care products suitable for the user using a generative AI model based on the hair condition evaluation and the emotional information analysis, and means for transmitting the recommended hair care products to the user terminal, thereby making it possible to recommend optimal hair care products that take into account the user's emotional state as well as their hair condition.

[0664] A "user terminal" is a device used by a user, including a smartphone, tablet, personal computer, smart glasses, etc.

[0665] "Image information" is visual data captured or acquired using a user terminal, and includes hair condition, facial expression, and the like.

[0666] "Hair condition" is information indicating attributes such as the user's hair type, dryness, whether or not it is damaged, and color tone.

[0667] "Emotion information" is information indicating the emotional state of the user obtained from the user's facial expression and voice, and includes, for example, joy, anger, sadness, surprise, fear, and the like.

[0668] A "generative AI model" is a technology that uses pre-trained AI algorithms to generate optimal outputs for specific inputs, and utilizes natural language processing technology and machine learning algorithms.

[0669] "Hair care products" are products used to maintain the health and beauty of hair, including shampoos, treatments, hairsprays, etc.

[0670] "Recommend" means presenting the best option based on specific criteria.

[0671] A "server" is a computer system that processes, stores, and distributes data, analyzes information received from user devices, and executes generative AI models.

[0672] A "database" is a system for organizing and storing information in digital form.

[0673] The "counseling result" is information that indicates advice or recommendations for the user, generated based on the analysis results of image information and emotion information.

[0674] The system for implementing this invention analyzes the user's hair condition and emotional information, and uses a generative AI model to recommend optimal hair care products. The specific configuration and operation procedure are shown below.

[0675] composition

[0676] User terminal

[0677] The user terminals are devices such as smartphones, tablets, personal computers, and smart glasses. These terminals are equipped with cameras and microphones to capture image information (hair condition and facial expression) and audio. Through a dedicated application or website, users can take photos of their hair condition and input emotional information.

[0678] server

[0679] The server is a computer system that processes and analyzes image and emotion information received from user devices. The server has the following functions:

[0680] 1. Database: Stores image information and emotion information received from users.

[0681] 2. Image analysis module: Evaluates hair condition and recognizes facial expressions.

[0682] 3. Sentiment Analysis Module: Evaluates the user's emotions using voice analysis.

[0683] 4. Generative AI Counseling Engine: Integrates the user's hair condition and emotional information to generate optimal counseling results and hair care products. Generative AI models used include GPT-3.

[0684] Operating Procedure

[0685] User terminal operation

[0686] Using a dedicated application, users can take photos of their hair condition and simultaneously input facial expressions and voice, and this information is sent from the user's device to the server.

[0687] Server Operation

[0688] The server stores the received image and emotion information in a database. It then uses an image analysis module to break down and evaluate the hair condition into factors such as dryness, damage, and color. It also uses an emotion analysis module to classify the user's emotional state into categories such as joy, anger, and sadness.

[0689] Generative AI Counseling Engine

[0690] The generative AI counseling engine operates based on the results of image analysis and emotion analysis. For example, if a user feels "stressed" and their hair condition is "dry" and "faded," it will generate recommended products such as "a highly moisturizing shampoo" or "a hairspray with a relaxing scent." Prompt text is used for this generation.

[0691] Example prompt sentence:

[0692] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[0693] Sending and viewing recommendations

[0694] The generated counseling results and recommended product list are sent from the server to the user's terminal and visually displayed on the user's terminal, allowing the user to easily purchase hair care products based on the recommendations.

[0695] Specific examples

[0696] Let's say User B is stressed about dry, faded hair. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, facial expression recognition and voice analysis are performed, and this information is sent to the server. The server evaluates the hair condition, and the emotion analysis module recognizes User B's stress level. Based on this information, the generative AI counseling engine recommends a "highly moisturizing shampoo" and a "hairspray with a relaxing scent," and sends the results to the user's device. User B can then purchase the products on the e-commerce site based on the recommendations.

[0697] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, providing more personalized care.

[0698] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0699] Step 1:

[0700] The user launches a dedicated application and takes a photo of the condition of their hair. At the same time, they input their facial expression and voice. This data (image information and voice information) is captured by the user's device. The input data is an image showing the condition of the hair, and facial expression and voice that capture the user's emotional state. This data is sent from the user's device to the server.

[0701] Step 2:

[0702] The server stores the image and audio information received from the user terminal in a database. The input data is the user's image and audio information, and each data is stored in the database. This saves the data for subsequent analysis processing.

[0703] Step 3:

[0704] The server's image analysis module evaluates hair condition based on image information stored in a database. The input data is image information, and the output data is the evaluation results for hair dryness, damage, color, etc. Specifically, the image is analyzed, pixel-based features are extracted, and the hair condition is classified.

[0705] Step 4:

[0706] At the same time, the server's emotion analysis module analyzes the voice information to assess the user's emotional state. The input data is the voice information, and the output data is the emotion assessment results, such as joy, anger, sadness, and stress. The voice analysis calculates emotion vectors of pitch, tone, and vocabulary to classify emotions.

[0707] Step 5:

[0708] The server sends the hair condition assessment results and emotion assessment results to the generative AI counseling engine. The generative AI counseling engine recommends optimal hair care products to the user based on this data. The input data are the hair condition assessment results and emotion assessment results, and the output data is a list of recommended products. For example, based on data such as "the user's emotional state is stressed and their hair is dry," the generative AI uses prompt sentences to select appropriate hair care products.

[0709] Step 6:

[0710] Example prompt sentence:

[0711] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[0712] The server's generative AI counseling engine uses the prompt sentences shown above to request the generative AI model to generate recommended products. This process allows the AI ​​model to use natural language processing technology to create a list of appropriate products.

[0713] Step 7:

[0714] The server sends the generated counseling results and recommended product list to the user's device. The input data are the recommendations made by the generative AI counseling engine, and the output data are the counseling results and product list in a format that is visually displayed to the user.

[0715] Step 8:

[0716] The user's device displays the received counseling results and a list of recommended products. The user can check this display and select hair care products that suit their preferences. This allows the user to purchase the recommended products directly on the e-commerce site. As a result, the user is provided with hair care products that are optimal for their hair condition and emotional state.

[0717] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0718] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0719] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0720] [Third embodiment]

[0721] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0722] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0724] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0725] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0727] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0728] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0729] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0730] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0731] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0732] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0733] This invention is a system that analyzes image information sent from a user's device and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, user's device, and generative AI counseling engine.

[0734] User terminal operation

[0735] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0736] Server Operation

[0737] The server receives image information and needs information sent from the user's device and stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0738] How the Generative AI Counseling Engine Works

[0739] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms and natural language processing technology. The generated counseling results and recommended product list are then sent back to the server.

[0740] Server Operation (cont.)

[0741] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easily understandable to the user, and sends it to the user's device.

[0742] User terminal operation (continued)

[0743] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0744] Specific examples

[0745] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0746] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0747] The processing flow will be explained below.

[0748] Step 1:

[0749] The user launches the app, selects the hair diagnosis option, and then uses the app's camera to take front, back, left, and right images of their hair.

[0750] Step 2:

[0751] Users input their hair concerns and needs (for example, "I'm concerned about dryness" or "My hair is very damaged"), then press the send button to upload the captured image and the input information to the server.

[0752] Step 3:

[0753] The server receives the image information and input information sent from the user terminal, and the received data is stored in a database.

[0754] Step 4:

[0755] The server then passes the stored image information to an analysis module, which uses AI image processing technology to analyze the hair condition, assessing dryness, damage, color, and other factors.

[0756] Step 5:

[0757] The server sends the analysis results to the AI ​​counseling engine, which includes the hair condition evaluation results and information on the user's concerns and needs.

[0758] Step 6:

[0759] The generative AI counseling engine analyzes the data received from the server and generates counseling results and a list of recommended hair care products tailored to the user's hair condition and needs.

[0760] Step 7:

[0761] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine.

[0762] Step 8:

[0763] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and transmits this data to the user's terminal.

[0764] Step 9:

[0765] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0766] Step 10:

[0767] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0768] Example 1

[0769] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0770] It is not easy for users to choose the right hair care products for their hair health and concerns. Conventional methods require a visit to a beauty salon or specialty store, which is time-consuming and labor-intensive. It is also difficult to receive advice at home to find the right products for a user's hair condition. Furthermore, there is a lack of an objective evaluation method for selecting products that suit individual hair conditions and concerns. This makes it difficult for users to care for their hair efficiently and effectively.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0772] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, and means for transmitting the recommended hair care products to the user terminal, thereby enabling the user to easily select and purchase products suitable for their hair condition from the comfort of their own home.

[0773] A "user terminal" is an electronic device that a user uses to take pictures of their hair condition and input information about their hair concerns and needs.

[0774] "Image information" refers to data of photographs or images showing the condition of hair taken by a user terminal.

[0775] "Hair condition" refers to information that indicates the quality and health of hair, such as dryness, damage, and color.

[0776] "Generative AI" refers to an artificial intelligence model that generates appropriate hair care products based on the user's hair condition and concerns.

[0777] "Hair care products" refers to products used for hair care, such as shampoo, treatment, and conditioner.

[0778] "Server" refers to a computer system that receives, stores, analyzes, and transmits information sent from a user terminal to the generative AI counseling engine.

[0779] "Counseling results" refers to the hair care product recommendations generated by the AI ​​based on the user's hair condition and concerns.

[0780] "Database" refers to a data storage system for storing image information and hair information received by the server.

[0781] "Image Analysis Module" refers to software or libraries for assessing hair condition using received image information.

[0782] This invention is a system that analyzes image information sent from a user terminal and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, a user terminal, and a generative AI counseling engine.

[0783] User terminal operation

[0784] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[0785] Server Operation

[0786] The server receives image information and hair needs information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module (such as OpenCV) to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[0787] How the Generative AI Counseling Engine Works

[0788] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms (e.g., TensorFlow and PyTorch) and natural language processing technologies (e.g., GPT-3). The generated counseling results and recommended product list are then sent back to the server.

[0789] Server Operation (cont.)

[0790] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine. This data is converted into a format that is easily understandable by the user (e.g., HTML or JSON format) and sent to the user's device.

[0791] User terminal operation (continued)

[0792] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[0793] Specific examples

[0794] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[0795] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[0796] Prompt Sentence Examples

[0797] You are a user who is concerned about dry and damaged hair. You launch the dedicated app, take photos of your hair from three angles (front, back, left, and right), and input your concerns such as "I'm worried about dryness" or "I'm very damaged." The server analyzes this information, and the AI ​​generator recommends the most suitable hair care products. What products will be recommended?

[0798] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0799] Step 1:

[0800] The user device begins operation when the user accesses a dedicated app or website. The user activates the camera function within the app and takes pictures of the condition of their hair from three directions: front, back, left, and right. They also input text about their hair concerns and needs (e.g., "I'm concerned about dryness" or "My hair is very damaged"). This input information (image and text) becomes the data required for subsequent processing. Before being sent to the server, the image and text data are converted into an appropriate format (e.g., JPEG image, JSON text).

[0801] input:

[0802] Hair images taken by users, hair concerns and needs entered by users

[0803] output:

[0804] Formatted image data and text data sent to the server

[0805] Specific behavior:

[0806] The user launches the app and taps the "Launch Camera" button.

[0807] Take three photos of your hair from the front, back, left and right and save them in the app.

[0808] Enter "I'm concerned about dryness" or "My skin is very damaged" in the text box on the screen and tap the send button.

[0809] Step 2:

[0810] The user device sends the captured image information and the entered text information about the user's worries to the server using an HTTP request. At this time, the format of the request header, etc. is checked to ensure that the image data and text data are sent correctly.

[0811] input:

[0812] Formatted image data and text data

[0813] output:

[0814] HTTP request sent to the server

[0815] Specific behavior:

[0816] The app generates an HTTP request and sends it to the server.

[0817] Notify the user that the request has been sent.

[0818] Step 3:

[0819] The server receives image and text information sent from the user's device and stores it in a database (e.g., MySQL, MongoDB). It then uses an image analysis module (e.g., OpenCV library) to evaluate the hair condition. This evaluation includes algorithms for analyzing features such as dryness, damage, and color.

[0820] input:

[0821] Image data and text data sent from the user's device

[0822] output:

[0823] Data stored in the database, analysis results (hair condition evaluation)

[0824] Specific behavior:

[0825] The server receives the HTTP request and stores the image and text data in a database.

[0826] Image analysis is performed using the OpenCV library to evaluate hair condition (e.g., "dryness" or "damage").

[0827] Step 4:

[0828] The server sends the analysis results and text information about the user's concerns to a generative AI counseling engine. The generative AI counseling engine uses machine learning algorithms (e.g., TensorFlow, PyTorch) and natural language processing techniques (e.g., GPT-3) to recommend the most suitable hair care products for the user. This process generates counseling results based on the evaluation results and the user's concerns.

[0829] input:

[0830] Analysis results (hair condition evaluation), text information (hair concerns and needs)

[0831] output:

[0832] Generated counseling results and recommended product list

[0833] Specific behavior:

[0834] The server sends the analysis results and text information to the generation AI.

[0835] The generative AI analyzes this data and generates a list of recommended products such as "shampoos with high moisturizing effects" and "treatments with damage repair effects."

[0836] Step 5:

[0837] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easy for the user to understand (e.g., HTML or JSON format), and sends it to the user's device.

[0838] input:

[0839] Counseling results and recommended product list returned by the AI

[0840] output:

[0841] Formatted data sent to the user device

[0842] Specific behavior:

[0843] The server receives the recommended product list and converts it into an appropriate format.

[0844] The converted data is sent to the user terminal as an HTTP response.

[0845] Step 6:

[0846] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the appropriate hair care products on the e-commerce site. If necessary, the user can enter their shipping address and payment method to complete the purchase process.

[0847] input:

[0848] Counseling results and recommended product list sent from the server

[0849] output:

[0850] Counseling results, recommended product list, and purchase procedure information displayed to users

[0851] Specific behavior:

[0852] The user terminal receives the response from the server and displays the recommended product list.

[0853] The user selects "Moisturizing Shampoo" and taps the purchase button.

[0854] Enter your shipping address and payment method to complete the purchase.

[0855] (Application example 1)

[0856] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0857] Conventional hair care product recommendation systems are limited to online use and do not support use in physical stores. As a result, it is not possible for customers to evaluate and analyze their hair condition in real time in physical stores and recommend the most suitable hair care products. Another problem is that it is difficult for customers to easily input their hair condition and concerns and receive on-the-spot counseling.

[0858] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0859] In this invention, the server includes means for analyzing image information received from a user terminal and evaluating the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, means for transmitting the recommended hair care products to the user terminal, means for photographing and analyzing the user's hair condition in real time using a smart mirror installed in a physical store, and means for displaying recommended product information to the user through the smart mirror. This makes it possible to evaluate a customer's hair condition even in a physical store and recommend optimal hair care products on the spot using a generation AI.

[0860] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[0861] "Image information" is data of photographs or videos of the user's hair condition.

[0862] The "means for assessing hair condition" is a function that analyzes the received image information to determine hair dryness, damage, color, etc.

[0863] "Generative AI" is artificial intelligence that uses machine learning algorithms and natural language processing technology to recommend optimal hair care products.

[0864] "Hair care products" are products for caring for hair, such as shampoo, treatment, and conditioner.

[0865] A "smart mirror" is an electronic device in the shape of a mirror that is installed in a physical store and can photograph and analyze the condition of a customer's hair in real time and display the results.

[0866] The "means for displaying recommended product information to the user" is a function for visually presenting recommended hair care products to the user.

[0867] A "database" is a digital storage system for storing and managing image information, analysis results, and user information.

[0868] The "counseling results" are hair care advice and a list of recommended products tailored to the user, analyzed and generated by AI.

[0869] The "analysis results" are the results of evaluating the condition of the hair based on image information.

[0870] "Information about concerns and needs" is data entered by the user regarding hair-related problems and desired hair care effects.

[0871] This invention provides a hair care product recommendation system suitable for use in brick-and-mortar stores. Specifically, it consists of a server, a user terminal, a generative AI counseling engine, and a smart mirror installed in the brick-and-mortar store.

[0872] The server analyzes image information received from the user's device to evaluate the condition of the hair. For example, a user stands in front of a smart mirror in a physical store, and the smart mirror uses a camera to capture the condition of the hair in real time. This image information is sent to the server, which then evaluates the hair condition using an image processing library such as OpenCV. The evaluation results are expressed as parameters such as dryness, damage, and color.

[0873] The server then sends the evaluation results to a generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the most suitable hair care products for the user. The generated list of recommended products is then sent back to the server, which then displays the list on the smart mirror, allowing the user to check the appropriate products on the spot.

[0874] Furthermore, image information received from the user's device and information about the user's hair concerns and needs are stored in a database. This stored data can be used in subsequent consultations, enabling more personalized suggestions.

[0875] As a concrete example, when a user visits a brick-and-mortar hair salon, they stand in front of a smart mirror. The smart mirror takes a photo and analyzes the user's hair condition, and based on the results, the AI ​​generator recommends "shampoos with high moisturizing effects" and "treatments with damage repair effects." This list of recommended products is displayed on the smart mirror, allowing the user to check the products on the spot. It is also possible for users to directly input their concerns and needs using the smart mirror.

[0876] Example prompt sentence:

[0877] "Recommend an appropriate hair care product based on image data showing the dryness of hair. The image file name is 'captured_image.jpg'. In your recommendation, please include the specific product name and the reason for your recommendation."

[0878] As described above, the present invention provides a system that can evaluate hair condition in real time even in a physical store and recommend optimal hair care products.

[0879] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0880] Step 1:

[0881] A user stands in front of a smart mirror in a physical store. The camera built into the smart mirror takes a picture of the user's hair condition. The captured image information is sent from the smart mirror to the server.

[0882] Input: Image of the user's hair condition

[0883] Output: Image data sent to the server

[0884] Step 2:

[0885] The server inputs the received image data into the image analysis module, which uses the OpenCV library to evaluate hair conditions such as dryness, damage, and color. The analysis results are extracted as parameters.

[0886] Input: Photographed image data

[0887] Output: Evaluated hair condition parameters (dryness, damage, color, etc.)

[0888] Step 3:

[0889] The server sends the evaluation results to the generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the best hair care products for the user. A list of recommended products is then sent back to the server.

[0890] Input: Hair condition parameters

[0891] Output: A list of recommended hair care products

[0892] Step 4:

[0893] The server sends the recommended product list returned by the generative AI counseling engine to the smart mirror, which then displays the information to the user.

[0894] Input: List of recommended hair care products

[0895] Output: Recommended product information displayed to the user

[0896] Step 5:

[0897] The smart mirror allows users to input information about their hair concerns and needs, which is then sent to a server.

[0898] Input: Information about hair concerns and needs entered by the user

[0899] Output: User concerns and needs sent to the server

[0900] Step 6:

[0901] The server stores the received information on the user's worries and needs in a database, which can be used for future counseling.

[0902] Input: Information about the user's concerns and needs

[0903] Output: Data on worries and needs recorded in a database

[0904] Step 7:

[0905] Furthermore, the server will re-launch the generative AI counseling engine based on the stored data to generate more personalized counseling results, which will also be displayed on the smart mirror.

[0906] Input: Information about user concerns and needs stored in a database

[0907] Output: More personalized counseling results

[0908] Through these steps, users can receive a real-time evaluation of their hair condition even in a physical store, and be recommended the most suitable hair care products.

[0909] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0910] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0911] User terminal operation

[0912] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and simultaneously input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information are sent from the user device to the server.

[0913] Server Operation

[0914] The server receives image and emotion information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and the results are also stored in the database. This evaluation result is sent to the generative AI counseling engine.

[0915] How the Generative AI Counseling Engine Works

[0916] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0917] Server Operation (cont.)

[0918] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and sends it to the user's device. The user's device displays the counseling results and recommended product list, allowing the user to easily select products on the EC site and complete the purchase procedure.

[0919] User terminal operation (continued)

[0920] The user can then review the displayed counseling results and recommended product list, select and purchase hair care products that suit their preferences, and the system will provide the best options for the user, taking into consideration the user's emotional state, such as stress and satisfaction.

[0921] Specific examples

[0922] For example, User B is troubled by dry and faded hair, and has recently been under a lot of stress. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate the user's emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B immediately purchases the recommended products on an e-commerce site.

[0923] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, resulting in more personalized care and increased user satisfaction.

[0924] The processing flow will be explained below.

[0925] Step 1:

[0926] The user launches the app and selects the hair diagnosis option. They then use the app's camera to take front, back, left, and right images of their hair. At this time, the app also captures an image of the user's face using facial recognition.

[0927] Step 2:

[0928] The user inputs emotional information. Emotional information is collected through simple questions within the app or through voice input via a microphone. For example, emotional data is collected by answering questions such as, "What is your stress level these days?"

[0929] Step 3:

[0930] The user presses a send button to send the captured hair image and emotion information to the server. The user terminal uploads the hair image data, emotion information, and face image data to the server.

[0931] Step 4:

[0932] The server receives the image information and emotion information sent from the user terminal, and the received data is stored in a database.

[0933] Step 5:

[0934] The server then passes the stored image information to an image analysis module, which analyzes the hair condition. This analysis uses AI image processing technology to evaluate hair dryness, damage, color, etc. At the same time, facial recognition technology is used to analyze emotional data.

[0935] Step 6:

[0936] The emotion engine uses facial recognition and voice analysis technologies to assess the user's emotions. The results of facial recognition and voice analysis are integrated to assess the user's stress level, satisfaction, and emotional state. These results are also stored in a database.

[0937] Step 7:

[0938] The server sends the hair condition evaluation results and emotion evaluation results to the AI ​​counseling engine. The data sent includes the hair condition evaluation results and the user's emotional state.

[0939] Step 8:

[0940] The generative AI counseling engine analyzes the data received from the server and generates counseling results and recommended hair care products suitable for the user. This generation process uses natural language processing technology and machine learning algorithms.

[0941] Step 9:

[0942] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine. The AI ​​also takes into account the user's emotional state, and adjusts the recommendation accordingly, for example, recommending products with a relaxing effect if the user is feeling stressed.

[0943] Step 10:

[0944] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and sends this data to the user's terminal. The user can then review it and select a product.

[0945] Step 11:

[0946] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[0947] Step 12:

[0948] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[0949] Example 2

[0950] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0951] Conventional hair care product recommendation systems recommend products based solely on the user's hair condition, without taking into account psychological factors such as the user's emotional state or stress level. This makes it difficult to recommend products that perfectly match the user's preferences and needs, resulting in a lack of user satisfaction. Furthermore, although recommendation systems that utilize user emotional information exist, these systems still have room for improvement in terms of accuracy and response speed.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0953] In this invention, the server includes a means for analyzing image information received from a user terminal to evaluate the hair condition, a means for analyzing emotional information received from the user terminal to evaluate the user's emotional state, and a means for using a generative AI to recommend hair care products suitable for the user based on the hair condition evaluation and the emotional state evaluation. This enables more personalized hair care product recommendations that take into account not only the user's hair condition but also their emotional state. This also increases user satisfaction and enables the rapid provision of products that meet the user's needs.

[0954] A "user terminal" is a device that allows a user to input information and acquire and transmit image information and emotional information. This includes smartphones and PCs.

[0955] "Image information" is visual data such as photos or videos showing the condition of the user's hair.

[0956] "Emotional information" is data that indicates the user's psychological state and emotions, obtained based on the user's facial expressions and voice analysis.

[0957] A "server" is a computing device that receives data sent from a user terminal, stores it in a database, and analyzes it.

[0958] A "database" is a structured information storage system for storing information received by a server.

[0959] The "image analysis module" is a software and hardware component for assessing hair condition, which can use machine learning algorithms.

[0960] An "emotion engine" is a system for analyzing emotional information and assessing a user's emotional state, including facial recognition and voice analysis technologies.

[0961] The "generative AI counseling engine" is a system that uses a generative AI model to recommend the most suitable hair care products to users based on the results of hair condition assessment and emotional assessment.

[0962] The "counseling results" are advice and a list of recommended hair care products based on the user's hair condition and emotional state.

[0963] The "recommended product list" is a list of hair care products suitable for the user, generated by the generative AI counseling engine.

[0964] "Natural language processing technology" is a technical field in computer science for understanding and generating text data.

[0965] A "machine learning algorithm" is a computational method for analyzing data, learning patterns, and making predictions and classifications.

[0966] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[0967] User terminal operation

[0968] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and also input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information is sent from the user device to the server.

[0969] Server Operation

[0970] The server receives image and emotion information sent from the user's device and first stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and stores the results in the database.

[0971] How the Generative AI Counseling Engine Works

[0972] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[0973] Server Operation (cont.)

[0974] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and transmits it to the user's terminal.

[0975] User terminal operation (continued)

[0976] The user terminal displays the counseling results and a list of recommended products, and the user can check the displayed information, select and purchase hair care products that suit their preferences. At this time, the user's emotional state is taken into consideration, and the best options are provided for the user.

[0977] Specific examples

[0978] For example, User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate their emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B can then purchase the recommended products on an e-commerce site.

[0979] Prompt Sentence Examples

[0980] Below are some example prompts for a generative AI model to recommend appropriate hair care products based on the user's hair condition and emotional information:

[0981] The user's hair is dry and faded. They've also been under a lot of stress lately. Recommend them the best hair care products. For example, you could recommend a moisturizing shampoo, a color-preserving treatment, or even a relaxing scented hairspray.

[0982] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0983] Step 1: The user accesses a dedicated app or website

[0984] The user accesses a dedicated app or website using a smartphone or PC. The app or website displays an interface for understanding the condition of the hair and instructs the user to take a photo of the condition of the hair. The input is the user's action (launching the app), and the output is a form that displays instructions for taking a photo.

[0985] Step 2: The user takes a photo of their hair condition and inputs emotional information.

[0986] The user follows the app's instructions to take a photo of the condition of their hair. The app also recognizes facial expressions through the camera and obtains emotional information by analyzing audio through the microphone. The input is the photo and audio data taken by the user, and the output is image information and emotional information. This information is sent from the user's device to the server.

[0987] Step 3: The server receives the image information and emotion information and stores them in the database.

[0988] The server receives image information and emotion information sent from the user terminal. It stores this data in a database. The input is the image information and emotion information received from the user terminal, and the output is the records stored in the database.

[0989] Step 4: The server uses the image analysis module to evaluate the hair condition

[0990] The server uses an image analysis module to analyze the received image information and evaluate the hair condition (dryness, damage, color, etc.). Specifically, it analyzes the image data using, for example, TensorFlow, a machine learning library, and outputs the hair condition as quantified data. The input is the image information stored in the database, and the output is the hair condition evaluation result.

[0991] Step 5: The server evaluates the emotional state using the emotion engine

[0992] The server uses an emotion engine to analyze the received emotion information and evaluate the user's emotional state (stress, relaxation, etc.). Specifically, it analyzes the emotion data using face recognition algorithms and voice analysis technology, and outputs the emotional state as numerically quantified and classified data. The input is the emotion information stored in the database, and the output is the emotional state evaluation result.

[0993] Step 6: The server sends the evaluation results to the generative AI counseling engine

[0994] The server sends the hair condition evaluation results and emotion evaluation results to the generative AI counseling engine. The inputs are the hair condition evaluation results and emotion state evaluation results, and the generative AI receives these data as outputs.

[0995] Step 7: The generative AI counseling engine generates appropriate counseling results and products.

[0996] The generative AI counseling engine generates counseling results and recommended hair care products suitable for the user based on the received evaluation results. This process uses natural language processing technology and machine learning algorithms. Specifically, the evaluation results are analyzed as input data, and the generative AI outputs counseling results and a product list based on the prompt text. The inputs are the hair condition evaluation results and emotional state evaluation results, and the output is counseling results and a list of recommended products.

[0997] Step 8: Send the results from the generative AI counseling engine back to the server

[0998] The generative AI counseling engine sends the generated counseling results and recommended product list to the server. The input is the counseling results and recommended product list generated by the generative AI, and the server receives these results as output.

[0999] Step 9: The server formats the consultation results and product list

[1000] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand. Specifically, it converts them into HTML or JSON format. The input is the counseling results and recommended product list received from the generation AI, and the output is the formatted data.

[1001] Step 10: The server sends the formatted results to the user's device.

[1002] The server sends the formatted counseling results and the product list to the user terminal. The formatted counseling results and the product list are input, and the output is sent to the user terminal.

[1003] Step 11: User reviews results and selects product

[1004] The user terminal displays the received counseling results and the recommended product list. The user checks the displayed information and selects hair care products that suit their preferences. The input is the counseling results and product list sent from the server, and the output is a product selection based on the user's preferences.

[1005] Step 12: The user purchases the product on the e-commerce site.

[1006] The user purchases the recommended product on an e-commerce website. Apps and websites often provide a purchase link for the recommended product, which the user can click to proceed directly with the purchase process. The input is product information selected by the user, and the output is information indicating that the purchase has been completed.

[1007] (Application example 2)

[1008] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1009] Conventional hair care product recommendation systems only evaluate the user's hair condition, making it difficult to provide personalized recommendations that reflect the user's emotional state. As a result, while they can recommend appropriate hair care products based on the user's hair condition, they cannot recommend products that match the user's emotional state at the time, which prevents them from fully improving user satisfaction.

[1010] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1011] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for analyzing the user's emotional information, means for recommending hair care products suitable for the user using a generative AI model based on the hair condition evaluation and the emotional information analysis, and means for transmitting the recommended hair care products to the user terminal, thereby making it possible to recommend optimal hair care products that take into account the user's emotional state as well as their hair condition.

[1012] A "user terminal" is a device used by a user, including a smartphone, tablet, personal computer, smart glasses, etc.

[1013] "Image information" is visual data captured or acquired using a user terminal, and includes hair condition, facial expression, and the like.

[1014] "Hair condition" is information indicating attributes such as the user's hair type, dryness, whether or not it is damaged, and color tone.

[1015] "Emotion information" is information indicating the emotional state of the user obtained from the user's facial expression and voice, and includes, for example, joy, anger, sadness, surprise, fear, and the like.

[1016] A "generative AI model" is a technology that uses pre-trained AI algorithms to generate optimal outputs for specific inputs, and utilizes natural language processing technology and machine learning algorithms.

[1017] "Hair care products" are products used to maintain the health and beauty of hair, including shampoos, treatments, hairsprays, etc.

[1018] "Recommend" means presenting the best option based on specific criteria.

[1019] A "server" is a computer system that processes, stores, and distributes data, analyzes information received from user devices, and executes generative AI models.

[1020] A "database" is a system for organizing and storing information in digital form.

[1021] The "counseling result" is information that indicates advice or recommendations for the user, generated based on the analysis results of image information and emotion information.

[1022] The system for implementing this invention analyzes the user's hair condition and emotional information, and uses a generative AI model to recommend optimal hair care products. The specific configuration and operation procedure are shown below.

[1023] composition

[1024] User terminal

[1025] The user terminals are devices such as smartphones, tablets, personal computers, and smart glasses. These terminals are equipped with cameras and microphones to capture image information (hair condition and facial expression) and audio. Through a dedicated application or website, users can take photos of their hair condition and input emotional information.

[1026] server

[1027] The server is a computer system that processes and analyzes image and emotion information received from user devices. The server has the following functions:

[1028] 1. Database: Stores image information and emotion information received from users.

[1029] 2. Image analysis module: Evaluates hair condition and recognizes facial expressions.

[1030] 3. Sentiment Analysis Module: Evaluates the user's emotions using voice analysis.

[1031] 4. Generative AI Counseling Engine: Integrates the user's hair condition and emotional information to generate optimal counseling results and hair care products. Generative AI models used include GPT-3.

[1032] Operating Procedure

[1033] User terminal operation

[1034] Using a dedicated application, users can take photos of their hair condition and simultaneously input facial expressions and voice, and this information is sent from the user's device to the server.

[1035] Server Operation

[1036] The server stores the received image and emotion information in a database. It then uses an image analysis module to break down and evaluate the hair condition into factors such as dryness, damage, and color. It also uses an emotion analysis module to classify the user's emotional state into categories such as joy, anger, and sadness.

[1037] Generative AI Counseling Engine

[1038] The generative AI counseling engine operates based on the results of image analysis and emotion analysis. For example, if a user feels "stressed" and their hair condition is "dry" and "faded," it will generate recommended products such as "a highly moisturizing shampoo" or "a hairspray with a relaxing scent." Prompt text is used for this generation.

[1039] Example prompt sentence:

[1040] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[1041] Sending and viewing recommendations

[1042] The generated counseling results and recommended product list are sent from the server to the user's terminal and visually displayed on the user's terminal, allowing the user to easily purchase hair care products based on the recommendations.

[1043] Specific examples

[1044] Let's say User B is stressed about dry, faded hair. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, facial expression recognition and voice analysis are performed, and this information is sent to the server. The server evaluates the hair condition, and the emotion analysis module recognizes User B's stress level. Based on this information, the generative AI counseling engine recommends a "highly moisturizing shampoo" and a "hairspray with a relaxing scent," and sends the results to the user's device. User B can then purchase the products on the e-commerce site based on the recommendations.

[1045] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, providing more personalized care.

[1046] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1047] Step 1:

[1048] The user launches a dedicated application and takes a photo of the condition of their hair. At the same time, they input their facial expression and voice. This data (image information and voice information) is captured by the user's device. The input data is an image showing the condition of the hair, and facial expression and voice that capture the user's emotional state. This data is sent from the user's device to the server.

[1049] Step 2:

[1050] The server stores the image and audio information received from the user terminal in a database. The input data is the user's image and audio information, and each data is stored in the database. This saves the data for subsequent analysis processing.

[1051] Step 3:

[1052] The server's image analysis module evaluates hair condition based on image information stored in a database. The input data is image information, and the output data is the evaluation results for hair dryness, damage, color, etc. Specifically, the image is analyzed, pixel-based features are extracted, and the hair condition is classified.

[1053] Step 4:

[1054] At the same time, the server's emotion analysis module analyzes the voice information to assess the user's emotional state. The input data is the voice information, and the output data is the emotion assessment results, such as joy, anger, sadness, and stress. The voice analysis calculates emotion vectors of pitch, tone, and vocabulary to classify emotions.

[1055] Step 5:

[1056] The server sends the hair condition assessment results and emotion assessment results to the generative AI counseling engine. The generative AI counseling engine recommends optimal hair care products to the user based on this data. The input data are the hair condition assessment results and emotion assessment results, and the output data is a list of recommended products. For example, based on data such as "the user's emotional state is stressed and their hair is dry," the generative AI uses prompt sentences to select appropriate hair care products.

[1057] Step 6:

[1058] Example prompt sentence:

[1059] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[1060] The server's generative AI counseling engine uses the prompt sentences shown above to request the generative AI model to generate recommended products. This process allows the AI ​​model to use natural language processing technology to create a list of appropriate products.

[1061] Step 7:

[1062] The server sends the generated counseling results and recommended product list to the user's device. The input data are the recommendations made by the generative AI counseling engine, and the output data are the counseling results and product list in a format that is visually displayed to the user.

[1063] Step 8:

[1064] The user's device displays the received counseling results and a list of recommended products. The user can check this display and select hair care products that suit their preferences. This allows the user to purchase the recommended products directly on the e-commerce site. As a result, the user is provided with hair care products that are optimal for their hair condition and emotional state.

[1065] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1066] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1067] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1068] [Fourth embodiment]

[1069] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1070] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1071] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1072] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1073] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1074] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1075] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1076] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1077] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1078] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1079] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1080] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1081] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1082] This invention is a system that analyzes image information sent from a user's device and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, user's device, and generative AI counseling engine.

[1083] User terminal operation

[1084] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[1085] Server Operation

[1086] The server receives image information and needs information sent from the user's device and stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[1087] How the Generative AI Counseling Engine Works

[1088] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms and natural language processing technology. The generated counseling results and recommended product list are then sent back to the server.

[1089] Server Operation (cont.)

[1090] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easily understandable to the user, and sends it to the user's device.

[1091] User terminal operation (continued)

[1092] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[1093] Specific examples

[1094] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[1095] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[1096] The processing flow will be explained below.

[1097] Step 1:

[1098] The user launches the app, selects the hair diagnosis option, and then uses the app's camera to take front, back, left, and right images of their hair.

[1099] Step 2:

[1100] Users input their hair concerns and needs (for example, "I'm concerned about dryness" or "My hair is very damaged"), then press the send button to upload the captured image and the input information to the server.

[1101] Step 3:

[1102] The server receives the image information and input information sent from the user terminal, and the received data is stored in a database.

[1103] Step 4:

[1104] The server then passes the stored image information to an analysis module, which uses AI image processing technology to analyze the hair condition, assessing dryness, damage, color, and other factors.

[1105] Step 5:

[1106] The server sends the analysis results to the AI ​​counseling engine, which includes the hair condition evaluation results and information on the user's concerns and needs.

[1107] Step 6:

[1108] The generative AI counseling engine analyzes the data received from the server and generates counseling results and a list of recommended hair care products tailored to the user's hair condition and needs.

[1109] Step 7:

[1110] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine.

[1111] Step 8:

[1112] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and transmits this data to the user's terminal.

[1113] Step 9:

[1114] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[1115] Step 10:

[1116] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[1117] Example 1

[1118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1119] It is not easy for users to choose the right hair care products for their hair health and concerns. Conventional methods require a visit to a beauty salon or specialty store, which is time-consuming and labor-intensive. It is also difficult to receive advice at home to find the right products for a user's hair condition. Furthermore, there is a lack of an objective evaluation method for selecting products that suit individual hair conditions and concerns. This makes it difficult for users to care for their hair efficiently and effectively.

[1120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1121] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, and means for transmitting the recommended hair care products to the user terminal, thereby enabling the user to easily select and purchase products suitable for their hair condition from the comfort of their own home.

[1122] A "user terminal" is an electronic device that a user uses to take pictures of their hair condition and input information about their hair concerns and needs.

[1123] "Image information" refers to data of photographs or images showing the condition of hair taken by a user terminal.

[1124] "Hair condition" refers to information that indicates the quality and health of hair, such as dryness, damage, and color.

[1125] "Generative AI" refers to an artificial intelligence model that generates appropriate hair care products based on the user's hair condition and concerns.

[1126] "Hair care products" refers to products used for hair care, such as shampoo, treatment, and conditioner.

[1127] "Server" refers to a computer system that receives, stores, analyzes, and transmits information sent from a user terminal to the generative AI counseling engine.

[1128] "Counseling results" refers to the hair care product recommendations generated by the AI ​​based on the user's hair condition and concerns.

[1129] "Database" refers to a data storage system for storing image information and hair information received by the server.

[1130] "Image Analysis Module" refers to software or libraries for assessing hair condition using received image information.

[1131] This invention is a system that analyzes image information sent from a user terminal and uses generative AI to recommend optimal hair care products. This system is operated mainly using a server, a user terminal, and a generative AI counseling engine.

[1132] User terminal operation

[1133] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and enter information about their hair concerns and needs. The captured image information and the entered information are sent from the user device to the server.

[1134] Server Operation

[1135] The server receives image information and hair needs information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module (such as OpenCV) to evaluate the hair condition (dryness, damage, color, etc.). The evaluation results are sent to the generative AI counseling engine. Information about the user's hair concerns and needs is also sent at the same time.

[1136] How the Generative AI Counseling Engine Works

[1137] The generative AI counseling engine generates counseling results and recommended products suitable for the user based on the evaluation results and needs information received from the server. This generation process uses machine learning algorithms (e.g., TensorFlow and PyTorch) and natural language processing technologies (e.g., GPT-3). The generated counseling results and recommended product list are then sent back to the server.

[1138] Server Operation (cont.)

[1139] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine. This data is converted into a format that is easily understandable by the user (e.g., HTML or JSON format) and sent to the user's device.

[1140] User terminal operation (continued)

[1141] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the recommended hair care products on the e-commerce site. Once the user has selected the product to purchase, they can complete the purchase process by entering their shipping address and payment method.

[1142] Specific examples

[1143] For example, User A is concerned about dry and damaged hair. User A launches a dedicated app and takes photos of the condition of their hair from three directions: front, back, left, and right. They then enter their concerns, such as "I'm concerned about dryness" or "It's very damaged," on the input screen. After this information is sent to the server, the server analyzes the hair condition and evaluates it as "dry" or "damaged." This evaluation result and the information about the concerns are sent to the generating AI, which then recommends to User A a "shampoo with high moisturizing effects" and a "treatment with damage repair effects." This information is then sent to User A's device, allowing User A to easily purchase appropriate hair care products on an e-commerce site.

[1144] This system allows users to select and purchase the hair care products that are best suited to them without spending time and effort, and also reduces the frequency of visits to the hair salon, enabling effective hair care at home.

[1145] Prompt Sentence Examples

[1146] You are a user who is concerned about dry and damaged hair. You launch the dedicated app, take photos of your hair from three angles (front, back, left, and right), and input your concerns such as "I'm worried about dryness" or "I'm very damaged." The server analyzes this information, and the AI ​​generator recommends the most suitable hair care products. What products will be recommended?

[1147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1148] Step 1:

[1149] The user device begins operation when the user accesses a dedicated app or website. The user activates the camera function within the app and takes pictures of the condition of their hair from three directions: front, back, left, and right. They also input text about their hair concerns and needs (e.g., "I'm concerned about dryness" or "My hair is very damaged"). This input information (image and text) becomes the data required for subsequent processing. Before being sent to the server, the image and text data are converted into an appropriate format (e.g., JPEG image, JSON text).

[1150] input:

[1151] Hair images taken by users, hair concerns and needs entered by users

[1152] output:

[1153] Formatted image data and text data sent to the server

[1154] Specific behavior:

[1155] The user launches the app and taps the "Launch Camera" button.

[1156] Take three photos of your hair from the front, back, left and right and save them in the app.

[1157] Enter "I'm concerned about dryness" or "My skin is very damaged" in the text box on the screen and tap the send button.

[1158] Step 2:

[1159] The user device sends the captured image information and the entered text information about the user's worries to the server using an HTTP request. At this time, the format of the request header, etc. is checked to ensure that the image data and text data are sent correctly.

[1160] input:

[1161] Formatted image data and text data

[1162] output:

[1163] HTTP request sent to the server

[1164] Specific behavior:

[1165] The app generates an HTTP request and sends it to the server.

[1166] Notify the user that the request has been sent.

[1167] Step 3:

[1168] The server receives image and text information sent from the user's device and stores it in a database (e.g., MySQL, MongoDB). It then uses an image analysis module (e.g., OpenCV library) to evaluate the hair condition. This evaluation includes algorithms for analyzing features such as dryness, damage, and color.

[1169] input:

[1170] Image data and text data sent from the user's device

[1171] output:

[1172] Data stored in the database, analysis results (hair condition evaluation)

[1173] Specific behavior:

[1174] The server receives the HTTP request and stores the image and text data in a database.

[1175] Image analysis is performed using the OpenCV library to evaluate hair condition (e.g., "dryness" or "damage").

[1176] Step 4:

[1177] The server sends the analysis results and text information about the user's concerns to a generative AI counseling engine. The generative AI counseling engine uses machine learning algorithms (e.g., TensorFlow, PyTorch) and natural language processing techniques (e.g., GPT-3) to recommend the most suitable hair care products for the user. This process generates counseling results based on the evaluation results and the user's concerns.

[1178] input:

[1179] Analysis results (hair condition evaluation), text information (hair concerns and needs)

[1180] output:

[1181] Generated counseling results and recommended product list

[1182] Specific behavior:

[1183] The server sends the analysis results and text information to the generation AI.

[1184] The generative AI analyzes this data and generates a list of recommended products such as "shampoos with high moisturizing effects" and "treatments with damage repair effects."

[1185] Step 5:

[1186] The server receives the counseling results and recommended product list returned by the AI ​​counseling engine, converts this data into a format that is easy for the user to understand (e.g., HTML or JSON format), and sends it to the user's device.

[1187] input:

[1188] Counseling results and recommended product list returned by the AI

[1189] output:

[1190] Formatted data sent to the user device

[1191] Specific behavior:

[1192] The server receives the recommended product list and converts it into an appropriate format.

[1193] The converted data is sent to the user terminal as an HTTP response.

[1194] Step 6:

[1195] The user's device receives and displays the counseling results and recommended product list sent from the server. Based on this information, the user can easily purchase the appropriate hair care products on the e-commerce site. If necessary, the user can enter their shipping address and payment method to complete the purchase process.

[1196] input:

[1197] Counseling results and recommended product list sent from the server

[1198] output:

[1199] Counseling results, recommended product list, and purchase procedure information displayed to users

[1200] Specific behavior:

[1201] The user terminal receives the response from the server and displays the recommended product list.

[1202] The user selects "Moisturizing Shampoo" and taps the purchase button.

[1203] Enter your shipping address and payment method to complete the purchase.

[1204] (Application example 1)

[1205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1206] Conventional hair care product recommendation systems are limited to online use and do not support use in physical stores. As a result, it is not possible for customers to evaluate and analyze their hair condition in real time in physical stores and recommend the most suitable hair care products. Another problem is that it is difficult for customers to easily input their hair condition and concerns and receive on-the-spot counseling.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1208] In this invention, the server includes means for analyzing image information received from a user terminal and evaluating the hair condition, means for using a generation AI to recommend hair care products suitable for the user based on the hair condition evaluation, means for transmitting the recommended hair care products to the user terminal, means for photographing and analyzing the user's hair condition in real time using a smart mirror installed in a physical store, and means for displaying recommended product information to the user through the smart mirror. This makes it possible to evaluate a customer's hair condition even in a physical store and recommend optimal hair care products on the spot using a generation AI.

[1209] A "user terminal" is an electronic device used by a user, such as a computer or smartphone.

[1210] "Image information" is data of photographs or videos of the user's hair condition.

[1211] The "means for assessing hair condition" is a function that analyzes the received image information to determine hair dryness, damage, color, etc.

[1212] "Generative AI" is artificial intelligence that uses machine learning algorithms and natural language processing technology to recommend optimal hair care products.

[1213] "Hair care products" are products for caring for hair, such as shampoo, treatment, and conditioner.

[1214] A "smart mirror" is an electronic device in the shape of a mirror that is installed in a physical store and can photograph and analyze the condition of a customer's hair in real time and display the results.

[1215] The "means for displaying recommended product information to the user" is a function for visually presenting recommended hair care products to the user.

[1216] A "database" is a digital storage system for storing and managing image information, analysis results, and user information.

[1217] The "counseling results" are hair care advice and a list of recommended products tailored to the user, analyzed and generated by AI.

[1218] The "analysis results" are the results of evaluating the condition of the hair based on image information.

[1219] "Information about concerns and needs" is data entered by the user regarding hair-related problems and desired hair care effects.

[1220] This invention provides a hair care product recommendation system suitable for use in brick-and-mortar stores. Specifically, it consists of a server, a user terminal, a generative AI counseling engine, and a smart mirror installed in the brick-and-mortar store.

[1221] The server analyzes image information received from the user's device to evaluate the condition of the hair. For example, a user stands in front of a smart mirror in a physical store, and the smart mirror uses a camera to capture the condition of the hair in real time. This image information is sent to the server, which then evaluates the hair condition using an image processing library such as OpenCV. The evaluation results are expressed as parameters such as dryness, damage, and color.

[1222] The server then sends the evaluation results to a generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the most suitable hair care products for the user. The generated list of recommended products is then sent back to the server, which then displays the list on the smart mirror, allowing the user to check the appropriate products on the spot.

[1223] Furthermore, image information received from the user's device and information about the user's hair concerns and needs are stored in a database. This stored data can be used in subsequent consultations, enabling more personalized suggestions.

[1224] As a concrete example, when a user visits a brick-and-mortar hair salon, they stand in front of a smart mirror. The smart mirror takes a photo and analyzes the user's hair condition, and based on the results, the AI ​​generator recommends "shampoos with high moisturizing effects" and "treatments with damage repair effects." This list of recommended products is displayed on the smart mirror, allowing the user to check the products on the spot. It is also possible for users to directly input their concerns and needs using the smart mirror.

[1225] Example prompt sentence:

[1226] "Recommend an appropriate hair care product based on image data showing the dryness of hair. The image file name is 'captured_image.jpg'. In your recommendation, please include the specific product name and the reason for your recommendation."

[1227] As described above, the present invention provides a system that can evaluate hair condition in real time even in a physical store and recommend optimal hair care products.

[1228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1229] Step 1:

[1230] A user stands in front of a smart mirror in a physical store. The camera built into the smart mirror takes a picture of the user's hair condition. The captured image information is sent from the smart mirror to the server.

[1231] Input: Image of the user's hair condition

[1232] Output: Image data sent to the server

[1233] Step 2:

[1234] The server inputs the received image data into the image analysis module, which uses the OpenCV library to evaluate hair conditions such as dryness, damage, and color. The analysis results are extracted as parameters.

[1235] Input: Photographed image data

[1236] Output: Evaluated hair condition parameters (dryness, damage, color, etc.)

[1237] Step 3:

[1238] The server sends the evaluation results to the generative AI counseling engine, which uses machine learning algorithms and natural language processing techniques to recommend the best hair care products for the user. A list of recommended products is then sent back to the server.

[1239] Input: Hair condition parameters

[1240] Output: A list of recommended hair care products

[1241] Step 4:

[1242] The server sends the recommended product list returned by the generative AI counseling engine to the smart mirror, which then displays the information to the user.

[1243] Input: List of recommended hair care products

[1244] Output: Recommended product information displayed to the user

[1245] Step 5:

[1246] The smart mirror allows users to input information about their hair concerns and needs, which is then sent to a server.

[1247] Input: Information about hair concerns and needs entered by the user

[1248] Output: User concerns and needs sent to the server

[1249] Step 6:

[1250] The server stores the received information on the user's worries and needs in a database, which can be used for future counseling.

[1251] Input: Information about the user's concerns and needs

[1252] Output: Data on worries and needs recorded in a database

[1253] Step 7:

[1254] Furthermore, the server will re-launch the generative AI counseling engine based on the stored data to generate more personalized counseling results, which will also be displayed on the smart mirror.

[1255] Input: Information about user concerns and needs stored in a database

[1256] Output: More personalized counseling results

[1257] Through these steps, users can receive a real-time evaluation of their hair condition even in a physical store, and be recommended the most suitable hair care products.

[1258] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1259] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[1260] User terminal operation

[1261] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and simultaneously input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information are sent from the user device to the server.

[1262] Server Operation

[1263] The server receives image and emotion information sent from the user's device and stores this data in a database. Next, the server uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and the results are also stored in the database. This evaluation result is sent to the generative AI counseling engine.

[1264] How the Generative AI Counseling Engine Works

[1265] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[1266] Server Operation (cont.)

[1267] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and sends it to the user's device. The user's device displays the counseling results and recommended product list, allowing the user to easily select products on the EC site and complete the purchase procedure.

[1268] User terminal operation (continued)

[1269] The user can then review the displayed counseling results and recommended product list, select and purchase hair care products that suit their preferences, and the system will provide the best options for the user, taking into consideration the user's emotional state, such as stress and satisfaction.

[1270] Specific examples

[1271] For example, User B is troubled by dry and faded hair, and has recently been under a lot of stress. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate the user's emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B immediately purchases the recommended products on an e-commerce site.

[1272] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, resulting in more personalized care and increased user satisfaction.

[1273] The processing flow will be explained below.

[1274] Step 1:

[1275] The user launches the app and selects the hair diagnosis option. They then use the app's camera to take front, back, left, and right images of their hair. At this time, the app also captures an image of the user's face using facial recognition.

[1276] Step 2:

[1277] The user inputs emotional information. Emotional information is collected through simple questions within the app or through voice input via a microphone. For example, emotional data is collected by answering questions such as, "What is your stress level these days?"

[1278] Step 3:

[1279] The user presses a send button to send the captured hair image and emotion information to the server. The user terminal uploads the hair image data, emotion information, and face image data to the server.

[1280] Step 4:

[1281] The server receives the image information and emotion information sent from the user terminal, and the received data is stored in a database.

[1282] Step 5:

[1283] The server then passes the stored image information to an image analysis module, which analyzes the hair condition. This analysis uses AI image processing technology to evaluate hair dryness, damage, color, etc. At the same time, facial recognition technology is used to analyze emotional data.

[1284] Step 6:

[1285] The emotion engine uses facial recognition and voice analysis technologies to assess the user's emotions. The results of facial recognition and voice analysis are integrated to assess the user's stress level, satisfaction, and emotional state. These results are also stored in a database.

[1286] Step 7:

[1287] The server sends the hair condition evaluation results and emotion evaluation results to the AI ​​counseling engine. The data sent includes the hair condition evaluation results and the user's emotional state.

[1288] Step 8:

[1289] The generative AI counseling engine analyzes the data received from the server and generates counseling results and recommended hair care products suitable for the user. This generation process uses natural language processing technology and machine learning algorithms.

[1290] Step 9:

[1291] The generated counseling results and recommended product list are sent back to the server from the generative AI counseling engine. The AI ​​also takes into account the user's emotional state, and adjusts the recommendation accordingly, for example, recommending products with a relaxing effect if the user is feeling stressed.

[1292] Step 10:

[1293] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand, and sends this data to the user's terminal. The user can then review it and select a product.

[1294] Step 11:

[1295] The user's device receives and displays the counseling results and recommended product list sent from the server. The user confirms the displayed information and selects to purchase the recommended hair care products on the e-commerce site.

[1296] Step 12:

[1297] The user selects the product they wish to purchase, enters their shipping address and payment information, and completes the purchase process. The e-commerce site then arranges for the product to be delivered to the user.

[1298] Example 2

[1299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1300] Conventional hair care product recommendation systems recommend products based solely on the user's hair condition, without taking into account psychological factors such as the user's emotional state or stress level. This makes it difficult to recommend products that perfectly match the user's preferences and needs, resulting in a lack of user satisfaction. Furthermore, although recommendation systems that utilize user emotional information exist, these systems still have room for improvement in terms of accuracy and response speed.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1302] In this invention, the server includes a means for analyzing image information received from a user terminal to evaluate the hair condition, a means for analyzing emotional information received from the user terminal to evaluate the user's emotional state, and a means for using a generative AI to recommend hair care products suitable for the user based on the hair condition evaluation and the emotional state evaluation. This enables more personalized hair care product recommendations that take into account not only the user's hair condition but also their emotional state. This also increases user satisfaction and enables the rapid provision of products that meet the user's needs.

[1303] A "user terminal" is a device that allows a user to input information and acquire and transmit image information and emotional information. This includes smartphones and PCs.

[1304] "Image information" is visual data such as photos or videos showing the condition of the user's hair.

[1305] "Emotional information" is data that indicates the user's psychological state and emotions, obtained based on the user's facial expressions and voice analysis.

[1306] A "server" is a computing device that receives data sent from a user terminal, stores it in a database, and analyzes it.

[1307] A "database" is a structured information storage system for storing information received by a server.

[1308] The "image analysis module" is a software and hardware component for assessing hair condition, which can use machine learning algorithms.

[1309] An "emotion engine" is a system for analyzing emotional information and assessing a user's emotional state, including facial recognition and voice analysis technologies.

[1310] The "generative AI counseling engine" is a system that uses a generative AI model to recommend the most suitable hair care products to users based on the results of hair condition assessment and emotional assessment.

[1311] The "counseling results" are advice and a list of recommended hair care products based on the user's hair condition and emotional state.

[1312] The "recommended product list" is a list of hair care products suitable for the user, generated by the generative AI counseling engine.

[1313] "Natural language processing technology" is a technical field in computer science for understanding and generating text data.

[1314] A "machine learning algorithm" is a computational method for analyzing data, learning patterns, and making predictions and classifications.

[1315] This invention relates to a hair care product recommendation system that combines a user emotion engine. This system analyzes image information and emotion information received from the user's device and uses generative AI to recommend optimal hair care products.

[1316] User terminal operation

[1317] Users access a dedicated app or website using a user device such as a smartphone or PC. They follow the app's instructions to take a photo of their hair condition and also input emotional information using facial recognition and voice analysis. Emotional information includes facial expression recognition via the camera and voice analysis via the microphone. This image information and emotional information is sent from the user device to the server.

[1318] Server Operation

[1319] The server receives image and emotion information sent from the user's device and first stores this data in a database. The server then uses an image analysis module to evaluate the hair condition (dryness, damage, color, etc.). In parallel, the emotion engine evaluates the user's emotions using facial expression recognition and voice analysis, and stores the results in the database.

[1320] How the Generative AI Counseling Engine Works

[1321] The generative AI counseling engine generates counseling results and recommended hair care products suited to the user based on the hair condition assessment results and emotion assessment results received from the server. This generation process uses natural language processing technology and machine learning algorithms. For example, if the user is feeling stressed, it can make recommendations based on emotion, such as recommending a shampoo with a relaxing scent. The generated counseling results and recommended product list are then sent back to the server.

[1322] Server Operation (cont.)

[1323] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand and transmits it to the user's terminal.

[1324] User terminal operation (continued)

[1325] The user terminal displays the counseling results and a list of recommended products, and the user can check the displayed information, select and purchase hair care products that suit their preferences. At this time, the user's emotional state is taken into consideration, and the best options are provided for the user.

[1326] Specific examples

[1327] For example, User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, the app uses a camera to recognize User B's facial expressions and uses voice analysis to evaluate their emotional state. When this information is sent to the server, the server evaluates the hair condition as "dry" and "faded," and the emotion engine determines that User B is feeling stressed. Based on these evaluation results, the generative AI counseling engine recommends to User B a "highly moisturizing shampoo," a "color-keeping treatment," and a "hairspray with a relaxing scent." This information is sent to User B's device, and User B can then purchase the recommended products on an e-commerce site.

[1328] Prompt Sentence Examples

[1329] Below are some example prompts for a generative AI model to recommend appropriate hair care products based on the user's hair condition and emotional information:

[1330] The user's hair is dry and faded. They've also been under a lot of stress lately. Recommend them the best hair care products. For example, you could recommend a moisturizing shampoo, a color-preserving treatment, or even a relaxing scented hairspray.

[1331] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1332] Step 1: The user accesses a dedicated app or website

[1333] The user accesses a dedicated app or website using a smartphone or PC. The app or website displays an interface for understanding the condition of the hair and instructs the user to take a photo of the condition of the hair. The input is the user's action (launching the app), and the output is a form that displays instructions for taking a photo.

[1334] Step 2: The user takes a photo of their hair condition and inputs emotional information.

[1335] The user follows the app's instructions to take a photo of the condition of their hair. The app also recognizes facial expressions through the camera and obtains emotional information by analyzing audio through the microphone. The input is the photo and audio data taken by the user, and the output is image information and emotional information. This information is sent from the user's device to the server.

[1336] Step 3: The server receives the image information and emotion information and stores them in the database.

[1337] The server receives image information and emotion information sent from the user terminal. It stores this data in a database. The input is the image information and emotion information received from the user terminal, and the output is the records stored in the database.

[1338] Step 4: The server uses the image analysis module to evaluate the hair condition

[1339] The server uses an image analysis module to analyze the received image information and evaluate the hair condition (dryness, damage, color, etc.). Specifically, it analyzes the image data using, for example, TensorFlow, a machine learning library, and outputs the hair condition as quantified data. The input is the image information stored in the database, and the output is the hair condition evaluation result.

[1340] Step 5: The server evaluates the emotional state using the emotion engine

[1341] The server uses an emotion engine to analyze the received emotion information and evaluate the user's emotional state (stress, relaxation, etc.). Specifically, it analyzes the emotion data using face recognition algorithms and voice analysis technology, and outputs the emotional state as numerically quantified and classified data. The input is the emotion information stored in the database, and the output is the emotional state evaluation result.

[1342] Step 6: The server sends the evaluation results to the generative AI counseling engine

[1343] The server sends the hair condition evaluation results and emotion evaluation results to the generative AI counseling engine. The inputs are the hair condition evaluation results and emotion state evaluation results, and the generative AI receives these data as outputs.

[1344] Step 7: The generative AI counseling engine generates appropriate counseling results and products.

[1345] The generative AI counseling engine generates counseling results and recommended hair care products suitable for the user based on the received evaluation results. This process uses natural language processing technology and machine learning algorithms. Specifically, the evaluation results are analyzed as input data, and the generative AI outputs counseling results and a product list based on the prompt text. The inputs are the hair condition evaluation results and emotional state evaluation results, and the output is counseling results and a list of recommended products.

[1346] Step 8: Send the results from the generative AI counseling engine back to the server

[1347] The generative AI counseling engine sends the generated counseling results and recommended product list to the server. The input is the counseling results and recommended product list generated by the generative AI, and the server receives these results as output.

[1348] Step 9: The server formats the consultation results and product list

[1349] The server formats the received counseling results and recommended product list into a format that is easy for the user to understand. Specifically, it converts them into HTML or JSON format. The input is the counseling results and recommended product list received from the generation AI, and the output is the formatted data.

[1350] Step 10: The server sends the formatted results to the user's device.

[1351] The server sends the formatted counseling results and the product list to the user terminal. The formatted counseling results and the product list are input, and the output is sent to the user terminal.

[1352] Step 11: User reviews results and selects product

[1353] The user terminal displays the received counseling results and the recommended product list. The user checks the displayed information and selects hair care products that suit their preferences. The input is the counseling results and product list sent from the server, and the output is a product selection based on the user's preferences.

[1354] Step 12: The user purchases the product on the e-commerce site.

[1355] The user purchases the recommended product on an e-commerce website. Apps and websites often provide a purchase link for the recommended product, which the user can click to proceed directly with the purchase process. The input is product information selected by the user, and the output is information indicating that the purchase has been completed.

[1356] (Application example 2)

[1357] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] Conventional hair care product recommendation systems only evaluate the user's hair condition, making it difficult to provide personalized recommendations that reflect the user's emotional state. As a result, while they can recommend appropriate hair care products based on the user's hair condition, they cannot recommend products that match the user's emotional state at the time, which prevents them from fully improving user satisfaction.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1360] In this invention, the server includes means for analyzing image information received from a user terminal to evaluate the hair condition, means for analyzing the user's emotional information, means for recommending hair care products suitable for the user using a generative AI model based on the hair condition evaluation and the emotional information analysis, and means for transmitting the recommended hair care products to the user terminal, thereby making it possible to recommend optimal hair care products that take into account the user's emotional state as well as their hair condition.

[1361] A "user terminal" is a device used by a user, including a smartphone, tablet, personal computer, smart glasses, etc.

[1362] "Image information" is visual data captured or acquired using a user terminal, and includes hair condition, facial expression, and the like.

[1363] "Hair condition" is information indicating attributes such as the user's hair type, dryness, whether or not it is damaged, and color tone.

[1364] "Emotion information" is information indicating the emotional state of the user obtained from the user's facial expression and voice, and includes, for example, joy, anger, sadness, surprise, fear, and the like.

[1365] A "generative AI model" is a technology that uses pre-trained AI algorithms to generate optimal outputs for specific inputs, and utilizes natural language processing technology and machine learning algorithms.

[1366] "Hair care products" are products used to maintain the health and beauty of hair, including shampoos, treatments, hairsprays, etc.

[1367] "Recommend" means presenting the best option based on specific criteria.

[1368] A "server" is a computer system that processes, stores, and distributes data, analyzes information received from user devices, and executes generative AI models.

[1369] A "database" is a system for organizing and storing information in digital form.

[1370] The "counseling result" is information that indicates advice or recommendations for the user, generated based on the analysis results of image information and emotion information.

[1371] The system for implementing this invention analyzes the user's hair condition and emotional information, and uses a generative AI model to recommend optimal hair care products. The specific configuration and operation procedure are shown below.

[1372] composition

[1373] User terminal

[1374] The user terminals are devices such as smartphones, tablets, personal computers, and smart glasses. These terminals are equipped with cameras and microphones to capture image information (hair condition and facial expression) and audio. Through a dedicated application or website, users can take photos of their hair condition and input emotional information.

[1375] server

[1376] The server is a computer system that processes and analyzes image and emotion information received from user devices. The server has the following functions:

[1377] 1. Database: Stores image information and emotion information received from users.

[1378] 2. Image analysis module: Evaluates hair condition and recognizes facial expressions.

[1379] 3. Sentiment Analysis Module: Evaluates the user's emotions using voice analysis.

[1380] 4. Generative AI Counseling Engine: Integrates the user's hair condition and emotional information to generate optimal counseling results and hair care products. Generative AI models used include GPT-3.

[1381] Operating Procedure

[1382] User terminal operation

[1383] Using a dedicated application, users can take photos of their hair condition and simultaneously input facial expressions and voice, and this information is sent from the user's device to the server.

[1384] Server Operation

[1385] The server stores the received image and emotion information in a database. It then uses an image analysis module to break down and evaluate the hair condition into factors such as dryness, damage, and color. It also uses an emotion analysis module to classify the user's emotional state into categories such as joy, anger, and sadness.

[1386] Generative AI Counseling Engine

[1387] The generative AI counseling engine operates based on the results of image analysis and emotion analysis. For example, if a user feels "stressed" and their hair condition is "dry" and "faded," it will generate recommended products such as "a highly moisturizing shampoo" or "a hairspray with a relaxing scent." Prompt text is used for this generation.

[1388] Example prompt sentence:

[1389] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[1390] Sending and viewing recommendations

[1391] The generated counseling results and recommended product list are sent from the server to the user's terminal and visually displayed on the user's terminal, allowing the user to easily purchase hair care products based on the recommendations.

[1392] Specific examples

[1393] Let's say User B is stressed about dry, faded hair. User B launches a dedicated app and takes a photo of the condition of their hair. At the same time, facial expression recognition and voice analysis are performed, and this information is sent to the server. The server evaluates the hair condition, and the emotion analysis module recognizes User B's stress level. Based on this information, the generative AI counseling engine recommends a "highly moisturizing shampoo" and a "hairspray with a relaxing scent," and sends the results to the user's device. User B can then purchase the products on the e-commerce site based on the recommendations.

[1394] This system allows users to easily select and purchase hair care products that best suit their hair condition and emotional state, providing more personalized care.

[1395] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1396] Step 1:

[1397] The user launches a dedicated application and takes a photo of the condition of their hair. At the same time, they input their facial expression and voice. This data (image information and voice information) is captured by the user's device. The input data is an image showing the condition of the hair, and facial expression and voice that capture the user's emotional state. This data is sent from the user's device to the server.

[1398] Step 2:

[1399] The server stores the image and audio information received from the user terminal in a database. The input data is the user's image and audio information, and each data is stored in the database. This saves the data for subsequent analysis processing.

[1400] Step 3:

[1401] The server's image analysis module evaluates hair condition based on image information stored in a database. The input data is image information, and the output data is the evaluation results for hair dryness, damage, color, etc. Specifically, the image is analyzed, pixel-based features are extracted, and the hair condition is classified.

[1402] Step 4:

[1403] At the same time, the server's emotion analysis module analyzes the voice information to assess the user's emotional state. The input data is the voice information, and the output data is the emotion assessment results, such as joy, anger, sadness, and stress. The voice analysis calculates emotion vectors of pitch, tone, and vocabulary to classify emotions.

[1404] Step 5:

[1405] The server sends the hair condition assessment results and emotion assessment results to the generative AI counseling engine. The generative AI counseling engine recommends optimal hair care products to the user based on this data. The input data are the hair condition assessment results and emotion assessment results, and the output data is a list of recommended products. For example, based on data such as "the user's emotional state is stressed and their hair is dry," the generative AI uses prompt sentences to select appropriate hair care products.

[1406] Step 6:

[1407] Example prompt sentence:

[1408] "User's emotional state is stressed. Hair condition shows dryness and color fading. Recommend appropriate haircare products."

[1409] The server's generative AI counseling engine uses the prompt sentences shown above to request the generative AI model to generate recommended products. This process allows the AI ​​model to use natural language processing technology to create a list of appropriate products.

[1410] Step 7:

[1411] The server sends the generated counseling results and recommended product list to the user's device. The input data are the recommendations made by the generative AI counseling engine, and the output data are the counseling results and product list in a format that is visually displayed to the user.

[1412] Step 8:

[1413] The user's device displays the received counseling results and a list of recommended products. The user can check this display and select hair care products that suit their preferences. This allows the user to purchase the recommended products directly on the e-commerce site. As a result, the user is provided with hair care products that are optimal for their hair condition and emotional state.

[1414] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1415] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1416] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1417] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1418] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1419] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1420] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1421] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1422] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1423] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1424] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1425] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1426] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1427] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1428] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1429] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1430] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1431] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1432] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1433] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1434] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1435] The following is further disclosed regarding the above embodiment.

[1436] (Claim 1)

[1437] A means for analyzing image information received from a user terminal and evaluating a hair condition;

[1438] A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation;

[1439] means for transmitting the recommended hair care products to a user terminal;

[1440] A system including:

[1441] (Claim 2)

[1442] means for storing image information received from a user terminal in a database;

[1443] A means for analyzing the condition of hair using the stored image information;

[1444] A means for generating a counseling result by a generation AI based on the analysis result;

[1445] The system of claim 1 further comprising:

[1446] (Claim 3)

[1447] the user terminal includes means for inputting information about the user's hair concerns and needs together with transmitting the image information;

[1448] The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs.

[1449] 10. The system of claim 1.

[1450] "Example 1"

[1451] (Claim 1)

[1452] A means for analyzing image information received from a user terminal and evaluating a hair condition;

[1453] A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation;

[1454] means for transmitting the recommended hair care products to a user terminal;

[1455] a means for displaying the recommended hair care products on a user terminal;

[1456] A means for the user to complete a purchase procedure for the recommended hair care product;

[1457] A system including:

[1458] (Claim 2)

[1459] means for storing image information received from a user terminal in a database;

[1460] A means for analyzing the condition of hair using the stored image information;

[1461] A means for generating a counseling result by a generation AI based on the analysis result;

[1462] means for generating recommended products based on the counseling results;

[1463] The system of claim 1 further comprising:

[1464] (Claim 3)

[1465] the user terminal includes means for inputting information about the user's hair concerns and needs together with transmitting the image information;

[1466] The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs.

[1467] 10. The system of claim 1.

[1468] "Application Example 1"

[1469] (Claim 1)

[1470] A means for analyzing image information received from a user terminal and evaluating a hair condition;

[1471] A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation;

[1472] means for transmitting the recommended hair care products to a user terminal;

[1473] A method to photograph and analyze the user's hair condition in real time using a smart mirror installed in a physical store,

[1474] means for displaying recommended product information to a user through the smart mirror;

[1475] A system including:

[1476] (Claim 2)

[1477] means for storing image information received from a user terminal in a database;

[1478] A means for analyzing the condition of hair using the stored image information;

[1479] A means for generating a counseling result by a generation AI based on the analysis result;

[1480] A means for storing image information captured by the smart mirror in the physical store in a database;

[1481] a means for displaying the analysis results returned by the smart mirror;

[1482] The system of claim 1 further comprising:

[1483] (Claim 3)

[1484] the user terminal includes means for inputting information about the user's hair concerns and needs together with transmitting the image information;

[1485] The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs.

[1486] means for a user to input information about hair concerns and needs through the smart mirror;

[1487] 10. The system of claim 1.

[1488] "Example 2: Combining Emotion Engines"

[1489] (Claim 1)

[1490] A means for analyzing image information received from a user terminal and evaluating a hair condition;

[1491] means for analyzing emotion information received from the user terminal and evaluating the user's emotion state;

[1492] A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation and the emotional state evaluation;

[1493] means for transmitting the recommended hair care products to a user terminal;

[1494] A system including:

[1495] (Claim 2)

[1496] means for storing image information and emotion information received from a user terminal in a database;

[1497] A means for analyzing the condition of hair using the stored image information;

[1498] means for analyzing an emotional state using the stored emotional information;

[1499] A means for generating a counseling result by a generation AI based on the analysis result;

[1500] The system of claim 1 further comprising:

[1501] (Claim 3)

[1502] the user terminal includes means for inputting information about the user's hair concerns and needs together with transmitting the image information and emotion information;

[1503] The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs.

[1504] 10. The system of claim 1.

[1505] "Application example 2 when combining emotion engines"

[1506] (Claim 1)

[1507] A means for analyzing image information received from a user terminal and evaluating a hair condition;

[1508] A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation;

[1509] A means for analyzing user emotion information;

[1510] A means for recommending hair care products suitable for the user using a generative AI model based on the hair condition evaluation and emotional information analysis;

[1511] means for transmitting the recommended hair care products to a user terminal;

[1512] A system including:

[1513] (Claim 2)

[1514] means for storing image information received from a user terminal in a database;

[1515] A means for analyzing the condition of hair using the stored image information;

[1516] means for storing the emotion information in a database;

[1517] A means for generating a counseling result by a generation AI based on the analysis result and emotional information;

[1518] The system of claim 1 further comprising:

[1519] (Claim 3)

[1520] the user terminal includes means for inputting information about the user's hair troubles and needs and emotional information together with transmitting the image information;

[1521] The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs as well as emotional information.

[1522] 10. The system of claim 1. [Explanation of symbols]

[1523] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for analyzing image information received from a user terminal and evaluating a hair condition; A means for recommending a hair care product suitable for the user using a generation AI based on the hair condition evaluation; means for transmitting the recommended hair care products to a user terminal; A system including:

2. means for storing image information received from a user terminal in a database; A means for analyzing the condition of hair using the stored image information; A means for generating a counseling result by a generation AI based on the analysis result; The system of claim 1 further comprising:

3. the user terminal includes means for inputting information regarding the user's hair concerns and needs together with transmitting the image information; The AI ​​counseling engine generates counseling results taking into consideration information about the hair worries and needs. The system of claim 1 .

Images