System

Abstract

To provide a system for generating a plurality of candidates by using AI on the basis of a hairstyle and a hair color requested from a customer, and for generating a 3D model by combining the candidates with face photograph.SOLUTION: The method includes the steps of receiving face photograph information provided by a customer, detecting a face portion from the face photograph based on the face photograph information, deleting a background, and extracting an outline of hair, generating a plurality of hairstyle and haircolor candidates based on the hairstyle and haircolor candidates, synthesizing the hairstyle and haircolor candidates with the face photograph information to generate a 3D model, transmitting the 3D model generated by the synthesizing step to customer terminals and beauty operator terminals, and displaying the 3D model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Choosing a hairstyle or hair color carries a significant risk for the customer because the results are irreversible. Traditionally, abstract images were shared mainly through magazines and catalogs, making it difficult for hairdressers and customers to share a concrete image of the finished product. This has led to problems such as customers being dissatisfied after the treatment or hairdressers not being able to accurately understand the customer's wishes. There is a need to solve this problem and ensure that a concrete image of the finished product is shared between customers and hairdressers. [Means for solving the problem]

[0005] This invention provides an image processing method that receives a customer's facial photo data, detects the face, removes the background, and extracts the hair outline. Furthermore, this system uses AI to generate multiple candidates based on the customer's desired hairstyle and hair color, and then combines these candidates with the facial photo data to generate a 3D model. The generated 3D model is sent to the customer's device and the hairdresser's device and displayed in real time. The system also provides a mechanism whereby the hairdresser and customer can review the 3D model and input corrections as necessary, generating and displaying a new 3D model. This allows the customer and hairdresser to share a specific image of the finished product in real time, enabling them to fine-tune their requests before the treatment.

[0006] "Customer" refers to an end user of this system who wishes to select a hairstyle or hair color.

[0007] "Facial photo data" is digital data containing an image of a customer's face, and is the basic data that the customer enters into the system.

[0008] "Image processing means" refers to a combination of hardware and software for performing specific functions (e.g., facial detection, background removal, hair outline extraction) on facial photograph data.

[0009] "Instruction data" refers to information entered in natural language by a customer detailing their desired hairstyle, hair color, and other details.

[0010] "Candidate generation means" refers to a process and device for using AI to generate multiple hairstyle and hair color candidates based on instruction data.

[0011] "Synthesis means" means a process and apparatus for integrating the generated hairstyle and hair color candidates with facial photograph data to create a 3D model.

[0012] "Communication means" refers to a system including the network infrastructure and protocols for transmitting the generated 3D model and other data to the customer terminal and the hairdresser terminal.

[0013] "Display Means" means a device or software for visually presenting 3D models or other visual data to a Customer or Hairdresser on a Terminal.

[0014] "Operation means" refers to the user interface and input device used to rotate the 3D model, zoom in and out, etc. on the customer's terminal, and to check the model in detail.

[0015] "Revision Request" means data representing specific changes that a stylist or customer may enter if they wish to make further adjustments to the 3D model. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating 3D models that are combined with customer photographs. This invention enables hairdressers and customers to accurately share specific images, improving customer satisfaction with the results of treatments.

[0038] System configuration

[0039] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a device such as a smartphone or tablet. The hairdresser terminal is also a digital device, and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0040] Program processing

[0041] Customer photo upload

[0042] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[0043] Image preprocessing on the server

[0044] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[0045] Inputting customer requests

[0046] The user (customer) inputs their desired hairstyle and hair color using natural language within the app, and the device sends this instruction data to the server.

[0047] Generate hairstyle and hair color candidates

[0048] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options, including analyzing the user's request using natural language processing (NLP) to generate 3D models of the options.

[0049] 3D model composition and transmission

[0050] The server combines the generated hairstyle and hair color candidates with the customer's facial photo data to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[0051] Display and check the 3D model

[0052] The device then displays the received 3D model for the user and hairdresser to review, allowing customers to see in detail which hairstyle and hair color will suit their face before the treatment.

[0053] Entering and regenerating correction requests

[0054] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I want the hair color to be a little lighter" can be included. The device sends these correction requests to the server, which then generates and updates the 3D model again.

[0055] Specific examples

[0056] Customer A Scenario

[0057] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0058] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0059] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0060] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0061] 5. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0062] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[0063] 7. The server receives the correction request and sends the regenerated 3D model to the device.

[0064] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[0065] This system allows customer A and the hairdresser to have a smooth consultation based on a specific and shared image.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and launches the camera to take a photo or selects an existing photo from the gallery.

[0069] Step 2:

[0070] The device temporarily stores the selected or taken photo, optimizes the image format and resolution, and sends the image data to the server.

[0071] Step 3:

[0072] The server receives the uploaded photo data and associates it with the designated customer account.

[0073] Step 4:

[0074] The server uses AI image processing models to detect faces in the received photos, remove backgrounds, and extract hair contours, providing the basis for changing hairstyles and hair colors.

[0075] Step 5:

[0076] The user inputs the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[0077] Step 6:

[0078] The server analyzes the user's request using natural language processing (NLP) and passes it as input to an AI generation model, which then generates candidate hairstyles and hair colors.

[0079] Step 7:

[0080] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0081] Step 8:

[0082] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0083] Step 9:

[0084] At the salon, the hairdresser and the user check the 3D model and input specific requests for corrections, such as making the hair color a little lighter. The requests are then sent from the device to the server.

[0085] Step 10:

[0086] The server receives the new request, updates the 3D model using the AI ​​model again, and sends the revised 3D model back to the client device.

[0087] Step 11:

[0088] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Traditionally, consultations about hairstyles and hair colors at beauty salons were conducted by the customer explaining the details verbally, and the hairdresser then performed the treatment based on that. This method could lead to dissatisfaction with the finished product, as the customer's intended hairstyle or hair color was not accurately conveyed. Furthermore, there was a lack of a way to share a specific image before the treatment, making it difficult to improve customer satisfaction.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for receiving facial photograph data provided by a customer, image processing means for detecting facial features from the facial photograph based on the facial photograph data, removing the background, and extracting the hair outline, means for receiving instruction data specifying the hairstyle and hair color desired by the customer, means for generating a plurality of hairstyle and hair color candidates, means for synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photograph data to generate a three-dimensional model, means for transmitting the three-dimensional model generated by the synthesis means to a customer terminal and a hairdresser terminal, means for the customer or hairdresser to input correction requests, and means for regenerating a corrected three-dimensional model based on the correction requests. This enables smooth counseling between the customer and the hairdresser based on a specific and shared image, and allows treatment that accurately reflects the customer's requests.

[0094] "Facial photo data provided by the customer" refers to image data of the customer's own face that has been photographed or selected by the customer using a device such as a smartphone or tablet.

[0095] "Image processing means for detecting facial areas, removing background, and extracting hair contours" is a function that uses a facial recognition algorithm within the server to identify facial areas, remove background, and clarify the outline of hair.

[0096] "Instruction data specifying the hairstyle and hair color desired by the customer" is information in which the customer selects the desired hairstyle and hair color through an input interface using natural language or from pre-prepared options.

[0097] The "means for generating multiple hairstyle and hair color candidates" is a function that uses a generative AI model in the server to create various hairstyle and hair color variations based on customer instruction data.

[0098] The "means for generating a three-dimensional model by combining the facial photograph data" is a function that combines the hairstyle and hair color data generated by the server with the customer's facial photograph data to create a three-dimensional hairstyle model that fits the customer's face.

[0099] The "communication means for transmitting to the customer terminal and the hairdresser terminal" is a function for transmitting the three-dimensional model generated by the server to the terminals of the customer and the hairdresser via data communication.

[0100] The "means for a customer or a hairdresser to input correction requests" is an interface that allows a customer or a hairdresser to input desired changes or corrections to the three-dimensional model.

[0101] The "means for regenerating a modified three-dimensional model" is a function in which the server receives a request for modification from a customer or a hairdresser and updates and regenerates the three-dimensional model.

[0102] "Display means" refers to a display or screen on which the customer and hairdresser can view the three-dimensional model.

[0103] The "operation means" is an interface for rotating and zooming in and out of the three-dimensional model.

[0104] This invention is a digital hair catalog system developed to improve customer satisfaction at beauty salons, and is a technology that generates a 3D model that is combined with a customer's facial photograph. This allows the hairdresser and customer to accurately share a specific image, improving customer satisfaction with the results of the treatment.

[0105] System configuration

[0106] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a digital device such as a smartphone or tablet. The hairdresser terminal is also a digital device and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0107] 1. Customer terminal

[0108] Customer terminals can be any commercially available digital device such as a smartphone or tablet.

[0109] The application has the ability to take photos and select photos from the gallery.

[0110] 2. Server

[0111] The server receives the photo data, processes the images, and generates a three-dimensional model.

[0112] The server can use common cloud services and uses AI image processing algorithms (e.g., OpenCV and TensorFlow) internally.

[0113] 3. Hairdresser's terminal

[0114] The hairdresser terminal uses the same digital device as the customer terminal.

[0115] A dedicated application is installed that allows hairdressers to view the 3D model and communicate with customers.

[0116] Process Details

[0117] 1. Upload customer photos

[0118] Users launch the app using their smartphone or tablet and take a photo of their face or select one from their gallery.

[0119] The device temporarily stores the photo data and sends it to the server.

[0120] 2. Image preprocessing on the server

[0121] The server receives the transmitted photo data and uses its internal AI image processing algorithm to detect the face area.

[0122] The background is removed and the hair outline is extracted, generating reference data for changing the hairstyle and hair color.

[0123] 3. Input customer requests

[0124] Users input their desired hairstyle and hair color using natural language within the app.

[0125] The terminal transmits this instruction data to the server.

[0126] 4. Generating hairstyle and hair color candidates

[0127] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options.

[0128] The server uses natural language processing (NLP) to analyze the user's request and generate candidate 3D models.

[0129] 5. Synthesis and transmission of 3D model

[0130] The server combines the generated hairstyle and hair color candidates with the customer's facial photograph data to generate a three-dimensional model.

[0131] The server transmits this three-dimensional model to the customer terminal and the hairdresser terminal.

[0132] 6. Display and check the 3D model

[0133] The terminal displays the received three-dimensional model, and the user and hairdresser confirm it.

[0134] This allows customers to check specifically which hairstyle and hair color suits their face before the treatment.

[0135] 7. Enter and regenerate correction requests

[0136] The user or hairdresser checks the three-dimensional model and inputs correction requests as necessary.

[0137] For example, this includes specific instructions such as "I want my hair color to be lighter."

[0138] The terminal sends this correction request to the server, which then generates and updates the 3D model again.

[0139] Specific examples

[0140] Customer A Scenario

[0141] 1. The user (Customer A) opens the app and uploads a photo of their face. For example, they take a photo of their face using the camera on their smartphone.

[0142] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0143] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0144] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0145] 5. The generated 3D model is sent to the customer terminal and the hairdresser terminal.

[0146] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[0147] 7. The server receives the modification request and sends the regenerated 3D model to the terminal.

[0148] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[0149] Prompt Sentence Examples

[0150] "Upload a photo of your face and enter your desired hairstyle and hair color. Example: 'Short bob', 'Reddish'"

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] Customer photo upload

[0154] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The user selects a photo within the app, and the app temporarily saves the photo data. The user's device then sends the photo data to the server. The input data is the photo of their face, and the output is the image data received by the server. Specifically, the user takes a photo of their face using the camera function on their smartphone and presses the "send" button on the app.

[0155] Step 2:

[0156] Image preprocessing on the server

[0157] The server receives the facial photo data sent by the user and temporarily stores it. Next, it automatically detects the face using an AI image processing algorithm (e.g., OpenCV or TensorFlow) inside the server. It also performs processing to remove the background and extract the outline of the hair. The input data is the facial photo data, and the output is image data with the background removed and the outline of the hair extracted. Specifically, the AI ​​algorithm identifies the facial area and performs processing to make the background in that area transparent.

[0158] Step 3:

[0159] Inputting customer requests

[0160] Within the app, the user uses natural language to input their desired hairstyle and hair color into a text box. The user enters, for example, "short bob" or "reddish," and the device sends this text data to the server. The input data is the user's natural language request, and the output is the instruction data received by the server. Specifically, the user enters the desired hairstyle and hair color into the app's input fields and presses the "send" button.

[0161] Step 4:

[0162] Generate hairstyle and hair color candidates

[0163] The server receives the request data sent by the user and analyzes it using an AI model (e.g., a natural language processing engine). The server generates multiple hairstyle and hair color candidates based on the analysis results. The input data is the request data, and the output is the generated hairstyle and hair color candidate data. Specifically, the AI ​​model analyzes requests such as "short bob" or "reddish" and creates appropriate hairstyle and hair color variations based on that.

[0164] Step 5:

[0165] 3D model synthesis and transmission

[0166] The server combines the generated hairstyle and hair color candidates with facial photo data to generate a 3D model. The server then sends the combined 3D model to the customer's device and the hairdresser's device. The input data is hairstyle and hair color candidate data and facial photo data, and the output is a 3D model. Specifically, the server uses 3D modeling software such as Blender to combine the facial and hairstyle data to generate a 3D model, which is then sent to the device.

[0167] Step 6:

[0168] Display and check the 3D model

[0169] The device receives and displays the 3D model data sent from the server. By checking the 3D model, the user and hairdresser can get a concrete idea of ​​the hairstyle and hair color that will suit the customer's face before the treatment. The input data is the 3D model data, and the output is the displayed 3D model. Specifically, the smartphone or tablet app displays the 3D model using a view function that allows the user to rotate, zoom in, and out.

[0170] Step 7:

[0171] Entering and regenerating correction requests

[0172] The user or hairdresser checks the 3D model and, if necessary, inputs any correction requests into the app. For example, specific instructions such as "I want my hair color lighter" can be included. The device sends this correction request data to the server, which then generates and updates the 3D model again. The input data is the correction request data, and the output is the corrected 3D model data. Specifically, the user or hairdresser enters the correction request into the app's input field and presses the "Send" button.

[0173] The above are the specific processing steps of the system that generates a three-dimensional hairstyle model using a customer's facial photograph.

[0174] (Application example 1)

[0175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] Conventional methods for proposing hairstyles at beauty salons make it difficult for customers and hairdressers to accurately share specific images, which can lead to a decrease in customer satisfaction after the treatment. There was also a need for a method that allows customers to try on new hairstyles and hair colors without leaving their homes. Furthermore, there were issues with usability and accuracy regarding systems for trying on hairstyles in virtual environments.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0178] In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; synthesis means for generating a 3D model by synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photo data; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and a therapist terminal; display means for displaying the 3D model; a generative AI model that analyzes natural language to understand the customer's request and generates and modifies the 3D model based on the request; and prompt generation means for inputting instruction data to be analyzed into the generative AI model. This allows customers to try new hairstyles and hair colors without leaving their homes and share specific images before the treatment, thereby improving customer satisfaction and improving the usability and accuracy of hairstyle try-on in a virtual environment.

[0179] "Customer terminal" refers to a digital device operated by a customer, including a smartphone, tablet, laptop, etc.

[0180] "Receiving means" refers to a means for receiving data, and in this case refers to a mechanism for receiving images, instruction data, etc. via Internet communication.

[0181] The "image processing means" refers to a means for performing processing to detect specific parts from received image data or remove backgrounds, and specifically includes algorithms for face detection and hair contour extraction.

[0182] "Instruction data" refers to data indicating information such as the hairstyle and hair color desired by the customer, and is input through the application.

[0183] The "candidate generation means" is a means for generating multiple hairstyle and hair color candidates based on instruction data, and is realized here by a generative AI model.

[0184] The "combining means" is a means for combining the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[0185] "Communication means" refers to the means for transmitting the generated 3D model and other data to the client terminal and the therapist terminal, primarily via the Internet.

[0186] The "display means" is a means for displaying the received 3D model on the terminal, and includes a graphical user interface (GUI).

[0187] A "professor terminal" is a digital device operated by the practitioner (hairdresser), and refers mainly to a computer or tablet used within the salon.

[0188] "Generative AI model" refers to an artificial intelligence (AI) model that analyzes natural language and generates and modifies 3D models based on customer requests.

[0189] The "prompt sentence generation means" is a means for generating sentences for inputting instruction data to be analyzed into the generative AI model.

[0190] The present invention relates to a digital hair catalog system for improving customer satisfaction in a virtual hair salon system. This system is comprised of a customer terminal, a therapist terminal, and a server, and transmits and receives data via the Internet.

[0191] Role of customer terminals

[0192] The customer terminal is a digital device such as a smartphone or tablet. The customer first launches the application and takes a photo of their face or selects one from the gallery. The photo is temporarily saved on the customer terminal and then sent to a server via the Internet.

[0193] Processing on the server

[0194] The server processes the data in the following steps:

[0195] 1. The server receives the facial photo data sent from the customer's device. The software used is Python's PIL (Python Imaging Library) and Requests library.

[0196] 2. Based on the received facial photo data, the system detects the face, removes the background, and extracts the hair outline. The OpenCV library is used here.

[0197] 3. Next, the customer receives instruction data specifying their desired hairstyle and hair color. This instruction data is entered by the customer in natural language within the app, and the server analyzes it using Hugging Face's Transformers library and T5 model.

[0198] 4. Generate multiple hairstyle and color options based on natural language analysis. This process uses a generative AI model to accurately reflect the customer's preferences in the 3D model.

[0199] 5. The generated hairstyle and hair color candidates are combined with facial photo data to generate a 3D model. The 3D model is created using the Open3D library.

[0200] 6. Finally, the generated 3D model can be sent to the client's device and the practitioner's device and displayed.

[0201] Role of the therapist terminal

[0202] The hairdresser's terminal is a digital device operated by the hairdresser. The hairdresser checks the 3D model sent from the server and selects the most suitable hairstyle and hair color together with the customer. If necessary, they can resubmit any correction requests to the server and generate and modify the 3D model again. This allows the customer and hairdresser to share a specific image.

[0203] Examples and prompts

[0204] For example, if a customer requests a short bob with red hair, the text is parsed with the prompt, "Please provide generated hairstyle and hair color options: Short bob with red hair." This prompt allows the generative AI model to generate multiple options based on the customer's request and synthesize them into a 3D model.

[0205] Example prompt sentence:

[0206] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[0207] This system allows customers to try on new hairstyles and hair colors without leaving their homes, and by sharing a concrete image of the desired look before the treatment, customer satisfaction is increased. In addition, the high-precision processing of natural language analysis and 3D model generation on the server improves the usability and accuracy of trying on hairstyles in a virtual environment.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The selected photo is temporarily saved on the customer's device and then sent to the server via the Internet. In this step, the customer's photo is used as input data, and the photo data sent to the server is obtained as output data.

[0211] Step 2:

[0212] The server internally processes the received facial photo data. First, it uses the OpenCV library to detect the face, remove the background, and extract the hair outline. At this stage, the facial photo data is used as input data, and image data with the background removed and the hair outline extracted is obtained as output data.

[0213] Step 3:

[0214] The user inputs the desired hairstyle and hair color in the app using natural language. This instruction data is sent from the customer terminal to the server. In this step, instruction text in natural language is used as input data, and the instruction data sent to the server is obtained as output data.

[0215] Step 4:

[0216] The server analyzes the received instruction data. Using Hugging Face's Transformers library and the T5 model, it generates hairstyle and hair color candidates based on natural language. The instruction data is used as input data, and multiple hairstyle and hair color candidates are generated as output data.

[0217] Step 5:

[0218] The generative AI model generates a 3D model by combining the generated hairstyle and hair color candidates with facial photo data. The 3D model is created using the Open3D library. In this step, hairstyle and hair color candidates and facial photo data with the background removed are used as input data, and a 3D model is generated as output data.

[0219] Step 6:

[0220] The server sends the generated 3D model to the client's device and the therapist's device. The client's device and the therapist's device display the 3D model, allowing the client and therapist to share and review it. In this step, the generated 3D model is used as input data, and the transmitted 3D model is obtained as output data.

[0221] Step 7:

[0222] The client and the therapist check the 3D model on the therapist's terminal and input any correction requests as necessary. The correction requests are sent from the therapist's terminal to the server. At this stage, instruction data for corrections is used as input data, and the correction requests are sent to the server as output data.

[0223] Step 8:

[0224] The server generates and modifies the 3D model again based on the modification request, and sends the results back to the client's device and the therapist's device. The modification request is used as input data, and the modified 3D model is generated as output data.

[0225] Example prompt sentence:

[0226] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating a 3D model that is combined with a customer's facial photograph, and technology for recognizing the user's emotions and adjusting suggested hairstyles and hair colors. This allows the hairdresser and customer to accurately share a specific image, further improving customer satisfaction with the results of the treatment.

[0229] System configuration

[0230] This system consists of a customer terminal, a server, a hairdresser terminal, and an emotion engine. The customer terminal is a digital device such as a smartphone or tablet, and the hairdresser terminal is also a digital device. The server is a central system that processes data and communicates between these terminals. The emotion engine is a combination of software and hardware for recognizing customer emotions.

[0231] Program processing

[0232] Customer photo upload

[0233] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[0234] Image preprocessing on the server

[0235] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[0236] Emotion analysis with emotion engine

[0237] The user's emotion engine analyzes the uploaded facial photo data and input data from the customer's device in real time to monitor the user's emotional state. For example, if the customer is smiling, it is analyzed as having a positive emotion.

[0238] Input hairstyle and hair color requests

[0239] The user inputs their desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[0240] Generate hairstyle and hair color candidates

[0241] The server analyzes the user's request (instruction data) using natural language processing (NLP) and passes it as input to the AI ​​generation model. The AI ​​generates candidate hairstyles and hair colors. Furthermore, based on the analysis results of the emotion engine, the candidate hairstyles and hair colors are adjusted to suit the user's emotional state.

[0242] 3D model composition and transmission

[0243] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0244] Display and check the 3D model

[0245] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0246] Entering and regenerating correction requests

[0247] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I'd like the hair color to be a little lighter" are included. The device sends these correction requests to the server. The server then generates and updates the 3D model again, and sends the newly generated 3D model to the device.

[0248] Specific examples

[0249] Customer A Scenario

[0250] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0251] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0252] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[0253] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0254] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[0255] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0256] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[0257] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[0258] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[0259] This system enables more personalized hairstyle suggestions that take into account the user's emotional state, enabling smooth counseling based on a specific and shared image between customer A and the hairdresser.

[0260] The processing flow will be explained below.

[0261] Step 1:

[0262] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and takes a photo of themselves or selects an existing photo from the gallery.

[0263] Step 2:

[0264] The device temporarily stores the selected or taken photo data, optimizes the image format and resolution, and sends it to the server.

[0265] Step 3:

[0266] The server receives the uploaded photo data and associates it with the designated customer account. The server then begins processing the image data and uses AI image processing models to detect facial features.

[0267] Step 4:

[0268] After detecting the face, the server performs image processing to remove the background and extract the hair outline, generating the basic data for changing the hairstyle and hair color.

[0269] Step 5:

[0270] The device receives a notification from the server that face detection and image processing are complete, and prompts the user to perform the next step: The user enters the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app.

[0271] Step 6:

[0272] The device sends instruction data to the server, which analyzes the user's request using natural language processing (NLP) and inputs the instruction data into an AI-generated model.

[0273] Step 7:

[0274] The server uses an AI generative model to generate candidates for a specific hairstyle and hair color. Multiple hairstyle and hair color candidates are generated.

[0275] Step 8:

[0276] The server activates the emotion engine and analyzes the uploaded facial photo and instruction data to recognize the user's emotional state, and adjusts the hairstyle and hair color suggestions based on the recognized emotional state.

[0277] Step 9:

[0278] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0279] Step 10:

[0280] The device displays the received 3D model, allowing the user to rotate it and zoom in and out to check the details. The hairdresser also checks the same model at the salon.

[0281] Step 11:

[0282] The user and hairdresser check the 3D model and input requests for corrections to specific parts (for example, "I want the hair color to be a little lighter"). The device then sends these requests to the server.

[0283] Step 12:

[0284] The server receives the new request and updates the 3D model using the AI ​​model again. The revised 3D model is then sent back to the customer's device and the hairdresser's device.

[0285] Step 13:

[0286] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[0287] Specific examples

[0288] Customer A Scenario

[0289] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0290] 2. The device sends the photo data to the server.

[0291] 3. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0292] 4. The server notifies the terminal that image preprocessing is complete.

[0293] 5. The user (customer A) inputs that he / she desires a "short bob" and "reddish" hairstyle and hair color.

[0294] 6. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0295] 7. The server uses the emotion engine to analyze the user's emotional state and adjusts the candidates suitable for a relaxed state.

[0296] 8. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0297] 9. The user (Customer A) and the hairdresser check the 3D model at the beauty salon.

[0298] 10. The user inputs a request for correction, such as "Make the hair color a little lighter."

[0299] 11. The server receives the correction request and sends the regenerated 3D model to the device.

[0300] 12. The user makes a final confirmation and, once they agree, the treatment begins.

[0301] This system enables personalized hairstyle suggestions that take into account the user's emotional state, realizing smooth counseling based on a specific and shared image between the customer and hairdresser.

[0302] Example 2

[0303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional hair catalog systems used in hair salons have made it difficult to improve customer satisfaction. In particular, the image of hairstyle shared between the customer and the hairdresser is vague, resulting in customers' expectations often not being met. In addition, it is difficult to reflect the customer's feelings and requests in real time, making it impossible to make optimal suggestions for each individual customer. A system that solves these issues was needed.

[0305] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; emotion analysis means for analyzing the customer's emotional data using a generative AI model and making adjustments based on the customer's emotions; synthesis means for synthesizing the hairstyle and hair color candidates adjusted by the candidate generation means and emotion analysis means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to the customer terminal and the hairdresser terminal; and display means for displaying the 3D model. This allows the customer and the hairdresser to have a specific and detailed shared image of the hairstyle and hair color during a smooth consultation. Furthermore, emotion analysis allows for optimal suggestions to be made to the customer, which is expected to improve customer satisfaction.

[0306] "Means for receiving facial photo data provided by customers" refers to a system or protocol that allows the server to receive facial photo data taken or selected by customers using a device such as a smartphone or tablet.

[0307] The "image processing means" refers to algorithms or software that detects the face, removes the background, and extracts the outline of the hair from the facial photograph data received by the server.

[0308] The "means for receiving instruction data" refers to a system or protocol that allows a customer to input their desired hairstyle and hair color within the app and for the server to receive that input data.

[0309] The "candidate generation means" refers to a generative AI model or algorithm for generating multiple hairstyle and hair color candidates based on the received instruction data.

[0310] An "emotion analysis method" is an algorithm or system that uses a generative AI model to analyze a customer's facial photograph and input data, determine the customer's emotions, and make adjustments based on the results.

[0311] The "combining means" refers to modeling software or algorithms that combine the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[0312] "Communication means" refers to the network infrastructure and protocols used to transmit the 3D model generated by the server to the customer terminal and the hairdresser terminal.

[0313] "Display means" refers to the interface and software for displaying the received 3D model on the customer terminal and the hairdresser terminal.

[0314] The program for this system is designed to improve customer satisfaction at beauty salons. Specifically, it provides technology that generates a 3D model based on a customer's facial photograph, analyzes the user's emotions, and adjusts suggested hairstyles and hair colors. Below, we will explain what hardware and software are used, and what data processing and calculations are performed.

[0315] Hardware and software used

[0316] Customer terminal

[0317] Customer terminals are primarily smartphones or tablets, which have application software installed on them and provide an interface for users to take a photo of their face and input their desired hairstyle and hair color.

[0318] Hairdresser's terminal

[0319] The hairdresser's terminal also uses a smartphone or tablet, which has application software installed on it that allows the hairdresser to view the customer's 3D model and input any correction requests.

[0320] server

[0321] The server acts as a central system, processing data sent from the customer and hairdresser devices. It implements advanced image processing algorithms and generative AI models, using libraries such as OpenCV and TensorFlow. It also uses Microsoft Azure's emotion recognition API to analyze customer emotions.

[0322] Data processing and calculation

[0323] Upload and pre-process face photos

[0324] The user takes a photo of their face on their device and sends it to the server, which then analyzes the received photo data to detect the face, remove the background, and extract the outline of the hair, using AI image processing algorithms.

[0325] Emotion analysis

[0326] The server runs an emotion engine that analyzes the uploaded facial photo data and customer input data to identify the user's emotions, which can then be classified as positive or negative.

[0327] Generate hairstyle and hair color candidates

[0328] The user inputs their desired hairstyle and hair color, which is then received by the server. The server then analyzes this data using NLP (natural language processing) and generates multiple candidates using a generative AI model. Based on the analysis results of the emotion engine, the server also adjusts the candidates to suit the user's emotional state.

[0329] 3D model compositing and display

[0330] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model. The generated 3D model is sent to the client's and stylist's devices using 3D modeling software such as Blender or Unity. The 3D model received on the device can be rotated and zoomed in and out, allowing the client and stylist to view it together.

[0331] Specific examples

[0332] Customer A Scenario

[0333] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0334] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0335] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[0336] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0337] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[0338] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0339] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[0340] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[0341] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[0342] Prompt Sentence Examples

[0343] "Based on a photo of the customer's face, please generate a 3D model with a short bob hairstyle and reddish hair color. Also, please display variations of the hairstyle that emphasize the relaxed look while the user is in a relaxed state."

[0344] This allows customers and hairdressers to have a specific and detailed shared image of the hairstyle they want, allowing for smooth counseling. Furthermore, emotion analysis can be used to provide optimal suggestions to customers, which is expected to improve customer satisfaction.

[0345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0346] Program processing flow

[0347] Step 1: Upload customer photos

[0348] The user launches the app and takes a photo of their face or selects one from the gallery. The device temporarily stores the photo data and sends it to the server.

[0349] Input: User's face photo

[0350] Output: Face photo data sent to the server

[0351] Specific behavior:

[0352] The user takes a photo of their face using the in-app camera, or selects an existing photo from the device's gallery.

[0353] The device temporarily saves the selected photo and sends it to the server using an HTTP POST request.

[0354] Step 2: Image preprocessing on the server

[0355] The server analyzes the received facial photo data, detects the face, removes the background, and extracts the outline of the hair.

[0356] Input: User's face photo data

[0357] Output: Data after face detection, background removal, and hair contour extraction

[0358] Specific behavior:

[0359] The image data received by the server is processed using an AI image processing algorithm (OpenCV or TensorFlow).

[0360] Detects the face and removes it by masking the background.

[0361] Identify and extract the hair contours.

[0362] Step 3: Sentiment Analysis

[0363] The server uses an emotion engine to analyze the uploaded facial photo and customer input data to identify the user's emotional state.

[0364] Input: Face photo data and customer input data

[0365] Output: Analysis of the user's emotional state

[0366] Specific behavior:

[0367] The server calls the emotion engine's API (e.g., emotion recognition API) and passes the data to be analyzed as input.

[0368] The emotion engine determines emotions such as smile or anger and returns the results to the server.

[0369] Step 4: Enter your hairstyle and hair color preferences

[0370] Users input their desired hairstyle and hair color within the app, and the device sends this data to the server.

[0371] Input: User's desired hairstyle and hair color instructions

[0372] Output: Instruction data sent to the server

[0373] Specific behavior:

[0374] The user enters their desired style, such as "short bob" or "reddish," into a text box within the app.

[0375] The device sends this input data to the server via an HTTP POST request.

[0376] Step 5: Generate hairstyle and hair color candidates

[0377] The server uses NLP (natural language processing) to analyze the user's request, then runs a generative AI model to generate multiple hairstyle and hair color options, and makes adjustments based on the results of sentiment analysis.

[0378] Input: User instruction data and sentiment analysis results

[0379] Output: Candidate data for hairstyle and hair color

[0380] Specific behavior:

[0381] The server uses an NLP model (e.g., a GPT model) to analyze the user's wishes.

[0382] The analysis results are input into a generative AI model to generate multiple hairstyle and hair color options.

[0383] Adjust candidates based on the results of sentiment analysis.

[0384] Step 6: Composite and send your 3D model

[0385] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[0386] Input: Face photo data, hairstyle and hair color candidate data

[0387] Output: The generated 3D model, and the client and hairdresser devices that receive it.

[0388] Specific behavior:

[0389] The server uses 3D modeling software such as Blender or Unity to integrate each piece of data.

[0390] The generated 3D model is sent to the customer's device and the hairdresser's device via an HTTP POST request.

[0391] Step 7: View and check the 3D model

[0392] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0393] Input: Received 3D model

[0394] Output: A displayed 3D model

[0395] Specific behavior:

[0396] The device launches a 3D viewer, reads the received model data, and displays it.

[0397] Users can use touch to rotate the 3D model and zoom to view details.

[0398] Step 8: Enter and Regenerate Correction Requests

[0399] The user or hairdresser checks the 3D model and inputs any additional requests or corrections. The device sends this data to the server, which then generates a new 3D model and sends it to the device.

[0400] Input: Additional requests and correction instructions

[0401] Output: Updated 3D model

[0402] Specific behavior:

[0403] Users input requests for corrections, such as "I would like the hair color to be a little lighter," into the app.

[0404] The device sends this correction data to the server via an HTTP POST request.

[0405] The server processes the data again, generates a new 3D model, and sends it to the device.

[0406] (Application example 2)

[0407] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0408] Conventional virtual try-on systems simply display the clothes and accessories selected by the user as they are, and are unable to improve the user's emotional state or the quality of the virtual experience in real time. This makes it difficult to provide optimal suggestions for the user, resulting in issues with customer satisfaction. Furthermore, it is time-consuming to combine facial photo data with candidate clothes and accessories, and it is difficult to quickly regenerate them based on requested modifications.

[0409] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing the background, and extracting the outline of the appearance; means for receiving instruction data specifying the appearance and color desired by the customer; candidate generation means for generating multiple appearance and color candidates based on the instruction data; synthesis means for synthesizing the appearance and color candidates generated by the candidate generation means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and an operation terminal; display means for displaying the 3D model; and means for analyzing the emotional state and adjusting the generated appearance and color candidates. This enables a system that provides an optimal virtual try-on experience based on user emotional analysis, improves customer satisfaction, and enables quick and easy modifications.

[0410] A "customer terminal" is a digital device operated by the customer, and is used to take a facial photograph, input instruction data, check 3D models, etc.

[0411] An "operation terminal" is a digital device operated by an operator, which is used to check the 3D model together with the customer and input any correction requests.

[0412] The "image processing means" is a collection of algorithms and software that detects facial features from received facial photograph data, removes background, and extracts the contours of the face.

[0413] "Instruction data" refers to data for inputting specific requests such as the appearance and color desired by the customer.

[0414] A "candidate generator" is a collection of algorithms and software that generates multiple appearance and color candidates based on instruction data.

[0415] "Synthesis means" refers to technology or software that integrates the generated appearance and color candidates with facial photograph data to generate a 3D model.

[0416] "Communication means" refers to the network communication technology and software used to transmit the generated 3D model to the customer terminal and the operation terminal.

[0417] "Display means" refers to a display device or software that allows a user or operator to visually check the received 3D model.

[0418] "Emotional state" is a mental health condition that analyzes a user's emotions and moods and monitors them in real time.

[0419] The "emotion analysis means" is a collection of technologies and algorithms for analyzing the user's emotional state based on facial photo data and input data, and adjusting the appearance and color candidates.

[0420] The present invention is a system for improving the fitting experience in a virtual fashion shop. The main components include a customer terminal, an operation terminal, a server, and a sentiment analysis engine. Each component is described below.

[0421] Customer terminal

[0422] The customer device refers to a smartphone or head-mounted display (HMD) operated by the customer. This device is used by the customer to take or upload a photo of their face and input their desired appearance and color. The customer device has the function of temporarily storing the face photo data and sending it to a server.

[0423] Operation terminal

[0424] The operation terminal is a tablet or PC operated by an operator. On this terminal, the operator can check the 3D model together with the customer and input any correction requests. The operation terminal also has the function of displaying the 3D model received from the server and inputting corrections as needed.

[0425] server

[0426] The server processes the received facial photo data, removing the background and extracting the contours of the appearance. The server uses image processing libraries such as OpenCV and Dlib. The server also uses natural language processing libraries (e.g., spaCy, NLTK) to analyze the customer's instruction data and input it into an AI generation model. The generated appearance and color candidates are integrated into a 3D model and sent to the customer's device and the operating device.

[0427] Sentiment Analysis Engine

[0428] The emotion analysis engine analyzes the user's emotional state based on their photo data and input data, using cloud-based emotion analysis services such as Amazon Rekognition and Microsoft Azure Face API, and then adjusts the optimal appearance and color options based on the analysis results.

[0429] Specific examples of processing

[0430] For example, if User B inputs that they want a "blue dress," the server analyzes this data and generates multiple blue dress candidates. If the user also requests that the color be "more vibrant," a new 3D model will be provided.

[0431] Prompt Sentence Examples

[0432] A user types: "I want to try on a blue dress."

[0433] The desired outfit type is "dress" and the color is "blue." Sentiment analysis indicates that the user is happy. Please present a more vibrant blue color.

[0434] The system of the present invention provides an optimal virtual try-on experience based on user sentiment analysis, improving customer satisfaction and enabling quick and easy modifications.

[0435] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0436] Step 1:

[0437] The user uses the customer's device to take a photo of their face or select one from a gallery, and then enters the photo data into the application. This photo data is temporarily saved in the customer's device's local storage. The device then sends this photo data to the server. The specific operation of sending the photo data involves communication using an HTTP request.

[0438] Step 2:

[0439] The server receives the facial photo data sent. Using OpenCV and Dlib, the server detects the face from the facial photo data, removes the background, and extracts the outline of the face. Specifically, the server converts the image to grayscale, detects facial landmarks, and then cuts out only the necessary parts. The input is the facial photo data sent from the customer's device, and the output is image data from which the face has been detected, the background removed, and the outline of the face extracted.

[0440] Step 3:

[0441] The user inputs the desired appearance and color in natural language using a customer terminal. This instruction data is temporarily stored in the local storage of the customer terminal and then sent to the server. The input is text data in natural language entered by the user, and the output is the instruction data sent to the server.

[0442] Step 4:

[0443] The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the instruction data it receives. The server parses the instruction data and identifies the desired appearance and color. Specific operations include tokenizing the text and extracting specific attributes (e.g., color, style). The input is the instruction data sent from the client device, and the output is the parsed desired appearance and color information.

[0444] Step 5:

[0445] The server inputs data into an AI generative model based on the analyzed appearance and color information. The generative AI model generates multiple appearance and color candidates. Specifically, it inputs the tensor data corresponding to the model and receives the generated candidate data. The input is the analyzed appearance and color information, and the output is the generated appearance and color candidates.

[0446] Step 6:

[0447] The server synthesizes the generated appearance and color candidates with the facial photo data to generate a 3D model. Specifically, it uses 3D rendering technology to integrate the photo data and candidate data. The input is the facial photo data and the generated candidate data, and the output is a 3D model.

[0448] Step 7:

[0449] The server sends the generated 3D model to the customer terminal and the operation terminal. The customer terminal and the operation terminal then receive and display the 3D model. Specifically, the transmission is performed via an HTTP request, and the received 3D model is displayed on the terminal's display. The input is the 3D model, and the output is the 3D model displayed on the terminal's display.

[0450] Step 8:

[0451] The user and operator check the 3D model and input any correction requests as necessary. Once the correction requests are entered, they are sent to the server from the customer terminal or the operation terminal. The input is text data of the correction requests, and the output is correction instruction data that is sent to the server.

[0452] Step 9:

[0453] The server regenerates the 3D model based on the correction request and resends the updated 3D model. Specifically, it generates new appearance and color candidates based on the correction instruction data and recomposes them. The input is the correction instruction data, and the output is the updated 3D model.

[0454] Step 10:

[0455] The customer terminal and the operation terminal display the received 3D model again for final confirmation. With final consent, the treatment (or purchase) begins. Specifically, the 3D model is displayed on the display again and the user's consent is sought. The input is the updated 3D model, and the output is the final confirmed 3D model.

[0456] In this way, a system is realized that provides an optimal virtual try-on experience based on user sentiment analysis, improves customer satisfaction, and allows for quick and easy adjustments.

[0457] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0458] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0459] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0460] [Second embodiment]

[0461] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0462] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0463] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0464] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0465] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0466] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0467] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0468] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0471] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0472] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0473] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating 3D models that are combined with customer photographs. This invention enables hairdressers and customers to accurately share specific images, improving customer satisfaction with the results of treatments.

[0474] System configuration

[0475] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a device such as a smartphone or tablet. The hairdresser terminal is also a digital device, and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0476] Program processing

[0477] Customer photo upload

[0478] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[0479] Image preprocessing on the server

[0480] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[0481] Inputting customer requests

[0482] The user (customer) inputs their desired hairstyle and hair color using natural language within the app, and the device sends this instruction data to the server.

[0483] Generate hairstyle and hair color candidates

[0484] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options, including analyzing the user's request using natural language processing (NLP) to generate 3D models of the options.

[0485] 3D model composition and transmission

[0486] The server combines the generated hairstyle and hair color candidates with the customer's facial photo data to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[0487] Display and check the 3D model

[0488] The device then displays the received 3D model for the user and hairdresser to review, allowing customers to see in detail which hairstyle and hair color will suit their face before the treatment.

[0489] Entering and regenerating correction requests

[0490] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I want the hair color to be a little lighter" can be included. The device sends these correction requests to the server, which then generates and updates the 3D model again.

[0491] Specific examples

[0492] Customer A Scenario

[0493] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0494] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0495] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0496] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0497] 5. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0498] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[0499] 7. The server receives the correction request and sends the regenerated 3D model to the device.

[0500] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[0501] This system allows customer A and the hairdresser to have a smooth consultation based on a specific and shared image.

[0502] The processing flow will be explained below.

[0503] Step 1:

[0504] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and launches the camera to take a photo or selects an existing photo from the gallery.

[0505] Step 2:

[0506] The device temporarily stores the selected or taken photo, optimizes the image format and resolution, and sends the image data to the server.

[0507] Step 3:

[0508] The server receives the uploaded photo data and associates it with the designated customer account.

[0509] Step 4:

[0510] The server uses AI image processing models to detect faces in the received photos, remove backgrounds, and extract hair contours, providing the basis for changing hairstyles and hair colors.

[0511] Step 5:

[0512] The user inputs the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[0513] Step 6:

[0514] The server analyzes the user's request using natural language processing (NLP) and passes it as input to an AI generation model, which then generates candidate hairstyles and hair colors.

[0515] Step 7:

[0516] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0517] Step 8:

[0518] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0519] Step 9:

[0520] At the salon, the hairdresser and the user check the 3D model and input specific requests for corrections, such as making the hair color a little lighter. The requests are then sent from the device to the server.

[0521] Step 10:

[0522] The server receives the new request, updates the 3D model using the AI ​​model again, and sends the revised 3D model back to the client device.

[0523] Step 11:

[0524] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[0525] Example 1

[0526] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0527] Traditionally, consultations about hairstyles and hair colors at beauty salons were conducted by the customer explaining the details verbally, and the hairdresser then performed the treatment based on that. This method could lead to dissatisfaction with the finished product, as the customer's intended hairstyle or hair color was not accurately conveyed. Furthermore, there was a lack of a way to share a specific image before the treatment, making it difficult to improve customer satisfaction.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0529] In this invention, the server includes means for receiving facial photograph data provided by a customer, image processing means for detecting facial features from the facial photograph based on the facial photograph data, removing the background, and extracting the hair outline, means for receiving instruction data specifying the hairstyle and hair color desired by the customer, means for generating a plurality of hairstyle and hair color candidates, means for synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photograph data to generate a three-dimensional model, means for transmitting the three-dimensional model generated by the synthesis means to a customer terminal and a hairdresser terminal, means for the customer or hairdresser to input correction requests, and means for regenerating a corrected three-dimensional model based on the correction requests. This enables smooth counseling between the customer and the hairdresser based on a specific and shared image, and allows treatment that accurately reflects the customer's requests.

[0530] "Facial photo data provided by the customer" refers to image data of the customer's own face that has been photographed or selected by the customer using a device such as a smartphone or tablet.

[0531] "Image processing means for detecting facial areas, removing background, and extracting hair contours" is a function that uses a facial recognition algorithm within the server to identify facial areas, remove background, and clarify the outline of hair.

[0532] "Instruction data specifying the hairstyle and hair color desired by the customer" is information in which the customer selects the desired hairstyle and hair color through an input interface using natural language or from pre-prepared options.

[0533] The "means for generating multiple hairstyle and hair color candidates" is a function that uses a generative AI model in the server to create various hairstyle and hair color variations based on customer instruction data.

[0534] The "means for generating a three-dimensional model by combining the facial photograph data" is a function that combines the hairstyle and hair color data generated by the server with the customer's facial photograph data to create a three-dimensional hairstyle model that fits the customer's face.

[0535] The "communication means for transmitting to the customer terminal and the hairdresser terminal" is a function for transmitting the three-dimensional model generated by the server to the terminals of the customer and the hairdresser via data communication.

[0536] The "means for a customer or a hairdresser to input correction requests" is an interface that allows a customer or a hairdresser to input desired changes or corrections to the three-dimensional model.

[0537] The "means for regenerating a modified three-dimensional model" is a function in which the server receives a request for modification from a customer or a hairdresser and updates and regenerates the three-dimensional model.

[0538] "Display means" refers to a display or screen on which the customer and hairdresser can view the three-dimensional model.

[0539] The "operation means" is an interface for rotating and zooming in and out of the three-dimensional model.

[0540] This invention is a digital hair catalog system developed to improve customer satisfaction at beauty salons, and is a technology that generates a 3D model that is combined with a customer's facial photograph. This allows the hairdresser and customer to accurately share a specific image, improving customer satisfaction with the results of the treatment.

[0541] System configuration

[0542] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a digital device such as a smartphone or tablet. The hairdresser terminal is also a digital device and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0543] 1. Customer terminal

[0544] Customer terminals can be any commercially available digital device such as a smartphone or tablet.

[0545] The application has the ability to take photos and select photos from the gallery.

[0546] 2. Server

[0547] The server receives the photo data, processes the images, and generates a three-dimensional model.

[0548] The server can use common cloud services and uses AI image processing algorithms (e.g., OpenCV and TensorFlow) internally.

[0549] 3. Hairdresser's terminal

[0550] The hairdresser terminal uses the same digital device as the customer terminal.

[0551] A dedicated application is installed that allows hairdressers to view the 3D model and communicate with customers.

[0552] Process Details

[0553] 1. Upload customer photos

[0554] Users launch the app using their smartphone or tablet and take a photo of their face or select one from their gallery.

[0555] The device temporarily stores the photo data and sends it to the server.

[0556] 2. Image preprocessing on the server

[0557] The server receives the transmitted photo data and uses its internal AI image processing algorithm to detect the face area.

[0558] The background is removed and the hair outline is extracted, generating reference data for changing the hairstyle and hair color.

[0559] 3. Input customer requests

[0560] Users input their desired hairstyle and hair color using natural language within the app.

[0561] The terminal transmits this instruction data to the server.

[0562] 4. Generating hairstyle and hair color candidates

[0563] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options.

[0564] The server uses natural language processing (NLP) to analyze the user's request and generate candidate 3D models.

[0565] 5. Synthesis and transmission of 3D model

[0566] The server combines the generated hairstyle and hair color candidates with the customer's facial photograph data to generate a three-dimensional model.

[0567] The server transmits this three-dimensional model to the customer terminal and the hairdresser terminal.

[0568] 6. Display and check the 3D model

[0569] The terminal displays the received three-dimensional model, and the user and hairdresser confirm it.

[0570] This allows customers to check specifically which hairstyle and hair color suits their face before the treatment.

[0571] 7. Enter and regenerate correction requests

[0572] The user or hairdresser checks the three-dimensional model and inputs correction requests as necessary.

[0573] For example, this includes specific instructions such as "I want my hair color to be lighter."

[0574] The terminal sends this correction request to the server, which then generates and updates the 3D model again.

[0575] Specific examples

[0576] Customer A Scenario

[0577] 1. The user (Customer A) opens the app and uploads a photo of their face. For example, they take a photo of their face using the camera on their smartphone.

[0578] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0579] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0580] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0581] 5. The generated 3D model is sent to the customer terminal and the hairdresser terminal.

[0582] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[0583] 7. The server receives the modification request and sends the regenerated 3D model to the terminal.

[0584] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[0585] Prompt Sentence Examples

[0586] "Upload a photo of your face and enter your desired hairstyle and hair color. Example: 'Short bob', 'Reddish'"

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Step 1:

[0589] Customer photo upload

[0590] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The user selects a photo within the app, and the app temporarily saves the photo data. The user's device then sends the photo data to the server. The input data is the photo of their face, and the output is the image data received by the server. Specifically, the user takes a photo of their face using the camera function on their smartphone and presses the "send" button on the app.

[0591] Step 2:

[0592] Image preprocessing on the server

[0593] The server receives the facial photo data sent by the user and temporarily stores it. Next, it automatically detects the face using an AI image processing algorithm (e.g., OpenCV or TensorFlow) inside the server. It also performs processing to remove the background and extract the outline of the hair. The input data is the facial photo data, and the output is image data with the background removed and the outline of the hair extracted. Specifically, the AI ​​algorithm identifies the facial area and performs processing to make the background in that area transparent.

[0594] Step 3:

[0595] Inputting customer requests

[0596] Within the app, the user uses natural language to input their desired hairstyle and hair color into a text box. The user enters, for example, "short bob" or "reddish," and the device sends this text data to the server. The input data is the user's natural language request, and the output is the instruction data received by the server. Specifically, the user enters the desired hairstyle and hair color into the app's input fields and presses the "send" button.

[0597] Step 4:

[0598] Generate hairstyle and hair color candidates

[0599] The server receives the request data sent by the user and analyzes it using an AI model (e.g., a natural language processing engine). The server generates multiple hairstyle and hair color candidates based on the analysis results. The input data is the request data, and the output is the generated hairstyle and hair color candidate data. Specifically, the AI ​​model analyzes requests such as "short bob" or "reddish" and creates appropriate hairstyle and hair color variations based on that.

[0600] Step 5:

[0601] 3D model synthesis and transmission

[0602] The server combines the generated hairstyle and hair color candidates with facial photo data to generate a 3D model. The server then sends the combined 3D model to the customer's device and the hairdresser's device. The input data is hairstyle and hair color candidate data and facial photo data, and the output is a 3D model. Specifically, the server uses 3D modeling software such as Blender to combine the facial and hairstyle data to generate a 3D model, which is then sent to the device.

[0603] Step 6:

[0604] Display and check the 3D model

[0605] The device receives and displays the 3D model data sent from the server. By checking the 3D model, the user and hairdresser can get a concrete idea of ​​the hairstyle and hair color that will suit the customer's face before the treatment. The input data is the 3D model data, and the output is the displayed 3D model. Specifically, the smartphone or tablet app displays the 3D model using a view function that allows the user to rotate, zoom in, and out.

[0606] Step 7:

[0607] Entering and regenerating correction requests

[0608] The user or hairdresser checks the 3D model and, if necessary, inputs any correction requests into the app. For example, specific instructions such as "I want my hair color lighter" can be included. The device sends this correction request data to the server, which then generates and updates the 3D model again. The input data is the correction request data, and the output is the corrected 3D model data. Specifically, the user or hairdresser enters the correction request into the app's input field and presses the "Send" button.

[0609] The above are the specific processing steps of the system that generates a three-dimensional hairstyle model using a customer's facial photograph.

[0610] (Application example 1)

[0611] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] Conventional methods for proposing hairstyles at beauty salons make it difficult for customers and hairdressers to accurately share specific images, which can lead to a decrease in customer satisfaction after the treatment. There was also a need for a method that allows customers to try on new hairstyles and hair colors without leaving their homes. Furthermore, there were issues with usability and accuracy regarding systems for trying on hairstyles in virtual environments.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0614] In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; synthesis means for generating a 3D model by synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photo data; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and a therapist terminal; display means for displaying the 3D model; a generative AI model that analyzes natural language to understand the customer's request and generates and modifies the 3D model based on the request; and prompt generation means for inputting instruction data to be analyzed into the generative AI model. This allows customers to try new hairstyles and hair colors without leaving their homes and share specific images before the treatment, thereby improving customer satisfaction and improving the usability and accuracy of hairstyle try-on in a virtual environment.

[0615] "Customer terminal" refers to a digital device operated by a customer, including a smartphone, tablet, laptop, etc.

[0616] "Receiving means" refers to a means for receiving data, and in this case refers to a mechanism for receiving images, instruction data, etc. via Internet communication.

[0617] The "image processing means" refers to a means for performing processing to detect specific parts from received image data or remove backgrounds, and specifically includes algorithms for face detection and hair contour extraction.

[0618] "Instruction data" refers to data indicating information such as the hairstyle and hair color desired by the customer, and is input through the application.

[0619] The "candidate generation means" is a means for generating multiple hairstyle and hair color candidates based on instruction data, and is realized here by a generative AI model.

[0620] The "combining means" is a means for combining the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[0621] "Communication means" refers to the means for transmitting the generated 3D model and other data to the client terminal and the therapist terminal, primarily via the Internet.

[0622] The "display means" is a means for displaying the received 3D model on the terminal, and includes a graphical user interface (GUI).

[0623] A "professor terminal" is a digital device operated by the practitioner (hairdresser), and refers mainly to a computer or tablet used within the salon.

[0624] "Generative AI model" refers to an artificial intelligence (AI) model that analyzes natural language and generates and modifies 3D models based on customer requests.

[0625] The "prompt sentence generation means" is a means for generating sentences for inputting instruction data to be analyzed into the generative AI model.

[0626] The present invention relates to a digital hair catalog system for improving customer satisfaction in a virtual hair salon system. This system is comprised of a customer terminal, a therapist terminal, and a server, and transmits and receives data via the Internet.

[0627] Role of customer terminals

[0628] The customer terminal is a digital device such as a smartphone or tablet. The customer first launches the application and takes a photo of their face or selects one from the gallery. The photo is temporarily saved on the customer terminal and then sent to a server via the Internet.

[0629] Processing on the server

[0630] The server processes the data in the following steps:

[0631] 1. The server receives the facial photo data sent from the customer's device. The software used is Python's PIL (Python Imaging Library) and Requests library.

[0632] 2. Based on the received facial photo data, the system detects the face, removes the background, and extracts the hair outline. The OpenCV library is used here.

[0633] 3. Next, the customer receives instruction data specifying their desired hairstyle and hair color. This instruction data is entered by the customer in natural language within the app, and the server analyzes it using Hugging Face's Transformers library and T5 model.

[0634] 4. Generate multiple hairstyle and color options based on natural language analysis. This process uses a generative AI model to accurately reflect the customer's preferences in the 3D model.

[0635] 5. The generated hairstyle and hair color candidates are combined with facial photo data to generate a 3D model. The 3D model is created using the Open3D library.

[0636] 6. Finally, the generated 3D model can be sent to the client's device and the practitioner's device and displayed.

[0637] Role of the therapist terminal

[0638] The hairdresser's terminal is a digital device operated by the hairdresser. The hairdresser checks the 3D model sent from the server and selects the most suitable hairstyle and hair color together with the customer. If necessary, they can resubmit any correction requests to the server and generate and modify the 3D model again. This allows the customer and hairdresser to share a specific image.

[0639] Examples and prompts

[0640] For example, if a customer requests a short bob with red hair, the text is parsed with the prompt, "Please provide generated hairstyle and hair color options: Short bob with red hair." This prompt allows the generative AI model to generate multiple options based on the customer's request and synthesize them into a 3D model.

[0641] Example prompt sentence:

[0642] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[0643] This system allows customers to try on new hairstyles and hair colors without leaving their homes, and by sharing a concrete image of the desired look before the treatment, customer satisfaction is increased. In addition, the high-precision processing of natural language analysis and 3D model generation on the server improves the usability and accuracy of trying on hairstyles in a virtual environment.

[0644] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0645] Step 1:

[0646] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The selected photo is temporarily saved on the customer's device and then sent to the server via the Internet. In this step, the customer's photo is used as input data, and the photo data sent to the server is obtained as output data.

[0647] Step 2:

[0648] The server internally processes the received facial photo data. First, it uses the OpenCV library to detect the face, remove the background, and extract the hair outline. At this stage, the facial photo data is used as input data, and image data with the background removed and the hair outline extracted is obtained as output data.

[0649] Step 3:

[0650] The user inputs the desired hairstyle and hair color in the app using natural language. This instruction data is sent from the customer terminal to the server. In this step, instruction text in natural language is used as input data, and the instruction data sent to the server is obtained as output data.

[0651] Step 4:

[0652] The server analyzes the received instruction data. Using Hugging Face's Transformers library and the T5 model, it generates hairstyle and hair color candidates based on natural language. The instruction data is used as input data, and multiple hairstyle and hair color candidates are generated as output data.

[0653] Step 5:

[0654] The generative AI model generates a 3D model by combining the generated hairstyle and hair color candidates with facial photo data. The 3D model is created using the Open3D library. In this step, hairstyle and hair color candidates and facial photo data with the background removed are used as input data, and a 3D model is generated as output data.

[0655] Step 6:

[0656] The server sends the generated 3D model to the client's device and the therapist's device. The client's device and the therapist's device display the 3D model, allowing the client and therapist to share and review it. In this step, the generated 3D model is used as input data, and the transmitted 3D model is obtained as output data.

[0657] Step 7:

[0658] The client and the therapist check the 3D model on the therapist's terminal and input any correction requests as necessary. The correction requests are sent from the therapist's terminal to the server. At this stage, instruction data for corrections is used as input data, and the correction requests are sent to the server as output data.

[0659] Step 8:

[0660] The server generates and modifies the 3D model again based on the modification request, and sends the results back to the client's device and the therapist's device. The modification request is used as input data, and the modified 3D model is generated as output data.

[0661] Example prompt sentence:

[0662] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[0663] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0664] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating a 3D model that is combined with a customer's facial photograph, and technology for recognizing the user's emotions and adjusting suggested hairstyles and hair colors. This allows the hairdresser and customer to accurately share a specific image, further improving customer satisfaction with the results of the treatment.

[0665] System configuration

[0666] This system consists of a customer terminal, a server, a hairdresser terminal, and an emotion engine. The customer terminal is a digital device such as a smartphone or tablet, and the hairdresser terminal is also a digital device. The server is a central system that processes data and communicates between these terminals. The emotion engine is a combination of software and hardware for recognizing customer emotions.

[0667] Program processing

[0668] Customer photo upload

[0669] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[0670] Image preprocessing on the server

[0671] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[0672] Emotion analysis with emotion engine

[0673] The user's emotion engine analyzes the uploaded facial photo data and input data from the customer's device in real time to monitor the user's emotional state. For example, if the customer is smiling, it is analyzed as having a positive emotion.

[0674] Input hairstyle and hair color requests

[0675] The user inputs their desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[0676] Generate hairstyle and hair color candidates

[0677] The server analyzes the user's request (instruction data) using natural language processing (NLP) and passes it as input to the AI ​​generation model. The AI ​​generates candidate hairstyles and hair colors. Furthermore, based on the analysis results of the emotion engine, the candidate hairstyles and hair colors are adjusted to suit the user's emotional state.

[0678] 3D model composition and transmission

[0679] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0680] Display and check the 3D model

[0681] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0682] Entering and regenerating correction requests

[0683] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I'd like the hair color to be a little lighter" are included. The device sends these correction requests to the server. The server then generates and updates the 3D model again, and sends the newly generated 3D model to the device.

[0684] Specific examples

[0685] Customer A Scenario

[0686] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0687] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0688] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[0689] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0690] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[0691] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0692] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[0693] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[0694] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[0695] This system enables more personalized hairstyle suggestions that take into account the user's emotional state, enabling smooth counseling based on a specific and shared image between customer A and the hairdresser.

[0696] The processing flow will be explained below.

[0697] Step 1:

[0698] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and takes a photo of themselves or selects an existing photo from the gallery.

[0699] Step 2:

[0700] The device temporarily stores the selected or taken photo data, optimizes the image format and resolution, and sends it to the server.

[0701] Step 3:

[0702] The server receives the uploaded photo data and associates it with the designated customer account. The server then begins processing the image data and uses AI image processing models to detect facial features.

[0703] Step 4:

[0704] After detecting the face, the server performs image processing to remove the background and extract the hair outline, generating the basic data for changing the hairstyle and hair color.

[0705] Step 5:

[0706] The device receives a notification from the server that face detection and image processing are complete, and prompts the user to perform the next step: The user enters the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app.

[0707] Step 6:

[0708] The device sends instruction data to the server, which analyzes the user's request using natural language processing (NLP) and inputs the instruction data into an AI-generated model.

[0709] Step 7:

[0710] The server uses an AI generative model to generate candidates for a specific hairstyle and hair color. Multiple hairstyle and hair color candidates are generated.

[0711] Step 8:

[0712] The server activates the emotion engine and analyzes the uploaded facial photo and instruction data to recognize the user's emotional state, and adjusts the hairstyle and hair color suggestions based on the recognized emotional state.

[0713] Step 9:

[0714] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0715] Step 10:

[0716] The device displays the received 3D model, allowing the user to rotate it and zoom in and out to check the details. The hairdresser also checks the same model at the salon.

[0717] Step 11:

[0718] The user and hairdresser check the 3D model and input requests for corrections to specific parts (for example, "I want the hair color to be a little lighter"). The device then sends these requests to the server.

[0719] Step 12:

[0720] The server receives the new request and updates the 3D model using the AI ​​model again. The revised 3D model is then sent back to the customer's device and the hairdresser's device.

[0721] Step 13:

[0722] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[0723] Specific examples

[0724] Customer A Scenario

[0725] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0726] 2. The device sends the photo data to the server.

[0727] 3. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0728] 4. The server notifies the terminal that image preprocessing is complete.

[0729] 5. The user (customer A) inputs that he / she desires a "short bob" and "reddish" hairstyle and hair color.

[0730] 6. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0731] 7. The server uses the emotion engine to analyze the user's emotional state and adjusts the candidates suitable for a relaxed state.

[0732] 8. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0733] 9. The user (Customer A) and the hairdresser check the 3D model at the beauty salon.

[0734] 10. The user inputs a request for correction, such as "Make the hair color a little lighter."

[0735] 11. The server receives the correction request and sends the regenerated 3D model to the device.

[0736] 12. The user makes a final confirmation and, once they agree, the treatment begins.

[0737] This system enables personalized hairstyle suggestions that take into account the user's emotional state, realizing smooth counseling based on a specific and shared image between the customer and hairdresser.

[0738] Example 2

[0739] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] Conventional hair catalog systems used in hair salons have made it difficult to improve customer satisfaction. In particular, the image of hairstyle shared between the customer and the hairdresser is vague, resulting in customers' expectations often not being met. In addition, it is difficult to reflect the customer's feelings and requests in real time, making it impossible to make optimal suggestions for each individual customer. A system that solves these issues was needed.

[0741] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; emotion analysis means for analyzing the customer's emotional data using a generative AI model and making adjustments based on the customer's emotions; synthesis means for synthesizing the hairstyle and hair color candidates adjusted by the candidate generation means and emotion analysis means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to the customer terminal and the hairdresser terminal; and display means for displaying the 3D model. This allows the customer and the hairdresser to have a specific and detailed shared image of the hairstyle and hair color during a smooth consultation. Furthermore, emotion analysis allows for optimal suggestions to be made to the customer, which is expected to improve customer satisfaction.

[0742] "Means for receiving facial photo data provided by customers" refers to a system or protocol that allows the server to receive facial photo data taken or selected by customers using a device such as a smartphone or tablet.

[0743] The "image processing means" refers to algorithms or software that detects the face, removes the background, and extracts the outline of the hair from the facial photograph data received by the server.

[0744] The "means for receiving instruction data" refers to a system or protocol that allows a customer to input their desired hairstyle and hair color within the app and for the server to receive that input data.

[0745] The "candidate generation means" refers to a generative AI model or algorithm for generating multiple hairstyle and hair color candidates based on the received instruction data.

[0746] An "emotion analysis method" is an algorithm or system that uses a generative AI model to analyze a customer's facial photograph and input data, determine the customer's emotions, and make adjustments based on the results.

[0747] The "combining means" refers to modeling software or algorithms that combine the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[0748] "Communication means" refers to the network infrastructure and protocols used to transmit the 3D model generated by the server to the customer terminal and the hairdresser terminal.

[0749] "Display means" refers to the interface and software for displaying the received 3D model on the customer terminal and the hairdresser terminal.

[0750] The program for this system is designed to improve customer satisfaction at beauty salons. Specifically, it provides technology that generates a 3D model based on a customer's facial photograph, analyzes the user's emotions, and adjusts suggested hairstyles and hair colors. Below, we will explain what hardware and software are used, and what data processing and calculations are performed.

[0751] Hardware and software used

[0752] Customer terminal

[0753] Customer terminals are primarily smartphones or tablets, which have application software installed on them and provide an interface for users to take a photo of their face and input their desired hairstyle and hair color.

[0754] Hairdresser's terminal

[0755] The hairdresser's terminal also uses a smartphone or tablet, which has application software installed on it that allows the hairdresser to view the customer's 3D model and input any correction requests.

[0756] server

[0757] The server acts as a central system, processing data sent from the customer and hairdresser devices. It implements advanced image processing algorithms and generative AI models, using libraries such as OpenCV and TensorFlow. It also uses Microsoft Azure's emotion recognition API to analyze customer emotions.

[0758] Data processing and calculation

[0759] Upload and pre-process face photos

[0760] The user takes a photo of their face on their device and sends it to the server, which then analyzes the received photo data to detect the face, remove the background, and extract the outline of the hair, using AI image processing algorithms.

[0761] Emotion analysis

[0762] The server runs an emotion engine that analyzes the uploaded facial photo data and customer input data to identify the user's emotions, which can then be classified as positive or negative.

[0763] Generate hairstyle and hair color candidates

[0764] The user inputs their desired hairstyle and hair color, which is then received by the server. The server then analyzes this data using NLP (natural language processing) and generates multiple candidates using a generative AI model. Based on the analysis results of the emotion engine, the server also adjusts the candidates to suit the user's emotional state.

[0765] 3D model compositing and display

[0766] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model. The generated 3D model is sent to the client's and stylist's devices using 3D modeling software such as Blender or Unity. The 3D model received on the device can be rotated and zoomed in and out, allowing the client and stylist to view it together.

[0767] Specific examples

[0768] Customer A Scenario

[0769] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0770] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0771] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[0772] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0773] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[0774] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0775] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[0776] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[0777] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[0778] Prompt Sentence Examples

[0779] "Based on a photo of the customer's face, please generate a 3D model with a short bob hairstyle and reddish hair color. Also, please display variations of the hairstyle that emphasize the relaxed look while the user is in a relaxed state."

[0780] This allows customers and hairdressers to have a specific and detailed shared image of the hairstyle they want, allowing for smooth counseling. Furthermore, emotion analysis can be used to provide optimal suggestions to customers, which is expected to improve customer satisfaction.

[0781] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0782] Program processing flow

[0783] Step 1: Upload customer photos

[0784] The user launches the app and takes a photo of their face or selects one from the gallery. The device temporarily stores the photo data and sends it to the server.

[0785] Input: User's face photo

[0786] Output: Face photo data sent to the server

[0787] Specific behavior:

[0788] The user takes a photo of their face using the in-app camera, or selects an existing photo from the device's gallery.

[0789] The device temporarily saves the selected photo and sends it to the server using an HTTP POST request.

[0790] Step 2: Image preprocessing on the server

[0791] The server analyzes the received facial photo data, detects the face, removes the background, and extracts the outline of the hair.

[0792] Input: User's face photo data

[0793] Output: Data after face detection, background removal, and hair contour extraction

[0794] Specific behavior:

[0795] The image data received by the server is processed using an AI image processing algorithm (OpenCV or TensorFlow).

[0796] Detects the face and removes it by masking the background.

[0797] Identify and extract the hair contours.

[0798] Step 3: Sentiment Analysis

[0799] The server uses an emotion engine to analyze the uploaded facial photo and customer input data to identify the user's emotional state.

[0800] Input: Face photo data and customer input data

[0801] Output: Analysis of the user's emotional state

[0802] Specific behavior:

[0803] The server calls the emotion engine's API (e.g., emotion recognition API) and passes the data to be analyzed as input.

[0804] The emotion engine determines emotions such as smile or anger and returns the results to the server.

[0805] Step 4: Enter your hairstyle and hair color preferences

[0806] Users input their desired hairstyle and hair color within the app, and the device sends this data to the server.

[0807] Input: User's desired hairstyle and hair color instructions

[0808] Output: Instruction data sent to the server

[0809] Specific behavior:

[0810] The user enters their desired style, such as "short bob" or "reddish," into a text box within the app.

[0811] The device sends this input data to the server via an HTTP POST request.

[0812] Step 5: Generate hairstyle and hair color candidates

[0813] The server uses NLP (natural language processing) to analyze the user's request, then runs a generative AI model to generate multiple hairstyle and hair color options, and makes adjustments based on the results of sentiment analysis.

[0814] Input: User instruction data and sentiment analysis results

[0815] Output: Candidate data for hairstyle and hair color

[0816] Specific behavior:

[0817] The server uses an NLP model (e.g., a GPT model) to analyze the user's wishes.

[0818] The analysis results are input into a generative AI model to generate multiple hairstyle and hair color options.

[0819] Adjust candidates based on the results of sentiment analysis.

[0820] Step 6: Composite and send your 3D model

[0821] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[0822] Input: Face photo data, hairstyle and hair color candidate data

[0823] Output: The generated 3D model, and the client and hairdresser devices that receive it.

[0824] Specific behavior:

[0825] The server uses 3D modeling software such as Blender or Unity to integrate each piece of data.

[0826] The generated 3D model is sent to the customer's device and the hairdresser's device via an HTTP POST request.

[0827] Step 7: View and check the 3D model

[0828] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0829] Input: Received 3D model

[0830] Output: A displayed 3D model

[0831] Specific behavior:

[0832] The device launches a 3D viewer, reads the received model data, and displays it.

[0833] Users can use touch to rotate the 3D model and zoom to view details.

[0834] Step 8: Enter and Regenerate Correction Requests

[0835] The user or hairdresser checks the 3D model and inputs any additional requests or corrections. The device sends this data to the server, which then generates a new 3D model and sends it to the device.

[0836] Input: Additional requests and correction instructions

[0837] Output: Updated 3D model

[0838] Specific behavior:

[0839] Users input requests for corrections, such as "I would like the hair color to be a little lighter," into the app.

[0840] The device sends this correction data to the server via an HTTP POST request.

[0841] The server processes the data again, generates a new 3D model, and sends it to the device.

[0842] (Application example 2)

[0843] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0844] Conventional virtual try-on systems simply display the clothes and accessories selected by the user as they are, and are unable to improve the user's emotional state or the quality of the virtual experience in real time. This makes it difficult to provide optimal suggestions for the user, resulting in issues with customer satisfaction. Furthermore, it is time-consuming to combine facial photo data with candidate clothes and accessories, and it is difficult to quickly regenerate them based on requested modifications.

[0845] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing the background, and extracting the outline of the appearance; means for receiving instruction data specifying the appearance and color desired by the customer; candidate generation means for generating multiple appearance and color candidates based on the instruction data; synthesis means for synthesizing the appearance and color candidates generated by the candidate generation means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and an operation terminal; display means for displaying the 3D model; and means for analyzing the emotional state and adjusting the generated appearance and color candidates. This enables a system that provides an optimal virtual try-on experience based on user emotional analysis, improves customer satisfaction, and enables quick and easy modifications.

[0846] A "customer terminal" is a digital device operated by the customer, and is used to take a facial photograph, input instruction data, check 3D models, etc.

[0847] An "operation terminal" is a digital device operated by an operator, which is used to check the 3D model together with the customer and input any correction requests.

[0848] The "image processing means" is a collection of algorithms and software that detects facial features from received facial photograph data, removes background, and extracts the contours of the face.

[0849] "Instruction data" refers to data for inputting specific requests such as the appearance and color desired by the customer.

[0850] A "candidate generator" is a collection of algorithms and software that generates multiple appearance and color candidates based on instruction data.

[0851] "Synthesis means" refers to technology or software that integrates the generated appearance and color candidates with facial photograph data to generate a 3D model.

[0852] "Communication means" refers to the network communication technology and software used to transmit the generated 3D model to the customer terminal and the operation terminal.

[0853] "Display means" refers to a display device or software that allows a user or operator to visually check the received 3D model.

[0854] "Emotional state" is a mental health condition that analyzes a user's emotions and moods and monitors them in real time.

[0855] The "emotion analysis means" is a collection of technologies and algorithms for analyzing the user's emotional state based on facial photo data and input data, and adjusting the appearance and color candidates.

[0856] The present invention is a system for improving the fitting experience in a virtual fashion shop. The main components include a customer terminal, an operation terminal, a server, and a sentiment analysis engine. Each component is described below.

[0857] Customer terminal

[0858] The customer device refers to a smartphone or head-mounted display (HMD) operated by the customer. This device is used by the customer to take or upload a photo of their face and input their desired appearance and color. The customer device has the function of temporarily storing the face photo data and sending it to a server.

[0859] Operation terminal

[0860] The operation terminal is a tablet or PC operated by an operator. On this terminal, the operator can check the 3D model together with the customer and input any correction requests. The operation terminal also has the function of displaying the 3D model received from the server and inputting corrections as needed.

[0861] server

[0862] The server processes the received facial photo data, removing the background and extracting the contours of the appearance. The server uses image processing libraries such as OpenCV and Dlib. The server also uses natural language processing libraries (e.g., spaCy, NLTK) to analyze the customer's instruction data and input it into an AI generation model. The generated appearance and color candidates are integrated into a 3D model and sent to the customer's device and the operating device.

[0863] Sentiment Analysis Engine

[0864] The emotion analysis engine analyzes the user's emotional state based on their photo data and input data, using cloud-based emotion analysis services such as Amazon Rekognition and Microsoft Azure Face API, and then adjusts the optimal appearance and color options based on the analysis results.

[0865] Specific examples of processing

[0866] For example, if User B inputs that they want a "blue dress," the server analyzes this data and generates multiple blue dress candidates. If the user also requests that the color be "more vibrant," a new 3D model will be provided.

[0867] Prompt Sentence Examples

[0868] A user types: "I want to try on a blue dress."

[0869] The desired outfit type is "dress" and the color is "blue." Sentiment analysis indicates that the user is happy. Please present a more vibrant blue color.

[0870] The system of the present invention provides an optimal virtual try-on experience based on user sentiment analysis, improving customer satisfaction and enabling quick and easy modifications.

[0871] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0872] Step 1:

[0873] The user uses the customer's device to take a photo of their face or select one from a gallery, and then enters the photo data into the application. This photo data is temporarily saved in the customer's device's local storage. The device then sends this photo data to the server. The specific operation of sending the photo data involves communication using an HTTP request.

[0874] Step 2:

[0875] The server receives the facial photo data sent. Using OpenCV and Dlib, the server detects the face from the facial photo data, removes the background, and extracts the outline of the face. Specifically, the server converts the image to grayscale, detects facial landmarks, and then cuts out only the necessary parts. The input is the facial photo data sent from the customer's device, and the output is image data from which the face has been detected, the background removed, and the outline of the face extracted.

[0876] Step 3:

[0877] The user inputs the desired appearance and color in natural language using a customer terminal. This instruction data is temporarily stored in the local storage of the customer terminal and then sent to the server. The input is text data in natural language entered by the user, and the output is the instruction data sent to the server.

[0878] Step 4:

[0879] The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the instruction data it receives. The server parses the instruction data and identifies the desired appearance and color. Specific operations include tokenizing the text and extracting specific attributes (e.g., color, style). The input is the instruction data sent from the client device, and the output is the parsed desired appearance and color information.

[0880] Step 5:

[0881] The server inputs data into an AI generative model based on the analyzed appearance and color information. The generative AI model generates multiple appearance and color candidates. Specifically, it inputs the tensor data corresponding to the model and receives the generated candidate data. The input is the analyzed appearance and color information, and the output is the generated appearance and color candidates.

[0882] Step 6:

[0883] The server synthesizes the generated appearance and color candidates with the facial photo data to generate a 3D model. Specifically, it uses 3D rendering technology to integrate the photo data and candidate data. The input is the facial photo data and the generated candidate data, and the output is a 3D model.

[0884] Step 7:

[0885] The server sends the generated 3D model to the customer terminal and the operation terminal. The customer terminal and the operation terminal then receive and display the 3D model. Specifically, the transmission is performed via an HTTP request, and the received 3D model is displayed on the terminal's display. The input is the 3D model, and the output is the 3D model displayed on the terminal's display.

[0886] Step 8:

[0887] The user and operator check the 3D model and input any correction requests as necessary. Once the correction requests are entered, they are sent to the server from the customer terminal or the operation terminal. The input is text data of the correction requests, and the output is correction instruction data that is sent to the server.

[0888] Step 9:

[0889] The server regenerates the 3D model based on the correction request and resends the updated 3D model. Specifically, it generates new appearance and color candidates based on the correction instruction data and recomposes them. The input is the correction instruction data, and the output is the updated 3D model.

[0890] Step 10:

[0891] The customer terminal and the operation terminal display the received 3D model again for final confirmation. With final consent, the treatment (or purchase) begins. Specifically, the 3D model is displayed on the display again and the user's consent is sought. The input is the updated 3D model, and the output is the final confirmed 3D model.

[0892] In this way, a system is realized that provides an optimal virtual try-on experience based on user sentiment analysis, improves customer satisfaction, and allows for quick and easy adjustments.

[0893] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0894] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0895] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0896] [Third embodiment]

[0897] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0898] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0899] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0900] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0901] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0902] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0903] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0904] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0905] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0906] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0907] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0908] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0909] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating 3D models that are combined with customer photographs. This invention enables hairdressers and customers to accurately share specific images, improving customer satisfaction with the results of treatments.

[0910] System configuration

[0911] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a device such as a smartphone or tablet. The hairdresser terminal is also a digital device, and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0912] Program processing

[0913] Customer photo upload

[0914] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[0915] Image preprocessing on the server

[0916] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[0917] Inputting customer requests

[0918] The user (customer) inputs their desired hairstyle and hair color using natural language within the app, and the device sends this instruction data to the server.

[0919] Generate hairstyle and hair color candidates

[0920] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options, including analyzing the user's request using natural language processing (NLP) to generate 3D models of the options.

[0921] 3D model composition and transmission

[0922] The server combines the generated hairstyle and hair color candidates with the customer's facial photo data to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[0923] Display and check the 3D model

[0924] The device then displays the received 3D model for the user and hairdresser to review, allowing customers to see in detail which hairstyle and hair color will suit their face before the treatment.

[0925] Entering and regenerating correction requests

[0926] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I want the hair color to be a little lighter" can be included. The device sends these correction requests to the server, which then generates and updates the 3D model again.

[0927] Specific examples

[0928] Customer A Scenario

[0929] 1. The user (Customer A) opens the app and uploads a photo of their face.

[0930] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[0931] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[0932] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[0933] 5. The generated 3D model is sent to the customer's device and the hairdresser's device.

[0934] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[0935] 7. The server receives the correction request and sends the regenerated 3D model to the device.

[0936] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[0937] This system allows customer A and the hairdresser to have a smooth consultation based on a specific and shared image.

[0938] The processing flow will be explained below.

[0939] Step 1:

[0940] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and launches the camera to take a photo or selects an existing photo from the gallery.

[0941] Step 2:

[0942] The device temporarily stores the selected or taken photo, optimizes the image format and resolution, and sends the image data to the server.

[0943] Step 3:

[0944] The server receives the uploaded photo data and associates it with the designated customer account.

[0945] Step 4:

[0946] The server uses AI image processing models to detect faces in the received photos, remove backgrounds, and extract hair contours, providing the basis for changing hairstyles and hair colors.

[0947] Step 5:

[0948] The user inputs the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[0949] Step 6:

[0950] The server analyzes the user's request using natural language processing (NLP) and passes it as input to an AI generation model, which then generates candidate hairstyles and hair colors.

[0951] Step 7:

[0952] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[0953] Step 8:

[0954] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[0955] Step 9:

[0956] At the salon, the hairdresser and the user check the 3D model and input specific requests for corrections, such as making the hair color a little lighter. The requests are then sent from the device to the server.

[0957] Step 10:

[0958] The server receives the new request, updates the 3D model using the AI ​​model again, and sends the revised 3D model back to the client device.

[0959] Step 11:

[0960] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[0961] Example 1

[0962] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0963] Traditionally, consultations about hairstyles and hair colors at beauty salons were conducted by the customer explaining the details verbally, and the hairdresser then performed the treatment based on that. This method could lead to dissatisfaction with the finished product, as the customer's intended hairstyle or hair color was not accurately conveyed. Furthermore, there was a lack of a way to share a specific image before the treatment, making it difficult to improve customer satisfaction.

[0964] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0965] In this invention, the server includes means for receiving facial photograph data provided by a customer, image processing means for detecting facial features from the facial photograph based on the facial photograph data, removing the background, and extracting the hair outline, means for receiving instruction data specifying the hairstyle and hair color desired by the customer, means for generating a plurality of hairstyle and hair color candidates, means for synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photograph data to generate a three-dimensional model, means for transmitting the three-dimensional model generated by the synthesis means to a customer terminal and a hairdresser terminal, means for the customer or hairdresser to input correction requests, and means for regenerating a corrected three-dimensional model based on the correction requests. This enables smooth counseling between the customer and the hairdresser based on a specific and shared image, and allows treatment that accurately reflects the customer's requests.

[0966] "Facial photo data provided by the customer" refers to image data of the customer's own face that has been photographed or selected by the customer using a device such as a smartphone or tablet.

[0967] "Image processing means for detecting facial areas, removing background, and extracting hair contours" is a function that uses a facial recognition algorithm within the server to identify facial areas, remove background, and clarify the outline of hair.

[0968] "Instruction data specifying the hairstyle and hair color desired by the customer" is information in which the customer selects the desired hairstyle and hair color through an input interface using natural language or from pre-prepared options.

[0969] The "means for generating multiple hairstyle and hair color candidates" is a function that uses a generative AI model in the server to create various hairstyle and hair color variations based on customer instruction data.

[0970] The "means for generating a three-dimensional model by combining the facial photograph data" is a function that combines the hairstyle and hair color data generated by the server with the customer's facial photograph data to create a three-dimensional hairstyle model that fits the customer's face.

[0971] The "communication means for transmitting to the customer terminal and the hairdresser terminal" is a function for transmitting the three-dimensional model generated by the server to the terminals of the customer and the hairdresser via data communication.

[0972] The "means for a customer or a hairdresser to input correction requests" is an interface that allows a customer or a hairdresser to input desired changes or corrections to the three-dimensional model.

[0973] The "means for regenerating a modified three-dimensional model" is a function in which the server receives a request for modification from a customer or a hairdresser and updates and regenerates the three-dimensional model.

[0974] "Display means" refers to a display or screen on which the customer and hairdresser can view the three-dimensional model.

[0975] The "operation means" is an interface for rotating and zooming in and out of the three-dimensional model.

[0976] This invention is a digital hair catalog system developed to improve customer satisfaction at beauty salons, and is a technology that generates a 3D model that is combined with a customer's facial photograph. This allows the hairdresser and customer to accurately share a specific image, improving customer satisfaction with the results of the treatment.

[0977] System configuration

[0978] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a digital device such as a smartphone or tablet. The hairdresser terminal is also a digital device and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[0979] 1. Customer terminal

[0980] Customer terminals can be any commercially available digital device such as a smartphone or tablet.

[0981] The application has the ability to take photos and select photos from the gallery.

[0982] 2. Server

[0983] The server receives the photo data, processes the images, and generates a three-dimensional model.

[0984] The server can use common cloud services and uses AI image processing algorithms (e.g., OpenCV and TensorFlow) internally.

[0985] 3. Hairdresser's terminal

[0986] The hairdresser terminal uses the same digital device as the customer terminal.

[0987] A dedicated application is installed that allows hairdressers to view the 3D model and communicate with customers.

[0988] Process Details

[0989] 1. Upload customer photos

[0990] Users launch the app using their smartphone or tablet and take a photo of their face or select one from their gallery.

[0991] The device temporarily stores the photo data and sends it to the server.

[0992] 2. Image preprocessing on the server

[0993] The server receives the transmitted photo data and uses its internal AI image processing algorithm to detect the face area.

[0994] The background is removed and the hair outline is extracted, generating reference data for changing the hairstyle and hair color.

[0995] 3. Input customer requests

[0996] Users input their desired hairstyle and hair color using natural language within the app.

[0997] The terminal transmits this instruction data to the server.

[0998] 4. Generating hairstyle and hair color candidates

[0999] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options.

[1000] The server uses natural language processing (NLP) to analyze the user's request and generate candidate 3D models.

[1001] 5. Synthesis and transmission of 3D model

[1002] The server combines the generated hairstyle and hair color candidates with the customer's facial photograph data to generate a three-dimensional model.

[1003] The server transmits this three-dimensional model to the customer terminal and the hairdresser terminal.

[1004] 6. Display and check the 3D model

[1005] The terminal displays the received three-dimensional model, and the user and hairdresser confirm it.

[1006] This allows customers to check specifically which hairstyle and hair color suits their face before the treatment.

[1007] 7. Enter and regenerate correction requests

[1008] The user or hairdresser checks the three-dimensional model and inputs correction requests as necessary.

[1009] For example, this includes specific instructions such as "I want my hair color to be lighter."

[1010] The terminal sends this correction request to the server, which then generates and updates the 3D model again.

[1011] Specific examples

[1012] Customer A Scenario

[1013] 1. The user (Customer A) opens the app and uploads a photo of their face. For example, they take a photo of their face using the camera on their smartphone.

[1014] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1015] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1016] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[1017] 5. The generated 3D model is sent to the customer terminal and the hairdresser terminal.

[1018] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[1019] 7. The server receives the modification request and sends the regenerated 3D model to the terminal.

[1020] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[1021] Prompt Sentence Examples

[1022] "Upload a photo of your face and enter your desired hairstyle and hair color. Example: 'Short bob', 'Reddish'"

[1023] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1024] Step 1:

[1025] Customer photo upload

[1026] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The user selects a photo within the app, and the app temporarily saves the photo data. The user's device then sends the photo data to the server. The input data is the photo of their face, and the output is the image data received by the server. Specifically, the user takes a photo of their face using the camera function on their smartphone and presses the "send" button on the app.

[1027] Step 2:

[1028] Image preprocessing on the server

[1029] The server receives the facial photo data sent by the user and temporarily stores it. Next, it automatically detects the face using an AI image processing algorithm (e.g., OpenCV or TensorFlow) inside the server. It also performs processing to remove the background and extract the outline of the hair. The input data is the facial photo data, and the output is image data with the background removed and the outline of the hair extracted. Specifically, the AI ​​algorithm identifies the facial area and performs processing to make the background in that area transparent.

[1030] Step 3:

[1031] Inputting customer requests

[1032] Within the app, the user uses natural language to input their desired hairstyle and hair color into a text box. The user enters, for example, "short bob" or "reddish," and the device sends this text data to the server. The input data is the user's natural language request, and the output is the instruction data received by the server. Specifically, the user enters the desired hairstyle and hair color into the app's input fields and presses the "send" button.

[1033] Step 4:

[1034] Generate hairstyle and hair color candidates

[1035] The server receives the request data sent by the user and analyzes it using an AI model (e.g., a natural language processing engine). The server generates multiple hairstyle and hair color candidates based on the analysis results. The input data is the request data, and the output is the generated hairstyle and hair color candidate data. Specifically, the AI ​​model analyzes requests such as "short bob" or "reddish" and creates appropriate hairstyle and hair color variations based on that.

[1036] Step 5:

[1037] 3D model synthesis and transmission

[1038] The server combines the generated hairstyle and hair color candidates with facial photo data to generate a 3D model. The server then sends the combined 3D model to the customer's device and the hairdresser's device. The input data is hairstyle and hair color candidate data and facial photo data, and the output is a 3D model. Specifically, the server uses 3D modeling software such as Blender to combine the facial and hairstyle data to generate a 3D model, which is then sent to the device.

[1039] Step 6:

[1040] Display and check the 3D model

[1041] The device receives and displays the 3D model data sent from the server. By checking the 3D model, the user and hairdresser can get a concrete idea of ​​the hairstyle and hair color that will suit the customer's face before the treatment. The input data is the 3D model data, and the output is the displayed 3D model. Specifically, the smartphone or tablet app displays the 3D model using a view function that allows the user to rotate, zoom in, and out.

[1042] Step 7:

[1043] Entering and regenerating correction requests

[1044] The user or hairdresser checks the 3D model and, if necessary, inputs any correction requests into the app. For example, specific instructions such as "I want my hair color lighter" can be included. The device sends this correction request data to the server, which then generates and updates the 3D model again. The input data is the correction request data, and the output is the corrected 3D model data. Specifically, the user or hairdresser enters the correction request into the app's input field and presses the "Send" button.

[1045] The above are the specific processing steps of the system that generates a three-dimensional hairstyle model using a customer's facial photograph.

[1046] (Application example 1)

[1047] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] Conventional methods for proposing hairstyles at beauty salons make it difficult for customers and hairdressers to accurately share specific images, which can lead to a decrease in customer satisfaction after the treatment. There was also a need for a method that allows customers to try on new hairstyles and hair colors without leaving their homes. Furthermore, there were issues with usability and accuracy regarding systems for trying on hairstyles in virtual environments.

[1049] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1050] In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; synthesis means for generating a 3D model by synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photo data; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and a therapist terminal; display means for displaying the 3D model; a generative AI model that analyzes natural language to understand the customer's request and generates and modifies the 3D model based on the request; and prompt generation means for inputting instruction data to be analyzed into the generative AI model. This allows customers to try new hairstyles and hair colors without leaving their homes and share specific images before the treatment, thereby improving customer satisfaction and improving the usability and accuracy of hairstyle try-on in a virtual environment.

[1051] "Customer terminal" refers to a digital device operated by a customer, including a smartphone, tablet, laptop, etc.

[1052] "Receiving means" refers to a means for receiving data, and in this case refers to a mechanism for receiving images, instruction data, etc. via Internet communication.

[1053] The "image processing means" refers to a means for performing processing to detect specific parts from received image data or remove backgrounds, and specifically includes algorithms for face detection and hair contour extraction.

[1054] "Instruction data" refers to data indicating information such as the hairstyle and hair color desired by the customer, and is input through the application.

[1055] The "candidate generation means" is a means for generating multiple hairstyle and hair color candidates based on instruction data, and is realized here by a generative AI model.

[1056] The "combining means" is a means for combining the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[1057] "Communication means" refers to the means for transmitting the generated 3D model and other data to the client terminal and the therapist terminal, primarily via the Internet.

[1058] The "display means" is a means for displaying the received 3D model on the terminal, and includes a graphical user interface (GUI).

[1059] A "professor terminal" is a digital device operated by the practitioner (hairdresser), and refers mainly to a computer or tablet used within the salon.

[1060] "Generative AI model" refers to an artificial intelligence (AI) model that analyzes natural language and generates and modifies 3D models based on customer requests.

[1061] The "prompt sentence generation means" is a means for generating sentences for inputting instruction data to be analyzed into the generative AI model.

[1062] The present invention relates to a digital hair catalog system for improving customer satisfaction in a virtual hair salon system. This system is comprised of a customer terminal, a therapist terminal, and a server, and transmits and receives data via the Internet.

[1063] Role of customer terminals

[1064] The customer terminal is a digital device such as a smartphone or tablet. The customer first launches the application and takes a photo of their face or selects one from the gallery. The photo is temporarily saved on the customer terminal and then sent to a server via the Internet.

[1065] Processing on the server

[1066] The server processes the data in the following steps:

[1067] 1. The server receives the facial photo data sent from the customer's device. The software used is Python's PIL (Python Imaging Library) and Requests library.

[1068] 2. Based on the received facial photo data, the system detects the face, removes the background, and extracts the hair outline. The OpenCV library is used here.

[1069] 3. Next, the customer receives instruction data specifying their desired hairstyle and hair color. This instruction data is entered by the customer in natural language within the app, and the server analyzes it using Hugging Face's Transformers library and T5 model.

[1070] 4. Generate multiple hairstyle and color options based on natural language analysis. This process uses a generative AI model to accurately reflect the customer's preferences in the 3D model.

[1071] 5. The generated hairstyle and hair color candidates are combined with facial photo data to generate a 3D model. The 3D model is created using the Open3D library.

[1072] 6. Finally, the generated 3D model can be sent to the client's device and the practitioner's device and displayed.

[1073] Role of the therapist terminal

[1074] The hairdresser's terminal is a digital device operated by the hairdresser. The hairdresser checks the 3D model sent from the server and selects the most suitable hairstyle and hair color together with the customer. If necessary, they can resubmit any correction requests to the server and generate and modify the 3D model again. This allows the customer and hairdresser to share a specific image.

[1075] Examples and prompts

[1076] For example, if a customer requests a short bob with red hair, the text is parsed with the prompt, "Please provide generated hairstyle and hair color options: Short bob with red hair." This prompt allows the generative AI model to generate multiple options based on the customer's request and synthesize them into a 3D model.

[1077] Example prompt sentence:

[1078] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[1079] This system allows customers to try on new hairstyles and hair colors without leaving their homes, and by sharing a concrete image of the desired look before the treatment, customer satisfaction is increased. In addition, the high-precision processing of natural language analysis and 3D model generation on the server improves the usability and accuracy of trying on hairstyles in a virtual environment.

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The selected photo is temporarily saved on the customer's device and then sent to the server via the Internet. In this step, the customer's photo is used as input data, and the photo data sent to the server is obtained as output data.

[1083] Step 2:

[1084] The server internally processes the received facial photo data. First, it uses the OpenCV library to detect the face, remove the background, and extract the hair outline. At this stage, the facial photo data is used as input data, and image data with the background removed and the hair outline extracted is obtained as output data.

[1085] Step 3:

[1086] The user inputs the desired hairstyle and hair color in the app using natural language. This instruction data is sent from the customer terminal to the server. In this step, instruction text in natural language is used as input data, and the instruction data sent to the server is obtained as output data.

[1087] Step 4:

[1088] The server analyzes the received instruction data. Using Hugging Face's Transformers library and the T5 model, it generates hairstyle and hair color candidates based on natural language. The instruction data is used as input data, and multiple hairstyle and hair color candidates are generated as output data.

[1089] Step 5:

[1090] The generative AI model generates a 3D model by combining the generated hairstyle and hair color candidates with facial photo data. The 3D model is created using the Open3D library. In this step, hairstyle and hair color candidates and facial photo data with the background removed are used as input data, and a 3D model is generated as output data.

[1091] Step 6:

[1092] The server sends the generated 3D model to the client's device and the therapist's device. The client's device and the therapist's device display the 3D model, allowing the client and therapist to share and review it. In this step, the generated 3D model is used as input data, and the transmitted 3D model is obtained as output data.

[1093] Step 7:

[1094] The client and the therapist check the 3D model on the therapist's terminal and input any correction requests as necessary. The correction requests are sent from the therapist's terminal to the server. At this stage, instruction data for corrections is used as input data, and the correction requests are sent to the server as output data.

[1095] Step 8:

[1096] The server generates and modifies the 3D model again based on the modification request, and sends the results back to the client's device and the therapist's device. The modification request is used as input data, and the modified 3D model is generated as output data.

[1097] Example prompt sentence:

[1098] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[1099] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1100] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating a 3D model that is combined with a customer's facial photograph, and technology for recognizing the user's emotions and adjusting suggested hairstyles and hair colors. This allows the hairdresser and customer to accurately share a specific image, further improving customer satisfaction with the results of the treatment.

[1101] System configuration

[1102] This system consists of a customer terminal, a server, a hairdresser terminal, and an emotion engine. The customer terminal is a digital device such as a smartphone or tablet, and the hairdresser terminal is also a digital device. The server is a central system that processes data and communicates between these terminals. The emotion engine is a combination of software and hardware for recognizing customer emotions.

[1103] Program processing

[1104] Customer photo upload

[1105] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[1106] Image preprocessing on the server

[1107] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[1108] Emotion analysis with emotion engine

[1109] The user's emotion engine analyzes the uploaded facial photo data and input data from the customer's device in real time to monitor the user's emotional state. For example, if the customer is smiling, it is analyzed as having a positive emotion.

[1110] Input hairstyle and hair color requests

[1111] The user inputs their desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[1112] Generate hairstyle and hair color candidates

[1113] The server analyzes the user's request (instruction data) using natural language processing (NLP) and passes it as input to the AI ​​generation model. The AI ​​generates candidate hairstyles and hair colors. Furthermore, based on the analysis results of the emotion engine, the candidate hairstyles and hair colors are adjusted to suit the user's emotional state.

[1114] 3D model composition and transmission

[1115] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[1116] Display and check the 3D model

[1117] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[1118] Entering and regenerating correction requests

[1119] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I'd like the hair color to be a little lighter" are included. The device sends these correction requests to the server. The server then generates and updates the 3D model again, and sends the newly generated 3D model to the device.

[1120] Specific examples

[1121] Customer A Scenario

[1122] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1123] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1124] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[1125] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1126] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[1127] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1128] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[1129] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[1130] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[1131] This system enables more personalized hairstyle suggestions that take into account the user's emotional state, enabling smooth counseling based on a specific and shared image between customer A and the hairdresser.

[1132] The processing flow will be explained below.

[1133] Step 1:

[1134] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and takes a photo of themselves or selects an existing photo from the gallery.

[1135] Step 2:

[1136] The device temporarily stores the selected or taken photo data, optimizes the image format and resolution, and sends it to the server.

[1137] Step 3:

[1138] The server receives the uploaded photo data and associates it with the designated customer account. The server then begins processing the image data and uses AI image processing models to detect facial features.

[1139] Step 4:

[1140] After detecting the face, the server performs image processing to remove the background and extract the hair outline, generating the basic data for changing the hairstyle and hair color.

[1141] Step 5:

[1142] The device receives a notification from the server that face detection and image processing are complete, and prompts the user to perform the next step: The user enters the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app.

[1143] Step 6:

[1144] The device sends instruction data to the server, which analyzes the user's request using natural language processing (NLP) and inputs the instruction data into an AI-generated model.

[1145] Step 7:

[1146] The server uses an AI generative model to generate candidates for a specific hairstyle and hair color. Multiple hairstyle and hair color candidates are generated.

[1147] Step 8:

[1148] The server activates the emotion engine and analyzes the uploaded facial photo and instruction data to recognize the user's emotional state, and adjusts the hairstyle and hair color suggestions based on the recognized emotional state.

[1149] Step 9:

[1150] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[1151] Step 10:

[1152] The device displays the received 3D model, allowing the user to rotate it and zoom in and out to check the details. The hairdresser also checks the same model at the salon.

[1153] Step 11:

[1154] The user and hairdresser check the 3D model and input requests for corrections to specific parts (for example, "I want the hair color to be a little lighter"). The device then sends these requests to the server.

[1155] Step 12:

[1156] The server receives the new request and updates the 3D model using the AI ​​model again. The revised 3D model is then sent back to the customer's device and the hairdresser's device.

[1157] Step 13:

[1158] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[1159] Specific examples

[1160] Customer A Scenario

[1161] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1162] 2. The device sends the photo data to the server.

[1163] 3. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1164] 4. The server notifies the terminal that image preprocessing is complete.

[1165] 5. The user (customer A) inputs that he / she desires a "short bob" and "reddish" hairstyle and hair color.

[1166] 6. The server analyzes the request and generates multiple short bob and reddish hair color options.

[1167] 7. The server uses the emotion engine to analyze the user's emotional state and adjusts the candidates suitable for a relaxed state.

[1168] 8. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1169] 9. The user (Customer A) and the hairdresser check the 3D model at the beauty salon.

[1170] 10. The user inputs a request for correction, such as "Make the hair color a little lighter."

[1171] 11. The server receives the correction request and sends the regenerated 3D model to the device.

[1172] 12. The user makes a final confirmation and, once they agree, the treatment begins.

[1173] This system enables personalized hairstyle suggestions that take into account the user's emotional state, realizing smooth counseling based on a specific and shared image between the customer and hairdresser.

[1174] Example 2

[1175] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1176] Conventional hair catalog systems used in hair salons have made it difficult to improve customer satisfaction. In particular, the image of hairstyle shared between the customer and the hairdresser is vague, resulting in customers' expectations often not being met. In addition, it is difficult to reflect the customer's feelings and requests in real time, making it impossible to make optimal suggestions for each individual customer. A system that solves these issues was needed.

[1177] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; emotion analysis means for analyzing the customer's emotional data using a generative AI model and making adjustments based on the customer's emotions; synthesis means for synthesizing the hairstyle and hair color candidates adjusted by the candidate generation means and emotion analysis means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to the customer terminal and the hairdresser terminal; and display means for displaying the 3D model. This allows the customer and the hairdresser to have a specific and detailed shared image of the hairstyle and hair color during a smooth consultation. Furthermore, emotion analysis allows for optimal suggestions to be made to the customer, which is expected to improve customer satisfaction.

[1178] "Means for receiving facial photo data provided by customers" refers to a system or protocol that allows the server to receive facial photo data taken or selected by customers using a device such as a smartphone or tablet.

[1179] The "image processing means" refers to algorithms or software that detects the face, removes the background, and extracts the outline of the hair from the facial photograph data received by the server.

[1180] The "means for receiving instruction data" refers to a system or protocol that allows a customer to input their desired hairstyle and hair color within the app and for the server to receive that input data.

[1181] The "candidate generation means" refers to a generative AI model or algorithm for generating multiple hairstyle and hair color candidates based on the received instruction data.

[1182] An "emotion analysis method" is an algorithm or system that uses a generative AI model to analyze a customer's facial photograph and input data, determine the customer's emotions, and make adjustments based on the results.

[1183] The "combining means" refers to modeling software or algorithms that combine the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[1184] "Communication means" refers to the network infrastructure and protocols used to transmit the 3D model generated by the server to the customer terminal and the hairdresser terminal.

[1185] "Display means" refers to the interface and software for displaying the received 3D model on the customer terminal and the hairdresser terminal.

[1186] The program for this system is designed to improve customer satisfaction at beauty salons. Specifically, it provides technology that generates a 3D model based on a customer's facial photograph, analyzes the user's emotions, and adjusts suggested hairstyles and hair colors. Below, we will explain what hardware and software are used, and what data processing and calculations are performed.

[1187] Hardware and software used

[1188] Customer terminal

[1189] Customer terminals are primarily smartphones or tablets, which have application software installed on them and provide an interface for users to take a photo of their face and input their desired hairstyle and hair color.

[1190] Hairdresser's terminal

[1191] The hairdresser's terminal also uses a smartphone or tablet, which has application software installed on it that allows the hairdresser to view the customer's 3D model and input any correction requests.

[1192] server

[1193] The server acts as a central system, processing data sent from the customer and hairdresser devices. It implements advanced image processing algorithms and generative AI models, using libraries such as OpenCV and TensorFlow. It also uses Microsoft Azure's emotion recognition API to analyze customer emotions.

[1194] Data processing and calculation

[1195] Upload and pre-process face photos

[1196] The user takes a photo of their face on their device and sends it to the server, which then analyzes the received photo data to detect the face, remove the background, and extract the outline of the hair, using AI image processing algorithms.

[1197] Emotion analysis

[1198] The server runs an emotion engine that analyzes the uploaded facial photo data and customer input data to identify the user's emotions, which can then be classified as positive or negative.

[1199] Generate hairstyle and hair color candidates

[1200] The user inputs their desired hairstyle and hair color, which is then received by the server. The server then analyzes this data using NLP (natural language processing) and generates multiple candidates using a generative AI model. Based on the analysis results of the emotion engine, the server also adjusts the candidates to suit the user's emotional state.

[1201] 3D model compositing and display

[1202] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model. The generated 3D model is sent to the client's and stylist's devices using 3D modeling software such as Blender or Unity. The 3D model received on the device can be rotated and zoomed in and out, allowing the client and stylist to view it together.

[1203] Specific examples

[1204] Customer A Scenario

[1205] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1206] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1207] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[1208] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1209] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[1210] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1211] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[1212] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[1213] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[1214] Prompt Sentence Examples

[1215] "Based on a photo of the customer's face, please generate a 3D model with a short bob hairstyle and reddish hair color. Also, please display variations of the hairstyle that emphasize the relaxed look while the user is in a relaxed state."

[1216] This allows customers and hairdressers to have a specific and detailed shared image of the hairstyle they want, allowing for smooth counseling. Furthermore, emotion analysis can be used to provide optimal suggestions to customers, which is expected to improve customer satisfaction.

[1217] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1218] Program processing flow

[1219] Step 1: Upload customer photos

[1220] The user launches the app and takes a photo of their face or selects one from the gallery. The device temporarily stores the photo data and sends it to the server.

[1221] Input: User's face photo

[1222] Output: Face photo data sent to the server

[1223] Specific behavior:

[1224] The user takes a photo of their face using the in-app camera, or selects an existing photo from the device's gallery.

[1225] The device temporarily saves the selected photo and sends it to the server using an HTTP POST request.

[1226] Step 2: Image preprocessing on the server

[1227] The server analyzes the received facial photo data, detects the face, removes the background, and extracts the outline of the hair.

[1228] Input: User's face photo data

[1229] Output: Data after face detection, background removal, and hair contour extraction

[1230] Specific behavior:

[1231] The image data received by the server is processed using an AI image processing algorithm (OpenCV or TensorFlow).

[1232] Detects the face and removes it by masking the background.

[1233] Identify and extract the hair contours.

[1234] Step 3: Sentiment Analysis

[1235] The server uses an emotion engine to analyze the uploaded facial photo and customer input data to identify the user's emotional state.

[1236] Input: Face photo data and customer input data

[1237] Output: Analysis of the user's emotional state

[1238] Specific behavior:

[1239] The server calls the emotion engine's API (e.g., emotion recognition API) and passes the data to be analyzed as input.

[1240] The emotion engine determines emotions such as smile or anger and returns the results to the server.

[1241] Step 4: Enter your hairstyle and hair color preferences

[1242] Users input their desired hairstyle and hair color within the app, and the device sends this data to the server.

[1243] Input: User's desired hairstyle and hair color instructions

[1244] Output: Instruction data sent to the server

[1245] Specific behavior:

[1246] The user enters their desired style, such as "short bob" or "reddish," into a text box within the app.

[1247] The device sends this input data to the server via an HTTP POST request.

[1248] Step 5: Generate hairstyle and hair color candidates

[1249] The server uses NLP (natural language processing) to analyze the user's request, then runs a generative AI model to generate multiple hairstyle and hair color options, and makes adjustments based on the results of sentiment analysis.

[1250] Input: User instruction data and sentiment analysis results

[1251] Output: Candidate data for hairstyle and hair color

[1252] Specific behavior:

[1253] The server uses an NLP model (e.g., a GPT model) to analyze the user's wishes.

[1254] The analysis results are input into a generative AI model to generate multiple hairstyle and hair color options.

[1255] Adjust candidates based on the results of sentiment analysis.

[1256] Step 6: Composite and send your 3D model

[1257] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[1258] Input: Face photo data, hairstyle and hair color candidate data

[1259] Output: The generated 3D model, and the client and hairdresser devices that receive it.

[1260] Specific behavior:

[1261] The server uses 3D modeling software such as Blender or Unity to integrate each piece of data.

[1262] The generated 3D model is sent to the customer's device and the hairdresser's device via an HTTP POST request.

[1263] Step 7: View and check the 3D model

[1264] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[1265] Input: Received 3D model

[1266] Output: A displayed 3D model

[1267] Specific behavior:

[1268] The device launches a 3D viewer, reads the received model data, and displays it.

[1269] Users can use touch to rotate the 3D model and zoom to view details.

[1270] Step 8: Enter and Regenerate Correction Requests

[1271] The user or hairdresser checks the 3D model and inputs any additional requests or corrections. The device sends this data to the server, which then generates a new 3D model and sends it to the device.

[1272] Input: Additional requests and correction instructions

[1273] Output: Updated 3D model

[1274] Specific behavior:

[1275] Users input requests for corrections, such as "I would like the hair color to be a little lighter," into the app.

[1276] The device sends this correction data to the server via an HTTP POST request.

[1277] The server processes the data again, generates a new 3D model, and sends it to the device.

[1278] (Application example 2)

[1279] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1280] Conventional virtual try-on systems simply display the clothes and accessories selected by the user as they are, and are unable to improve the user's emotional state or the quality of the virtual experience in real time. This makes it difficult to provide optimal suggestions for the user, resulting in issues with customer satisfaction. Furthermore, it is time-consuming to combine facial photo data with candidate clothes and accessories, and it is difficult to quickly regenerate them based on requested modifications.

[1281] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing the background, and extracting the outline of the appearance; means for receiving instruction data specifying the appearance and color desired by the customer; candidate generation means for generating multiple appearance and color candidates based on the instruction data; synthesis means for synthesizing the appearance and color candidates generated by the candidate generation means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and an operation terminal; display means for displaying the 3D model; and means for analyzing the emotional state and adjusting the generated appearance and color candidates. This enables a system that provides an optimal virtual try-on experience based on user emotional analysis, improves customer satisfaction, and enables quick and easy modifications.

[1282] A "customer terminal" is a digital device operated by the customer, and is used to take a facial photograph, input instruction data, check 3D models, etc.

[1283] An "operation terminal" is a digital device operated by an operator, which is used to check the 3D model together with the customer and input any correction requests.

[1284] The "image processing means" is a collection of algorithms and software that detects facial features from received facial photograph data, removes background, and extracts the contours of the face.

[1285] "Instruction data" refers to data for inputting specific requests such as the appearance and color desired by the customer.

[1286] A "candidate generator" is a collection of algorithms and software that generates multiple appearance and color candidates based on instruction data.

[1287] "Synthesis means" refers to technology or software that integrates the generated appearance and color candidates with facial photograph data to generate a 3D model.

[1288] "Communication means" refers to the network communication technology and software used to transmit the generated 3D model to the customer terminal and the operation terminal.

[1289] "Display means" refers to a display device or software that allows a user or operator to visually check the received 3D model.

[1290] "Emotional state" is a mental health condition that analyzes a user's emotions and moods and monitors them in real time.

[1291] The "emotion analysis means" is a collection of technologies and algorithms for analyzing the user's emotional state based on facial photo data and input data, and adjusting the appearance and color candidates.

[1292] The present invention is a system for improving the fitting experience in a virtual fashion shop. The main components include a customer terminal, an operation terminal, a server, and a sentiment analysis engine. Each component is described below.

[1293] Customer terminal

[1294] The customer device refers to a smartphone or head-mounted display (HMD) operated by the customer. This device is used by the customer to take or upload a photo of their face and input their desired appearance and color. The customer device has the function of temporarily storing the face photo data and sending it to a server.

[1295] Operation terminal

[1296] The operation terminal is a tablet or PC operated by an operator. On this terminal, the operator can check the 3D model together with the customer and input any correction requests. The operation terminal also has the function of displaying the 3D model received from the server and inputting corrections as needed.

[1297] server

[1298] The server processes the received facial photo data, removing the background and extracting the contours of the appearance. The server uses image processing libraries such as OpenCV and Dlib. The server also uses natural language processing libraries (e.g., spaCy, NLTK) to analyze the customer's instruction data and input it into an AI generation model. The generated appearance and color candidates are integrated into a 3D model and sent to the customer's device and the operating device.

[1299] Sentiment Analysis Engine

[1300] The emotion analysis engine analyzes the user's emotional state based on their photo data and input data, using cloud-based emotion analysis services such as Amazon Rekognition and Microsoft Azure Face API, and then adjusts the optimal appearance and color options based on the analysis results.

[1301] Specific examples of processing

[1302] For example, if User B inputs that they want a "blue dress," the server analyzes this data and generates multiple blue dress candidates. If the user also requests that the color be "more vibrant," a new 3D model will be provided.

[1303] Prompt Sentence Examples

[1304] A user types: "I want to try on a blue dress."

[1305] The desired outfit type is "dress" and the color is "blue." Sentiment analysis indicates that the user is happy. Please present a more vibrant blue color.

[1306] The system of the present invention provides an optimal virtual try-on experience based on user sentiment analysis, improving customer satisfaction and enabling quick and easy modifications.

[1307] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1308] Step 1:

[1309] The user uses the customer's device to take a photo of their face or select one from a gallery, and then enters the photo data into the application. This photo data is temporarily saved in the customer's device's local storage. The device then sends this photo data to the server. The specific operation of sending the photo data involves communication using an HTTP request.

[1310] Step 2:

[1311] The server receives the facial photo data sent. Using OpenCV and Dlib, the server detects the face from the facial photo data, removes the background, and extracts the outline of the face. Specifically, the server converts the image to grayscale, detects facial landmarks, and then cuts out only the necessary parts. The input is the facial photo data sent from the customer's device, and the output is image data from which the face has been detected, the background removed, and the outline of the face extracted.

[1312] Step 3:

[1313] The user inputs the desired appearance and color in natural language using a customer terminal. This instruction data is temporarily stored in the local storage of the customer terminal and then sent to the server. The input is text data in natural language entered by the user, and the output is the instruction data sent to the server.

[1314] Step 4:

[1315] The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the instruction data it receives. The server parses the instruction data and identifies the desired appearance and color. Specific operations include tokenizing the text and extracting specific attributes (e.g., color, style). The input is the instruction data sent from the client device, and the output is the parsed desired appearance and color information.

[1316] Step 5:

[1317] The server inputs data into an AI generative model based on the analyzed appearance and color information. The generative AI model generates multiple appearance and color candidates. Specifically, it inputs the tensor data corresponding to the model and receives the generated candidate data. The input is the analyzed appearance and color information, and the output is the generated appearance and color candidates.

[1318] Step 6:

[1319] The server synthesizes the generated appearance and color candidates with the facial photo data to generate a 3D model. Specifically, it uses 3D rendering technology to integrate the photo data and candidate data. The input is the facial photo data and the generated candidate data, and the output is a 3D model.

[1320] Step 7:

[1321] The server sends the generated 3D model to the customer terminal and the operation terminal. The customer terminal and the operation terminal then receive and display the 3D model. Specifically, the transmission is performed via an HTTP request, and the received 3D model is displayed on the terminal's display. The input is the 3D model, and the output is the 3D model displayed on the terminal's display.

[1322] Step 8:

[1323] The user and operator check the 3D model and input any correction requests as necessary. Once the correction requests are entered, they are sent to the server from the customer terminal or the operation terminal. The input is text data of the correction requests, and the output is correction instruction data that is sent to the server.

[1324] Step 9:

[1325] The server regenerates the 3D model based on the correction request and resends the updated 3D model. Specifically, it generates new appearance and color candidates based on the correction instruction data and recomposes them. The input is the correction instruction data, and the output is the updated 3D model.

[1326] Step 10:

[1327] The customer terminal and the operation terminal display the received 3D model again for final confirmation. With final consent, the treatment (or purchase) begins. Specifically, the 3D model is displayed on the display again and the user's consent is sought. The input is the updated 3D model, and the output is the final confirmed 3D model.

[1328] In this way, a system is realized that provides an optimal virtual try-on experience based on user sentiment analysis, improves customer satisfaction, and allows for quick and easy adjustments.

[1329] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1330] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1331] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1332] [Fourth embodiment]

[1333] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1334] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1335] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1336] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1337] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1338] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1339] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1340] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1341] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1342] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1343] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1344] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1345] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1346] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating 3D models that are combined with customer photographs. This invention enables hairdressers and customers to accurately share specific images, improving customer satisfaction with the results of treatments.

[1347] System configuration

[1348] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a device such as a smartphone or tablet. The hairdresser terminal is also a digital device, and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[1349] Program processing

[1350] Customer photo upload

[1351] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[1352] Image preprocessing on the server

[1353] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[1354] Inputting customer requests

[1355] The user (customer) inputs their desired hairstyle and hair color using natural language within the app, and the device sends this instruction data to the server.

[1356] Generate hairstyle and hair color candidates

[1357] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options, including analyzing the user's request using natural language processing (NLP) to generate 3D models of the options.

[1358] 3D model composition and transmission

[1359] The server combines the generated hairstyle and hair color candidates with the customer's facial photo data to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[1360] Display and check the 3D model

[1361] The device then displays the received 3D model for the user and hairdresser to review, allowing customers to see in detail which hairstyle and hair color will suit their face before the treatment.

[1362] Entering and regenerating correction requests

[1363] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I want the hair color to be a little lighter" can be included. The device sends these correction requests to the server, which then generates and updates the 3D model again.

[1364] Specific examples

[1365] Customer A Scenario

[1366] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1367] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1368] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1369] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[1370] 5. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1371] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[1372] 7. The server receives the correction request and sends the regenerated 3D model to the device.

[1373] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[1374] This system allows customer A and the hairdresser to have a smooth consultation based on a specific and shared image.

[1375] The processing flow will be explained below.

[1376] Step 1:

[1377] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and launches the camera to take a photo or selects an existing photo from the gallery.

[1378] Step 2:

[1379] The device temporarily stores the selected or taken photo, optimizes the image format and resolution, and sends the image data to the server.

[1380] Step 3:

[1381] The server receives the uploaded photo data and associates it with the designated customer account.

[1382] Step 4:

[1383] The server uses AI image processing models to detect faces in the received photos, remove backgrounds, and extract hair contours, providing the basis for changing hairstyles and hair colors.

[1384] Step 5:

[1385] The user inputs the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[1386] Step 6:

[1387] The server analyzes the user's request using natural language processing (NLP) and passes it as input to an AI generation model, which then generates candidate hairstyles and hair colors.

[1388] Step 7:

[1389] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[1390] Step 8:

[1391] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[1392] Step 9:

[1393] At the salon, the hairdresser and the user check the 3D model and input specific requests for corrections, such as making the hair color a little lighter. The requests are then sent from the device to the server.

[1394] Step 10:

[1395] The server receives the new request, updates the 3D model using the AI ​​model again, and sends the revised 3D model back to the client device.

[1396] Step 11:

[1397] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[1398] Example 1

[1399] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1400] Traditionally, consultations about hairstyles and hair colors at beauty salons were conducted by the customer explaining the details verbally, and the hairdresser then performed the treatment based on that. This method could lead to dissatisfaction with the finished product, as the customer's intended hairstyle or hair color was not accurately conveyed. Furthermore, there was a lack of a way to share a specific image before the treatment, making it difficult to improve customer satisfaction.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1402] In this invention, the server includes means for receiving facial photograph data provided by a customer, image processing means for detecting facial features from the facial photograph based on the facial photograph data, removing the background, and extracting the hair outline, means for receiving instruction data specifying the hairstyle and hair color desired by the customer, means for generating a plurality of hairstyle and hair color candidates, means for synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photograph data to generate a three-dimensional model, means for transmitting the three-dimensional model generated by the synthesis means to a customer terminal and a hairdresser terminal, means for the customer or hairdresser to input correction requests, and means for regenerating a corrected three-dimensional model based on the correction requests. This enables smooth counseling between the customer and the hairdresser based on a specific and shared image, and allows treatment that accurately reflects the customer's requests.

[1403] "Facial photo data provided by the customer" refers to image data of the customer's own face that has been photographed or selected by the customer using a device such as a smartphone or tablet.

[1404] "Image processing means for detecting facial areas, removing background, and extracting hair contours" is a function that uses a facial recognition algorithm within the server to identify facial areas, remove background, and clarify the outline of hair.

[1405] "Instruction data specifying the hairstyle and hair color desired by the customer" is information in which the customer selects the desired hairstyle and hair color through an input interface using natural language or from pre-prepared options.

[1406] The "means for generating multiple hairstyle and hair color candidates" is a function that uses a generative AI model in the server to create various hairstyle and hair color variations based on customer instruction data.

[1407] The "means for generating a three-dimensional model by combining the facial photograph data" is a function that combines the hairstyle and hair color data generated by the server with the customer's facial photograph data to create a three-dimensional hairstyle model that fits the customer's face.

[1408] The "communication means for transmitting to the customer terminal and the hairdresser terminal" is a function for transmitting the three-dimensional model generated by the server to the terminals of the customer and the hairdresser via data communication.

[1409] The "means for a customer or a hairdresser to input correction requests" is an interface that allows a customer or a hairdresser to input desired changes or corrections to the three-dimensional model.

[1410] The "means for regenerating a modified three-dimensional model" is a function in which the server receives a request for modification from a customer or a hairdresser and updates and regenerates the three-dimensional model.

[1411] "Display means" refers to a display or screen on which the customer and hairdresser can view the three-dimensional model.

[1412] The "operation means" is an interface for rotating and zooming in and out of the three-dimensional model.

[1413] This invention is a digital hair catalog system developed to improve customer satisfaction at beauty salons, and is a technology that generates a 3D model that is combined with a customer's facial photograph. This allows the hairdresser and customer to accurately share a specific image, improving customer satisfaction with the results of the treatment.

[1414] System configuration

[1415] This system consists of a customer terminal, a server, and a hairdresser terminal. The customer terminal is a digital device such as a smartphone or tablet. The hairdresser terminal is also a digital device and is primarily used by hairdressers. The server plays a central role in processing data and communicating between these terminals.

[1416] 1. Customer terminal

[1417] Customer terminals can be any commercially available digital device such as a smartphone or tablet.

[1418] The application has the ability to take photos and select photos from the gallery.

[1419] 2. Server

[1420] The server receives the photo data, processes the images, and generates a three-dimensional model.

[1421] The server can use common cloud services and uses AI image processing algorithms (e.g., OpenCV and TensorFlow) internally.

[1422] 3. Hairdresser's terminal

[1423] The hairdresser terminal uses the same digital device as the customer terminal.

[1424] A dedicated application is installed that allows hairdressers to view the 3D model and communicate with customers.

[1425] Process Details

[1426] 1. Upload customer photos

[1427] Users launch the app using their smartphone or tablet and take a photo of their face or select one from their gallery.

[1428] The device temporarily stores the photo data and sends it to the server.

[1429] 2. Image preprocessing on the server

[1430] The server receives the transmitted photo data and uses its internal AI image processing algorithm to detect the face area.

[1431] The background is removed and the hair outline is extracted, generating reference data for changing the hairstyle and hair color.

[1432] 3. Input customer requests

[1433] Users input their desired hairstyle and hair color using natural language within the app.

[1434] The terminal transmits this instruction data to the server.

[1435] 4. Generating hairstyle and hair color candidates

[1436] The server receives the instruction data and uses the AI ​​model to generate multiple hairstyle and hair color options.

[1437] The server uses natural language processing (NLP) to analyze the user's request and generate candidate 3D models.

[1438] 5. Synthesis and transmission of 3D model

[1439] The server combines the generated hairstyle and hair color candidates with the customer's facial photograph data to generate a three-dimensional model.

[1440] The server transmits this three-dimensional model to the customer terminal and the hairdresser terminal.

[1441] 6. Display and check the 3D model

[1442] The terminal displays the received three-dimensional model, and the user and hairdresser confirm it.

[1443] This allows customers to check specifically which hairstyle and hair color suits their face before the treatment.

[1444] 7. Enter and regenerate correction requests

[1445] The user or hairdresser checks the three-dimensional model and inputs correction requests as necessary.

[1446] For example, this includes specific instructions such as "I want my hair color to be lighter."

[1447] The terminal sends this correction request to the server, which then generates and updates the 3D model again.

[1448] Specific examples

[1449] Customer A Scenario

[1450] 1. The user (Customer A) opens the app and uploads a photo of their face. For example, they take a photo of their face using the camera on their smartphone.

[1451] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1452] 3. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1453] 4. The server analyzes the request and generates multiple short bob and reddish hair color options.

[1454] 5. The generated 3D model is sent to the customer terminal and the hairdresser terminal.

[1455] 6. At the beauty salon, Customer A and the hairdresser check the 3D model and input a request for correction, such as "make the hair color lighter."

[1456] 7. The server receives the modification request and sends the regenerated 3D model to the terminal.

[1457] 8. Customer A makes a final confirmation and, once he or she agrees, the treatment is carried out.

[1458] Prompt Sentence Examples

[1459] "Upload a photo of your face and enter your desired hairstyle and hair color. Example: 'Short bob', 'Reddish'"

[1460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1461] Step 1:

[1462] Customer photo upload

[1463] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The user selects a photo within the app, and the app temporarily saves the photo data. The user's device then sends the photo data to the server. The input data is the photo of their face, and the output is the image data received by the server. Specifically, the user takes a photo of their face using the camera function on their smartphone and presses the "send" button on the app.

[1464] Step 2:

[1465] Image preprocessing on the server

[1466] The server receives the facial photo data sent by the user and temporarily stores it. Next, it automatically detects the face using an AI image processing algorithm (e.g., OpenCV or TensorFlow) inside the server. It also performs processing to remove the background and extract the outline of the hair. The input data is the facial photo data, and the output is image data with the background removed and the outline of the hair extracted. Specifically, the AI ​​algorithm identifies the facial area and performs processing to make the background in that area transparent.

[1467] Step 3:

[1468] Inputting customer requests

[1469] Within the app, the user uses natural language to input their desired hairstyle and hair color into a text box. The user enters, for example, "short bob" or "reddish," and the device sends this text data to the server. The input data is the user's natural language request, and the output is the instruction data received by the server. Specifically, the user enters the desired hairstyle and hair color into the app's input fields and presses the "send" button.

[1470] Step 4:

[1471] Generate hairstyle and hair color candidates

[1472] The server receives the request data sent by the user and analyzes it using an AI model (e.g., a natural language processing engine). The server generates multiple hairstyle and hair color candidates based on the analysis results. The input data is the request data, and the output is the generated hairstyle and hair color candidate data. Specifically, the AI ​​model analyzes requests such as "short bob" or "reddish" and creates appropriate hairstyle and hair color variations based on that.

[1473] Step 5:

[1474] 3D model synthesis and transmission

[1475] The server combines the generated hairstyle and hair color candidates with facial photo data to generate a 3D model. The server then sends the combined 3D model to the customer's device and the hairdresser's device. The input data is hairstyle and hair color candidate data and facial photo data, and the output is a 3D model. Specifically, the server uses 3D modeling software such as Blender to combine the facial and hairstyle data to generate a 3D model, which is then sent to the device.

[1476] Step 6:

[1477] Display and check the 3D model

[1478] The device receives and displays the 3D model data sent from the server. By checking the 3D model, the user and hairdresser can get a concrete idea of ​​the hairstyle and hair color that will suit the customer's face before the treatment. The input data is the 3D model data, and the output is the displayed 3D model. Specifically, the smartphone or tablet app displays the 3D model using a view function that allows the user to rotate, zoom in, and out.

[1479] Step 7:

[1480] Entering and regenerating correction requests

[1481] The user or hairdresser checks the 3D model and, if necessary, inputs any correction requests into the app. For example, specific instructions such as "I want my hair color lighter" can be included. The device sends this correction request data to the server, which then generates and updates the 3D model again. The input data is the correction request data, and the output is the corrected 3D model data. Specifically, the user or hairdresser enters the correction request into the app's input field and presses the "Send" button.

[1482] The above are the specific processing steps of the system that generates a three-dimensional hairstyle model using a customer's facial photograph.

[1483] (Application example 1)

[1484] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1485] Conventional methods for proposing hairstyles at beauty salons make it difficult for customers and hairdressers to accurately share specific images, which can lead to a decrease in customer satisfaction after the treatment. There was also a need for a method that allows customers to try on new hairstyles and hair colors without leaving their homes. Furthermore, there were issues with usability and accuracy regarding systems for trying on hairstyles in virtual environments.

[1486] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1487] In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; synthesis means for generating a 3D model by synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photo data; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and a therapist terminal; display means for displaying the 3D model; a generative AI model that analyzes natural language to understand the customer's request and generates and modifies the 3D model based on the request; and prompt generation means for inputting instruction data to be analyzed into the generative AI model. This allows customers to try new hairstyles and hair colors without leaving their homes and share specific images before the treatment, thereby improving customer satisfaction and improving the usability and accuracy of hairstyle try-on in a virtual environment.

[1488] "Customer terminal" refers to a digital device operated by a customer, including a smartphone, tablet, laptop, etc.

[1489] "Receiving means" refers to a means for receiving data, and in this case refers to a mechanism for receiving images, instruction data, etc. via Internet communication.

[1490] The "image processing means" refers to a means for performing processing to detect specific parts from received image data or remove backgrounds, and specifically includes algorithms for face detection and hair contour extraction.

[1491] "Instruction data" refers to data indicating information such as the hairstyle and hair color desired by the customer, and is input through the application.

[1492] The "candidate generation means" is a means for generating multiple hairstyle and hair color candidates based on instruction data, and is realized here by a generative AI model.

[1493] The "combining means" is a means for combining the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[1494] "Communication means" refers to the means for transmitting the generated 3D model and other data to the client terminal and the therapist terminal, primarily via the Internet.

[1495] The "display means" is a means for displaying the received 3D model on the terminal, and includes a graphical user interface (GUI).

[1496] A "professor terminal" is a digital device operated by the practitioner (hairdresser), and refers mainly to a computer or tablet used within the salon.

[1497] "Generative AI model" refers to an artificial intelligence (AI) model that analyzes natural language and generates and modifies 3D models based on customer requests.

[1498] The "prompt sentence generation means" is a means for generating sentences for inputting instruction data to be analyzed into the generative AI model.

[1499] The present invention relates to a digital hair catalog system for improving customer satisfaction in a virtual hair salon system. This system is comprised of a customer terminal, a therapist terminal, and a server, and transmits and receives data via the Internet.

[1500] Role of customer terminals

[1501] The customer terminal is a digital device such as a smartphone or tablet. The customer first launches the application and takes a photo of their face or selects one from the gallery. The photo is temporarily saved on the customer terminal and then sent to a server via the Internet.

[1502] Processing on the server

[1503] The server processes the data in the following steps:

[1504] 1. The server receives the facial photo data sent from the customer's device. The software used is Python's PIL (Python Imaging Library) and Requests library.

[1505] 2. Based on the received facial photo data, the system detects the face, removes the background, and extracts the hair outline. The OpenCV library is used here.

[1506] 3. Next, the customer receives instruction data specifying their desired hairstyle and hair color. This instruction data is entered by the customer in natural language within the app, and the server analyzes it using Hugging Face's Transformers library and T5 model.

[1507] 4. Generate multiple hairstyle and color options based on natural language analysis. This process uses a generative AI model to accurately reflect the customer's preferences in the 3D model.

[1508] 5. The generated hairstyle and hair color candidates are combined with facial photo data to generate a 3D model. The 3D model is created using the Open3D library.

[1509] 6. Finally, the generated 3D model can be sent to the client's device and the practitioner's device and displayed.

[1510] Role of the therapist terminal

[1511] The hairdresser's terminal is a digital device operated by the hairdresser. The hairdresser checks the 3D model sent from the server and selects the most suitable hairstyle and hair color together with the customer. If necessary, they can resubmit any correction requests to the server and generate and modify the 3D model again. This allows the customer and hairdresser to share a specific image.

[1512] Examples and prompts

[1513] For example, if a customer requests a short bob with red hair, the text is parsed with the prompt, "Please provide generated hairstyle and hair color options: Short bob with red hair." This prompt allows the generative AI model to generate multiple options based on the customer's request and synthesize them into a 3D model.

[1514] Example prompt sentence:

[1515] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[1516] This system allows customers to try on new hairstyles and hair colors without leaving their homes, and by sharing a concrete image of the desired look before the treatment, customer satisfaction is increased. In addition, the high-precision processing of natural language analysis and 3D model generation on the server improves the usability and accuracy of trying on hairstyles in a virtual environment.

[1517] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1518] Step 1:

[1519] The user launches the app using a smartphone or tablet and takes a photo of their face or selects one from the gallery. The selected photo is temporarily saved on the customer's device and then sent to the server via the Internet. In this step, the customer's photo is used as input data, and the photo data sent to the server is obtained as output data.

[1520] Step 2:

[1521] The server internally processes the received facial photo data. First, it uses the OpenCV library to detect the face, remove the background, and extract the hair outline. At this stage, the facial photo data is used as input data, and image data with the background removed and the hair outline extracted is obtained as output data.

[1522] Step 3:

[1523] The user inputs the desired hairstyle and hair color in the app using natural language. This instruction data is sent from the customer terminal to the server. In this step, instruction text in natural language is used as input data, and the instruction data sent to the server is obtained as output data.

[1524] Step 4:

[1525] The server analyzes the received instruction data. Using Hugging Face's Transformers library and the T5 model, it generates hairstyle and hair color candidates based on natural language. The instruction data is used as input data, and multiple hairstyle and hair color candidates are generated as output data.

[1526] Step 5:

[1527] The generative AI model generates a 3D model by combining the generated hairstyle and hair color candidates with facial photo data. The 3D model is created using the Open3D library. In this step, hairstyle and hair color candidates and facial photo data with the background removed are used as input data, and a 3D model is generated as output data.

[1528] Step 6:

[1529] The server sends the generated 3D model to the client's device and the therapist's device. The client's device and the therapist's device display the 3D model, allowing the client and therapist to share and review it. In this step, the generated 3D model is used as input data, and the transmitted 3D model is obtained as output data.

[1530] Step 7:

[1531] The client and the therapist check the 3D model on the therapist's terminal and input any correction requests as necessary. The correction requests are sent from the therapist's terminal to the server. At this stage, instruction data for corrections is used as input data, and the correction requests are sent to the server as output data.

[1532] Step 8:

[1533] The server generates and modifies the 3D model again based on the modification request, and sends the results back to the client's device and the therapist's device. The modification request is used as input data, and the modified 3D model is generated as output data.

[1534] Example prompt sentence:

[1535] Please provide the generated hairstyle and hair color suggestions: I want a short bob with red hair color

[1536] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1537] This invention relates to a digital hair catalog system for improving customer satisfaction at beauty salons, and in particular provides technology for generating a 3D model that is combined with a customer's facial photograph, and technology for recognizing the user's emotions and adjusting suggested hairstyles and hair colors. This allows the hairdresser and customer to accurately share a specific image, further improving customer satisfaction with the results of the treatment.

[1538] System configuration

[1539] This system consists of a customer terminal, a server, a hairdresser terminal, and an emotion engine. The customer terminal is a digital device such as a smartphone or tablet, and the hairdresser terminal is also a digital device. The server is a central system that processes data and communicates between these terminals. The emotion engine is a combination of software and hardware for recognizing customer emotions.

[1540] Program processing

[1541] Customer photo upload

[1542] The user (customer) starts the app on their smartphone or tablet and takes a photo of themselves or selects one from the gallery. The device temporarily saves the photo data and sends it to the server.

[1543] Image preprocessing on the server

[1544] The server receives the photo data and uses its internal AI image processing algorithm to detect the face, remove the background, and extract the hair contours, generating reference data for changing the hairstyle and hair color.

[1545] Emotion analysis with emotion engine

[1546] The user's emotion engine analyzes the uploaded facial photo data and input data from the customer's device in real time to monitor the user's emotional state. For example, if the customer is smiling, it is analyzed as having a positive emotion.

[1547] Input hairstyle and hair color requests

[1548] The user inputs their desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app. The device then sends this instruction data to the server.

[1549] Generate hairstyle and hair color candidates

[1550] The server analyzes the user's request (instruction data) using natural language processing (NLP) and passes it as input to the AI ​​generation model. The AI ​​generates candidate hairstyles and hair colors. Furthermore, based on the analysis results of the emotion engine, the candidate hairstyles and hair colors are adjusted to suit the user's emotional state.

[1551] 3D model composition and transmission

[1552] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[1553] Display and check the 3D model

[1554] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[1555] Entering and regenerating correction requests

[1556] The user (customer) or hairdresser checks the 3D model and inputs any correction requests as necessary. For example, specific instructions such as "I'd like the hair color to be a little lighter" are included. The device sends these correction requests to the server. The server then generates and updates the 3D model again, and sends the newly generated 3D model to the device.

[1557] Specific examples

[1558] Customer A Scenario

[1559] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1560] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1561] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[1562] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1563] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[1564] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1565] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[1566] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[1567] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[1568] This system enables more personalized hairstyle suggestions that take into account the user's emotional state, enabling smooth counseling based on a specific and shared image between customer A and the hairdresser.

[1569] The processing flow will be explained below.

[1570] Step 1:

[1571] The user launches the app and takes a photo or selects an image from the gallery. The user taps the "Upload Photo" button and takes a photo of themselves or selects an existing photo from the gallery.

[1572] Step 2:

[1573] The device temporarily stores the selected or taken photo data, optimizes the image format and resolution, and sends it to the server.

[1574] Step 3:

[1575] The server receives the uploaded photo data and associates it with the designated customer account. The server then begins processing the image data and uses AI image processing models to detect facial features.

[1576] Step 4:

[1577] After detecting the face, the server performs image processing to remove the background and extract the hair outline, generating the basic data for changing the hairstyle and hair color.

[1578] Step 5:

[1579] The device receives a notification from the server that face detection and image processing are complete, and prompts the user to perform the next step: The user enters the desired hairstyle (e.g., "short bob") and hair color (e.g., "reddish") using natural language within the app.

[1580] Step 6:

[1581] The device sends instruction data to the server, which analyzes the user's request using natural language processing (NLP) and inputs the instruction data into an AI-generated model.

[1582] Step 7:

[1583] The server uses an AI generative model to generate candidates for a specific hairstyle and hair color. Multiple hairstyle and hair color candidates are generated.

[1584] Step 8:

[1585] The server activates the emotion engine and analyzes the uploaded facial photo and instruction data to recognize the user's emotional state, and adjusts the hairstyle and hair color suggestions based on the recognized emotional state.

[1586] Step 9:

[1587] The server combines the photo with the generated hairstyle and hair color candidates to generate multiple 3D models, which are then sent to the customer's device and the hairdresser's device.

[1588] Step 10:

[1589] The device displays the received 3D model, allowing the user to rotate it and zoom in and out to check the details. The hairdresser also checks the same model at the salon.

[1590] Step 11:

[1591] The user and hairdresser check the 3D model and input requests for corrections to specific parts (for example, "I want the hair color to be a little lighter"). The device then sends these requests to the server.

[1592] Step 12:

[1593] The server receives the new request and updates the 3D model using the AI ​​model again. The revised 3D model is then sent back to the customer's device and the hairdresser's device.

[1594] Step 13:

[1595] The device displays the updated 3D model, and the stylist and user review the new model for a final confirmation. The user confirms whether they are satisfied with the final hairstyle and color, and if they agree, the treatment begins.

[1596] Specific examples

[1597] Customer A Scenario

[1598] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1599] 2. The device sends the photo data to the server.

[1600] 3. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1601] 4. The server notifies the terminal that image preprocessing is complete.

[1602] 5. The user (customer A) inputs that he / she desires a "short bob" and "reddish" hairstyle and hair color.

[1603] 6. The server analyzes the request and generates multiple short bob and reddish hair color options.

[1604] 7. The server uses the emotion engine to analyze the user's emotional state and adjusts the candidates suitable for a relaxed state.

[1605] 8. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1606] 9. The user (Customer A) and the hairdresser check the 3D model at the beauty salon.

[1607] 10. The user inputs a request for correction, such as "Make the hair color a little lighter."

[1608] 11. The server receives the correction request and sends the regenerated 3D model to the device.

[1609] 12. The user makes a final confirmation and, once they agree, the treatment begins.

[1610] This system enables personalized hairstyle suggestions that take into account the user's emotional state, realizing smooth counseling based on a specific and shared image between the customer and hairdresser.

[1611] Example 2

[1612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1613] Conventional hair catalog systems used in hair salons have made it difficult to improve customer satisfaction. In particular, the image of hairstyle shared between the customer and the hairdresser is vague, resulting in customers' expectations often not being met. In addition, it is difficult to reflect the customer's feelings and requests in real time, making it impossible to make optimal suggestions for each individual customer. A system that solves these issues was needed.

[1614] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo, removing background, and extracting hair contours based on the facial photo data; means for receiving instruction data specifying the customer's desired hairstyle and hair color; candidate generation means for generating multiple hairstyle and hair color candidates based on the instruction data; emotion analysis means for analyzing the customer's emotional data using a generative AI model and making adjustments based on the customer's emotions; synthesis means for synthesizing the hairstyle and hair color candidates adjusted by the candidate generation means and emotion analysis means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to the customer terminal and the hairdresser terminal; and display means for displaying the 3D model. This allows the customer and the hairdresser to have a specific and detailed shared image of the hairstyle and hair color during a smooth consultation. Furthermore, emotion analysis allows for optimal suggestions to be made to the customer, which is expected to improve customer satisfaction.

[1615] "Means for receiving facial photo data provided by customers" refers to a system or protocol that allows the server to receive facial photo data taken or selected by customers using a device such as a smartphone or tablet.

[1616] The "image processing means" refers to algorithms or software that detects the face, removes the background, and extracts the outline of the hair from the facial photograph data received by the server.

[1617] The "means for receiving instruction data" refers to a system or protocol that allows a customer to input their desired hairstyle and hair color within the app and for the server to receive that input data.

[1618] The "candidate generation means" refers to a generative AI model or algorithm for generating multiple hairstyle and hair color candidates based on the received instruction data.

[1619] An "emotion analysis method" is an algorithm or system that uses a generative AI model to analyze a customer's facial photograph and input data, determine the customer's emotions, and make adjustments based on the results.

[1620] The "combining means" refers to modeling software or algorithms that combine the generated hairstyle and hair color candidates with facial photograph data to generate a 3D model.

[1621] "Communication means" refers to the network infrastructure and protocols used to transmit the 3D model generated by the server to the customer terminal and the hairdresser terminal.

[1622] "Display means" refers to the interface and software for displaying the received 3D model on the customer terminal and the hairdresser terminal.

[1623] The program for this system is designed to improve customer satisfaction at beauty salons. Specifically, it provides technology that generates a 3D model based on a customer's facial photograph, analyzes the user's emotions, and adjusts suggested hairstyles and hair colors. Below, we will explain what hardware and software are used, and what data processing and calculations are performed.

[1624] Hardware and software used

[1625] Customer terminal

[1626] Customer terminals are primarily smartphones or tablets, which have application software installed on them and provide an interface for users to take a photo of their face and input their desired hairstyle and hair color.

[1627] Hairdresser's terminal

[1628] The hairdresser's terminal also uses a smartphone or tablet, which has application software installed on it that allows the hairdresser to view the customer's 3D model and input any correction requests.

[1629] server

[1630] The server acts as a central system, processing data sent from the customer and hairdresser devices. It implements advanced image processing algorithms and generative AI models, using libraries such as OpenCV and TensorFlow. It also uses Microsoft Azure's emotion recognition API to analyze customer emotions.

[1631] Data processing and calculation

[1632] Upload and pre-process face photos

[1633] The user takes a photo of their face on their device and sends it to the server, which then analyzes the received photo data to detect the face, remove the background, and extract the outline of the hair, using AI image processing algorithms.

[1634] Emotion analysis

[1635] The server runs an emotion engine that analyzes the uploaded facial photo data and customer input data to identify the user's emotions, which can then be classified as positive or negative.

[1636] Generate hairstyle and hair color candidates

[1637] The user inputs their desired hairstyle and hair color, which is then received by the server. The server then analyzes this data using NLP (natural language processing) and generates multiple candidates using a generative AI model. Based on the analysis results of the emotion engine, the server also adjusts the candidates to suit the user's emotional state.

[1638] 3D model compositing and display

[1639] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model. The generated 3D model is sent to the client's and stylist's devices using 3D modeling software such as Blender or Unity. The 3D model received on the device can be rotated and zoomed in and out, allowing the client and stylist to view it together.

[1640] Specific examples

[1641] Customer A Scenario

[1642] 1. The user (Customer A) opens the app and uploads a photo of their face.

[1643] 2. The server receives the photo and performs preprocessing to detect faces, remove backgrounds, and extract hair contours.

[1644] 3. The emotion engine analyzes the photo data and customer input data and determines that the user is relaxed.

[1645] 4. Customer A enters that he / she wants a "short bob" and a "reddish" hair color.

[1646] 5. The server analyzes the request and generates multiple short bob and reddish hair color options. It also adjusts the options to best suit the relaxed state based on the emotion engine results.

[1647] 6. The generated 3D model is sent to the customer's device and the hairdresser's device.

[1648] 7. At the salon, Customer A and the hairdresser review the 3D model and input specific requests for revisions, such as making the hair color a little lighter.

[1649] 8. The server receives the correction request and sends the regenerated 3D model to the device.

[1650] 9. Customer A makes a final confirmation and, once he or she agrees, the treatment begins.

[1651] Prompt Sentence Examples

[1652] "Based on a photo of the customer's face, please generate a 3D model with a short bob hairstyle and reddish hair color. Also, please display variations of the hairstyle that emphasize the relaxed look while the user is in a relaxed state."

[1653] This allows customers and hairdressers to have a specific and detailed shared image of the hairstyle they want, allowing for smooth counseling. Furthermore, emotion analysis can be used to provide optimal suggestions to customers, which is expected to improve customer satisfaction.

[1654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1655] Program processing flow

[1656] Step 1: Upload customer photos

[1657] The user launches the app and takes a photo of their face or selects one from the gallery. The device temporarily stores the photo data and sends it to the server.

[1658] Input: User's face photo

[1659] Output: Face photo data sent to the server

[1660] Specific behavior:

[1661] The user takes a photo of their face using the in-app camera, or selects an existing photo from the device's gallery.

[1662] The device temporarily saves the selected photo and sends it to the server using an HTTP POST request.

[1663] Step 2: Image preprocessing on the server

[1664] The server analyzes the received facial photo data, detects the face, removes the background, and extracts the outline of the hair.

[1665] Input: User's face photo data

[1666] Output: Data after face detection, background removal, and hair contour extraction

[1667] Specific behavior:

[1668] The image data received by the server is processed using an AI image processing algorithm (OpenCV or TensorFlow).

[1669] Detects the face and removes it by masking the background.

[1670] Identify and extract the hair contours.

[1671] Step 3: Sentiment Analysis

[1672] The server uses an emotion engine to analyze the uploaded facial photo and customer input data to identify the user's emotional state.

[1673] Input: Face photo data and customer input data

[1674] Output: Analysis of the user's emotional state

[1675] Specific behavior:

[1676] The server calls the emotion engine's API (e.g., emotion recognition API) and passes the data to be analyzed as input.

[1677] The emotion engine determines emotions such as smile or anger and returns the results to the server.

[1678] Step 4: Enter your hairstyle and hair color preferences

[1679] Users input their desired hairstyle and hair color within the app, and the device sends this data to the server.

[1680] Input: User's desired hairstyle and hair color instructions

[1681] Output: Instruction data sent to the server

[1682] Specific behavior:

[1683] The user enters their desired style, such as "short bob" or "reddish," into a text box within the app.

[1684] The device sends this input data to the server via an HTTP POST request.

[1685] Step 5: Generate hairstyle and hair color candidates

[1686] The server uses NLP (natural language processing) to analyze the user's request, then runs a generative AI model to generate multiple hairstyle and hair color options, and makes adjustments based on the results of sentiment analysis.

[1687] Input: User instruction data and sentiment analysis results

[1688] Output: Candidate data for hairstyle and hair color

[1689] Specific behavior:

[1690] The server uses an NLP model (e.g., a GPT model) to analyze the user's wishes.

[1691] The analysis results are input into a generative AI model to generate multiple hairstyle and hair color options.

[1692] Adjust candidates based on the results of sentiment analysis.

[1693] Step 6: Composite and send your 3D model

[1694] The server combines the generated hairstyle and hair color candidates with the user's facial photo to generate a 3D model, which is then sent to the customer's device and the hairdresser's device.

[1695] Input: Face photo data, hairstyle and hair color candidate data

[1696] Output: The generated 3D model, and the client and hairdresser devices that receive it.

[1697] Specific behavior:

[1698] The server uses 3D modeling software such as Blender or Unity to integrate each piece of data.

[1699] The generated 3D model is sent to the customer's device and the hairdresser's device via an HTTP POST request.

[1700] Step 7: View and check the 3D model

[1701] The device displays the received 3D model for the user and hairdresser to review. The user can rotate the 3D model and zoom in and out.

[1702] Input: Received 3D model

[1703] Output: A displayed 3D model

[1704] Specific behavior:

[1705] The device launches a 3D viewer, reads the received model data, and displays it.

[1706] Users can use touch to rotate the 3D model and zoom to view details.

[1707] Step 8: Enter and Regenerate Correction Requests

[1708] The user or hairdresser checks the 3D model and inputs any additional requests or corrections. The device sends this data to the server, which then generates a new 3D model and sends it to the device.

[1709] Input: Additional requests and correction instructions

[1710] Output: Updated 3D model

[1711] Specific behavior:

[1712] Users input requests for corrections, such as "I would like the hair color to be a little lighter," into the app.

[1713] The device sends this correction data to the server via an HTTP POST request.

[1714] The server processes the data again, generates a new 3D model, and sends it to the device.

[1715] (Application example 2)

[1716] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1717] Conventional virtual try-on systems simply display the clothes and accessories selected by the user as they are, and are unable to improve the user's emotional state or the quality of the virtual experience in real time. This makes it difficult to provide optimal suggestions for the user, resulting in issues with customer satisfaction. Furthermore, it is time-consuming to combine facial photo data with candidate clothes and accessories, and it is difficult to quickly regenerate them based on requested modifications.

[1718] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving facial photo data provided by a customer; image processing means for detecting facial features from the facial photo based on the facial photo data, removing the background, and extracting the outline of the appearance; means for receiving instruction data specifying the appearance and color desired by the customer; candidate generation means for generating multiple appearance and color candidates based on the instruction data; synthesis means for synthesizing the appearance and color candidates generated by the candidate generation means with the facial photo data to generate a 3D model; communication means for transmitting the 3D model generated by the synthesis means to a customer terminal and an operation terminal; display means for displaying the 3D model; and means for analyzing the emotional state and adjusting the generated appearance and color candidates. This enables a system that provides an optimal virtual try-on experience based on user emotional analysis, improves customer satisfaction, and enables quick and easy modifications.

[1719] A "customer terminal" is a digital device operated by the customer, and is used to take a facial photograph, input instruction data, check 3D models, etc.

[1720] An "operation terminal" is a digital device operated by an operator, which is used to check the 3D model together with the customer and input any correction requests.

[1721] The "image processing means" is a collection of algorithms and software that detects facial features from received facial photograph data, removes background, and extracts the contours of the face.

[1722] "Instruction data" refers to data for inputting specific requests such as the appearance and color desired by the customer.

[1723] A "candidate generator" is a collection of algorithms and software that generates multiple appearance and color candidates based on instruction data.

[1724] "Synthesis means" refers to technology or software that integrates the generated appearance and color candidates with facial photograph data to generate a 3D model.

[1725] "Communication means" refers to the network communication technology and software used to transmit the generated 3D model to the customer terminal and the operation terminal.

[1726] "Display means" refers to a display device or software that allows a user or operator to visually check the received 3D model.

[1727] "Emotional state" is a mental health condition that analyzes a user's emotions and moods and monitors them in real time.

[1728] The "emotion analysis means" is a collection of technologies and algorithms for analyzing the user's emotional state based on facial photo data and input data, and adjusting the appearance and color candidates.

[1729] The present invention is a system for improving the fitting experience in a virtual fashion shop. The main components include a customer terminal, an operation terminal, a server, and a sentiment analysis engine. Each component is described below.

[1730] Customer terminal

[1731] The customer device refers to a smartphone or head-mounted display (HMD) operated by the customer. This device is used by the customer to take or upload a photo of their face and input their desired appearance and color. The customer device has the function of temporarily storing the face photo data and sending it to a server.

[1732] Operation terminal

[1733] The operation terminal is a tablet or PC operated by an operator. On this terminal, the operator can check the 3D model together with the customer and input any correction requests. The operation terminal also has the function of displaying the 3D model received from the server and inputting corrections as needed.

[1734] server

[1735] The server processes the received facial photo data, removing the background and extracting the contours of the appearance. The server uses image processing libraries such as OpenCV and Dlib. The server also uses natural language processing libraries (e.g., spaCy, NLTK) to analyze the customer's instruction data and input it into an AI generation model. The generated appearance and color candidates are integrated into a 3D model and sent to the customer's device and the operating device.

[1736] Sentiment Analysis Engine

[1737] The emotion analysis engine analyzes the user's emotional state based on their photo data and input data, using cloud-based emotion analysis services such as Amazon Rekognition and Microsoft Azure Face API, and then adjusts the optimal appearance and color options based on the analysis results.

[1738] Specific examples of processing

[1739] For example, if User B inputs that they want a "blue dress," the server analyzes this data and generates multiple blue dress candidates. If the user also requests that the color be "more vibrant," a new 3D model will be provided.

[1740] Prompt Sentence Examples

[1741] A user types: "I want to try on a blue dress."

[1742] The desired outfit type is "dress" and the color is "blue." Sentiment analysis indicates that the user is happy. Please present a more vibrant blue color.

[1743] The system of the present invention provides an optimal virtual try-on experience based on user sentiment analysis, improving customer satisfaction and enabling quick and easy modifications.

[1744] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1745] Step 1:

[1746] The user uses the customer's device to take a photo of their face or select one from a gallery, and then enters the photo data into the application. This photo data is temporarily saved in the customer's device's local storage. The device then sends this photo data to the server. The specific operation of sending the photo data involves communication using an HTTP request.

[1747] Step 2:

[1748] The server receives the facial photo data sent. Using OpenCV and Dlib, the server detects the face from the facial photo data, removes the background, and extracts the outline of the face. Specifically, the server converts the image to grayscale, detects facial landmarks, and then cuts out only the necessary parts. The input is the facial photo data sent from the customer's device, and the output is image data from which the face has been detected, the background removed, and the outline of the face extracted.

[1749] Step 3:

[1750] The user inputs the desired appearance and color in natural language using a customer terminal. This instruction data is temporarily stored in the local storage of the customer terminal and then sent to the server. The input is text data in natural language entered by the user, and the output is the instruction data sent to the server.

[1751] Step 4:

[1752] The server uses a natural language processing library (e.g., spaCy, NLTK) to parse the instruction data it receives. The server parses the instruction data and identifies the desired appearance and color. Specific operations include tokenizing the text and extracting specific attributes (e.g., color, style). The input is the instruction data sent from the client device, and the output is the parsed desired appearance and color information.

[1753] Step 5:

[1754] The server inputs data into an AI generative model based on the analyzed appearance and color information. The generative AI model generates multiple appearance and color candidates. Specifically, it inputs the tensor data corresponding to the model and receives the generated candidate data. The input is the analyzed appearance and color information, and the output is the generated appearance and color candidates.

[1755] Step 6:

[1756] The server synthesizes the generated appearance and color candidates with the facial photo data to generate a 3D model. Specifically, it uses 3D rendering technology to integrate the photo data and candidate data. The input is the facial photo data and the generated candidate data, and the output is a 3D model.

[1757] Step 7:

[1758] The server sends the generated 3D model to the customer terminal and the operation terminal. The customer terminal and the operation terminal then receive and display the 3D model. Specifically, the transmission is performed via an HTTP request, and the received 3D model is displayed on the terminal's display. The input is the 3D model, and the output is the 3D model displayed on the terminal's display.

[1759] Step 8:

[1760] The user and operator check the 3D model and input any correction requests as necessary. Once the correction requests are entered, they are sent to the server from the customer terminal or the operation terminal. The input is text data of the correction requests, and the output is correction instruction data that is sent to the server.

[1761] Step 9:

[1762] The server regenerates the 3D model based on the correction request and resends the updated 3D model. Specifically, it generates new appearance and color candidates based on the correction instruction data and recomposes them. The input is the correction instruction data, and the output is the updated 3D model.

[1763] Step 10:

[1764] The customer terminal and the operation terminal display the received 3D model again for final confirmation. With final consent, the treatment (or purchase) begins. Specifically, the 3D model is displayed on the display again and the user's consent is sought. The input is the updated 3D model, and the output is the final confirmed 3D model.

[1765] In this way, a system is realized that provides an optimal virtual try-on experience based on user sentiment analysis, improves customer satisfaction, and allows for quick and easy adjustments.

[1766] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1767] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1768] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1769] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1770] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1771] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1772] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1773] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1774] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1775] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1776] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1777] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the...

Claims

1. A means for receiving customer-provided facial photograph data; an image processing means for detecting a face portion from the face photograph, removing a background, and extracting a hair outline based on the face photograph data; means for receiving instruction data specifying a hairstyle and hair color desired by the customer; candidate generating means for generating a plurality of candidates for hairstyle and hair color based on the instruction data; a synthesis means for synthesizing the hairstyle and hair color candidates generated by the candidate generation means with the facial photograph data to generate a 3D model; a communication means for transmitting the 3D model generated by the synthesis means to the customer terminal and the hairdresser terminal; a display means for displaying the 3D model; A system including:

2. A means for the customer and the hairdresser to check the 3D model and input desired corrections at the hairdresser terminal; a means for regenerating a 3D model based on the corrections; and a communication means for transmitting the corrected 3D model again to the customer terminal and the hairdresser terminal. The system of claim 1 .

3. an operating means for rotating and zooming in and out of the 3D model on the client terminal; a display means for displaying the 3D model operated by the operation means in real time; The system of claim 1 .

Images