system

Abstract

Provide a system. [Solution] means for receiving facial image data; means for generating a 3D model based on the image data; A means of interactively gathering details about the user's desired appearance; means for modifying the 3D model based on the collected preferences; A system including means for providing said modified 3D model to a user and a practitioner.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern cosmetic surgery, it is difficult for patients to accurately communicate their desired appearance to their surgeon. As a result of not being able to share a specific image, post-procedure satisfaction is often low, and a gap between expectations and reality often arises. Furthermore, if patients are unable to accurately verbalize their wishes, it becomes difficult for doctors to create an appropriate treatment plan. For this reason, there is a need for a method to visually materialize patients' wishes and improve the precision of the treatment, so that both patients and doctors can be satisfied. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including a means for receiving facial image data, a means for generating a 3D model based on the image data, a means for interactively collecting details of a user's desired appearance, a means for modifying the 3D model based on the collected preferences, and a means for providing the modified 3D model to the user and the practitioner. Specifically, the system modifies the 3D model based on the collected user preferences and repeatedly receives feedback from the user to determine the final ideal appearance. The system also includes a question generation module as a means for interactively collecting details of the preferences, generating additional specific questions based on the user's answers to accurately grasp and visualize the user's preferences. This allows for a specific and objective consideration of the patient's desired appearance, thereby improving the accuracy of the treatment.

[0006] "Facial image data" is digital image information that visually captures the facial features of a user.

[0007] A "3D model" refers to digital data that represents the shape and features of a user's face in three dimensions.

[0008] "Dialogue" refers to a method of collecting information through a question-and-answer exchange between the system and the user.

[0009] "Means for collecting desired details" refers to interactive tools and processes that allow users to capture their desired cosmetic changes and specific requests.

[0010] "Means of modification" refers to algorithms and functions that modify and update existing 3D models based on user preferences.

[0011] "Means of Providing" refers to the methods and tools for displaying and sharing modified 3D models with users and practitioners.

[0012] "Question Generation Module" refers to a function or program that automatically generates specific follow-up questions based on the user's answers.

[0013] "User Feedback" refers to any opinions or additional requests provided by a User to the System.

[0014] "Practitioner" refers to a professional doctor or technician who performs cosmetic surgery.

[0015] A "treatment plan" is a specific surgery or treatment plan that the practitioner creates based on the user's wishes and the 3D model. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. The system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, modifies the 3D model based on the collected information, and provides the 3D model to the user and the practitioner.

[0038] 1. Image upload and initial settings

[0039] 1. Upload an image

[0040] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0041] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[0042] 2. Image Reception and Analysis

[0043] The server receives the uploaded facial photos and stores them in a database.

[0044] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[0045] 2. Building an ideal image through dialogue

[0046] 1. Generating the initial question

[0047] The server prepares the interactive questions and presents the first question to the user.

[0048] For example, ask the user, "What changes would you like to see?"

[0049] 2. Obtaining and elaborating user answers

[0050] The user responds, "I want to make my nose taller."

[0051] The server generates additional, more specific questions based on the answers and presents them to the user.

[0052] For example: Ask specifically, "How tall would you like your nose to be?"

[0053] The user responds, "I'd like it to be about 2 mm higher."

[0054] 3. Iteration

[0055] This interactive question and answer process applies to other facial features as well (eyes, mouth, chin, etc.).

[0056] 3. Creating and checking the 3D model

[0057] 1. Creating the initial 3D model

[0058] The server generates an initial 3D model based on the collected user preferences.

[0059] The server sends the generated 3D model to the terminal and displays it to the user.

[0060] 2. User Review and Feedback

[0061] The device presents the 3D model to the user and prompts them to confirm it.

[0062] The user provides feedback such as, "Make the nose 1 mm higher and thinner."

[0063] The server revises the 3D model based on the feedback and displays it again.

[0064] 4. Sharing and implementing the final model

[0065] 1. Final confirmation and model confirmation

[0066] The user is satisfied with the final 3D model and decides, "This is it."

[0067] The terminal sends this information to the server.

[0068] 2. Save and share your final model

[0069] The server stores the finalized 3D model and sends it to the practitioner (doctor).

[0070] 3. Planning the treatment plan

[0071] The practitioner will create a specific treatment plan based on the final 3D model.

[0072] In this way, by allowing users to concretize their desired appearance and visualize it as a 3D model, communication between the patient and the practitioner can be facilitated and the precision of the treatment can be improved.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] Uploading an image

[0076] Users upload a photo of their face to the system from their terminal.

[0077] The terminal sends this facial photo to the server.

[0078] Step 2:

[0079] Image reception and analysis

[0080] The server receives the uploaded facial photos and stores them in a database.

[0081] The server uses an image processing module to extract facial feature points.

[0082] Step 3:

[0083] Generate initial questions

[0084] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0085] The terminal displays this question to the user.

[0086] Step 4:

[0087] Get user answers

[0088] The user responds, "I want to make my nose taller."

[0089] The terminal sends this response to the server.

[0090] Step 5:

[0091] Generate follow-up questions

[0092] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0093] The terminal displays this question to the user.

[0094] Step 6:

[0095] Getting specific answers

[0096] The user responds, "Make it about 2 mm higher."

[0097] The terminal sends this response to the server.

[0098] Step 7:

[0099] Repeated dialogue

[0100] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0101] The user responds, "I'd like the eyes to be a little bigger."

[0102] *This process is repeated for all changes the user desires.

[0103] Step 8:

[0104] Generating the initial 3D model

[0105] The server generates an initial 3D model based on the collected user preferences.

[0106] The server sends the generated 3D model to the terminal.

[0107] Step 9:

[0108] Check the 3D model

[0109] The device displays the initial 3D model to the user.

[0110] The user checks the 3D model and provides feedback such as, "Make the nose 1 mm higher and thinner."

[0111] The terminal sends this feedback to the server.

[0112] Step 10:

[0113] Modifying the 3D model

[0114] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0115] This process is repeated until the user is satisfied.

[0116] Step 11:

[0117] Final confirmation and model confirmation

[0118] The user is satisfied with the final 3D model and decides, "This is it."

[0119] The terminal sends this information to the server.

[0120] Step 12:

[0121] Save and share your final model

[0122] The server stores the final 3D model.

[0123] The server shares the finalized 3D model with the practitioner.

[0124] Step 13:

[0125] Treatment plan planning

[0126] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[0127] In this way, by collecting and analyzing the user's wishes at each step and generating, modifying and providing specific 3D models, it is possible to increase user satisfaction and the accuracy of the treatment.

[0128] Example 1

[0129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In conventional cosmetic medicine, it has been difficult for users to accurately communicate their desired appearance to the practitioner. There is a need for a system that can collect detailed information about the specific changes desired by the user and generate and modify a 3D model based on that information. In addition, there is a high need for a system that can repeatedly check and modify the 3D model based on user feedback.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0132] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for including a generative AI model for generating interactive questions, means for generating additional specific questions based on the user's answers to the generated questions, means for repeatedly modifying the 3D model based on the collected preferences, and means for redisplaying the 3D model. This allows the user to collect details of the user's desired appearance and repeatedly confirm and modify them, thereby providing the practitioner with accurate information and improving the accuracy of the treatment.

[0133] "Facial image data" refers to an image file of a user's face that is used by the system to analyze facial features.

[0134] A "3D model" is a computer-generated, three-dimensional digital model that reflects a user's facial features and desired changes.

[0135] "Dialogue" is a form of communication between a person and a system through questions and answers, and is a method of gathering detailed information about the user's wishes.

[0136] A "generative AI model" is a model that uses artificial intelligence technology to generate appropriate questions and answers based on user input, and examples include AI for processing natural language.

[0137] "Feedback" refers to the input of desired changes or modifications made by the user to the system, which the system uses to reflect this and modify the 3D model.

[0138] "User" refers to a person who utilizes the system to input their wishes to change their appearance.

[0139] "Practitioner" refers to a professional who performs cosmetic medical treatments based on the user's wishes.

[0140] This invention is a system that helps users accurately communicate their desired appearance to a practitioner before undergoing cosmetic medical treatment. The system receives facial image data from the user, generates a 3D model based on that data, interactively collects detailed requests, modifies the 3D model based on the collected requests, and provides the results to the user and the practitioner.

[0141] The system uses the following hardware and software:

[0142] Hardware: Servers, devices (smartphones, computers)

[0143] Software: OpenCV (image processing library), Blender (3D modeling software), generative AI models (e.g., GPT-3 (registered trademark))

[0144] Users take a photo of their face using their device and upload the image data to the system via a dedicated app or web page. The server receives the image data and stores it in a database. The server then uses an image processing library such as OpenCV to recognize facial feature points, which allows for the generation of an initial 3D model.

[0145] Next, the server uses the generative AI model to create dialogue-style questions and present them to the user. For example, it generates a question such as "What changes would you like to make?" and obtains the user's answer. The user inputs a specific answer such as "I want my nose to be higher." The server then uses the generative AI model again to generate an additional specific question such as "How high do you want your nose to be?" and presents it to the user. All detailed wishes collected through the dialogue with the user are recorded.

[0146] The server uses 3D modeling software such as Blender to generate a 3D model based on the collected preferences and sends the model to the device. The device then displays the 3D model to the user, allowing the user to provide feedback. For example, the user may enter specific feedback such as, "Make the nose 1 mm higher and thinner." The server then modifies the 3D model based on this feedback, sends it back to the device, and displays it to the user. This process is repeated until the user is satisfied.

[0147] Finally, the server stores the 3D model finalized by the user and provides it to the practitioner, who then uses it to create a specific treatment plan. This prevents any discrepancies between the user and the practitioner regarding the desired appearance and improves the accuracy of the treatment.

[0148] As a concrete example, the following exchange occurs:

[0149] Prompt: "What would you like to change?"

[0150] User Answer: "I want to make my nose taller."

[0151] Next prompt: "How tall would you like your nose to be?"

[0152] User answer: "I want it to be about 2mm higher."

[0153] In this way, the present invention provides a system that can facilitate communication between the user and the practitioner and improve the accuracy of treatment by specifically collecting the user's desired appearance and sequentially reflecting it in a three-dimensional model.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1: Upload an image

[0156] Users use a device (smartphone or computer) to take a photo of their face and upload the image data to the system via a dedicated app or web page.

[0157] Input: Face image data (JPEG, PNG, etc.)

[0158] Specific behavior: The user takes a photo of their face using the camera app and clicks the upload button to upload the image file via the app or browser.

[0159] Output: Facial image data sent to the system.

[0160] Step 2: Receiving and analyzing images

[0161] The server receives the uploaded facial photo, stores it in a database, and uses an image processing module to recognize facial features (the positions of the eyes, nose, mouth, etc.).

[0162] Input: Image data uploaded by the user

[0163] How it works: The server receives the image data sent in the HTTP request and saves it in storage. It then uses an image processing library such as OpenCV to detect facial feature points.

[0164] Output: Feature point coordinate data and image path stored in database.

[0165] Step 3: Generate initial questions

[0166] The server uses the generative AI model to generate initial interactive questions and present them to the user.

[0167] Input: Feature point coordinate data and user information

[0168] What it does: The server calls a generative AI model (e.g., GPT-3) and generates questions such as, "What changes would you like to make?"

[0169] Output: Initial question.

[0170] Step 4: Obtaining and elaborating user responses

[0171] The user enters an answer to the question, and the server then generates a more specific question based on the answer and presents it to the user.

[0172] Input: Initial question and user's answer

[0173] Specific operation: The user enters an answer (e.g., "I want to make my nose taller") into the input form and submits it. The server uses the generative AI model to generate the next question (e.g., "How tall do you want your nose to be?").

[0174] Output: Detailed follow-up questions.

[0175] Step 5: Iterate

[0176] The dialogue-style question and answer process is repeated for other facial features (eyes, mouth, chin, etc.).

[0177] Input: Previous answers and generated questions

[0178] What happens: The server receives each answer and generates additional questions as needed and presents them to the user.

[0179] Output: A list of questions and answers with all preferences collected in detail.

[0180] Step 6: Creating the initial 3D model

[0181] The server uses three-dimensional modeling software to generate an initial 3D model based on the collected user preferences.

[0182] Input: User's desired details

[0183] Specific operation: The server calls 3D modeling software such as Blender and generates a 3D model based on the user's desired shape.

[0184] Output: Data of the initial 3D model.

[0185] Step 7: User review and feedback

[0186] The device displays the generated 3D model to the user, who then provides feedback.

[0187] Input: Initial 3D model

[0188] What it does: Render and display a 3D model on the user's device, and the user can provide feedback such as "Make the nose 1mm taller and thinner."

[0189] Output: User feedback.

[0190] Step 8: Rework the 3D model

[0191] The server revises the 3D model based on the user's feedback and displays it again, repeating this process until the user is satisfied.

[0192] Input: User feedback

[0193] Specific operation: The server analyzes the feedback, revises the model in the 3D modeling software, and sends the revisited model data back to the user's device.

[0194] Output: The modified 3D model.

[0195] Step 9: Final check and finalize the model

[0196] The user is satisfied with the final 3D model and decides, "This is it."

[0197] Input: Modified 3D model

[0198] Specific behavior: The user makes a final confirmation and clicks the "Confirm" button.

[0199] Output: Model confirmation notice.

[0200] Step 10: Save and share your final model

[0201] The server stores the finalized 3D model and provides it to the practitioner.

[0202] Input: Model confirmation notification and 3D model data

[0203] Specific operation: The server saves the 3D model data in a database and notifies the practitioner via email or a dedicated portal.

[0204] Output: 3D model data provided to the practitioner.

[0205] Step 11: Planning your treatment

[0206] The practitioner will create a specific treatment plan based on the final 3D model.

[0207] Input: Provided 3D model data

[0208] Specific operations: The surgeon simulates surgery based on the provided 3D model and creates a specific treatment plan.

[0209] Output: Treatment plan.

[0210] (Application example 1)

[0211] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0212] In traditional plastic surgery and cosmetic procedures, it has been difficult for patients to accurately communicate their desired appearance to the doctor, resulting in unsatisfactory results. Similarly, in food delivery, it has been difficult for customers to communicate the specific customizations they desire, resulting in unsatisfactory orders. There is a need for a system that can solve these issues and accurately respond to users' requests.

[0213] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0214] In this invention, the server includes means for receiving image data of a face and food, means for generating and modifying a 3D model, means for interactively collecting details of the user's desired appearance and food, and means for modifying the 3D model according to the user's wishes and providing it to the user, a practitioner, or a restaurant, thereby enabling the user to accurately communicate specific customizations of the user's desired appearance and food, and for treatment and cooking to be performed based on those customizations.

[0215] "Facial image data" refers to photographs and image information of the user's face, and is the basic data used to generate a 3D model.

[0216] "Means for generating 3D models" refers to the process and technology for creating three-dimensional model data based on image data of faces or food.

[0217] "A means for interactively gathering details of a user's desired appearance" refers to an interface and system that gathers the specifications and features desired by the user in the form of questions.

[0218] "Means for modifying 3D models based on collected preferences" refers to techniques for modifying or updating existing 3D models based on preferences or feedback collected from users.

[0219] "Means for providing modified 3D models to users and practitioners" refers to a system for displaying or transmitting the completed 3D model to users and practitioners.

[0220] "Food image data" refers to photos and image information of the dishes and ingredients desired by the user, and is the basic data used to generate a 3D model of the dish.

[0221] "Means for generating and modifying 3D food models" refers to technology that creates three-dimensional food model data based on food image data and modifies or updates it.

[0222] "Means for interactively collecting details of the food desired by the user" refers to an interface and system for collecting the specifications and characteristics of the food desired by the user in the form of questions.

[0223] "Means for modifying 3D food models according to user preferences and providing the finalized models to restaurants" refers to a system for modifying 3D food models based on preferences and feedback collected from users and displaying or transmitting the finalized models to restaurants.

[0224] The present invention provides a system that allows users to specifically communicate their desired appearance and food customization details. This system operates between a server and a user terminal, and the process proceeds as follows:

[0225] 1. Image upload and initial settings

[0226] Users take a photo of themselves or a reference image of the dish they want to eat with their smartphone, and then send the image to the server via the application. The server then stores the received image data and begins analyzing it.

[0227] The server uses image processing modules such as OpenCV to analyze facial features and food ingredients, and then generates an initial 3D model. The generated 3D model is then sent to the user's device and displayed to the user.

[0228] 2. Creating ideal appearances and dishes through dialogue

[0229] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The user enters answers, and the server generates additional, more specific questions based on those answers.

[0230] For example, if the user answers "I want a taller nose" or "I want more cheese," the server generates and displays more specific questions such as "How tall do you want your nose to be?" or "What type of cheese would you like more of?"

[0231] 3. Creating and checking the 3D model

[0232] The server generates an initial 3D model based on the user's preferences and sends it to the user's device. The user can review the 3D model and provide feedback as needed. For example, they can provide feedback such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes."

[0233] The server then modifies the 3D model based on this feedback and displays it again to the user, repeating this process until the user is satisfied.

[0234] 4. Sharing the final model and ordering / treatment

[0235] Once the user is satisfied with the final 3D model and confirms it, the device sends this information to the server. The server saves the finalized 3D model and sends it to the therapist or restaurant. The therapist uses this model to create a specific treatment plan, and the restaurant begins cooking based on this model.

[0236] Specific examples and prompts for the generative AI model

[0237] Specific examples

[0238] 1. In the case of cosmetic surgery: The user requests "I want my nose to be straighter and higher," and the 3D model is updated based on that request.

[0239] 2. Food delivery: A user requests "more cheese on my pizza," and the 3D model is updated based on that request.

[0240] Generative AI model prompt

[0241] A user wants to customize a cosmetic procedure or food. Please generate a 3D model using the following instructions:

[0242] 1. Cosmetic procedure: Nose lift (specifically, 2mm higher)

[0243] 2. Cooking: Add more cheese (mozzarella and parmesan)

[0244] Output: Updated 3D model data and its visual representation.

[0245] Using this prompt, the AI ​​model can generate a 3D model that accurately reflects the user's wishes. This system accurately conveys the user's wishes, improving the accuracy of treatment and cooking.

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] Users take a photo of themselves or an image of the dish they want to eat with their smartphone and upload it to the server through the application. The server receives this image data and stores it in a database. Here, the input is the image file uploaded by the user, and the output is the image data stored on the server.

[0249] Step 2:

[0250] The server analyzes the received image data using an image processing module such as OpenCV. It extracts facial features and food ingredients and generates an initial 3D model based on them. Here, the input is the image data saved in step 1, and the output is the generated initial 3D model data.

[0251] Step 3:

[0252] The server sends the generated initial 3D model to the terminal and displays it to the user. The input is the 3D model data generated in step 2, and the output is the 3D model displayed on the user's terminal.

[0253] Step 4:

[0254] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The input is the user's preference, and the output is the interactive questions.

[0255] Step 5:

[0256] The user answers questions interactively, for example, by inputting specific wishes such as "I want a taller nose" or "I want more cheese." The input is the user's answer, and the output is the answer data sent to the server.

[0257] Step 6:

[0258] The server analyzes the collected response data from the user and generates additional specific questions, such as "How tall do you want your nose to be?" or "Which type of cheese would you like more of?", and presents them to the user again. The input is the user's response data, and the output is the additional questions.

[0259] Step 7:

[0260] The server modifies the 3D model based on the user's detailed wishes and sends it back. For example, creating a 3D model of a pizza with a longer nose or more cheese. The input is the user's wishes, and the output is the modified 3D model data.

[0261] Step 8:

[0262] The user checks the modified 3D model and provides further feedback, such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes." The input is the user's feedback, and the output is the feedback data received by the server.

[0263] Step 9:

[0264] The server revises the 3D model based on this feedback and resends it. This process is repeated until the user is satisfied. The input is the user's feedback data, and the output is the revisited 3D model.

[0265] Step 10:

[0266] When the user is satisfied with the final 3D model and confirms, "This is it," the device sends this information to the server. The input is the user's final decision, and the output is the decision data received by the server.

[0267] Step 11:

[0268] The server stores the finalized 3D model and sends it to the practitioner or restaurant. The practitioner makes a specific treatment plan based on this model, and the restaurant starts cooking based on this model. The input is the finalized 3D model data, and the output is the model data sent to the practitioner or restaurant.

[0269] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0270] This invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between users and physicians. This system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, and modifies the 3D model based on those details, providing it to the user and the practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[0271] 1. Image upload and initial settings

[0272] 1. Upload an image

[0273] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0274] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[0275] 2. Image Reception and Analysis

[0276] The server receives the uploaded facial photos and stores them in a database.

[0277] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[0278] 2. Building an ideal image through dialogue

[0279] 1. Generating the initial question

[0280] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0281] The terminal displays this question to the user.

[0282] 2. Obtaining user answers

[0283] The user responds, "I want to make my nose taller."

[0284] The terminal sends this response to the server.

[0285] 3. Generating follow-up questions

[0286] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0287] The terminal displays this question to the user.

[0288] 4. Getting specific answers

[0289] The user responds, "Make it about 2 mm higher."

[0290] The terminal sends this response to the server.

[0291] 5. Repeated dialogue

[0292] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0293] The user responds, "I'd like the eyes to be a little bigger."

[0294] *This process is repeated for all changes the user desires.

[0295] 3. Dialogue adjustment using emotion engine

[0296] 1. User Emotion Recognition

[0297] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[0298] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[0299] 2. Emotion-based dialogue adjustment

[0300] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[0301] The device displays these tailored questions or messages to the user.

[0302] 4. Creating and checking the 3D model

[0303] 1. Generate the initial 3D model

[0304] The server generates an initial 3D model based on the collected user preferences.

[0305] The server sends the generated 3D model to the terminal and displays it to the user.

[0306] 2. User Review and Feedback

[0307] The device will present the user with an initial 3D model and ask them to confirm it.

[0308] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[0309] The terminal sends this feedback to the server.

[0310] 3. Modifying the 3D model

[0311] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0312] This process is repeated until the user is satisfied.

[0313] 5. Final confirmation and treatment plan

[0314] 1. Final confirmation and model confirmation

[0315] The user is satisfied with the final 3D model and decides, "This is it."

[0316] The terminal sends this information to the server.

[0317] 2. Save and share your final model

[0318] The server saves the final 3D model and sends it to the practitioner (doctor).

[0319] 3. Planning the treatment plan

[0320] The practitioner will create a specific treatment plan based on the final 3D model.

[0321] In this way, by allowing users to specify their desired appearance and visualizing it as a 3D model, communication between patients and practitioners can be facilitated and the precision of treatment can be improved. In addition, by taking the user's emotional state into account using an emotion engine, user satisfaction can be further improved.

[0322] The processing flow will be explained below.

[0323] Step 1:

[0324] Uploading an image

[0325] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0326] The terminal sends this facial photo to the server.

[0327] Step 2:

[0328] Image reception and analysis

[0329] The server receives the uploaded facial photos and stores them in a database.

[0330] The server uses an image processing module to extract facial feature points.

[0331] Step 3:

[0332] Generate initial questions

[0333] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0334] The terminal displays this question to the user.

[0335] Step 4:

[0336] Get user answers

[0337] The user responds, "I want to make my nose taller."

[0338] The terminal sends this response to the server.

[0339] Step 5:

[0340] Generate follow-up questions

[0341] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0342] The terminal displays this question to the user.

[0343] Step 6:

[0344] Getting specific answers

[0345] The user responds, "Make it about 2 mm higher."

[0346] The terminal sends this response to the server.

[0347] Step 7:

[0348] Repeated dialogue

[0349] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0350] The user responds, "I'd like the eyes to be a little bigger."

[0351] *This process is repeated for all changes the user desires.

[0352] Step 8:

[0353] User emotion recognition

[0354] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[0355] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[0356] Step 9:

[0357] Emotion-based dialogue adjustment

[0358] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[0359] The device displays these tailored questions or messages to the user.

[0360] Step 10:

[0361] Generating the initial 3D model

[0362] The server generates an initial 3D model based on the collected user preferences.

[0363] The server sends the generated 3D model to the terminal and displays it to the user.

[0364] Step 11:

[0365] Check the 3D model

[0366] The device will present the user with an initial 3D model and ask them to confirm it.

[0367] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[0368] The terminal sends this feedback to the server.

[0369] Step 12:

[0370] Modifying the 3D model

[0371] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0372] This process is repeated until the user is satisfied.

[0373] Step 13:

[0374] Final confirmation and model confirmation

[0375] The user is satisfied with the final 3D model and decides, "This is it."

[0376] The terminal sends this information to the server.

[0377] Step 14:

[0378] Save and share your final model

[0379] The server stores the final 3D model.

[0380] The server shares the finalized 3D model with the practitioner.

[0381] Step 15:

[0382] Treatment plan planning

[0383] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[0384] In this way, by collecting and analyzing the user's wishes at each step, generating and modifying specific 3D models, and further taking the user's emotional state into account using an emotion engine, it is possible to increase user satisfaction and the accuracy of the treatment.

[0385] Example 2

[0386] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] In conventional beauty treatments, it is difficult for users to accurately communicate their desired appearance to the practitioner, resulting in treatments that do not meet the user's expectations. Furthermore, the user's emotional state is not fully taken into consideration, resulting in poor communication quality and lower user satisfaction. Furthermore, the inefficient dialogue-based interaction makes it difficult to quickly and accurately gather detailed specifications for the desired appearance.

[0388] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0389] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for recognizing the user's emotional state and adjusting the dialogue, and means for adjusting the dialogue based on the recognized emotional state. This allows the user's desired appearance to be accurately expressed in a 3D model and information to be efficiently shared with the practitioner. Furthermore, by conducting dialogue taking the user's emotional state into consideration, the quality of communication can be improved and user satisfaction can be increased.

[0390] "Facial image data" is digital image data that records the features and shape of a user's face in detail.

[0391] A "3D model" is a digital model that is reproduced in three dimensions based on image data of a face.

[0392] "Dialogue" is a format in which the user and the system exchange information through a question-and-answer exchange.

[0393] The "means for collecting details of the user's desired appearance" refers to the means used to collect specific details of the user's desired changes to appearance.

[0394] "Means for modifying 3D models" means means for modifying and changing existing 3D models according to the user's desired appearance.

[0395] "Means provided to users and practitioners" refers to the means for displaying and sharing modified 3D models with users and practitioners.

[0396] "Means for recognizing the user's emotional state" refers to means for detecting and analyzing emotions from the user's facial expressions and voice.

[0397] A "means for adjusting a dialogue" is a means for changing the content or manner of a dialogue based on the recognized emotional state of the user.

[0398] The "question generation module" is a module for generating additional specific questions based on the user's answers.

[0399] "Sensors and cameras" are devices for collecting the user's facial expressions and voice.

[0400] "Emotional state analysis" is the process of analyzing a user's emotions based on collected facial and voice data.

[0401] "Feedback" means opinions or requests provided by users.

[0402] A "practitioner" is a professional who performs treatments to make the user's appearance as desired.

[0403] This invention is a system that allows users to accurately communicate their desired appearance to practitioners, improving the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between the user and practitioner. This system includes functions to generate a 3D model based on facial image data, interactively collect details of the user's desired appearance, modify the 3D model based on those details, and provide it to the user and practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[0404] Specifically, the following means are used:

[0405] 1. Image upload and initial settings

[0406] Users upload their facial photos to the system using a smartphone or computer. For example, a user can take a photo of their face with their smartphone and send the image to the system through an application.

[0407] The server receives the uploaded facial photo and stores it in a database. At this time, the image data is sent to the server via an HTTP request and stored in a database management system (e.g., MySQL (registered trademark), PostgreSQL).

[0408] The server uses an image processing module (e.g., OpenCV, TENSORFLOW (registered trademark)) to recognize facial feature points and prepare for generating an initial 3D model. As a specific example, the server uses OpenCV, a Python library, to extract feature points such as the eyes, nose, and mouth from the uploaded image.

[0409] 2. Building an ideal image through dialogue

[0410] The server generates an initial question based on the user's basic information and face photo. For example, it generates an initial question such as "What changes would you like to make?", and the device displays this question to the user.

[0411] The user responds, "I want to make my nose taller," and the terminal sends this response to the server.

[0412] The server generates a detailed question (e.g., "How tall do you want your nose to be?") based on the user's answers, and the device displays the question.

[0413] The user responds, "Please raise it about 2 mm," and the device sends the response to the server.

[0414] This interactive process is repeated for all changes the user desires.

[0415] 3. Dialogue adjustment using emotion engine

[0416] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time, using devices such as a webcam and microphone.

[0417] The server analyzes the user's emotional state using an emotion engine (e.g., Affectiva, Microsoft® Emotion API). For example, the server sends the recognized emotion data to the emotion engine, which analyzes the user's emotional state as "excitement," "stress," "anxiety," etc.

[0418] The server adjusts the dialogue based on the user's emotional state. For example, if the user is feeling stressed, it generates additional questions or messages to help them relax, which the device then displays to the user.

[0419] 4. Creating and checking the 3D model

[0420] The server generates an initial 3D model based on the collected user preferences and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[0421] The server sends the generated 3D model to the terminal, which displays it to the user.

[0422] The user reviews the initial 3D model and provides feedback, such as "Make the nose 1 mm taller and thinner," which is then sent from the device to the server.

[0423] The server modifies the 3D model based on the user's feedback and sends it back to the device, and this process is repeated until the user is satisfied.

[0424] 5. Final confirmation and treatment plan

[0425] The user is satisfied with the final 3D model and decides, "This is it."

[0426] The device sends this information to a server, which stores the final 3D model and sends it to the practitioner (doctor).

[0427] The practitioner will create a specific treatment plan based on the final 3D model.

[0428] This allows users to specify their desired appearance and visualize it as a 3D model, facilitating communication between patients and practitioners and improving the precision of treatment. Furthermore, by using an emotion engine, the system can take into account the user's emotional state, improving the quality of communication and increasing user satisfaction.

[0429] Examples of prompts:

[0430] 1. Prompt: "What facial changes would you like to see?"

[0431] 2. Prompt: "Please tell us more about the changes you would make. For example, make the nose higher and the eyes bigger."

[0432] 3. Prompt: What are the desired dimensions after the change?

[0433] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0434] Step 1:

[0435] Uploading an image

[0436] Users upload a photo of their face to the system from their smartphone or computer.

[0437] Input: Data file of face photos

[0438] Output: Facial photo data uploaded to the system

[0439] Specific operation: When a user takes a photo of their face using the smartphone's camera app and presses the "upload" button through the application, the photo is sent to the server.

[0440] Step 2:

[0441] Image reception and analysis

[0442] The server receives the uploaded facial photos and stores them in a database.

[0443] Input: Uploaded face photo data

[0444] Output: Facial photo data stored in the database

[0445] Specific operation: The server receives the HTTP request and stores the image data in a database (e.g., MySQL, PostgreSQL).

[0446] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial feature points and prepare for generating an initial 3D model.

[0447] Input: Facial photo data stored in the database

[0448] Output: Data containing facial feature points

[0449] Specific operation: The server uses OpenCV to detect facial feature points and extracts the data as components.

[0450] Step 3:

[0451] Generate initial questions

[0452] The server generates initial questions based on the user's basic information and facial photo.

[0453] Input: User's basic information and face photo data

[0454] Output: First question text

[0455] What it does: The server retrieves the user's basic information from the database and uses a text generation engine to generate an initial question such as "What changes would you like to make?"

[0456] Step 4:

[0457] Get user answers

[0458] The terminal displays the generated question to the user.

[0459] Input: First question text

[0460] Output: Question printed to terminal

[0461] What it does: Displays a question in the application's interface and allows the user to enter an answer.

[0462] The user inputs the desired changes (e.g., "I want to make my nose taller").

[0463] Input: The question displayed on the terminal

[0464] Output: User's answer (text data)

[0465] Specific behavior: The user enters the desired appearance changes in the text box and presses the "Submit" button to send the answer to the server.

[0466] Step 5:

[0467] Generate follow-up questions

[0468] The server generates detailed follow-up questions based on the user's answers (e.g., "How tall would you like your nose to be?").

[0469] Input: User's answer (text data)

[0470] Output: Detailed follow-up question text

[0471] Specific actions: Use natural language processing to analyze the user's answers and generate specific instructions or follow-up questions.

[0472] Step 6:

[0473] Getting specific answers

[0474] The terminal displays the generated follow-up questions to the user.

[0475] Input: Detailed follow-up question text

[0476] Output: Additional questions displayed on the terminal

[0477] What it does: It displays the question in the application and allows the user to enter a detailed answer.

[0478] The user enters a detailed response (e.g., "Please make it about 2 mm higher").

[0479] Input: Additional question displayed on the terminal

[0480] Output: Detailed user responses (text data)

[0481] Specific operation: The user enters the answer to the additional question in the text box and presses the submit button to send it to the server.

[0482] Step 7:

[0483] Repeated dialogue

[0484] The server generates additional questions for other changes as well, to obtain more detailed answers (e.g., "What size do you want your eyes to be?").

[0485] Input: Each user's answer (text data)

[0486] Output: Each follow-up question text and its answer

[0487] Specific operation: Repeat the same interactive process to collect all changes desired by the user.

[0488] Step 8:

[0489] User emotion recognition

[0490] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time.

[0491] Input: User's facial expressions and voice data

[0492] Output: Emotional state data

[0493] What it does: It uses a webcam and microphone to capture data in real time and sends it to an emotion recognition algorithm.

[0494] Step 9:

[0495] Emotion-based dialogue adjustment

[0496] The server adjusts the dialogue (e.g., generates follow-up questions or messages to relax the user) based on the emotional state analyzed using an emotion engine (e.g., Affectiva, Microsoft Emotion API).

[0497] Input: Emotional state data

[0498] Output: Adjusted dialogue (questions and messages)

[0499] Specific operation: Dynamically adjust the dialogue content based on the emotional state obtained from the emotion engine.

[0500] Step 10:

[0501] Generating the initial 3D model

[0502] The server generates an initial 3D model based on the collected user preferences, and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[0503] Input: User's desired changes data

[0504] Output: Initial 3D model data

[0505] Specific operation: Run the auto-generation script in the 3D modeling tool to create the initial 3D model.

[0506] Step 11:

[0507] User Review and Feedback

[0508] The device presents the generated initial 3D model to the user and prompts them to confirm it.

[0509] Input: Initial 3D model data

[0510] Output: The initial 3D model displayed on the device.

[0511] What it does: Displays a rendering of a 3D model and provides an interface where users can provide feedback.

[0512] The user provides feedback (e.g., "Make the nose 1 mm higher and thinner").

[0513] Input: Initial 3D model displayed on the device

[0514] Output: User feedback (text data)

[0515] Specific behavior: The user fills in the feedback form and presses the submit button to send the feedback to the server.

[0516] Step 12:

[0517] Modifying the 3D model

[0518] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0519] Input: User feedback (text data)

[0520] Output: Modified 3D model data

[0521] What happens: Run an automated correction script in the 3D modeling tool based on the collected feedback, then create and submit the model again.

[0522] Step 13:

[0523] Final confirmation and treatment plan

[0524] The user is satisfied with the final 3D model and decides, "This is it."

[0525] Input: Final 3D model data

[0526] Output: User's confirmed decision (text data)

[0527] Specific actions: Includes pressing the confirmation button.

[0528] The server stores the final 3D model and sends it to the practitioner.

[0529] Input: User's confirmed intention (text data)

[0530] Output: Final 3D model data

[0531] Specific operation: The data is saved in a database, and notifications are sent to the practitioner's device and data is shared.

[0532] The practitioner will create a specific treatment plan based on the final 3D model.

[0533] Input: Final 3D model data

[0534] Output: Specific treatment plan

[0535] Specific actions: Using the 3D model as a reference, we will develop a specific schedule and procedures for the treatment.

[0536] (Application example 2)

[0537] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0538] Conventional systems make it difficult for users to accurately communicate the details of their desired appearance to the therapist, resulting in lower satisfaction with the treatment. Furthermore, dialogue conducted without taking the user's emotional state into consideration can increase stress and anxiety for the user. Furthermore, the process of generating a 3D model based on the user's wishes is not completed all at once, requiring multiple revisions, increasing the time and effort required for treatment. A system that can resolve these issues and achieve higher user satisfaction and more accurate treatments was needed.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0540] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for analyzing the user's emotions using sensors and cameras, and means for adjusting the dialogue based on the results of the emotion analysis. This allows for accurate visualization of the user's desired appearance and facilitates smooth communication between the user and the practitioner. Furthermore, dialogue that takes the user's emotions into consideration reduces stress and anxiety, allowing for the provision of a more satisfying service.

[0541] "Means for receiving facial image data" refers to a series of functions that allow the server to receive facial image data captured by a user using a device such as a smartphone or camera.

[0542] "Means for generating a 3D model" means a collection of software and algorithms for creating a three-dimensional model of a user's face based on the received facial image data.

[0543] "Means for interactively gathering details of a user's desired appearance" means an interface and related functionality that allows a user to interactively input specific improvements or changes to their desired appearance.

[0544] The "means for modifying the 3D model based on the collected wishes" refers to a function that edits an existing 3D model based on the desired data collected from the user, and performs processing to bring it closer to the user's ideal.

[0545] "Means for providing the modified 3D model to users and practitioners" means an interface and related functionality for displaying and sharing the modified 3D model so that users and practitioners can view it.

[0546] "Means for analyzing user emotions using sensors and cameras" refers to technology that uses sensors and cameras on a device to analyze emotions in real time from a user's facial expressions and voice.

[0547] The "means for adjusting the dialogue based on the emotion analysis results" is a function for generating and adjusting questions and messages to more appropriately progress the dialogue based on the analyzed emotional state of the user.

[0548] This invention is a system that allows users to accurately communicate their desired appearance to practitioners and improve the quality of treatment. This system generates a 3D model based on the user's facial image, collects preferences in an interactive format, and provides the user and practitioner with a modified 3D model based on those preferences. It also takes the user's emotions into account and adjusts the dialogue to improve user satisfaction.

[0549] 1. Image upload and initial settings

[0550] The server receives image data of the face taken by the user using a device such as a smartphone or camera. When the user uploads the image data through the application, the server receives and stores the data. The software used is OpenCV and dlib, which analyze the facial feature points and prepare to generate a 3D model.

[0551] 2. Generating 3D models

[0552] The server uses OpenCV and dlib to generate a 3D model of the user's face based on the received facial image data. This 3D model serves as the basis for incorporating specific changes desired by the user. For example, dlib's Face Points can be used to extract feature points and generate a 3D point cloud based on them.

[0553] 3. Building an ideal image through dialogue

[0554] The device interactively collects details of the user's desired appearance. It includes a question generation module for generating additional, more specific questions based on the user's answers. This interaction is conducted by the user through a smartphone application. For example, if the user inputs, "I want a higher nose," the system adds a more specific question: "How high do you want your nose to be?"

[0555] 4. Dialogue adjustment using emotion engine

[0556] The device uses sensors and cameras to recognize the user's emotions in real time. The server then analyzes the user's emotional state using emotion analysis software such as DeepFace. The dialogue is adjusted based on the analysis results. For example, if the user is feeling stressed, a message is generated saying, "We recommend that you receive treatment in a relaxing environment. Please let us know if you have any concerns."

[0557] 5. Modifying and checking the 3D model

[0558] The server modifies the 3D model based on the user's preferences collected through the interaction. The modified 3D model is then sent back to the device and presented to the user. The user provides feedback through the operation interface, and the server further modifies the 3D model based on that feedback. This process is repeated until the user is satisfied. For example, if the user provides feedback such as "Make the nose 1 mm higher and thinner," the server regenerates the 3D model based on that feedback.

[0559] 6. Final confirmation and treatment plan

[0560] If the user is satisfied with the final 3D model, they can confirm it via their device by clicking "OK." This information is then shared with the practitioner via the server, who will then create a specific treatment plan based on the final 3D model.

[0561] Examples of prompt statements

[0562] "How can I help users relax if they're feeling stressed?"

[0563] "Please tell us more about the user's desired appearance."

[0564] This allows the user's wishes to be accurately visualized, facilitating smooth communication with the therapist. Furthermore, dialogue using an emotion engine can reduce the user's stress and anxiety, improving satisfaction.

[0565] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0566] Step 1:

[0567] Users take a picture of their face using a smartphone or camera and upload the image data to the system through the application, which then sends the image data to the server.

[0568] Input: Face image data

[0569] Output: Uploaded image data

[0570] Step 2:

[0571] The server receives the uploaded facial image data and analyzes the facial feature points using image processing libraries (OpenCV, dlib), and generates an initial 3D model based on the feature points.

[0572] Input: Uploaded image data

[0573] Data processing: facial feature point analysis, 3D model generation

[0574] Output: Initial 3D model

[0575] Step 3:

[0576] The device interactively collects details of the user's desired appearance along with an initial 3D model sent from the server, and the user inputs desired changes through the application.

[0577] Input: Initial 3D model, detailed questions about the user's desired appearance

[0578] Data Entry: User Answers

[0579] Output: User's desired data

[0580] Step 4:

[0581] The server modifies the 3D model based on the collected user data. Specifically, it changes the shape of the 3D model according to the user's feedback.

[0582] Input: User's desired data

[0583] Data calculation: Modifying 3D models

[0584] Output: Modified 3D model

[0585] Step 5:

[0586] The device presents the modified 3D model to the user and prompts them for confirmation, and the user provides additional feedback, which is then sent back to the server. This process is repeated until the user is satisfied.

[0587] Input: Modified 3D model, user feedback

[0588] Data Entry: Additional User Feedback

[0589] Output: Next change request based on feedback

[0590] Step 6:

[0591] The device uses sensors and cameras to monitor the user's facial expressions and voice, analyzing their emotions in real time. The server then adjusts the content of the conversation based on the results of the emotion analysis.

[0592] Input: User's facial expression and voice data

[0593] Data Processing: Real-time Sentiment Analysis

[0594] Output: Adjustment of dialogue content based on emotion analysis results

[0595] Step 7:

[0596] The server sends the 3D model that the user is finally satisfied with and the treatment plan based on it to the practitioner, who then creates a specific treatment plan based on the model.

[0597] Input: Final 3D model, user's final decision

[0598] Output: Sharing treatment plans, planning treatment plans

[0599] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0600] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0601] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0602] [Second embodiment]

[0603] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0604] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0605] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0606] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0607] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0608] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0609] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0610] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0611] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0614] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0615] The present invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. The system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, modifies the 3D model based on the collected information, and provides the 3D model to the user and the practitioner.

[0616] 1. Image upload and initial settings

[0617] 1. Upload an image

[0618] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0619] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[0620] 2. Image Reception and Analysis

[0621] The server receives the uploaded facial photos and stores them in a database.

[0622] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[0623] 2. Building an ideal image through dialogue

[0624] 1. Generating the initial question

[0625] The server prepares the interactive questions and presents the first question to the user.

[0626] For example, ask the user, "What changes would you like to see?"

[0627] 2. Obtaining and elaborating user answers

[0628] The user responds, "I want to make my nose taller."

[0629] The server generates additional, more specific questions based on the answers and presents them to the user.

[0630] For example: Ask specifically, "How tall would you like your nose to be?"

[0631] The user responds, "I'd like it to be about 2 mm higher."

[0632] 3. Iteration

[0633] This interactive question and answer process applies to other facial features as well (eyes, mouth, chin, etc.).

[0634] 3. Creating and checking the 3D model

[0635] 1. Creating the initial 3D model

[0636] The server generates an initial 3D model based on the collected user preferences.

[0637] The server sends the generated 3D model to the terminal and displays it to the user.

[0638] 2. User Review and Feedback

[0639] The device presents the 3D model to the user and prompts them to confirm it.

[0640] The user provides feedback such as, "Make the nose 1 mm higher and thinner."

[0641] The server revises the 3D model based on the feedback and displays it again.

[0642] 4. Sharing and implementing the final model

[0643] 1. Final confirmation and model confirmation

[0644] The user is satisfied with the final 3D model and decides, "This is it."

[0645] The terminal sends this information to the server.

[0646] 2. Save and share your final model

[0647] The server stores the finalized 3D model and sends it to the practitioner (doctor).

[0648] 3. Planning the treatment plan

[0649] The practitioner will create a specific treatment plan based on the final 3D model.

[0650] In this way, by allowing users to concretize their desired appearance and visualize it as a 3D model, communication between the patient and the practitioner can be facilitated and the precision of the treatment can be improved.

[0651] The processing flow will be explained below.

[0652] Step 1:

[0653] Uploading an image

[0654] Users upload a photo of their face to the system from their terminal.

[0655] The terminal sends this facial photo to the server.

[0656] Step 2:

[0657] Image reception and analysis

[0658] The server receives the uploaded facial photos and stores them in a database.

[0659] The server uses an image processing module to extract facial feature points.

[0660] Step 3:

[0661] Generate initial questions

[0662] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0663] The terminal displays this question to the user.

[0664] Step 4:

[0665] Get user answers

[0666] The user responds, "I want to make my nose taller."

[0667] The terminal sends this response to the server.

[0668] Step 5:

[0669] Generate follow-up questions

[0670] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0671] The terminal displays this question to the user.

[0672] Step 6:

[0673] Getting specific answers

[0674] The user responds, "Make it about 2 mm higher."

[0675] The terminal sends this response to the server.

[0676] Step 7:

[0677] Repeated dialogue

[0678] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0679] The user responds, "I'd like the eyes to be a little bigger."

[0680] *This process is repeated for all changes the user desires.

[0681] Step 8:

[0682] Generating the initial 3D model

[0683] The server generates an initial 3D model based on the collected user preferences.

[0684] The server sends the generated 3D model to the terminal.

[0685] Step 9:

[0686] Check the 3D model

[0687] The device displays the initial 3D model to the user.

[0688] The user checks the 3D model and provides feedback such as, "Make the nose 1 mm higher and thinner."

[0689] The terminal sends this feedback to the server.

[0690] Step 10:

[0691] Modifying the 3D model

[0692] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0693] This process is repeated until the user is satisfied.

[0694] Step 11:

[0695] Final confirmation and model confirmation

[0696] The user is satisfied with the final 3D model and decides, "This is it."

[0697] The terminal sends this information to the server.

[0698] Step 12:

[0699] Save and share your final model

[0700] The server stores the final 3D model.

[0701] The server shares the finalized 3D model with the practitioner.

[0702] Step 13:

[0703] Treatment plan planning

[0704] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[0705] In this way, by collecting and analyzing the user's wishes at each step and generating, modifying and providing specific 3D models, it is possible to increase user satisfaction and the accuracy of the treatment.

[0706] Example 1

[0707] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] In conventional cosmetic medicine, it has been difficult for users to accurately communicate their desired appearance to the practitioner. There is a need for a system that can collect detailed information about the specific changes desired by the user and generate and modify a 3D model based on that information. In addition, there is a high need for a system that can repeatedly check and modify the 3D model based on user feedback.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0710] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for including a generative AI model for generating interactive questions, means for generating additional specific questions based on the user's answers to the generated questions, means for repeatedly modifying the 3D model based on the collected preferences, and means for redisplaying the 3D model. This allows the user to collect details of the user's desired appearance and repeatedly confirm and modify them, thereby providing the practitioner with accurate information and improving the accuracy of the treatment.

[0711] "Facial image data" refers to an image file of a user's face that is used by the system to analyze facial features.

[0712] A "3D model" is a computer-generated, three-dimensional digital model that reflects a user's facial features and desired changes.

[0713] "Dialogue" is a form of communication between a person and a system through questions and answers, and is a method of gathering detailed information about the user's wishes.

[0714] A "generative AI model" is a model that uses artificial intelligence technology to generate appropriate questions and answers based on user input, and examples include AI for processing natural language.

[0715] "Feedback" refers to the input of desired changes or modifications made by the user to the system, which the system uses to reflect this and modify the 3D model.

[0716] "User" refers to a person who utilizes the system to input their wishes to change their appearance.

[0717] "Practitioner" refers to a professional who performs cosmetic medical treatments based on the user's wishes.

[0718] This invention is a system that helps users accurately communicate their desired appearance to a practitioner before undergoing cosmetic medical treatment. The system receives facial image data from the user, generates a 3D model based on that data, interactively collects detailed requests, modifies the 3D model based on the collected requests, and provides the results to the user and the practitioner.

[0719] The system uses the following hardware and software:

[0720] Hardware: Servers, devices (smartphones, computers)

[0721] Software: OpenCV (image processing library), Blender (3D modeling software), generative AI models (e.g., GPT-3)

[0722] Users take a photo of their face using their device and upload the image data to the system via a dedicated app or web page. The server receives the image data and stores it in a database. The server then uses an image processing library such as OpenCV to recognize facial feature points, which allows for the generation of an initial 3D model.

[0723] Next, the server uses the generative AI model to create dialogue-style questions and present them to the user. For example, it generates a question such as "What changes would you like to make?" and obtains the user's answer. The user inputs a specific answer such as "I want my nose to be higher." The server then uses the generative AI model again to generate an additional specific question such as "How high do you want your nose to be?" and presents it to the user. All detailed wishes collected through the dialogue with the user are recorded.

[0724] The server uses 3D modeling software such as Blender to generate a 3D model based on the collected preferences and sends the model to the device. The device then displays the 3D model to the user, allowing the user to provide feedback. For example, the user may enter specific feedback such as, "Make the nose 1 mm higher and thinner." The server then modifies the 3D model based on this feedback, sends it back to the device, and displays it to the user. This process is repeated until the user is satisfied.

[0725] Finally, the server stores the 3D model finalized by the user and provides it to the practitioner, who then uses it to create a specific treatment plan. This prevents any discrepancies between the user and the practitioner regarding the desired appearance and improves the accuracy of the treatment.

[0726] As a concrete example, the following exchange occurs:

[0727] Prompt: "What would you like to change?"

[0728] User Answer: "I want to make my nose taller."

[0729] Next prompt: "How tall would you like your nose to be?"

[0730] User answer: "I want it to be about 2mm higher."

[0731] In this way, the present invention provides a system that can facilitate communication between the user and the practitioner and improve the accuracy of treatment by specifically collecting the user's desired appearance and sequentially reflecting it in a three-dimensional model.

[0732] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0733] Step 1: Upload an image

[0734] Users use a device (smartphone or computer) to take a photo of their face and upload the image data to the system via a dedicated app or web page.

[0735] Input: Face image data (JPEG, PNG, etc.)

[0736] Specific behavior: The user takes a photo of their face using the camera app and clicks the upload button to upload the image file via the app or browser.

[0737] Output: Facial image data sent to the system.

[0738] Step 2: Receiving and analyzing images

[0739] The server receives the uploaded facial photo, stores it in a database, and uses an image processing module to recognize facial features (the positions of the eyes, nose, mouth, etc.).

[0740] Input: Image data uploaded by the user

[0741] How it works: The server receives the image data sent in the HTTP request and saves it in storage. It then uses an image processing library such as OpenCV to detect facial feature points.

[0742] Output: Feature point coordinate data and image path stored in database.

[0743] Step 3: Generate initial questions

[0744] The server uses the generative AI model to generate initial interactive questions and present them to the user.

[0745] Input: Feature point coordinate data and user information

[0746] What it does: The server calls a generative AI model (e.g., GPT-3) and generates questions such as, "What changes would you like to make?"

[0747] Output: Initial question.

[0748] Step 4: Obtaining and elaborating user responses

[0749] The user enters an answer to the question, and the server then generates a more specific question based on the answer and presents it to the user.

[0750] Input: Initial question and user's answer

[0751] Specific operation: The user enters an answer (e.g., "I want to make my nose taller") into the input form and submits it. The server uses the generative AI model to generate the next question (e.g., "How tall do you want your nose to be?").

[0752] Output: Detailed follow-up questions.

[0753] Step 5: Iterate

[0754] The dialogue-style question and answer process is repeated for other facial features (eyes, mouth, chin, etc.).

[0755] Input: Previous answers and generated questions

[0756] What happens: The server receives each answer and generates additional questions as needed and presents them to the user.

[0757] Output: A list of questions and answers with all preferences collected in detail.

[0758] Step 6: Creating the initial 3D model

[0759] The server uses three-dimensional modeling software to generate an initial 3D model based on the collected user preferences.

[0760] Input: User's desired details

[0761] Specific operation: The server calls 3D modeling software such as Blender and generates a 3D model based on the user's desired shape.

[0762] Output: Data of the initial 3D model.

[0763] Step 7: User review and feedback

[0764] The device displays the generated 3D model to the user, who then provides feedback.

[0765] Input: Initial 3D model

[0766] What it does: Render and display a 3D model on the user's device, and the user can provide feedback such as "Make the nose 1mm taller and thinner."

[0767] Output: User feedback.

[0768] Step 8: Rework the 3D model

[0769] The server revises the 3D model based on the user's feedback and displays it again, repeating this process until the user is satisfied.

[0770] Input: User feedback

[0771] Specific operation: The server analyzes the feedback, revises the model in the 3D modeling software, and sends the revisited model data back to the user's device.

[0772] Output: The modified 3D model.

[0773] Step 9: Final check and finalize the model

[0774] The user is satisfied with the final 3D model and decides, "This is it."

[0775] Input: Modified 3D model

[0776] Specific behavior: The user makes a final confirmation and clicks the "Confirm" button.

[0777] Output: Model confirmation notice.

[0778] Step 10: Save and share your final model

[0779] The server stores the finalized 3D model and provides it to the practitioner.

[0780] Input: Model confirmation notification and 3D model data

[0781] Specific operation: The server saves the 3D model data in a database and notifies the practitioner via email or a dedicated portal.

[0782] Output: 3D model data provided to the practitioner.

[0783] Step 11: Planning your treatment

[0784] The practitioner will create a specific treatment plan based on the final 3D model.

[0785] Input: Provided 3D model data

[0786] Specific operations: The surgeon simulates surgery based on the provided 3D model and creates a specific treatment plan.

[0787] Output: Treatment plan.

[0788] (Application example 1)

[0789] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0790] In traditional plastic surgery and cosmetic procedures, it has been difficult for patients to accurately communicate their desired appearance to the doctor, resulting in unsatisfactory results. Similarly, in food delivery, it has been difficult for customers to communicate the specific customizations they desire, resulting in unsatisfactory orders. There is a need for a system that can solve these issues and accurately respond to users' requests.

[0791] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0792] In this invention, the server includes means for receiving image data of a face and food, means for generating and modifying a 3D model, means for interactively collecting details of the user's desired appearance and food, and means for modifying the 3D model according to the user's wishes and providing it to the user, a practitioner, or a restaurant, thereby enabling the user to accurately communicate specific customizations of the user's desired appearance and food, and for treatment and cooking to be performed based on those customizations.

[0793] "Facial image data" refers to photographs and image information of the user's face, and is the basic data used to generate a 3D model.

[0794] "Means for generating 3D models" refers to the process and technology for creating three-dimensional model data based on image data of faces or food.

[0795] "A means for interactively gathering details of a user's desired appearance" refers to an interface and system that gathers the specifications and features desired by the user in the form of questions.

[0796] "Means for modifying 3D models based on collected preferences" refers to techniques for modifying or updating existing 3D models based on preferences or feedback collected from users.

[0797] "Means for providing modified 3D models to users and practitioners" refers to a system for displaying or transmitting the completed 3D model to users and practitioners.

[0798] "Food image data" refers to photos and image information of the dishes and ingredients desired by the user, and is the basic data used to generate a 3D model of the dish.

[0799] "Means for generating and modifying 3D food models" refers to technology that creates three-dimensional food model data based on food image data and modifies or updates it.

[0800] "Means for interactively collecting details of the food desired by the user" refers to an interface and system for collecting the specifications and characteristics of the food desired by the user in the form of questions.

[0801] "Means for modifying 3D food models according to user preferences and providing the finalized models to restaurants" refers to a system for modifying 3D food models based on preferences and feedback collected from users and displaying or transmitting the finalized models to restaurants.

[0802] The present invention provides a system that allows users to specifically communicate their desired appearance and food customization details. This system operates between a server and a user terminal, and the process proceeds as follows:

[0803] 1. Image upload and initial settings

[0804] Users take a photo of themselves or a reference image of the dish they want to eat with their smartphone, and then send the image to the server via the application. The server then stores the received image data and begins analyzing it.

[0805] The server uses image processing modules such as OpenCV to analyze facial features and food ingredients, and then generates an initial 3D model. The generated 3D model is then sent to the user's device and displayed to the user.

[0806] 2. Creating ideal appearances and dishes through dialogue

[0807] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The user enters answers, and the server generates additional, more specific questions based on those answers.

[0808] For example, if the user answers "I want a taller nose" or "I want more cheese," the server generates and displays more specific questions such as "How tall do you want your nose to be?" or "What type of cheese would you like more of?"

[0809] 3. Creating and checking the 3D model

[0810] The server generates an initial 3D model based on the user's preferences and sends it to the user's device. The user can review the 3D model and provide feedback as needed. For example, they can provide feedback such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes."

[0811] The server then modifies the 3D model based on this feedback and displays it again to the user, repeating this process until the user is satisfied.

[0812] 4. Sharing the final model and ordering / treatment

[0813] Once the user is satisfied with the final 3D model and confirms it, the device sends this information to the server. The server saves the finalized 3D model and sends it to the therapist or restaurant. The therapist uses this model to create a specific treatment plan, and the restaurant begins cooking based on this model.

[0814] Specific examples and prompts for the generative AI model

[0815] Specific examples

[0816] 1. In the case of cosmetic surgery: The user requests "I want my nose to be straighter and higher," and the 3D model is updated based on that request.

[0817] 2. Food delivery: A user requests "more cheese on my pizza," and the 3D model is updated based on that request.

[0818] Generative AI model prompt

[0819] A user wants to customize a cosmetic procedure or food. Please generate a 3D model using the following instructions:

[0820] 1. Cosmetic procedure: Nose lift (specifically, 2mm higher)

[0821] 2. Cooking: Add more cheese (mozzarella and parmesan)

[0822] Output: Updated 3D model data and its visual representation.

[0823] Using this prompt, the AI ​​model can generate a 3D model that accurately reflects the user's wishes. This system accurately conveys the user's wishes, improving the accuracy of treatment and cooking.

[0824] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0825] Step 1:

[0826] Users take a photo of themselves or an image of the dish they want to eat with their smartphone and upload it to the server through the application. The server receives this image data and stores it in a database. Here, the input is the image file uploaded by the user, and the output is the image data stored on the server.

[0827] Step 2:

[0828] The server analyzes the received image data using an image processing module such as OpenCV. It extracts facial features and food ingredients and generates an initial 3D model based on them. Here, the input is the image data saved in step 1, and the output is the generated initial 3D model data.

[0829] Step 3:

[0830] The server sends the generated initial 3D model to the terminal and displays it to the user. The input is the 3D model data generated in step 2, and the output is the 3D model displayed on the user's terminal.

[0831] Step 4:

[0832] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The input is the user's preference, and the output is the interactive questions.

[0833] Step 5:

[0834] The user answers questions interactively, for example, by inputting specific wishes such as "I want a taller nose" or "I want more cheese." The input is the user's answer, and the output is the answer data sent to the server.

[0835] Step 6:

[0836] The server analyzes the collected response data from the user and generates additional specific questions, such as "How tall do you want your nose to be?" or "Which type of cheese would you like more of?", and presents them to the user again. The input is the user's response data, and the output is the additional questions.

[0837] Step 7:

[0838] The server modifies the 3D model based on the user's detailed wishes and sends it back. For example, creating a 3D model of a pizza with a longer nose or more cheese. The input is the user's wishes, and the output is the modified 3D model data.

[0839] Step 8:

[0840] The user checks the modified 3D model and provides further feedback, such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes." The input is the user's feedback, and the output is the feedback data received by the server.

[0841] Step 9:

[0842] The server revises the 3D model based on this feedback and resends it. This process is repeated until the user is satisfied. The input is the user's feedback data, and the output is the revisited 3D model.

[0843] Step 10:

[0844] When the user is satisfied with the final 3D model and confirms, "This is it," the device sends this information to the server. The input is the user's final decision, and the output is the decision data received by the server.

[0845] Step 11:

[0846] The server stores the finalized 3D model and sends it to the practitioner or restaurant. The practitioner makes a specific treatment plan based on this model, and the restaurant starts cooking based on this model. The input is the finalized 3D model data, and the output is the model data sent to the practitioner or restaurant.

[0847] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0848] This invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between users and physicians. This system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, and modifies the 3D model based on those details, providing it to the user and the practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[0849] 1. Image upload and initial settings

[0850] 1. Upload an image

[0851] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0852] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[0853] 2. Image Reception and Analysis

[0854] The server receives the uploaded facial photos and stores them in a database.

[0855] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[0856] 2. Building an ideal image through dialogue

[0857] 1. Generating the initial question

[0858] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0859] The terminal displays this question to the user.

[0860] 2. Obtaining user answers

[0861] The user responds, "I want to make my nose taller."

[0862] The terminal sends this response to the server.

[0863] 3. Generating follow-up questions

[0864] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0865] The terminal displays this question to the user.

[0866] 4. Getting specific answers

[0867] The user responds, "Make it about 2 mm higher."

[0868] The terminal sends this response to the server.

[0869] 5. Repeated dialogue

[0870] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0871] The user responds, "I'd like the eyes to be a little bigger."

[0872] *This process is repeated for all changes the user desires.

[0873] 3. Dialogue adjustment using emotion engine

[0874] 1. User Emotion Recognition

[0875] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[0876] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[0877] 2. Emotion-based dialogue adjustment

[0878] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[0879] The device displays these tailored questions or messages to the user.

[0880] 4. Creating and checking the 3D model

[0881] 1. Generate the initial 3D model

[0882] The server generates an initial 3D model based on the collected user preferences.

[0883] The server sends the generated 3D model to the terminal and displays it to the user.

[0884] 2. User Review and Feedback

[0885] The device will present the user with an initial 3D model and ask them to confirm it.

[0886] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[0887] The terminal sends this feedback to the server.

[0888] 3. Modifying the 3D model

[0889] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0890] This process is repeated until the user is satisfied.

[0891] 5. Final confirmation and treatment plan

[0892] 1. Final confirmation and model confirmation

[0893] The user is satisfied with the final 3D model and decides, "This is it."

[0894] The terminal sends this information to the server.

[0895] 2. Save and share your final model

[0896] The server saves the final 3D model and sends it to the practitioner (doctor).

[0897] 3. Planning the treatment plan

[0898] The practitioner will create a specific treatment plan based on the final 3D model.

[0899] In this way, by allowing users to specify their desired appearance and visualizing it as a 3D model, communication between patients and practitioners can be facilitated and the precision of treatment can be improved. In addition, by taking the user's emotional state into account using an emotion engine, user satisfaction can be further improved.

[0900] The processing flow will be explained below.

[0901] Step 1:

[0902] Uploading an image

[0903] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[0904] The terminal sends this facial photo to the server.

[0905] Step 2:

[0906] Image reception and analysis

[0907] The server receives the uploaded facial photos and stores them in a database.

[0908] The server uses an image processing module to extract facial feature points.

[0909] Step 3:

[0910] Generate initial questions

[0911] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[0912] The terminal displays this question to the user.

[0913] Step 4:

[0914] Get user answers

[0915] The user responds, "I want to make my nose taller."

[0916] The terminal sends this response to the server.

[0917] Step 5:

[0918] Generate follow-up questions

[0919] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[0920] The terminal displays this question to the user.

[0921] Step 6:

[0922] Getting specific answers

[0923] The user responds, "Make it about 2 mm higher."

[0924] The terminal sends this response to the server.

[0925] Step 7:

[0926] Repeated dialogue

[0927] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[0928] The user responds, "I'd like the eyes to be a little bigger."

[0929] *This process is repeated for all changes the user desires.

[0930] Step 8:

[0931] User emotion recognition

[0932] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[0933] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[0934] Step 9:

[0935] Emotion-based dialogue adjustment

[0936] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[0937] The device displays these tailored questions or messages to the user.

[0938] Step 10:

[0939] Generating the initial 3D model

[0940] The server generates an initial 3D model based on the collected user preferences.

[0941] The server sends the generated 3D model to the terminal and displays it to the user.

[0942] Step 11:

[0943] Check the 3D model

[0944] The device will present the user with an initial 3D model and ask them to confirm it.

[0945] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[0946] The terminal sends this feedback to the server.

[0947] Step 12:

[0948] Modifying the 3D model

[0949] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[0950] This process is repeated until the user is satisfied.

[0951] Step 13:

[0952] Final confirmation and model confirmation

[0953] The user is satisfied with the final 3D model and decides, "This is it."

[0954] The terminal sends this information to the server.

[0955] Step 14:

[0956] Save and share your final model

[0957] The server stores the final 3D model.

[0958] The server shares the finalized 3D model with the practitioner.

[0959] Step 15:

[0960] Treatment plan planning

[0961] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[0962] In this way, by collecting and analyzing the user's wishes at each step, generating and modifying specific 3D models, and further taking the user's emotional state into account using an emotion engine, it is possible to increase user satisfaction and the accuracy of the treatment.

[0963] Example 2

[0964] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0965] In conventional beauty treatments, it is difficult for users to accurately communicate their desired appearance to the practitioner, resulting in treatments that do not meet the user's expectations. Furthermore, the user's emotional state is not fully taken into consideration, resulting in poor communication quality and lower user satisfaction. Furthermore, the inefficient dialogue-based interaction makes it difficult to quickly and accurately gather detailed specifications for the desired appearance.

[0966] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0967] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for recognizing the user's emotional state and adjusting the dialogue, and means for adjusting the dialogue based on the recognized emotional state. This allows the user's desired appearance to be accurately expressed in a 3D model and information to be efficiently shared with the practitioner. Furthermore, by conducting dialogue taking the user's emotional state into consideration, the quality of communication can be improved and user satisfaction can be increased.

[0968] "Facial image data" is digital image data that records the features and shape of a user's face in detail.

[0969] A "3D model" is a digital model that is reproduced in three dimensions based on image data of a face.

[0970] "Dialogue" is a format in which the user and the system exchange information through a question-and-answer exchange.

[0971] The "means for collecting details of the user's desired appearance" refers to the means used to collect specific details of the user's desired changes to appearance.

[0972] "Means for modifying 3D models" means means for modifying and changing existing 3D models according to the user's desired appearance.

[0973] "Means provided to users and practitioners" refers to the means for displaying and sharing modified 3D models with users and practitioners.

[0974] "Means for recognizing the user's emotional state" refers to means for detecting and analyzing emotions from the user's facial expressions and voice.

[0975] A "means for adjusting a dialogue" is a means for changing the content or manner of a dialogue based on the recognized emotional state of the user.

[0976] The "question generation module" is a module for generating additional specific questions based on the user's answers.

[0977] "Sensors and cameras" are devices for collecting the user's facial expressions and voice.

[0978] "Emotional state analysis" is the process of analyzing a user's emotions based on collected facial and voice data.

[0979] "Feedback" means opinions or requests provided by users.

[0980] A "practitioner" is a professional who performs treatments to make the user's appearance as desired.

[0981] This invention is a system that allows users to accurately communicate their desired appearance to practitioners, improving the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between the user and practitioner. This system includes functions to generate a 3D model based on facial image data, interactively collect details of the user's desired appearance, modify the 3D model based on those details, and provide it to the user and practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[0982] Specifically, the following means are used:

[0983] 1. Image upload and initial settings

[0984] Users upload their facial photos to the system using a smartphone or computer. For example, a user can take a photo of their face with their smartphone and send the image to the system through an application.

[0985] The server receives the uploaded face photo and stores it in a database. At this time, the image data is sent to the server via an HTTP request and stored in a database management system (e.g., MySQL, PostgreSQL).

[0986] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial features and prepare for generating an initial 3D model. For example, the server uses the Python library OpenCV to extract features such as the eyes, nose, and mouth from the uploaded image.

[0987] 2. Building an ideal image through dialogue

[0988] The server generates an initial question based on the user's basic information and face photo. For example, it generates an initial question such as "What changes would you like to make?", and the device displays this question to the user.

[0989] The user responds, "I want to make my nose taller," and the terminal sends this response to the server.

[0990] The server generates a detailed question (e.g., "How tall do you want your nose to be?") based on the user's answers, and the device displays the question.

[0991] The user responds, "Please raise it about 2 mm," and the device sends the response to the server.

[0992] This interactive process is repeated for all changes the user desires.

[0993] 3. Dialogue adjustment using emotion engine

[0994] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time, using devices such as a webcam and microphone.

[0995] The server uses an emotion engine (e.g., Affectiva, Microsoft Emotion API) to analyze the user's emotional state. For example, the server sends the recognized emotion data to the emotion engine, which analyzes the emotional state as "excitement," "stress," "anxiety," etc.

[0996] The server adjusts the dialogue based on the user's emotional state. For example, if the user is feeling stressed, it generates additional questions or messages to help them relax, which the device then displays to the user.

[0997] 4. Creating and checking the 3D model

[0998] The server generates an initial 3D model based on the collected user preferences and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[0999] The server sends the generated 3D model to the terminal, which displays it to the user.

[1000] The user reviews the initial 3D model and provides feedback, such as "Make the nose 1 mm taller and thinner," which is then sent from the device to the server.

[1001] The server modifies the 3D model based on the user's feedback and sends it back to the device, and this process is repeated until the user is satisfied.

[1002] 5. Final confirmation and treatment plan

[1003] The user is satisfied with the final 3D model and decides, "This is it."

[1004] The device sends this information to a server, which stores the final 3D model and sends it to the practitioner (doctor).

[1005] The practitioner will create a specific treatment plan based on the final 3D model.

[1006] This allows users to specify their desired appearance and visualize it as a 3D model, facilitating communication between patients and practitioners and improving the precision of treatment. Furthermore, by using an emotion engine, the system can take into account the user's emotional state, improving the quality of communication and increasing user satisfaction.

[1007] Examples of prompts:

[1008] 1. Prompt: "What facial changes would you like to see?"

[1009] 2. Prompt: "Please tell us more about the changes you would make. For example, make the nose higher and the eyes bigger."

[1010] 3. Prompt: What are the desired dimensions after the change?

[1011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1012] Step 1:

[1013] Uploading an image

[1014] Users upload a photo of their face to the system from their smartphone or computer.

[1015] Input: Data file of face photos

[1016] Output: Facial photo data uploaded to the system

[1017] Specific operation: When a user takes a photo of their face using the smartphone's camera app and presses the "upload" button through the application, the photo is sent to the server.

[1018] Step 2:

[1019] Image reception and analysis

[1020] The server receives the uploaded facial photos and stores them in a database.

[1021] Input: Uploaded face photo data

[1022] Output: Facial photo data stored in the database

[1023] Specific operation: The server receives the HTTP request and stores the image data in a database (e.g., MySQL, PostgreSQL).

[1024] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial feature points and prepare for generating an initial 3D model.

[1025] Input: Facial photo data stored in the database

[1026] Output: Data containing facial feature points

[1027] Specific operation: The server uses OpenCV to detect facial feature points and extracts the data as components.

[1028] Step 3:

[1029] Generate initial questions

[1030] The server generates initial questions based on the user's basic information and facial photo.

[1031] Input: User's basic information and face photo data

[1032] Output: First question text

[1033] What it does: The server retrieves the user's basic information from the database and uses a text generation engine to generate an initial question such as "What changes would you like to make?"

[1034] Step 4:

[1035] Get user answers

[1036] The terminal displays the generated question to the user.

[1037] Input: First question text

[1038] Output: Question printed to terminal

[1039] What it does: Displays a question in the application's interface and allows the user to enter an answer.

[1040] The user inputs the desired changes (e.g., "I want to make my nose taller").

[1041] Input: The question displayed on the terminal

[1042] Output: User's answer (text data)

[1043] Specific behavior: The user enters the desired appearance changes in the text box and presses the "Submit" button to send the answer to the server.

[1044] Step 5:

[1045] Generate follow-up questions

[1046] The server generates detailed follow-up questions based on the user's answers (e.g., "How tall would you like your nose to be?").

[1047] Input: User's answer (text data)

[1048] Output: Detailed follow-up question text

[1049] Specific actions: Use natural language processing to analyze the user's answers and generate specific instructions or follow-up questions.

[1050] Step 6:

[1051] Getting specific answers

[1052] The terminal displays the generated follow-up questions to the user.

[1053] Input: Detailed follow-up question text

[1054] Output: Additional questions displayed on the terminal

[1055] What it does: It displays the question in the application and allows the user to enter a detailed answer.

[1056] The user enters a detailed response (e.g., "Please make it about 2 mm higher").

[1057] Input: Additional question displayed on the terminal

[1058] Output: Detailed user responses (text data)

[1059] Specific operation: The user enters the answer to the additional question in the text box and presses the submit button to send it to the server.

[1060] Step 7:

[1061] Repeated dialogue

[1062] The server generates additional questions for other changes as well, to obtain more detailed answers (e.g., "What size do you want your eyes to be?").

[1063] Input: Each user's answer (text data)

[1064] Output: Each follow-up question text and its answer

[1065] Specific operation: Repeat the same interactive process to collect all changes desired by the user.

[1066] Step 8:

[1067] User emotion recognition

[1068] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time.

[1069] Input: User's facial expressions and voice data

[1070] Output: Emotional state data

[1071] What it does: It uses a webcam and microphone to capture data in real time and sends it to an emotion recognition algorithm.

[1072] Step 9:

[1073] Emotion-based dialogue adjustment

[1074] The server adjusts the dialogue (e.g., generates follow-up questions or messages to relax the user) based on the emotional state analyzed using an emotion engine (e.g., Affectiva, Microsoft Emotion API).

[1075] Input: Emotional state data

[1076] Output: Adjusted dialogue (questions and messages)

[1077] Specific operation: Dynamically adjust the dialogue content based on the emotional state obtained from the emotion engine.

[1078] Step 10:

[1079] Generating the initial 3D model

[1080] The server generates an initial 3D model based on the collected user preferences, and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[1081] Input: User's desired changes data

[1082] Output: Initial 3D model data

[1083] Specific operation: Run the auto-generation script in the 3D modeling tool to create the initial 3D model.

[1084] Step 11:

[1085] User Review and Feedback

[1086] The device presents the generated initial 3D model to the user and prompts them to confirm it.

[1087] Input: Initial 3D model data

[1088] Output: The initial 3D model displayed on the device.

[1089] What it does: Displays a rendering of a 3D model and provides an interface where users can provide feedback.

[1090] The user provides feedback (e.g., "Make the nose 1 mm higher and thinner").

[1091] Input: Initial 3D model displayed on the device

[1092] Output: User feedback (text data)

[1093] Specific behavior: The user fills in the feedback form and presses the submit button to send the feedback to the server.

[1094] Step 12:

[1095] Modifying the 3D model

[1096] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1097] Input: User feedback (text data)

[1098] Output: Modified 3D model data

[1099] What happens: Run an automated correction script in the 3D modeling tool based on the collected feedback, then create and submit the model again.

[1100] Step 13:

[1101] Final confirmation and treatment plan

[1102] The user is satisfied with the final 3D model and decides, "This is it."

[1103] Input: Final 3D model data

[1104] Output: User's confirmed decision (text data)

[1105] Specific actions: Includes pressing the confirmation button.

[1106] The server stores the final 3D model and sends it to the practitioner.

[1107] Input: User's confirmed intention (text data)

[1108] Output: Final 3D model data

[1109] Specific operation: The data is saved in a database, and notifications are sent to the practitioner's device and data is shared.

[1110] The practitioner will create a specific treatment plan based on the final 3D model.

[1111] Input: Final 3D model data

[1112] Output: Specific treatment plan

[1113] Specific actions: Using the 3D model as a reference, we will develop a specific schedule and procedures for the treatment.

[1114] (Application example 2)

[1115] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1116] Conventional systems make it difficult for users to accurately communicate the details of their desired appearance to the therapist, resulting in lower satisfaction with the treatment. Furthermore, dialogue conducted without taking the user's emotional state into consideration can increase stress and anxiety for the user. Furthermore, the process of generating a 3D model based on the user's wishes is not completed all at once, requiring multiple revisions, increasing the time and effort required for treatment. A system that can resolve these issues and achieve higher user satisfaction and more accurate treatments was needed.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1118] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for analyzing the user's emotions using sensors and cameras, and means for adjusting the dialogue based on the results of the emotion analysis. This allows for accurate visualization of the user's desired appearance and facilitates smooth communication between the user and the practitioner. Furthermore, dialogue that takes the user's emotions into consideration reduces stress and anxiety, allowing for the provision of a more satisfying service.

[1119] "Means for receiving facial image data" refers to a series of functions that allow the server to receive facial image data captured by a user using a device such as a smartphone or camera.

[1120] "Means for generating a 3D model" means a collection of software and algorithms for creating a three-dimensional model of a user's face based on the received facial image data.

[1121] "Means for interactively gathering details of a user's desired appearance" means an interface and related functionality that allows a user to interactively input specific improvements or changes to their desired appearance.

[1122] The "means for modifying the 3D model based on the collected wishes" refers to a function that edits an existing 3D model based on the desired data collected from the user, and performs processing to bring it closer to the user's ideal.

[1123] "Means for providing the modified 3D model to users and practitioners" means an interface and related functionality for displaying and sharing the modified 3D model so that users and practitioners can view it.

[1124] "Means for analyzing user emotions using sensors and cameras" refers to technology that uses sensors and cameras on a device to analyze emotions in real time from a user's facial expressions and voice.

[1125] The "means for adjusting the dialogue based on the emotion analysis results" is a function for generating and adjusting questions and messages to more appropriately progress the dialogue based on the analyzed emotional state of the user.

[1126] This invention is a system that allows users to accurately communicate their desired appearance to practitioners and improve the quality of treatment. This system generates a 3D model based on the user's facial image, collects preferences in an interactive format, and provides the user and practitioner with a modified 3D model based on those preferences. It also takes the user's emotions into account and adjusts the dialogue to improve user satisfaction.

[1127] 1. Image upload and initial settings

[1128] The server receives image data of the face taken by the user using a device such as a smartphone or camera. When the user uploads the image data through the application, the server receives and stores the data. The software used is OpenCV and dlib, which analyze the facial feature points and prepare to generate a 3D model.

[1129] 2. Generating 3D models

[1130] The server uses OpenCV and dlib to generate a 3D model of the user's face based on the received facial image data. This 3D model serves as the basis for incorporating specific changes desired by the user. For example, dlib's Face Points can be used to extract feature points and generate a 3D point cloud based on them.

[1131] 3. Building an ideal image through dialogue

[1132] The device interactively collects details of the user's desired appearance. It includes a question generation module for generating additional, more specific questions based on the user's answers. This interaction is conducted by the user through a smartphone application. For example, if the user inputs, "I want a higher nose," the system adds a more specific question: "How high do you want your nose to be?"

[1133] 4. Dialogue adjustment using emotion engine

[1134] The device uses sensors and cameras to recognize the user's emotions in real time. The server then analyzes the user's emotional state using emotion analysis software such as DeepFace. The dialogue is adjusted based on the analysis results. For example, if the user is feeling stressed, a message is generated saying, "We recommend that you receive treatment in a relaxing environment. Please let us know if you have any concerns."

[1135] 5. Modifying and checking the 3D model

[1136] The server modifies the 3D model based on the user's preferences collected through the interaction. The modified 3D model is then sent back to the device and presented to the user. The user provides feedback through the operation interface, and the server further modifies the 3D model based on that feedback. This process is repeated until the user is satisfied. For example, if the user provides feedback such as "Make the nose 1 mm higher and thinner," the server regenerates the 3D model based on that feedback.

[1137] 6. Final confirmation and treatment plan

[1138] If the user is satisfied with the final 3D model, they can confirm it via their device by clicking "OK." This information is then shared with the practitioner via the server, who will then create a specific treatment plan based on the final 3D model.

[1139] Examples of prompt statements

[1140] "How can I help users relax if they're feeling stressed?"

[1141] "Please tell us more about the user's desired appearance."

[1142] This allows the user's wishes to be accurately visualized, facilitating smooth communication with the therapist. Furthermore, dialogue using an emotion engine can reduce the user's stress and anxiety, improving satisfaction.

[1143] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1144] Step 1:

[1145] Users take a picture of their face using a smartphone or camera and upload the image data to the system through the application, which then sends the image data to the server.

[1146] Input: Face image data

[1147] Output: Uploaded image data

[1148] Step 2:

[1149] The server receives the uploaded facial image data and analyzes the facial feature points using image processing libraries (OpenCV, dlib), and generates an initial 3D model based on the feature points.

[1150] Input: Uploaded image data

[1151] Data processing: facial feature point analysis, 3D model generation

[1152] Output: Initial 3D model

[1153] Step 3:

[1154] The device interactively collects details of the user's desired appearance along with an initial 3D model sent from the server, and the user inputs desired changes through the application.

[1155] Input: Initial 3D model, detailed questions about the user's desired appearance

[1156] Data Entry: User Answers

[1157] Output: User's desired data

[1158] Step 4:

[1159] The server modifies the 3D model based on the collected user data. Specifically, it changes the shape of the 3D model according to the user's feedback.

[1160] Input: User's desired data

[1161] Data calculation: Modifying 3D models

[1162] Output: Modified 3D model

[1163] Step 5:

[1164] The device presents the modified 3D model to the user and prompts them for confirmation, and the user provides additional feedback, which is then sent back to the server. This process is repeated until the user is satisfied.

[1165] Input: Modified 3D model, user feedback

[1166] Data Entry: Additional User Feedback

[1167] Output: Next change request based on feedback

[1168] Step 6:

[1169] The device uses sensors and cameras to monitor the user's facial expressions and voice, analyzing their emotions in real time. The server then adjusts the content of the conversation based on the results of the emotion analysis.

[1170] Input: User's facial expression and voice data

[1171] Data Processing: Real-time Sentiment Analysis

[1172] Output: Adjustment of dialogue content based on emotion analysis results

[1173] Step 7:

[1174] The server sends the 3D model that the user is finally satisfied with and the treatment plan based on it to the practitioner, who then creates a specific treatment plan based on the model.

[1175] Input: Final 3D model, user's final decision

[1176] Output: Sharing treatment plans, planning treatment plans

[1177] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1178] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1179] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1180] [Third embodiment]

[1181] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1182] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1183] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1184] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1185] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1186] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1187] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1188] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1189] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1190] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1191] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1192] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1193] The present invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. The system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, modifies the 3D model based on the collected information, and provides the 3D model to the user and the practitioner.

[1194] 1. Image upload and initial settings

[1195] 1. Upload an image

[1196] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[1197] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[1198] 2. Image Reception and Analysis

[1199] The server receives the uploaded facial photos and stores them in a database.

[1200] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[1201] 2. Building an ideal image through dialogue

[1202] 1. Generating the initial question

[1203] The server prepares the interactive questions and presents the first question to the user.

[1204] For example, ask the user, "What changes would you like to see?"

[1205] 2. Obtaining and elaborating user answers

[1206] The user responds, "I want to make my nose taller."

[1207] The server generates additional, more specific questions based on the answers and presents them to the user.

[1208] For example: Ask specifically, "How tall would you like your nose to be?"

[1209] The user responds, "I'd like it to be about 2 mm higher."

[1210] 3. Iteration

[1211] This interactive question and answer process applies to other facial features as well (eyes, mouth, chin, etc.).

[1212] 3. Creating and checking the 3D model

[1213] 1. Creating the initial 3D model

[1214] The server generates an initial 3D model based on the collected user preferences.

[1215] The server sends the generated 3D model to the terminal and displays it to the user.

[1216] 2. User Review and Feedback

[1217] The device presents the 3D model to the user and prompts them to confirm it.

[1218] The user provides feedback such as, "Make the nose 1 mm higher and thinner."

[1219] The server revises the 3D model based on the feedback and displays it again.

[1220] 4. Sharing and implementing the final model

[1221] 1. Final confirmation and model confirmation

[1222] The user is satisfied with the final 3D model and decides, "This is it."

[1223] The terminal sends this information to the server.

[1224] 2. Save and share your final model

[1225] The server stores the finalized 3D model and sends it to the practitioner (doctor).

[1226] 3. Planning the treatment plan

[1227] The practitioner will create a specific treatment plan based on the final 3D model.

[1228] In this way, by allowing users to concretize their desired appearance and visualize it as a 3D model, communication between the patient and the practitioner can be facilitated and the precision of the treatment can be improved.

[1229] The processing flow will be explained below.

[1230] Step 1:

[1231] Uploading an image

[1232] Users upload a photo of their face to the system from their terminal.

[1233] The terminal sends this facial photo to the server.

[1234] Step 2:

[1235] Image reception and analysis

[1236] The server receives the uploaded facial photos and stores them in a database.

[1237] The server uses an image processing module to extract facial feature points.

[1238] Step 3:

[1239] Generate initial questions

[1240] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[1241] The terminal displays this question to the user.

[1242] Step 4:

[1243] Get user answers

[1244] The user responds, "I want to make my nose taller."

[1245] The terminal sends this response to the server.

[1246] Step 5:

[1247] Generate follow-up questions

[1248] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[1249] The terminal displays this question to the user.

[1250] Step 6:

[1251] Getting specific answers

[1252] The user responds, "Make it about 2 mm higher."

[1253] The terminal sends this response to the server.

[1254] Step 7:

[1255] Repeated dialogue

[1256] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[1257] The user responds, "I'd like the eyes to be a little bigger."

[1258] *This process is repeated for all changes the user desires.

[1259] Step 8:

[1260] Generating the initial 3D model

[1261] The server generates an initial 3D model based on the collected user preferences.

[1262] The server sends the generated 3D model to the terminal.

[1263] Step 9:

[1264] Check the 3D model

[1265] The device displays the initial 3D model to the user.

[1266] The user checks the 3D model and provides feedback such as, "Make the nose 1 mm higher and thinner."

[1267] The terminal sends this feedback to the server.

[1268] Step 10:

[1269] Modifying the 3D model

[1270] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1271] This process is repeated until the user is satisfied.

[1272] Step 11:

[1273] Final confirmation and model confirmation

[1274] The user is satisfied with the final 3D model and decides, "This is it."

[1275] The terminal sends this information to the server.

[1276] Step 12:

[1277] Save and share your final model

[1278] The server stores the final 3D model.

[1279] The server shares the finalized 3D model with the practitioner.

[1280] Step 13:

[1281] Treatment plan planning

[1282] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[1283] In this way, by collecting and analyzing the user's wishes at each step and generating, modifying and providing specific 3D models, it is possible to increase user satisfaction and the accuracy of the treatment.

[1284] Example 1

[1285] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] In conventional cosmetic medicine, it has been difficult for users to accurately communicate their desired appearance to the practitioner. There is a need for a system that can collect detailed information about the specific changes desired by the user and generate and modify a 3D model based on that information. In addition, there is a high need for a system that can repeatedly check and modify the 3D model based on user feedback.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1288] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for including a generative AI model for generating interactive questions, means for generating additional specific questions based on the user's answers to the generated questions, means for repeatedly modifying the 3D model based on the collected preferences, and means for redisplaying the 3D model. This allows the user to collect details of the user's desired appearance and repeatedly confirm and modify them, thereby providing the practitioner with accurate information and improving the accuracy of the treatment.

[1289] "Facial image data" refers to an image file of a user's face that is used by the system to analyze facial features.

[1290] A "3D model" is a computer-generated, three-dimensional digital model that reflects a user's facial features and desired changes.

[1291] "Dialogue" is a form of communication between a person and a system through questions and answers, and is a method of gathering detailed information about the user's wishes.

[1292] A "generative AI model" is a model that uses artificial intelligence technology to generate appropriate questions and answers based on user input, and examples include AI for processing natural language.

[1293] "Feedback" refers to the input of desired changes or modifications made by the user to the system, which the system uses to reflect this and modify the 3D model.

[1294] "User" refers to a person who utilizes the system to input their wishes to change their appearance.

[1295] "Practitioner" refers to a professional who performs cosmetic medical treatments based on the user's wishes.

[1296] This invention is a system that helps users accurately communicate their desired appearance to a practitioner before undergoing cosmetic medical treatment. The system receives facial image data from the user, generates a 3D model based on that data, interactively collects detailed requests, modifies the 3D model based on the collected requests, and provides the results to the user and the practitioner.

[1297] The system uses the following hardware and software:

[1298] Hardware: Servers, devices (smartphones, computers)

[1299] Software: OpenCV (image processing library), Blender (3D modeling software), generative AI models (e.g., GPT-3)

[1300] Users take a photo of their face using their device and upload the image data to the system via a dedicated app or web page. The server receives the image data and stores it in a database. The server then uses an image processing library such as OpenCV to recognize facial feature points, which allows for the generation of an initial 3D model.

[1301] Next, the server uses the generative AI model to create dialogue-style questions and present them to the user. For example, it generates a question such as "What changes would you like to make?" and obtains the user's answer. The user inputs a specific answer such as "I want my nose to be higher." The server then uses the generative AI model again to generate an additional specific question such as "How high do you want your nose to be?" and presents it to the user. All detailed wishes collected through the dialogue with the user are recorded.

[1302] The server uses 3D modeling software such as Blender to generate a 3D model based on the collected preferences and sends the model to the device. The device then displays the 3D model to the user, allowing the user to provide feedback. For example, the user may enter specific feedback such as, "Make the nose 1 mm higher and thinner." The server then modifies the 3D model based on this feedback, sends it back to the device, and displays it to the user. This process is repeated until the user is satisfied.

[1303] Finally, the server stores the 3D model finalized by the user and provides it to the practitioner, who then uses it to create a specific treatment plan. This prevents any discrepancies between the user and the practitioner regarding the desired appearance and improves the accuracy of the treatment.

[1304] As a concrete example, the following exchange occurs:

[1305] Prompt: "What would you like to change?"

[1306] User Answer: "I want to make my nose taller."

[1307] Next prompt: "How tall would you like your nose to be?"

[1308] User answer: "I want it to be about 2mm higher."

[1309] In this way, the present invention provides a system that can facilitate communication between the user and the practitioner and improve the accuracy of treatment by specifically collecting the user's desired appearance and sequentially reflecting it in a three-dimensional model.

[1310] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1311] Step 1: Upload an image

[1312] Users use a device (smartphone or computer) to take a photo of their face and upload the image data to the system via a dedicated app or web page.

[1313] Input: Face image data (JPEG, PNG, etc.)

[1314] Specific behavior: The user takes a photo of their face using the camera app and clicks the upload button to upload the image file via the app or browser.

[1315] Output: Facial image data sent to the system.

[1316] Step 2: Receiving and analyzing images

[1317] The server receives the uploaded facial photo, stores it in a database, and uses an image processing module to recognize facial features (the positions of the eyes, nose, mouth, etc.).

[1318] Input: Image data uploaded by the user

[1319] How it works: The server receives the image data sent in the HTTP request and saves it in storage. It then uses an image processing library such as OpenCV to detect facial feature points.

[1320] Output: Feature point coordinate data and image path stored in database.

[1321] Step 3: Generate initial questions

[1322] The server uses the generative AI model to generate initial interactive questions and present them to the user.

[1323] Input: Feature point coordinate data and user information

[1324] What it does: The server calls a generative AI model (e.g., GPT-3) and generates questions such as, "What changes would you like to make?"

[1325] Output: Initial question.

[1326] Step 4: Obtaining and elaborating user responses

[1327] The user enters an answer to the question, and the server then generates a more specific question based on the answer and presents it to the user.

[1328] Input: Initial question and user's answer

[1329] Specific operation: The user enters an answer (e.g., "I want to make my nose taller") into the input form and submits it. The server uses the generative AI model to generate the next question (e.g., "How tall do you want your nose to be?").

[1330] Output: Detailed follow-up questions.

[1331] Step 5: Iterate

[1332] The dialogue-style question and answer process is repeated for other facial features (eyes, mouth, chin, etc.).

[1333] Input: Previous answers and generated questions

[1334] What happens: The server receives each answer and generates additional questions as needed and presents them to the user.

[1335] Output: A list of questions and answers with all preferences collected in detail.

[1336] Step 6: Creating the initial 3D model

[1337] The server uses three-dimensional modeling software to generate an initial 3D model based on the collected user preferences.

[1338] Input: User's desired details

[1339] Specific operation: The server calls 3D modeling software such as Blender and generates a 3D model based on the user's desired shape.

[1340] Output: Data of the initial 3D model.

[1341] Step 7: User review and feedback

[1342] The device displays the generated 3D model to the user, who then provides feedback.

[1343] Input: Initial 3D model

[1344] What it does: Render and display a 3D model on the user's device, and the user can provide feedback such as "Make the nose 1mm taller and thinner."

[1345] Output: User feedback.

[1346] Step 8: Rework the 3D model

[1347] The server revises the 3D model based on the user's feedback and displays it again, repeating this process until the user is satisfied.

[1348] Input: User feedback

[1349] Specific operation: The server analyzes the feedback, revises the model in the 3D modeling software, and sends the revisited model data back to the user's device.

[1350] Output: The modified 3D model.

[1351] Step 9: Final check and finalize the model

[1352] The user is satisfied with the final 3D model and decides, "This is it."

[1353] Input: Modified 3D model

[1354] Specific behavior: The user makes a final confirmation and clicks the "Confirm" button.

[1355] Output: Model confirmation notice.

[1356] Step 10: Save and share your final model

[1357] The server stores the finalized 3D model and provides it to the practitioner.

[1358] Input: Model confirmation notification and 3D model data

[1359] Specific operation: The server saves the 3D model data in a database and notifies the practitioner via email or a dedicated portal.

[1360] Output: 3D model data provided to the practitioner.

[1361] Step 11: Planning your treatment

[1362] The practitioner will create a specific treatment plan based on the final 3D model.

[1363] Input: Provided 3D model data

[1364] Specific operations: The surgeon simulates surgery based on the provided 3D model and creates a specific treatment plan.

[1365] Output: Treatment plan.

[1366] (Application example 1)

[1367] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1368] In traditional plastic surgery and cosmetic procedures, it has been difficult for patients to accurately communicate their desired appearance to the doctor, resulting in unsatisfactory results. Similarly, in food delivery, it has been difficult for customers to communicate the specific customizations they desire, resulting in unsatisfactory orders. There is a need for a system that can solve these issues and accurately respond to users' requests.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1370] In this invention, the server includes means for receiving image data of a face and food, means for generating and modifying a 3D model, means for interactively collecting details of the user's desired appearance and food, and means for modifying the 3D model according to the user's wishes and providing it to the user, a practitioner, or a restaurant, thereby enabling the user to accurately communicate specific customizations of the user's desired appearance and food, and for treatment and cooking to be performed based on those customizations.

[1371] "Facial image data" refers to photographs and image information of the user's face, and is the basic data used to generate a 3D model.

[1372] "Means for generating 3D models" refers to the process and technology for creating three-dimensional model data based on image data of faces or food.

[1373] "A means for interactively gathering details of a user's desired appearance" refers to an interface and system that gathers the specifications and features desired by the user in the form of questions.

[1374] "Means for modifying 3D models based on collected preferences" refers to techniques for modifying or updating existing 3D models based on preferences or feedback collected from users.

[1375] "Means for providing modified 3D models to users and practitioners" refers to a system for displaying or transmitting the completed 3D model to users and practitioners.

[1376] "Food image data" refers to photos and image information of the dishes and ingredients desired by the user, and is the basic data used to generate a 3D model of the dish.

[1377] "Means for generating and modifying 3D food models" refers to technology that creates three-dimensional food model data based on food image data and modifies or updates it.

[1378] "Means for interactively collecting details of the food desired by the user" refers to an interface and system for collecting the specifications and characteristics of the food desired by the user in the form of questions.

[1379] "Means for modifying 3D food models according to user preferences and providing the finalized models to restaurants" refers to a system for modifying 3D food models based on preferences and feedback collected from users and displaying or transmitting the finalized models to restaurants.

[1380] The present invention provides a system that allows users to specifically communicate their desired appearance and food customization details. This system operates between a server and a user terminal, and the process proceeds as follows:

[1381] 1. Image upload and initial settings

[1382] Users take a photo of themselves or a reference image of the dish they want to eat with their smartphone, and then send the image to the server via the application. The server then stores the received image data and begins analyzing it.

[1383] The server uses image processing modules such as OpenCV to analyze facial features and food ingredients, and then generates an initial 3D model. The generated 3D model is then sent to the user's device and displayed to the user.

[1384] 2. Creating ideal appearances and dishes through dialogue

[1385] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The user enters answers, and the server generates additional, more specific questions based on those answers.

[1386] For example, if the user answers "I want a taller nose" or "I want more cheese," the server generates and displays more specific questions such as "How tall do you want your nose to be?" or "What type of cheese would you like more of?"

[1387] 3. Creating and checking the 3D model

[1388] The server generates an initial 3D model based on the user's preferences and sends it to the user's device. The user can review the 3D model and provide feedback as needed. For example, they can provide feedback such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes."

[1389] The server then modifies the 3D model based on this feedback and displays it again to the user, repeating this process until the user is satisfied.

[1390] 4. Sharing the final model and ordering / treatment

[1391] Once the user is satisfied with the final 3D model and confirms it, the device sends this information to the server. The server saves the finalized 3D model and sends it to the therapist or restaurant. The therapist uses this model to create a specific treatment plan, and the restaurant begins cooking based on this model.

[1392] Specific examples and prompts for the generative AI model

[1393] Specific examples

[1394] 1. In the case of cosmetic surgery: The user requests "I want my nose to be straighter and higher," and the 3D model is updated based on that request.

[1395] 2. Food delivery: A user requests "more cheese on my pizza," and the 3D model is updated based on that request.

[1396] Generative AI model prompt

[1397] A user wants to customize a cosmetic procedure or food. Please generate a 3D model using the following instructions:

[1398] 1. Cosmetic procedure: Nose lift (specifically, 2mm higher)

[1399] 2. Cooking: Add more cheese (mozzarella and parmesan)

[1400] Output: Updated 3D model data and its visual representation.

[1401] Using this prompt, the AI ​​model can generate a 3D model that accurately reflects the user's wishes. This system accurately conveys the user's wishes, improving the accuracy of treatment and cooking.

[1402] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1403] Step 1:

[1404] Users take a photo of themselves or an image of the dish they want to eat with their smartphone and upload it to the server through the application. The server receives this image data and stores it in a database. Here, the input is the image file uploaded by the user, and the output is the image data stored on the server.

[1405] Step 2:

[1406] The server analyzes the received image data using an image processing module such as OpenCV. It extracts facial features and food ingredients and generates an initial 3D model based on them. Here, the input is the image data saved in step 1, and the output is the generated initial 3D model data.

[1407] Step 3:

[1408] The server sends the generated initial 3D model to the terminal and displays it to the user. The input is the 3D model data generated in step 2, and the output is the 3D model displayed on the user's terminal.

[1409] Step 4:

[1410] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The input is the user's preference, and the output is the interactive questions.

[1411] Step 5:

[1412] The user answers questions interactively, for example, by inputting specific wishes such as "I want a taller nose" or "I want more cheese." The input is the user's answer, and the output is the answer data sent to the server.

[1413] Step 6:

[1414] The server analyzes the collected response data from the user and generates additional specific questions, such as "How tall do you want your nose to be?" or "Which type of cheese would you like more of?", and presents them to the user again. The input is the user's response data, and the output is the additional questions.

[1415] Step 7:

[1416] The server modifies the 3D model based on the user's detailed wishes and sends it back. For example, creating a 3D model of a pizza with a longer nose or more cheese. The input is the user's wishes, and the output is the modified 3D model data.

[1417] Step 8:

[1418] The user checks the modified 3D model and provides further feedback, such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes." The input is the user's feedback, and the output is the feedback data received by the server.

[1419] Step 9:

[1420] The server revises the 3D model based on this feedback and resends it. This process is repeated until the user is satisfied. The input is the user's feedback data, and the output is the revisited 3D model.

[1421] Step 10:

[1422] When the user is satisfied with the final 3D model and confirms, "This is it," the device sends this information to the server. The input is the user's final decision, and the output is the decision data received by the server.

[1423] Step 11:

[1424] The server stores the finalized 3D model and sends it to the practitioner or restaurant. The practitioner makes a specific treatment plan based on this model, and the restaurant starts cooking based on this model. The input is the finalized 3D model data, and the output is the model data sent to the practitioner or restaurant.

[1425] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1426] This invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between users and physicians. This system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, and modifies the 3D model based on those details, providing it to the user and the practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[1427] 1. Image upload and initial settings

[1428] 1. Upload an image

[1429] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[1430] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[1431] 2. Image Reception and Analysis

[1432] The server receives the uploaded facial photos and stores them in a database.

[1433] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[1434] 2. Building an ideal image through dialogue

[1435] 1. Generating the initial question

[1436] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[1437] The terminal displays this question to the user.

[1438] 2. Obtaining user answers

[1439] The user responds, "I want to make my nose taller."

[1440] The terminal sends this response to the server.

[1441] 3. Generating follow-up questions

[1442] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[1443] The terminal displays this question to the user.

[1444] 4. Getting specific answers

[1445] The user responds, "Make it about 2 mm higher."

[1446] The terminal sends this response to the server.

[1447] 5. Repeated dialogue

[1448] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[1449] The user responds, "I'd like the eyes to be a little bigger."

[1450] *This process is repeated for all changes the user desires.

[1451] 3. Dialogue adjustment using emotion engine

[1452] 1. User Emotion Recognition

[1453] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[1454] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[1455] 2. Emotion-based dialogue adjustment

[1456] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[1457] The device displays these tailored questions or messages to the user.

[1458] 4. Creating and checking the 3D model

[1459] 1. Generate the initial 3D model

[1460] The server generates an initial 3D model based on the collected user preferences.

[1461] The server sends the generated 3D model to the terminal and displays it to the user.

[1462] 2. User Review and Feedback

[1463] The device will present the user with an initial 3D model and ask them to confirm it.

[1464] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[1465] The terminal sends this feedback to the server.

[1466] 3. Modifying the 3D model

[1467] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1468] This process is repeated until the user is satisfied.

[1469] 5. Final confirmation and treatment plan

[1470] 1. Final confirmation and model confirmation

[1471] The user is satisfied with the final 3D model and decides, "This is it."

[1472] The terminal sends this information to the server.

[1473] 2. Save and share your final model

[1474] The server saves the final 3D model and sends it to the practitioner (doctor).

[1475] 3. Planning the treatment plan

[1476] The practitioner will create a specific treatment plan based on the final 3D model.

[1477] In this way, by allowing users to specify their desired appearance and visualizing it as a 3D model, communication between patients and practitioners can be facilitated and the precision of treatment can be improved. In addition, by taking the user's emotional state into account using an emotion engine, user satisfaction can be further improved.

[1478] The processing flow will be explained below.

[1479] Step 1:

[1480] Uploading an image

[1481] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[1482] The terminal sends this facial photo to the server.

[1483] Step 2:

[1484] Image reception and analysis

[1485] The server receives the uploaded facial photos and stores them in a database.

[1486] The server uses an image processing module to extract facial feature points.

[1487] Step 3:

[1488] Generate initial questions

[1489] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[1490] The terminal displays this question to the user.

[1491] Step 4:

[1492] Get user answers

[1493] The user responds, "I want to make my nose taller."

[1494] The terminal sends this response to the server.

[1495] Step 5:

[1496] Generate follow-up questions

[1497] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[1498] The terminal displays this question to the user.

[1499] Step 6:

[1500] Getting specific answers

[1501] The user responds, "Make it about 2 mm higher."

[1502] The terminal sends this response to the server.

[1503] Step 7:

[1504] Repeated dialogue

[1505] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[1506] The user responds, "I'd like the eyes to be a little bigger."

[1507] *This process is repeated for all changes the user desires.

[1508] Step 8:

[1509] User emotion recognition

[1510] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[1511] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[1512] Step 9:

[1513] Emotion-based dialogue adjustment

[1514] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[1515] The device displays these tailored questions or messages to the user.

[1516] Step 10:

[1517] Generating the initial 3D model

[1518] The server generates an initial 3D model based on the collected user preferences.

[1519] The server sends the generated 3D model to the terminal and displays it to the user.

[1520] Step 11:

[1521] Check the 3D model

[1522] The device will present the user with an initial 3D model and ask them to confirm it.

[1523] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[1524] The terminal sends this feedback to the server.

[1525] Step 12:

[1526] Modifying the 3D model

[1527] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1528] This process is repeated until the user is satisfied.

[1529] Step 13:

[1530] Final confirmation and model confirmation

[1531] The user is satisfied with the final 3D model and decides, "This is it."

[1532] The terminal sends this information to the server.

[1533] Step 14:

[1534] Save and share your final model

[1535] The server stores the final 3D model.

[1536] The server shares the finalized 3D model with the practitioner.

[1537] Step 15:

[1538] Treatment plan planning

[1539] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[1540] In this way, by collecting and analyzing the user's wishes at each step, generating and modifying specific 3D models, and further taking the user's emotional state into account using an emotion engine, it is possible to increase user satisfaction and the accuracy of the treatment.

[1541] Example 2

[1542] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1543] In conventional beauty treatments, it is difficult for users to accurately communicate their desired appearance to the practitioner, resulting in treatments that do not meet the user's expectations. Furthermore, the user's emotional state is not fully taken into consideration, resulting in poor communication quality and lower user satisfaction. Furthermore, the inefficient dialogue-based interaction makes it difficult to quickly and accurately gather detailed specifications for the desired appearance.

[1544] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1545] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for recognizing the user's emotional state and adjusting the dialogue, and means for adjusting the dialogue based on the recognized emotional state. This allows the user's desired appearance to be accurately expressed in a 3D model and information to be efficiently shared with the practitioner. Furthermore, by conducting dialogue taking the user's emotional state into consideration, the quality of communication can be improved and user satisfaction can be increased.

[1546] "Facial image data" is digital image data that records the features and shape of a user's face in detail.

[1547] A "3D model" is a digital model that is reproduced in three dimensions based on image data of a face.

[1548] "Dialogue" is a format in which the user and the system exchange information through a question-and-answer exchange.

[1549] The "means for collecting details of the user's desired appearance" refers to the means used to collect specific details of the user's desired changes to appearance.

[1550] "Means for modifying 3D models" means means for modifying and changing existing 3D models according to the user's desired appearance.

[1551] "Means provided to users and practitioners" refers to the means for displaying and sharing modified 3D models with users and practitioners.

[1552] "Means for recognizing the user's emotional state" refers to means for detecting and analyzing emotions from the user's facial expressions and voice.

[1553] A "means for adjusting a dialogue" is a means for changing the content or manner of a dialogue based on the recognized emotional state of the user.

[1554] The "question generation module" is a module for generating additional specific questions based on the user's answers.

[1555] "Sensors and cameras" are devices for collecting the user's facial expressions and voice.

[1556] "Emotional state analysis" is the process of analyzing a user's emotions based on collected facial and voice data.

[1557] "Feedback" means opinions or requests provided by users.

[1558] A "practitioner" is a professional who performs treatments to make the user's appearance as desired.

[1559] This invention is a system that allows users to accurately communicate their desired appearance to practitioners, improving the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between the user and practitioner. This system includes functions to generate a 3D model based on facial image data, interactively collect details of the user's desired appearance, modify the 3D model based on those details, and provide it to the user and practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[1560] Specifically, the following means are used:

[1561] 1. Image upload and initial settings

[1562] Users upload their facial photos to the system using a smartphone or computer. For example, a user can take a photo of their face with their smartphone and send the image to the system through an application.

[1563] The server receives the uploaded face photo and stores it in a database. At this time, the image data is sent to the server via an HTTP request and stored in a database management system (e.g., MySQL, PostgreSQL).

[1564] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial features and prepare for generating an initial 3D model. For example, the server uses the Python library OpenCV to extract features such as the eyes, nose, and mouth from the uploaded image.

[1565] 2. Building an ideal image through dialogue

[1566] The server generates an initial question based on the user's basic information and face photo. For example, it generates an initial question such as "What changes would you like to make?", and the device displays this question to the user.

[1567] The user responds, "I want to make my nose taller," and the terminal sends this response to the server.

[1568] The server generates a detailed question (e.g., "How tall do you want your nose to be?") based on the user's answers, and the device displays the question.

[1569] The user responds, "Please raise it about 2 mm," and the device sends the response to the server.

[1570] This interactive process is repeated for all changes the user desires.

[1571] 3. Dialogue adjustment using emotion engine

[1572] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time, using devices such as a webcam and microphone.

[1573] The server uses an emotion engine (e.g., Affectiva, Microsoft Emotion API) to analyze the user's emotional state. For example, the server sends the recognized emotion data to the emotion engine, which analyzes the emotional state as "excitement," "stress," "anxiety," etc.

[1574] The server adjusts the dialogue based on the user's emotional state. For example, if the user is feeling stressed, it generates additional questions or messages to help them relax, which the device then displays to the user.

[1575] 4. Creating and checking the 3D model

[1576] The server generates an initial 3D model based on the collected user preferences and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[1577] The server sends the generated 3D model to the terminal, which displays it to the user.

[1578] The user reviews the initial 3D model and provides feedback, such as "Make the nose 1 mm taller and thinner," which is then sent from the device to the server.

[1579] The server modifies the 3D model based on the user's feedback and sends it back to the device, and this process is repeated until the user is satisfied.

[1580] 5. Final confirmation and treatment plan

[1581] The user is satisfied with the final 3D model and decides, "This is it."

[1582] The device sends this information to a server, which stores the final 3D model and sends it to the practitioner (doctor).

[1583] The practitioner will create a specific treatment plan based on the final 3D model.

[1584] This allows users to specify their desired appearance and visualize it as a 3D model, facilitating communication between patients and practitioners and improving the precision of treatment. Furthermore, by using an emotion engine, the system can take into account the user's emotional state, improving the quality of communication and increasing user satisfaction.

[1585] Examples of prompts:

[1586] 1. Prompt: "What facial changes would you like to see?"

[1587] 2. Prompt: "Please tell us more about the changes you would make. For example, make the nose higher and the eyes bigger."

[1588] 3. Prompt: What are the desired dimensions after the change?

[1589] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1590] Step 1:

[1591] Uploading an image

[1592] Users upload a photo of their face to the system from their smartphone or computer.

[1593] Input: Data file of face photos

[1594] Output: Facial photo data uploaded to the system

[1595] Specific operation: When a user takes a photo of their face using the smartphone's camera app and presses the "upload" button through the application, the photo is sent to the server.

[1596] Step 2:

[1597] Image reception and analysis

[1598] The server receives the uploaded facial photos and stores them in a database.

[1599] Input: Uploaded face photo data

[1600] Output: Facial photo data stored in the database

[1601] Specific operation: The server receives the HTTP request and stores the image data in a database (e.g., MySQL, PostgreSQL).

[1602] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial feature points and prepare for generating an initial 3D model.

[1603] Input: Facial photo data stored in the database

[1604] Output: Data containing facial feature points

[1605] Specific operation: The server uses OpenCV to detect facial feature points and extracts the data as components.

[1606] Step 3:

[1607] Generate initial questions

[1608] The server generates initial questions based on the user's basic information and facial photo.

[1609] Input: User's basic information and face photo data

[1610] Output: First question text

[1611] What it does: The server retrieves the user's basic information from the database and uses a text generation engine to generate an initial question such as "What changes would you like to make?"

[1612] Step 4:

[1613] Get user answers

[1614] The terminal displays the generated question to the user.

[1615] Input: First question text

[1616] Output: Question printed to terminal

[1617] What it does: Displays a question in the application's interface and allows the user to enter an answer.

[1618] The user inputs the desired changes (e.g., "I want to make my nose taller").

[1619] Input: The question displayed on the terminal

[1620] Output: User's answer (text data)

[1621] Specific behavior: The user enters the desired appearance changes in the text box and presses the "Submit" button to send the answer to the server.

[1622] Step 5:

[1623] Generate follow-up questions

[1624] The server generates detailed follow-up questions based on the user's answers (e.g., "How tall would you like your nose to be?").

[1625] Input: User's answer (text data)

[1626] Output: Detailed follow-up question text

[1627] Specific actions: Use natural language processing to analyze the user's answers and generate specific instructions or follow-up questions.

[1628] Step 6:

[1629] Getting specific answers

[1630] The terminal displays the generated follow-up questions to the user.

[1631] Input: Detailed follow-up question text

[1632] Output: Additional questions displayed on the terminal

[1633] What it does: It displays the question in the application and allows the user to enter a detailed answer.

[1634] The user enters a detailed response (e.g., "Please make it about 2 mm higher").

[1635] Input: Additional question displayed on the terminal

[1636] Output: Detailed user responses (text data)

[1637] Specific operation: The user enters the answer to the additional question in the text box and presses the submit button to send it to the server.

[1638] Step 7:

[1639] Repeated dialogue

[1640] The server generates additional questions for other changes as well, to obtain more detailed answers (e.g., "What size do you want your eyes to be?").

[1641] Input: Each user's answer (text data)

[1642] Output: Each follow-up question text and its answer

[1643] Specific operation: Repeat the same interactive process to collect all changes desired by the user.

[1644] Step 8:

[1645] User emotion recognition

[1646] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time.

[1647] Input: User's facial expressions and voice data

[1648] Output: Emotional state data

[1649] What it does: It uses a webcam and microphone to capture data in real time and sends it to an emotion recognition algorithm.

[1650] Step 9:

[1651] Emotion-based dialogue adjustment

[1652] The server adjusts the dialogue (e.g., generates follow-up questions or messages to relax the user) based on the emotional state analyzed using an emotion engine (e.g., Affectiva, Microsoft Emotion API).

[1653] Input: Emotional state data

[1654] Output: Adjusted dialogue (questions and messages)

[1655] Specific operation: Dynamically adjust the dialogue content based on the emotional state obtained from the emotion engine.

[1656] Step 10:

[1657] Generating the initial 3D model

[1658] The server generates an initial 3D model based on the collected user preferences, and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[1659] Input: User's desired changes data

[1660] Output: Initial 3D model data

[1661] Specific operation: Run the auto-generation script in the 3D modeling tool to create the initial 3D model.

[1662] Step 11:

[1663] User Review and Feedback

[1664] The device presents the generated initial 3D model to the user and prompts them to confirm it.

[1665] Input: Initial 3D model data

[1666] Output: The initial 3D model displayed on the device.

[1667] What it does: Displays a rendering of a 3D model and provides an interface where users can provide feedback.

[1668] The user provides feedback (e.g., "Make the nose 1 mm higher and thinner").

[1669] Input: Initial 3D model displayed on the device

[1670] Output: User feedback (text data)

[1671] Specific behavior: The user fills in the feedback form and presses the submit button to send the feedback to the server.

[1672] Step 12:

[1673] Modifying the 3D model

[1674] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1675] Input: User feedback (text data)

[1676] Output: Modified 3D model data

[1677] What happens: Run an automated correction script in the 3D modeling tool based on the collected feedback, then create and submit the model again.

[1678] Step 13:

[1679] Final confirmation and treatment plan

[1680] The user is satisfied with the final 3D model and decides, "This is it."

[1681] Input: Final 3D model data

[1682] Output: User's confirmed decision (text data)

[1683] Specific actions: Includes pressing the confirmation button.

[1684] The server stores the final 3D model and sends it to the practitioner.

[1685] Input: User's confirmed intention (text data)

[1686] Output: Final 3D model data

[1687] Specific operation: The data is saved in a database, and notifications are sent to the practitioner's device and data is shared.

[1688] The practitioner will create a specific treatment plan based on the final 3D model.

[1689] Input: Final 3D model data

[1690] Output: Specific treatment plan

[1691] Specific actions: Using the 3D model as a reference, we will develop a specific schedule and procedures for the treatment.

[1692] (Application example 2)

[1693] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1694] Conventional systems make it difficult for users to accurately communicate the details of their desired appearance to the therapist, resulting in lower satisfaction with the treatment. Furthermore, dialogue conducted without taking the user's emotional state into consideration can increase stress and anxiety for the user. Furthermore, the process of generating a 3D model based on the user's wishes is not completed all at once, requiring multiple revisions, increasing the time and effort required for treatment. A system that can resolve these issues and achieve higher user satisfaction and more accurate treatments was needed.

[1695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1696] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for analyzing the user's emotions using sensors and cameras, and means for adjusting the dialogue based on the results of the emotion analysis. This allows for accurate visualization of the user's desired appearance and facilitates smooth communication between the user and the practitioner. Furthermore, dialogue that takes the user's emotions into consideration reduces stress and anxiety, allowing for the provision of a more satisfying service.

[1697] "Means for receiving facial image data" refers to a series of functions that allow the server to receive facial image data captured by a user using a device such as a smartphone or camera.

[1698] "Means for generating a 3D model" means a collection of software and algorithms for creating a three-dimensional model of a user's face based on the received facial image data.

[1699] "Means for interactively gathering details of a user's desired appearance" means an interface and related functionality that allows a user to interactively input specific improvements or changes to their desired appearance.

[1700] The "means for modifying the 3D model based on the collected wishes" refers to a function that edits an existing 3D model based on the desired data collected from the user, and performs processing to bring it closer to the user's ideal.

[1701] "Means for providing the modified 3D model to users and practitioners" means an interface and related functionality for displaying and sharing the modified 3D model so that users and practitioners can view it.

[1702] "Means for analyzing user emotions using sensors and cameras" refers to technology that uses sensors and cameras on a device to analyze emotions in real time from a user's facial expressions and voice.

[1703] The "means for adjusting the dialogue based on the emotion analysis results" is a function for generating and adjusting questions and messages to more appropriately progress the dialogue based on the analyzed emotional state of the user.

[1704] This invention is a system that allows users to accurately communicate their desired appearance to practitioners and improve the quality of treatment. This system generates a 3D model based on the user's facial image, collects preferences in an interactive format, and provides the user and practitioner with a modified 3D model based on those preferences. It also takes the user's emotions into account and adjusts the dialogue to improve user satisfaction.

[1705] 1. Image upload and initial settings

[1706] The server receives image data of the face taken by the user using a device such as a smartphone or camera. When the user uploads the image data through the application, the server receives and stores the data. The software used is OpenCV and dlib, which analyze the facial feature points and prepare to generate a 3D model.

[1707] 2. Generating 3D models

[1708] The server uses OpenCV and dlib to generate a 3D model of the user's face based on the received facial image data. This 3D model serves as the basis for incorporating specific changes desired by the user. For example, dlib's Face Points can be used to extract feature points and generate a 3D point cloud based on them.

[1709] 3. Building an ideal image through dialogue

[1710] The device interactively collects details of the user's desired appearance. It includes a question generation module for generating additional, more specific questions based on the user's answers. This interaction is conducted by the user through a smartphone application. For example, if the user inputs, "I want a higher nose," the system adds a more specific question: "How high do you want your nose to be?"

[1711] 4. Dialogue adjustment using emotion engine

[1712] The device uses sensors and cameras to recognize the user's emotions in real time. The server then analyzes the user's emotional state using emotion analysis software such as DeepFace. The dialogue is adjusted based on the analysis results. For example, if the user is feeling stressed, a message is generated saying, "We recommend that you receive treatment in a relaxing environment. Please let us know if you have any concerns."

[1713] 5. Modifying and checking the 3D model

[1714] The server modifies the 3D model based on the user's preferences collected through the interaction. The modified 3D model is then sent back to the device and presented to the user. The user provides feedback through the operation interface, and the server further modifies the 3D model based on that feedback. This process is repeated until the user is satisfied. For example, if the user provides feedback such as "Make the nose 1 mm higher and thinner," the server regenerates the 3D model based on that feedback.

[1715] 6. Final confirmation and treatment plan

[1716] If the user is satisfied with the final 3D model, they can confirm it via their device by clicking "OK." This information is then shared with the practitioner via the server, who will then create a specific treatment plan based on the final 3D model.

[1717] Examples of prompt statements

[1718] "How can I help users relax if they're feeling stressed?"

[1719] "Please tell us more about the user's desired appearance."

[1720] This allows the user's wishes to be accurately visualized, facilitating smooth communication with the therapist. Furthermore, dialogue using an emotion engine can reduce the user's stress and anxiety, improving satisfaction.

[1721] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1722] Step 1:

[1723] Users take a picture of their face using a smartphone or camera and upload the image data to the system through the application, which then sends the image data to the server.

[1724] Input: Face image data

[1725] Output: Uploaded image data

[1726] Step 2:

[1727] The server receives the uploaded facial image data and analyzes the facial feature points using image processing libraries (OpenCV, dlib), and generates an initial 3D model based on the feature points.

[1728] Input: Uploaded image data

[1729] Data processing: facial feature point analysis, 3D model generation

[1730] Output: Initial 3D model

[1731] Step 3:

[1732] The device interactively collects details of the user's desired appearance along with an initial 3D model sent from the server, and the user inputs desired changes through the application.

[1733] Input: Initial 3D model, detailed questions about the user's desired appearance

[1734] Data Entry: User Answers

[1735] Output: User's desired data

[1736] Step 4:

[1737] The server modifies the 3D model based on the collected user data. Specifically, it changes the shape of the 3D model according to the user's feedback.

[1738] Input: User's desired data

[1739] Data calculation: Modifying 3D models

[1740] Output: Modified 3D model

[1741] Step 5:

[1742] The device presents the modified 3D model to the user and prompts them for confirmation, and the user provides additional feedback, which is then sent back to the server. This process is repeated until the user is satisfied.

[1743] Input: Modified 3D model, user feedback

[1744] Data Entry: Additional User Feedback

[1745] Output: Next change request based on feedback

[1746] Step 6:

[1747] The device uses sensors and cameras to monitor the user's facial expressions and voice, analyzing their emotions in real time. The server then adjusts the content of the conversation based on the results of the emotion analysis.

[1748] Input: User's facial expression and voice data

[1749] Data Processing: Real-time Sentiment Analysis

[1750] Output: Adjustment of dialogue content based on emotion analysis results

[1751] Step 7:

[1752] The server sends the 3D model that the user is finally satisfied with and the treatment plan based on it to the practitioner, who then creates a specific treatment plan based on the model.

[1753] Input: Final 3D model, user's final decision

[1754] Output: Sharing treatment plans, planning treatment plans

[1755] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1756] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1757] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1758] [Fourth embodiment]

[1759] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1760] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1761] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1762] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1763] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1764] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1765] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1766] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1767] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1768] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1769] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1770] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1771] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1772] The present invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. The system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, modifies the 3D model based on the collected information, and provides the 3D model to the user and the practitioner.

[1773] 1. Image upload and initial settings

[1774] 1. Upload an image

[1775] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[1776] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[1777] 2. Image Reception and Analysis

[1778] The server receives the uploaded facial photos and stores them in a database.

[1779] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[1780] 2. Building an ideal image through dialogue

[1781] 1. Generating the initial question

[1782] The server prepares the interactive questions and presents the first question to the user.

[1783] For example, ask the user, "What changes would you like to see?"

[1784] 2. Obtaining and elaborating user answers

[1785] The user responds, "I want to make my nose taller."

[1786] The server generates additional, more specific questions based on the answers and presents them to the user.

[1787] For example: Ask specifically, "How tall would you like your nose to be?"

[1788] The user responds, "I'd like it to be about 2 mm higher."

[1789] 3. Iteration

[1790] This interactive question and answer process applies to other facial features as well (eyes, mouth, chin, etc.).

[1791] 3. Creating and checking the 3D model

[1792] 1. Creating the initial 3D model

[1793] The server generates an initial 3D model based on the collected user preferences.

[1794] The server sends the generated 3D model to the terminal and displays it to the user.

[1795] 2. User Review and Feedback

[1796] The device presents the 3D model to the user and prompts them to confirm it.

[1797] The user provides feedback such as, "Make the nose 1 mm higher and thinner."

[1798] The server revises the 3D model based on the feedback and displays it again.

[1799] 4. Sharing and implementing the final model

[1800] 1. Final confirmation and model confirmation

[1801] The user is satisfied with the final 3D model and decides, "This is it."

[1802] The terminal sends this information to the server.

[1803] 2. Save and share your final model

[1804] The server stores the finalized 3D model and sends it to the practitioner (doctor).

[1805] 3. Planning the treatment plan

[1806] The practitioner will create a specific treatment plan based on the final 3D model.

[1807] In this way, by allowing users to concretize their desired appearance and visualize it as a 3D model, communication between the patient and the practitioner can be facilitated and the precision of the treatment can be improved.

[1808] The processing flow will be explained below.

[1809] Step 1:

[1810] Uploading an image

[1811] Users upload a photo of their face to the system from their terminal.

[1812] The terminal sends this facial photo to the server.

[1813] Step 2:

[1814] Image reception and analysis

[1815] The server receives the uploaded facial photos and stores them in a database.

[1816] The server uses an image processing module to extract facial feature points.

[1817] Step 3:

[1818] Generate initial questions

[1819] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[1820] The terminal displays this question to the user.

[1821] Step 4:

[1822] Get user answers

[1823] The user responds, "I want to make my nose taller."

[1824] The terminal sends this response to the server.

[1825] Step 5:

[1826] Generate follow-up questions

[1827] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[1828] The terminal displays this question to the user.

[1829] Step 6:

[1830] Getting specific answers

[1831] The user responds, "Make it about 2 mm higher."

[1832] The terminal sends this response to the server.

[1833] Step 7:

[1834] Repeated dialogue

[1835] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[1836] The user responds, "I'd like the eyes to be a little bigger."

[1837] *This process is repeated for all changes the user desires.

[1838] Step 8:

[1839] Generating the initial 3D model

[1840] The server generates an initial 3D model based on the collected user preferences.

[1841] The server sends the generated 3D model to the terminal.

[1842] Step 9:

[1843] Check the 3D model

[1844] The device displays the initial 3D model to the user.

[1845] The user checks the 3D model and provides feedback such as, "Make the nose 1 mm higher and thinner."

[1846] The terminal sends this feedback to the server.

[1847] Step 10:

[1848] Modifying the 3D model

[1849] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[1850] This process is repeated until the user is satisfied.

[1851] Step 11:

[1852] Final confirmation and model confirmation

[1853] The user is satisfied with the final 3D model and decides, "This is it."

[1854] The terminal sends this information to the server.

[1855] Step 12:

[1856] Save and share your final model

[1857] The server stores the final 3D model.

[1858] The server shares the finalized 3D model with the practitioner.

[1859] Step 13:

[1860] Treatment plan planning

[1861] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[1862] In this way, by collecting and analyzing the user's wishes at each step and generating, modifying and providing specific 3D models, it is possible to increase user satisfaction and the accuracy of the treatment.

[1863] Example 1

[1864] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1865] In conventional cosmetic medicine, it has been difficult for users to accurately communicate their desired appearance to the practitioner. There is a need for a system that can collect detailed information about the specific changes desired by the user and generate and modify a 3D model based on that information. In addition, there is a high need for a system that can repeatedly check and modify the 3D model based on user feedback.

[1866] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1867] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for including a generative AI model for generating interactive questions, means for generating additional specific questions based on the user's answers to the generated questions, means for repeatedly modifying the 3D model based on the collected preferences, and means for redisplaying the 3D model. This allows the user to collect details of the user's desired appearance and repeatedly confirm and modify them, thereby providing the practitioner with accurate information and improving the accuracy of the treatment.

[1868] "Facial image data" refers to an image file of a user's face that is used by the system to analyze facial features.

[1869] A "3D model" is a computer-generated, three-dimensional digital model that reflects a user's facial features and desired changes.

[1870] "Dialogue" is a form of communication between a person and a system through questions and answers, and is a method of gathering detailed information about the user's wishes.

[1871] A "generative AI model" is a model that uses artificial intelligence technology to generate appropriate questions and answers based on user input, and examples include AI for processing natural language.

[1872] "Feedback" refers to the input of desired changes or modifications made by the user to the system, which the system uses to reflect this and modify the 3D model.

[1873] "User" refers to a person who utilizes the system to input their wishes to change their appearance.

[1874] "Practitioner" refers to a professional who performs cosmetic medical treatments based on the user's wishes.

[1875] This invention is a system that helps users accurately communicate their desired appearance to a practitioner before undergoing cosmetic medical treatment. The system receives facial image data from the user, generates a 3D model based on that data, interactively collects detailed requests, modifies the 3D model based on the collected requests, and provides the results to the user and the practitioner.

[1876] The system uses the following hardware and software:

[1877] Hardware: Servers, devices (smartphones, computers)

[1878] Software: OpenCV (image processing library), Blender (3D modeling software), generative AI models (e.g., GPT-3)

[1879] Users take a photo of their face using their device and upload the image data to the system via a dedicated app or web page. The server receives the image data and stores it in a database. The server then uses an image processing library such as OpenCV to recognize facial feature points, which allows for the generation of an initial 3D model.

[1880] Next, the server uses the generative AI model to create dialogue-style questions and present them to the user. For example, it generates a question such as "What changes would you like to make?" and obtains the user's answer. The user inputs a specific answer such as "I want my nose to be higher." The server then uses the generative AI model again to generate an additional specific question such as "How high do you want your nose to be?" and presents it to the user. All detailed wishes collected through the dialogue with the user are recorded.

[1881] The server uses 3D modeling software such as Blender to generate a 3D model based on the collected preferences and sends the model to the device. The device then displays the 3D model to the user, allowing the user to provide feedback. For example, the user may enter specific feedback such as, "Make the nose 1 mm higher and thinner." The server then modifies the 3D model based on this feedback, sends it back to the device, and displays it to the user. This process is repeated until the user is satisfied.

[1882] Finally, the server stores the 3D model finalized by the user and provides it to the practitioner, who then uses it to create a specific treatment plan. This prevents any discrepancies between the user and the practitioner regarding the desired appearance and improves the accuracy of the treatment.

[1883] As a concrete example, the following exchange occurs:

[1884] Prompt: "What would you like to change?"

[1885] User Answer: "I want to make my nose taller."

[1886] Next prompt: "How tall would you like your nose to be?"

[1887] User answer: "I want it to be about 2mm higher."

[1888] In this way, the present invention provides a system that can facilitate communication between the user and the practitioner and improve the accuracy of treatment by specifically collecting the user's desired appearance and sequentially reflecting it in a three-dimensional model.

[1889] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1890] Step 1: Upload an image

[1891] Users use a device (smartphone or computer) to take a photo of their face and upload the image data to the system via a dedicated app or web page.

[1892] Input: Face image data (JPEG, PNG, etc.)

[1893] Specific behavior: The user takes a photo of their face using the camera app and clicks the upload button to upload the image file via the app or browser.

[1894] Output: Facial image data sent to the system.

[1895] Step 2: Receiving and analyzing images

[1896] The server receives the uploaded facial photo, stores it in a database, and uses an image processing module to recognize facial features (the positions of the eyes, nose, mouth, etc.).

[1897] Input: Image data uploaded by the user

[1898] How it works: The server receives the image data sent in the HTTP request and saves it in storage. It then uses an image processing library such as OpenCV to detect facial feature points.

[1899] Output: Feature point coordinate data and image path stored in database.

[1900] Step 3: Generate initial questions

[1901] The server uses the generative AI model to generate initial interactive questions and present them to the user.

[1902] Input: Feature point coordinate data and user information

[1903] What it does: The server calls a generative AI model (e.g., GPT-3) and generates questions such as, "What changes would you like to make?"

[1904] Output: Initial question.

[1905] Step 4: Obtaining and elaborating user responses

[1906] The user enters an answer to the question, and the server then generates a more specific question based on the answer and presents it to the user.

[1907] Input: Initial question and user's answer

[1908] Specific operation: The user enters an answer (e.g., "I want to make my nose taller") into the input form and submits it. The server uses the generative AI model to generate the next question (e.g., "How tall do you want your nose to be?").

[1909] Output: Detailed follow-up questions.

[1910] Step 5: Iterate

[1911] The dialogue-style question and answer process is repeated for other facial features (eyes, mouth, chin, etc.).

[1912] Input: Previous answers and generated questions

[1913] What happens: The server receives each answer and generates additional questions as needed and presents them to the user.

[1914] Output: A list of questions and answers with all preferences collected in detail.

[1915] Step 6: Creating the initial 3D model

[1916] The server uses three-dimensional modeling software to generate an initial 3D model based on the collected user preferences.

[1917] Input: User's desired details

[1918] Specific operation: The server calls 3D modeling software such as Blender and generates a 3D model based on the user's desired shape.

[1919] Output: Data of the initial 3D model.

[1920] Step 7: User review and feedback

[1921] The device displays the generated 3D model to the user, who then provides feedback.

[1922] Input: Initial 3D model

[1923] What it does: Render and display a 3D model on the user's device, and the user can provide feedback such as "Make the nose 1mm taller and thinner."

[1924] Output: User feedback.

[1925] Step 8: Rework the 3D model

[1926] The server revises the 3D model based on the user's feedback and displays it again, repeating this process until the user is satisfied.

[1927] Input: User feedback

[1928] Specific operation: The server analyzes the feedback, revises the model in the 3D modeling software, and sends the revisited model data back to the user's device.

[1929] Output: The modified 3D model.

[1930] Step 9: Final check and finalize the model

[1931] The user is satisfied with the final 3D model and decides, "This is it."

[1932] Input: Modified 3D model

[1933] Specific behavior: The user makes a final confirmation and clicks the "Confirm" button.

[1934] Output: Model confirmation notice.

[1935] Step 10: Save and share your final model

[1936] The server stores the finalized 3D model and provides it to the practitioner.

[1937] Input: Model confirmation notification and 3D model data

[1938] Specific operation: The server saves the 3D model data in a database and notifies the practitioner via email or a dedicated portal.

[1939] Output: 3D model data provided to the practitioner.

[1940] Step 11: Planning your treatment

[1941] The practitioner will create a specific treatment plan based on the final 3D model.

[1942] Input: Provided 3D model data

[1943] Specific operations: The surgeon simulates surgery based on the provided 3D model and creates a specific treatment plan.

[1944] Output: Treatment plan.

[1945] (Application example 1)

[1946] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1947] In traditional plastic surgery and cosmetic procedures, it has been difficult for patients to accurately communicate their desired appearance to the doctor, resulting in unsatisfactory results. Similarly, in food delivery, it has been difficult for customers to communicate the specific customizations they desire, resulting in unsatisfactory orders. There is a need for a system that can solve these issues and accurately respond to users' requests.

[1948] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1949] In this invention, the server includes means for receiving image data of a face and food, means for generating and modifying a 3D model, means for interactively collecting details of the user's desired appearance and food, and means for modifying the 3D model according to the user's wishes and providing it to the user, a practitioner, or a restaurant, thereby enabling the user to accurately communicate specific customizations of the user's desired appearance and food, and for treatment and cooking to be performed based on those customizations.

[1950] "Facial image data" refers to photographs and image information of the user's face, and is the basic data used to generate a 3D model.

[1951] "Means for generating 3D models" refers to the process and technology for creating three-dimensional model data based on image data of faces or food.

[1952] "A means for interactively gathering details of a user's desired appearance" refers to an interface and system that gathers the specifications and features desired by the user in the form of questions.

[1953] "Means for modifying 3D models based on collected preferences" refers to techniques for modifying or updating existing 3D models based on preferences or feedback collected from users.

[1954] "Means for providing modified 3D models to users and practitioners" refers to a system for displaying or transmitting the completed 3D model to users and practitioners.

[1955] "Food image data" refers to photos and image information of the dishes and ingredients desired by the user, and is the basic data used to generate a 3D model of the dish.

[1956] "Means for generating and modifying 3D food models" refers to technology that creates three-dimensional food model data based on food image data and modifies or updates it.

[1957] "Means for interactively collecting details of the food desired by the user" refers to an interface and system for collecting the specifications and characteristics of the food desired by the user in the form of questions.

[1958] "Means for modifying 3D food models according to user preferences and providing the finalized models to restaurants" refers to a system for modifying 3D food models based on preferences and feedback collected from users and displaying or transmitting the finalized models to restaurants.

[1959] The present invention provides a system that allows users to specifically communicate their desired appearance and food customization details. This system operates between a server and a user terminal, and the process proceeds as follows:

[1960] 1. Image upload and initial settings

[1961] Users take a photo of themselves or a reference image of the dish they want to eat with their smartphone, and then send the image to the server via the application. The server then stores the received image data and begins analyzing it.

[1962] The server uses image processing modules such as OpenCV to analyze facial features and food ingredients, and then generates an initial 3D model. The generated 3D model is then sent to the user's device and displayed to the user.

[1963] 2. Creating ideal appearances and dishes through dialogue

[1964] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The user enters answers, and the server generates additional, more specific questions based on those answers.

[1965] For example, if the user answers "I want a taller nose" or "I want more cheese," the server generates and displays more specific questions such as "How tall do you want your nose to be?" or "What type of cheese would you like more of?"

[1966] 3. Creating and checking the 3D model

[1967] The server generates an initial 3D model based on the user's preferences and sends it to the user's device. The user can review the 3D model and provide feedback as needed. For example, they can provide feedback such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes."

[1968] The server then modifies the 3D model based on this feedback and displays it again to the user, repeating this process until the user is satisfied.

[1969] 4. Sharing the final model and ordering / treatment

[1970] Once the user is satisfied with the final 3D model and confirms it, the device sends this information to the server. The server saves the finalized 3D model and sends it to the therapist or restaurant. The therapist uses this model to create a specific treatment plan, and the restaurant begins cooking based on this model.

[1971] Specific examples and prompts for the generative AI model

[1972] Specific examples

[1973] 1. In the case of cosmetic surgery: The user requests "I want my nose to be straighter and higher," and the 3D model is updated based on that request.

[1974] 2. Food delivery: A user requests "more cheese on my pizza," and the 3D model is updated based on that request.

[1975] Generative AI model prompt

[1976] A user wants to customize a cosmetic procedure or food. Please generate a 3D model using the following instructions:

[1977] 1. Cosmetic procedure: Nose lift (specifically, 2mm higher)

[1978] 2. Cooking: Add more cheese (mozzarella and parmesan)

[1979] Output: Updated 3D model data and its visual representation.

[1980] Using this prompt, the AI ​​model can generate a 3D model that accurately reflects the user's wishes. This system accurately conveys the user's wishes, improving the accuracy of treatment and cooking.

[1981] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1982] Step 1:

[1983] Users take a photo of themselves or an image of the dish they want to eat with their smartphone and upload it to the server through the application. The server receives this image data and stores it in a database. Here, the input is the image file uploaded by the user, and the output is the image data stored on the server.

[1984] Step 2:

[1985] The server analyzes the received image data using an image processing module such as OpenCV. It extracts facial features and food ingredients and generates an initial 3D model based on them. Here, the input is the image data saved in step 1, and the output is the generated initial 3D model data.

[1986] Step 3:

[1987] The server sends the generated initial 3D model to the terminal and displays it to the user. The input is the 3D model data generated in step 2, and the output is the 3D model displayed on the user's terminal.

[1988] Step 4:

[1989] The server generates interactive questions and presents them to the user, such as "What changes would you like to make?" or "Would you like to change the ingredients?" The input is the user's preference, and the output is the interactive questions.

[1990] Step 5:

[1991] The user answers questions interactively, for example, by inputting specific wishes such as "I want a taller nose" or "I want more cheese." The input is the user's answer, and the output is the answer data sent to the server.

[1992] Step 6:

[1993] The server analyzes the collected response data from the user and generates additional specific questions, such as "How tall do you want your nose to be?" or "Which type of cheese would you like more of?", and presents them to the user again. The input is the user's response data, and the output is the additional questions.

[1994] Step 7:

[1995] The server modifies the 3D model based on the user's detailed wishes and sends it back. For example, creating a 3D model of a pizza with a longer nose or more cheese. The input is the user's wishes, and the output is the modified 3D model data.

[1996] Step 8:

[1997] The user checks the modified 3D model and provides further feedback, such as "Make the nose 1 mm taller and thinner" or "Add more tomatoes." The input is the user's feedback, and the output is the feedback data received by the server.

[1998] Step 9:

[1999] The server revises the 3D model based on this feedback and resends it. This process is repeated until the user is satisfied. The input is the user's feedback data, and the output is the revisited 3D model.

[2000] Step 10:

[2001] When the user is satisfied with the final 3D model and confirms, "This is it," the device sends this information to the server. The input is the user's final decision, and the output is the decision data received by the server.

[2002] Step 11:

[2003] The server stores the finalized 3D model and sends it to the practitioner or restaurant. The practitioner makes a specific treatment plan based on this model, and the restaurant starts cooking based on this model. The input is the finalized 3D model data, and the output is the model data sent to the practitioner or restaurant.

[2004] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2005] This invention is a system that allows users to accurately communicate their desired appearance to physicians and improve the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between users and physicians. This system generates a 3D model based on facial image data, interactively collects details of the user's desired appearance, and modifies the 3D model based on those details, providing it to the user and the practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[2006] 1. Image upload and initial settings

[2007] 1. Upload an image

[2008] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[2009] Example: A user takes a photo of their face with their smartphone and sends the image to the system through an application.

[2010] 2. Image Reception and Analysis

[2011] The server receives the uploaded facial photos and stores them in a database.

[2012] The server uses an image processing module to recognize facial feature points and prepare for the generation of an initial 3D model.

[2013] 2. Building an ideal image through dialogue

[2014] 1. Generating the initial question

[2015] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[2016] The terminal displays this question to the user.

[2017] 2. Obtaining user answers

[2018] The user responds, "I want to make my nose taller."

[2019] The terminal sends this response to the server.

[2020] 3. Generating follow-up questions

[2021] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[2022] The terminal displays this question to the user.

[2023] 4. Getting specific answers

[2024] The user responds, "Make it about 2 mm higher."

[2025] The terminal sends this response to the server.

[2026] 5. Repeated dialogue

[2027] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[2028] The user responds, "I'd like the eyes to be a little bigger."

[2029] *This process is repeated for all changes the user desires.

[2030] 3. Dialogue adjustment using emotion engine

[2031] 1. User Emotion Recognition

[2032] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[2033] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[2034] 2. Emotion-based dialogue adjustment

[2035] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[2036] The device displays these tailored questions or messages to the user.

[2037] 4. Creating and checking the 3D model

[2038] 1. Generate the initial 3D model

[2039] The server generates an initial 3D model based on the collected user preferences.

[2040] The server sends the generated 3D model to the terminal and displays it to the user.

[2041] 2. User Review and Feedback

[2042] The device will present the user with an initial 3D model and ask them to confirm it.

[2043] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[2044] The terminal sends this feedback to the server.

[2045] 3. Modifying the 3D model

[2046] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[2047] This process is repeated until the user is satisfied.

[2048] 5. Final confirmation and treatment plan

[2049] 1. Final confirmation and model confirmation

[2050] The user is satisfied with the final 3D model and decides, "This is it."

[2051] The terminal sends this information to the server.

[2052] 2. Save and share your final model

[2053] The server saves the final 3D model and sends it to the practitioner (doctor).

[2054] 3. Planning the treatment plan

[2055] The practitioner will create a specific treatment plan based on the final 3D model.

[2056] In this way, by allowing users to specify their desired appearance and visualizing it as a 3D model, communication between patients and practitioners can be facilitated and the precision of treatment can be improved. In addition, by taking the user's emotional state into account using an emotion engine, user satisfaction can be further improved.

[2057] The processing flow will be explained below.

[2058] Step 1:

[2059] Uploading an image

[2060] Users upload a photo of their face to the system from their smartphone or computer (terminal).

[2061] The terminal sends this facial photo to the server.

[2062] Step 2:

[2063] Image reception and analysis

[2064] The server receives the uploaded facial photos and stores them in a database.

[2065] The server uses an image processing module to extract facial feature points.

[2066] Step 3:

[2067] Generate initial questions

[2068] The server generates the first question, "What changes would you like to make?" based on the user's basic information and face photo.

[2069] The terminal displays this question to the user.

[2070] Step 4:

[2071] Get user answers

[2072] The user responds, "I want to make my nose taller."

[2073] The terminal sends this response to the server.

[2074] Step 5:

[2075] Generate follow-up questions

[2076] The server generates detailed, specific questions based on the user's answers, e.g., "How tall do you want your nose to be?"

[2077] The terminal displays this question to the user.

[2078] Step 6:

[2079] Getting specific answers

[2080] The user responds, "Make it about 2 mm higher."

[2081] The terminal sends this response to the server.

[2082] Step 7:

[2083] Repeated dialogue

[2084] The server repeats the same question and answer for other changes, e.g., "What size do you want your eyes to be?"

[2085] The user responds, "I'd like the eyes to be a little bigger."

[2086] *This process is repeated for all changes the user desires.

[2087] Step 8:

[2088] User emotion recognition

[2089] The device uses sensors and cameras to recognize the user's emotions in real time from their facial expressions and voice.

[2090] The server uses an emotion engine to analyze the user's emotional state (e.g., excitement, stress, anxiety).

[2091] Step 9:

[2092] Emotion-based dialogue adjustment

[2093] The server adjusts the dialogue based on the user's emotional state, e.g., if the user is feeling stressed, it generates additional questions or messages to help them relax.

[2094] The device displays these tailored questions or messages to the user.

[2095] Step 10:

[2096] Generating the initial 3D model

[2097] The server generates an initial 3D model based on the collected user preferences.

[2098] The server sends the generated 3D model to the terminal and displays it to the user.

[2099] Step 11:

[2100] Check the 3D model

[2101] The device will present the user with an initial 3D model and ask them to confirm it.

[2102] The user provides feedback such as "Make the nose 1 mm higher and thinner."

[2103] The terminal sends this feedback to the server.

[2104] Step 12:

[2105] Modifying the 3D model

[2106] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[2107] This process is repeated until the user is satisfied.

[2108] Step 13:

[2109] Final confirmation and model confirmation

[2110] The user is satisfied with the final 3D model and decides, "This is it."

[2111] The terminal sends this information to the server.

[2112] Step 14:

[2113] Save and share your final model

[2114] The server stores the final 3D model.

[2115] The server shares the finalized 3D model with the practitioner.

[2116] Step 15:

[2117] Treatment plan planning

[2118] The practitioner will review the final 3D model and create a specific treatment plan based on it.

[2119] In this way, by collecting and analyzing the user's wishes at each step, generating and modifying specific 3D models, and further taking the user's emotional state into account using an emotion engine, it is possible to increase user satisfaction and the accuracy of the treatment.

[2120] Example 2

[2121] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2122] In conventional beauty treatments, it is difficult for users to accurately communicate their desired appearance to the practitioner, resulting in treatments that do not meet the user's expectations. Furthermore, the user's emotional state is not fully taken into consideration, resulting in poor communication quality and lower user satisfaction. Furthermore, the inefficient dialogue-based interaction makes it difficult to quickly and accurately gather detailed specifications for the desired appearance.

[2123] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2124] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for recognizing the user's emotional state and adjusting the dialogue, and means for adjusting the dialogue based on the recognized emotional state. This allows the user's desired appearance to be accurately expressed in a 3D model and information to be efficiently shared with the practitioner. Furthermore, by conducting dialogue taking the user's emotional state into consideration, the quality of communication can be improved and user satisfaction can be increased.

[2125] "Facial image data" is digital image data that records the features and shape of a user's face in detail.

[2126] A "3D model" is a digital model that is reproduced in three dimensions based on image data of a face.

[2127] "Dialogue" is a format in which the user and the system exchange information through a question-and-answer exchange.

[2128] The "means for collecting details of the user's desired appearance" refers to the means used to collect specific details of the user's desired changes to appearance.

[2129] "Means for modifying 3D models" means means for modifying and changing existing 3D models according to the user's desired appearance.

[2130] "Means provided to users and practitioners" refers to the means for displaying and sharing modified 3D models with users and practitioners.

[2131] "Means for recognizing the user's emotional state" refers to means for detecting and analyzing emotions from the user's facial expressions and voice.

[2132] A "means for adjusting a dialogue" is a means for changing the content or manner of a dialogue based on the recognized emotional state of the user.

[2133] The "question generation module" is a module for generating additional specific questions based on the user's answers.

[2134] "Sensors and cameras" are devices for collecting the user's facial expressions and voice.

[2135] "Emotional state analysis" is the process of analyzing a user's emotions based on collected facial and voice data.

[2136] "Feedback" means opinions or requests provided by users.

[2137] A "practitioner" is a professional who performs treatments to make the user's appearance as desired.

[2138] This invention is a system that allows users to accurately communicate their desired appearance to practitioners, improving the quality of treatment. Furthermore, by recognizing and utilizing the user's emotions, more personalized dialogue is possible, improving communication between the user and practitioner. This system includes functions to generate a 3D model based on facial image data, interactively collect details of the user's desired appearance, modify the 3D model based on those details, and provide it to the user and practitioner. It also incorporates an emotion engine to adjust the dialogue based on the user's emotions.

[2139] Specifically, the following means are used:

[2140] 1. Image upload and initial settings

[2141] Users upload their facial photos to the system using a smartphone or computer. For example, a user can take a photo of their face with their smartphone and send the image to the system through an application.

[2142] The server receives the uploaded face photo and stores it in a database. At this time, the image data is sent to the server via an HTTP request and stored in a database management system (e.g., MySQL, PostgreSQL).

[2143] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial features and prepare for generating an initial 3D model. For example, the server uses the Python library OpenCV to extract features such as the eyes, nose, and mouth from the uploaded image.

[2144] 2. Building an ideal image through dialogue

[2145] The server generates an initial question based on the user's basic information and face photo. For example, it generates an initial question such as "What changes would you like to make?", and the device displays this question to the user.

[2146] The user responds, "I want to make my nose taller," and the terminal sends this response to the server.

[2147] The server generates a detailed question (e.g., "How tall do you want your nose to be?") based on the user's answers, and the device displays the question.

[2148] The user responds, "Please raise it about 2 mm," and the device sends the response to the server.

[2149] This interactive process is repeated for all changes the user desires.

[2150] 3. Dialogue adjustment using emotion engine

[2151] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time, using devices such as a webcam and microphone.

[2152] The server uses an emotion engine (e.g., Affectiva, Microsoft Emotion API) to analyze the user's emotional state. For example, the server sends the recognized emotion data to the emotion engine, which analyzes the emotional state as "excitement," "stress," "anxiety," etc.

[2153] The server adjusts the dialogue based on the user's emotional state. For example, if the user is feeling stressed, it generates additional questions or messages to help them relax, which the device then displays to the user.

[2154] 4. Creating and checking the 3D model

[2155] The server generates an initial 3D model based on the collected user preferences and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[2156] The server sends the generated 3D model to the terminal, which displays it to the user.

[2157] The user reviews the initial 3D model and provides feedback, such as "Make the nose 1 mm taller and thinner," which is then sent from the device to the server.

[2158] The server modifies the 3D model based on the user's feedback and sends it back to the device, and this process is repeated until the user is satisfied.

[2159] 5. Final confirmation and treatment plan

[2160] The user is satisfied with the final 3D model and decides, "This is it."

[2161] The device sends this information to a server, which stores the final 3D model and sends it to the practitioner (doctor).

[2162] The practitioner will create a specific treatment plan based on the final 3D model.

[2163] This allows users to specify their desired appearance and visualize it as a 3D model, facilitating communication between patients and practitioners and improving the precision of treatment. Furthermore, by using an emotion engine, the system can take into account the user's emotional state, improving the quality of communication and increasing user satisfaction.

[2164] Examples of prompts:

[2165] 1. Prompt: "What facial changes would you like to see?"

[2166] 2. Prompt: "Please tell us more about the changes you would make. For example, make the nose higher and the eyes bigger."

[2167] 3. Prompt: What are the desired dimensions after the change?

[2168] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2169] Step 1:

[2170] Uploading an image

[2171] Users upload a photo of their face to the system from their smartphone or computer.

[2172] Input: Data file of face photos

[2173] Output: Facial photo data uploaded to the system

[2174] Specific operation: When a user takes a photo of their face using the smartphone's camera app and presses the "upload" button through the application, the photo is sent to the server.

[2175] Step 2:

[2176] Image reception and analysis

[2177] The server receives the uploaded facial photos and stores them in a database.

[2178] Input: Uploaded face photo data

[2179] Output: Facial photo data stored in the database

[2180] Specific operation: The server receives the HTTP request and stores the image data in a database (e.g., MySQL, PostgreSQL).

[2181] The server uses image processing modules (e.g., OpenCV, TensorFlow) to recognize facial feature points and prepare for generating an initial 3D model.

[2182] Input: Facial photo data stored in the database

[2183] Output: Data containing facial feature points

[2184] Specific operation: The server uses OpenCV to detect facial feature points and extracts the data as components.

[2185] Step 3:

[2186] Generate initial questions

[2187] The server generates initial questions based on the user's basic information and facial photo.

[2188] Input: User's basic information and face photo data

[2189] Output: First question text

[2190] What it does: The server retrieves the user's basic information from the database and uses a text generation engine to generate an initial question such as "What changes would you like to make?"

[2191] Step 4:

[2192] Get user answers

[2193] The terminal displays the generated question to the user.

[2194] Input: First question text

[2195] Output: Question printed to terminal

[2196] What it does: Displays a question in the application's interface and allows the user to enter an answer.

[2197] The user inputs the desired changes (e.g., "I want to make my nose taller").

[2198] Input: The question displayed on the terminal

[2199] Output: User's answer (text data)

[2200] Specific behavior: The user enters the desired appearance changes in the text box and presses the "Submit" button to send the answer to the server.

[2201] Step 5:

[2202] Generate follow-up questions

[2203] The server generates detailed follow-up questions based on the user's answers (e.g., "How tall would you like your nose to be?").

[2204] Input: User's answer (text data)

[2205] Output: Detailed follow-up question text

[2206] Specific actions: Use natural language processing to analyze the user's answers and generate specific instructions or follow-up questions.

[2207] Step 6:

[2208] Getting specific answers

[2209] The terminal displays the generated follow-up questions to the user.

[2210] Input: Detailed follow-up question text

[2211] Output: Additional questions displayed on the terminal

[2212] What it does: It displays the question in the application and allows the user to enter a detailed answer.

[2213] The user enters a detailed response (e.g., "Please make it about 2 mm higher").

[2214] Input: Additional question displayed on the terminal

[2215] Output: Detailed user responses (text data)

[2216] Specific operation: The user enters the answer to the additional question in the text box and presses the submit button to send it to the server.

[2217] Step 7:

[2218] Repeated dialogue

[2219] The server generates additional questions for other changes as well, to obtain more detailed answers (e.g., "What size do you want your eyes to be?").

[2220] Input: Each user's answer (text data)

[2221] Output: Each follow-up question text and its answer

[2222] Specific operation: Repeat the same interactive process to collect all changes desired by the user.

[2223] Step 8:

[2224] User emotion recognition

[2225] The device uses sensors and cameras to recognize emotions from the user's facial expressions and voice in real time.

[2226] Input: User's facial expressions and voice data

[2227] Output: Emotional state data

[2228] What it does: It uses a webcam and microphone to capture data in real time and sends it to an emotion recognition algorithm.

[2229] Step 9:

[2230] Emotion-based dialogue adjustment

[2231] The server adjusts the dialogue (e.g., generates follow-up questions or messages to relax the user) based on the emotional state analyzed using an emotion engine (e.g., Affectiva, Microsoft Emotion API).

[2232] Input: Emotional state data

[2233] Output: Adjusted dialogue (questions and messages)

[2234] Specific operation: Dynamically adjust the dialogue content based on the emotional state obtained from the emotion engine.

[2235] Step 10:

[2236] Generating the initial 3D model

[2237] The server generates an initial 3D model based on the collected user preferences, and creates a three-dimensional model using 3D modeling software (e.g., Blender, Maya).

[2238] Input: User's desired changes data

[2239] Output: Initial 3D model data

[2240] Specific operation: Run the auto-generation script in the 3D modeling tool to create the initial 3D model.

[2241] Step 11:

[2242] User Review and Feedback

[2243] The device presents the generated initial 3D model to the user and prompts them to confirm it.

[2244] Input: Initial 3D model data

[2245] Output: The initial 3D model displayed on the device.

[2246] What it does: Displays a rendering of a 3D model and provides an interface where users can provide feedback.

[2247] The user provides feedback (e.g., "Make the nose 1 mm higher and thinner").

[2248] Input: Initial 3D model displayed on the device

[2249] Output: User feedback (text data)

[2250] Specific behavior: The user fills in the feedback form and presses the submit button to send the feedback to the server.

[2251] Step 12:

[2252] Modifying the 3D model

[2253] The server modifies the 3D model based on the user's feedback and sends it back to the device.

[2254] Input: User feedback (text data)

[2255] Output: Modified 3D model data

[2256] What happens: Run an automated correction script in the 3D modeling tool based on the collected feedback, then create and submit the model again.

[2257] Step 13:

[2258] Final confirmation and treatment plan

[2259] The user is satisfied with the final 3D model and decides, "This is it."

[2260] Input: Final 3D model data

[2261] Output: User's confirmed decision (text data)

[2262] Specific actions: Includes pressing the confirmation button.

[2263] The server stores the final 3D model and sends it to the practitioner.

[2264] Input: User's confirmed intention (text data)

[2265] Output: Final 3D model data

[2266] Specific operation: The data is saved in a database, and notifications are sent to the practitioner's device and data is shared.

[2267] The practitioner will create a specific treatment plan based on the final 3D model.

[2268] Input: Final 3D model data

[2269] Output: Specific treatment plan

[2270] Specific actions: Using the 3D model as a reference, we will develop a specific schedule and procedures for the treatment.

[2271] (Application example 2)

[2272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2273] Conventional systems make it difficult for users to accurately communicate the details of their desired appearance to the therapist, resulting in lower satisfaction with the treatment. Furthermore, dialogue conducted without taking the user's emotional state into consideration can increase stress and anxiety for the user. Furthermore, the process of generating a 3D model based on the user's wishes is not completed all at once, requiring multiple revisions, increasing the time and effort required for treatment. A system that can resolve these issues and achieve higher user satisfaction and more accurate treatments was needed.

[2274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2275] In this invention, the server includes means for receiving facial image data, means for generating a 3D model based on the image data, means for interactively collecting details of the user's desired appearance, means for modifying the 3D model based on the collected preferences, means for providing the modified 3D model to the user and the practitioner, means for analyzing the user's emotions using sensors and cameras, and means for adjusting the dialogue based on the results of the emotion analysis. This allows for accurate visualization of the user's desired appearance and facilitates smooth communication between the user and the practitioner. Furthermore, dialogue that takes the user's emotions into consideration reduces stress and anxiety, allowing for the provision of a more satisfying service.

[2276] "Means for receiving facial image data" refers to a series of functions that allow the server to receive facial image data captured by a user using a device such as a smartphone or camera.

[2277] "Means for generating a 3D model" means a collection of software and algorithms for creating a three-dimensional model of a user's face based on the received facial image data.

[2278] "Means for interactively gathering details of a user's desired appearance" means an interface and related functionality that allows a user to interactively input specific improvements or changes to their desired appearance.

[2279] The "means for modifying the 3D model based on the collected wishes" refers to a function that edits an existing 3D model based on the desired data collected from the user, and performs processing to bring it closer to the user's ideal.

[2280] "Means for providing the modified 3D model to users and practitioners" means an interface and related functionality for displaying and sharing the modified 3D model so that users and practitioners can view it.

[2281] "Means for analyzing user emotions using sensors and cameras" refers to technology that uses sensors and cameras on a device to analyze emotions in real time from a user's facial expressions and voice.

[2282] The "means for adjusting the dialogue based on the emotion analysis results" is a function for generating and adjusting questions and messages to more appropriately progress the dialogue based on the analyzed emotional state of the user.

[2283] This invention is a system that allows users to accurately communicate their desired appearance to practitioners and improve the quality of treatment. This system generates a 3D model based on the user's facial image, collects preferences in an interactive format, and provides the user and practitioner with a modified 3D model based on those preferences. It also takes the user's emotions into account and adjusts the dialogue to improve user satisfaction.

[2284] 1. Image upload and initial settings

[2285] The server receives image data of the face taken by the user using a device such as a smartphone or camera. When the user uploads the image data through the application, the server receives and stores the data. The software used is OpenCV and dlib, which analyze the facial feature points and prepare to generate a 3D model.

[2286] 2. Generating 3D models

[2287] The server uses OpenCV and dlib to generate a 3D model of the user's face based on the received facial image data. This 3D model serves as the basis for incorporating specific changes desired by the user. For example, dlib's Face Points can be used to extract feature points and generate a 3D point cloud based on them.

[2288] 3. Building an ideal image through dialogue

[2289] The device interactively collects details of the user's desired appearance. It includes a question generation module for generating additional, more specific questions based on the user's answers. This interaction is conducted by the user through a smartphone application. For example, if the user inputs, "I want a higher nose," the system adds a more specific question: "How high do you want your nose to be?"

[2290] 4. Dialogue adjustment using emotion engine

[2291] The device uses sensors and cameras to recognize the user's emotions in real time. The server then analyzes the user's emotional state using emotion analysis software such as DeepFace. The dialogue is adjusted based on the analysis results. For example, if the user is feeling stressed, a message is generated saying, "We recommend that you receive treatment in a relaxing environment. Please let us know if you have any concerns."

[2292] 5. Modifying and checking the 3D model

[2293] The server modifies the 3D model based on the user's preferences collected through the interaction. The modified 3D model is then sent back to the device and presented to the user. The user provides feedback through the operation interface, and the server further modifies the 3D model based on that feedback. This process is repeated until the user is satisfied. For example, if the user provides feedback such as "Make the nose 1 mm higher and thinner," the server regenerates the 3D model based on that feedback.

[2294] 6. Final confirmation and treatment plan

[2295] If the user is satisfied with the final 3D model, they can confirm it via their device by clicking "OK." This information is then shared with the practitioner via the server, who will then create a specific treatment plan based on the final 3D model.

[2296] Examples of prompt statements

[2297] "How can I help users relax if they're feeling stressed?"

[2298] "Please tell us more about the user's desired appearance."

[2299] This allows the user's wishes to be accurately visualized, facilitating smooth communication with the therapist. Furthermore, dialogue using an emotion engine can reduce the user's stress and anxiety, improving satisfaction.

[2300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2301] Step 1:

[2302] Users take a picture of their face using a smartphone or camera and upload the image data to the system through the application, which then sends the image data to the server.

[2303] Input: Face image data

[2304] Output: Uploaded image data

[2305] Step 2:

[2306] The server receives the uploaded facial image data and analyzes the facial feature points using image processing libraries (OpenCV, dlib), and generates an initial 3D model based on the feature points.

[2307] Input: Uploaded image data

[2308] Data processing: facial feature point analysis, 3D model generation

[2309] Output: Initial 3D model

[2310] Step 3:

[2311] The device interactively collects details of the user's desired appearance along with an initial 3D model sent from the server, and the user inputs desired changes through the application.

[2312] Input: Initial 3D model, detailed questions about the user's desired appearance

[2313] Data Entry: User Answers

[2314] Output: User's desired data

[2315] Step 4:

[2316] The server modifies the 3D model based on the collected user data. Specifically, it changes the shape of the 3D model according to the user's feedback.

[2317] Input: User's desired data

[2318] Data calculation: Modifying 3D models

[2319] Output: Modified 3D model

[2320] Step 5:

[2321] The device presents the modified 3D model to the user and prompts them for confirmation, and the user provides additional feedback, which is then sent back to the server. This process is repeated until the user is satisfied.

[2322] Input: Modified 3D model, user feedback

[2323] Data Entry: Additional User Feedback

[2324] Output: Next change request based on feedback

[2325] Step 6:

[2326] The device uses sensors and cameras to monitor the user's facial expressions and voice, analyzing their emotions in real time. The server then adjusts the content of the conversation based on the results of the emotion analysis.

[2327] Input: User's facial expression and voice data

[2328] Data Processing: Real-time Sentiment Analysis

[2329] Output: Adjustment of dialogue content based on emotion analysis results

[2330] Step 7:

[2331] The server sends the 3D model that the user is finally satisfied with and the treatment plan based on it to the practitioner, who then creates a specific treatment plan based on the model.

[2332] Input: Final 3D model, user's final decision

[2333] Output: Sharing treatment plans, planning treatment plans

[2334] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2335] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2336] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2337] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2338] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2339] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2340] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2341] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2342] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2343] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2344] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2345] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2346] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2347] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2348] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2349] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2350] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2351] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2352] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2353] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2354] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2355] The following is further disclosed regarding the above embodiment.

[2356] (Claim 1)

[2357] means for receiving facial image data;

[2358] means for generating a 3D model based on the image data;

[2359] A means of interactively gathering details about the user's desired appearance;

[2360] means for modifying the 3D model based on the collected preferences;

[2361] A system including means for providing said modified 3D model to a user and a practitioner.

[2362] (Claim 2)

[2363] The system of claim 1, further comprising means for iteratively modifying the 3D model based on user feedback.

[2364] (Claim 3)

[2365] 10. The system of claim 1, wherein the interactively gathering desired details means includes a question generation module, the question generation module including means for generating additional specific questions based on user responses.

[2366] "Example 1"

[2367] (Claim 1)

[2368] means for receiving facial image data;

[2369] means for generating a three-dimensional model based on the image data;

[2370] means for interactively gathering details of a user's desired appearance;

[2371] means for modifying the three-dimensional model based on the collected preferences;

[2372] means for providing the modified three-dimensional model to a user and a practitioner;

[2373] a means for generating interactive questions, the means including a generative AI model;

[2374] means for generating additional specific questions based on user answers to the generated questions;

[2375] means for iteratively modifying the three-dimensional model based on the collected preferences;

[2376] means for displaying the three-dimensional model again;

[2377] Including system.

[2378] (Claim 2)

[2379] 10. The system of claim 1, further comprising means for iteratively modifying the three-dimensional model based on user feedback.

[2380] (Claim 3)

[2381] 10. The system of claim 1, wherein the interactively gathering desired details means includes a question generation module, the question generation module including means for generating additional specific questions based on user answers.

[2382] "Application Example 1"

[2383] (Claim 1)

[2384] means for receiving facial image data;

[2385] means for generating a 3D model based on the image data;

[2386] A means of interactively gathering details about the user's desired appearance;

[2387] means for modifying the 3D model based on the collected preferences;

[2388] means for providing the modified 3D model to a user and a practitioner;

[2389] means for receiving image data of the food;

[2390] A means for generating and modifying 3D models of food;

[2391] A means of interactively collecting details of the user's food preferences;

[2392] The system includes means for modifying the 3D model of the food according to the user's wishes and providing the finalized model to the restaurant.

[2393] (Claim 2)

[2394] The system of claim 1, further comprising means for iteratively modifying the 3D model based on user feedback.

[2395] (Claim 3)

[2396] 10. The system of claim 1, wherein the interactively gathering desired details means includes a question generation module, the question generation module including means for generating additional specific questions based on user responses.

[2397] "Example 2: Combining Emotion Engines"

[2398] (Claim 1)

[2399] means for receiving facial image data;

[2400] means for generating a 3D model based on the image data;

[2401] A means of interactively gathering details about the user's desired appearance;

[2402] means for modifying the 3D model based on the collected preferences;

[2403] means for providing the modified 3D model to a user and a practitioner;

[2404] a means of recognizing the user's emotional state and adjusting the interaction;

[2405] The system includes means for adjusting interaction based on the recognized emotional state.

[2406] (Claim 2)

[2407] The system of claim 1, further...

Claims

1. means for receiving facial image data; means for generating a 3D model based on the image data; A means of interactively gathering details about the user's desired appearance; means for modifying the 3D model based on the collected preferences; A system including means for providing said modified 3D model to a user and a practitioner.

2. The system of claim 1 , further comprising means for iteratively modifying the 3D model based on user feedback.

3. 2. The system of claim 1, wherein said interactively gathering desired details means includes a question generation module, said question generation module including means for generating additional specific questions based on user responses.

Images