System

Abstract

A system is provided.SOLUTION: A system comprising: means for performing real-time voice recognition; means for dynamically generating a question based on the recognized voice; means for displaying the generated question in a 3D hologram to present the question to a user; means for receiving an answer of the user by voice and recognizing the answer again; and means for recording a history of the recognition result and the question.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional medical interview systems often have difficulty accurately conveying patients' symptoms, and interviews are often insufficient, especially for children and the elderly. Furthermore, the collected interview data is incomplete, making it difficult for medical professionals to provide diagnosis and follow-up care. Furthermore, even after consultation, there is a problem in that follow-up care, such as medication management, insurance guidance, and schedule management, is not adequately provided in the home environment. The present invention aims to solve these problems and realize accurate and comprehensive health management and follow-up care. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for performing real-time speech recognition, a means for dynamically generating questions based on recognized speech data, a means for displaying the generated questions as 3D holograms and presenting them to the user, a means for receiving the user's voice responses and re-recognizing them, a means for recording the recognition results and a history of the questions, a means for sharing the recognition results and recorded data with medical professionals, and a means for performing post-consultation follow-up based on health data. This allows for an accurate understanding of the user's symptoms and allows for appropriate medical intervention. Furthermore, comprehensive health data management and follow-up ensures reliable health management after consultations.

[0006] "Real-time" refers to the immediacy of time, allowing for immediate processing and response to a user's voice input.

[0007] "Speech recognition" refers to the technology that analyzes a user's voice data and converts it into corresponding text data.

[0008] "Generating questions" refers to the process of dynamically creating the next necessary question using an AI model or other means based on the user's answer obtained through voice recognition.

[0009] "3D hologram" refers to a virtual character that is displayed realistically using three-dimensional images in order to visually interact with the user.

[0010] "Recognition results" refers to the text data obtained by speech recognition and the analysis results thereof.

[0011] "Question History" means a record of a series of questions asked to a User and the User's answers to those questions.

[0012] "Medical professionals" refers to professionals with specialized medical knowledge, such as doctors, nurses, and pharmacists.

[0013] "Follow-up" refers to ongoing support and guidance to maintain and improve a user's health.

[0014] "Medication management" refers to the process of setting schedules and reminders to ensure users take their medications appropriately, and monitoring and supporting their implementation.

[0015] "Insurance guidance" refers to providing users with guidelines for using health insurance and related systems, and providing guidance to encourage appropriate use.

[0016] "Schedule management" refers to the process of managing a user's medical-related schedules (checkups, medical examinations, prescription renewals, etc.) and notifying the user at the appropriate time. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, data sharing, and post-consultation follow-up.

[0039] Overview of program processing

[0040] server

[0041] 1. User registration and initial data settings

[0042] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[0043] Create and manage your own healthcare profile.

[0044] 2. Dynamic Questionnaire Generation

[0045] When a user initiates a consultation, the server generates a predefined initial list of questions.

[0046] The AI ​​model receives the speech recognition results and dynamically generates the next question based on those results.

[0047] This process allows for appropriate medical interviews to be conducted based on the user's symptoms.

[0048] 3. Data sharing and post-examination support

[0049] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[0050] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[0051] Terminal

[0052] 1. 3D hologram generation

[0053] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[0054] The hologram presents the user with a series of generated questions.

[0055] 2. Voice Recognition and Feedback

[0056] It receives the user's voice response and performs real-time voice recognition.

[0057] The voice recognition results are sent to the server, which receives the next question and displays it on the hologram.

[0058] 3. Healthcare data management and notification

[0059] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[0060] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[0061] User

[0062] 1. Starting and answering the medical interview

[0063] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[0064] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[0065] 2. Entering daily health data

[0066] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[0067] The data is sent to the server and the user's profile is updated.

[0068] 3. Notification and follow-up

[0069] Receive notifications from the hologram when it's time to take your medication and take it as instructed.

[0070] If you have a scheduled health check or medical examination, you will be notified the day before or just before.

[0071] Specific examples

[0072] If user A has a history of high blood pressure, the following specific process is executed:

[0073] 1. Initial data settings

[0074] User A uses a terminal to enter his / her name, age, and medical history of high blood pressure, and sends the information to the server.

[0075] The server receives this information and creates a healthcare profile for User A.

[0076] 2. Start of interview

[0077] When user A begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[0078] User A answers by voice, "My blood pressure has been high recently."

[0079] 3. Dynamic Questionnaire Generation

[0080] The server inputs the speech recognition results into the AI ​​model and dynamically generates the next question: "Do you exercise?"

[0081] This question is displayed on the device via a hologram, and User A answers, "I haven't been exercising much lately."

[0082] 4. Data recording and sharing

[0083] Each response is sent in real time to a server and recorded.

[0084] This data will be shared with your family doctor and used as diagnostic material.

[0085] 5. Post-examination support

[0086] User A's medication schedule is generated on the server and sent to the terminal.

[0087] When it's time to take your medication, a hologram on the device will notify you, "Take your medication now."

[0088] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[0089] The processing flow will be explained below.

[0090] Step 1:

[0091] The user presses the start button on the terminal, which starts the interview process.

[0092] Step 2:

[0093] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[0094] Step 3:

[0095] The user answers the question verbally, for example, "I'm a little tired."

[0096] Step 4:

[0097] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[0098] Step 5:

[0099] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[0100] Step 6:

[0101] The device receives the next question from the server and displays it via a 3D hologram.

[0102] Step 7:

[0103] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[0104] Step 8:

[0105] The device performs voice recognition again, converts it into text data, and sends it to the server.

[0106] Step 9:

[0107] The server repeats the process of generating more questions based on the new recognition results, and continues interrogating until all the necessary information is collected.

[0108] Step 10:

[0109] The server stores all collected data in a database and updates the user's healthcare profile.

[0110] Step 11:

[0111] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[0112] Step 12:

[0113] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[0114] Step 13:

[0115] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[0116] Step 14:

[0117] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[0118] Step 15:

[0119] When it's time to take your medication or when your health checkup is scheduled, the device will notify you via a 3D hologram, displaying a reminder such as "Take your medicine now."

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Conventional health management systems require a lot of effort for user data entry and follow-up, making it difficult to accurately interview patients and share data in real time. Furthermore, there are limitations in the accuracy of voice recognition and the generation of dynamic questions, making it difficult to provide appropriate responses based on the user's symptoms. Furthermore, follow-up after consultations is often insufficient, resulting in inadequate ongoing health management.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes a means for receiving user registration and basic information and storing it in a database, a means for performing real-time speech recognition, and a means for dynamically generating questions using prompts generated based on the recognized speech data, thereby enabling accurate collection of user health data in real time, continuous follow-up, and comprehensive health management.

[0125] "User registration" is the process of entering basic user information into the system and storing it in the database.

[0126] "Speech recognition" is a technology that receives a user's voice and interprets it as digital data.

[0127] A "prompt" is the text for the next question generated based on an AI model.

[0128] "Dynamic question generation" is the process of generating the next question in real time based on the user's speech recognition results.

[0129] "3D hologram display" is a method of visually presenting questions and notifications to users using three-dimensional imaging technology.

[0130] "Recognition result" refers to text data obtained by speech recognition.

[0131] "Question History" is a record of all questions posed to a user and their answers.

[0132] "Health professionals" are people whose job it is to evaluate health data and make diagnoses and treatment decisions.

[0133] "Data sharing" is the process of sharing collected data with another user or system.

[0134] "Follow-up" refers to ongoing support activities to help users maintain their health after a medical examination.

[0135] "Healthcare data" refers to data that represents the user's daily health condition (body temperature, blood pressure, weight, etc.).

[0136] "Drug management" is the process of properly managing the timing and amount of medication a user takes.

[0137] "Insurance guidance" is the process of providing information and advice related to a user's health insurance.

[0138] "Appointment management" is the process of scheduling a user's health-related appointments (such as doctor's appointments and checkups) and providing appropriate notifications.

[0139] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, generative AI models, data sharing, and follow-up functions. The specific hardware and software usage and data processing procedures are described below.

[0140] Server configuration and processing

[0141] 1. User registration and initial data settings

[0142] The server receives basic user information (such as name, age, medical history, etc.) from the registration form and stores it in a database. Based on this data, a healthcare profile is generated for each user.

[0143] 2. Dynamic Questionnaire Generation

[0144] When the user starts the medical interview, the server sends an initial list of questions to the device. After receiving the user's voice responses, the server analyzes them using speech recognition software (e.g., Google Cloud Speech-to-Text API). Based on the analysis results, a generative AI model (e.g., BERT or GPT-3) generates the next questions.

[0145] 3. Data sharing and post-examination support

[0146] The collected medical interview data and daily health care data are shared with the patient's primary care physician, and a schedule for follow-up visits after the visit is generated and sent to the device.

[0147] Terminal configuration and handling

[0148] 1. 3D hologram generation

[0149] When the user begins the consultation, the device activates a hologram using 3D holographic technology. The first question is displayed as a hologram and presented to the user. An example of the hardware used is a display from the Looking Glass Factory.

[0150] 2. Voice Recognition and Feedback

[0151] The device receives the user's voice response through a microphone, analyzes it using speech recognition software, and sends the results to the server, which then receives the next dynamically generated question and displays it on the hologram.

[0152] 3. Healthcare data management and notification

[0153] The device allows users to input their daily health data (body temperature, blood pressure, weight, etc.) and synchronizes it with a server. It notifies users using holograms when it's time to take their medication or when medical appointments are approaching.

[0154] User operations

[0155] 1. Starting and answering the medical interview

[0156] The user presses a button on the device to start the medical interview and answers the questions posed by the hologram by voice. For example, the initial question, "How are you feeling right now?" is answered with, "I'm a little tired."

[0157] 2. Entering daily health data

[0158] Users input their daily health data into the device, for example, by measuring their body temperature with a thermometer, and the data is then sent to the server.

[0159] 3. Notification and follow-up

[0160] When it's time to take their medication, the user will receive a notification from the hologram on the device saying, "Take your medicine now," and will follow the instructions to take their medicine. They will also receive notifications the day before or just before scheduled medical checkups or appointments.

[0161] Specific examples

[0162] Prompt Sentence Examples

[0163] Initial question: "How are you feeling right now?"

[0164] Voice response: "I'm a little tired"

[0165] Next question: "Do you exercise?"

[0166] Voice response: "I haven't been exercising much lately."

[0167] As described above, this system utilizes voice recognition and generative AI models to dynamically grasp the user's health status and provide appropriate questions and follow-ups, enabling comprehensive health management.

[0168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0169] Step 1:

[0170] User registration and initial data settings

[0171] Users enter basic information such as their name, age, and medical history into a registration form on their device. The device then sends this information to the server, which then stores the received information in a database and creates a healthcare profile for each user. Specifically, the server generates a user ID and associates the entered data with that ID and stores it.

[0172] Input: User's basic information (name, age, medical history)

[0173] Output: User ID, healthcare profile

[0174] Step 2:

[0175] Beginning of medical interview

[0176] The user presses the "Start medical interview" button on the device. The device retrieves a list of initial questions from the server and activates a 3D hologram to display the initial questions. The user answers the questions verbally. The device receives the user's voice responses through a microphone and performs voice recognition in real time. The voice recognition results are sent to the server. Specifically, the voice data is converted into text data, and that text data is sent to the server.

[0177] Input: Press the start button and answer by voice

[0178] Output: Initial question list, speech-recognized text data

[0179] Step 3:

[0180] Dynamic interview generation

[0181] The server receives the speech recognition results and inputs them into a generative AI model. The AI ​​model (e.g., BERT or GPT-3) generates the next question based on the speech recognition results and the prompt. The server then sends the generated question to the device. The device displays the next question as a 3D hologram and presents it to the user. This allows the server to dynamically generate and display appropriate questions based on the user's specific symptoms.

[0182] Input: Speech recognition result (text), prompt

[0183] Output: Next question (text)

[0184] Step 4:

[0185] Voice Recognition and Feedback

[0186] The device receives the user's new voice response and performs voice recognition again. It sends the voice recognition result to the server and waits for the next appropriate question. This enables continuous dynamic interviews. Specifically, the device converts voice into text in real time and sends the text data to the server.

[0187] Input: Voice response

[0188] Output: Recognized text data

[0189] Step 5:

[0190] Data recording and sharing

[0191] The server records all medical interview data and voice recognition results. The recorded data is shared with medical professionals as needed. Medical professionals can use this data as a reference for diagnosis and treatment. Specifically, the server stores the data in a database and makes some or all of the data accessible to medical professionals.

[0192] Input: Voice recognition results, medical interview data

[0193] Output: Recorded interview data, data shared with medical professionals

[0194] Step 6:

[0195] Healthcare data management and notification

[0196] Users enter their daily health data (such as temperature, blood pressure, and weight) into the device. The device then sends this data to a server, which updates the user's health profile. The device also uses 3D holograms to notify users when it's time to take medication or when medical appointments are approaching. For example, a notification like "Take your medicine now" may appear on the hologram.

[0197] Input: Daily health data (temperature, blood pressure, weight, etc.), schedule data

[0198] Output: Updated health profile, notification message

[0199] Step 7:

[0200] Follow-up after consultation

[0201] The server generates a schedule for follow-ups after the consultation and sends it to the device. The user manages medication and health checkup schedules according to notifications from the device. The server sends follow-up notifications in a timely manner based on the user's healthcare data and schedule. Specifically, the server calculates medication schedules and health checkup schedules, and sends them to the device for display.

[0202] Input: Follow-up schedule, healthcare data

[0203] Output: Notification messages, management schedules

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Conventional health management systems lack the ability to dynamically generate questions in response to user inquiries or real-time voice recognition and feedback using 3D holograms, making it difficult to provide comprehensive and personalized support for users' health management. They also lack a system for sharing generated data with medical professionals for appropriate follow-up. Furthermore, they are unable to provide individually customized health advice based on each user's health information, resulting in a lack of accuracy and usefulness of the health advice.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for performing real-time speech recognition, means for dynamically generating questions based on the recognized speech data, means for displaying the generated questions as 3D holograms and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and a history of the questions, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for generating and displaying individually customized health advice based on each user's health information, thereby enabling accurate and comprehensive management of the user's health condition in real time and providing individually customized health advice.

[0209] "Means for real-time speech recognition" refers to technology that can instantly convert a user's speech into digital data and analyze its content.

[0210] "Means for dynamically generating questions based on recognized voice data" refers to a technology in which an AI model automatically creates appropriate questions based on the user's current state, based on analyzed voice data.

[0211] "Means of displaying and presenting to users as 3D holograms" refers to a technology that visually provides users with content generated using three-dimensional video technology.

[0212] "Means for receiving and re-recognizing the user's voice response" refers to a technology that analyzes the user's voice response again using voice recognition technology and reflects the results in the system.

[0213] The "means for recording the recognition results and the history of the questions" is a technology for saving the speech recognition results and the history of the questions generated as data.

[0214] "Means for sharing recorded data with medical professionals" refers to technology that securely and accurately transfers stored voice recognition results and question history data to medical professionals.

[0215] "Means for post-medical follow-up based on health data" refers to technology that analyzes a user's daily health data and, based on that data, provides appropriate guidance and advice after a medical visit.

[0216] "A means for generating and displaying individually customized health advice based on each user's health information" refers to a technology that automatically generates specific health advice based on each user's individual health condition and history, and presents it using 3D holograms, etc.

[0217] This invention is a system that performs real-time voice recognition, dynamically generates questions based on the data, and presents them to the user using a 3D hologram. The system also receives the user's answers via voice and records the recognition results and question history. It also shares the recognition results and recorded data with medical professionals, who use the health data to provide follow-up after consultations. It also includes a means for generating and displaying individually customized health advice based on each user's health information.

[0218] Server Operation

[0219] The server is responsible for user registration and initial data configuration. It receives the user's basic information (name, age, medical history) and stores it in a database. It is also equipped with an AI model that receives voice recognition results and dynamically generates the next question. The generated questions and voice recognition results are recorded in a database and shared with medical professionals along with health data. After the consultation, a schedule for medication management and appointment management is generated and sent to the user's device.

[0220] Device behavior

[0221] The device uses 3D holograms to display dynamically generated questions to the user. It recognizes the user's answers in real time and sends the results to the server. The user's daily health data (body temperature, blood pressure, weight, etc.) is also entered into the device and synchronized with the server. The device notifies the user via 3D holograms when it is time to take medication or when medical appointments are approaching.

[0222] User Actions

[0223] The user presses the button to start the medical interview, answers the questions posed by the hologram by voice, and enters their daily health data into the device. For example, to the question, "How are you feeling right now?", the user can reply, "I'm a little tired." The voice response is recognized in real time, and the next question is presented through the hologram. The user also receives notifications and follows instructions when it's time to take medication or when medical appointments are approaching.

[0224] Specific Examples

[0225] If User A has a history of high blood pressure, the following process is executed. User A enters their name, age, and history of high blood pressure and sends them to the server. The server uses this information to create a healthcare profile for User A. When User A begins the medical interview, a 3D hologram on the device displays the question, "How is your current health?" User A replies, "My blood pressure has been high recently." The server inputs the voice recognition results into an AI model, which dynamically generates the next question, "Are you exercising?" This question is displayed via the hologram, and User A replies, "I haven't been exercising much recently."

[0226] Prompt Sentence Examples

[0227] User: "How are you feeling these days?"

[0228] Answer: "I have high blood pressure and get tired easily."

[0229] Question to prompt: "Do you exercise?"

[0230] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1: User Registration

[0233] Users enter basic information such as their name, age, and medical history into the device and press the registration button. The device then sends this data to the server, which stores the received information in a database and creates a healthcare profile for each user.

[0234] Input: Name, age, medical history

[0235] Output: Generate a healthcare profile

[0236] Specific behavior: Inserts information into the database and creates an initial profile

[0237] Step 2: Begin the interview

[0238] The user presses the button to start the medical interview, and the device uses a 3D hologram to display the first question: "How are you feeling right now?" The user then answers this question verbally. The device recognizes the user's voice in real time, converts it into digital data, and sends it to the server.

[0239] Input: Voice input "I'm a little tired"

[0240] Output: Speech recognition results as digital data

[0241] Specific operation: Converts speech into text using a speech recognition engine and sends it to the server

[0242] Step 3: Dynamic Questionnaire Generation

[0243] The server analyzes the received voice recognition data and inputs it into the AI ​​model. The AI ​​model dynamically generates the next question: "Do you exercise?" The generated question is sent to the device.

[0244] Input: Speech recognition result: "I'm a little tired"

[0245] Output: Next question: "Do you exercise?"

[0246] How it works: Uses AI models to dynamically generate the next question

[0247] Step 4: Ask the next question

[0248] The device then displays the next question as a 3D hologram and presents it to the user. The user answers, "I haven't been exercising much lately." The device again recognizes the voice in real time and sends the results to the server.

[0249] Input: Voice input "I haven't been exercising much lately"

[0250] Output: The following speech recognition results stored in the database:

[0251] Specific operation: Displaying questions using 3D holograms, recognizing voice and sending it to the server

[0252] Step 5: Record data and share with medical professionals

[0253] The server records the voice recognition results and question history in a database, which is shared with medical professionals periodically or upon request.

[0254] Input: Speech recognition result: "I haven't been exercising much lately."

[0255] Output: Data shared with medical professionals

[0256] Specific actions: Recorded in a database and shared with medical professionals through a dedicated interface

[0257] Step 6: Enter and sync your daily health data

[0258] Users input their daily health data, such as body temperature, blood pressure, and weight, into the device, which then sends this data to the server, which stores it in a database.

[0259] Input: Health data such as body temperature, blood pressure, and weight

[0260] Output: Updated healthcare profile

[0261] Specific operation: Enter health data, send it to the server and save it

[0262] Step 7: Follow-up after the visit

[0263] The server generates a schedule for medication management and schedule management as follow-up after the consultation and sends it to the terminal. The terminal checks this schedule and notifies the user at the specified time using a 3D hologram.

[0264] Input: Health data, feedback from medical professionals

[0265] Output: Follow-up schedule and notifications

[0266] Specific actions: Follow-up schedule generation and notifications using 3D holograms

[0267] Step 8: Generate personalized health advice

[0268] The server uses AI models to generate personalized health advice based on each user's health information, which is then sent to the device and displayed to the user as a 3D hologram.

[0269] Input: User health information and daily data

[0270] Output: Customized health advice

[0271] How it works: Uses AI models to generate health advice and displays it in a 3D hologram

[0272] This series of steps enables accurate and comprehensive management of the user's health status in real time and provides individually customized health advice.

[0273] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0274] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system integrates real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up to provide accurate and comprehensive health management.

[0275] Overview of program processing

[0276] server

[0277] 1. User registration and initial data settings

[0278] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[0279] Create and manage your own healthcare profile.

[0280] 2. Dynamic Questionnaire Generation

[0281] When a user initiates a consultation, the server generates a predefined initial list of questions.

[0282] The AI ​​model receives the speech recognition results and the emotion data recognized by the emotion engine, and dynamically generates the next question based on the results.

[0283] This process allows for appropriate interviews to be conducted based on the user's symptoms and emotions.

[0284] 3. Data sharing and post-examination support

[0285] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[0286] Sharing emotional data will also help medical professionals make more accurate diagnoses.

[0287] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[0288] Terminal

[0289] 1. 3D hologram generation

[0290] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[0291] The hologram presents the user with a series of generated questions.

[0292] 2. Speech and Emotion Recognition

[0293] It receives the user's voice response and performs real-time voice recognition.

[0294] The emotion engine analyzes the user's voice and facial expressions when answering and generates emotional data.

[0295] 3. Feedback

[0296] The user's voice recognition results and emotion data are sent to the server, which receives the next question and displays it on the hologram.

[0297] If the emotional data indicates "anxiety" or "tension," the hologram provides feedback to help the user relax.

[0298] 4. Healthcare Data Management and Notification

[0299] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[0300] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[0301] User

[0302] 1. Starting and answering the medical interview

[0303] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[0304] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[0305] 2. Entering daily health data

[0306] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[0307] The data is sent to the server and the user's profile is updated.

[0308] 3. Collecting emotional data and receiving feedback

[0309] Receives feedback based on emotional data from the hologram and engages in adaptive dialogue.

[0310] For example, if a user responds, "I'm feeling a little anxious right now," the hologram will offer encouraging words such as, "It sounds like you're feeling anxious. I suggest you take some time off."

[0311] Specific examples

[0312] Assuming that user B is experiencing mental stress, the following specific process will be carried out:

[0313] 1. Initial data settings

[0314] User B uses the terminal to enter his / her name, age, and medical history of mental stress, and sends the information to the server.

[0315] The server receives this information and creates a healthcare profile for User B.

[0316] 2. Start of interview

[0317] When User B begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[0318] User B responds verbally, "I've been feeling stressed lately."

[0319] 3. Dynamic Questionnaire Generation and Emotion Recognition

[0320] The server inputs the voice recognition results and the "anxiety" emotion generated by the emotion engine into the AI ​​model, and dynamically generates the next question: "In what situations do you feel stressed?"

[0321] This question is displayed on the device via a hologram, and User B answers, "It's mainly pressure at work."

[0322] 4. Data recording and sharing

[0323] Each response and emotional data is sent to a server in real time and recorded.

[0324] The collected data will be shared with your family doctor and used as diagnostic material.

[0325] 5. Post-examination support

[0326] A stress management schedule for user B is generated on the server and sent to the terminal.

[0327] When it's time to take medication or engage in relaxation activities, a hologram on the device will notify you, saying, "Please perform your relaxation exercises now."

[0328] In this way, the present invention comprehensively manages the user's health status and emotional data, realizing accurate medical interviews and continuous follow-up.

[0329] The processing flow will be explained below.

[0330] Step 1:

[0331] The user presses the start button on the terminal, which starts the interview process.

[0332] Step 2:

[0333] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[0334] Step 3:

[0335] The user answers the question verbally, for example, "I'm a little tired."

[0336] Step 4:

[0337] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[0338] Step 5:

[0339] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[0340] Step 6:

[0341] The device receives the next question from the server and displays it via a 3D hologram.

[0342] Step 7:

[0343] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[0344] Step 8:

[0345] The device sends the user's voice and facial expressions to the emotion engine, which analyzes the user's emotions and generates emotion data indicating, for example, "anxiety" or "tension."

[0346] Step 9:

[0347] The device performs voice recognition again and transmits the text data and emotion data to the server.

[0348] Step 10:

[0349] The server repeats the process of generating further questions based on new recognition results and emotion data, continuing the interview process until all necessary information is collected. For example, if the emotion data indicates "anxiety," it generates additional questions that take emotion into consideration, such as "Have you been sleeping well recently?"

[0350] Step 11:

[0351] The server stores all collected data in a database and updates the user's healthcare profile.

[0352] Step 12:

[0353] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[0354] Step 13:

[0355] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[0356] Step 14:

[0357] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[0358] Step 15:

[0359] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[0360] Step 16:

[0361] The device will notify the user via a 3D hologram when it is time to take medication, when a medical checkup is scheduled, etc. For example, if the emotional data indicates "anxiety" or "stress," a reminder such as "We recommend you perform a relaxation exercise now" will be displayed.

[0362] Example 2

[0363] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Conventional health management systems lack the functionality to analyze users' voice and emotional data in real time and provide appropriate medical feedback. Furthermore, they lack a system that can centrally manage follow-up after medical examinations and daily health data management. As a result, it is difficult for users to accurately and comprehensively understand their own health status.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0366] In this invention, the server includes means for performing real-time voice recognition, means for dynamically generating questions based on the recognized voice data and emotion data, means for displaying the generated questions as a 3D image and presenting them to the user, means for receiving the user's voice responses and re-recognizing them, means for recording the recognition results and emotion data, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for providing feedback according to the user's emotion data. This enables comprehensive health management, including analyzing the user's health condition in real time, providing appropriate feedback based on the emotion data, and post-examination follow-ups.

[0367] "Means for real-time speech recognition" refers to devices or software that have the ability to instantly convert a user's voice into text data and process it in real time.

[0368] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a system that has the function of automatically generating the next question based on the situation using the results of voice recognition and emotional analysis.

[0369] "Means for displaying generated questions in 3D images and presenting them to the user" refers to a system that has the function of visually presenting questions generated for dialogue with the user using a 3D image display device.

[0370] "Means for receiving and re-recognizing the user's voice response" refers to devices or software that have the function of converting the user's voice response into text data using voice recognition technology and analyzing it.

[0371] "Means for recording the recognition results and emotion data" refers to a system that has the function of storing the voice recognition results and emotion analysis results in a recording medium such as a database.

[0372] "Means for sharing the recognition results and recorded data with medical professionals" refers to a system that has the function of securely transferring and sharing speech recognition results and recorded data with medical professionals.

[0373] "Means for post-visit follow-up based on health data" refers to a system that has the functionality to manage medication, guidance, and schedules after a visit based on the user's health data.

[0374] "Means for providing feedback based on the user's emotional data" refers to a system that has the function of analyzing the user's emotional data and providing appropriate advice or words of encouragement based on the results.

[0375] "Means for integrated management of health data" refers to a system that has the function of centrally collecting, analyzing, and managing users' daily health data.

[0376] "Medication management" refers to a system that manages a user's medication schedule and notifies them to take their medication appropriately.

[0377] "Instruction management" refers to a system that has the function of managing the content of instruction provided by medical professionals and providing appropriate instructions and advice to users.

[0378] "Schedule management" refers to a system that has the function of managing a user's medical-related schedules and notifying them at the appropriate time.

[0379] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up.

[0380] Server Roles

[0381] The server receives the user's basic information and stores it in a database. This information includes name, age, and medical history. For example, data such as "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure" is entered. The server uses this information to create a healthcare profile for each user.

[0382] Furthermore, when the user begins the medical interview, the server generates a predefined list of initial questions. For example, it generates a question such as, "How are you feeling right now?". The AI ​​model then dynamically generates the next question based on the speech recognition results and emotion data obtained by the emotion engine. This results in a dynamic question such as, "Since when have you been feeling tired?"

[0383] The server shares the collected medical interview data and daily health care data with the patient's primary care physician. This includes voice and emotional data. Schedules for medication management, instruction management, and schedule management are then generated and sent to the device. For example, instructions such as "Take your medicine at 8:00 every morning" are generated.

[0384] Device Role

[0385] When the user begins the medical interview, the device activates a 3D hologram and displays the first question. The hologram speaks, "Hello, how are you feeling right now?" The device recognizes the user's voice response in real time, and the emotion engine analyzes the emotion at the time of the response. For example, if the user responds, "I've been feeling stressed lately," the voice is converted into text and emotion data is generated.

[0386] The device then sends the user's voice recognition results and emotional data to the server, which then asks the next question and displays it on the hologram. For example, if the emotional data indicates anxiety or tension, the hologram might suggest, "Would you like to know how to relax?"

[0387] The device inputs the user's daily health data, such as temperature, blood pressure, and weight, and synchronizes it with the server. In addition, when it is time to take medication or when medical appointments are approaching, the device notifies the user via a hologram. For example, it issues an alert saying, "It's time to take your medicine."

[0388] User Roles

[0389] The user presses a button on the device to start the medical interview and answers the questions from the hologram by voice. For example, the hologram might ask, "How are you feeling right now?" and the user might reply, "I'm a little tired." Next, the user enters their daily health data (e.g., body temperature, blood pressure, weight) into the device, which then sends that data to the server to update their profile. For example, they might enter, "Today's body temperature is 36.5 degrees."

[0390] The hologram responds by providing feedback based on the user's emotional data, allowing for adaptive interactions. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will respond with encouragement, "It seems you're feeling anxious. Perhaps it would be good for you to take a short break."

[0391] Specific examples

[0392] If user B is experiencing mental stress, the following specific process will occur:

[0393] User B uses the device to enter their name, age, and medical history of mental stress, and sends this to the server. The server creates a healthcare profile. When the medical interview begins, a 3D hologram on the device displays the question, "How is your current health condition?" User B responds verbally, "I've been feeling very stressed lately." Based on the voice recognition results and the "anxiety" emotion generated by the emotion engine, the server dynamically generates the next question: "In what situations do you feel stressed?"

[0394] Each response and emotional data are sent to the server in real time and recorded. This data is then provided to the user's doctor as diagnostic information. User B's stress management schedule is generated by the server and sent to the device. For example, when it is time to take medication or engage in relaxation activities, a hologram on the device will notify the user, saying, "Please perform relaxation exercises now."

[0395] Example prompts for generative AI models

[0396] 1. "If a user has been stressed recently, what follow-up questions should be generated?"

[0397] 2. "If the emotion engine determines that the user is anxious, please suggest the appropriate feedback to provide next."

[0398] 3. "Generate follow-up questions based on the user's daily health data (e.g., body temperature, blood pressure)."

[0399] As described above, this system comprehensively manages the user's health status and emotional data, enabling accurate medical interviews and continuous follow-up.

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] User registration and initial data settings

[0403] The server receives the user's basic information (name, age, medical history) as input. It then stores this information in a database and creates a healthcare profile for each user. This profile serves as the foundation for the user to begin their medical interview. For example, if a user enters "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure," this data will be registered in the database.

[0404] Step 2:

[0405] Start of interview and generation of initial questions

[0406] When the user presses the start interview button, the terminal activates a 3D hologram. The hologram displays the first question to the user, "How is your current health condition?", and asks the question aloud. The input here is "pressing the start interview button," and the output is "displaying the initial question."

[0407] Step 3:

[0408] Speech and Emotion Recognition

[0409] The device receives the user's voice response as input and performs real-time voice recognition. The emotion engine then analyzes the user's voice and facial expressions to generate emotion data. At this stage, the input is the "voice response," and the output is the "recognized text data and emotion data." For example, if the user responds "I'm a little tired," the voice is converted into text data saying "I'm a little tired," and the emotion data generated is "anxiety."

[0410] Step 4:

[0411] Dynamic generation of next question

[0412] The server receives the speech recognition results and emotion data sent from the device as input. The AI ​​model then analyzes these data and dynamically generates the next question. At this stage, the input is the "speech recognition results and emotion data," and the output is the "next question." For example, the next question generated might be, "Since when have you been feeling tired?"

[0413] Step 5:

[0414] View next question and receive answer

[0415] The device receives the next question from the server, displays it on a 3D hologram, and presents it to the user. The user responds vocally, and the device again performs voice and emotion recognition. The input / output cycle is repeated. The input is "generation of the next question," and the output is "display of the next question and a vocal response."

[0416] Step 6:

[0417] Data recording and sharing

[0418] The server records the collected speech recognition results, emotion data, and medical history data in a database. Furthermore, a secure data transmission protocol is used to share important data with medical professionals. The input is "collected data," and the output is "recording in the database and sharing with medical professionals."

[0419] Step 7:

[0420] Follow-up schedule generation

[0421] The server generates a schedule for post-consultation medication management, instruction management, and schedule management based on the user's health data and medical interview results. The input is "health data and medical interview results," and the output is "follow-up schedule." For example, a medication instruction such as "Take your medicine every morning at 8 o'clock" is generated.

[0422] Step 8:

[0423] Providing Feedback

[0424] The device notifies the user of the generated follow-up schedule via a hologram. It also provides feedback based on the user's emotional data. The input is "follow-up schedule and emotional data," and the output is "notification and feedback." For example, the hologram may notify the user, "Please perform relaxation exercises now."

[0425] In this way, it is possible to comprehensively manage the user's health status and emotional data, and realize accurate medical interviews and continuous follow-up.

[0426] (Application example 2)

[0427] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] In modern society, there is a demand for efficient health management and related follow-up for users. However, conventional health management systems do not adequately provide comprehensive support that takes into account users' emotional states or personalized product recommendations. Therefore, a new system is needed to improve the quality of users' health management.

[0429] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0430] In this invention, the server includes means for recognizing the user's voice in real time, means for dynamically generating questions based on the recognized voice data and emotional data, means for displaying the generated questions in 3D images and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and question history, means for sharing the recognition results and recorded data and emotional data, means for conducting post-examination follow-up based on health data, and means for recommending products based on preferences. This makes it possible to conduct medical interviews and follow-ups that reflect the user's emotional state, and further improve the quality of the user's health management through personalized product recommendations.

[0431] "Means for real-time speech recognition" refers to a function that instantly converts voice data provided by a user into text data.

[0432] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a function that analyzes the voice recognition results and the user's emotional state, and automatically generates appropriate questions based on these.

[0433] "Means for displaying the generated question in a 3D image and presenting it to the user" refers to a function that uses hologram technology or other 3D image display technology to visually show the generated question to the user.

[0434] The "means for receiving and re-recognizing the user's voice response" is a function for re-recognizing the user's verbal response and converting it into data.

[0435] The "means for recording the recognition results and question history" is a function for saving the speech recognition results and the history of questions and answers generated at that time in a database or the like.

[0436] "Means for sharing recognition results, recorded data, and emotional data" refers to the function of sharing recognition results, recorded question and answer history, and emotional data with other systems and experts.

[0437] "Means for post-consultation follow-up based on health data" is a function that utilizes health data to take appropriate measures to manage and support the user's health after a consultation.

[0438] "Preference-based product recommendation means" is a function that automatically suggests optimal products based on the user's preferences and health condition.

[0439] MODE FOR CARRYING OUT THE INVENTION

[0440] This is a health management system for online shopping sites that combines 3D holograms, voice recognition, AI models, an emotion engine, data sharing, and follow-up functions. The system grasps the user's health and emotional state in real time, and based on that, conducts appropriate medical interviews, follow-ups, and recommends related products.

[0441] System Overview

[0442] The system consists of three main components:

[0443] 1. Server

[0444] 2. Device (smartphone, etc.)

[0445] 3. Users

[0446] server

[0447] The server's main responsibilities are:

[0448] User registration and initial data setup: The server receives basic information provided by the user (such as name, age, and health status) and stores it in a database, which then creates a health profile for each user.

[0449] Dynamic Question Generation: Based on the speech recognition results and emotion data, the next question is dynamically generated using an AI model. In this process, the most appropriate question is presented to the user.

[0450] Data sharing: Collected voice data, emotional data, health data, etc. will be shared with the user's permission, allowing for more accurate support to be provided to the user.

[0451] Post-consultation follow-up and product recommendations: Based on the user's health and preference data, necessary follow-up and related product recommendations are provided.

[0452] Terminal

[0453] The device (e.g., smartphone) primarily performs the following functions:

[0454] 3D hologram generation and display: At the start of the interview, a 3D hologram is activated and presents questions to the user. The hologram displays the generated questions one after another, ensuring a smooth dialogue.

[0455] Speech and emotion recognition: The device recognizes the user's voice in real time and generates emotion data using an emotion engine. This data is sent to the server and used to generate the next question.

[0456] Feedback display: The voice recognition results and emotional data are sent to the server, and the next question is received from the server and displayed on the hologram. If the emotional data indicates "anxiety" or "tension," the device has the function of displaying feedback to relax the user.

[0457] User

[0458] The user performs the following actions:

[0459] Starting and answering the medical interview: Press the start button on the device and answer the questions from the hologram by voice. For example, in response to the initial question, "How are you feeling lately?", you can answer, "I'm a little tired."

[0460] Daily health data input: Enter your daily health data (body temperature, blood pressure, weight, etc.) into your device and synchronize it with the server, which will update your health profile in real time.

[0461] Collecting emotional data and receiving feedback: The hologram receives feedback based on emotional data collected during the interview and engages in adaptive dialogue. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will provide encouraging words such as, "It seems you're feeling anxious. I recommend you take a break."

[0462] Hardware and software used

[0463] Hardware: Smartphone (iOS or Android device)

[0464] software:

[0465] Speech Recognition: Converts speech to text using the speech_recognition library.

[0466] Emotion Recognition: Analyze emotions from speech and text using third-party emotion recognition libraries.

[0467] Generative AI Models: Use a generative AI model library to generate the next question.

[0468] Specific examples

[0469] For example, if a user enters "I've been feeling more stressed lately" in the default settings, the system will act as follows:

[0470] 1. The user answers verbally, "I've been busy at work lately and it's stressful."

[0471] 2. Speech to text: "Work has been busy and stressful lately."

[0472] 3. Sentiment analysis: Identify "stress."

[0473] 4. The AI ​​model generates the following question: "What situations make you feel particularly stressed?"

[0474] Prompt Sentence Examples

[0475] "Based on the analysis of the user's voice content and emotions, please generate the following appropriate questions. Voice content: 'Work has been busy lately and I'm stressed.' Emotion: 'Stressed.' Next question:"

[0476] This allows the medical interview process to proceed smoothly and allows for specific follow-up and product recommendations tailored to the user's health condition.

[0477] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0478] Step 1:

[0479] User Registration

[0480] 1. The server receives basic information entered by the user (name, age, health status) and stores it in a database.

[0481] Input: Basic information such as name, age, and health status.

[0482] Data processing: The entered information is registered in a database according to the format.

[0483] Output: User profile created.

[0484] Step 2:

[0485] Interview begins

[0486] 1. The user launches the app on their device and presses the button to start the medical interview.

[0487] Input: User operation (pressing the button to start the medical interview).

[0488] Data processing: Set the interview start flag.

[0489] Output: Start of the interview process.

[0490] 2. The device will activate a 3D hologram and display the first question.

[0491] Input: Interview start flag.

[0492] Data processing: Get the first question.

[0493] Output: Hologram of the question.

[0494] Step 3:

[0495] Speech and Emotion Recognition

[0496] 1. The user answers the first question by voice.

[0497] Input: Audio data.

[0498] Data processing: Acquisition of audio data.

[0499] Output: Save audio data.

[0500] 2. The device performs voice recognition and converts it into text data.

[0501] Input: Audio data.

[0502] Data processing: Text conversion using speech recognition (using the speech_recognition library).

[0503] Output: Text data.

[0504] 3. The emotion engine analyzes the voice and text data and generates emotion data.

[0505] Input: Audio and text data.

[0506] Data processing: Sentiment analysis (using emotion recognition libraries).

[0507] Output: Emotion data.

[0508] Step 4:

[0509] Dynamic interview generation

[0510] 1. The server receives the speech recognition results and emotion data and inputs a prompt sentence to the generative AI model to generate the next question.

[0511] Input: Speech recognition results, emotion data.

[0512] Data processing: Prompt sentence generation.

[0513] Output: The prompt statement.

[0514] 2. The server uses a generative AI model to dynamically generate the next question.

[0515] Input: prompt statement.

[0516] Data processing: The AI ​​model generates the next question.

[0517] Output: Next question.

[0518] Step 5:

[0519] Presenting the next question

[0520] 1. The server sends the following question to the terminal:

[0521] Input: Next question.

[0522] Data processing: Sending query data.

[0523] Output: Sends the next question to the terminal.

[0524] 2. The device uses a hologram to display the next question.

[0525] Input: Next question.

[0526] Data processing: Hologram generation and display.

[0527] Output: Hologram of the question.

[0528] Step 6:

[0529] Data recording and sharing

[0530] 1. The server records each response and emotion data in real time.

[0531] Input: Speech recognition results, emotion data, question history.

[0532] Data processing: Saving to database.

[0533] Output: Recorded data.

[0534] 2. The server shares the recorded data.

[0535] Input: Recorded data.

[0536] Data processing: Generating data for sharing.

[0537] Output: Sending shared data.

[0538] Step 7:

[0539] Follow-up and product recommendations

[0540] 1. The server generates a schedule for follow-up visits and preference-based product recommendations.

[0541] Input: Health data, preference data.

[0542] Data processing: Applying schedule generation and product recommendation algorithms.

[0543] Output: Follow-up schedule, product recommendation list.

[0544] 2. The device displays follow-up notifications and product recommendations to the user.

[0545] Input: Follow-up schedule, product recommendation list.

[0546] Data processing: generation of notification and display data.

[0547] Output: Follow-up notification, product recommendation display.

[0548] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0549] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0550] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0551] [Second embodiment]

[0552] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0553] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0554] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0555] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0556] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0557] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0558] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0559] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0560] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0561] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0562] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0563] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0564] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, data sharing, and post-consultation follow-up.

[0565] Overview of program processing

[0566] server

[0567] 1. User registration and initial data settings

[0568] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[0569] Create and manage your own healthcare profile.

[0570] 2. Dynamic Questionnaire Generation

[0571] When a user initiates a consultation, the server generates a predefined initial list of questions.

[0572] The AI ​​model receives the speech recognition results and dynamically generates the next question based on those results.

[0573] This process allows for appropriate medical interviews to be conducted based on the user's symptoms.

[0574] 3. Data sharing and post-examination support

[0575] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[0576] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[0577] Terminal

[0578] 1. 3D hologram generation

[0579] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[0580] The hologram presents the user with a series of generated questions.

[0581] 2. Voice Recognition and Feedback

[0582] It receives the user's voice response and performs real-time voice recognition.

[0583] The voice recognition results are sent to the server, which receives the next question and displays it on the hologram.

[0584] 3. Healthcare data management and notification

[0585] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[0586] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[0587] User

[0588] 1. Starting and answering the medical interview

[0589] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[0590] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[0591] 2. Entering daily health data

[0592] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[0593] The data is sent to the server and the user's profile is updated.

[0594] 3. Notification and follow-up

[0595] Receive notifications from the hologram when it's time to take your medication and take it as instructed.

[0596] If you have a scheduled health check or medical examination, you will be notified the day before or just before.

[0597] Specific examples

[0598] If user A has a history of high blood pressure, the following specific process is executed:

[0599] 1. Initial data settings

[0600] User A uses a terminal to enter his / her name, age, and medical history of high blood pressure, and sends the information to the server.

[0601] The server receives this information and creates a healthcare profile for User A.

[0602] 2. Start of interview

[0603] When user A begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[0604] User A answers by voice, "My blood pressure has been high recently."

[0605] 3. Dynamic Questionnaire Generation

[0606] The server inputs the speech recognition results into the AI ​​model and dynamically generates the next question: "Do you exercise?"

[0607] This question is displayed on the device via a hologram, and User A answers, "I haven't been exercising much lately."

[0608] 4. Data recording and sharing

[0609] Each response is sent in real time to a server and recorded.

[0610] This data will be shared with your family doctor and used as diagnostic material.

[0611] 5. Post-examination support

[0612] User A's medication schedule is generated on the server and sent to the terminal.

[0613] When it's time to take your medication, a hologram on the device will notify you, "Take your medication now."

[0614] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[0615] The processing flow will be explained below.

[0616] Step 1:

[0617] The user presses the start button on the terminal, which starts the interview process.

[0618] Step 2:

[0619] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[0620] Step 3:

[0621] The user answers the question verbally, for example, "I'm a little tired."

[0622] Step 4:

[0623] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[0624] Step 5:

[0625] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[0626] Step 6:

[0627] The device receives the next question from the server and displays it via a 3D hologram.

[0628] Step 7:

[0629] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[0630] Step 8:

[0631] The device performs voice recognition again, converts it into text data, and sends it to the server.

[0632] Step 9:

[0633] The server repeats the process of generating more questions based on the new recognition results, and continues interrogating until all the necessary information is collected.

[0634] Step 10:

[0635] The server stores all collected data in a database and updates the user's healthcare profile.

[0636] Step 11:

[0637] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[0638] Step 12:

[0639] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[0640] Step 13:

[0641] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[0642] Step 14:

[0643] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[0644] Step 15:

[0645] When it's time to take your medication or when your health checkup is scheduled, the device will notify you via a 3D hologram, displaying a reminder such as "Take your medicine now."

[0646] Example 1

[0647] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0648] Conventional health management systems require a lot of effort for user data entry and follow-up, making it difficult to accurately interview patients and share data in real time. Furthermore, there are limitations in the accuracy of voice recognition and the generation of dynamic questions, making it difficult to provide appropriate responses based on the user's symptoms. Furthermore, follow-up after consultations is often insufficient, resulting in inadequate ongoing health management.

[0649] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0650] In this invention, the server includes a means for receiving user registration and basic information and storing it in a database, a means for performing real-time speech recognition, and a means for dynamically generating questions using prompts generated based on the recognized speech data, thereby enabling accurate collection of user health data in real time, continuous follow-up, and comprehensive health management.

[0651] "User registration" is the process of entering basic user information into the system and storing it in the database.

[0652] "Speech recognition" is a technology that receives a user's voice and interprets it as digital data.

[0653] A "prompt" is the text for the next question generated based on an AI model.

[0654] "Dynamic question generation" is the process of generating the next question in real time based on the user's speech recognition results.

[0655] "3D hologram display" is a method of visually presenting questions and notifications to users using three-dimensional imaging technology.

[0656] "Recognition result" refers to text data obtained by speech recognition.

[0657] "Question History" is a record of all questions posed to a user and their answers.

[0658] "Health professionals" are people whose job it is to evaluate health data and make diagnoses and treatment decisions.

[0659] "Data sharing" is the process of sharing collected data with another user or system.

[0660] "Follow-up" refers to ongoing support activities to help users maintain their health after a medical examination.

[0661] "Healthcare data" refers to data that represents the user's daily health condition (body temperature, blood pressure, weight, etc.).

[0662] "Drug management" is the process of properly managing the timing and amount of medication a user takes.

[0663] "Insurance guidance" is the process of providing information and advice related to a user's health insurance.

[0664] "Appointment management" is the process of scheduling a user's health-related appointments (such as doctor's appointments and checkups) and providing appropriate notifications.

[0665] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, generative AI models, data sharing, and follow-up functions. The specific hardware and software usage and data processing procedures are described below.

[0666] Server configuration and processing

[0667] 1. User registration and initial data settings

[0668] The server receives basic user information (such as name, age, medical history, etc.) from the registration form and stores it in a database. Based on this data, a healthcare profile is generated for each user.

[0669] 2. Dynamic Questionnaire Generation

[0670] When the user starts the medical interview, the server sends an initial list of questions to the device. After receiving the user's voice responses, the server analyzes them using speech recognition software (e.g., Google Cloud Speech-to-Text API). Based on the analysis results, a generative AI model (e.g., BERT or GPT-3) generates the next questions.

[0671] 3. Data sharing and post-examination support

[0672] The collected medical interview data and daily health care data are shared with the patient's primary care physician, and a schedule for follow-up visits after the visit is generated and sent to the device.

[0673] Terminal configuration and handling

[0674] 1. 3D hologram generation

[0675] When the user begins the consultation, the device activates a hologram using 3D holographic technology. The first question is displayed as a hologram and presented to the user. An example of the hardware used is a display from the Looking Glass Factory.

[0676] 2. Voice Recognition and Feedback

[0677] The device receives the user's voice response through a microphone, analyzes it using speech recognition software, and sends the results to the server, which then receives the next dynamically generated question and displays it on the hologram.

[0678] 3. Healthcare data management and notification

[0679] The device allows users to input their daily health data (body temperature, blood pressure, weight, etc.) and synchronizes it with a server. It notifies users using holograms when it's time to take their medication or when medical appointments are approaching.

[0680] User operations

[0681] 1. Starting and answering the medical interview

[0682] The user presses a button on the device to start the medical interview and answers the questions posed by the hologram by voice. For example, the initial question, "How are you feeling right now?" is answered with, "I'm a little tired."

[0683] 2. Entering daily health data

[0684] Users input their daily health data into the device, for example, by measuring their body temperature with a thermometer, and the data is then sent to the server.

[0685] 3. Notification and follow-up

[0686] When it's time to take their medication, the user will receive a notification from the hologram on the device saying, "Take your medicine now," and will follow the instructions to take their medicine. They will also receive notifications the day before or just before scheduled medical checkups or appointments.

[0687] Specific examples

[0688] Prompt Sentence Examples

[0689] Initial question: "How are you feeling right now?"

[0690] Voice response: "I'm a little tired"

[0691] Next question: "Do you exercise?"

[0692] Voice response: "I haven't been exercising much lately."

[0693] As described above, this system utilizes voice recognition and generative AI models to dynamically grasp the user's health status and provide appropriate questions and follow-ups, enabling comprehensive health management.

[0694] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0695] Step 1:

[0696] User registration and initial data settings

[0697] Users enter basic information such as their name, age, and medical history into a registration form on their device. The device then sends this information to the server, which then stores the received information in a database and creates a healthcare profile for each user. Specifically, the server generates a user ID and associates the entered data with that ID and stores it.

[0698] Input: User's basic information (name, age, medical history)

[0699] Output: User ID, healthcare profile

[0700] Step 2:

[0701] Beginning of medical interview

[0702] The user presses the "Start medical interview" button on the device. The device retrieves a list of initial questions from the server and activates a 3D hologram to display the initial questions. The user answers the questions verbally. The device receives the user's voice responses through a microphone and performs voice recognition in real time. The voice recognition results are sent to the server. Specifically, the voice data is converted into text data, and that text data is sent to the server.

[0703] Input: Press the start button and answer by voice

[0704] Output: Initial question list, speech-recognized text data

[0705] Step 3:

[0706] Dynamic interview generation

[0707] The server receives the speech recognition results and inputs them into a generative AI model. The AI ​​model (e.g., BERT or GPT-3) generates the next question based on the speech recognition results and the prompt. The server then sends the generated question to the device. The device displays the next question as a 3D hologram and presents it to the user. This allows the server to dynamically generate and display appropriate questions based on the user's specific symptoms.

[0708] Input: Speech recognition result (text), prompt

[0709] Output: Next question (text)

[0710] Step 4:

[0711] Voice Recognition and Feedback

[0712] The device receives the user's new voice response and performs voice recognition again. It sends the voice recognition result to the server and waits for the next appropriate question. This enables continuous dynamic interviews. Specifically, the device converts voice into text in real time and sends the text data to the server.

[0713] Input: Voice response

[0714] Output: Recognized text data

[0715] Step 5:

[0716] Data recording and sharing

[0717] The server records all medical interview data and voice recognition results. The recorded data is shared with medical professionals as needed. Medical professionals can use this data as a reference for diagnosis and treatment. Specifically, the server stores the data in a database and makes some or all of the data accessible to medical professionals.

[0718] Input: Voice recognition results, medical interview data

[0719] Output: Recorded interview data, data shared with medical professionals

[0720] Step 6:

[0721] Healthcare data management and notification

[0722] Users enter their daily health data (such as temperature, blood pressure, and weight) into the device. The device then sends this data to a server, which updates the user's health profile. The device also uses 3D holograms to notify users when it's time to take medication or when medical appointments are approaching. For example, a notification like "Take your medicine now" may appear on the hologram.

[0723] Input: Daily health data (temperature, blood pressure, weight, etc.), schedule data

[0724] Output: Updated health profile, notification message

[0725] Step 7:

[0726] Follow-up after consultation

[0727] The server generates a schedule for follow-ups after the consultation and sends it to the device. The user manages medication and health checkup schedules according to notifications from the device. The server sends follow-up notifications in a timely manner based on the user's healthcare data and schedule. Specifically, the server calculates medication schedules and health checkup schedules, and sends them to the device for display.

[0728] Input: Follow-up schedule, healthcare data

[0729] Output: Notification messages, management schedules

[0730] (Application example 1)

[0731] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0732] Conventional health management systems lack the ability to dynamically generate questions in response to user inquiries or real-time voice recognition and feedback using 3D holograms, making it difficult to provide comprehensive and personalized support for users' health management. They also lack a system for sharing generated data with medical professionals for appropriate follow-up. Furthermore, they are unable to provide individually customized health advice based on each user's health information, resulting in a lack of accuracy and usefulness of the health advice.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0734] In this invention, the server includes means for performing real-time speech recognition, means for dynamically generating questions based on the recognized speech data, means for displaying the generated questions as 3D holograms and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and a history of the questions, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for generating and displaying individually customized health advice based on each user's health information, thereby enabling accurate and comprehensive management of the user's health condition in real time and providing individually customized health advice.

[0735] "Means for real-time speech recognition" refers to technology that can instantly convert a user's speech into digital data and analyze its content.

[0736] "Means for dynamically generating questions based on recognized voice data" refers to a technology in which an AI model automatically creates appropriate questions based on the user's current state, based on analyzed voice data.

[0737] "Means of displaying and presenting to users as 3D holograms" refers to a technology that visually provides users with content generated using three-dimensional video technology.

[0738] "Means for receiving and re-recognizing the user's voice response" refers to a technology that analyzes the user's voice response again using voice recognition technology and reflects the results in the system.

[0739] The "means for recording the recognition results and the history of the questions" is a technology for saving the speech recognition results and the history of the questions generated as data.

[0740] "Means for sharing recorded data with medical professionals" refers to technology that securely and accurately transfers stored voice recognition results and question history data to medical professionals.

[0741] "Means for post-medical follow-up based on health data" refers to technology that analyzes a user's daily health data and, based on that data, provides appropriate guidance and advice after a medical visit.

[0742] "A means for generating and displaying individually customized health advice based on each user's health information" refers to a technology that automatically generates specific health advice based on each user's individual health condition and history, and presents it using 3D holograms, etc.

[0743] This invention is a system that performs real-time voice recognition, dynamically generates questions based on the data, and presents them to the user using a 3D hologram. The system also receives the user's answers via voice and records the recognition results and question history. It also shares the recognition results and recorded data with medical professionals, who use the health data to provide follow-up after consultations. It also includes a means for generating and displaying individually customized health advice based on each user's health information.

[0744] Server Operation

[0745] The server is responsible for user registration and initial data configuration. It receives the user's basic information (name, age, medical history) and stores it in a database. It is also equipped with an AI model that receives voice recognition results and dynamically generates the next question. The generated questions and voice recognition results are recorded in a database and shared with medical professionals along with health data. After the consultation, a schedule for medication management and appointment management is generated and sent to the user's device.

[0746] Device behavior

[0747] The device uses 3D holograms to display dynamically generated questions to the user. It recognizes the user's answers in real time and sends the results to the server. The user's daily health data (body temperature, blood pressure, weight, etc.) is also entered into the device and synchronized with the server. The device notifies the user via 3D holograms when it is time to take medication or when medical appointments are approaching.

[0748] User Actions

[0749] The user presses the button to start the medical interview, answers the questions posed by the hologram by voice, and enters their daily health data into the device. For example, to the question, "How are you feeling right now?", the user can reply, "I'm a little tired." The voice response is recognized in real time, and the next question is presented through the hologram. The user also receives notifications and follows instructions when it's time to take medication or when medical appointments are approaching.

[0750] Specific Examples

[0751] If User A has a history of high blood pressure, the following process is executed. User A enters their name, age, and history of high blood pressure and sends them to the server. The server uses this information to create a healthcare profile for User A. When User A begins the medical interview, a 3D hologram on the device displays the question, "How is your current health?" User A replies, "My blood pressure has been high recently." The server inputs the voice recognition results into an AI model, which dynamically generates the next question, "Are you exercising?" This question is displayed via the hologram, and User A replies, "I haven't been exercising much recently."

[0752] Prompt Sentence Examples

[0753] User: "How are you feeling these days?"

[0754] Answer: "I have high blood pressure and get tired easily."

[0755] Question to prompt: "Do you exercise?"

[0756] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[0757] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0758] Step 1: User Registration

[0759] Users enter basic information such as their name, age, and medical history into the device and press the registration button. The device then sends this data to the server, which stores the received information in a database and creates a healthcare profile for each user.

[0760] Input: Name, age, medical history

[0761] Output: Generate a healthcare profile

[0762] Specific behavior: Inserts information into the database and creates an initial profile

[0763] Step 2: Begin the interview

[0764] The user presses the button to start the medical interview, and the device uses a 3D hologram to display the first question: "How are you feeling right now?" The user then answers this question verbally. The device recognizes the user's voice in real time, converts it into digital data, and sends it to the server.

[0765] Input: Voice input "I'm a little tired"

[0766] Output: Speech recognition results as digital data

[0767] Specific operation: Converts speech into text using a speech recognition engine and sends it to the server

[0768] Step 3: Dynamic Questionnaire Generation

[0769] The server analyzes the received voice recognition data and inputs it into the AI ​​model. The AI ​​model dynamically generates the next question: "Do you exercise?" The generated question is sent to the device.

[0770] Input: Speech recognition result: "I'm a little tired"

[0771] Output: Next question: "Do you exercise?"

[0772] How it works: Uses AI models to dynamically generate the next question

[0773] Step 4: Ask the next question

[0774] The device then displays the next question as a 3D hologram and presents it to the user. The user answers, "I haven't been exercising much lately." The device again recognizes the voice in real time and sends the results to the server.

[0775] Input: Voice input "I haven't been exercising much lately"

[0776] Output: The following speech recognition results stored in the database:

[0777] Specific operation: Displaying questions using 3D holograms, recognizing voice and sending it to the server

[0778] Step 5: Record data and share with medical professionals

[0779] The server records the voice recognition results and question history in a database, which is shared with medical professionals periodically or upon request.

[0780] Input: Speech recognition result: "I haven't been exercising much lately."

[0781] Output: Data shared with medical professionals

[0782] Specific actions: Recorded in a database and shared with medical professionals through a dedicated interface

[0783] Step 6: Enter and sync your daily health data

[0784] Users input their daily health data, such as body temperature, blood pressure, and weight, into the device, which then sends this data to the server, which stores it in a database.

[0785] Input: Health data such as body temperature, blood pressure, and weight

[0786] Output: Updated healthcare profile

[0787] Specific operation: Enter health data, send it to the server and save it

[0788] Step 7: Follow-up after the visit

[0789] The server generates a schedule for medication management and schedule management as follow-up after the consultation and sends it to the terminal. The terminal checks this schedule and notifies the user at the specified time using a 3D hologram.

[0790] Input: Health data, feedback from medical professionals

[0791] Output: Follow-up schedule and notifications

[0792] Specific actions: Follow-up schedule generation and notifications using 3D holograms

[0793] Step 8: Generate personalized health advice

[0794] The server uses AI models to generate personalized health advice based on each user's health information, which is then sent to the device and displayed to the user as a 3D hologram.

[0795] Input: User health information and daily data

[0796] Output: Customized health advice

[0797] How it works: Uses AI models to generate health advice and displays it in a 3D hologram

[0798] This series of steps enables accurate and comprehensive management of the user's health status in real time and provides individually customized health advice.

[0799] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0800] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system integrates real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up to provide accurate and comprehensive health management.

[0801] Overview of program processing

[0802] server

[0803] 1. User registration and initial data settings

[0804] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[0805] Create and manage your own healthcare profile.

[0806] 2. Dynamic Questionnaire Generation

[0807] When a user initiates a consultation, the server generates a predefined initial list of questions.

[0808] The AI ​​model receives the speech recognition results and the emotion data recognized by the emotion engine, and dynamically generates the next question based on the results.

[0809] This process allows for appropriate interviews to be conducted based on the user's symptoms and emotions.

[0810] 3. Data sharing and post-examination support

[0811] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[0812] Sharing emotional data will also help medical professionals make more accurate diagnoses.

[0813] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[0814] Terminal

[0815] 1. 3D hologram generation

[0816] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[0817] The hologram presents the user with a series of generated questions.

[0818] 2. Speech and Emotion Recognition

[0819] It receives the user's voice response and performs real-time voice recognition.

[0820] The emotion engine analyzes the user's voice and facial expressions when answering and generates emotional data.

[0821] 3. Feedback

[0822] The user's voice recognition results and emotion data are sent to the server, which receives the next question and displays it on the hologram.

[0823] If the emotional data indicates "anxiety" or "tension," the hologram provides feedback to help the user relax.

[0824] 4. Healthcare Data Management and Notification

[0825] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[0826] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[0827] User

[0828] 1. Starting and answering the medical interview

[0829] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[0830] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[0831] 2. Entering daily health data

[0832] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[0833] The data is sent to the server and the user's profile is updated.

[0834] 3. Collecting emotional data and receiving feedback

[0835] Receives feedback based on emotional data from the hologram and engages in adaptive dialogue.

[0836] For example, if a user responds, "I'm feeling a little anxious right now," the hologram will offer encouraging words such as, "It sounds like you're feeling anxious. I suggest you take some time off."

[0837] Specific examples

[0838] Assuming that user B is experiencing mental stress, the following specific process will be carried out:

[0839] 1. Initial data settings

[0840] User B uses the terminal to enter his / her name, age, and medical history of mental stress, and sends the information to the server.

[0841] The server receives this information and creates a healthcare profile for User B.

[0842] 2. Start of interview

[0843] When User B begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[0844] User B responds verbally, "I've been feeling stressed lately."

[0845] 3. Dynamic Questionnaire Generation and Emotion Recognition

[0846] The server inputs the voice recognition results and the "anxiety" emotion generated by the emotion engine into the AI ​​model, and dynamically generates the next question: "In what situations do you feel stressed?"

[0847] This question is displayed on the device via a hologram, and User B answers, "It's mainly pressure at work."

[0848] 4. Data recording and sharing

[0849] Each response and emotional data is sent to a server in real time and recorded.

[0850] The collected data will be shared with your family doctor and used as diagnostic material.

[0851] 5. Post-examination support

[0852] A stress management schedule for user B is generated on the server and sent to the terminal.

[0853] When it's time to take medication or engage in relaxation activities, a hologram on the device will notify you, saying, "Please perform your relaxation exercises now."

[0854] In this way, the present invention comprehensively manages the user's health status and emotional data, realizing accurate medical interviews and continuous follow-up.

[0855] The processing flow will be explained below.

[0856] Step 1:

[0857] The user presses the start button on the terminal, which starts the interview process.

[0858] Step 2:

[0859] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[0860] Step 3:

[0861] The user answers the question verbally, for example, "I'm a little tired."

[0862] Step 4:

[0863] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[0864] Step 5:

[0865] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[0866] Step 6:

[0867] The device receives the next question from the server and displays it via a 3D hologram.

[0868] Step 7:

[0869] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[0870] Step 8:

[0871] The device sends the user's voice and facial expressions to the emotion engine, which analyzes the user's emotions and generates emotion data indicating, for example, "anxiety" or "tension."

[0872] Step 9:

[0873] The device performs voice recognition again and transmits the text data and emotion data to the server.

[0874] Step 10:

[0875] The server repeats the process of generating further questions based on new recognition results and emotion data, continuing the interview process until all necessary information is collected. For example, if the emotion data indicates "anxiety," it generates additional questions that take emotion into consideration, such as "Have you been sleeping well recently?"

[0876] Step 11:

[0877] The server stores all collected data in a database and updates the user's healthcare profile.

[0878] Step 12:

[0879] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[0880] Step 13:

[0881] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[0882] Step 14:

[0883] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[0884] Step 15:

[0885] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[0886] Step 16:

[0887] The device will notify the user via a 3D hologram when it is time to take medication, when a medical checkup is scheduled, etc. For example, if the emotional data indicates "anxiety" or "stress," a reminder such as "We recommend you perform a relaxation exercise now" will be displayed.

[0888] Example 2

[0889] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0890] Conventional health management systems lack the functionality to analyze users' voice and emotional data in real time and provide appropriate medical feedback. Furthermore, they lack a system that can centrally manage follow-up after medical examinations and daily health data management. As a result, it is difficult for users to accurately and comprehensively understand their own health status.

[0891] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0892] In this invention, the server includes means for performing real-time voice recognition, means for dynamically generating questions based on the recognized voice data and emotion data, means for displaying the generated questions as a 3D image and presenting them to the user, means for receiving the user's voice responses and re-recognizing them, means for recording the recognition results and emotion data, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for providing feedback according to the user's emotion data. This enables comprehensive health management, including analyzing the user's health condition in real time, providing appropriate feedback based on the emotion data, and post-examination follow-ups.

[0893] "Means for real-time speech recognition" refers to devices or software that have the ability to instantly convert a user's voice into text data and process it in real time.

[0894] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a system that has the function of automatically generating the next question based on the situation using the results of voice recognition and emotional analysis.

[0895] "Means for displaying generated questions in 3D images and presenting them to the user" refers to a system that has the function of visually presenting questions generated for dialogue with the user using a 3D image display device.

[0896] "Means for receiving and re-recognizing the user's voice response" refers to devices or software that have the function of converting the user's voice response into text data using voice recognition technology and analyzing it.

[0897] "Means for recording the recognition results and emotion data" refers to a system that has the function of storing the voice recognition results and emotion analysis results in a recording medium such as a database.

[0898] "Means for sharing the recognition results and recorded data with medical professionals" refers to a system that has the function of securely transferring and sharing speech recognition results and recorded data with medical professionals.

[0899] "Means for post-visit follow-up based on health data" refers to a system that has the functionality to manage medication, guidance, and schedules after a visit based on the user's health data.

[0900] "Means for providing feedback based on the user's emotional data" refers to a system that has the function of analyzing the user's emotional data and providing appropriate advice or words of encouragement based on the results.

[0901] "Means for integrated management of health data" refers to a system that has the function of centrally collecting, analyzing, and managing users' daily health data.

[0902] "Medication management" refers to a system that manages a user's medication schedule and notifies them to take their medication appropriately.

[0903] "Instruction management" refers to a system that has the function of managing the content of instruction provided by medical professionals and providing appropriate instructions and advice to users.

[0904] "Schedule management" refers to a system that has the function of managing a user's medical-related schedules and notifying them at the appropriate time.

[0905] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up.

[0906] Server Roles

[0907] The server receives the user's basic information and stores it in a database. This information includes name, age, and medical history. For example, data such as "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure" is entered. The server uses this information to create a healthcare profile for each user.

[0908] Furthermore, when the user begins the medical interview, the server generates a predefined list of initial questions. For example, it generates a question such as, "How are you feeling right now?". The AI ​​model then dynamically generates the next question based on the speech recognition results and emotion data obtained by the emotion engine. This results in a dynamic question such as, "Since when have you been feeling tired?"

[0909] The server shares the collected medical interview data and daily health care data with the patient's primary care physician. This includes voice and emotional data. Schedules for medication management, instruction management, and schedule management are then generated and sent to the device. For example, instructions such as "Take your medicine at 8:00 every morning" are generated.

[0910] Device Role

[0911] When the user begins the medical interview, the device activates a 3D hologram and displays the first question. The hologram speaks, "Hello, how are you feeling right now?" The device recognizes the user's voice response in real time, and the emotion engine analyzes the emotion at the time of the response. For example, if the user responds, "I've been feeling stressed lately," the voice is converted into text and emotion data is generated.

[0912] The device then sends the user's voice recognition results and emotional data to the server, which then asks the next question and displays it on the hologram. For example, if the emotional data indicates anxiety or tension, the hologram might suggest, "Would you like to know how to relax?"

[0913] The device inputs the user's daily health data, such as temperature, blood pressure, and weight, and synchronizes it with the server. In addition, when it is time to take medication or when medical appointments are approaching, the device notifies the user via a hologram. For example, it issues an alert saying, "It's time to take your medicine."

[0914] User Roles

[0915] The user presses a button on the device to start the medical interview and answers the questions from the hologram by voice. For example, the hologram might ask, "How are you feeling right now?" and the user might reply, "I'm a little tired." Next, the user enters their daily health data (e.g., body temperature, blood pressure, weight) into the device, which then sends that data to the server to update their profile. For example, they might enter, "Today's body temperature is 36.5 degrees."

[0916] The hologram responds by providing feedback based on the user's emotional data, allowing for adaptive interactions. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will respond with encouragement, "It seems you're feeling anxious. Perhaps it would be good for you to take a short break."

[0917] Specific examples

[0918] If user B is experiencing mental stress, the following specific process will occur:

[0919] User B uses the device to enter their name, age, and medical history of mental stress, and sends this to the server. The server creates a healthcare profile. When the medical interview begins, a 3D hologram on the device displays the question, "How is your current health condition?" User B responds verbally, "I've been feeling very stressed lately." Based on the voice recognition results and the "anxiety" emotion generated by the emotion engine, the server dynamically generates the next question: "In what situations do you feel stressed?"

[0920] Each response and emotional data are sent to the server in real time and recorded. This data is then provided to the user's doctor as diagnostic information. User B's stress management schedule is generated by the server and sent to the device. For example, when it is time to take medication or engage in relaxation activities, a hologram on the device will notify the user, saying, "Please perform relaxation exercises now."

[0921] Example prompts for generative AI models

[0922] 1. "If a user has been stressed recently, what follow-up questions should be generated?"

[0923] 2. "If the emotion engine determines that the user is anxious, please suggest the appropriate feedback to provide next."

[0924] 3. "Generate follow-up questions based on the user's daily health data (e.g., body temperature, blood pressure)."

[0925] As described above, this system comprehensively manages the user's health status and emotional data, enabling accurate medical interviews and continuous follow-up.

[0926] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0927] Step 1:

[0928] User registration and initial data settings

[0929] The server receives the user's basic information (name, age, medical history) as input. It then stores this information in a database and creates a healthcare profile for each user. This profile serves as the foundation for the user to begin their medical interview. For example, if a user enters "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure," this data will be registered in the database.

[0930] Step 2:

[0931] Start of interview and generation of initial questions

[0932] When the user presses the start interview button, the terminal activates a 3D hologram. The hologram displays the first question to the user, "How is your current health condition?", and asks the question aloud. The input here is "pressing the start interview button," and the output is "displaying the initial question."

[0933] Step 3:

[0934] Speech and Emotion Recognition

[0935] The device receives the user's voice response as input and performs real-time voice recognition. The emotion engine then analyzes the user's voice and facial expressions to generate emotion data. At this stage, the input is the "voice response," and the output is the "recognized text data and emotion data." For example, if the user responds "I'm a little tired," the voice is converted into text data saying "I'm a little tired," and the emotion data generated is "anxiety."

[0936] Step 4:

[0937] Dynamic generation of next question

[0938] The server receives the speech recognition results and emotion data sent from the device as input. The AI ​​model then analyzes these data and dynamically generates the next question. At this stage, the input is the "speech recognition results and emotion data," and the output is the "next question." For example, the next question generated might be, "Since when have you been feeling tired?"

[0939] Step 5:

[0940] View next question and receive answer

[0941] The device receives the next question from the server, displays it on a 3D hologram, and presents it to the user. The user responds vocally, and the device again performs voice and emotion recognition. The input / output cycle is repeated. The input is "generation of the next question," and the output is "display of the next question and a vocal response."

[0942] Step 6:

[0943] Data recording and sharing

[0944] The server records the collected speech recognition results, emotion data, and medical history data in a database. Furthermore, a secure data transmission protocol is used to share important data with medical professionals. The input is "collected data," and the output is "recording in the database and sharing with medical professionals."

[0945] Step 7:

[0946] Follow-up schedule generation

[0947] The server generates a schedule for post-consultation medication management, instruction management, and schedule management based on the user's health data and medical interview results. The input is "health data and medical interview results," and the output is "follow-up schedule." For example, a medication instruction such as "Take your medicine every morning at 8 o'clock" is generated.

[0948] Step 8:

[0949] Providing Feedback

[0950] The device notifies the user of the generated follow-up schedule via a hologram. It also provides feedback based on the user's emotional data. The input is "follow-up schedule and emotional data," and the output is "notification and feedback." For example, the hologram may notify the user, "Please perform relaxation exercises now."

[0951] In this way, it is possible to comprehensively manage the user's health status and emotional data, and realize accurate medical interviews and continuous follow-up.

[0952] (Application example 2)

[0953] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0954] In modern society, there is a demand for efficient health management and related follow-up for users. However, conventional health management systems do not adequately provide comprehensive support that takes into account users' emotional states or personalized product recommendations. Therefore, a new system is needed to improve the quality of users' health management.

[0955] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0956] In this invention, the server includes means for recognizing the user's voice in real time, means for dynamically generating questions based on the recognized voice data and emotional data, means for displaying the generated questions in 3D images and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and question history, means for sharing the recognition results and recorded data and emotional data, means for conducting post-examination follow-up based on health data, and means for recommending products based on preferences. This makes it possible to conduct medical interviews and follow-ups that reflect the user's emotional state, and further improve the quality of the user's health management through personalized product recommendations.

[0957] "Means for real-time speech recognition" refers to a function that instantly converts voice data provided by a user into text data.

[0958] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a function that analyzes the voice recognition results and the user's emotional state, and automatically generates appropriate questions based on these.

[0959] "Means for displaying the generated question in a 3D image and presenting it to the user" refers to a function that uses hologram technology or other 3D image display technology to visually show the generated question to the user.

[0960] The "means for receiving and re-recognizing the user's voice response" is a function for re-recognizing the user's verbal response and converting it into data.

[0961] The "means for recording the recognition results and question history" is a function for saving the speech recognition results and the history of questions and answers generated at that time in a database or the like.

[0962] "Means for sharing recognition results, recorded data, and emotional data" refers to the function of sharing recognition results, recorded question and answer history, and emotional data with other systems and experts.

[0963] "Means for post-consultation follow-up based on health data" is a function that utilizes health data to take appropriate measures to manage and support the user's health after a consultation.

[0964] "Preference-based product recommendation means" is a function that automatically suggests optimal products based on the user's preferences and health condition.

[0965] MODE FOR CARRYING OUT THE INVENTION

[0966] This is a health management system for online shopping sites that combines 3D holograms, voice recognition, AI models, an emotion engine, data sharing, and follow-up functions. The system grasps the user's health and emotional state in real time, and based on that, conducts appropriate medical interviews, follow-ups, and recommends related products.

[0967] System Overview

[0968] The system consists of three main components:

[0969] 1. Server

[0970] 2. Device (smartphone, etc.)

[0971] 3. Users

[0972] server

[0973] The server's main responsibilities are:

[0974] User registration and initial data setup: The server receives basic information provided by the user (such as name, age, and health status) and stores it in a database, which then creates a health profile for each user.

[0975] Dynamic Question Generation: Based on the speech recognition results and emotion data, the next question is dynamically generated using an AI model. In this process, the most appropriate question is presented to the user.

[0976] Data sharing: Collected voice data, emotional data, health data, etc. will be shared with the user's permission, allowing for more accurate support to be provided to the user.

[0977] Post-consultation follow-up and product recommendations: Based on the user's health and preference data, necessary follow-up and related product recommendations are provided.

[0978] Terminal

[0979] The device (e.g., smartphone) primarily performs the following functions:

[0980] 3D hologram generation and display: At the start of the interview, a 3D hologram is activated and presents questions to the user. The hologram displays the generated questions one after another, ensuring a smooth dialogue.

[0981] Speech and emotion recognition: The device recognizes the user's voice in real time and generates emotion data using an emotion engine. This data is sent to the server and used to generate the next question.

[0982] Feedback display: The voice recognition results and emotional data are sent to the server, and the next question is received from the server and displayed on the hologram. If the emotional data indicates "anxiety" or "tension," the device has the function of displaying feedback to relax the user.

[0983] User

[0984] The user performs the following actions:

[0985] Starting and answering the medical interview: Press the start button on the device and answer the questions from the hologram by voice. For example, in response to the initial question, "How are you feeling lately?", you can answer, "I'm a little tired."

[0986] Daily health data input: Enter your daily health data (body temperature, blood pressure, weight, etc.) into your device and synchronize it with the server, which will update your health profile in real time.

[0987] Collecting emotional data and receiving feedback: The hologram receives feedback based on emotional data collected during the interview and engages in adaptive dialogue. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will provide encouraging words such as, "It seems you're feeling anxious. I recommend you take a break."

[0988] Hardware and software used

[0989] Hardware: Smartphone (iOS or Android device)

[0990] software:

[0991] Speech Recognition: Converts speech to text using the speech_recognition library.

[0992] Emotion Recognition: Analyze emotions from speech and text using third-party emotion recognition libraries.

[0993] Generative AI Models: Use a generative AI model library to generate the next question.

[0994] Specific examples

[0995] For example, if a user enters "I've been feeling more stressed lately" in the default settings, the system will act as follows:

[0996] 1. The user answers verbally, "I've been busy at work lately and it's stressful."

[0997] 2. Speech to text: "Work has been busy and stressful lately."

[0998] 3. Sentiment analysis: Identify "stress."

[0999] 4. The AI ​​model generates the following question: "What situations make you feel particularly stressed?"

[1000] Prompt Sentence Examples

[1001] "Based on the analysis of the user's voice content and emotions, please generate the following appropriate questions. Voice content: 'Work has been busy lately and I'm stressed.' Emotion: 'Stressed.' Next question:"

[1002] This allows the medical interview process to proceed smoothly and allows for specific follow-up and product recommendations tailored to the user's health condition.

[1003] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1004] Step 1:

[1005] User Registration

[1006] 1. The server receives basic information entered by the user (name, age, health status) and stores it in a database.

[1007] Input: Basic information such as name, age, and health status.

[1008] Data processing: The entered information is registered in a database according to the format.

[1009] Output: User profile created.

[1010] Step 2:

[1011] Interview begins

[1012] 1. The user launches the app on their device and presses the button to start the medical interview.

[1013] Input: User operation (pressing the button to start the medical interview).

[1014] Data processing: Set the interview start flag.

[1015] Output: Start of the interview process.

[1016] 2. The device will activate a 3D hologram and display the first question.

[1017] Input: Interview start flag.

[1018] Data processing: Get the first question.

[1019] Output: Holographic question display.

[1020] Step 3:

[1021] Speech and Emotion Recognition

[1022] 1. The user answers the first question by voice.

[1023] Input: Audio data.

[1024] Data processing: Acquisition of audio data.

[1025] Output: Save audio data.

[1026] 2. The device performs voice recognition and converts it into text data.

[1027] Input: Audio data.

[1028] Data processing: Text conversion using speech recognition (using the speech_recognition library).

[1029] Output: Text data.

[1030] 3. The emotion engine analyzes the voice and text data and generates emotion data.

[1031] Input: Audio and text data.

[1032] Data processing: Sentiment analysis (using emotion recognition libraries).

[1033] Output: Emotion data.

[1034] Step 4:

[1035] Dynamic interview generation

[1036] 1. The server receives the speech recognition results and emotion data and inputs a prompt sentence to the generative AI model to generate the next question.

[1037] Input: Speech recognition results, emotion data.

[1038] Data processing: Prompt sentence generation.

[1039] Output: The prompt statement.

[1040] 2. The server uses a generative AI model to dynamically generate the next question.

[1041] Input: prompt statement.

[1042] Data processing: The AI ​​model generates the next question.

[1043] Output: Next question.

[1044] Step 5:

[1045] Presenting the next question

[1046] 1. The server sends the following question to the terminal:

[1047] Input: Next question.

[1048] Data processing: Sending query data.

[1049] Output: Sends the next question to the terminal.

[1050] 2. The device uses a hologram to display the next question.

[1051] Input: Next question.

[1052] Data processing: Hologram generation and display.

[1053] Output: Hologram of the question.

[1054] Step 6:

[1055] Data recording and sharing

[1056] 1. The server records each response and emotion data in real time.

[1057] Input: Speech recognition results, emotion data, question history.

[1058] Data processing: Saving to database.

[1059] Output: Recorded data.

[1060] 2. The server shares the recorded data.

[1061] Input: Recorded data.

[1062] Data processing: Generating data for sharing.

[1063] Output: Sending shared data.

[1064] Step 7:

[1065] Follow-up and product recommendations

[1066] 1. The server generates a schedule for follow-up visits and preference-based product recommendations.

[1067] Input: Health data, preference data.

[1068] Data processing: Applying schedule generation and product recommendation algorithms.

[1069] Output: Follow-up schedule, product recommendation list.

[1070] 2. The device displays follow-up notifications and product recommendations to the user.

[1071] Input: Follow-up schedule, product recommendation list.

[1072] Data processing: generation of notification and display data.

[1073] Output: Follow-up notification, product recommendation display.

[1074] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1075] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1076] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1077] [Third embodiment]

[1078] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1079] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1080] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1081] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1082] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1083] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1084] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1085] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1086] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1087] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1088] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1089] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1090] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, data sharing, and post-consultation follow-up.

[1091] Overview of program processing

[1092] server

[1093] 1. User registration and initial data settings

[1094] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[1095] Create and manage your own healthcare profile.

[1096] 2. Dynamic Questionnaire Generation

[1097] When a user initiates a consultation, the server generates a predefined initial list of questions.

[1098] The AI ​​model receives the speech recognition results and dynamically generates the next question based on those results.

[1099] This process allows for appropriate medical interviews to be conducted based on the user's symptoms.

[1100] 3. Data sharing and post-examination support

[1101] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[1102] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[1103] Terminal

[1104] 1. 3D hologram generation

[1105] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[1106] The hologram presents the user with a series of generated questions.

[1107] 2. Voice Recognition and Feedback

[1108] It receives the user's voice response and performs real-time voice recognition.

[1109] The voice recognition results are sent to the server, which receives the next question and displays it on the hologram.

[1110] 3. Healthcare data management and notification

[1111] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[1112] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[1113] User

[1114] 1. Starting and answering the medical interview

[1115] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[1116] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[1117] 2. Entering daily health data

[1118] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[1119] The data is sent to the server and the user's profile is updated.

[1120] 3. Notification and follow-up

[1121] Receive notifications from the hologram when it's time to take your medication and take it as instructed.

[1122] If you have a scheduled health check or medical examination, you will be notified the day before or just before.

[1123] Specific examples

[1124] If user A has a history of high blood pressure, the following specific process is executed:

[1125] 1. Initial data settings

[1126] User A uses a terminal to enter his / her name, age, and medical history of high blood pressure, and sends the information to the server.

[1127] The server receives this information and creates a healthcare profile for User A.

[1128] 2. Start of interview

[1129] When user A begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[1130] User A answers by voice, "My blood pressure has been high recently."

[1131] 3. Dynamic Questionnaire Generation

[1132] The server inputs the speech recognition results into the AI ​​model and dynamically generates the next question: "Do you exercise?"

[1133] This question is displayed on the device via a hologram, and User A answers, "I haven't been exercising much lately."

[1134] 4. Data recording and sharing

[1135] Each response is sent in real time to a server and recorded.

[1136] This data will be shared with your family doctor and used as diagnostic material.

[1137] 5. Post-examination support

[1138] User A's medication schedule is generated on the server and sent to the terminal.

[1139] When it's time to take your medication, a hologram on the device will notify you, "Take your medication now."

[1140] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[1141] The processing flow will be explained below.

[1142] Step 1:

[1143] The user presses the start button on the terminal, which starts the interview process.

[1144] Step 2:

[1145] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[1146] Step 3:

[1147] The user answers the question verbally, for example, "I'm a little tired."

[1148] Step 4:

[1149] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[1150] Step 5:

[1151] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[1152] Step 6:

[1153] The device receives the next question from the server and displays it via a 3D hologram.

[1154] Step 7:

[1155] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[1156] Step 8:

[1157] The device performs voice recognition again, converts it into text data, and sends it to the server.

[1158] Step 9:

[1159] The server repeats the process of generating more questions based on the new recognition results, and continues interrogating until all the necessary information is collected.

[1160] Step 10:

[1161] The server stores all collected data in a database and updates the user's healthcare profile.

[1162] Step 11:

[1163] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[1164] Step 12:

[1165] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[1166] Step 13:

[1167] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[1168] Step 14:

[1169] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[1170] Step 15:

[1171] When it's time to take your medication or when your health checkup is scheduled, the device will notify you via a 3D hologram, displaying a reminder such as "Take your medicine now."

[1172] Example 1

[1173] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1174] Conventional health management systems require a lot of effort for user data entry and follow-up, making it difficult to accurately interview patients and share data in real time. Furthermore, there are limitations in the accuracy of voice recognition and the generation of dynamic questions, making it difficult to provide appropriate responses based on the user's symptoms. Furthermore, follow-up after consultations is often insufficient, resulting in inadequate ongoing health management.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1176] In this invention, the server includes a means for receiving user registration and basic information and storing it in a database, a means for performing real-time speech recognition, and a means for dynamically generating questions using prompts generated based on the recognized speech data, thereby enabling accurate collection of user health data in real time, continuous follow-up, and comprehensive health management.

[1177] "User registration" is the process of entering basic user information into the system and storing it in the database.

[1178] "Speech recognition" is a technology that receives a user's voice and interprets it as digital data.

[1179] A "prompt" is the text for the next question generated based on an AI model.

[1180] "Dynamic question generation" is the process of generating the next question in real time based on the user's speech recognition results.

[1181] "3D hologram display" is a method of visually presenting questions and notifications to users using three-dimensional imaging technology.

[1182] "Recognition result" refers to text data obtained by speech recognition.

[1183] "Question History" is a record of all questions posed to a user and their answers.

[1184] "Health professionals" are people whose job it is to evaluate health data and make diagnoses and treatment decisions.

[1185] "Data sharing" is the process of sharing collected data with another user or system.

[1186] "Follow-up" refers to ongoing support activities to help users maintain their health after a medical examination.

[1187] "Healthcare data" refers to data that represents the user's daily health condition (body temperature, blood pressure, weight, etc.).

[1188] "Drug management" is the process of properly managing the timing and amount of medication a user takes.

[1189] "Insurance guidance" is the process of providing information and advice related to a user's health insurance.

[1190] "Appointment management" is the process of scheduling a user's health-related appointments (such as doctor's appointments and checkups) and providing appropriate notifications.

[1191] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, generative AI models, data sharing, and follow-up functions. The specific hardware and software usage and data processing procedures are described below.

[1192] Server configuration and processing

[1193] 1. User registration and initial data settings

[1194] The server receives basic user information (such as name, age, medical history, etc.) from the registration form and stores it in a database. Based on this data, a healthcare profile is generated for each user.

[1195] 2. Dynamic Questionnaire Generation

[1196] When the user starts the medical interview, the server sends an initial list of questions to the device. After receiving the user's voice responses, the server analyzes them using speech recognition software (e.g., Google Cloud Speech-to-Text API). Based on the analysis results, a generative AI model (e.g., BERT or GPT-3) generates the next questions.

[1197] 3. Data sharing and post-examination support

[1198] The collected medical interview data and daily health care data are shared with the patient's primary care physician, and a schedule for follow-up visits after the visit is generated and sent to the device.

[1199] Terminal configuration and handling

[1200] 1. 3D hologram generation

[1201] When the user begins the consultation, the device activates a hologram using 3D holographic technology. The first question is displayed as a hologram and presented to the user. An example of the hardware used is a display from the Looking Glass Factory.

[1202] 2. Voice Recognition and Feedback

[1203] The device receives the user's voice response through a microphone, analyzes it using speech recognition software, and sends the results to the server, which then receives the next dynamically generated question and displays it on the hologram.

[1204] 3. Healthcare data management and notification

[1205] The device allows users to input their daily health data (body temperature, blood pressure, weight, etc.) and synchronizes it with a server. It notifies users using holograms when it's time to take their medication or when medical appointments are approaching.

[1206] User operations

[1207] 1. Starting and answering the medical interview

[1208] The user presses a button on the device to start the medical interview and answers the questions posed by the hologram by voice. For example, the initial question, "How are you feeling right now?" is answered with, "I'm a little tired."

[1209] 2. Entering daily health data

[1210] Users input their daily health data into the device, for example, by measuring their body temperature with a thermometer, and the data is then sent to the server.

[1211] 3. Notification and follow-up

[1212] When it's time to take their medication, the user will receive a notification from the hologram on the device saying, "Take your medicine now," and will follow the instructions to take their medicine. They will also receive notifications the day before or just before scheduled medical checkups or appointments.

[1213] Specific examples

[1214] Prompt Sentence Examples

[1215] Initial question: "How are you feeling right now?"

[1216] Voice response: "I'm a little tired"

[1217] Next question: "Do you exercise?"

[1218] Voice response: "I haven't been exercising much lately."

[1219] As described above, this system utilizes voice recognition and generative AI models to dynamically grasp the user's health status and provide appropriate questions and follow-ups, enabling comprehensive health management.

[1220] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1221] Step 1:

[1222] User registration and initial data settings

[1223] Users enter basic information such as their name, age, and medical history into a registration form on their device. The device then sends this information to the server, which then stores the received information in a database and creates a healthcare profile for each user. Specifically, the server generates a user ID and associates the entered data with that ID and stores it.

[1224] Input: User's basic information (name, age, medical history)

[1225] Output: User ID, healthcare profile

[1226] Step 2:

[1227] Beginning of medical interview

[1228] The user presses the "Start medical interview" button on the device. The device retrieves a list of initial questions from the server and activates a 3D hologram to display the initial questions. The user answers the questions verbally. The device receives the user's voice responses through a microphone and performs voice recognition in real time. The voice recognition results are sent to the server. Specifically, the voice data is converted into text data, and that text data is sent to the server.

[1229] Input: Press the start button and answer by voice

[1230] Output: Initial question list, speech-recognized text data

[1231] Step 3:

[1232] Dynamic interview generation

[1233] The server receives the speech recognition results and inputs them into a generative AI model. The AI ​​model (e.g., BERT or GPT-3) generates the next question based on the speech recognition results and the prompt. The server then sends the generated question to the device. The device displays the next question as a 3D hologram and presents it to the user. This allows the server to dynamically generate and display appropriate questions based on the user's specific symptoms.

[1234] Input: Speech recognition result (text), prompt

[1235] Output: Next question (text)

[1236] Step 4:

[1237] Voice Recognition and Feedback

[1238] The device receives the user's new voice response and performs voice recognition again. It sends the voice recognition result to the server and waits for the next appropriate question. This enables continuous dynamic interviews. Specifically, the device converts voice into text in real time and sends the text data to the server.

[1239] Input: Voice response

[1240] Output: Recognized text data

[1241] Step 5:

[1242] Data recording and sharing

[1243] The server records all medical interview data and voice recognition results. The recorded data is shared with medical professionals as needed. Medical professionals can use this data as a reference for diagnosis and treatment. Specifically, the server stores the data in a database and makes some or all of the data accessible to medical professionals.

[1244] Input: Voice recognition results, medical interview data

[1245] Output: Recorded interview data, data shared with medical professionals

[1246] Step 6:

[1247] Healthcare data management and notification

[1248] Users enter their daily health data (such as temperature, blood pressure, and weight) into the device. The device then sends this data to a server, which updates the user's health profile. The device also uses 3D holograms to notify users when it's time to take medication or when medical appointments are approaching. For example, a notification like "Take your medicine now" may appear on the hologram.

[1249] Input: Daily health data (temperature, blood pressure, weight, etc.), schedule data

[1250] Output: Updated health profile, notification message

[1251] Step 7:

[1252] Follow-up after consultation

[1253] The server generates a schedule for follow-ups after the consultation and sends it to the device. The user manages medication and health checkup schedules according to notifications from the device. The server sends follow-up notifications in a timely manner based on the user's healthcare data and schedule. Specifically, the server calculates medication schedules and health checkup schedules, and sends them to the device for display.

[1254] Input: Follow-up schedule, healthcare data

[1255] Output: Notification messages, management schedules

[1256] (Application example 1)

[1257] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] Conventional health management systems lack the ability to dynamically generate questions in response to user inquiries or real-time voice recognition and feedback using 3D holograms, making it difficult to provide comprehensive and personalized support for users' health management. They also lack a system for sharing generated data with medical professionals for appropriate follow-up. Furthermore, they are unable to provide individually customized health advice based on each user's health information, resulting in a lack of accuracy and usefulness of the health advice.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1260] In this invention, the server includes means for performing real-time speech recognition, means for dynamically generating questions based on the recognized speech data, means for displaying the generated questions as 3D holograms and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and a history of the questions, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for generating and displaying individually customized health advice based on each user's health information, thereby enabling accurate and comprehensive management of the user's health condition in real time and providing individually customized health advice.

[1261] "Means for real-time speech recognition" refers to technology that can instantly convert a user's speech into digital data and analyze its content.

[1262] "Means for dynamically generating questions based on recognized voice data" refers to a technology in which an AI model automatically creates appropriate questions based on the user's current state, based on analyzed voice data.

[1263] "Means of displaying and presenting to users as 3D holograms" refers to a technology that visually provides users with content generated using three-dimensional video technology.

[1264] "Means for receiving and re-recognizing the user's voice response" refers to a technology that analyzes the user's voice response again using voice recognition technology and reflects the results in the system.

[1265] The "means for recording the recognition results and the history of the questions" is a technology for saving the speech recognition results and the history of the questions generated as data.

[1266] "Means for sharing recorded data with medical professionals" refers to technology that securely and accurately transfers stored voice recognition results and question history data to medical professionals.

[1267] "Means for post-medical follow-up based on health data" refers to technology that analyzes a user's daily health data and, based on that data, provides appropriate guidance and advice after a medical visit.

[1268] "A means for generating and displaying individually customized health advice based on each user's health information" refers to a technology that automatically generates specific health advice based on each user's individual health condition and history, and presents it using 3D holograms, etc.

[1269] This invention is a system that performs real-time voice recognition, dynamically generates questions based on the data, and presents them to the user using a 3D hologram. The system also receives the user's answers via voice and records the recognition results and question history. It also shares the recognition results and recorded data with medical professionals, who use the health data to provide follow-up after consultations. It also includes a means for generating and displaying individually customized health advice based on each user's health information.

[1270] Server Operation

[1271] The server is responsible for user registration and initial data configuration. It receives the user's basic information (name, age, medical history) and stores it in a database. It is also equipped with an AI model that receives voice recognition results and dynamically generates the next question. The generated questions and voice recognition results are recorded in a database and shared with medical professionals along with health data. After the consultation, a schedule for medication management and appointment management is generated and sent to the user's device.

[1272] Device behavior

[1273] The device uses 3D holograms to display dynamically generated questions to the user. It recognizes the user's answers in real time and sends the results to the server. The user's daily health data (body temperature, blood pressure, weight, etc.) is also entered into the device and synchronized with the server. The device notifies the user via 3D holograms when it is time to take medication or when medical appointments are approaching.

[1274] User Actions

[1275] The user presses the button to start the medical interview, answers the questions posed by the hologram by voice, and enters their daily health data into the device. For example, to the question, "How are you feeling right now?", the user can reply, "I'm a little tired." The voice response is recognized in real time, and the next question is presented through the hologram. The user also receives notifications and follows instructions when it's time to take medication or when medical appointments are approaching.

[1276] Specific Examples

[1277] If User A has a history of high blood pressure, the following process is executed. User A enters their name, age, and history of high blood pressure and sends them to the server. The server uses this information to create a healthcare profile for User A. When User A begins the medical interview, a 3D hologram on the device displays the question, "How is your current health?" User A replies, "My blood pressure has been high recently." The server inputs the voice recognition results into an AI model, which dynamically generates the next question, "Are you exercising?" This question is displayed via the hologram, and User A replies, "I haven't been exercising much recently."

[1278] Prompt Sentence Examples

[1279] User: "How are you feeling these days?"

[1280] Answer: "I have high blood pressure and get tired easily."

[1281] Question to prompt: "Do you exercise?"

[1282] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[1283] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1284] Step 1: User Registration

[1285] Users enter basic information such as their name, age, and medical history into the device and press the registration button. The device then sends this data to the server, which stores the received information in a database and creates a healthcare profile for each user.

[1286] Input: Name, age, medical history

[1287] Output: Generate a healthcare profile

[1288] Specific behavior: Inserts information into the database and creates an initial profile

[1289] Step 2: Begin the interview

[1290] The user presses the button to start the medical interview, and the device uses a 3D hologram to display the first question: "How are you feeling right now?" The user then answers this question verbally. The device recognizes the user's voice in real time, converts it into digital data, and sends it to the server.

[1291] Input: Voice input "I'm a little tired"

[1292] Output: Speech recognition results as digital data

[1293] Specific operation: Converts speech into text using a speech recognition engine and sends it to the server

[1294] Step 3: Dynamic Questionnaire Generation

[1295] The server analyzes the received voice recognition data and inputs it into the AI ​​model. The AI ​​model dynamically generates the next question: "Do you exercise?" The generated question is sent to the device.

[1296] Input: Speech recognition result: "I'm a little tired"

[1297] Output: Next question: "Do you exercise?"

[1298] How it works: Uses AI models to dynamically generate the next question

[1299] Step 4: Ask the next question

[1300] The device then displays the next question as a 3D hologram and presents it to the user. The user answers, "I haven't been exercising much lately." The device again recognizes the voice in real time and sends the results to the server.

[1301] Input: Voice input "I haven't been exercising much lately"

[1302] Output: The following speech recognition results stored in the database:

[1303] Specific operation: Displaying questions using 3D holograms, recognizing voice and sending it to the server

[1304] Step 5: Record data and share with medical professionals

[1305] The server records the voice recognition results and question history in a database, which is shared with medical professionals periodically or upon request.

[1306] Input: Speech recognition result: "I haven't been exercising much lately."

[1307] Output: Data shared with medical professionals

[1308] Specific actions: Recorded in a database and shared with medical professionals through a dedicated interface

[1309] Step 6: Enter and sync your daily health data

[1310] Users input their daily health data, such as body temperature, blood pressure, and weight, into the device, which then sends this data to the server, which stores it in a database.

[1311] Input: Health data such as body temperature, blood pressure, and weight

[1312] Output: Updated healthcare profile

[1313] Specific operation: Enter health data, send it to the server and save it

[1314] Step 7: Follow-up after the visit

[1315] The server generates a schedule for medication management and schedule management as follow-up after the consultation and sends it to the terminal. The terminal checks this schedule and notifies the user at the specified time using a 3D hologram.

[1316] Input: Health data, feedback from medical professionals

[1317] Output: Follow-up schedule and notifications

[1318] Specific actions: Follow-up schedule generation and notifications using 3D holograms

[1319] Step 8: Generate personalized health advice

[1320] The server uses AI models to generate personalized health advice based on each user's health information, which is then sent to the device and displayed to the user as a 3D hologram.

[1321] Input: User health information and daily data

[1322] Output: Customized health advice

[1323] How it works: Uses AI models to generate health advice and displays it in a 3D hologram

[1324] This series of steps enables accurate and comprehensive management of the user's health status in real time and provides individually customized health advice.

[1325] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1326] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system integrates real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up to provide accurate and comprehensive health management.

[1327] Overview of program processing

[1328] server

[1329] 1. User registration and initial data settings

[1330] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[1331] Create and manage your own healthcare profile.

[1332] 2. Dynamic Questionnaire Generation

[1333] When a user initiates a consultation, the server generates a predefined initial list of questions.

[1334] The AI ​​model receives the speech recognition results and the emotion data recognized by the emotion engine, and dynamically generates the next question based on the results.

[1335] This process allows for appropriate interviews to be conducted based on the user's symptoms and emotions.

[1336] 3. Data sharing and post-examination support

[1337] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[1338] Sharing emotional data will also help medical professionals make more accurate diagnoses.

[1339] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[1340] Terminal

[1341] 1. 3D hologram generation

[1342] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[1343] The hologram presents the user with a series of generated questions.

[1344] 2. Speech and Emotion Recognition

[1345] It receives the user's voice response and performs real-time voice recognition.

[1346] The emotion engine analyzes the user's voice and facial expressions when answering and generates emotional data.

[1347] 3. Feedback

[1348] The user's voice recognition results and emotion data are sent to the server, which receives the next question and displays it on the hologram.

[1349] If the emotional data indicates "anxiety" or "tension," the hologram provides feedback to help the user relax.

[1350] 4. Healthcare Data Management and Notification

[1351] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[1352] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[1353] User

[1354] 1. Starting and answering the medical interview

[1355] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[1356] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[1357] 2. Entering daily health data

[1358] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[1359] The data is sent to the server and the user's profile is updated.

[1360] 3. Collecting emotional data and receiving feedback

[1361] Receives feedback based on emotional data from the hologram and engages in adaptive dialogue.

[1362] For example, if a user responds, "I'm feeling a little anxious right now," the hologram will offer encouraging words such as, "It sounds like you're feeling anxious. I suggest you take some time off."

[1363] Specific examples

[1364] Assuming that user B is experiencing mental stress, the following specific process will be carried out:

[1365] 1. Initial data settings

[1366] User B uses the terminal to enter his / her name, age, and medical history of mental stress, and sends the information to the server.

[1367] The server receives this information and creates a healthcare profile for User B.

[1368] 2. Start of interview

[1369] When User B begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[1370] User B responds verbally, "I've been feeling stressed lately."

[1371] 3. Dynamic Questionnaire Generation and Emotion Recognition

[1372] The server inputs the voice recognition results and the "anxiety" emotion generated by the emotion engine into the AI ​​model, and dynamically generates the next question: "In what situations do you feel stressed?"

[1373] This question is displayed on the device via a hologram, and User B answers, "It's mainly pressure at work."

[1374] 4. Data recording and sharing

[1375] Each response and emotional data is sent to a server in real time and recorded.

[1376] The collected data will be shared with your family doctor and used as diagnostic material.

[1377] 5. Post-examination support

[1378] A stress management schedule for user B is generated on the server and sent to the terminal.

[1379] When it's time to take medication or engage in relaxation activities, a hologram on the device will notify you, saying, "Please perform your relaxation exercises now."

[1380] In this way, the present invention comprehensively manages the user's health status and emotional data, realizing accurate medical interviews and continuous follow-up.

[1381] The processing flow will be explained below.

[1382] Step 1:

[1383] The user presses the start button on the terminal, which starts the interview process.

[1384] Step 2:

[1385] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[1386] Step 3:

[1387] The user answers the question verbally, for example, "I'm a little tired."

[1388] Step 4:

[1389] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[1390] Step 5:

[1391] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[1392] Step 6:

[1393] The device receives the next question from the server and displays it via a 3D hologram.

[1394] Step 7:

[1395] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[1396] Step 8:

[1397] The device sends the user's voice and facial expressions to the emotion engine, which analyzes the user's emotions and generates emotion data indicating, for example, "anxiety" or "tension."

[1398] Step 9:

[1399] The device performs voice recognition again and transmits the text data and emotion data to the server.

[1400] Step 10:

[1401] The server repeats the process of generating further questions based on new recognition results and emotion data, continuing the interview process until all necessary information is collected. For example, if the emotion data indicates "anxiety," it generates additional questions that take emotion into consideration, such as "Have you been sleeping well recently?"

[1402] Step 11:

[1403] The server stores all collected data in a database and updates the user's healthcare profile.

[1404] Step 12:

[1405] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[1406] Step 13:

[1407] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[1408] Step 14:

[1409] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[1410] Step 15:

[1411] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[1412] Step 16:

[1413] The device will notify the user via a 3D hologram when it is time to take medication, when a medical checkup is scheduled, etc. For example, if the emotional data indicates "anxiety" or "stress," a reminder such as "We recommend you perform a relaxation exercise now" will be displayed.

[1414] Example 2

[1415] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1416] Conventional health management systems lack the functionality to analyze users' voice and emotional data in real time and provide appropriate medical feedback. Furthermore, they lack a system that can centrally manage follow-up after medical examinations and daily health data management. As a result, it is difficult for users to accurately and comprehensively understand their own health status.

[1417] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1418] In this invention, the server includes means for performing real-time voice recognition, means for dynamically generating questions based on the recognized voice data and emotion data, means for displaying the generated questions as a 3D image and presenting them to the user, means for receiving the user's voice responses and re-recognizing them, means for recording the recognition results and emotion data, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for providing feedback according to the user's emotion data. This enables comprehensive health management, including analyzing the user's health condition in real time, providing appropriate feedback based on the emotion data, and post-examination follow-ups.

[1419] "Means for real-time speech recognition" refers to devices or software that have the ability to instantly convert a user's voice into text data and process it in real time.

[1420] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a system that has the function of automatically generating the next question based on the situation using the results of voice recognition and emotional analysis.

[1421] "Means for displaying generated questions in 3D images and presenting them to the user" refers to a system that has the function of visually presenting questions generated for dialogue with the user using a 3D image display device.

[1422] "Means for receiving and re-recognizing the user's voice response" refers to devices or software that have the function of converting the user's voice response into text data using voice recognition technology and analyzing it.

[1423] "Means for recording the recognition results and emotion data" refers to a system that has the function of storing the voice recognition results and emotion analysis results in a recording medium such as a database.

[1424] "Means for sharing the recognition results and recorded data with medical professionals" refers to a system that has the function of securely transferring and sharing speech recognition results and recorded data with medical professionals.

[1425] "Means for post-visit follow-up based on health data" refers to a system that has the functionality to manage medication, guidance, and schedules after a visit based on the user's health data.

[1426] "Means for providing feedback based on the user's emotional data" refers to a system that has the function of analyzing the user's emotional data and providing appropriate advice or words of encouragement based on the results.

[1427] "Means for integrated management of health data" refers to a system that has the function of centrally collecting, analyzing, and managing users' daily health data.

[1428] "Medication management" refers to a system that manages a user's medication schedule and notifies them to take their medication appropriately.

[1429] "Instruction management" refers to a system that has the function of managing the content of instruction provided by medical professionals and providing appropriate instructions and advice to users.

[1430] "Schedule management" refers to a system that has the function of managing a user's medical-related schedules and notifying them at the appropriate time.

[1431] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up.

[1432] Server Roles

[1433] The server receives the user's basic information and stores it in a database. This information includes name, age, and medical history. For example, data such as "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure" is entered. The server uses this information to create a healthcare profile for each user.

[1434] Furthermore, when the user begins the medical interview, the server generates a predefined list of initial questions. For example, it generates a question such as, "How are you feeling right now?". The AI ​​model then dynamically generates the next question based on the speech recognition results and emotion data obtained by the emotion engine. This results in a dynamic question such as, "Since when have you been feeling tired?"

[1435] The server shares the collected medical interview data and daily health care data with the patient's primary care physician. This includes voice and emotional data. Schedules for medication management, instruction management, and schedule management are then generated and sent to the device. For example, instructions such as "Take your medicine at 8:00 every morning" are generated.

[1436] Device Role

[1437] When the user begins the medical interview, the device activates a 3D hologram and displays the first question. The hologram speaks, "Hello, how are you feeling right now?" The device recognizes the user's voice response in real time, and the emotion engine analyzes the emotion at the time of the response. For example, if the user responds, "I've been feeling stressed lately," the voice is converted into text and emotion data is generated.

[1438] The device then sends the user's voice recognition results and emotional data to the server, which then asks the next question and displays it on the hologram. For example, if the emotional data indicates anxiety or tension, the hologram might suggest, "Would you like to know how to relax?"

[1439] The device inputs the user's daily health data, such as temperature, blood pressure, and weight, and synchronizes it with the server. In addition, when it is time to take medication or when medical appointments are approaching, the device notifies the user via a hologram. For example, it issues an alert saying, "It's time to take your medicine."

[1440] User Roles

[1441] The user presses a button on the device to start the medical interview and answers the questions from the hologram by voice. For example, the hologram might ask, "How are you feeling right now?" and the user might reply, "I'm a little tired." Next, the user enters their daily health data (e.g., body temperature, blood pressure, weight) into the device, which then sends that data to the server to update their profile. For example, they might enter, "Today's body temperature is 36.5 degrees."

[1442] The hologram responds by providing feedback based on the user's emotional data, allowing for adaptive interactions. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will respond with encouragement, "It seems you're feeling anxious. Perhaps it would be good for you to take a short break."

[1443] Specific examples

[1444] If user B is experiencing mental stress, the following specific process will occur:

[1445] User B uses the device to enter their name, age, and medical history of mental stress, and sends this to the server. The server creates a healthcare profile. When the medical interview begins, a 3D hologram on the device displays the question, "How is your current health condition?" User B responds verbally, "I've been feeling very stressed lately." Based on the voice recognition results and the "anxiety" emotion generated by the emotion engine, the server dynamically generates the next question: "In what situations do you feel stressed?"

[1446] Each response and emotional data are sent to the server in real time and recorded. This data is then provided to the user's doctor as diagnostic information. User B's stress management schedule is generated by the server and sent to the device. For example, when it is time to take medication or engage in relaxation activities, a hologram on the device will notify the user, saying, "Please perform relaxation exercises now."

[1447] Example prompts for generative AI models

[1448] 1. "If a user has been stressed recently, what follow-up questions should be generated?"

[1449] 2. "If the emotion engine determines that the user is anxious, please suggest the appropriate feedback to provide next."

[1450] 3. "Generate follow-up questions based on the user's daily health data (e.g., body temperature, blood pressure)."

[1451] As described above, this system comprehensively manages the user's health status and emotional data, enabling accurate medical interviews and continuous follow-up.

[1452] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1453] Step 1:

[1454] User registration and initial data settings

[1455] The server receives the user's basic information (name, age, medical history) as input. It then stores this information in a database and creates a healthcare profile for each user. This profile serves as the foundation for the user to begin their medical interview. For example, if a user enters "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure," this data will be registered in the database.

[1456] Step 2:

[1457] Start of interview and generation of initial questions

[1458] When the user presses the start interview button, the terminal activates a 3D hologram. The hologram displays the first question to the user, "How is your current health condition?", and asks the question aloud. The input here is "pressing the start interview button," and the output is "displaying the initial question."

[1459] Step 3:

[1460] Speech and Emotion Recognition

[1461] The device receives the user's voice response as input and performs real-time voice recognition. The emotion engine then analyzes the user's voice and facial expressions to generate emotion data. At this stage, the input is the "voice response," and the output is the "recognized text data and emotion data." For example, if the user responds "I'm a little tired," the voice is converted into text data saying "I'm a little tired," and the emotion data generated is "anxiety."

[1462] Step 4:

[1463] Dynamic generation of next question

[1464] The server receives the speech recognition results and emotion data sent from the device as input. The AI ​​model then analyzes these data and dynamically generates the next question. At this stage, the input is the "speech recognition results and emotion data," and the output is the "next question." For example, the next question generated might be, "Since when have you been feeling tired?"

[1465] Step 5:

[1466] View next question and receive answer

[1467] The device receives the next question from the server, displays it on a 3D hologram, and presents it to the user. The user responds vocally, and the device again performs voice and emotion recognition. The input / output cycle is repeated. The input is "generation of the next question," and the output is "display of the next question and a vocal response."

[1468] Step 6:

[1469] Data recording and sharing

[1470] The server records the collected speech recognition results, emotion data, and medical history data in a database. Furthermore, a secure data transmission protocol is used to share important data with medical professionals. The input is "collected data," and the output is "recording in the database and sharing with medical professionals."

[1471] Step 7:

[1472] Follow-up schedule generation

[1473] The server generates a schedule for post-consultation medication management, instruction management, and schedule management based on the user's health data and medical interview results. The input is "health data and medical interview results," and the output is "follow-up schedule." For example, a medication instruction such as "Take your medicine every morning at 8 o'clock" is generated.

[1474] Step 8:

[1475] Providing Feedback

[1476] The device notifies the user of the generated follow-up schedule via a hologram. It also provides feedback based on the user's emotional data. The input is "follow-up schedule and emotional data," and the output is "notification and feedback." For example, the hologram may notify the user, "Please perform relaxation exercises now."

[1477] In this way, it is possible to comprehensively manage the user's health status and emotional data, and realize accurate medical interviews and continuous follow-up.

[1478] (Application example 2)

[1479] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1480] In modern society, there is a demand for efficient health management and related follow-up for users. However, conventional health management systems do not adequately provide comprehensive support that takes into account users' emotional states or personalized product recommendations. Therefore, a new system is needed to improve the quality of users' health management.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1482] In this invention, the server includes means for recognizing the user's voice in real time, means for dynamically generating questions based on the recognized voice data and emotional data, means for displaying the generated questions in 3D images and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and question history, means for sharing the recognition results and recorded data and emotional data, means for conducting post-examination follow-up based on health data, and means for recommending products based on preferences. This makes it possible to conduct medical interviews and follow-ups that reflect the user's emotional state, and further improve the quality of the user's health management through personalized product recommendations.

[1483] "Means for real-time speech recognition" refers to a function that instantly converts voice data provided by a user into text data.

[1484] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a function that analyzes the voice recognition results and the user's emotional state, and automatically generates appropriate questions based on these.

[1485] "Means for displaying the generated question in a 3D image and presenting it to the user" refers to a function that uses hologram technology or other 3D image display technology to visually show the generated question to the user.

[1486] The "means for receiving and re-recognizing the user's voice response" is a function for re-recognizing the user's verbal response and converting it into data.

[1487] The "means for recording the recognition results and question history" is a function for saving the speech recognition results and the history of questions and answers generated at that time in a database or the like.

[1488] "Means for sharing recognition results, recorded data, and emotional data" refers to the function of sharing recognition results, recorded question and answer history, and emotional data with other systems and experts.

[1489] "Means for post-consultation follow-up based on health data" is a function that utilizes health data to take appropriate measures to manage and support the user's health after a consultation.

[1490] "Preference-based product recommendation means" is a function that automatically suggests optimal products based on the user's preferences and health condition.

[1491] MODE FOR CARRYING OUT THE INVENTION

[1492] This is a health management system for online shopping sites that combines 3D holograms, voice recognition, AI models, an emotion engine, data sharing, and follow-up functions. The system grasps the user's health and emotional state in real time, and based on that, conducts appropriate medical interviews, follow-ups, and recommends related products.

[1493] System Overview

[1494] The system consists of three main components:

[1495] 1. Server

[1496] 2. Device (smartphone, etc.)

[1497] 3. Users

[1498] server

[1499] The server's main responsibilities are:

[1500] User registration and initial data setup: The server receives basic information provided by the user (such as name, age, and health status) and stores it in a database, which then creates a health profile for each user.

[1501] Dynamic Question Generation: Based on the speech recognition results and emotion data, the next question is dynamically generated using an AI model. In this process, the most appropriate question is presented to the user.

[1502] Data sharing: Collected voice data, emotional data, health data, etc. will be shared with the user's permission, allowing for more accurate support to be provided to the user.

[1503] Post-consultation follow-up and product recommendations: Based on the user's health and preference data, necessary follow-up and related product recommendations are provided.

[1504] Terminal

[1505] The device (e.g., smartphone) primarily performs the following functions:

[1506] 3D hologram generation and display: At the start of the interview, a 3D hologram is activated and presents questions to the user. The hologram displays the generated questions one after another, ensuring a smooth dialogue.

[1507] Speech and emotion recognition: The device recognizes the user's voice in real time and generates emotion data using an emotion engine. This data is sent to the server and used to generate the next question.

[1508] Feedback display: The voice recognition results and emotional data are sent to the server, and the next question is received from the server and displayed on the hologram. If the emotional data indicates "anxiety" or "tension," the device has the function of displaying feedback to relax the user.

[1509] User

[1510] The user performs the following actions:

[1511] Starting and answering the medical interview: Press the start button on the device and answer the questions from the hologram by voice. For example, in response to the initial question, "How are you feeling lately?", you can answer, "I'm a little tired."

[1512] Daily health data input: Enter your daily health data (body temperature, blood pressure, weight, etc.) into your device and synchronize it with the server, which will update your health profile in real time.

[1513] Collecting emotional data and receiving feedback: The hologram receives feedback based on emotional data collected during the interview and engages in adaptive dialogue. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will provide encouraging words such as, "It seems you're feeling anxious. I recommend you take a break."

[1514] Hardware and software used

[1515] Hardware: Smartphone (iOS or Android device)

[1516] software:

[1517] Speech Recognition: Converts speech to text using the speech_recognition library.

[1518] Emotion Recognition: Analyze emotions from speech and text using third-party emotion recognition libraries.

[1519] Generative AI Models: Use a generative AI model library to generate the next question.

[1520] Specific examples

[1521] For example, if a user enters "I've been feeling more stressed lately" in the default settings, the system will act as follows:

[1522] 1. The user answers verbally, "I've been busy at work lately and it's stressful."

[1523] 2. Speech to text: "Work has been busy and stressful lately."

[1524] 3. Sentiment analysis: Identify "stress."

[1525] 4. The AI ​​model generates the following question: "What situations make you feel particularly stressed?"

[1526] Prompt Sentence Examples

[1527] "Based on the analysis of the user's voice content and emotions, please generate the following appropriate questions. Voice content: 'Work has been busy lately and I'm stressed.' Emotion: 'Stressed.' Next question:"

[1528] This allows the medical interview process to proceed smoothly and allows for specific follow-up and product recommendations tailored to the user's health condition.

[1529] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1530] Step 1:

[1531] User Registration

[1532] 1. The server receives basic information entered by the user (name, age, health status) and stores it in a database.

[1533] Input: Basic information such as name, age, and health status.

[1534] Data processing: The entered information is registered in a database according to the format.

[1535] Output: User profile created.

[1536] Step 2:

[1537] Interview begins

[1538] 1. The user launches the app on their device and presses the button to start the medical interview.

[1539] Input: User operation (pressing the button to start the medical interview).

[1540] Data processing: Set the interview start flag.

[1541] Output: Start of the interview process.

[1542] 2. The device will activate a 3D hologram and display the first question.

[1543] Input: Interview start flag.

[1544] Data processing: Get the first question.

[1545] Output: Hologram of the question.

[1546] Step 3:

[1547] Speech and Emotion Recognition

[1548] 1. The user answers the first question by voice.

[1549] Input: Audio data.

[1550] Data processing: Acquisition of audio data.

[1551] Output: Save audio data.

[1552] 2. The device performs voice recognition and converts it into text data.

[1553] Input: Audio data.

[1554] Data processing: Text conversion using speech recognition (using the speech_recognition library).

[1555] Output: Text data.

[1556] 3. The emotion engine analyzes the voice and text data and generates emotion data.

[1557] Input: Audio and text data.

[1558] Data processing: Sentiment analysis (using emotion recognition libraries).

[1559] Output: Emotion data.

[1560] Step 4:

[1561] Dynamic interview generation

[1562] 1. The server receives the speech recognition results and emotion data and inputs a prompt sentence to the generative AI model to generate the next question.

[1563] Input: Speech recognition results, emotion data.

[1564] Data processing: Prompt sentence generation.

[1565] Output: The prompt statement.

[1566] 2. The server uses a generative AI model to dynamically generate the next question.

[1567] Input: prompt statement.

[1568] Data processing: The AI ​​model generates the next question.

[1569] Output: Next question.

[1570] Step 5:

[1571] Presenting the next question

[1572] 1. The server sends the following question to the terminal:

[1573] Input: Next question.

[1574] Data processing: Sending query data.

[1575] Output: Sends the next question to the terminal.

[1576] 2. The device uses a hologram to display the next question.

[1577] Input: Next question.

[1578] Data processing: Hologram generation and display.

[1579] Output: Hologram of the question.

[1580] Step 6:

[1581] Data recording and sharing

[1582] 1. The server records each response and emotion data in real time.

[1583] Input: Speech recognition results, emotion data, question history.

[1584] Data processing: Saving to database.

[1585] Output: Recorded data.

[1586] 2. The server shares the recorded data.

[1587] Input: Recorded data.

[1588] Data processing: Generating data for sharing.

[1589] Output: Sending shared data.

[1590] Step 7:

[1591] Follow-up and product recommendations

[1592] 1. The server generates a schedule for follow-up visits and preference-based product recommendations.

[1593] Input: Health data, preference data.

[1594] Data processing: Applying schedule generation and product recommendation algorithms.

[1595] Output: Follow-up schedule, product recommendation list.

[1596] 2. The device displays follow-up notifications and product recommendations to the user.

[1597] Input: Follow-up schedule, product recommendation list.

[1598] Data processing: generation of notification and display data.

[1599] Output: Follow-up notification, product recommendation display.

[1600] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1601] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1602] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1603] [Fourth embodiment]

[1604] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1605] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1606] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1607] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1608] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1609] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1610] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1611] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1612] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1613] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1614] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1615] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1616] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1617] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, data sharing, and post-consultation follow-up.

[1618] Overview of program processing

[1619] server

[1620] 1. User registration and initial data settings

[1621] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[1622] Create and manage your own healthcare profile.

[1623] 2. Dynamic Questionnaire Generation

[1624] When a user initiates a consultation, the server generates a predefined initial list of questions.

[1625] The AI ​​model receives the speech recognition results and dynamically generates the next question based on those results.

[1626] This process allows for appropriate medical interviews to be conducted based on the user's symptoms.

[1627] 3. Data sharing and post-examination support

[1628] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[1629] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[1630] Terminal

[1631] 1. 3D hologram generation

[1632] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[1633] The hologram presents the user with a series of generated questions.

[1634] 2. Voice Recognition and Feedback

[1635] It receives the user's voice response and performs real-time voice recognition.

[1636] The voice recognition results are sent to the server, which receives the next question and displays it on the hologram.

[1637] 3. Healthcare data management and notification

[1638] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[1639] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[1640] User

[1641] 1. Starting and answering the medical interview

[1642] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[1643] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[1644] 2. Entering daily health data

[1645] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[1646] The data is sent to the server and the user's profile is updated.

[1647] 3. Notification and follow-up

[1648] Receive notifications from the hologram when it's time to take your medication and take it as instructed.

[1649] If you have a scheduled health check or medical examination, you will be notified the day before or just before.

[1650] Specific examples

[1651] If user A has a history of high blood pressure, the following specific process is executed:

[1652] 1. Initial data settings

[1653] User A uses a terminal to enter his / her name, age, and medical history of high blood pressure, and sends the information to the server.

[1654] The server receives this information and creates a healthcare profile for User A.

[1655] 2. Start of interview

[1656] When user A begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[1657] User A answers by voice, "My blood pressure has been high recently."

[1658] 3. Dynamic Questionnaire Generation

[1659] The server inputs the speech recognition results into the AI ​​model and dynamically generates the next question: "Do you exercise?"

[1660] This question is displayed on the device via a hologram, and User A answers, "I haven't been exercising much lately."

[1661] 4. Data recording and sharing

[1662] Each response is sent in real time to a server and recorded.

[1663] This data will be shared with your family doctor and used as diagnostic material.

[1664] 5. Post-examination support

[1665] User A's medication schedule is generated on the server and sent to the terminal.

[1666] When it's time to take your medication, a hologram on the device will notify you, "Take your medication now."

[1667] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[1668] The processing flow will be explained below.

[1669] Step 1:

[1670] The user presses the start button on the terminal, which starts the interview process.

[1671] Step 2:

[1672] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[1673] Step 3:

[1674] The user answers the question verbally, for example, "I'm a little tired."

[1675] Step 4:

[1676] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[1677] Step 5:

[1678] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[1679] Step 6:

[1680] The device receives the next question from the server and displays it via a 3D hologram.

[1681] Step 7:

[1682] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[1683] Step 8:

[1684] The device performs voice recognition again, converts it into text data, and sends it to the server.

[1685] Step 9:

[1686] The server repeats the process of generating more questions based on the new recognition results, and continues interrogating until all the necessary information is collected.

[1687] Step 10:

[1688] The server stores all collected data in a database and updates the user's healthcare profile.

[1689] Step 11:

[1690] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[1691] Step 12:

[1692] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[1693] Step 13:

[1694] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[1695] Step 14:

[1696] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[1697] Step 15:

[1698] When it's time to take your medication or when your health checkup is scheduled, the device will notify you via a 3D hologram, displaying a reminder such as "Take your medicine now."

[1699] Example 1

[1700] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1701] Conventional health management systems require a lot of effort for user data entry and follow-up, making it difficult to accurately interview patients and share data in real time. Furthermore, there are limitations in the accuracy of voice recognition and the generation of dynamic questions, making it difficult to provide appropriate responses based on the user's symptoms. Furthermore, follow-up after consultations is often insufficient, resulting in inadequate ongoing health management.

[1702] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1703] In this invention, the server includes a means for receiving user registration and basic information and storing it in a database, a means for performing real-time speech recognition, and a means for dynamically generating questions using prompts generated based on the recognized speech data, thereby enabling accurate collection of user health data in real time, continuous follow-up, and comprehensive health management.

[1704] "User registration" is the process of entering basic user information into the system and storing it in the database.

[1705] "Speech recognition" is a technology that receives a user's voice and interprets it as digital data.

[1706] A "prompt" is the text for the next question generated based on an AI model.

[1707] "Dynamic question generation" is the process of generating the next question in real time based on the user's speech recognition results.

[1708] "3D hologram display" is a method of visually presenting questions and notifications to users using three-dimensional imaging technology.

[1709] "Recognition result" refers to text data obtained by speech recognition.

[1710] "Question History" is a record of all questions posed to a user and their answers.

[1711] "Health professionals" are people whose job it is to evaluate health data and make diagnoses and treatment decisions.

[1712] "Data sharing" is the process of sharing collected data with another user or system.

[1713] "Follow-up" refers to ongoing support activities to help users maintain their health after a medical examination.

[1714] "Healthcare data" refers to data that represents the user's daily health condition (body temperature, blood pressure, weight, etc.).

[1715] "Drug management" is the process of properly managing the timing and amount of medication a user takes.

[1716] "Insurance guidance" is the process of providing information and advice related to a user's health insurance.

[1717] "Appointment management" is the process of scheduling a user's health-related appointments (such as doctor's appointments and checkups) and providing appropriate notifications.

[1718] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, generative AI models, data sharing, and follow-up functions. The specific hardware and software usage and data processing procedures are described below.

[1719] Server configuration and processing

[1720] 1. User registration and initial data settings

[1721] The server receives basic user information (such as name, age, medical history, etc.) from the registration form and stores it in a database. Based on this data, a healthcare profile is generated for each user.

[1722] 2. Dynamic Questionnaire Generation

[1723] When the user starts the medical interview, the server sends an initial list of questions to the device. After receiving the user's voice responses, the server analyzes them using speech recognition software (e.g., Google Cloud Speech-to-Text API). Based on the analysis results, a generative AI model (e.g., BERT or GPT-3) generates the next questions.

[1724] 3. Data sharing and post-examination support

[1725] The collected medical interview data and daily health care data are shared with the patient's primary care physician, and a schedule for follow-up visits after the visit is generated and sent to the device.

[1726] Terminal configuration and handling

[1727] 1. 3D hologram generation

[1728] When the user begins the consultation, the device activates a hologram using 3D holographic technology. The first question is displayed as a hologram and presented to the user. An example of the hardware used is a display from the Looking Glass Factory.

[1729] 2. Voice Recognition and Feedback

[1730] The device receives the user's voice response through a microphone, analyzes it using speech recognition software, and sends the results to the server, which then receives the next dynamically generated question and displays it on the hologram.

[1731] 3. Healthcare data management and notification

[1732] The device allows users to input their daily health data (body temperature, blood pressure, weight, etc.) and synchronizes it with a server. It notifies users using holograms when it's time to take their medication or when medical appointments are approaching.

[1733] User operations

[1734] 1. Starting and answering the medical interview

[1735] The user presses a button on the device to start the medical interview and answers the questions posed by the hologram by voice. For example, the initial question, "How are you feeling right now?" is answered with, "I'm a little tired."

[1736] 2. Entering daily health data

[1737] Users input their daily health data into the device, for example, by measuring their body temperature with a thermometer, and the data is then sent to the server.

[1738] 3. Notification and follow-up

[1739] When it's time to take their medication, the user will receive a notification from the hologram on the device saying, "Take your medicine now," and will follow the instructions to take their medicine. They will also receive notifications the day before or just before scheduled medical checkups or appointments.

[1740] Specific examples

[1741] Prompt Sentence Examples

[1742] Initial question: "How are you feeling right now?"

[1743] Voice response: "I'm a little tired"

[1744] Next question: "Do you exercise?"

[1745] Voice response: "I haven't been exercising much lately."

[1746] As described above, this system utilizes voice recognition and generative AI models to dynamically grasp the user's health status and provide appropriate questions and follow-ups, enabling comprehensive health management.

[1747] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1748] Step 1:

[1749] User registration and initial data settings

[1750] Users enter basic information such as their name, age, and medical history into a registration form on their device. The device then sends this information to the server, which then stores the received information in a database and creates a healthcare profile for each user. Specifically, the server generates a user ID and associates the entered data with that ID and stores it.

[1751] Input: User's basic information (name, age, medical history)

[1752] Output: User ID, healthcare profile

[1753] Step 2:

[1754] Beginning of medical interview

[1755] The user presses the "Start medical interview" button on the device. The device retrieves a list of initial questions from the server and activates a 3D hologram to display the initial questions. The user answers the questions verbally. The device receives the user's voice responses through a microphone and performs voice recognition in real time. The voice recognition results are sent to the server. Specifically, the voice data is converted into text data, and that text data is sent to the server.

[1756] Input: Press the start button and answer by voice

[1757] Output: Initial question list, speech-recognized text data

[1758] Step 3:

[1759] Dynamic interview generation

[1760] The server receives the speech recognition results and inputs them into a generative AI model. The AI ​​model (e.g., BERT or GPT-3) generates the next question based on the speech recognition results and the prompt. The server then sends the generated question to the device. The device displays the next question as a 3D hologram and presents it to the user. This allows the server to dynamically generate and display appropriate questions based on the user's specific symptoms.

[1761] Input: Speech recognition result (text), prompt

[1762] Output: Next question (text)

[1763] Step 4:

[1764] Voice Recognition and Feedback

[1765] The device receives the user's new voice response and performs voice recognition again. It sends the voice recognition result to the server and waits for the next appropriate question. This enables continuous dynamic interviews. Specifically, the device converts voice into text in real time and sends the text data to the server.

[1766] Input: Voice response

[1767] Output: Recognized text data

[1768] Step 5:

[1769] Data recording and sharing

[1770] The server records all medical interview data and voice recognition results. The recorded data is shared with medical professionals as needed. Medical professionals can use this data as a reference for diagnosis and treatment. Specifically, the server stores the data in a database and makes some or all of the data accessible to medical professionals.

[1771] Input: Voice recognition results, medical interview data

[1772] Output: Recorded interview data, data shared with medical professionals

[1773] Step 6:

[1774] Healthcare data management and notification

[1775] Users enter their daily health data (such as temperature, blood pressure, and weight) into the device. The device then sends this data to a server, which updates the user's health profile. The device also uses 3D holograms to notify users when it's time to take medication or when medical appointments are approaching. For example, a notification like "Take your medicine now" may appear on the hologram.

[1776] Input: Daily health data (temperature, blood pressure, weight, etc.), schedule data

[1777] Output: Updated health profile, notification message

[1778] Step 7:

[1779] Follow-up after consultation

[1780] The server generates a schedule for follow-ups after the consultation and sends it to the device. The user manages medication and health checkup schedules according to notifications from the device. The server sends follow-up notifications in a timely manner based on the user's healthcare data and schedule. Specifically, the server calculates medication schedules and health checkup schedules, and sends them to the device for display.

[1781] Input: Follow-up schedule, healthcare data

[1782] Output: Notification messages, management schedules

[1783] (Application example 1)

[1784] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1785] Conventional health management systems lack the ability to dynamically generate questions in response to user inquiries or real-time voice recognition and feedback using 3D holograms, making it difficult to provide comprehensive and personalized support for users' health management. They also lack a system for sharing generated data with medical professionals for appropriate follow-up. Furthermore, they are unable to provide individually customized health advice based on each user's health information, resulting in a lack of accuracy and usefulness of the health advice.

[1786] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1787] In this invention, the server includes means for performing real-time speech recognition, means for dynamically generating questions based on the recognized speech data, means for displaying the generated questions as 3D holograms and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and a history of the questions, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for generating and displaying individually customized health advice based on each user's health information, thereby enabling accurate and comprehensive management of the user's health condition in real time and providing individually customized health advice.

[1788] "Means for real-time speech recognition" refers to technology that can instantly convert a user's speech into digital data and analyze its content.

[1789] "Means for dynamically generating questions based on recognized voice data" refers to a technology in which an AI model automatically creates appropriate questions based on the user's current state, based on analyzed voice data.

[1790] "Means of displaying and presenting to users as 3D holograms" refers to a technology that visually provides users with content generated using three-dimensional video technology.

[1791] "Means for receiving and re-recognizing the user's voice response" refers to a technology that analyzes the user's voice response again using voice recognition technology and reflects the results in the system.

[1792] The "means for recording the recognition results and the history of the questions" is a technology for saving the speech recognition results and the history of the questions generated as data.

[1793] "Means for sharing recorded data with medical professionals" refers to technology that securely and accurately transfers stored voice recognition results and question history data to medical professionals.

[1794] "Means for post-medical follow-up based on health data" refers to technology that analyzes a user's daily health data and, based on that data, provides appropriate guidance and advice after a medical visit.

[1795] "A means for generating and displaying individually customized health advice based on each user's health information" refers to a technology that automatically generates specific health advice based on each user's individual health condition and history, and presents it using 3D holograms, etc.

[1796] This invention is a system that performs real-time voice recognition, dynamically generates questions based on the data, and presents them to the user using a 3D hologram. The system also receives the user's answers via voice and records the recognition results and question history. It also shares the recognition results and recorded data with medical professionals, who use the health data to provide follow-up after consultations. It also includes a means for generating and displaying individually customized health advice based on each user's health information.

[1797] Server Operation

[1798] The server is responsible for user registration and initial data configuration. It receives the user's basic information (name, age, medical history) and stores it in a database. It is also equipped with an AI model that receives voice recognition results and dynamically generates the next question. The generated questions and voice recognition results are recorded in a database and shared with medical professionals along with health data. After the consultation, a schedule for medication management and appointment management is generated and sent to the user's device.

[1799] Device behavior

[1800] The device uses 3D holograms to display dynamically generated questions to the user. It recognizes the user's answers in real time and sends the results to the server. The user's daily health data (body temperature, blood pressure, weight, etc.) is also entered into the device and synchronized with the server. The device notifies the user via 3D holograms when it is time to take medication or when medical appointments are approaching.

[1801] User Actions

[1802] The user presses the button to start the medical interview, answers the questions posed by the hologram by voice, and enters their daily health data into the device. For example, to the question, "How are you feeling right now?", the user can reply, "I'm a little tired." The voice response is recognized in real time, and the next question is presented through the hologram. The user also receives notifications and follows instructions when it's time to take medication or when medical appointments are approaching.

[1803] Specific Examples

[1804] If User A has a history of high blood pressure, the following process is executed. User A enters their name, age, and history of high blood pressure and sends them to the server. The server uses this information to create a healthcare profile for User A. When User A begins the medical interview, a 3D hologram on the device displays the question, "How is your current health?" User A replies, "My blood pressure has been high recently." The server inputs the voice recognition results into an AI model, which dynamically generates the next question, "Are you exercising?" This question is displayed via the hologram, and User A replies, "I haven't been exercising much recently."

[1805] Prompt Sentence Examples

[1806] User: "How are you feeling these days?"

[1807] Answer: "I have high blood pressure and get tired easily."

[1808] Question to prompt: "Do you exercise?"

[1809] In this way, the present invention comprehensively supports the user's health management, enabling accurate medical interviews and continuous follow-up.

[1810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1811] Step 1: User Registration

[1812] Users enter basic information such as their name, age, and medical history into the device and press the registration button. The device then sends this data to the server, which stores the received information in a database and creates a healthcare profile for each user.

[1813] Input: Name, age, medical history

[1814] Output: Generate a healthcare profile

[1815] Specific behavior: Inserts information into the database and creates an initial profile

[1816] Step 2: Begin the interview

[1817] The user presses the button to start the medical interview, and the device uses a 3D hologram to display the first question: "How are you feeling right now?" The user then answers this question verbally. The device recognizes the user's voice in real time, converts it into digital data, and sends it to the server.

[1818] Input: Voice input "I'm a little tired"

[1819] Output: Speech recognition results as digital data

[1820] Specific operation: Converts speech into text using a speech recognition engine and sends it to the server

[1821] Step 3: Dynamic Questionnaire Generation

[1822] The server analyzes the received voice recognition data and inputs it into the AI ​​model. The AI ​​model dynamically generates the next question: "Do you exercise?" The generated question is sent to the device.

[1823] Input: Speech recognition result: "I'm a little tired"

[1824] Output: Next question: "Do you exercise?"

[1825] How it works: Uses AI models to dynamically generate the next question

[1826] Step 4: Ask the next question

[1827] The device then displays the next question as a 3D hologram and presents it to the user. The user answers, "I haven't been exercising much lately." The device again recognizes the voice in real time and sends the results to the server.

[1828] Input: Voice input "I haven't been exercising much lately"

[1829] Output: The following speech recognition results stored in the database:

[1830] Specific operation: Displaying questions using 3D holograms, recognizing voice and sending it to the server

[1831] Step 5: Record data and share with medical professionals

[1832] The server records the voice recognition results and question history in a database, which is shared with medical professionals periodically or upon request.

[1833] Input: Speech recognition result: "I haven't been exercising much lately."

[1834] Output: Data shared with medical professionals

[1835] Specific actions: Recorded in a database and shared with medical professionals through a dedicated interface

[1836] Step 6: Enter and sync your daily health data

[1837] Users input their daily health data, such as body temperature, blood pressure, and weight, into the device, which then sends this data to the server, which stores it in a database.

[1838] Input: Health data such as body temperature, blood pressure, and weight

[1839] Output: Updated healthcare profile

[1840] Specific operation: Enter health data, send it to the server and save it

[1841] Step 7: Follow-up after the visit

[1842] The server generates a schedule for medication management and schedule management as follow-up after the consultation and sends it to the terminal. The terminal checks this schedule and notifies the user at the specified time using a 3D hologram.

[1843] Input: Health data, feedback from medical professionals

[1844] Output: Follow-up schedule and notifications

[1845] Specific actions: Follow-up schedule generation and notifications using 3D holograms

[1846] Step 8: Generate personalized health advice

[1847] The server uses AI models to generate personalized health advice based on each user's health information, which is then sent to the device and displayed to the user as a 3D hologram.

[1848] Input: User health information and daily data

[1849] Output: Customized health advice

[1850] How it works: Uses AI models to generate health advice and displays it in a 3D hologram

[1851] This series of steps enables accurate and comprehensive management of the user's health status in real time and provides individually customized health advice.

[1852] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1853] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system integrates real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up to provide accurate and comprehensive health management.

[1854] Overview of program processing

[1855] server

[1856] 1. User registration and initial data settings

[1857] The server receives the user's basic information (name, age, medical history) and stores it in a database.

[1858] Create and manage your own healthcare profile.

[1859] 2. Dynamic Questionnaire Generation

[1860] When a user initiates a consultation, the server generates a predefined initial list of questions.

[1861] The AI ​​model receives the speech recognition results and the emotion data recognized by the emotion engine, and dynamically generates the next question based on the results.

[1862] This process allows for appropriate interviews to be conducted based on the user's symptoms and emotions.

[1863] 3. Data sharing and post-examination support

[1864] The collected medical interview data and the user's daily healthcare data will be shared with the user's primary care physician.

[1865] Sharing emotional data will also help medical professionals make more accurate diagnoses.

[1866] After the consultation, a schedule for medication management, insurance guidance, and schedule management is generated and sent to the terminal.

[1867] Terminal

[1868] 1. 3D hologram generation

[1869] When the user begins the interview, a 3D hologram activates on the device and displays the first question.

[1870] The hologram presents the user with a series of generated questions.

[1871] 2. Speech and Emotion Recognition

[1872] It receives the user's voice response and performs real-time voice recognition.

[1873] The emotion engine analyzes the user's voice and facial expressions when answering and generates emotional data.

[1874] 3. Feedback

[1875] The user's voice recognition results and emotion data are sent to the server, which receives the next question and displays it on the hologram.

[1876] If the emotional data indicates "anxiety" or "tension," the hologram provides feedback to help the user relax.

[1877] 4. Healthcare Data Management and Notification

[1878] Users enter their daily health data, such as body temperature, blood pressure, and weight, and synchronize it with the server.

[1879] The hologram notifies the user when it's time to take medication or when a medical appointment is approaching.

[1880] User

[1881] 1. Starting and answering the medical interview

[1882] Press the button to start the medical interview on the device and answer the questions from the hologram by voice.

[1883] For example, in response to the initial question, "How are you feeling right now?", you might respond, "I'm a little tired."

[1884] 2. Entering daily health data

[1885] Enter your daily health data (body temperature, blood pressure, weight, etc.) into the device.

[1886] The data is sent to the server and the user's profile is updated.

[1887] 3. Collecting emotional data and receiving feedback

[1888] Receives feedback based on emotional data from the hologram and engages in adaptive dialogue.

[1889] For example, if a user responds, "I'm feeling a little anxious right now," the hologram will offer encouraging words such as, "It sounds like you're feeling anxious. I suggest you take some time off."

[1890] Specific examples

[1891] Assuming that user B is experiencing mental stress, the following specific process will be carried out:

[1892] 1. Initial data settings

[1893] User B uses the terminal to enter his / her name, age, and medical history of mental stress, and sends the information to the server.

[1894] The server receives this information and creates a healthcare profile for User B.

[1895] 2. Start of interview

[1896] When User B begins the medical interview, a 3D hologram appears on the device asking, "How are you feeling right now?"

[1897] User B responds verbally, "I've been feeling stressed lately."

[1898] 3. Dynamic Questionnaire Generation and Emotion Recognition

[1899] The server inputs the voice recognition results and the "anxiety" emotion generated by the emotion engine into the AI ​​model, and dynamically generates the next question: "In what situations do you feel stressed?"

[1900] This question is displayed on the device via a hologram, and User B answers, "It's mainly pressure at work."

[1901] 4. Data recording and sharing

[1902] Each response and emotional data is sent to a server in real time and recorded.

[1903] The collected data will be shared with your family doctor and used as diagnostic material.

[1904] 5. Post-examination support

[1905] A stress management schedule for user B is generated on the server and sent to the terminal.

[1906] When it's time to take medication or engage in relaxation activities, a hologram on the device will notify you, saying, "Please perform your relaxation exercises now."

[1907] In this way, the present invention comprehensively manages the user's health status and emotional data, realizing accurate medical interviews and continuous follow-up.

[1908] The processing flow will be explained below.

[1909] Step 1:

[1910] The user presses the start button on the terminal, which starts the interview process.

[1911] Step 2:

[1912] The device will activate a 3D hologram and ask some initial questions, such as "How are you feeling right now?"

[1913] Step 3:

[1914] The user answers the question verbally, for example, "I'm a little tired."

[1915] Step 4:

[1916] The device sends the user's voice to a voice recognition module, which converts the voice data into text.

[1917] Step 5:

[1918] The server receives the speech recognition results and has the AI ​​model analyze them. The next question is generated based on the speech recognition results. For example, an additional question such as "Are you still feeling tired?" is created.

[1919] Step 6:

[1920] The device receives the next question from the server and displays it via a 3D hologram.

[1921] Step 7:

[1922] The user answers the follow-up question verbally, for example, by saying "Yes, it's still going on."

[1923] Step 8:

[1924] The device sends the user's voice and facial expressions to the emotion engine, which analyzes the user's emotions and generates emotion data indicating, for example, "anxiety" or "tension."

[1925] Step 9:

[1926] The device performs voice recognition again and transmits the text data and emotion data to the server.

[1927] Step 10:

[1928] The server repeats the process of generating further questions based on new recognition results and emotion data, continuing the interview process until all necessary information is collected. For example, if the emotion data indicates "anxiety," it generates additional questions that take emotion into consideration, such as "Have you been sleeping well recently?"

[1929] Step 11:

[1930] The server stores all collected data in a database and updates the user's healthcare profile.

[1931] Step 12:

[1932] The server sends the updated data to share with your doctor, who will then use it to make a diagnosis.

[1933] Step 13:

[1934] The server generates a follow-up schedule (medication management, schedule management, etc.) after the consultation and sends it to the terminal.

[1935] Step 14:

[1936] The device manages the user's daily health data (body temperature, blood pressure, weight, etc.) and displays reminders and notifications via 3D holograms.

[1937] Step 15:

[1938] Users enter their daily healthcare data into the device, and the data is synchronized with the server in real time.

[1939] Step 16:

[1940] The device will notify the user via a 3D hologram when it is time to take medication, when a medical checkup is scheduled, etc. For example, if the emotional data indicates "anxiety" or "stress," a reminder such as "We recommend you perform a relaxation exercise now" will be displayed.

[1941] Example 2

[1942] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1943] Conventional health management systems lack the functionality to analyze users' voice and emotional data in real time and provide appropriate medical feedback. Furthermore, they lack a system that can centrally manage follow-up after medical examinations and daily health data management. As a result, it is difficult for users to accurately and comprehensively understand their own health status.

[1944] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1945] In this invention, the server includes means for performing real-time voice recognition, means for dynamically generating questions based on the recognized voice data and emotion data, means for displaying the generated questions as a 3D image and presenting them to the user, means for receiving the user's voice responses and re-recognizing them, means for recording the recognition results and emotion data, means for sharing the recognition results and recorded data with medical professionals, means for performing post-examination follow-ups based on the health data, and means for providing feedback according to the user's emotion data. This enables comprehensive health management, including analyzing the user's health condition in real time, providing appropriate feedback based on the emotion data, and post-examination follow-ups.

[1946] "Means for real-time speech recognition" refers to devices or software that have the ability to instantly convert a user's voice into text data and process it in real time.

[1947] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a system that has the function of automatically generating the next question based on the situation using the results of voice recognition and emotional analysis.

[1948] "Means for displaying generated questions in 3D images and presenting them to the user" refers to a system that has the function of visually presenting questions generated for dialogue with the user using a 3D image display device.

[1949] "Means for receiving and re-recognizing the user's voice response" refers to devices or software that have the function of converting the user's voice response into text data using voice recognition technology and analyzing it.

[1950] "Means for recording the recognition results and emotion data" refers to a system that has the function of storing the voice recognition results and emotion analysis results in a recording medium such as a database.

[1951] "Means for sharing the recognition results and recorded data with medical professionals" refers to a system that has the function of securely transferring and sharing speech recognition results and recorded data with medical professionals.

[1952] "Means for post-visit follow-up based on health data" refers to a system that has the functionality to manage medication, guidance, and schedules after a visit based on the user's health data.

[1953] "Means for providing feedback based on the user's emotional data" refers to a system that has the function of analyzing the user's emotional data and providing appropriate advice or words of encouragement based on the results.

[1954] "Means for integrated management of health data" refers to a system that has the function of centrally collecting, analyzing, and managing users' daily health data.

[1955] "Medication management" refers to a system that manages a user's medication schedule and notifies them to take their medication appropriately.

[1956] "Instruction management" refers to a system that has the function of managing the content of instruction provided by medical professionals and providing appropriate instructions and advice to users.

[1957] "Schedule management" refers to a system that has the function of managing a user's medical-related schedules and notifying them at the appropriate time.

[1958] This invention is a comprehensive health management system that combines 3D holograms, voice recognition, AI models, emotion engines, data sharing, and follow-up functions. The system provides accurate and comprehensive health management by integrating real-time medical interviews, user emotion recognition, data sharing, and post-consultation follow-up.

[1959] Server Roles

[1960] The server receives the user's basic information and stores it in a database. This information includes name, age, and medical history. For example, data such as "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure" is entered. The server uses this information to create a healthcare profile for each user.

[1961] Furthermore, when the user begins the medical interview, the server generates a predefined list of initial questions. For example, it generates a question such as, "How are you feeling right now?". The AI ​​model then dynamically generates the next question based on the speech recognition results and emotion data obtained by the emotion engine. This results in a dynamic question such as, "Since when have you been feeling tired?"

[1962] The server shares the collected medical interview data and daily health care data with the patient's primary care physician. This includes voice and emotional data. Schedules for medication management, instruction management, and schedule management are then generated and sent to the device. For example, instructions such as "Take your medicine at 8:00 every morning" are generated.

[1963] Device Role

[1964] When the user begins the medical interview, the device activates a 3D hologram and displays the first question. The hologram speaks, "Hello, how are you feeling right now?" The device recognizes the user's voice response in real time, and the emotion engine analyzes the emotion at the time of the response. For example, if the user responds, "I've been feeling stressed lately," the voice is converted into text and emotion data is generated.

[1965] The device then sends the user's voice recognition results and emotional data to the server, which then asks the next question and displays it on the hologram. For example, if the emotional data indicates anxiety or tension, the hologram might suggest, "Would you like to know how to relax?"

[1966] The device inputs the user's daily health data, such as temperature, blood pressure, and weight, and synchronizes it with the server. In addition, when it is time to take medication or when medical appointments are approaching, the device notifies the user via a hologram. For example, it issues an alert saying, "It's time to take your medicine."

[1967] User Roles

[1968] The user presses a button on the device to start the medical interview and answers the questions from the hologram by voice. For example, the hologram might ask, "How are you feeling right now?" and the user might reply, "I'm a little tired." Next, the user enters their daily health data (e.g., body temperature, blood pressure, weight) into the device, which then sends that data to the server to update their profile. For example, they might enter, "Today's body temperature is 36.5 degrees."

[1969] The hologram responds by providing feedback based on the user's emotional data, allowing for adaptive interactions. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will respond with encouragement, "It seems you're feeling anxious. Perhaps it would be good for you to take a short break."

[1970] Specific examples

[1971] If user B is experiencing mental stress, the following specific process will occur:

[1972] User B uses the device to enter their name, age, and medical history of mental stress, and sends this to the server. The server creates a healthcare profile. When the medical interview begins, a 3D hologram on the device displays the question, "How is your current health condition?" User B responds verbally, "I've been feeling very stressed lately." Based on the voice recognition results and the "anxiety" emotion generated by the emotion engine, the server dynamically generates the next question: "In what situations do you feel stressed?"

[1973] Each response and emotional data are sent to the server in real time and recorded. This data is then provided to the user's doctor as diagnostic information. User B's stress management schedule is generated by the server and sent to the device. For example, when it is time to take medication or engage in relaxation activities, a hologram on the device will notify the user, saying, "Please perform relaxation exercises now."

[1974] Example prompts for generative AI models

[1975] 1. "If a user has been stressed recently, what follow-up questions should be generated?"

[1976] 2. "If the emotion engine determines that the user is anxious, please suggest the appropriate feedback to provide next."

[1977] 3. "Generate follow-up questions based on the user's daily health data (e.g., body temperature, blood pressure)."

[1978] As described above, this system comprehensively manages the user's health status and emotional data, enabling accurate medical interviews and continuous follow-up.

[1979] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1980] Step 1:

[1981] User registration and initial data settings

[1982] The server receives the user's basic information (name, age, medical history) as input. It then stores this information in a database and creates a healthcare profile for each user. This profile serves as the foundation for the user to begin their medical interview. For example, if a user enters "Taro Tanaka, 30 years old, previously diagnosed with high blood pressure," this data will be registered in the database.

[1983] Step 2:

[1984] Start of interview and generation of initial questions

[1985] When the user presses the start interview button, the terminal activates a 3D hologram. The hologram displays the first question to the user, "How is your current health condition?", and asks the question aloud. The input here is "pressing the start interview button," and the output is "displaying the initial question."

[1986] Step 3:

[1987] Speech and Emotion Recognition

[1988] The device receives the user's voice response as input and performs real-time voice recognition. The emotion engine then analyzes the user's voice and facial expressions to generate emotion data. At this stage, the input is the "voice response," and the output is the "recognized text data and emotion data." For example, if the user responds "I'm a little tired," the voice is converted into text data saying "I'm a little tired," and the emotion data generated is "anxiety."

[1989] Step 4:

[1990] Dynamic generation of next question

[1991] The server receives the speech recognition results and emotion data sent from the device as input. The AI ​​model then analyzes these data and dynamically generates the next question. At this stage, the input is the "speech recognition results and emotion data," and the output is the "next question." For example, the next question generated might be, "Since when have you been feeling tired?"

[1992] Step 5:

[1993] View next question and receive answer

[1994] The device receives the next question from the server, displays it on a 3D hologram, and presents it to the user. The user responds vocally, and the device again performs voice and emotion recognition. The input / output cycle is repeated. The input is "generation of the next question," and the output is "display of the next question and a vocal response."

[1995] Step 6:

[1996] Data recording and sharing

[1997] The server records the collected speech recognition results, emotion data, and medical history data in a database. Furthermore, a secure data transmission protocol is used to share important data with medical professionals. The input is "collected data," and the output is "recording in the database and sharing with medical professionals."

[1998] Step 7:

[1999] Follow-up schedule generation

[2000] The server generates a schedule for post-consultation medication management, instruction management, and schedule management based on the user's health data and medical interview results. The input is "health data and medical interview results," and the output is "follow-up schedule." For example, a medication instruction such as "Take your medicine every morning at 8 o'clock" is generated.

[2001] Step 8:

[2002] Providing Feedback

[2003] The device notifies the user of the generated follow-up schedule via a hologram. It also provides feedback based on the user's emotional data. The input is "follow-up schedule and emotional data," and the output is "notification and feedback." For example, the hologram may notify the user, "Please perform relaxation exercises now."

[2004] In this way, it is possible to comprehensively manage the user's health status and emotional data, and realize accurate medical interviews and continuous follow-up.

[2005] (Application example 2)

[2006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2007] In modern society, there is a demand for efficient health management and related follow-up for users. However, conventional health management systems do not adequately provide comprehensive support that takes into account users' emotional states or personalized product recommendations. Therefore, a new system is needed to improve the quality of users' health management.

[2008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2009] In this invention, the server includes means for recognizing the user's voice in real time, means for dynamically generating questions based on the recognized voice data and emotional data, means for displaying the generated questions in 3D images and presenting them to the user, means for receiving the user's answers by voice and re-recognizing them, means for recording the recognition results and question history, means for sharing the recognition results and recorded data and emotional data, means for conducting post-examination follow-up based on health data, and means for recommending products based on preferences. This makes it possible to conduct medical interviews and follow-ups that reflect the user's emotional state, and further improve the quality of the user's health management through personalized product recommendations.

[2010] "Means for real-time speech recognition" refers to a function that instantly converts voice data provided by a user into text data.

[2011] "Means for dynamically generating questions based on recognized voice data and emotional data" refers to a function that analyzes the voice recognition results and the user's emotional state, and automatically generates appropriate questions based on these.

[2012] "Means for displaying the generated question in a 3D image and presenting it to the user" refers to a function that uses hologram technology or other 3D image display technology to visually show the generated question to the user.

[2013] The "means for receiving and re-recognizing the user's voice response" is a function for re-recognizing the user's verbal response and converting it into data.

[2014] The "means for recording the recognition results and question history" is a function for saving the speech recognition results and the history of questions and answers generated at that time in a database or the like.

[2015] "Means for sharing recognition results, recorded data, and emotional data" refers to the function of sharing recognition results, recorded question and answer history, and emotional data with other systems and experts.

[2016] "Means for post-consultation follow-up based on health data" is a function that utilizes health data to take appropriate measures to manage and support the user's health after a consultation.

[2017] "Preference-based product recommendation means" is a function that automatically suggests optimal products based on the user's preferences and health condition.

[2018] MODE FOR CARRYING OUT THE INVENTION

[2019] This is a health management system for online shopping sites that combines 3D holograms, voice recognition, AI models, an emotion engine, data sharing, and follow-up functions. The system grasps the user's health and emotional state in real time, and based on that, conducts appropriate medical interviews, follow-ups, and recommends related products.

[2020] System Overview

[2021] The system consists of three main components:

[2022] 1. Server

[2023] 2. Device (smartphone, etc.)

[2024] 3. Users

[2025] server

[2026] The server's main responsibilities are:

[2027] User registration and initial data setup: The server receives basic information provided by the user (such as name, age, and health status) and stores it in a database, which then creates a health profile for each user.

[2028] Dynamic Question Generation: Based on the speech recognition results and emotion data, the next question is dynamically generated using an AI model. In this process, the most appropriate question is presented to the user.

[2029] Data sharing: Collected voice data, emotional data, health data, etc. will be shared with the user's permission, allowing for more accurate support to be provided to the user.

[2030] Post-consultation follow-up and product recommendations: Based on the user's health and preference data, necessary follow-up and related product recommendations are provided.

[2031] Terminal

[2032] The device (e.g., smartphone) primarily performs the following functions:

[2033] 3D hologram generation and display: At the start of the interview, a 3D hologram is activated and presents questions to the user. The hologram displays the generated questions one after another, ensuring a smooth dialogue.

[2034] Speech and emotion recognition: The device recognizes the user's voice in real time and generates emotion data using an emotion engine. This data is sent to the server and used to generate the next question.

[2035] Feedback display: The voice recognition results and emotional data are sent to the server, and the next question is received from the server and displayed on the hologram. If the emotional data indicates "anxiety" or "tension," the device has the function of displaying feedback to relax the user.

[2036] User

[2037] The user performs the following actions:

[2038] Starting and answering the medical interview: Press the start button on the device and answer the questions from the hologram by voice. For example, in response to the initial question, "How are you feeling lately?", you can answer, "I'm a little tired."

[2039] Daily health data input: Enter your daily health data (body temperature, blood pressure, weight, etc.) into your device and synchronize it with the server, which will update your health profile in real time.

[2040] Collecting emotional data and receiving feedback: The hologram receives feedback based on emotional data collected during the interview and engages in adaptive dialogue. For example, if the user responds, "I'm feeling a little anxious right now," the hologram will provide encouraging words such as, "It seems you're feeling anxious. I recommend you take a break."

[2041] Hardware and software used

[2042] Hardware: Smartphone (iOS or Android device)

[2043] software:

[2044] Speech Recognition: Converts speech to text using the speech_recognition library.

[2045] Emotion Recognition: Analyze emotions from speech and text using third-party emotion recognition libraries.

[2046] Generative AI Models: Use a generative AI model library to generate the next question.

[2047] Specific examples

[2048] For example, if a user enters "I've been feeling more stressed lately" in the default settings, the system will act as follows:

[2049] 1. The user answers verbally, "I've been busy at work lately and it's stressful."

[2050] 2. Speech to text: "Work has been busy and stressful lately."

[2051] 3. Sentiment analysis: Identify "stress."

[2052] 4. The AI ​​model generates the following question: "What situations make you feel particularly stressed?"

[2053] Prompt Sentence Examples

[2054] "Based on the analysis of the user's voice content and emotions, please generate the following appropriate questions. Voice content: 'Work has been busy lately and I'm stressed.' Emotion: 'Stressed.' Next question:"

[2055] This allows the medical interview process to proceed smoothly and allows for specific follow-up and product recommendations tailored to the user's health condition.

[2056] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2057] Step 1:

[2058] User Registration

[2059] 1. The server receives basic information entered by the user (name, age, health status) and stores it in a database.

[2060] Input: Basic information such as name, age, and health status.

[2061] Data processing: The entered information is registered in a database according to the format.

[2062] Output: User profile created.

[2063] Step 2:

[2064] Interview begins

[2065] 1. The user launches the app on their device and presses the button to start the medical interview.

[2066] Input: User operation (pressing the button to start the medical interview).

[2067] Data processing: Set the interview start flag.

[2068] Output: Start of the interview process.

[2069] 2. The device will activate a 3D hologram and display the first question.

[2070] Input: Interview start flag.

[2071] Data processing: Get the first question.

[2072] Output: Hologram of the question.

[2073] Step 3:

[2074] Speech and Emotion Recognition

[2075] 1. The user answers the first question by voice.

[2076] Input: Audio data.

[2077] Data processing: Acquisition of audio data.

[2078] Output: Save audio data.

[2079] 2. The device performs voice recognition and converts it into text data.

[2080] Input: Audio data.

[2081] Data processing: Text conversion using speech recognition (using the speech_recognition library).

[2082] Output: Text data.

[2083] 3. The emotion engine analyzes the voice and text data and generates emotion data.

[2084] Input: Audio and text data.

[2085] Data processing: Sentiment analysis (using emotion recognition libraries).

[2086] Output: Emotion data.

[2087] Step 4:

[2088] Dynamic interview generation

[2089] 1. The server receives the speech recognition results and emotion data and inputs a prompt sentence to the generative AI model to generate the next question.

[2090] Input: Speech recognition results, emotion data.

[2091] Data processing: Prompt sentence generation.

[2092] Output: The prompt statement.

[2093] 2. The server uses a generative AI model to dynamically generate the next question.

[2094] Input: prompt statement.

[2095] Data processing: The AI ​​model generates the next question.

[2096] Output: Next question.

[2097] Step 5:

[2098] Presenting the next question

[2099] 1. The server sends the following question to the terminal:

[2100] Input: Next question.

[2101] Data processing: Sending query data.

[2102] Output: Sends the next question to the terminal.

[2103] 2. The device uses a hologram to display the next question.

[2104] Input: Next question.

[2105] Data processing: Hologram generation and display.

[2106] Output: Hologram of the question.

[2107] Step 6:

[2108] Data recording and sharing

[2109] 1. The server records each response and emotion data in real time.

[2110] Input: Speech recognition results, emotion data, question history.

[2111] Data processing: Saving to database.

[2112] Output: Recorded data.

[2113] 2. The server shares the recorded data.

[2114] Input: Recorded data.

[2115] Data processing: Generating data for sharing.

[2116] Output: Sending shared data.

[2117] Step 7:

[2118] Follow-up and product recommendations

[2119] 1. The server generates a schedule for follow-up visits and preference-based product recommendations.

[2120] Input: Health data, preference data.

[2121] Data processing: Applying schedule generation and product recommendation algorithms.

[2122] Output: Follow-up schedule, product recommendation list.

[2123] 2. The device displays follow-up notifications and product recommendations to the user.

[2124] Input: Follow-up schedule, product recommendation list.

[2125] Data processing: generation of notification and display data.

[2126] Output: Follow-up notification, product recommendation display.

[2127] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2128] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2129] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2130] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2131] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2132] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2133] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2134] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2135] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2136] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2137] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2138] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not lim...

Claims

1. a means for real-time speech recognition; A means for dynamically generating questions based on the recognized speech data; A means for displaying the generated questions as 3D holograms and presenting them to the user; A means of receiving and re-recognizing the user's response by voice, a means for recording the recognition results and the history of the questions; means for sharing the recognition results and recorded data with a medical professional; A means of following up after a medical examination based on health data, and A system including:

2. The system according to claim 1 , further comprising means for integrating and managing health data.

3. The system according to claim 1 , wherein the follow-up means after the consultation includes medication management, insurance guidance, and schedule management.

Images