System

Abstract

To provide a system.SOLUTION: A system includes: means that makes a pre-inquiry; means that prepares a doctor's questions on the basis of a result of the pre-inquiry; means that creates a clinical chart by integrating the doctor's questions and questions from a patient; means that causes the clinical chart to cooperate with the doctor; means that extracts a keyword and a symptom by using natural language processing; and means that prepares the doctor's questions by using a generative AI model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the medical field, there is a need to reduce patient consultation and waiting times, and to reduce the burden on patients and medical staff. However, the current system is not able to fully resolve these issues. [Means for solving the problem]

[0005] The present invention provides a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the doctor's questions and questions from the patient, and a means for sharing the medical record with the doctor. This allows the doctor to propose anticipated symptoms, procedures, treatment times, etc. based on advance confirmation before the patient visit, and to examine the patient based on the suggestions, thereby shortening the time for patients to visit the hospital and waiting, and reducing the burden on patients and medical staff. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The present invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and takes part in a preliminary interview by online automatic transcription with an avatar or by filling out a form. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0029] "Example 2"

[0030] As a specific example, if a patient complains of cold symptoms, the preliminary interview will ask details about the symptoms and when they began. Based on this, the system will generate questions that the doctor is likely to ask at the time of the consultation, such as "What's your fever?" or "Do you have a cough?" It will also accept questions from the patient, such as "What cold medicine should I take?" These questions are integrated into a medical record and shared with the doctor. The doctor will use this as a reference when examining the patient, ensuring efficient medical care.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The patient accesses a dedicated URL.

[0034] Step 2: Conduct a pre-interview with an avatar using online automated transcription or form filling.

[0035] Step 3: Based on the results of the pre-interview, the system generates questions that the doctor is likely to ask during the visit.

[0036] Step 4: Also receive questions from the patient and consolidate these questions to create a medical record.

[0037] Step 5: The created medical record is shared with the doctor, who will use it as a reference when examining the patient.

[0038] "Example 2"

[0039] Step 1: If the patient complains of cold symptoms, the preliminary interview involves asking about details of the symptoms and when they began.

[0040] Step 2: Based on this, the system generates questions that a doctor might ask during a consultation, such as "What's your fever?" or "Do you have a cough?"

[0041] Step 3: Also, answer any questions from the patient, such as "What cold medicine should I take?"

[0042] Step 4: Integrate these questions into the patient's chart and communicate with the doctor.

[0043] Step 5: The doctor will use this as a reference when examining the patient, ensuring efficient medical treatment.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0046] In conventional medical systems, there was a problem that it was time-consuming for medical staff to ask questions before patients visit the clinic, making it difficult to conduct efficient examinations. In addition, it was difficult to accurately grasp the details of the patient's questions and symptoms, which could lead to a decline in the quality of the examination. Furthermore, the doctor could not quickly obtain the information necessary for the examination, which sometimes prolonged the examination time. To solve these problems, a system was needed that would improve the efficiency of the pre-examination and enable doctors to quickly obtain the information necessary for the examination.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for extracting keywords and symptoms using natural language processing, and a means for generating questions for the doctor using a generative AI model. This makes it possible to improve the efficiency of the preliminary interview and allows the doctor to quickly obtain information required for the examination, thereby improving the quality of the examination and shortening the examination time.

[0049] A "pre-medical interview" is a process in which a patient enters their symptoms and condition before visiting a doctor, and medical professionals collect the necessary information before the examination.

[0050] The "doctor's questions" are questions that the doctor asks the patient during the consultation, which are generated based on the results of the preliminary interview.

[0051] A "medical record" is a medical document that records a patient's symptoms, examination results, treatment details, etc.

[0052] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[0053] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new information or data.

[0054] "Keywords" are important words or phrases within text data that are extracted using natural language processing.

[0055] A "symptom" is any physical or mental abnormality or discomfort experienced by a patient.

[0056] An "avatar" is a virtual character that allows users to interact online.

[0057] "Online automatic transcription" is a technology that converts audio into text in real time.

[0058] "Form entry" is a method in which a user manually enters text data into fields on a web page.

[0059] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0060] Hardware and Software Used

[0061] This system is realized using the following hardware and software.

[0062] Hardware: Servers, terminals (smartphones, PCs, etc.)

[0063] Software: Python, natural language processing libraries (NLTK, spaCy), database manipulation libraries (SQLAlchemy), web interface framework (Flask), generative AI models (e.g., GPT-3® by OpenAI®)

[0064] Program Processing

[0065] The user (patient) accesses a dedicated URL. Once accessed, an avatar appears on the device and the preliminary interview begins through online automatic transcription or form input. Specifically, the text data entered by the user is sent from the device to the server.

[0066] The server analyzes the received text data using natural language processing libraries (NLTK and spaCy) to extract keywords and symptoms. Based on the results of this analysis, a generative AI model (for example, OpenAI's GPT-3) is used to generate questions that the doctor may ask during the consultation.

[0067] In addition, the terminal accepts additional questions from the user, and the questions entered on the terminal are sent back to the server, which then consolidates these questions and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner).

[0068] The created medical records are sent from the server to the doctor's terminal, and the doctor uses the records as a reference when examining the patient.

[0069] Examples of concrete examples and prompts

[0070] As a concrete example, consider the case where a user inputs, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." This text data is sent from the device to the server, and the server uses a natural language processing library to extract keywords such as "headache" and "when I wake up in the morning."

[0071] Next, feed the generative AI model the following prompt:

[0072] User symptoms: Headache, severe pain upon waking in the morning

[0073] Generate questions your doctor may have during a consultation.

[0074] A generative AI model might generate questions like:

[0075] 1. How often do your headaches occur?

[0076] 2. How severe is your headache?

[0077] 3. Do you have any other symptoms besides headaches?

[0078] These questions and any additional questions from the user are integrated to create a medical record in a format suitable for the electronic medical record system. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0079] In this way, the system acts as a substitute for medical professionals to support patients.

[0080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0081] Step 1:

[0082] The user accesses a special URL.

[0083] The user accesses the dedicated URL provided by the medical institution. For example, the user inputs the URL using a browser on a smartphone or computer. Input: Dedicated URL. Output: Display of the web page.

[0084] Step 2:

[0085] The terminal will display an avatar and begin the preliminary interview.

[0086] The terminal displays a web page based on the accessed URL. This web page displays an avatar and prompts the user to start the pre-interview. Input: Dedicated URL. Output: Display of avatar and start of pre-interview.

[0087] Step 3:

[0088] The user inputs text data.

[0089] The user follows the avatar's instructions to input their symptoms and condition in text format. For example, the user might input, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." Input: User's symptoms and condition. Output: Text data.

[0090] Step 4:

[0091] The terminal transmits the input data to the server.

[0092] The terminal sends the text data entered by the user to the server. This transmission is done securely using the HTTPS protocol. Input: Text data. Output: Data transmission to the server.

[0093] Step 5:

[0094] The server performs natural language processing to extract keywords and symptoms.

[0095] The server analyzes the received text data using a natural language processing library (NLTK or spaCy). For example, it extracts keywords such as "headache" and "when waking up in the morning." Input: Text data. Output: Extracted keywords and symptoms.

[0096] Step 6:

[0097] The server uses the generative AI model to generate questions for the doctor.

[0098] The server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-3) based on the extracted keywords, and generates questions that a doctor may expect to ask when visiting the patient. For example, a prompt sentence such as "User's symptoms: headache, severe pain when waking up in the morning. Please generate questions that a doctor may expect to ask when visiting the patient" can be used. Input: Extraction results of keywords and symptoms. Output: Questions asked by the doctor.

[0099] Step 7:

[0100] The user enters a follow-up question.

[0101] The user inputs additional questions or information for the generated question. For example, the user answers "About three times a week" to the question "How often do you get headaches?". Input: Doctor's question. Output: Additional questions or information.

[0102] Step 8:

[0103] The terminal sends a follow-up question to the server.

[0104] The terminal sends any additional questions or information entered by the user back to the server. Input: Additional questions or information. Output: Sending data to the server.

[0105] Step 9:

[0106] The server consolidates the questions and creates a medical record.

[0107] The server merges the initial interview data and the follow-up question data and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner). Input: Initial interview data, follow-up question data. Output: Medical record.

[0108] Step 10:

[0109] The server links the medical records to the doctor's terminal.

[0110] The server links the created medical records to the doctor's terminal. The doctor can view the medical records through a dedicated medical system. Input: Medical records. Output: Linking medical records to the doctor's terminal.

[0111] Step 11:

[0112] The doctor will examine the patient using the medical records.

[0113] The doctor will consult the linked medical records when examining the patient. The medical records contain detailed information about the user's symptoms and the results of the preliminary interview, allowing the examination to proceed smoothly. Input: Medical records. Output: Examination results.

[0114] (Application example 1)

[0115] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, and doctors often lacked the information they needed to examine patients. In addition, there was no way to understand the patient's symptoms or questions in advance when consulting about medicines at pharmacies or drug stores, making it difficult to respond efficiently. This resulted in longer waiting times for patients and increased burdens on medical professionals.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0118] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to a doctor, a means for supporting preliminary interviews and consultations about medicines at pharmacies and drug stores, and a means for generating questions using a generative AI model. This allows a patient to collect necessary information before visiting a medical institution or a pharmacy, enabling efficient examinations and consultations.

[0119] A "pre-medication interview" is a process in which medical professionals collect necessary information in advance by having patients enter their symptoms and questions online before visiting a medical institution or pharmacy.

[0120] The "means for generating questions for a doctor" is a system that automatically generates questions that a doctor should ask during a consultation based on the results of a preliminary interview.

[0121] The "means of creating a medical record" refers to a system that integrates the results of the preliminary interview, questions from the doctor, and questions from the patient to create a medical record that compiles all the information necessary for the examination.

[0122] "Means of sharing medical records with doctors" refers to a system that provides the created medical records to doctors so that they can refer to them during consultations.

[0123] The "means of supporting advance medical interviews and medication consultations at pharmacies and drug stores" is a system that allows patients to input their symptoms and questions in advance at pharmacies and drug stores, enabling pharmacists to respond efficiently.

[0124] A "generative AI model" is an algorithm that uses artificial intelligence to generate appropriate questions and answers based on input data.

[0125] The "means for generating questions" refers to a system that uses a generative AI model to automatically generate appropriate questions based on the results of the preliminary interview.

[0126] The system for implementing this invention is configured as follows. First, a user accesses a dedicated URL using a terminal such as a smartphone or a personal computer. After accessing the URL, a preliminary interview is conducted by online automatic transcription with an avatar or by filling out a form. The results of this preliminary interview are sent to the server.

[0127] Based on the results of the preliminary interview, the server generates questions for the doctor using a generative AI model that uses advanced natural language processing algorithms, such as OpenAI's GPT-3. The generated questions are stored on the server.

[0128] Next, the server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. This medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0129] In addition, the system also has a function to support pre-medication and drug consultations at pharmacies and drug stores. Before going to a pharmacy or drug store, the user performs a pre-medication interview using a similar procedure, and the generative AI model generates questions for the pharmacist based on the results. This allows the pharmacist to efficiently respond to patient consultations.

[0130] For example, if a user enters, "I have a headache and a sore throat," the generative AI model generates questions such as, "When did the headache start?" and "How bad is your throat?" These questions are integrated into the medical record and provided to doctors and pharmacists.

[0131] An example of a prompt is as follows:

[0132] "Generate questions the pharmacist might have based on the following symptoms: I have a headache and a sore throat."

[0133] In this way, a system can be realized that enables efficient medical examinations and consultations based on the results of the preliminary interview.

[0134] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0135] Step 1:

[0136] The user accesses a dedicated URL using a device such as a smartphone or PC. The user completes a preliminary interview by using an avatar to automatically transcribe online or by filling out a form. The data entered is the user's symptoms and questions. This data is then sent from the device to the server.

[0137] Step 2:

[0138] The server uses a generative AI model to generate questions for the doctor based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated doctor's question.

[0139] Step 3:

[0140] The server creates a medical record by integrating the generated doctor's questions and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[0141] Step 4:

[0142] The server transmits the created medical record to the doctor. The medical record is provided to the doctor so that the doctor can refer to it when examining the patient. The input is the integrated medical record, and the output is the medical record provided to the doctor.

[0143] Step 5:

[0144] Before a user goes to a pharmacy or drugstore, the user performs a pre-medical interview using a similar procedure. The user inputs symptoms and questions using a terminal and sends them to the server. The input data is the user's symptoms and questions.

[0145] Step 6:

[0146] The server uses a generative AI model to generate questions for the pharmacist based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated questions for the pharmacist.

[0147] Step 7:

[0148] The server creates a medical record by integrating the generated questions from the pharmacist and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[0149] Step 8:

[0150] The server communicates the created chart to the pharmacist. The chart is provided to the pharmacist so that the pharmacist can refer to it during consultation. The input is the integrated chart, and the output is the chart provided to the pharmacist.

[0151] Example 2

[0152] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] In conventional medical systems, the process from the patient's preliminary interview to the doctor's examination was not carried out efficiently, which often resulted in long consultation times. In addition, it was difficult to generate appropriate questions based on the patient's symptoms, and doctors were sometimes unable to fully grasp the necessary information before the examination. This resulted in poor efficiency in medical treatment and poor patient satisfaction.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0155] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor using a generative AI model based on the results of the preliminary interview, a means for creating a medical record by integrating the doctor's questions and questions from the patient, and a means for sharing the medical record with the doctor. This allows appropriate questions based on the patient's symptoms to be automatically generated, enabling the doctor to grasp the necessary information before the consultation. This improves the efficiency of medical care and also improves patient satisfaction.

[0156] A "pre-medical interview" is the process in which a patient provides information about their symptoms, medical history, etc. before an appointment.

[0157] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate outputs for a specific task.

[0158] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[0159] "Patient Questions" are questions that patients ask their physicians about their symptoms or treatment.

[0160] A "medical record" is a document that contains a patient's medical records, including symptoms, diagnoses, and treatment details.

[0161] "Collaboration" refers to different systems and processes sharing information and working together.

[0162] A "web page" is a page of information accessible on the Internet.

[0163] An "avatar" is a character or icon that represents a user online.

[0164] "Online automatic transcription" is a technology that converts audio into text in real time.

[0165] "Form filling" is the process by which a user enters information into fields on a web page.

[0166] "Medical Condition" refers to a patient's health condition or symptoms.

[0167] "Procedure" refers to the medical treatment or care that a doctor provides to a patient.

[0168] "Treatment time" refers to the time it takes a doctor to treat a patient.

[0169] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0170] This invention is a system that performs a preliminary interview, generates questions for the doctor using a generative AI model, creates a medical record by integrating the questions from the patient, and communicates with the doctor. A specific embodiment of this system is described below.

[0171] Hardware and Software Used

[0172] Hardware: Servers, terminals (PCs, tablets, smartphones, etc.)

[0173] Software: Generative AI models (e.g., GPT-4®), database management systems (e.g., MySQL®), web application frameworks (e.g., Django)

[0174] System Configuration

[0175] 1. Server: The server is responsible for generating questions to conduct a pre-medical interview. It uses a generative AI model (e.g. GPT-4) to generate questions based on the patient's symptoms. The server also receives answers from the user (patient) and stores them in a database. The server also integrates the collected information, creates a medical record, and notifies the doctor.

[0176] 2. Terminal: The terminal provides an interface for the user (patient) to conduct a preliminary interview. The terminal displays the questions received from the server to the user, collects the user's answers, and sends them to the server.

[0177] 3. User: The user (patient) uses the terminal to conduct a preliminary interview. The user answers the questions displayed on the terminal and, if necessary, enters his / her own questions.

[0178] Examples

[0179] 1. Question generation for pre-medical interview: The server uses a generative AI model (e.g., GPT-4) to generate questions for pre-medical interviews that are necessary when a patient complains of cold symptoms. For example, questions such as "When did your symptoms start?" and "What symptoms are you experiencing?" are generated. The generated questions are stored in a database.

[0180] 2. Displaying questions on the terminal: The terminal retrieves the questions from the pre-medical interview from the server and displays them to the user. For example, questions such as "What is your fever?" and "Do you have a cough?" are displayed on the terminal screen.

[0181] 3. Collect user answers: The user answers the questions displayed on the terminal. For example, they input information such as "My fever is 37.5 degrees" or "I have a cough." The user can also input their own questions. For example, they input questions such as "What cold medicine should I take?"

[0182] 4. Sending and storing answers: The terminal sends the user's answers to the server. The server stores the received answers in a database. This allows the centralized management of patient information.

[0183] 5. Creating a medical record: The server consolidates the collected information and creates a medical record. The medical record includes details of the patient's symptoms, when they first appeared, questions that the doctor anticipates, and questions from the patient. The created medical record is stored in a database.

[0184] 6. Medical record sharing with doctor: The server sends the created medical record to the doctor's terminal. The doctor performs the examination using the received medical record as a reference. For example, by looking at the medical record, the doctor can know in advance information such as "The patient has a fever of 37.5 degrees and is coughing." This allows for more efficient medical treatment.

[0185] Examples of prompt statements

[0186] "A patient is complaining of cold symptoms. Using the information below, generate questions that a doctor might ask during a consultation. Please include details of the patient's symptoms and when they began."

[0187] This system allows doctors to understand patients' symptoms in advance, improving the efficiency of medical treatment and increasing patient satisfaction.

[0188] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0189] Step 1:

[0190] The server generates the pre-screening questions.

[0191] Input: Information about the patient's initial symptoms (e.g. cold symptoms)

[0192] Data processing: Input a prompt sentence into a generative AI model (e.g. GPT-4) to generate appropriate questions.

[0193] Output: Generated questions (e.g., "When did your symptoms start?", "What symptoms are you experiencing?")

[0194] Specific operation: The server inputs a prompt sentence into the generative AI model: "The patient is complaining of cold symptoms. Based on the information below, please generate questions that a doctor may expect to ask when visiting the patient. Please also include details of the patient's symptoms and when they began." and stores the generated questions in a database.

[0195] Step 2:

[0196] The terminal obtains the questions for the preliminary medical examination from the server and displays them to the user.

[0197] Input: A generated query from the server

[0198] Data processing: Converting the question into a format for display to the user.

[0199] Output: The question that is displayed to the user (e.g. "What's your fever?", "Do you have a cough?")

[0200] Specific operation: The terminal displays the questions obtained from the server on the screen and formats them in a way that makes it easy for the user to answer.

[0201] Step 3:

[0202] The user answers the questions displayed on the terminal.

[0203] Input: the question displayed on the terminal

[0204] Data processing: Enter the user's answers into the input form.

[0205] Output: User's answer (e.g. "My temperature is 37.5 degrees" or "I have a cough")

[0206] Specific operation: The user inputs text in response to the questions displayed on the terminal, and if necessary, inputs his or her own question.

[0207] Step 4:

[0208] The terminal transmits the user's answer to the server.

[0209] Input: User's answer

[0210] Data processing: The response data is converted into a format for sending to the server.

[0211] Output: Answer data sent to the server

[0212] Specific operation: The terminal transmits the answers entered by the user to the server in real time.

[0213] Step 5:

[0214] The server stores the received answers in a database.

[0215] Input: Answer data sent from the device

[0216] Data processing: Convert the response data into a format that can be stored in the database.

[0217] Output: Answer data stored in a database

[0218] Specific operations: The server stores the received response data in a database for subsequent processing.

[0219] Step 6:

[0220] The server consolidates the collected information and creates a medical record.

[0221] Input: Answer data stored in the database

[0222] Data processing: Integrate the response data and create a medical record.

[0223] Output: Created medical record

[0224] Specific operation: The server creates a medical record by integrating the patient's symptoms and questions based on the information stored in the database.

[0225] Step 7:

[0226] The server transmits the created chart to the doctor's terminal.

[0227] Input: Created medical record

[0228] Data processing: Converting medical record data into a format for sending to the doctor's terminal.

[0229] Output: Medical record sent to doctor's terminal

[0230] Specific operation: The server sends the created medical record to the doctor's terminal so that the doctor can review the information before the examination.

[0231] Step 8:

[0232] The doctor will use the medical records he or she receives to conduct the examination.

[0233] Input: Medical record sent to doctor's terminal

[0234] Data processing: Check the contents of the medical records and extract the information necessary for the examination.

[0235] Output: Examination results and treatment plan

[0236] Specific actions: The doctor reviews the patient's medical records to understand the patient's symptoms and questions, and conducts an efficient examination.

[0237] (Application example 2)

[0238] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, which often led to long consultation times for doctors. In addition, even in brick-and-mortar stores, there was no way for customers to ask questions before visiting the store, which led to inefficient responses from store staff.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0241] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for receiving symptoms reported by a customer as an input and generating questions to be asked by store staff, a means for generating questions for the customer to ask the store staff, a means for integrating the generated questions to create a medical record, and a means for outputting the created medical record. This enables the efficiency of medical examinations at medical institutions and the efficiency of customer service at physical stores.

[0242] A "pre-medical interview" is a process in which patients or customers enter their symptoms and requests in advance before visiting a medical institution or physical store.

[0243] "Doctor's questions" are questions that the doctor should ask the patient based on the results of the preliminary interview.

[0244] "Patient questions" are questions that patients want to ask their doctors.

[0245] A "medical record" is a record that integrates the symptoms and questions of patients or customers, as well as questions from doctors and store staff.

[0246] "Symptoms reported by customers" refers to the physical discomfort and requests felt by customers who visit physical stores.

[0247] "Questions that store staff should ask" are questions that store staff should ask customers based on the customer's symptoms and requests.

[0248] "Questions that customers should ask store staff" are questions that customers would like to ask store staff.

[0249] A "generated question" is a question that is automatically generated based on the results of a preliminary interview.

[0250] The "means of creating a medical record" refers to the process of creating a medical record by integrating the results of the preliminary interview and the questions generated.

[0251] "Means for outputting medical records" refers to the process of providing the created medical records to doctors and store staff.

[0252] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for receiving as input symptoms reported by a customer and generating questions to be asked by store staff, means for generating questions for the customer to ask the store staff, means for integrating the generated questions to create a medical record, and means for outputting the created medical record.

[0253] The server receives the results of the preliminary interview and uses a generative AI model to generate questions for the doctor and store staff. Specifically, it uses OpenAI's API to generate appropriate questions based on the results of the preliminary interview. The generated questions are processed using a Python program and integrated into the medical record.

[0254] The terminal (smartphone or tablet) provides an interface for the user to conduct a preliminary medical interview. The user accesses a dedicated URL and conducts the preliminary medical interview by online automatic transcription with an avatar or by filling out a form. The user's symptoms and requests are then sent to the server.

[0255] When a user inputs the symptoms they are complaining about, the server uses a generative AI model to generate questions that store staff should ask, or questions that customers should ask store staff. The generated questions are integrated into a medical record and provided to store staff.

[0256] As a specific example, the prompt sentence when a customer complains of cold symptoms is shown below.

[0257] "A customer is complaining of the following symptoms: Cold symptoms. Generate questions for store staff to ask."

[0258] When this prompt is executed, the server will generate a question like this:

[0259] "What's your fever? Do you have a cough?"

[0260] It also generates questions for customers to ask store staff.

[0261] "What cold medicine should I take?"

[0262] This will enable more efficient medical examinations at medical institutions and more efficient customer service at physical stores.

[0263] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0264] Step 1:

[0265] The user accesses a dedicated URL using a device (smartphone or tablet). The user completes a preliminary interview by using an avatar to automatically transcribe text online or by filling out a form. The input data is the user's symptoms and requests. The output data is sent to the server as the results of the preliminary interview.

[0266] Step 2:

[0267] The server analyzes the received results of the pre-interview. The input data is information about the user's symptoms and requests. The server uses a generative AI model (e.g., OpenAI's API) to generate questions for the doctor and questions for the store staff based on the results of the pre-interview. The output data are the generated questions.

[0268] Step 3:

[0269] The server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. The input data are the generated questions and the results of the preliminary interview. The server integrates these data and configures them as a medical record. The output data is the created medical record.

[0270] Step 4:

[0271] The server shares the created medical record with the doctor. The input data is the created medical record. The server sends the medical record to the doctor's terminal so that the doctor can refer to it when examining the patient. The output data is the medical record provided to the doctor.

[0272] Step 5:

[0273] When a user inputs the symptoms they are experiencing, the server uses a generative AI model to generate questions for store staff to ask. The input data is information about the user's symptoms. The server uses a generative AI model to generate appropriate questions. The output data is the questions that store staff should ask.

[0274] Step 6:

[0275] The server generates questions that customers should ask store staff. The input data is information about the user's symptoms. The server uses a generative AI model to generate questions that customers should ask. The output data is questions that customers should ask store staff.

[0276] Step 7:

[0277] The server integrates the generated questions to create a medical record. The input data are the generated questions. The server integrates these questions and configures them as a medical record. The output data is the created medical record.

[0278] Step 8:

[0279] The server outputs the created medical record. The input data is the created medical record. The server provides the medical record to the store staff so that they can refer to it when dealing with customers. The output data is the medical record provided to the store staff.

[0280] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0281] "Example 1"

[0282] In one embodiment of the present invention, an emotion engine is introduced. The emotion engine recognizes emotions from the user's voice, facial expression, or text input. Specifically, when the user conducts a pre-interview, the emotion is estimated from the tone, strength, and speed of the user's voice. If the user is using a webcam, the emotion is also estimated from facial expression. Furthermore, if the user inputs text, the emotion is estimated from the content and expression of the text.

[0283] "Example 2"

[0284] As a concrete example, consider a case where a user complains of cold symptoms. When the user says "My head hurts," emotions such as "pain" or "suffering" are inferred from the voice. Also, if the user is using a webcam, emotions such as "pain" or "discomfort" are inferred from the facial expression. Furthermore, if the user inputs the text "My head hurts," emotions such as "pain" or "suffering" are also inferred from that text. Based on the recognition results of these emotion engines, the doctor's questions are adjusted. For example, a more specific question such as "How bad is your headache?" is generated. This allows the doctor to grasp the user's condition more accurately and perform an appropriate examination.

[0285] The process flow of each embodiment will be described below.

[0286] "Example 1"

[0287] Step 1: The user accesses the URL for the pre-consultation questionnaire.

[0288] Step 2: The user answers the medical questions, while the emotion engine analyzes the user's voice, facial expressions, or text input.

[0289] Step 3: The emotion engine recognizes the user's emotions and shares the results with the doctor.

[0290] "Example 2"

[0291] Step 1: When a user reports cold symptoms, their voice, facial expressions, or text input are analyzed by the emotion engine.

[0292] Step 2: The emotion engine recognizes the user's emotions, such as "pain" or "suffering."

[0293] Step 3: Based on the emotion engine's recognition results, the doctor's question is adjusted, for example, a more specific question is generated, such as "How bad is your headache?"

[0294] Example 1

[0295] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0296] In the conventional medical system, when a patient conducts a pre-consultation before visiting a doctor, medical staff must directly respond to the patient, which takes time and effort. It is also difficult to accurately grasp the patient's emotions, and there are cases where the information required for an appropriate examination is insufficient. Furthermore, doctors are unable to prepare questions to ask at the time of the consultation in advance, which can reduce the efficiency of examinations.

[0297] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0298] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from the user's voice, facial expression, or text input, and a means for generating questions for the doctor using a generative AI model. This allows the patient to efficiently conduct a preliminary interview before a medical examination, and the doctor to prepare appropriate questions in advance. In addition, by accurately grasping the patient's emotions, information for a more appropriate medical examination can be provided.

[0299] A "pre-medical interview" is an interview conducted by a patient before visiting a doctor, and is the process of collecting information such as the patient's symptoms and medical history.

[0300] "Doctor's questions" are questions that a doctor asks a patient, and are intended to understand the patient's symptoms and condition in detail.

[0301] A "medical record" is a document that compiles a patient's medical records and serves as a reference for doctors when examining and treating patients.

[0302] "Means for recognizing emotions" refers to technology or devices for inferring emotions from a user's voice, facial expression, or text input.

[0303] A "generative AI model" is a model that uses artificial intelligence to generate text and questions, and examples include GPT-3.

[0304] An "avatar" is a virtual character that interacts with a user and assists in online communication.

[0305] "Online automatic transcription" is a technology that converts voice input into text, automatically recording what a user says as text.

[0306] "Form entry" refers to a method in which a user enters information into a web form, and is used during a pre-consultation interview.

[0307] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0308] Hardware and Software Used

[0309] The system's program is developed using Python. The main libraries used are NLTK for natural language processing, TextBlob for sentiment analysis, SpeechRecognition for voice recognition, and OpenCV for facial expression recognition. Advanced natural language generation models such as GPT-3 are used as generative AI models.

[0310] Program processing explanation

[0311] When a user accesses a dedicated URL, the server displays an avatar to the user, which then interacts with the user to ask questions. The user can answer the questions by voice input, text input, or form input.

[0312] The server receives the user's input data and performs the following operations:

[0313] For speech input: Use the SpeechRecognition library to convert speech to text.

[0314] For text input: Process it as text data as is.

[0315] For form filling: Collect data for each field in the form.

[0316] The server then analyzes the user's emotions using an emotion engine, which estimates emotions in the following way:

[0317] For voice input: Analyzes the tone, volume and speaking rate of the voice.

[0318] For facial expression input: OpenCV is used to analyze the video captured by the webcam and estimate emotions from facial expressions.

[0319] For text input: Use TextBlob to infer sentiment from the text content and expression.

[0320] Based on this data, the server generates questions that the doctor is likely to ask during the consultation. It uses a generative AI model (e.g., GPT-3) to generate questions based on the user's pre-medical interview results. It also accepts questions from the user and integrates these questions to create a medical record.

[0321] The created chart is sent from the server to the doctor, who then uses it as a reference when examining the patient.

[0322] Examples of concrete examples and prompts

[0323] As a specific example, consider a case where a user complains of a headache and inputs, by voice, "I've been having a really bad headache lately."

[0324] The server does the following:

[0325] 1. Convert speech to text using the SpeechRecognition library.

[0326] 2. Use TextBlob to infer the user's emotion (e.g., distress or anxiety) from the text "I have a bad headache."

[0327] 3. Use GPT-3 to generate questions a doctor might ask during a visit, such as "When did your headache start?" or "Where does it hurt?"

[0328] Example prompt:

[0329] If a user types, "I've been having really bad headaches lately," generate questions a doctor might ask during a visit.

[0330] In this way, the system acts as a substitute for medical professionals to support patients.

[0331] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0332] Step 1:

[0333] The user accesses a special URL.

[0334] Input: User's access request

[0335] Output: Login screen displayed

[0336] Specific operation: A user opens a browser and accesses a dedicated URL. The server receives the user's access request and displays a login screen.

[0337] Step 2:

[0338] The user enters login information and is authenticated.

[0339] Input: User ID and password

[0340] Output: Authentication result (success or failure)

[0341] Specific operation: The user enters the ID and password and clicks the "Login" button. The server accesses the database and verifies the entered ID and password. If authentication is successful, an avatar is displayed to the user.

[0342] Step 3:

[0343] The user conducts a preliminary interview through dialogue with an avatar.

[0344] Input: User voice input, text input, or form input

[0345] Output: Pre-interview data (text format)

[0346] Specific operation: The user speaks to the avatar, saying, "I've been having a really bad headache lately." The server uses the SpeechRecognition library to convert the speech to text, and obtains the text data, "I've been having a really bad headache lately."

[0347] Step 4:

[0348] The server uses an emotion engine to analyze the user's emotions.

[0349] Input: Pre-medical interview data (text format)

[0350] Output: Sentiment analysis results (e.g., distress, anxiety)

[0351] Specific operation: The server uses TextBlob to analyze the user's emotions from the text "I've been having a lot of headaches lately." For example, it estimates emotions such as pain and anxiety.

[0352] Step 5:

[0353] The server uses the generative AI model to generate questions for the doctor.

[0354] Input: Pre-interview data and sentiment analysis results

[0355] Output: Doctor's questions (text format)

[0356] Specific operation: The server inputs a prompt sentence to the generative AI model (GPT-3). Example of a prompt sentence: "If a user inputs, 'I've been having a lot of headaches lately,' please generate a question that a doctor would ask when visiting the doctor." The server takes the generated question and generates a question such as, "When did your headache start?" or "Where does it hurt?"

[0357] Step 6:

[0358] The server creates a medical record by integrating the results of the preliminary interview with the generated doctor's questions.

[0359] Input: Pre-medication data and doctor's questions

[0360] Output: Medical record (text format)

[0361] Specific operation: The server integrates the user's preliminary interview results with the generated doctor's questions and creates a medical record.

[0362] Step 7:

[0363] The medical records created by the server are shared with the doctor.

[0364] Input: Medical record (text format)

[0365] Output: Send medical record to doctor

[0366] Specific operation: The server sends the created chart to the doctor. The doctor uses the chart as a reference when examining the patient.

[0367] (Application example 1)

[0368] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] In conventional medical systems, the patient's pre-consultation and medical record creation were done manually, which placed a heavy burden on medical staff and was inefficient. In addition, in brick-and-mortar stores, the response to customer questions and requests was insufficient, which led to a decrease in customer satisfaction. Furthermore, there was also the issue of it being difficult to appropriately recognize and respond to customer emotions.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0371] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, a means for accepting questions and requests from customers in a physical store, recognizing emotions, and providing appropriate responses, and a means for recording questions and requests from customers and linking them to store staff. This reduces the burden on medical staff and enables efficient creation of medical records, while also enabling proper recognition of customer emotions in a physical store and providing highly satisfying responses.

[0372] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, in order to understand the patient's symptoms and medical history in advance.

[0373] "Doctor's questions" are questions that a doctor asks a patient to confirm in detail the patient's symptoms and condition.

[0374] A "medical chart" is a compilation of a patient's medical records, which serve as a reference for doctors when examining and treating patients.

[0375] An "emotion engine" is a system for recognizing emotions from a user's voice, facial expressions, or text input.

[0376] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[0377] "Customer questions and requests" are questions and requests made by customers to store staff or support robots in physical stores.

[0378] "Store staff" refers to employees who work in physical stores and provide products and services to customers.

[0379] "Online automatic transcription" is a technology that converts speech into text, automatically recording what a user says as text.

[0380] "Form input" refers to a method in which a user inputs information into an input field on a web page, and is used for things like advance interviews and questionnaires.

[0381] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, means for accepting customer questions and requests in a physical store, recognizing emotions and responding appropriately, and means for recording customer questions and requests and linking them to store staff.

[0382] System Programming

[0383] The system program performs the following processing.

[0384] 1. Pre-medication:

[0385] The server will ask the user a preliminary interview via online automatic transcription with an avatar or through form entry when the user accesses a dedicated URL. The information entered by the user will be stored in a database.

[0386] 2. Doctor Question Generation:

[0387] The server generates questions that are expected to be asked by the doctor at the time of the visit based on the results of the preliminary interview. The generated questions are added to the user's medical record.

[0388] 3. Creation and collaboration of medical records:

[0389] The server creates a medical record by integrating the generated doctor's questions with the user's questions. The created medical record is shared with the doctor and used as a reference during consultations.

[0390] 4. Emotion recognition:

[0391] The server uses an emotion engine to recognize emotions from the user's voice, facial expressions, or text input. For emotion recognition, it uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model).

[0392] 5. Customer service in physical stores:

[0393] In a brick-and-mortar store, the server will receive customer questions and requests, recognize emotions, and respond appropriately. Customer questions and requests are recorded and communicated to store staff.

[0394] Hardware and Software Used

[0395] Hardware:

[0396] Webcam: Used to capture the customer's face.

[0397] Microphone: Used to capture the customer's voice.

[0398] software:

[0399] OpenCV: A library for face recognition.

[0400] SpeechRecognition: A library for speech recognition.

[0401] Transformers: Generative AI models for emotion recognition (using the Hugging Face pipeline).

[0402] Examples

[0403] For example, if a customer in a physical store asks a robot, "I want to know more about this product," the server will recognize the customer's emotions from their voice and facial expressions and respond appropriately. The customer's questions and requests are recorded and shared with the store staff.

[0404] Examples of prompt statements

[0405] "Estimate how your customers feel when they say, 'I want to know more about this product.'"

[0406] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0407] Step 1:

[0408] The server performs a preliminary interview with the user who accesses a dedicated URL through online automatic transcription with an avatar or form input. The information entered by the user is stored in a database. The input is information about the user's symptoms and medical history, and the output is the interview results stored in the database.

[0409] Step 2:

[0410] The server generates questions that are expected to be asked by the doctor at the time of the consultation based on the results of the preliminary interview. The generated questions are added to the user's medical record. The input is the results of the preliminary interview, and the output is the generated questions from the doctor. For data processing, an algorithm is used to analyze the interview results and generate appropriate questions.

[0411] Step 3:

[0412] The server creates a medical record by integrating the generated doctor's questions and the user's questions. The created medical record is shared with the doctor and used as a reference during consultations. The input is the doctor's questions and the user's questions, and the output is the integrated medical record. Data calculations include integrating the questions and formatting the medical record.

[0413] Step 4:

[0414] The server uses an emotion engine to recognize emotions from the user's voice, facial expression, or text input. Emotion recognition uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model). The input is the user's voice, facial expression, or text, and the output is the recognized emotion. Data processing involves analysis of voice and image data.

[0415] Step 5:

[0416] In a physical store, the server receives customer questions and requests, recognizes emotions, and responds appropriately. Customer questions and requests are recorded and communicated to store staff. The input is the customer's question or request, and the output is the recorded data and the response. Data calculations are performed to analyze the question or request and generate an appropriate response.

[0417] Step 6:

[0418] The server records the customer's questions and requests and shares them with the store staff. The input is the customer's questions and requests, and the output is the information shared with the store staff. Data processing involves recording and sharing of questions and requests.

[0419] Example 2

[0420] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] In conventional medical systems, doctors were not sufficiently prepared to ask appropriate questions based on the patient's pre-interview information, which led to problems with the efficiency of medical examinations. In addition, questions were not adjusted to take into account the patient's feelings, making it difficult to accurately grasp the patient's condition. This resulted in a decline in the quality of medical care and a decrease in patient satisfaction.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0423] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for estimating emotions from audio input and video input, and a means for adjusting the questions of the doctor based on the emotions. This makes it possible to generate appropriate questions based on the patient's preliminary interview information and adjust the questions taking the patient's emotions into consideration.

[0424] A "pre-medical interview" is a process in which a patient inputs information about their symptoms, the time of onset, etc., before their appointment.

[0425] The "doctor's questions" are questions that the doctor asks the patient, which are generated based on the results of the preliminary interview.

[0426] A "medical chart" is a medical record that is created by integrating information from the patient's prior medical interview, questions from the doctor, and questions from the patient.

[0427] The "means for estimating emotions" refers to technology for analyzing and estimating a patient's emotions from audio and video input.

[0428] The "means for adjusting questions" refers to a technique for appropriately changing the content of questions posed to a doctor based on the estimated emotions.

[0429] "Web Page" means an information page accessible via the Internet that provides an interface for conducting a pre-consultation interview.

[0430] "Online automatic transcription" is a technology that converts voice input into text and is used as part of the pre-interview.

[0431] "Form entry" is a method by which a user enters information into fields on a web page.

[0432] A "generative AI model" is an algorithm that uses artificial intelligence to perform natural language processing and generate appropriate questions and answers.

[0433] The present invention is a system that generates questions for a doctor based on information about a patient's advance medical interview and adjusts the questions in consideration of the patient's feelings. A specific embodiment of this system will be described below.

[0434] Hardware and Software Used

[0435] Hardware

[0436] Server: Use a server machine with a powerful processor and sufficient memory.

[0437] Terminal: A device such as a PC or smartphone is used for the user to input pre-interview information.

[0438] software

[0439] Generative AI models: Using GPT-4 as an example, for natural language processing.

[0440] Database: A database is used to store pre-interview information, generated questions, and medical records.

[0441] Emotion Engine: Uses software to infer emotions from audio and video inputs.

[0442] System processing flow

[0443] 1. Enter your pre-medical interview information

[0444] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms, such as when they first began to show symptoms, etc. For example, the user inputs information such as "I have a headache" or "I've had a fever since yesterday."

[0445] 2. Saving pre-interview information

[0446] The server receives the pre-interview information sent by the user and stores it in a database for use in subsequent processing.

[0447] 3. Doctor Question Generation

[0448] The server uses the generative AI model to generate questions for the doctor to ask during the consultation based on the pre-medical interview information. For example, if a user enters "I have a headache," the server generates questions such as "How bad is your headache?" and "What's your fever?"

[0449] 4. Accepting questions from patients

[0450] A user uses a terminal to input a question to a doctor, such as "What kind of cold medicine should I take?" The server receives these questions and stores them in a database.

[0451] 5. Creating a medical record

[0452] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the user's questions. The medical record includes the user's symptoms, the doctor's questions, and the user's questions. This medical record is shared with the doctor.

[0453] 6. Using the Emotion Engine

[0454] If the user uses voice input or a webcam, the server uses an emotion engine to infer the user's emotions. For example, when the user says "I have a headache," the server infers emotions such as "pain" and "suffering" from the voice, and emotions such as "distress" and "discomfort" from the webcam footage. Based on these emotions, the doctor's questions are adjusted.

[0455] 7. Information sharing with doctors

[0456] The server then shares the created medical records and the results of the emotion engine with the doctor. The doctor will use these as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical records, such as "How bad is your headache?"

[0457] Examples and prompts

[0458] As a specific example, consider the case where a user complains of cold symptoms.

[0459] User input: "My head hurts."

[0460] Server action:

[0461] Estimate emotions such as "pain" and "suffering" from voice input.

[0462] Estimates emotions such as "pain" and "discomfort" from webcam footage.

[0463] Estimate emotions such as "pain" and "suffering" from text input.

[0464] Based on this information, it generates specific questions such as, "How bad is your headache?"

[0465] Example prompt:

[0466] If a patient complains of a "headache," infer their emotion from voice, facial expressions, and text and generate appropriate questions.

[0467] This system will enable doctors to grasp the user's condition more accurately and provide more efficient medical care.

[0468] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0469] Step 1:

[0470] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms and the time of onset. For example, they input information such as "I have a headache" or "I've had a fever since yesterday." The input data is sent from the terminal to the server.

[0471] Step 2:

[0472] The server receives the pre-interview information sent by the user. The received information is stored in the database. Specifically, the server connects to the database and stores the user's input data in an appropriate format. After the storage is completed, the server records the status of successful storage in the log.

[0473] Step 3:

[0474] The server uses the generative AI model to generate questions that the doctor will ask at the time of the visit based on the pre-medical interview information. For example, if the user inputs "I have a headache," the server inputs the prompt "I have a headache" into the generative AI model and generates questions such as "How bad is your headache?" and "What's your fever?" The generated questions are stored in a database.

[0475] Step 4:

[0476] A user uses a terminal to input a question to a doctor. For example, a question such as "What cold medicine should I take?" The input question is sent from the terminal to the server. The server receives the question and stores it in a database.

[0477] Step 5:

[0478] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the questions from the user. Specifically, the server retrieves the necessary information from the database and integrates the information according to the format of the medical record. The created medical record is saved in the database.

[0479] Step 6:

[0480] If the user uses voice input or a webcam, the server uses the emotion engine to estimate the user's emotion. For example, when the user says "I have a headache," the server inputs the voice data into the emotion engine to estimate the emotion of "pain" or "suffering." It also estimates the emotion of "distress" or "discomfort" from the webcam footage. The estimated emotion data is stored in a database.

[0481] Step 7:

[0482] The server communicates the created medical record and the results of the emotion engine to the doctor. Specifically, the server sends the medical record and emotion data to the doctor's terminal. The doctor uses this as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical record, such as "How bad is your headache?"

[0483] (Application example 2)

[0484] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0485] Conventional medical systems require a lot of time and effort to accurately grasp the symptoms of patients, making it difficult to monitor the health of passengers in real time, especially in autonomous vehicles. In addition, when a passenger complains of feeling unwell, it is necessary to generate appropriate questions and respond quickly, but current systems do not do this sufficiently. This can lead to the inability to properly manage the passenger's health, which can delay emergency responses.

[0486] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0487] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the result of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for performing voice recognition, a means for performing emotion analysis, a means for acquiring camera images, and a means for adjusting the questions based on the result of the emotion analysis. This makes it possible to monitor the health condition of a passenger in an autonomous vehicle in real time, and to generate appropriate questions and quickly respond when an abnormality is detected.

[0488] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, and is the process of collecting information such as symptoms and medical history.

[0489] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[0490] A "medical record" is a document that compiles a patient's medical records and includes details of the examination and prescriptions.

[0491] "Speech recognition" is the technology that converts speech into text.

[0492] "Sentiment analysis" is a technology that infers emotions from text, voice, facial expressions, etc.

[0493] "Camera video" refers to video data captured by a camera.

[0494] "Question tailoring" is the process of generating or modifying appropriate questions based on the results of sentiment analysis.

[0495] An "autonomous vehicle" is a vehicle that can drive automatically without a driver.

[0496] "Passenger health condition" refers to a passenger's physical condition, medical condition, or other health-related condition.

[0497] "Real-time monitoring" means monitoring the current state instantly.

[0498] A system for carrying out the present invention is configured as follows: The server includes means for conducting a preliminary interview, means for generating questions for a doctor based on the result of the preliminary interview, means for creating a medical record by integrating the questions of the doctor and questions from the patient, means for linking the medical record to the doctor, means for performing voice recognition, means for performing emotion analysis, means for acquiring camera images, and means for adjusting the questions based on the result of the emotion analysis.

[0499] Hardware and Software Used

[0500] Hardware: microphones, webcams, on-board computers in autonomous vehicles

[0501] Software: speech_recognition library, pipeline of transformers library, cv2 (OpenCV)

[0502] Data processing and calculation

[0503] 1. Speech recognition: The server captures the audio from the microphone and converts it to text using the speech_recognition library.

[0504] 2. Sentiment analysis: The server performs sentiment analysis on the retrieved text using a pipeline from the transformers library to estimate the sentiment.

[0505] 3. Acquiring camera images: The server acquires images from the webcam and displays them in real time using cv2 (OpenCV).

[0506] 4. Question Adjustment: The server generates or modifies appropriate questions based on the results of sentiment analysis.

[0507] Examples

[0508] For example, if a passenger says "I feel sick," the server recognizes the voice and performs emotion analysis. If the emotion is determined to be "negative," the server generates the question, "Which part hurts?" This makes it possible to monitor the health of passengers in autonomous vehicles in real time and respond quickly if an abnormality is detected.

[0509] Examples of prompt statements

[0510] If the user says "I feel sick", do sentiment analysis and if the sentiment is determined to be "negative", generate the question "Which part hurts?"

[0511] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0512] Step 1:

[0513] The server acquires voice from the microphone. When the user says "I feel sick," the voice data is input. The server converts the voice data into text using the speech_recognition library. Through this process, the voice data is output as text data.

[0514] Step 2:

[0515] The server uses the text data obtained in step 1 as input and performs sentiment analysis using the pipeline of the transformers library. The sentiment analysis model estimates emotions from the text data and outputs the results. For example, if the text is "feeling bad," the emotion is output as "negative."

[0516] Step 3:

[0517] The server acquires video data from a webcam. The camera video is acquired in real time and processed using cv2 (OpenCV). The video data is used to analyze the user's facial expressions. The analysis results are output as emotion data estimated from the user's facial expressions.

[0518] Step 4:

[0519] The server integrates the emotion data obtained in steps 2 and 3 to perform an overall emotion evaluation. For example, if the voice is judged to be "negative" and the facial expression is judged to be "painful," the overall emotion evaluation will be "very bad." This overall evaluation is output.

[0520] Step 5:

[0521] The server generates an appropriate question based on the overall emotion evaluation. Using a generative AI model, a prompt sentence is input and an appropriate question is generated. For example, if you say "I feel bad" and the emotion is determined to be "negative," the question "Which part hurts?" is generated. This question is output.

[0522] Step 6:

[0523] The server presents the generated questions to the user. When the user enters an answer, the answer data is sent to the server. The server integrates the answer data into the medical record and communicates it to the doctor. This process updates the medical record and provides it to the doctor.

[0524] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0525] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0526] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0527] In the above embodiment, an example has been given in which the specific process is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0528] [Second embodiment]

[0529] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0530] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0531] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0532] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0533] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0534] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0535] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0536] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0537] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0538] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0539] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0540] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0541] "Example 1"

[0542] The present invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and takes part in a preliminary interview by online automatic transcription with an avatar or by filling out a form. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0543] "Example 2"

[0544] As a specific example, if a patient complains of cold symptoms, the preliminary interview will ask details about the symptoms and when they began. Based on this, the system will generate questions that the doctor is likely to ask at the time of the consultation, such as "What's your fever?" or "Do you have a cough?" It will also accept questions from the patient, such as "What cold medicine should I take?" These questions are integrated into a medical record and shared with the doctor. The doctor will use this as a reference when examining the patient, ensuring efficient medical care.

[0545] The process flow of each embodiment will be described below.

[0546] "Example 1"

[0547] Step 1: The patient accesses a dedicated URL.

[0548] Step 2: Conduct a pre-interview with an avatar using online automated transcription or form filling.

[0549] Step 3: Based on the results of the pre-interview, the system generates questions that the doctor is likely to ask during the visit.

[0550] Step 4: Also receive questions from the patient and consolidate these questions to create a medical record.

[0551] Step 5: The created medical record is shared with the doctor, who will use it as a reference when examining the patient.

[0552] "Example 2"

[0553] Step 1: If the patient complains of cold symptoms, the preliminary interview involves asking about details of the symptoms and when they began.

[0554] Step 2: Based on this, the system generates questions that a doctor might ask during a consultation, such as "What's your fever?" or "Do you have a cough?"

[0555] Step 3: Also, answer any questions from the patient, such as "What cold medicine should I take?"

[0556] Step 4: Integrate these questions into the patient's chart and communicate with the doctor.

[0557] Step 5: The doctor will use this as a reference when examining the patient, ensuring efficient medical treatment.

[0558] Example 1

[0559] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0560] In conventional medical systems, there was a problem that it was time-consuming for medical staff to ask questions before patients visit the clinic, making it difficult to conduct efficient examinations. In addition, it was difficult to accurately grasp the details of the patient's questions and symptoms, which could lead to a decline in the quality of the examination. Furthermore, the doctor could not quickly obtain the information necessary for the examination, which sometimes prolonged the examination time. To solve these problems, a system was needed that would improve the efficiency of the pre-examination and enable doctors to quickly obtain the information necessary for the examination.

[0561] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0562] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for extracting keywords and symptoms using natural language processing, and a means for generating questions for the doctor using a generative AI model. This makes it possible to improve the efficiency of the preliminary interview and allows the doctor to quickly obtain information required for the examination, thereby improving the quality of the examination and shortening the examination time.

[0563] A "pre-medical interview" is a process in which a patient enters their symptoms and condition before visiting a doctor, and medical professionals collect the necessary information before the examination.

[0564] The "doctor's questions" are questions that the doctor asks the patient during the consultation, which are generated based on the results of the preliminary interview.

[0565] A "medical record" is a medical document that records a patient's symptoms, examination results, treatment details, etc.

[0566] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[0567] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new information or data.

[0568] "Keywords" are important words or phrases within text data that are extracted using natural language processing.

[0569] A "symptom" is any physical or mental abnormality or discomfort experienced by a patient.

[0570] An "avatar" is a virtual character that allows users to interact online.

[0571] "Online automatic transcription" is a technology that converts audio into text in real time.

[0572] "Form entry" is a method in which a user manually enters text data into fields on a web page.

[0573] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0574] Hardware and Software Used

[0575] This system is realized using the following hardware and software.

[0576] Hardware: Servers, terminals (smartphones, PCs, etc.)

[0577] Software: Python, natural language processing libraries (NLTK, spaCy), database manipulation libraries (SQLAlchemy), web interface framework (Flask), generative AI models (e.g., OpenAI's GPT-3)

[0578] Program Processing

[0579] The user (patient) accesses a dedicated URL. Once accessed, an avatar appears on the device and the preliminary interview begins through online automatic transcription or form input. Specifically, the text data entered by the user is sent from the device to the server.

[0580] The server analyzes the received text data using natural language processing libraries (NLTK and spaCy) to extract keywords and symptoms. Based on the results of this analysis, a generative AI model (for example, OpenAI's GPT-3) is used to generate questions that the doctor may ask during the consultation.

[0581] In addition, the terminal accepts additional questions from the user, and the questions entered on the terminal are sent back to the server, which then consolidates these questions and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner).

[0582] The created medical records are sent from the server to the doctor's terminal, and the doctor uses the records as a reference when examining the patient.

[0583] Examples of concrete examples and prompts

[0584] As a concrete example, consider the case where a user inputs, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." This text data is sent from the device to the server, and the server uses a natural language processing library to extract keywords such as "headache" and "when I wake up in the morning."

[0585] Next, feed the generative AI model the following prompt:

[0586] User symptoms: Headache, severe pain upon waking in the morning

[0587] Generate questions your doctor may have during a consultation.

[0588] A generative AI model might generate questions like:

[0589] 1. How often do your headaches occur?

[0590] 2. How severe is your headache?

[0591] 3. Do you have any other symptoms besides headaches?

[0592] These questions and any additional questions from the user are integrated to create a medical record in a format suitable for the electronic medical record system. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0593] In this way, the system acts as a substitute for medical professionals to support patients.

[0594] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0595] Step 1:

[0596] The user accesses a special URL.

[0597] The user accesses the dedicated URL provided by the medical institution. For example, the user inputs the URL using a browser on a smartphone or computer. Input: Dedicated URL. Output: Display of the web page.

[0598] Step 2:

[0599] The terminal will display an avatar and begin the preliminary interview.

[0600] The terminal displays a web page based on the accessed URL. This web page displays an avatar and prompts the user to start the pre-interview. Input: Dedicated URL. Output: Display of avatar and start of pre-interview.

[0601] Step 3:

[0602] The user inputs text data.

[0603] The user follows the avatar's instructions to input their symptoms and condition in text format. For example, the user might input, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." Input: User's symptoms and condition. Output: Text data.

[0604] Step 4:

[0605] The terminal transmits the input data to the server.

[0606] The terminal sends the text data entered by the user to the server. This transmission is done securely using the HTTPS protocol. Input: Text data. Output: Data transmission to the server.

[0607] Step 5:

[0608] The server performs natural language processing to extract keywords and symptoms.

[0609] The server analyzes the received text data using a natural language processing library (NLTK or spaCy). For example, it extracts keywords such as "headache" and "when waking up in the morning." Input: Text data. Output: Extracted keywords and symptoms.

[0610] Step 6:

[0611] The server uses the generative AI model to generate questions for the doctor.

[0612] The server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-3) based on the extracted keywords, and generates questions that a doctor may expect to ask when visiting the patient. For example, a prompt sentence such as "User's symptoms: headache, severe pain when waking up in the morning. Please generate questions that a doctor may expect to ask when visiting the patient" can be used. Input: Extraction results of keywords and symptoms. Output: Questions asked by the doctor.

[0613] Step 7:

[0614] The user enters a follow-up question.

[0615] The user inputs additional questions or information for the generated question. For example, the user answers "About three times a week" to the question "How often do you get headaches?". Input: Doctor's question. Output: Additional questions or information.

[0616] Step 8:

[0617] The terminal sends a follow-up question to the server.

[0618] The terminal sends any additional questions or information entered by the user back to the server. Input: Additional questions or information. Output: Sending data to the server.

[0619] Step 9:

[0620] The server consolidates the questions and creates a medical record.

[0621] The server merges the initial interview data and the follow-up question data and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner). Input: Initial interview data, follow-up question data. Output: Medical record.

[0622] Step 10:

[0623] The server links the medical records to the doctor's terminal.

[0624] The server links the created medical records to the doctor's terminal. The doctor can view the medical records through a dedicated medical system. Input: Medical records. Output: Linking medical records to the doctor's terminal.

[0625] Step 11:

[0626] The doctor will examine the patient using the medical records.

[0627] The doctor will consult the linked medical records when examining the patient. The medical records contain detailed information about the user's symptoms and the results of the preliminary interview, allowing the examination to proceed smoothly. Input: Medical records. Output: Examination results.

[0628] (Application example 1)

[0629] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0630] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, and doctors often lacked the information they needed to examine patients. In addition, there was no way to understand the patient's symptoms or questions in advance when consulting about medicines at pharmacies or drug stores, making it difficult to respond efficiently. This resulted in longer waiting times for patients and increased burdens on medical professionals.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0632] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to a doctor, a means for supporting preliminary interviews and consultations about medicines at pharmacies and drug stores, and a means for generating questions using a generative AI model. This allows a patient to collect necessary information before visiting a medical institution or a pharmacy, enabling efficient examinations and consultations.

[0633] A "pre-medication interview" is a process in which medical professionals collect necessary information in advance by having patients enter their symptoms and questions online before visiting a medical institution or pharmacy.

[0634] The "means for generating questions for a doctor" is a system that automatically generates questions that a doctor should ask during a consultation based on the results of a preliminary interview.

[0635] The "means of creating a medical record" refers to a system that integrates the results of the preliminary interview, questions from the doctor, and questions from the patient to create a medical record that compiles all the information necessary for the examination.

[0636] "Means of sharing medical records with doctors" refers to a system that provides the created medical records to doctors so that they can refer to them during consultations.

[0637] The "means of supporting advance medical interviews and medication consultations at pharmacies and drug stores" is a system that allows patients to input their symptoms and questions in advance at pharmacies and drug stores, enabling pharmacists to respond efficiently.

[0638] A "generative AI model" is an algorithm that uses artificial intelligence to generate appropriate questions and answers based on input data.

[0639] The "means for generating questions" refers to a system that uses a generative AI model to automatically generate appropriate questions based on the results of the preliminary interview.

[0640] The system for implementing this invention is configured as follows. First, a user accesses a dedicated URL using a terminal such as a smartphone or a personal computer. After accessing the URL, a preliminary interview is conducted by online automatic transcription with an avatar or by filling out a form. The results of this preliminary interview are sent to the server.

[0641] Based on the results of the preliminary interview, the server generates questions for the doctor using a generative AI model that uses advanced natural language processing algorithms, such as OpenAI's GPT-3. The generated questions are stored on the server.

[0642] Next, the server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. This medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0643] In addition, the system also has a function to support pre-medication and drug consultations at pharmacies and drug stores. Before going to a pharmacy or drug store, the user performs a pre-medication interview using a similar procedure, and the generative AI model generates questions for the pharmacist based on the results. This allows the pharmacist to efficiently respond to patient consultations.

[0644] For example, if a user enters, "I have a headache and a sore throat," the generative AI model generates questions such as, "When did the headache start?" and "How bad is your throat?" These questions are integrated into the medical record and provided to doctors and pharmacists.

[0645] An example of a prompt is as follows:

[0646] "Generate questions the pharmacist might have based on the following symptoms: I have a headache and a sore throat."

[0647] In this way, a system can be realized that enables efficient medical examinations and consultations based on the results of the preliminary interview.

[0648] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0649] Step 1:

[0650] The user accesses a dedicated URL using a device such as a smartphone or PC. The user completes a preliminary interview by using an avatar to automatically transcribe online or by filling out a form. The data entered is the user's symptoms and questions. This data is then sent from the device to the server.

[0651] Step 2:

[0652] The server uses a generative AI model to generate questions for the doctor based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated doctor's question.

[0653] Step 3:

[0654] The server creates a medical record by integrating the generated doctor's questions and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[0655] Step 4:

[0656] The server transmits the created medical record to the doctor. The medical record is provided so that the doctor can refer to it when examining the patient. The input is the integrated medical record, and the output is the medical record provided to the doctor.

[0657] Step 5:

[0658] Before a user goes to a pharmacy or drugstore, the user performs a pre-medical interview using a similar procedure. The user inputs symptoms and questions using a terminal and sends them to the server. The input data is the user's symptoms and questions.

[0659] Step 6:

[0660] The server uses a generative AI model to generate questions for the pharmacist based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated questions for the pharmacist.

[0661] Step 7:

[0662] The server creates a medical record by integrating the generated questions from the pharmacist and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[0663] Step 8:

[0664] The server communicates the created chart to the pharmacist. The chart is provided to the pharmacist so that the pharmacist can refer to it during consultation. The input is the integrated chart, and the output is the chart provided to the pharmacist.

[0665] Example 2

[0666] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0667] In conventional medical systems, the process from the patient's preliminary interview to the doctor's examination was not carried out efficiently, which often resulted in long consultation times. In addition, it was difficult to generate appropriate questions based on the patient's symptoms, and doctors were sometimes unable to fully grasp the necessary information before the examination. This resulted in poor efficiency in medical treatment and poor patient satisfaction.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0669] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor using a generative AI model based on the results of the preliminary interview, a means for creating a medical record by integrating the doctor's questions and questions from the patient, and a means for sharing the medical record with the doctor. This allows appropriate questions based on the patient's symptoms to be automatically generated, enabling the doctor to grasp the necessary information before the consultation. This improves the efficiency of medical care and also improves patient satisfaction.

[0670] A "pre-medical interview" is the process in which a patient provides information about their symptoms, medical history, etc. before an appointment.

[0671] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate outputs for a specific task.

[0672] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[0673] "Patient Questions" are questions that patients ask their physicians about their symptoms or treatment.

[0674] A "medical record" is a document that contains a patient's medical records, including symptoms, diagnoses, and treatment details.

[0675] "Collaboration" refers to different systems and processes sharing information and working together.

[0676] A "web page" is a page of information accessible on the Internet.

[0677] An "avatar" is a character or icon that represents a user online.

[0678] "Online automatic transcription" is a technology that converts audio into text in real time.

[0679] "Form filling" is the process by which a user enters information into fields on a web page.

[0680] "Medical Condition" refers to a patient's health condition or symptoms.

[0681] "Procedure" refers to the medical treatment or care that a doctor provides to a patient.

[0682] "Treatment time" refers to the time it takes a doctor to treat a patient.

[0683] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0684] This invention is a system that performs a preliminary interview, generates questions for the doctor using a generative AI model, creates a medical record by integrating the questions from the patient, and communicates with the doctor. A specific embodiment of this system is described below.

[0685] Hardware and Software Used

[0686] Hardware: Servers, terminals (PCs, tablets, smartphones, etc.)

[0687] Software: Generative AI models (e.g. GPT-4), database management systems (e.g. MySQL), web application frameworks (e.g. Django)

[0688] System Configuration

[0689] 1. Server: The server is responsible for generating questions to conduct a pre-medical interview. It uses a generative AI model (e.g. GPT-4) to generate questions based on the patient's symptoms. The server also receives answers from the user (patient) and stores them in a database. The server also integrates the collected information, creates a medical record, and notifies the doctor.

[0690] 2. Terminal: The terminal provides an interface for the user (patient) to conduct a preliminary interview. The terminal displays the questions received from the server to the user, collects the user's answers, and sends them to the server.

[0691] 3. User: The user (patient) uses the terminal to conduct a preliminary interview. The user answers the questions displayed on the terminal and, if necessary, enters his / her own questions.

[0692] Examples

[0693] 1. Question generation for pre-medical interview: The server uses a generative AI model (e.g., GPT-4) to generate questions for pre-medical interviews that are necessary when a patient complains of cold symptoms. For example, questions such as "When did your symptoms start?" and "What symptoms are you experiencing?" are generated. The generated questions are stored in a database.

[0694] 2. Displaying questions on the terminal: The terminal retrieves the questions from the pre-medical interview from the server and displays them to the user. For example, questions such as "What is your fever?" and "Do you have a cough?" are displayed on the terminal screen.

[0695] 3. Collect user answers: The user answers the questions displayed on the terminal. For example, they input information such as "My fever is 37.5 degrees" or "I have a cough." The user can also input their own questions. For example, they input questions such as "What cold medicine should I take?"

[0696] 4. Sending and storing answers: The terminal sends the user's answers to the server. The server stores the received answers in a database. This allows the centralized management of patient information.

[0697] 5. Creating a medical record: The server consolidates the collected information and creates a medical record. The medical record includes details of the patient's symptoms, when they first appeared, questions that the doctor anticipates, and questions from the patient. The created medical record is stored in a database.

[0698] 6. Medical record sharing with doctor: The server sends the created medical record to the doctor's terminal. The doctor performs the examination using the received medical record as a reference. For example, by looking at the medical record, the doctor can know in advance information such as "The patient has a fever of 37.5 degrees and is coughing." This allows for more efficient medical treatment.

[0699] Examples of prompt statements

[0700] "A patient is complaining of cold symptoms. Using the information below, generate questions that a doctor might ask during a consultation. Please include details of the patient's symptoms and when they began."

[0701] This system allows doctors to understand patients' symptoms in advance, improving the efficiency of medical treatment and increasing patient satisfaction.

[0702] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0703] Step 1:

[0704] The server generates the pre-screening questions.

[0705] Input: Information about the patient's initial symptoms (e.g. cold symptoms)

[0706] Data processing: Input a prompt sentence into a generative AI model (e.g. GPT-4) to generate appropriate questions.

[0707] Output: Generated questions (e.g., "When did your symptoms start?", "What symptoms are you experiencing?")

[0708] Specific operation: The server inputs a prompt sentence into the generative AI model: "The patient is complaining of cold symptoms. Based on the information below, please generate questions that a doctor may expect to ask when visiting the patient. Please also include details of the patient's symptoms and when they began." and stores the generated questions in a database.

[0709] Step 2:

[0710] The terminal obtains the questions for the preliminary medical examination from the server and displays them to the user.

[0711] Input: A generated query from the server

[0712] Data processing: Converting the question into a format for display to the user.

[0713] Output: The question that is displayed to the user (e.g. "What's your fever?", "Do you have a cough?")

[0714] Specific operation: The terminal displays the questions obtained from the server on the screen and formats them in a way that makes it easy for the user to answer.

[0715] Step 3:

[0716] The user answers the questions displayed on the terminal.

[0717] Input: the question displayed on the terminal

[0718] Data processing: Enter the user's answers into the input form.

[0719] Output: User's answer (e.g. "My temperature is 37.5 degrees" or "I have a cough")

[0720] Specific operation: The user inputs text in response to the questions displayed on the terminal, and if necessary, inputs his or her own question.

[0721] Step 4:

[0722] The terminal transmits the user's answer to the server.

[0723] Input: User's answer

[0724] Data processing: The response data is converted into a format for sending to the server.

[0725] Output: Answer data sent to the server

[0726] Specific operation: The terminal transmits the answers entered by the user to the server in real time.

[0727] Step 5:

[0728] The server stores the received answers in a database.

[0729] Input: Answer data sent from the device

[0730] Data processing: Convert the response data into a format that can be stored in the database.

[0731] Output: Answer data stored in a database

[0732] Specific operations: The server stores the received response data in a database for subsequent processing.

[0733] Step 6:

[0734] The server consolidates the collected information and creates a medical record.

[0735] Input: Answer data stored in the database

[0736] Data processing: Integrate the response data and create a medical record.

[0737] Output: Created medical record

[0738] Specific operation: The server creates a medical record by integrating the patient's symptoms and questions based on the information stored in the database.

[0739] Step 7:

[0740] The server transmits the created chart to the doctor's terminal.

[0741] Input: Created medical record

[0742] Data processing: Converting medical record data into a format for sending to the doctor's terminal.

[0743] Output: Medical record sent to doctor's terminal

[0744] Specific operation: The server sends the created medical record to the doctor's terminal so that the doctor can review the information before the examination.

[0745] Step 8:

[0746] The doctor will use the medical records he or she receives to conduct the examination.

[0747] Input: Medical record sent to doctor's terminal

[0748] Data processing: Check the contents of the medical records and extract the information necessary for the examination.

[0749] Output: Examination results and treatment plan

[0750] Specific actions: The doctor reviews the patient's medical records to understand the patient's symptoms and questions, and conducts an efficient examination.

[0751] (Application example 2)

[0752] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0753] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, which often led to long consultation times for doctors. In addition, even in brick-and-mortar stores, there was no way for customers to ask questions before visiting the store, which led to inefficient responses from store staff.

[0754] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0755] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for receiving symptoms reported by a customer as an input and generating questions to be asked by store staff, a means for generating questions for the customer to ask the store staff, a means for integrating the generated questions to create a medical record, and a means for outputting the created medical record. This enables the efficiency of medical examinations at medical institutions and the efficiency of customer service at physical stores.

[0756] A "pre-medical interview" is a process in which patients or customers enter their symptoms and requests in advance before visiting a medical institution or physical store.

[0757] "Doctor's questions" are questions that the doctor should ask the patient based on the results of the preliminary interview.

[0758] "Patient questions" are questions that patients want to ask their doctors.

[0759] A "medical record" is a record that integrates the symptoms and questions of patients or customers, as well as questions from doctors and store staff.

[0760] "Symptoms reported by customers" refers to the physical discomfort and requests felt by customers who visit physical stores.

[0761] "Questions that store staff should ask" are questions that store staff should ask customers based on the customer's symptoms and requests.

[0762] "Questions that customers should ask store staff" are questions that customers would like to ask store staff.

[0763] A "generated question" is a question that is automatically generated based on the results of a preliminary interview.

[0764] The "means of creating a medical record" refers to the process of creating a medical record by integrating the results of the preliminary interview and the questions generated.

[0765] "Means for outputting medical records" refers to the process of providing the created medical records to doctors and store staff.

[0766] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for receiving as input symptoms reported by a customer and generating questions to be asked by store staff, means for generating questions for the customer to ask the store staff, means for integrating the generated questions to create a medical record, and means for outputting the created medical record.

[0767] The server receives the results of the preliminary interview and uses a generative AI model to generate questions for the doctor and store staff. Specifically, it uses OpenAI's API to generate appropriate questions based on the results of the preliminary interview. The generated questions are processed using a Python program and integrated into the medical record.

[0768] The terminal (smartphone or tablet) provides an interface for the user to conduct a preliminary medical interview. The user accesses a dedicated URL and conducts the preliminary medical interview by online automatic transcription with an avatar or by filling out a form. The user's symptoms and requests are then sent to the server.

[0769] When a user inputs the symptoms they are complaining about, the server uses a generative AI model to generate questions that store staff should ask, or questions that customers should ask store staff. The generated questions are integrated into a medical record and provided to store staff.

[0770] As a specific example, the prompt sentence when a customer complains of cold symptoms is shown below.

[0771] "A customer is complaining of the following symptoms: Cold symptoms. Generate questions for store staff to ask."

[0772] When this prompt is executed, the server will generate a question like this:

[0773] "What's your fever? Do you have a cough?"

[0774] It also generates questions for customers to ask store staff.

[0775] "What cold medicine should I take?"

[0776] This will enable more efficient medical examinations at medical institutions and more efficient customer service at physical stores.

[0777] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0778] Step 1:

[0779] The user accesses a dedicated URL using a device (smartphone or tablet). The user completes a preliminary interview by using an avatar to automatically transcribe text online or by filling out a form. The input data is the user's symptoms and requests. The output data is sent to the server as the results of the preliminary interview.

[0780] Step 2:

[0781] The server analyzes the received results of the pre-interview. The input data is information about the user's symptoms and requests. The server uses a generative AI model (e.g., OpenAI's API) to generate questions for the doctor and questions for the store staff based on the results of the pre-interview. The output data are the generated questions.

[0782] Step 3:

[0783] The server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. The input data are the generated questions and the results of the preliminary interview. The server integrates these data and configures them as a medical record. The output data is the created medical record.

[0784] Step 4:

[0785] The server shares the created medical record with the doctor. The input data is the created medical record. The server sends the medical record to the doctor's terminal so that the doctor can refer to it when examining the patient. The output data is the medical record provided to the doctor.

[0786] Step 5:

[0787] When a user inputs the symptoms they are experiencing, the server uses a generative AI model to generate questions for store staff to ask. The input data is information about the user's symptoms. The server uses a generative AI model to generate appropriate questions. The output data is the questions that store staff should ask.

[0788] Step 6:

[0789] The server generates questions that customers should ask store staff. The input data is information about the user's symptoms. The server uses a generative AI model to generate questions that customers should ask. The output data is questions that customers should ask store staff.

[0790] Step 7:

[0791] The server integrates the generated questions to create a medical record. The input data are the generated questions. The server integrates these questions and configures them as a medical record. The output data is the created medical record.

[0792] Step 8:

[0793] The server outputs the created medical record. The input data is the created medical record. The server provides the medical record to the store staff so that they can refer to it when dealing with customers. The output data is the medical record provided to the store staff.

[0794] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0795] "Example 1"

[0796] In one embodiment of the present invention, an emotion engine is introduced. The emotion engine recognizes emotions from the user's voice, facial expression, or text input. Specifically, when the user conducts a pre-interview, the emotion is estimated from the tone, strength, and speed of the user's voice. If the user is using a webcam, the emotion is also estimated from facial expression. Furthermore, if the user inputs text, the emotion is estimated from the content and expression of the text.

[0797] "Example 2"

[0798] As a concrete example, consider a case where a user complains of cold symptoms. When the user says "My head hurts," emotions such as "pain" or "suffering" are inferred from the voice. Also, if the user is using a webcam, emotions such as "pain" or "discomfort" are inferred from the facial expression. Furthermore, if the user inputs the text "My head hurts," emotions such as "pain" or "suffering" are also inferred from that text. Based on the recognition results of these emotion engines, the doctor's questions are adjusted. For example, a more specific question such as "How bad is your headache?" is generated. This allows the doctor to grasp the user's condition more accurately and perform an appropriate examination.

[0799] The process flow of each embodiment will be described below.

[0800] "Example 1"

[0801] Step 1: The user accesses the URL for the pre-consultation questionnaire.

[0802] Step 2: The user answers the medical questions, while the emotion engine analyzes the user's voice, facial expressions, or text input.

[0803] Step 3: The emotion engine recognizes the user's emotions and shares the results with the doctor.

[0804] "Example 2"

[0805] Step 1: When a user reports cold symptoms, their voice, facial expressions, or text input are analyzed by the emotion engine.

[0806] Step 2: The emotion engine recognizes the user's emotions, such as "pain" or "suffering."

[0807] Step 3: Based on the emotion engine's recognition results, the doctor's question is adjusted, for example, a more specific question is generated, such as "How bad is your headache?"

[0808] Example 1

[0809] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0810] In the conventional medical system, when a patient conducts a pre-consultation before visiting a doctor, medical staff must directly respond to the patient, which takes time and effort. It is also difficult to accurately grasp the patient's emotions, and there are cases where the information required for an appropriate examination is insufficient. Furthermore, doctors are unable to prepare questions to ask at the time of the consultation in advance, which can reduce the efficiency of examinations.

[0811] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0812] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from the user's voice, facial expression, or text input, and a means for generating questions for the doctor using a generative AI model. This allows the patient to efficiently conduct a preliminary interview before a medical examination, and the doctor to prepare appropriate questions in advance. In addition, by accurately grasping the patient's emotions, information for a more appropriate medical examination can be provided.

[0813] A "pre-medical interview" is an interview conducted by a patient before visiting a doctor, and is the process of collecting information such as the patient's symptoms and medical history.

[0814] "Doctor's questions" are questions that a doctor asks a patient, and are intended to understand the patient's symptoms and condition in detail.

[0815] A "medical record" is a document that compiles a patient's medical records and serves as a reference for doctors when examining and treating patients.

[0816] "Means for recognizing emotions" refers to technology or devices for inferring emotions from a user's voice, facial expression, or text input.

[0817] A "generative AI model" is a model that uses artificial intelligence to generate text and questions, and examples include GPT-3.

[0818] An "avatar" is a virtual character that interacts with a user and assists in online communication.

[0819] "Online automatic transcription" is a technology that converts voice input into text, automatically recording what a user says as text.

[0820] "Form entry" refers to a method in which a user enters information into a web form, and is used during a pre-consultation interview.

[0821] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[0822] Hardware and Software Used

[0823] The system's program is developed using Python. The main libraries used are NLTK for natural language processing, TextBlob for sentiment analysis, SpeechRecognition for voice recognition, and OpenCV for facial expression recognition. Advanced natural language generation models such as GPT-3 are used as generative AI models.

[0824] Program processing explanation

[0825] When a user accesses a dedicated URL, the server displays an avatar to the user, which then interacts with the user to ask questions. The user can answer the questions by voice input, text input, or form input.

[0826] The server receives the user's input data and performs the following operations:

[0827] For speech input: Use the SpeechRecognition library to convert speech to text.

[0828] For text input: Process it as text data as is.

[0829] For form filling: Collect data for each field in the form.

[0830] The server then analyzes the user's emotions using an emotion engine, which estimates emotions in the following way:

[0831] For voice input: Analyzes the tone, volume and speaking rate of the voice.

[0832] For facial expression input: OpenCV is used to analyze the video captured by the webcam and estimate emotions from facial expressions.

[0833] For text input: Use TextBlob to infer sentiment from the text content and expression.

[0834] Based on this data, the server generates questions that the doctor is likely to ask during the consultation. It uses a generative AI model (e.g., GPT-3) to generate questions based on the user's pre-medical interview results. It also accepts questions from the user and integrates these questions to create a medical record.

[0835] The created chart is sent from the server to the doctor, who then uses it as a reference when examining the patient.

[0836] Examples of concrete examples and prompts

[0837] As a specific example, consider a case where a user complains of a headache and inputs, by voice, "I've been having a really bad headache lately."

[0838] The server does the following:

[0839] 1. Convert speech to text using the SpeechRecognition library.

[0840] 2. Use TextBlob to infer the user's emotion (e.g., distress or anxiety) from the text "I have a bad headache."

[0841] 3. Use GPT-3 to generate questions a doctor might ask during a visit, such as "When did your headache start?" or "Where does it hurt?"

[0842] Example prompt:

[0843] If a user types, "I've been having really bad headaches lately," generate questions a doctor might ask during a visit.

[0844] In this way, the system acts as a substitute for medical professionals to support patients.

[0845] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0846] Step 1:

[0847] The user accesses a special URL.

[0848] Input: User's access request

[0849] Output: Login screen displayed

[0850] Specific operation: A user opens a browser and accesses a dedicated URL. The server receives the user's access request and displays a login screen.

[0851] Step 2:

[0852] The user enters login information and is authenticated.

[0853] Input: User ID and password

[0854] Output: Authentication result (success or failure)

[0855] Specific operation: The user enters the ID and password and clicks the "Login" button. The server accesses the database and verifies the entered ID and password. If authentication is successful, an avatar is displayed to the user.

[0856] Step 3:

[0857] The user conducts a preliminary interview through dialogue with an avatar.

[0858] Input: User voice input, text input, or form input

[0859] Output: Pre-interview data (text format)

[0860] Specific operation: The user speaks to the avatar, saying, "I've been having a really bad headache lately." The server uses the SpeechRecognition library to convert the speech to text, and obtains the text data, "I've been having a really bad headache lately."

[0861] Step 4:

[0862] The server uses an emotion engine to analyze the user's emotions.

[0863] Input: Pre-medical interview data (text format)

[0864] Output: Sentiment analysis results (e.g., distress, anxiety)

[0865] Specific operation: The server uses TextBlob to analyze the user's emotions from the text "I've been having a lot of headaches lately." For example, it estimates emotions such as pain and anxiety.

[0866] Step 5:

[0867] The server uses the generative AI model to generate questions for the doctor.

[0868] Input: Pre-interview data and sentiment analysis results

[0869] Output: Doctor's questions (text format)

[0870] Specific operation: The server inputs a prompt sentence to the generative AI model (GPT-3). Example of a prompt sentence: "If a user inputs, 'I've been having a lot of headaches lately,' please generate a question that a doctor would ask when visiting the doctor." The server takes the generated question and generates a question such as, "When did your headache start?" or "Where does it hurt?"

[0871] Step 6:

[0872] The server creates a medical record by integrating the results of the preliminary interview with the generated doctor's questions.

[0873] Input: Pre-medication data and doctor's questions

[0874] Output: Medical record (text format)

[0875] Specific operation: The server integrates the user's preliminary interview results with the generated doctor's questions and creates a medical record.

[0876] Step 7:

[0877] The medical records created by the server are shared with the doctor.

[0878] Input: Medical record (text format)

[0879] Output: Send medical record to doctor

[0880] Specific operation: The server sends the created chart to the doctor. The doctor uses the chart as a reference when examining the patient.

[0881] (Application example 1)

[0882] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0883] In conventional medical systems, the patient's pre-consultation and medical record creation were done manually, which placed a heavy burden on medical staff and was inefficient. In addition, in brick-and-mortar stores, the response to customer questions and requests was insufficient, which led to a decrease in customer satisfaction. Furthermore, there was also the issue of it being difficult to appropriately recognize and respond to customer emotions.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0885] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, a means for accepting questions and requests from customers in a physical store, recognizing emotions, and providing appropriate responses, and a means for recording questions and requests from customers and linking them to store staff. This reduces the burden on medical staff and enables efficient creation of medical records, while also enabling proper recognition of customer emotions in a physical store and providing highly satisfying responses.

[0886] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, in order to understand the patient's symptoms and medical history in advance.

[0887] "Doctor's questions" are questions that a doctor asks a patient to confirm in detail the patient's symptoms and condition.

[0888] A "medical chart" is a compilation of a patient's medical records, which serve as a reference for doctors when examining and treating patients.

[0889] An "emotion engine" is a system for recognizing emotions from a user's voice, facial expressions, or text input.

[0890] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[0891] "Customer questions and requests" are questions and requests made by customers to store staff or support robots in physical stores.

[0892] "Store staff" refers to employees who work in physical stores and provide products and services to customers.

[0893] "Online automatic transcription" is a technology that converts speech into text, automatically recording what a user says as text.

[0894] "Form input" refers to a method in which a user inputs information into an input field on a web page, and is used for things like advance interviews and questionnaires.

[0895] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, means for accepting customer questions and requests in a physical store, recognizing emotions and responding appropriately, and means for recording customer questions and requests and linking them to store staff.

[0896] System Programming

[0897] The system program performs the following processing.

[0898] 1. Pre-medication:

[0899] The server will ask the user a preliminary interview via online automatic transcription with an avatar or through form entry when the user accesses a dedicated URL. The information entered by the user will be stored in a database.

[0900] 2. Doctor Question Generation:

[0901] The server generates questions that are expected to be asked by the doctor at the time of the visit based on the results of the preliminary interview. The generated questions are added to the user's medical record.

[0902] 3. Creation and collaboration of medical records:

[0903] The server creates a medical record by integrating the generated doctor's questions with the user's questions. The created medical record is shared with the doctor and used as a reference during consultations.

[0904] 4. Emotion recognition:

[0905] The server uses an emotion engine to recognize emotions from the user's voice, facial expressions, or text input. For emotion recognition, it uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model).

[0906] 5. Customer service in physical stores:

[0907] In a brick-and-mortar store, the server will receive customer questions and requests, recognize emotions, and respond appropriately. Customer questions and requests are recorded and communicated to store staff.

[0908] Hardware and Software Used

[0909] Hardware:

[0910] Webcam: Used to capture the customer's face.

[0911] Microphone: Used to capture the customer's voice.

[0912] software:

[0913] OpenCV: A library for face recognition.

[0914] SpeechRecognition: A library for speech recognition.

[0915] Transformers: Generative AI models for emotion recognition (using the Hugging Face pipeline).

[0916] Examples

[0917] For example, if a customer in a physical store asks a robot, "I want to know more about this product," the server will recognize the customer's emotions from their voice and facial expressions and respond appropriately. The customer's questions and requests are recorded and shared with the store staff.

[0918] Examples of prompt statements

[0919] "Estimate how your customers feel when they say, 'I want to know more about this product.'"

[0920] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0921] Step 1:

[0922] The server performs a preliminary interview with the user who accesses a dedicated URL through online automatic transcription with an avatar or form input. The information entered by the user is stored in a database. The input is information about the user's symptoms and medical history, and the output is the interview results stored in the database.

[0923] Step 2:

[0924] The server generates questions that are expected to be asked by the doctor at the time of the consultation based on the results of the preliminary interview. The generated questions are added to the user's medical record. The input is the results of the preliminary interview, and the output is the generated questions from the doctor. For data processing, an algorithm is used to analyze the interview results and generate appropriate questions.

[0925] Step 3:

[0926] The server creates a medical record by integrating the generated doctor's questions and the user's questions. The created medical record is shared with the doctor and used as a reference during consultations. The input is the doctor's questions and the user's questions, and the output is the integrated medical record. Data calculations include integrating the questions and formatting the medical record.

[0927] Step 4:

[0928] The server uses an emotion engine to recognize emotions from the user's voice, facial expression, or text input. Emotion recognition uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model). The input is the user's voice, facial expression, or text, and the output is the recognized emotion. Data processing involves analysis of voice and image data.

[0929] Step 5:

[0930] In a physical store, the server receives customer questions and requests, recognizes emotions, and responds appropriately. Customer questions and requests are recorded and communicated to store staff. The input is the customer's question or request, and the output is the recorded data and the response. Data calculations are performed to analyze the question or request and generate an appropriate response.

[0931] Step 6:

[0932] The server records the customer's questions and requests and shares them with the store staff. The input is the customer's questions and requests, and the output is the information shared with the store staff. Data processing involves recording and sharing of questions and requests.

[0933] Example 2

[0934] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0935] In conventional medical systems, doctors were not sufficiently prepared to ask appropriate questions based on the patient's pre-interview information, which led to problems with the efficiency of medical examinations. In addition, questions were not adjusted to take into account the patient's feelings, making it difficult to accurately grasp the patient's condition. This resulted in a decline in the quality of medical care and a decrease in patient satisfaction.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0937] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for estimating emotions from audio input and video input, and a means for adjusting the questions of the doctor based on the emotions. This makes it possible to generate appropriate questions based on the patient's preliminary interview information and adjust the questions taking the patient's emotions into consideration.

[0938] A "pre-medical interview" is a process in which a patient inputs information about their symptoms, the time of onset, etc., before their appointment.

[0939] The "doctor's questions" are questions that the doctor asks the patient, which are generated based on the results of the preliminary interview.

[0940] A "medical chart" is a medical record that is created by integrating information from the patient's prior medical interview, questions from the doctor, and questions from the patient.

[0941] The "means for estimating emotions" refers to technology for analyzing and estimating a patient's emotions from audio and video input.

[0942] The "means for adjusting questions" refers to a technique for appropriately changing the content of questions posed to a doctor based on the estimated emotions.

[0943] "Web Page" means an information page accessible via the Internet that provides an interface for conducting a pre-consultation interview.

[0944] "Online automatic transcription" is a technology that converts voice input into text and is used as part of the pre-interview.

[0945] "Form entry" is a method by which a user enters information into fields on a web page.

[0946] A "generative AI model" is an algorithm that uses artificial intelligence to perform natural language processing and generate appropriate questions and answers.

[0947] The present invention is a system that generates questions for a doctor based on information about a patient's advance medical interview and adjusts the questions in consideration of the patient's feelings. A specific embodiment of this system will be described below.

[0948] Hardware and Software Used

[0949] Hardware

[0950] Server: Use a server machine with a powerful processor and sufficient memory.

[0951] Terminal: A device such as a PC or smartphone is used for the user to input pre-interview information.

[0952] software

[0953] Generative AI models: Using GPT-4 as an example, for natural language processing.

[0954] Database: A database is used to store pre-interview information, generated questions, and medical records.

[0955] Emotion Engine: Uses software to infer emotions from audio and video inputs.

[0956] System processing flow

[0957] 1. Enter your pre-medical interview information

[0958] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms, such as when they first began to show symptoms, etc. For example, the user inputs information such as "I have a headache" or "I've had a fever since yesterday."

[0959] 2. Saving pre-interview information

[0960] The server receives the pre-interview information sent by the user and stores it in a database for use in subsequent processing.

[0961] 3. Doctor Question Generation

[0962] The server uses the generative AI model to generate questions for the doctor to ask during the consultation based on the pre-medical interview information. For example, if a user enters "I have a headache," the server generates questions such as "How bad is your headache?" and "What's your fever?"

[0963] 4. Accepting questions from patients

[0964] A user uses a terminal to input a question to a doctor, such as "What kind of cold medicine should I take?" The server receives these questions and stores them in a database.

[0965] 5. Creating a medical record

[0966] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the user's questions. The medical record includes the user's symptoms, the doctor's questions, and the user's questions. This medical record is shared with the doctor.

[0967] 6. Using the Emotion Engine

[0968] If the user uses voice input or a webcam, the server uses an emotion engine to infer the user's emotions. For example, when the user says "I have a headache," the server infers emotions such as "pain" and "suffering" from the voice, and emotions such as "distress" and "discomfort" from the webcam footage. Based on these emotions, the doctor's questions are adjusted.

[0969] 7. Information sharing with doctors

[0970] The server then shares the created medical records and the results of the emotion engine with the doctor. The doctor will use these as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical records, such as "How bad is your headache?"

[0971] Examples and prompts

[0972] As a specific example, consider the case where a user complains of cold symptoms.

[0973] User input: "My head hurts."

[0974] Server action:

[0975] Estimate emotions such as "pain" and "suffering" from voice input.

[0976] Estimates emotions such as "pain" and "discomfort" from webcam footage.

[0977] Estimate emotions such as "pain" and "suffering" from text input.

[0978] Based on this information, it generates specific questions such as, "How bad is your headache?"

[0979] Example prompt:

[0980] If a patient complains of a "headache," infer their emotion from voice, facial expressions, and text and generate appropriate questions.

[0981] This system will enable doctors to grasp the user's condition more accurately and provide more efficient medical care.

[0982] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0983] Step 1:

[0984] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms and the time of onset. For example, they input information such as "I have a headache" or "I've had a fever since yesterday." The input data is sent from the terminal to the server.

[0985] Step 2:

[0986] The server receives the pre-interview information sent by the user. The received information is stored in the database. Specifically, the server connects to the database and stores the user's input data in an appropriate format. After the storage is completed, the server records the status of successful storage in the log.

[0987] Step 3:

[0988] The server uses the generative AI model to generate questions that the doctor will ask at the time of the visit based on the pre-medical interview information. For example, if the user inputs "I have a headache," the server inputs the prompt "I have a headache" into the generative AI model and generates questions such as "How bad is your headache?" and "What's your fever?" The generated questions are stored in a database.

[0989] Step 4:

[0990] A user uses a terminal to input a question to a doctor. For example, a question such as "What cold medicine should I take?" The input question is sent from the terminal to the server. The server receives the question and stores it in a database.

[0991] Step 5:

[0992] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the questions from the user. Specifically, the server retrieves the necessary information from the database and integrates the information according to the format of the medical record. The created medical record is saved in the database.

[0993] Step 6:

[0994] If the user uses voice input or a webcam, the server uses the emotion engine to estimate the user's emotion. For example, when the user says "I have a headache," the server inputs the voice data into the emotion engine to estimate the emotion of "pain" or "suffering." It also estimates the emotion of "distress" or "discomfort" from the webcam footage. The estimated emotion data is stored in a database.

[0995] Step 7:

[0996] The server communicates the created medical record and the results of the emotion engine to the doctor. Specifically, the server sends the medical record and emotion data to the doctor's terminal. The doctor uses this as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical record, such as "How bad is your headache?"

[0997] (Application example 2)

[0998] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0999] Conventional medical systems require a lot of time and effort to accurately grasp the symptoms of patients, making it difficult to monitor the health of passengers in real time, especially in autonomous vehicles. In addition, when a passenger complains of feeling unwell, it is necessary to generate appropriate questions and respond quickly, but current systems do not do this sufficiently. This can lead to the inability to properly manage the passenger's health, which can delay emergency responses.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1001] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the result of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for performing voice recognition, a means for performing emotion analysis, a means for acquiring camera images, and a means for adjusting the questions based on the result of the emotion analysis. This makes it possible to monitor the health condition of a passenger in an autonomous vehicle in real time, and to generate appropriate questions and quickly respond when an abnormality is detected.

[1002] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, and is the process of collecting information such as symptoms and medical history.

[1003] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[1004] A "medical record" is a document that compiles a patient's medical records and includes details of the examination and prescriptions.

[1005] "Speech recognition" is the technology that converts speech into text.

[1006] "Sentiment analysis" is a technology that infers emotions from text, voice, facial expressions, etc.

[1007] "Camera video" refers to video data captured by a camera.

[1008] "Question tailoring" is the process of generating or modifying appropriate questions based on the results of sentiment analysis.

[1009] An "autonomous vehicle" is a vehicle that can drive automatically without a driver.

[1010] "Passenger health condition" refers to a passenger's physical condition, medical condition, or other health-related condition.

[1011] "Real-time monitoring" means monitoring the current state instantly.

[1012] A system for carrying out the present invention is configured as follows: The server includes means for conducting a preliminary interview, means for generating questions for a doctor based on the result of the preliminary interview, means for creating a medical record by integrating the questions of the doctor and questions from the patient, means for linking the medical record to the doctor, means for performing voice recognition, means for performing emotion analysis, means for acquiring camera images, and means for adjusting the questions based on the result of the emotion analysis.

[1013] Hardware and Software Used

[1014] Hardware: microphones, webcams, on-board computers in autonomous vehicles

[1015] Software: speech_recognition library, pipeline of transformers library, cv2 (OpenCV)

[1016] Data processing and calculation

[1017] 1. Speech recognition: The server captures the audio from the microphone and converts it to text using the speech_recognition library.

[1018] 2. Sentiment analysis: The server performs sentiment analysis on the retrieved text using a pipeline from the transformers library to estimate the sentiment.

[1019] 3. Acquiring camera images: The server acquires images from the webcam and displays them in real time using cv2 (OpenCV).

[1020] 4. Question Adjustment: The server generates or modifies appropriate questions based on the results of sentiment analysis.

[1021] Examples

[1022] For example, if a passenger says "I feel sick," the server recognizes the voice and performs emotion analysis. If the emotion is determined to be "negative," the server generates the question, "Which part hurts?" This makes it possible to monitor the health of passengers in autonomous vehicles in real time and respond quickly if an abnormality is detected.

[1023] Examples of prompt statements

[1024] If the user says "I feel sick", do sentiment analysis and if the sentiment is determined to be "negative", generate the question "Which part hurts?"

[1025] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1026] Step 1:

[1027] The server acquires voice from the microphone. When the user says "I feel sick," the voice data is input. The server converts the voice data into text using the speech_recognition library. Through this process, the voice data is output as text data.

[1028] Step 2:

[1029] The server uses the text data obtained in step 1 as input and performs sentiment analysis using the pipeline of the transformers library. The sentiment analysis model estimates emotions from the text data and outputs the results. For example, if the text is "feeling bad," the emotion is output as "negative."

[1030] Step 3:

[1031] The server acquires video data from a webcam. The camera video is acquired in real time and processed using cv2 (OpenCV). The video data is used to analyze the user's facial expressions. The analysis results are output as emotion data estimated from the user's facial expressions.

[1032] Step 4:

[1033] The server integrates the emotion data obtained in steps 2 and 3 to perform an overall emotion evaluation. For example, if the voice is judged to be "negative" and the facial expression is judged to be "painful," the overall emotion evaluation will be "very bad." This overall evaluation is output.

[1034] Step 5:

[1035] The server generates an appropriate question based on the overall emotion evaluation. Using a generative AI model, a prompt sentence is input and an appropriate question is generated. For example, if you say "I feel bad" and the emotion is determined to be "negative," the question "Which part hurts?" is generated. This question is output.

[1036] Step 6:

[1037] The server presents the generated questions to the user. When the user enters an answer, the answer data is sent to the server. The server integrates the answer data into the medical record and communicates it to the doctor. This process updates the medical record and provides it to the doctor.

[1038] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1039] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1040] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1041] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1042] [Third embodiment]

[1043] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1044] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1045] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1046] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1047] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1048] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1049] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1050] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1051] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1052] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1053] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1054] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1055] "Example 1"

[1056] The present invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and takes part in a preliminary interview by online automatic transcription with an avatar or by filling out a form. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1057] "Example 2"

[1058] As a specific example, if a patient complains of cold symptoms, the preliminary interview will ask details about the symptoms and when they began. Based on this, the system will generate questions that the doctor is likely to ask at the time of the consultation, such as "What's your fever?" or "Do you have a cough?" It will also accept questions from the patient, such as "What cold medicine should I take?" These questions are integrated into a medical record and shared with the doctor. The doctor will use this as a reference when examining the patient, ensuring efficient medical care.

[1059] The process flow of each embodiment will be described below.

[1060] "Example 1"

[1061] Step 1: The patient accesses a dedicated URL.

[1062] Step 2: Conduct a pre-interview with an avatar using online automated transcription or form filling.

[1063] Step 3: Based on the results of the pre-interview, the system generates questions that the doctor is likely to ask during the visit.

[1064] Step 4: Also receive questions from the patient and consolidate these questions to create a medical record.

[1065] Step 5: The created medical record is shared with the doctor, who will use it as a reference when examining the patient.

[1066] "Example 2"

[1067] Step 1: If the patient complains of cold symptoms, the preliminary interview involves asking about details of the symptoms and when they began.

[1068] Step 2: Based on this, the system generates questions that a doctor might ask during a consultation, such as "What's your fever?" or "Do you have a cough?"

[1069] Step 3: Also, answer any questions the patient may have, such as "What cold medicine should I take?"

[1070] Step 4: Integrate these questions into the patient's chart and communicate with the doctor.

[1071] Step 5: The doctor will use this as a reference when examining the patient, ensuring efficient medical treatment.

[1072] Example 1

[1073] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1074] In conventional medical systems, there was a problem that it was time-consuming for medical staff to ask questions before patients visit the clinic, making it difficult to conduct efficient examinations. In addition, it was difficult to accurately grasp the details of the patient's questions and symptoms, which could lead to a decline in the quality of the examination. Furthermore, the doctor could not quickly obtain the information necessary for the examination, which sometimes prolonged the examination time. To solve these problems, a system was needed that would improve the efficiency of the pre-examination and enable doctors to quickly obtain the information necessary for the examination.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1076] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for extracting keywords and symptoms using natural language processing, and a means for generating questions for the doctor using a generative AI model. This makes it possible to improve the efficiency of the preliminary interview and allows the doctor to quickly obtain information required for the examination, thereby improving the quality of the examination and shortening the examination time.

[1077] A "pre-medical interview" is a process in which a patient enters their symptoms and condition before visiting a doctor, and medical professionals collect the necessary information before the examination.

[1078] The "doctor's questions" are questions that the doctor asks the patient during the consultation, which are generated based on the results of the preliminary interview.

[1079] A "medical record" is a medical document that records a patient's symptoms, examination results, treatment details, etc.

[1080] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[1081] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new information or data.

[1082] "Keywords" are important words or phrases within text data that are extracted using natural language processing.

[1083] A "symptom" is any physical or mental abnormality or discomfort experienced by a patient.

[1084] An "avatar" is a virtual character that allows users to interact online.

[1085] "Online automatic transcription" is a technology that converts audio into text in real time.

[1086] "Form entry" is a method in which a user manually enters text data into fields on a web page.

[1087] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1088] Hardware and Software Used

[1089] This system is realized using the following hardware and software.

[1090] Hardware: Servers, terminals (smartphones, PCs, etc.)

[1091] Software: Python, natural language processing libraries (NLTK, spaCy), database manipulation libraries (SQLAlchemy), web interface framework (Flask), generative AI models (e.g., OpenAI's GPT-3)

[1092] Program Processing

[1093] The user (patient) accesses a dedicated URL. Once accessed, an avatar appears on the device and the preliminary interview begins through online automatic transcription or form input. Specifically, the text data entered by the user is sent from the device to the server.

[1094] The server analyzes the received text data using natural language processing libraries (NLTK and spaCy) to extract keywords and symptoms. Based on the results of this analysis, a generative AI model (for example, OpenAI's GPT-3) is used to generate questions that the doctor may ask during the consultation.

[1095] In addition, the terminal accepts additional questions from the user, and the questions entered on the terminal are sent back to the server, which then consolidates these questions and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner).

[1096] The created medical records are sent from the server to the doctor's terminal, and the doctor uses the records as a reference when examining the patient.

[1097] Examples of concrete examples and prompts

[1098] As a concrete example, consider the case where a user inputs, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." This text data is sent from the device to the server, and the server uses a natural language processing library to extract keywords such as "headache" and "when I wake up in the morning."

[1099] Next, feed the generative AI model the following prompt:

[1100] User symptoms: Headache, severe pain upon waking in the morning

[1101] Generate questions your doctor may have during a consultation.

[1102] A generative AI model might generate questions like:

[1103] 1. How often do your headaches occur?

[1104] 2. How severe is your headache?

[1105] 3. Do you have any other symptoms besides headaches?

[1106] These questions and any additional questions from the user are integrated to create a medical record in a format suitable for the electronic medical record system. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1107] In this way, the system acts as a substitute for medical professionals to support patients.

[1108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1109] Step 1:

[1110] The user accesses a special URL.

[1111] The user accesses the dedicated URL provided by the medical institution. For example, the user inputs the URL using a browser on a smartphone or computer. Input: Dedicated URL. Output: Display of the web page.

[1112] Step 2:

[1113] The terminal will display an avatar and begin the preliminary interview.

[1114] The terminal displays a web page based on the accessed URL. This web page displays an avatar and prompts the user to start the pre-interview. Input: Dedicated URL. Output: Display of avatar and start of pre-interview.

[1115] Step 3:

[1116] The user inputs text data.

[1117] The user inputs his / her symptoms and condition in text format according to the avatar's instructions. For example, the user might input, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." Input: User's symptoms and condition. Output: Text data.

[1118] Step 4:

[1119] The terminal transmits the input data to the server.

[1120] The terminal sends the text data entered by the user to the server. This transmission is done securely using the HTTPS protocol. Input: Text data. Output: Data transmission to the server.

[1121] Step 5:

[1122] The server performs natural language processing to extract keywords and symptoms.

[1123] The server analyzes the received text data using a natural language processing library (NLTK or spaCy). For example, it extracts keywords such as "headache" and "when waking up in the morning." Input: Text data. Output: Extracted keywords and symptoms.

[1124] Step 6:

[1125] The server uses the generative AI model to generate questions for the doctor.

[1126] The server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-3) based on the extracted keywords, and generates questions that a doctor may expect to ask when visiting the patient. For example, a prompt sentence such as "User's symptoms: headache, severe pain when waking up in the morning. Please generate questions that a doctor may expect to ask when visiting the patient" can be used. Input: Extraction results of keywords and symptoms. Output: Questions asked by the doctor.

[1127] Step 7:

[1128] The user enters a follow-up question.

[1129] The user inputs additional questions or information for the generated question. For example, the user answers "About three times a week" to the question "How often do you get headaches?". Input: Doctor's question. Output: Additional questions or information.

[1130] Step 8:

[1131] The terminal sends a follow-up question to the server.

[1132] The terminal sends any additional questions or information entered by the user back to the server. Input: Additional questions or information. Output: Sending data to the server.

[1133] Step 9:

[1134] The server consolidates the questions and creates a medical record.

[1135] The server merges the initial interview data and the follow-up question data and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner). Input: Initial interview data, follow-up question data. Output: Medical record.

[1136] Step 10:

[1137] The server links the medical records to the doctor's terminal.

[1138] The server links the created medical records to the doctor's terminal. The doctor can view the medical records through a dedicated medical system. Input: Medical records. Output: Linking medical records to the doctor's terminal.

[1139] Step 11:

[1140] The doctor will examine the patient using the medical records.

[1141] The doctor will consult the linked medical records when examining the patient. The medical records contain detailed information about the user's symptoms and the results of the preliminary interview, allowing the examination to proceed smoothly. Input: Medical records. Output: Examination results.

[1142] (Application example 1)

[1143] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1144] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, and doctors often lacked the information they needed to examine patients. In addition, there was no way to understand the patient's symptoms or questions in advance when consulting about medicines at pharmacies or drug stores, making it difficult to respond efficiently. This resulted in longer waiting times for patients and increased burdens on medical professionals.

[1145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1146] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to a doctor, a means for supporting preliminary interviews and consultations about medicines at pharmacies and drug stores, and a means for generating questions using a generative AI model. This allows a patient to collect necessary information before visiting a medical institution or a pharmacy, enabling efficient examinations and consultations.

[1147] A "pre-medication interview" is a process in which medical professionals collect necessary information in advance by having patients enter their symptoms and questions online before visiting a medical institution or pharmacy.

[1148] The "means for generating questions for a doctor" is a system that automatically generates questions that a doctor should ask during a consultation based on the results of a preliminary interview.

[1149] The "means of creating a medical record" refers to a system that integrates the results of the preliminary interview, questions from the doctor, and questions from the patient to create a medical record that compiles all the information necessary for the examination.

[1150] "Means of sharing medical records with doctors" refers to a system that provides the created medical records to doctors so that they can refer to them during consultations.

[1151] The "means of supporting advance medical interviews and medication consultations at pharmacies and drug stores" is a system that allows patients to input their symptoms and questions in advance at pharmacies and drug stores, enabling pharmacists to respond efficiently.

[1152] A "generative AI model" is an algorithm that uses artificial intelligence to generate appropriate questions and answers based on input data.

[1153] The "means for generating questions" refers to a system that uses a generative AI model to automatically generate appropriate questions based on the results of the preliminary interview.

[1154] The system for implementing this invention is configured as follows. First, a user accesses a dedicated URL using a terminal such as a smartphone or a personal computer. After accessing the URL, a preliminary interview is conducted by online automatic transcription with an avatar or by filling out a form. The results of this preliminary interview are sent to the server.

[1155] Based on the results of the preliminary interview, the server generates questions for the doctor using a generative AI model that uses advanced natural language processing algorithms, such as OpenAI's GPT-3. The generated questions are stored on the server.

[1156] Next, the server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. This medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1157] In addition, the system also has a function to support pre-medication and drug consultations at pharmacies and drug stores. Before going to a pharmacy or drug store, the user performs a pre-medication interview using a similar procedure, and the generative AI model generates questions for the pharmacist based on the results. This allows the pharmacist to efficiently respond to patient consultations.

[1158] For example, if a user enters, "I have a headache and a sore throat," the generative AI model generates questions such as, "When did the headache start?" and "How bad is your throat?" These questions are integrated into the medical record and provided to doctors and pharmacists.

[1159] An example of a prompt is as follows:

[1160] "Generate questions the pharmacist might have based on the following symptoms: I have a headache and a sore throat."

[1161] In this way, a system can be realized that enables efficient medical examinations and consultations based on the results of the preliminary interview.

[1162] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1163] Step 1:

[1164] The user accesses a dedicated URL using a device such as a smartphone or PC. The user completes a preliminary interview by using an avatar to automatically transcribe online or by filling out a form. The data entered is the user's symptoms and questions. This data is then sent from the device to the server.

[1165] Step 2:

[1166] The server uses a generative AI model to generate questions for the doctor based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated doctor's question.

[1167] Step 3:

[1168] The server creates a medical record by integrating the generated doctor's questions and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[1169] Step 4:

[1170] The server transmits the created medical record to the doctor. The medical record is provided so that the doctor can refer to it when examining the patient. The input is the integrated medical record, and the output is the medical record provided to the doctor.

[1171] Step 5:

[1172] Before a user goes to a pharmacy or drugstore, the user performs a pre-medical interview using a similar procedure. The user inputs symptoms and questions using a terminal and sends them to the server. The input data is the user's symptoms and questions.

[1173] Step 6:

[1174] The server uses a generative AI model to generate questions for the pharmacist based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated questions for the pharmacist.

[1175] Step 7:

[1176] The server creates a medical record by integrating the generated questions from the pharmacist and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[1177] Step 8:

[1178] The server communicates the created chart to the pharmacist. The chart is provided to the pharmacist so that the pharmacist can refer to it during consultation. The input is the integrated chart, and the output is the chart provided to the pharmacist.

[1179] Example 2

[1180] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1181] In conventional medical systems, the process from the patient's preliminary interview to the doctor's examination was not carried out efficiently, which often resulted in long consultation times. In addition, it was difficult to generate appropriate questions based on the patient's symptoms, and doctors were sometimes unable to fully grasp the necessary information before the examination. This resulted in poor efficiency in medical treatment and poor patient satisfaction.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1183] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor using a generative AI model based on the results of the preliminary interview, a means for creating a medical record by integrating the doctor's questions and questions from the patient, and a means for sharing the medical record with the doctor. This allows appropriate questions based on the patient's symptoms to be automatically generated, enabling the doctor to grasp the necessary information before the consultation. This improves the efficiency of medical care and also improves patient satisfaction.

[1184] A "pre-medical interview" is the process in which a patient provides information about their symptoms, medical history, etc. before an appointment.

[1185] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate outputs for a specific task.

[1186] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[1187] "Patient Questions" are questions that patients ask their physicians about their symptoms or treatment.

[1188] A "medical record" is a document that contains a patient's medical records, including symptoms, diagnoses, and treatment details.

[1189] "Collaboration" refers to different systems and processes sharing information and working together.

[1190] A "web page" is a page of information accessible on the Internet.

[1191] An "avatar" is a character or icon that represents a user online.

[1192] "Online automatic transcription" is a technology that converts audio into text in real time.

[1193] "Form filling" is the process by which a user enters information into fields on a web page.

[1194] "Medical Condition" refers to a patient's health condition or symptoms.

[1195] "Procedure" refers to the medical treatment or care that a doctor provides to a patient.

[1196] "Treatment time" refers to the time it takes a doctor to treat a patient.

[1197] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1198] This invention is a system that performs a preliminary interview, generates questions for the doctor using a generative AI model, creates a medical record by integrating the questions from the patient, and communicates with the doctor. A specific embodiment of this system is described below.

[1199] Hardware and Software Used

[1200] Hardware: Servers, terminals (PCs, tablets, smartphones, etc.)

[1201] Software: Generative AI models (e.g. GPT-4), database management systems (e.g. MySQL), web application frameworks (e.g. Django)

[1202] System Configuration

[1203] 1. Server: The server is responsible for generating questions to conduct a pre-medical interview. It uses a generative AI model (e.g. GPT-4) to generate questions based on the patient's symptoms. The server also receives answers from the user (patient) and stores them in a database. The server also integrates the collected information, creates a medical record, and notifies the doctor.

[1204] 2. Terminal: The terminal provides an interface for the user (patient) to conduct a preliminary interview. The terminal displays the questions received from the server to the user, collects the user's answers, and sends them to the server.

[1205] 3. User: The user (patient) uses the terminal to conduct a preliminary interview. The user answers the questions displayed on the terminal and, if necessary, enters his / her own questions.

[1206] Examples

[1207] 1. Question generation for pre-medical interview: The server uses a generative AI model (e.g., GPT-4) to generate questions for pre-medical interviews that are necessary when a patient complains of cold symptoms. For example, questions such as "When did your symptoms start?" and "What symptoms are you experiencing?" are generated. The generated questions are stored in a database.

[1208] 2. Displaying questions on the terminal: The terminal retrieves the questions from the pre-medical interview from the server and displays them to the user. For example, questions such as "What is your fever?" and "Do you have a cough?" are displayed on the terminal screen.

[1209] 3. Collect user answers: The user answers the questions displayed on the terminal. For example, they input information such as "My fever is 37.5 degrees" or "I have a cough." The user can also input their own questions. For example, they input questions such as "What cold medicine should I take?"

[1210] 4. Sending and storing answers: The terminal sends the user's answers to the server. The server stores the received answers in a database. This allows the centralized management of patient information.

[1211] 5. Creating a medical record: The server consolidates the collected information and creates a medical record. The medical record includes details of the patient's symptoms, when they first appeared, questions that the doctor anticipates, and questions from the patient. The created medical record is stored in a database.

[1212] 6. Medical record sharing with doctor: The server sends the created medical record to the doctor's terminal. The doctor performs the examination using the received medical record as a reference. For example, by looking at the medical record, the doctor can know in advance information such as "The patient has a fever of 37.5 degrees and is coughing." This allows for more efficient medical treatment.

[1213] Examples of prompt statements

[1214] "A patient is complaining of cold symptoms. Using the information below, generate questions that a doctor might ask during a consultation. Please include details of the patient's symptoms and when they began."

[1215] This system allows doctors to understand patients' symptoms in advance, improving the efficiency of medical treatment and increasing patient satisfaction.

[1216] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1217] Step 1:

[1218] The server generates the pre-screening questions.

[1219] Input: Information about the patient's initial symptoms (e.g. cold symptoms)

[1220] Data processing: Input a prompt sentence into a generative AI model (e.g. GPT-4) to generate appropriate questions.

[1221] Output: Generated questions (e.g., "When did your symptoms start?", "What symptoms are you experiencing?")

[1222] Specific operation: The server inputs a prompt sentence into the generative AI model: "The patient is complaining of cold symptoms. Based on the information below, please generate questions that a doctor may expect to ask when visiting the patient. Please also include details of the patient's symptoms and when they began." and stores the generated questions in a database.

[1223] Step 2:

[1224] The terminal obtains the questions for the preliminary medical examination from the server and displays them to the user.

[1225] Input: A generated query from the server

[1226] Data processing: Converting the question into a format for display to the user.

[1227] Output: The question that is displayed to the user (e.g. "What's your fever?", "Do you have a cough?")

[1228] Specific operation: The terminal displays the questions obtained from the server on the screen and formats them in a way that makes it easy for the user to answer.

[1229] Step 3:

[1230] The user answers the questions displayed on the terminal.

[1231] Input: the question displayed on the terminal

[1232] Data processing: Enter the user's answers into the input form.

[1233] Output: User's answer (e.g. "My temperature is 37.5 degrees" or "I have a cough")

[1234] Specific operation: The user inputs text in response to the questions displayed on the terminal, and if necessary, inputs his or her own question.

[1235] Step 4:

[1236] The terminal transmits the user's answer to the server.

[1237] Input: User's answer

[1238] Data processing: The response data is converted into a format for sending to the server.

[1239] Output: Answer data sent to the server

[1240] Specific operation: The terminal transmits the answers entered by the user to the server in real time.

[1241] Step 5:

[1242] The server stores the received answers in a database.

[1243] Input: Answer data sent from the device

[1244] Data processing: Convert the response data into a format that can be stored in the database.

[1245] Output: Answer data stored in a database

[1246] Specific operations: The server stores the received response data in a database for subsequent processing.

[1247] Step 6:

[1248] The server consolidates the collected information and creates a medical record.

[1249] Input: Answer data stored in the database

[1250] Data processing: Integrate the response data and create a medical record.

[1251] Output: Created medical record

[1252] Specific operation: The server creates a medical record by integrating the patient's symptoms and questions based on the information stored in the database.

[1253] Step 7:

[1254] The server transmits the created chart to the doctor's terminal.

[1255] Input: Created medical record

[1256] Data processing: Converting medical record data into a format for sending to the doctor's terminal.

[1257] Output: Medical record sent to doctor's terminal

[1258] Specific operation: The server sends the created medical record to the doctor's terminal so that the doctor can review the information before the examination.

[1259] Step 8:

[1260] The doctor will use the medical records he or she receives to conduct the examination.

[1261] Input: Medical record sent to doctor's terminal

[1262] Data processing: Check the contents of the medical records and extract the information necessary for the examination.

[1263] Output: Examination results and treatment plan

[1264] Specific actions: The doctor reviews the patient's medical records to understand the patient's symptoms and questions, and conducts an efficient examination.

[1265] (Application example 2)

[1266] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1267] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, which often led to long consultation times for doctors. In addition, even in brick-and-mortar stores, there was no way for customers to ask questions before visiting the store, which led to inefficient responses from store staff.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1269] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for receiving symptoms reported by a customer as an input and generating questions to be asked by store staff, a means for generating questions for the customer to ask the store staff, a means for integrating the generated questions to create a medical record, and a means for outputting the created medical record. This enables the efficiency of medical examinations at medical institutions and the efficiency of customer service at physical stores.

[1270] A "pre-medical interview" is a process in which patients or customers enter their symptoms and requests in advance before visiting a medical institution or physical store.

[1271] "Doctor's questions" are questions that the doctor should ask the patient based on the results of the preliminary interview.

[1272] "Patient questions" are questions that patients want to ask their doctors.

[1273] A "medical record" is a record that integrates the symptoms and questions of patients or customers, as well as questions from doctors and store staff.

[1274] "Symptoms reported by customers" refers to the physical discomfort and requests felt by customers who visit physical stores.

[1275] "Questions that store staff should ask" are questions that store staff should ask customers based on the customer's symptoms and requests.

[1276] "Questions that customers should ask store staff" are questions that customers would like to ask store staff.

[1277] A "generated question" is a question that is automatically generated based on the results of a preliminary interview.

[1278] The "means of creating a medical record" refers to the process of creating a medical record by integrating the results of the preliminary interview and the questions generated.

[1279] "Means for outputting medical records" refers to the process of providing the created medical records to doctors and store staff.

[1280] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for receiving as input symptoms reported by a customer and generating questions to be asked by store staff, means for generating questions for the customer to ask the store staff, means for integrating the generated questions to create a medical record, and means for outputting the created medical record.

[1281] The server receives the results of the preliminary interview and uses a generative AI model to generate questions for the doctor and store staff. Specifically, it uses OpenAI's API to generate appropriate questions based on the results of the preliminary interview. The generated questions are processed using a Python program and integrated into the medical record.

[1282] The terminal (smartphone or tablet) provides an interface for the user to conduct a preliminary medical interview. The user accesses a dedicated URL and conducts the preliminary medical interview by online automatic transcription with an avatar or by filling out a form. The user's symptoms and requests are then sent to the server.

[1283] When a user inputs the symptoms they are complaining about, the server uses a generative AI model to generate questions that store staff should ask, or questions that customers should ask store staff. The generated questions are integrated into a medical record and provided to store staff.

[1284] As a specific example, the prompt sentence when a customer complains of cold symptoms is shown below.

[1285] "A customer is complaining of the following symptoms: Cold symptoms. Generate questions for store staff to ask."

[1286] When this prompt is executed, the server will generate a question like this:

[1287] "What's your fever? Do you have a cough?"

[1288] It also generates questions for customers to ask store staff.

[1289] "What cold medicine should I take?"

[1290] This will enable more efficient medical examinations at medical institutions and more efficient customer service at physical stores.

[1291] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1292] Step 1:

[1293] The user accesses a dedicated URL using a device (smartphone or tablet). The user completes a preliminary interview by using an avatar to automatically transcribe text online or by filling out a form. The input data is the user's symptoms and requests. The output data is sent to the server as the results of the preliminary interview.

[1294] Step 2:

[1295] The server analyzes the received results of the pre-interview. The input data is information about the user's symptoms and requests. The server uses a generative AI model (e.g., OpenAI's API) to generate questions for the doctor and questions for the store staff based on the results of the pre-interview. The output data are the generated questions.

[1296] Step 3:

[1297] The server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. The input data are the generated questions and the results of the preliminary interview. The server integrates these data and configures them as a medical record. The output data is the created medical record.

[1298] Step 4:

[1299] The server shares the created medical record with the doctor. The input data is the created medical record. The server sends the medical record to the doctor's terminal so that the doctor can refer to it when examining the patient. The output data is the medical record provided to the doctor.

[1300] Step 5:

[1301] When a user inputs the symptoms they are experiencing, the server uses a generative AI model to generate questions for store staff to ask. The input data is information about the user's symptoms. The server uses a generative AI model to generate appropriate questions. The output data is the questions that store staff should ask.

[1302] Step 6:

[1303] The server generates questions that customers should ask store staff. The input data is information about the user's symptoms. The server uses a generative AI model to generate questions that customers should ask. The output data is questions that customers should ask store staff.

[1304] Step 7:

[1305] The server integrates the generated questions to create a medical record. The input data are the generated questions. The server integrates these questions and configures them as a medical record. The output data is the created medical record.

[1306] Step 8:

[1307] The server outputs the created medical record. The input data is the created medical record. The server provides the medical record to the store staff so that they can refer to it when dealing with customers. The output data is the medical record provided to the store staff.

[1308] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1309] "Example 1"

[1310] In one embodiment of the present invention, an emotion engine is introduced. The emotion engine recognizes emotions from the user's voice, facial expression, or text input. Specifically, when the user conducts a pre-interview, the emotion is estimated from the tone, strength, and speed of the user's voice. If the user is using a webcam, the emotion is also estimated from facial expression. Furthermore, if the user inputs text, the emotion is estimated from the content and expression of the text.

[1311] "Example 2"

[1312] As a concrete example, consider a case where a user complains of cold symptoms. When the user says "My head hurts," emotions such as "pain" or "suffering" are inferred from the voice. Also, if the user is using a webcam, emotions such as "pain" or "discomfort" are inferred from the facial expression. Furthermore, if the user inputs the text "My head hurts," emotions such as "pain" or "suffering" are also inferred from that text. Based on the recognition results of these emotion engines, the doctor's questions are adjusted. For example, a more specific question such as "How bad is your headache?" is generated. This allows the doctor to grasp the user's condition more accurately and perform an appropriate examination.

[1313] The process flow of each embodiment will be described below.

[1314] "Example 1"

[1315] Step 1: The user accesses the URL for the pre-consultation questionnaire.

[1316] Step 2: The user answers the medical questions, while the emotion engine analyzes the user's voice, facial expressions, or text input.

[1317] Step 3: The emotion engine recognizes the user's emotions and shares the results with the doctor.

[1318] "Example 2"

[1319] Step 1: When a user reports cold symptoms, their voice, facial expressions, or text input are analyzed by the emotion engine.

[1320] Step 2: The emotion engine recognizes the user's emotions, such as "pain" or "suffering."

[1321] Step 3: Based on the emotion engine's recognition results, the doctor's question is adjusted, for example, a more specific question is generated, such as "How bad is your headache?"

[1322] Example 1

[1323] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1324] In the conventional medical system, when a patient conducts a pre-consultation before visiting a doctor, medical staff must directly respond to the patient, which takes time and effort. It is also difficult to accurately grasp the patient's emotions, and there are cases where the information required for an appropriate examination is insufficient. Furthermore, doctors are unable to prepare questions to ask at the time of the consultation in advance, which can reduce the efficiency of examinations.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1326] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from the user's voice, facial expression, or text input, and a means for generating questions for the doctor using a generative AI model. This allows the patient to efficiently conduct a preliminary interview before a medical examination, and the doctor to prepare appropriate questions in advance. In addition, by accurately grasping the patient's emotions, information for a more appropriate medical examination can be provided.

[1327] A "pre-medical interview" is an interview conducted by a patient before visiting a doctor, and is the process of collecting information such as the patient's symptoms and medical history.

[1328] "Doctor's questions" are questions that a doctor asks a patient, and are intended to understand the patient's symptoms and condition in detail.

[1329] A "medical record" is a document that compiles a patient's medical records and serves as a reference for doctors when examining and treating patients.

[1330] "Means for recognizing emotions" refers to technology or devices for inferring emotions from a user's voice, facial expression, or text input.

[1331] A "generative AI model" is a model that uses artificial intelligence to generate text and questions, and examples include GPT-3.

[1332] An "avatar" is a virtual character that interacts with a user and assists in online communication.

[1333] "Online automatic transcription" is a technology that converts voice input into text, automatically recording what a user says as text.

[1334] "Form entry" refers to a method in which a user enters information into a web form, and is used during a pre-consultation interview.

[1335] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1336] Hardware and Software Used

[1337] The system's program is developed using Python. The main libraries used are NLTK for natural language processing, TextBlob for sentiment analysis, SpeechRecognition for voice recognition, and OpenCV for facial expression recognition. Advanced natural language generation models such as GPT-3 are used as generative AI models.

[1338] Program processing explanation

[1339] When a user accesses a dedicated URL, the server displays an avatar to the user, which then interacts with the user to ask questions. The user can answer the questions by voice input, text input, or form input.

[1340] The server receives the user's input data and performs the following operations:

[1341] For speech input: Use the SpeechRecognition library to convert speech to text.

[1342] For text input: Process it as text data as is.

[1343] For form filling: Collect data for each field in the form.

[1344] The server then analyzes the user's emotions using an emotion engine, which estimates emotions in the following way:

[1345] For voice input: Analyzes the tone, volume and speaking rate of the voice.

[1346] For facial expression input: OpenCV is used to analyze the video captured by the webcam and estimate emotions from facial expressions.

[1347] For text input: Use TextBlob to infer sentiment from the text content and expression.

[1348] Based on this data, the server generates questions that the doctor is likely to ask during the consultation. It uses a generative AI model (e.g., GPT-3) to generate questions based on the user's pre-medical interview results. It also accepts questions from the user and integrates these questions to create a medical record.

[1349] The created chart is sent from the server to the doctor, who then uses it as a reference when examining the patient.

[1350] Examples of concrete examples and prompts

[1351] As a specific example, consider a case where a user complains of a headache and inputs, by voice, "I've been having a really bad headache lately."

[1352] The server does the following:

[1353] 1. Convert speech to text using the SpeechRecognition library.

[1354] 2. Use TextBlob to infer the user's emotion (e.g., distress or anxiety) from the text "I have a bad headache."

[1355] 3. Use GPT-3 to generate questions a doctor might ask during a visit, such as "When did your headache start?" or "Where does it hurt?"

[1356] Example prompt:

[1357] If a user types, "I've been having really bad headaches lately," generate questions a doctor might ask during a visit.

[1358] In this way, the system acts as a substitute for medical professionals to support patients.

[1359] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1360] Step 1:

[1361] The user accesses a special URL.

[1362] Input: User's access request

[1363] Output: Login screen displayed

[1364] Specific operation: A user opens a browser and accesses a dedicated URL. The server receives the user's access request and displays a login screen.

[1365] Step 2:

[1366] The user enters login information and is authenticated.

[1367] Input: User ID and password

[1368] Output: Authentication result (success or failure)

[1369] Specific operation: The user enters the ID and password and clicks the "Login" button. The server accesses the database and verifies the entered ID and password. If authentication is successful, an avatar is displayed to the user.

[1370] Step 3:

[1371] The user conducts a preliminary interview through dialogue with an avatar.

[1372] Input: User voice input, text input, or form input

[1373] Output: Pre-interview data (text format)

[1374] Specific operation: The user speaks to the avatar, saying, "I've been having a really bad headache lately." The server uses the SpeechRecognition library to convert the speech to text, and obtains the text data, "I've been having a really bad headache lately."

[1375] Step 4:

[1376] The server uses an emotion engine to analyze the user's emotions.

[1377] Input: Pre-medical interview data (text format)

[1378] Output: Sentiment analysis results (e.g., distress, anxiety)

[1379] Specific operation: The server uses TextBlob to analyze the user's emotions from the text "I've been having a lot of headaches lately." For example, it estimates emotions such as pain and anxiety.

[1380] Step 5:

[1381] The server uses the generative AI model to generate questions for the doctor.

[1382] Input: Pre-interview data and sentiment analysis results

[1383] Output: Doctor's questions (text format)

[1384] Specific operation: The server inputs a prompt sentence to the generative AI model (GPT-3). Example of a prompt sentence: "If a user inputs, 'I've been having a lot of headaches lately,' please generate a question that a doctor would ask when visiting the doctor." The server takes the generated question and generates a question such as, "When did your headache start?" or "Where does it hurt?"

[1385] Step 6:

[1386] The server creates a medical record by integrating the results of the preliminary interview with the generated doctor's questions.

[1387] Input: Pre-medication data and doctor's questions

[1388] Output: Medical record (text format)

[1389] Specific operation: The server integrates the user's preliminary interview results with the generated doctor's questions and creates a medical record.

[1390] Step 7:

[1391] The medical records created by the server are shared with the doctor.

[1392] Input: Medical record (text format)

[1393] Output: Send medical record to doctor

[1394] Specific operation: The server sends the created chart to the doctor. The doctor uses the chart as a reference when examining the patient.

[1395] (Application example 1)

[1396] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1397] In conventional medical systems, the patient's pre-consultation and medical record creation were done manually, which placed a heavy burden on medical staff and was inefficient. In addition, in brick-and-mortar stores, the response to customer questions and requests was insufficient, which led to a decrease in customer satisfaction. Furthermore, there was also the issue of it being difficult to appropriately recognize and respond to customer emotions.

[1398] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1399] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, a means for accepting questions and requests from customers in a physical store, recognizing emotions, and providing appropriate responses, and a means for recording questions and requests from customers and linking them to store staff. This reduces the burden on medical staff and enables efficient creation of medical records, while also enabling proper recognition of customer emotions in a physical store and providing highly satisfying responses.

[1400] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, in order to understand the patient's symptoms and medical history in advance.

[1401] "Doctor's questions" are questions that a doctor asks a patient to confirm in detail the patient's symptoms and condition.

[1402] A "medical chart" is a compilation of a patient's medical records, which serve as a reference for doctors when examining and treating patients.

[1403] An "emotion engine" is a system for recognizing emotions from a user's voice, facial expressions, or text input.

[1404] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[1405] "Customer questions and requests" are questions and requests made by customers to store staff or support robots in physical stores.

[1406] "Store staff" refers to employees who work in physical stores and provide products and services to customers.

[1407] "Online automatic transcription" is a technology that converts speech into text, automatically recording what a user says as text.

[1408] "Form input" refers to a method in which a user inputs information into an input field on a web page, and is used for things like advance interviews and questionnaires.

[1409] A system for implementing this invention includes means for conducting a preliminary interview, means for generating questions for a doctor based on the results of the preliminary interview, means for creating a medical record by integrating the doctor's questions with questions from the patient, means for linking the medical record to the doctor, means for recognizing emotions from a user's voice, facial expression, or text input using an emotion engine, means for accepting customer questions and requests in a physical store, recognizing emotions and responding appropriately, and means for recording customer questions and requests and linking them to store staff.

[1410] System Programming

[1411] The system program performs the following processing.

[1412] 1. Pre-medication:

[1413] The server will ask the user a preliminary interview via online automatic transcription with an avatar or through form entry when the user accesses a dedicated URL. The information entered by the user will be stored in a database.

[1414] 2. Doctor Question Generation:

[1415] The server generates questions that are expected to be asked by the doctor at the time of the visit based on the results of the preliminary interview. The generated questions are added to the user's medical record.

[1416] 3. Creation and collaboration of medical records:

[1417] The server creates a medical record by integrating the generated doctor's questions with the user's questions. The created medical record is shared with the doctor and used as a reference during consultations.

[1418] 4. Emotion recognition:

[1419] The server uses an emotion engine to recognize emotions from the user's voice, facial expressions, or text input. For emotion recognition, it uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model).

[1420] 5. Customer service in physical stores:

[1421] In a brick-and-mortar store, the server will receive customer questions and requests, recognize emotions, and respond appropriately. Customer questions and requests are recorded and communicated to store staff.

[1422] Hardware and Software Used

[1423] Hardware:

[1424] Webcam: Used to capture the customer's face.

[1425] Microphone: Used to capture the customer's voice.

[1426] software:

[1427] OpenCV: A library for face recognition.

[1428] SpeechRecognition: A library for speech recognition.

[1429] Transformers: Generative AI models for emotion recognition (using the Hugging Face pipeline).

[1430] Examples

[1431] For example, if a customer in a physical store asks a robot, "I want to know more about this product," the server will recognize the customer's emotions from their voice and facial expressions and respond appropriately. The customer's questions and requests are recorded and shared with the store staff.

[1432] Examples of prompt statements

[1433] "Estimate how your customers feel when they say, 'I want to know more about this product.'"

[1434] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1435] Step 1:

[1436] The server performs a preliminary interview with the user who accesses a dedicated URL through online automatic transcription with an avatar or form input. The information entered by the user is stored in a database. The input is information about the user's symptoms and medical history, and the output is the interview results stored in the database.

[1437] Step 2:

[1438] The server generates questions that are expected to be asked by the doctor at the time of the consultation based on the results of the preliminary interview. The generated questions are added to the user's medical record. The input is the results of the preliminary interview, and the output is the generated questions from the doctor. For data processing, an algorithm is used to analyze the interview results and generate appropriate questions.

[1439] Step 3:

[1440] The server creates a medical record by integrating the generated doctor's questions and the user's questions. The created medical record is shared with the doctor and used as a reference during consultations. The input is the doctor's questions and the user's questions, and the output is the integrated medical record. Data calculations include integrating the questions and formatting the medical record.

[1441] Step 4:

[1442] The server uses an emotion engine to recognize emotions from the user's voice, facial expression, or text input. Emotion recognition uses OpenCV (face recognition), SpeechRecognition (voice recognition), and Transformers (generative AI model). The input is the user's voice, facial expression, or text, and the output is the recognized emotion. Data processing involves analysis of voice and image data.

[1443] Step 5:

[1444] In a physical store, the server receives customer questions and requests, recognizes emotions, and responds appropriately. Customer questions and requests are recorded and communicated to store staff. The input is the customer's question or request, and the output is the recorded data and the response. Data calculations are performed to analyze the question or request and generate an appropriate response.

[1445] Step 6:

[1446] The server records the customer's questions and requests and shares them with the store staff. The input is the customer's questions and requests, and the output is the information shared with the store staff. Data processing involves recording and sharing of questions and requests.

[1447] Example 2

[1448] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1449] In conventional medical systems, doctors were not sufficiently prepared to ask appropriate questions based on the patient's pre-interview information, which led to problems with the efficiency of medical examinations. In addition, questions were not adjusted to take into account the patient's feelings, making it difficult to accurately grasp the patient's condition. This resulted in a decline in the quality of medical care and a decrease in patient satisfaction.

[1450] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1451] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for estimating emotions from audio input and video input, and a means for adjusting the questions of the doctor based on the emotions. This makes it possible to generate appropriate questions based on the patient's preliminary interview information and adjust the questions taking the patient's emotions into consideration.

[1452] A "pre-medical interview" is a process in which a patient inputs information about their symptoms, the time of onset, etc., before their appointment.

[1453] The "doctor's questions" are questions that the doctor asks the patient, which are generated based on the results of the preliminary interview.

[1454] A "medical chart" is a medical record that is created by integrating information from the patient's prior medical interview, questions from the doctor, and questions from the patient.

[1455] The "means for estimating emotions" refers to technology for analyzing and estimating a patient's emotions from audio and video input.

[1456] The "means for adjusting questions" refers to a technique for appropriately changing the content of questions posed to a doctor based on the estimated emotions.

[1457] "Web Page" means an information page accessible via the Internet that provides an interface for conducting a pre-consultation interview.

[1458] "Online automatic transcription" is a technology that converts voice input into text and is used as part of the pre-interview.

[1459] "Form entry" is a method by which a user enters information into fields on a web page.

[1460] A "generative AI model" is an algorithm that uses artificial intelligence to perform natural language processing and generate appropriate questions and answers.

[1461] The present invention is a system that generates questions for a doctor based on information about a patient's advance medical interview and adjusts the questions in consideration of the patient's feelings. A specific embodiment of this system will be described below.

[1462] Hardware and Software Used

[1463] Hardware

[1464] Server: Use a server machine with a powerful processor and sufficient memory.

[1465] Terminal: A device such as a PC or smartphone is used for the user to input pre-interview information.

[1466] software

[1467] Generative AI models: Using GPT-4 as an example, for natural language processing.

[1468] Database: A database is used to store pre-interview information, generated questions, and medical records.

[1469] Emotion Engine: Uses software to infer emotions from audio and video inputs.

[1470] System processing flow

[1471] 1. Enter your pre-medical interview information

[1472] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms, such as when they first began to show symptoms, etc. For example, the user inputs information such as "I have a headache" or "I've had a fever since yesterday."

[1473] 2. Saving pre-interview information

[1474] The server receives the pre-interview information sent by the user and stores it in a database for use in subsequent processing.

[1475] 3. Doctor Question Generation

[1476] The server uses the generative AI model to generate questions for the doctor to ask during the consultation based on the pre-medical interview information. For example, if a user enters "I have a headache," the server generates questions such as "How bad is your headache?" and "What's your fever?"

[1477] 4. Accepting questions from patients

[1478] A user uses a terminal to input a question to a doctor, such as "What kind of cold medicine should I take?" The server receives these questions and stores them in a database.

[1479] 5. Creating a medical record

[1480] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the user's questions. The medical record includes the user's symptoms, the doctor's questions, and the user's questions. This medical record is shared with the doctor.

[1481] 6. Using the Emotion Engine

[1482] If the user uses voice input or a webcam, the server uses an emotion engine to infer the user's emotions. For example, when the user says "I have a headache," the server infers emotions such as "pain" and "suffering" from the voice, and emotions such as "distress" and "discomfort" from the webcam footage. Based on these emotions, the doctor's questions are adjusted.

[1483] 7. Information sharing with doctors

[1484] The server then shares the created medical records and the results of the emotion engine with the doctor. The doctor will use these as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical records, such as "How bad is your headache?"

[1485] Examples and prompts

[1486] As a specific example, consider the case where a user complains of cold symptoms.

[1487] User input: "My head hurts."

[1488] Server action:

[1489] Estimate emotions such as "pain" and "suffering" from voice input.

[1490] Estimates emotions such as "pain" and "discomfort" from webcam footage.

[1491] Estimate emotions such as "pain" and "suffering" from text input.

[1492] Based on this information, it generates specific questions such as, "How bad is your headache?"

[1493] Example prompt:

[1494] If a patient complains of a "headache," infer their emotion from voice, facial expressions, and text and generate appropriate questions.

[1495] This system will enable doctors to grasp the user's condition more accurately and provide more efficient medical care.

[1496] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1497] Step 1:

[1498] The user uses a terminal to access the system's medical interview form. The user inputs details of their symptoms and the time of onset. For example, they input information such as "I have a headache" or "I've had a fever since yesterday." The input data is sent from the terminal to the server.

[1499] Step 2:

[1500] The server receives the pre-interview information sent by the user. The received information is stored in the database. Specifically, the server connects to the database and stores the user's input data in an appropriate format. After the storage is completed, the server records the status of successful storage in the log.

[1501] Step 3:

[1502] The server uses the generative AI model to generate questions that the doctor will ask at the time of the visit based on the pre-medical interview information. For example, if the user inputs "I have a headache," the server inputs the prompt "I have a headache" into the generative AI model and generates questions such as "How bad is your headache?" and "What's your fever?" The generated questions are stored in a database.

[1503] Step 4:

[1504] A user uses a terminal to input a question to a doctor. For example, a question such as "What cold medicine should I take?" The input question is sent from the terminal to the server. The server receives the question and stores it in a database.

[1505] Step 5:

[1506] The server creates a medical record by integrating the collected pre-medical interview information, the generated doctor's questions, and the questions from the user. Specifically, the server retrieves the necessary information from the database and integrates the information according to the format of the medical record. The created medical record is saved in the database.

[1507] Step 6:

[1508] If the user uses voice input or a webcam, the server uses the emotion engine to estimate the user's emotion. For example, when the user says "I have a headache," the server inputs the voice data into the emotion engine to estimate the emotion of "pain" or "suffering." It also estimates the emotion of "distress" or "discomfort" from the webcam footage. The estimated emotion data is stored in a database.

[1509] Step 7:

[1510] The server communicates the created medical record and the results of the emotion engine to the doctor. Specifically, the server sends the medical record and emotion data to the doctor's terminal. The doctor uses this as a reference when examining the patient. For example, the doctor can ask specific questions based on the medical record, such as "How bad is your headache?"

[1511] (Application example 2)

[1512] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1513] Conventional medical systems require a lot of time and effort to accurately grasp the symptoms of patients, making it difficult to monitor the health of passengers in real time, especially in autonomous vehicles. In addition, when a passenger complains of feeling unwell, it is necessary to generate appropriate questions and respond quickly, but current systems do not do this sufficiently. This can lead to the inability to properly manage the passenger's health, which can delay emergency responses.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1515] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the result of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for performing voice recognition, a means for performing emotion analysis, a means for acquiring camera images, and a means for adjusting the questions based on the result of the emotion analysis. This makes it possible to monitor the health condition of a passenger in an autonomous vehicle in real time, and to generate appropriate questions and quickly respond when an abnormality is detected.

[1516] A "preliminary medical interview" is an interview conducted before a patient visits a medical institution, and is the process of collecting information such as symptoms and medical history.

[1517] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[1518] A "medical record" is a document that compiles a patient's medical records and includes details of the examination and prescriptions.

[1519] "Speech recognition" is the technology that converts speech into text.

[1520] "Sentiment analysis" is a technology that infers emotions from text, voice, facial expressions, etc.

[1521] "Camera video" refers to video data captured by a camera.

[1522] "Question tailoring" is the process of generating or modifying appropriate questions based on the results of sentiment analysis.

[1523] An "autonomous vehicle" is a vehicle that can drive automatically without a driver.

[1524] "Passenger health condition" refers to a passenger's physical condition, medical condition, or other health-related condition.

[1525] "Real-time monitoring" means monitoring the current state instantly.

[1526] A system for carrying out the present invention is configured as follows: The server includes means for conducting a preliminary interview, means for generating questions for a doctor based on the result of the preliminary interview, means for creating a medical record by integrating the questions of the doctor and questions from the patient, means for linking the medical record to the doctor, means for performing voice recognition, means for performing emotion analysis, means for acquiring camera images, and means for adjusting the questions based on the result of the emotion analysis.

[1527] Hardware and Software Used

[1528] Hardware: microphones, webcams, on-board computers in autonomous vehicles

[1529] Software: speech_recognition library, pipeline of transformers library, cv2 (OpenCV)

[1530] Data processing and calculation

[1531] 1. Speech recognition: The server captures the audio from the microphone and converts it to text using the speech_recognition library.

[1532] 2. Sentiment analysis: The server performs sentiment analysis on the retrieved text using a pipeline from the transformers library to estimate the sentiment.

[1533] 3. Acquiring camera images: The server acquires images from the webcam and displays them in real time using cv2 (OpenCV).

[1534] 4. Question Adjustment: The server generates or modifies appropriate questions based on the results of sentiment analysis.

[1535] Examples

[1536] For example, if a passenger says "I feel sick," the server recognizes the voice and performs emotion analysis. If the emotion is determined to be "negative," the server generates the question, "Which part hurts?" This makes it possible to monitor the health of passengers in autonomous vehicles in real time and respond quickly if an abnormality is detected.

[1537] Examples of prompt statements

[1538] If the user says "I feel sick", do sentiment analysis and if the sentiment is determined to be "negative", generate the question "Which part hurts?"

[1539] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1540] Step 1:

[1541] The server acquires voice from the microphone. When the user says "I feel sick," the voice data is input. The server converts the voice data into text using the speech_recognition library. Through this process, the voice data is output as text data.

[1542] Step 2:

[1543] The server uses the text data obtained in step 1 as input and performs sentiment analysis using the pipeline of the transformers library. The sentiment analysis model estimates emotions from the text data and outputs the results. For example, if the text is "feeling bad," the emotion is output as "negative."

[1544] Step 3:

[1545] The server acquires video data from a webcam. The camera video is acquired in real time and processed using cv2 (OpenCV). The video data is used to analyze the user's facial expressions. The analysis results are output as emotion data estimated from the user's facial expressions.

[1546] Step 4:

[1547] The server integrates the emotion data obtained in steps 2 and 3 to perform an overall emotion evaluation. For example, if the voice is judged to be "negative" and the facial expression is judged to be "painful," the overall emotion evaluation will be "very bad." This overall evaluation is output.

[1548] Step 5:

[1549] The server generates an appropriate question based on the overall emotion evaluation. Using a generative AI model, a prompt sentence is input and an appropriate question is generated. For example, if you say "I feel bad" and the emotion is determined to be "negative," the question "Which part hurts?" is generated. This question is output.

[1550] Step 6:

[1551] The server presents the generated questions to the user. When the user enters an answer, the answer data is sent to the server. The server integrates the answer data into the medical record and communicates it to the doctor. This process updates the medical record and provides it to the doctor.

[1552] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1554] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1555] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1556] [Fourth embodiment]

[1557] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1558] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1559] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1560] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1561] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1562] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1563] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1564] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1565] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1566] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1567] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1568] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1569] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1570] "Example 1"

[1571] The present invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and takes part in a preliminary interview by online automatic transcription with an avatar or by filling out a form. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1572] "Example 2"

[1573] As a specific example, if a patient complains of cold symptoms, the preliminary interview will ask details about the symptoms and when they began. Based on this, the system will generate questions that the doctor is likely to ask at the time of the consultation, such as "What's your fever?" or "Do you have a cough?" It will also accept questions from the patient, such as "What cold medicine should I take?" These questions are integrated into a medical record and shared with the doctor. The doctor will use this as a reference when examining the patient, ensuring efficient medical care.

[1574] The process flow of each embodiment will be described below.

[1575] "Example 1"

[1576] Step 1: The patient accesses a dedicated URL.

[1577] Step 2: Conduct a pre-interview with an avatar using online automated transcription or form filling.

[1578] Step 3: Based on the results of the pre-interview, the system generates questions that the doctor is likely to ask during the visit.

[1579] Step 4: Also receive questions from the patient and consolidate these questions to create a medical record.

[1580] Step 5: The created medical record is shared with the doctor, who will use it as a reference when examining the patient.

[1581] "Example 2"

[1582] Step 1: If the patient complains of cold symptoms, the preliminary interview involves asking about details of the symptoms and when they began.

[1583] Step 2: Based on this, the system generates questions that a doctor might ask during a consultation, such as "What's your fever?" or "Do you have a cough?"

[1584] Step 3: Also, answer any questions from the patient, such as "What cold medicine should I take?"

[1585] Step 4: Integrate these questions into the patient's chart and communicate with the doctor.

[1586] Step 5: The doctor will use this as a reference when examining the patient, ensuring efficient medical treatment.

[1587] Example 1

[1588] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1589] In conventional medical systems, there was a problem that it was time-consuming for medical staff to ask questions before patients visit the clinic, making it difficult to conduct efficient examinations. In addition, it was difficult to accurately grasp the details of the patient's questions and symptoms, which could lead to a decline in the quality of the examination. Furthermore, the doctor could not quickly obtain the information necessary for the examination, which sometimes prolonged the examination time. To solve these problems, a system was needed that would improve the efficiency of the pre-examination and enable doctors to quickly obtain the information necessary for the examination.

[1590] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1591] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to the doctor, a means for extracting keywords and symptoms using natural language processing, and a means for generating questions for the doctor using a generative AI model. This makes it possible to improve the efficiency of the preliminary interview and allows the doctor to quickly obtain information required for the examination, thereby improving the quality of the examination and shortening the examination time.

[1592] A "pre-medical interview" is a process in which a patient enters their symptoms and condition before visiting a doctor, and medical professionals collect the necessary information before the examination.

[1593] The "doctor's questions" are questions that the doctor asks the patient during the consultation, which are generated based on the results of the preliminary interview.

[1594] A "medical record" is a medical document that records a patient's symptoms, examination results, treatment details, etc.

[1595] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[1596] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new information or data.

[1597] "Keywords" are important words or phrases within text data that are extracted using natural language processing.

[1598] A "symptom" is any physical or mental abnormality or discomfort experienced by a patient.

[1599] An "avatar" is a virtual character that allows users to interact online.

[1600] "Online automatic transcription" is a technology that converts audio into text in real time.

[1601] "Form entry" is a method in which a user manually enters text data into fields on a web page.

[1602] This invention is a system that supports patients on behalf of medical professionals. Specifically, before a patient visits the hospital, the patient accesses a dedicated URL and completes a preliminary interview using an avatar and online automatic transcription or form input. Based on the results of this preliminary interview, the system generates questions that the doctor is likely to ask at the time of the visit. In addition, the system accepts questions from the patient and creates a medical record by integrating these questions. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1603] Hardware and Software Used

[1604] This system is realized using the following hardware and software.

[1605] Hardware: Servers, terminals (smartphones, PCs, etc.)

[1606] Software: Python, natural language processing libraries (NLTK, spaCy), database manipulation libraries (SQLAlchemy), web interface framework (Flask), generative AI models (e.g., OpenAI's GPT-3)

[1607] Program Processing

[1608] The user (patient) accesses a dedicated URL. Once accessed, an avatar appears on the device and the preliminary interview begins through online automatic transcription or form input. Specifically, the text data entered by the user is sent from the device to the server.

[1609] The server analyzes the received text data using natural language processing libraries (NLTK and spaCy) to extract keywords and symptoms. Based on the results of this analysis, a generative AI model (for example, OpenAI's GPT-3) is used to generate questions that the doctor may ask during the consultation.

[1610] In addition, the terminal accepts additional questions from the user, and the questions entered on the terminal are sent back to the server, which then consolidates these questions and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner).

[1611] The created medical records are sent from the server to the doctor's terminal, and the doctor uses the records as a reference when examining the patient.

[1612] Examples of concrete examples and prompts

[1613] As a concrete example, consider the case where a user inputs, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." This text data is sent from the device to the server, and the server uses a natural language processing library to extract keywords such as "headache" and "when I wake up in the morning."

[1614] Next, feed the generative AI model the following prompt:

[1615] User symptoms: Headache, severe pain upon waking in the morning

[1616] Generate questions your doctor may have during a consultation.

[1617] A generative AI model might generate questions like:

[1618] 1. How often do your headaches occur?

[1619] 2. How severe is your headache?

[1620] 3. Do you have any other symptoms besides headaches?

[1621] These questions and any additional questions from the user are integrated to create a medical record in a format suitable for the electronic medical record system. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1622] In this way, the system acts as a substitute for medical professionals to support patients.

[1623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1624] Step 1:

[1625] The user accesses a special URL.

[1626] The user accesses the dedicated URL provided by the medical institution. For example, the user inputs the URL using a browser on a smartphone or computer. Input: Dedicated URL. Output: Display of the web page.

[1627] Step 2:

[1628] The terminal will display an avatar and begin the preliminary interview.

[1629] The terminal displays a web page based on the accessed URL. This web page displays an avatar and prompts the user to start the pre-interview. Input: Dedicated URL. Output: Display of avatar and start of pre-interview.

[1630] Step 3:

[1631] The user inputs text data.

[1632] The user inputs his / her symptoms and condition in text format according to the avatar's instructions. For example, the user might input, "I've been having headaches lately. The pain is especially severe when I wake up in the morning." Input: User's symptoms and condition. Output: Text data.

[1633] Step 4:

[1634] The terminal transmits the input data to the server.

[1635] The terminal sends the text data entered by the user to the server. This transmission is done securely using the HTTPS protocol. Input: Text data. Output: Data transmission to the server.

[1636] Step 5:

[1637] The server performs natural language processing to extract keywords and symptoms.

[1638] The server analyzes the received text data using a natural language processing library (NLTK or spaCy). For example, it extracts keywords such as "headache" and "when waking up in the morning." Input: Text data. Output: Extracted keywords and symptoms.

[1639] Step 6:

[1640] The server uses the generative AI model to generate questions for the doctor.

[1641] The server inputs a prompt sentence into a generative AI model (for example, OpenAI's GPT-3) based on the extracted keywords, and generates questions that a doctor may expect to ask when visiting the patient. For example, a prompt sentence such as "User's symptoms: headache, severe pain when waking up in the morning. Please generate questions that a doctor may expect to ask when visiting the patient" can be used. Input: Extraction results of keywords and symptoms. Output: Questions asked by the doctor.

[1642] Step 7:

[1643] The user enters a follow-up question.

[1644] The user inputs additional questions or information for the generated question. For example, the user answers "About three times a week" to the question "How often do you get headaches?". Input: Doctor's question. Output: Additional questions or information.

[1645] Step 8:

[1646] The terminal sends a follow-up question to the server.

[1647] The terminal sends any additional questions or information entered by the user back to the server. Input: Additional questions or information. Output: Sending data to the server.

[1648] Step 9:

[1649] The server consolidates the questions and creates a medical record.

[1650] The server merges the initial interview data and the follow-up question data and creates a medical record in a format suitable for the electronic medical record system (e.g., Epic or Cerner). Input: Initial interview data, follow-up question data. Output: Medical record.

[1651] Step 10:

[1652] The server links the medical records to the doctor's terminal.

[1653] The server links the created medical records to the doctor's terminal. The doctor can view the medical records through a dedicated medical system. Input: Medical records. Output: Linking medical records to the doctor's terminal.

[1654] Step 11:

[1655] The doctor will examine the patient using the medical records.

[1656] The doctor will consult the linked medical records when examining the patient. The medical records contain detailed information about the user's symptoms and the results of the preliminary interview, allowing the examination to proceed smoothly. Input: Medical records. Output: Examination results.

[1657] (Application example 1)

[1658] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1659] In conventional medical systems, there were limited ways for patients to ask questions before visiting a medical institution, and doctors often lacked the information they needed to examine patients. In addition, there was no way to understand the patient's symptoms or questions in advance when consulting about medicines at pharmacies or drug stores, making it difficult to respond efficiently. This resulted in longer waiting times for patients and increased burdens on medical professionals.

[1660] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1661] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for a doctor based on the results of the preliminary interview, a means for creating a medical record by integrating the questions of the doctor and questions from the patient, a means for linking the medical record to a doctor, a means for supporting preliminary interviews and consultations about medicines at pharmacies and drug stores, and a means for generating questions using a generative AI model. This allows a patient to collect necessary information before visiting a medical institution or a pharmacy, enabling efficient examinations and consultations.

[1662] A "pre-medication interview" is a process in which medical professionals collect necessary information in advance by having patients enter their symptoms and questions online before visiting a medical institution or pharmacy.

[1663] The "means for generating questions for a doctor" is a system that automatically generates questions that a doctor should ask during a consultation based on the results of a preliminary interview.

[1664] The "means of creating a medical record" refers to a system that integrates the results of the preliminary interview, questions from the doctor, and questions from the patient to create a medical record that compiles all the information necessary for the examination.

[1665] "Means of sharing medical records with doctors" refers to a system that provides the created medical records to doctors so that they can refer to them during consultations.

[1666] The "means of supporting advance medical interviews and medication consultations at pharmacies and drug stores" is a system that allows patients to input their symptoms and questions in advance at pharmacies and drug stores, enabling pharmacists to respond efficiently.

[1667] A "generative AI model" is an algorithm that uses artificial intelligence to generate appropriate questions and answers based on input data.

[1668] The "means for generating questions" refers to a system that uses a generative AI model to automatically generate appropriate questions based on the results of the preliminary interview.

[1669] The system for implementing this invention is configured as follows. First, a user accesses a dedicated URL using a terminal such as a smartphone or a personal computer. After accessing the URL, a preliminary interview is conducted by online automatic transcription with an avatar or by filling out a form. The results of this preliminary interview are sent to the server.

[1670] Based on the results of the preliminary interview, the server generates questions for the doctor using a generative AI model that uses advanced natural language processing algorithms, such as OpenAI's GPT-3. The generated questions are stored on the server.

[1671] Next, the server creates a medical record by integrating the generated questions from the doctor and the questions from the patient. This medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The created medical record is shared with the doctor, who uses it as a reference when examining the patient.

[1672] In addition, the system also has a function to support pre-medication and drug consultations at pharmacies and drug stores. Before going to a pharmacy or drug store, the user performs a pre-medication interview using a similar procedure, and the generative AI model generates questions for the pharmacist based on the results. This allows the pharmacist to efficiently respond to patient consultations.

[1673] For example, if a user enters, "I have a headache and a sore throat," the generative AI model generates questions such as, "When did the headache start?" and "How bad is your throat?" These questions are integrated into the medical record and provided to doctors and pharmacists.

[1674] An example of a prompt is as follows:

[1675] "Generate questions the pharmacist might have based on the following symptoms: I have a headache and a sore throat."

[1676] In this way, a system can be realized that enables efficient medical examinations and consultations based on the results of the preliminary interview.

[1677] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1678] Step 1:

[1679] The user accesses a dedicated URL using a device such as a smartphone or PC. The user completes a preliminary interview by using an avatar to automatically transcribe online or by filling out a form. The data entered is the user's symptoms and questions. This data is then sent from the device to the server.

[1680] Step 2:

[1681] The server uses a generative AI model to generate questions for the doctor based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated doctor's question.

[1682] Step 3:

[1683] The server creates a medical record by integrating the generated doctor's questions and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[1684] Step 4:

[1685] The server transmits the created medical record to the doctor. The medical record is provided so that the doctor can refer to it when examining the patient. The input is the integrated medical record, and the output is the medical record provided to the doctor.

[1686] Step 5:

[1687] Before a user goes to a pharmacy or drugstore, the user performs a pre-medical interview using a similar procedure. The user inputs symptoms and questions using a terminal and sends them to the server. The input data is the user's symptoms and questions.

[1688] Step 6:

[1689] The server uses a generative AI model to generate questions for the pharmacist based on the results of the pre-interview received. Specifically, the results of the pre-interview are input to the generative AI model as prompts to generate appropriate questions. The input is the results of the pre-interview, and the output is the generated questions for the pharmacist.

[1690] Step 7:

[1691] The server creates a medical record by integrating the generated questions from the pharmacist and the questions from the patient. The medical record includes the results of the preliminary interview, the generated questions, and the questions from the patient. The input is the results of the preliminary interview, the generated questions, and the questions from the patient, and the output is the integrated medical record.

[1692] Step 8:

[1693] The server communicates the created chart to the pharmacist. The chart is provided to the pharmacist so that the pharmacist can refer to it during consultation. The input is the integrated chart, and the output is the chart provided to the pharmacist.

[1694] Example 2

[1695] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1696] In conventional medical systems, the process from the patient's preliminary interview to the doctor's examination was not carried out efficiently, which often resulted in long consultation times. In addition, it was difficult to generate appropriate questions based on the patient's symptoms, and doctors were sometimes unable to fully grasp the necessary information before the examination. This resulted in poor efficiency in medical treatment and poor patient satisfaction.

[1697] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1698] In this invention, the server includes a means for conducting a preliminary interview, a means for generating questions for the doctor using a generative AI model based on the results of the preliminary interview, a means for creating a medical record by integrating the doctor's questions and questions from the patient, and a means for sharing the medical record with the doctor. This allows appropriate questions based on the patient's symptoms to be automatically generated, enabling the doctor to grasp the necessary information before the consultation. This improves the efficiency of medical care and also improves patient satisfaction.

[1699] A "pre-medical interview" is the process in which a patient provides information about their symptoms, medical history, etc. before an appointment.

[1700] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate outputs for a specific task.

[1701] "Doctor's questions" are questions that a doctor asks to understand a patient's symptoms and condition.

[1702] "Patient Questions" are questions that patients ask their physicians about their symptoms or treatment.

[1703] A "medical record" is a document that contains a patient's medical records, including symptoms, diagnoses, and treatment details.

[1704] "Collaboration" refers to different systems and processes sharing information and working together.

[1705] A "web page" is a page of information accessible on the Internet.

[1706] An "avatar" is a character or icon that represents a user online.

[1707] "Online automatic transcription" is a technology that converts audio into text in real time.

[1708] "Form filling" is the process by which a user enters information into fields on a web page.

[1709] "Medical Condition" refers to a patient's health condition or symptoms.

[1710] "Procedure" refers to the medical treatment or care that a doctor provides to a patient.

[1711] "Treatment time" refers to the time it takes a doctor to treat a patient.

[1712] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1713] This invention is a system that performs a preliminary interview, generates questions for the doctor using a generative AI model, creates a medical record by integrating the questions from the patient, and communicates with the doctor. A specific embodiment of this system is described below.

[1714] Hardware and Software Used

[1715] Hardware: Servers, terminals (PCs, tablets, smartphones, etc.)

[1716] Software: Generative AI models (e.g. GPT-4), database management systems (e.g. MySQL), web application frameworks (e.g. Django)

[1717] System Configuration

[1718] 1. Server: The server is responsible for generating questions to conduct a pre-medical interview. It uses a generative AI model (e.g. GPT-4) to generate questions based on the patient's symptoms. The server also receives answers from the user (patient) and stores them in a database. The server also integrates the collected information, creates a medical record, and notifies the doctor.

[1719] 2. Terminal: The terminal provides an interface for the user (patient) to conduct a preliminary interview. The terminal displays the questions received from the server to the user, collects the user's answers, and sends them to the server.

[1720] 3. User: The user (patient) uses the terminal to conduct a preliminary interview. The user answers the questions displayed on the terminal and, if necessary, enters his / her own questions.

[1721] Examples

[1722] 1. Question generation for pre-medical interview: The server uses a generative AI model (e.g., GPT-4) to generate questions for pre-medical interviews that are necessary when a patient complains of cold symptoms. For example, questions such as "When did your symptoms start?" and "What symptoms are you experiencing?" are generated. The generated questions are stored in a database.

[1723] 2. Displaying questions on the terminal: The terminal retrieves the questions from the pre-medical interview from the server and displays them to the user. For example, questions such as "What is your fever?" and "Do you have a cough?" are displayed on the terminal screen.

[1724] 3. Collect user answers: The user answers the questions displayed on the terminal. For example, they input information such as "My fever is 37.5 degrees" or "I have a cough." The user can also input their own questions. For example, they input questions such as "What cold medicine should I take?"

[1725] 4. Sending and storing answers: The terminal sends the user's answers to the server. The server stores the received answers in a database. This allows the centralized management of patient information.

[1726] 5. Creating a medical record: The server consolidates the collected information and creates a medical record. The medical record includes details of the patient's symptoms, when they first appeared, questions that the doctor anticipates, and questions from the patient. The created medical record is stored in a database.

[1727] 6. Medical record sharing with doctor: The server sends the created medical record to the doctor's terminal. The doctor performs the examination using the received medical record as a reference. For example, by looking at the medical record, the doctor can know in advance information such as "The patient has a fever of 37.5 degrees and is coughing." This allows for more efficient medical treatment.

[1728] Examples of prompt statements

[1729] "A patient is complaining of cold symptoms. Using the information below, generate questions that a doctor might ask during a consultation. Please include details of the patient's symptoms and when they began."

[1730] This system allows doctors to understand patients' symptoms in advance, improving the efficiency of medical treatment and increasing patient satisfaction.

[1731] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1732] Step 1:

[1733] The server generates the pre-screening questions.

[1734] Input: Information about the patient's initial symptoms (e.g. cold symptoms)

[1735] Data processing: Input a prompt sentence into a generative AI model (e.g. GPT-4) to generate appropriate questions.

[1736] Output: Generated questions (e.g., "When did your symptoms start?", "What symptoms are you experiencing?")

[1737] Specific operation: The server inputs a prompt sentence into the generative AI model: "The patient is complaining of cold symptoms. Based on the information below, please generate questions that a doctor may expect to ask when visiting the patient. Please also include details of the patient's symptoms and when they began." and stores the generated questions in a database.

[1738] Step 2:

[1739] The terminal obtains the questions for the preliminary medical examination from the server and displays them to the user.

[1740] Input: A generated query from the server

[1741] Data processing: Converting the question into a format for display to the user.

[1742] Output: The question that is dis...

Claims

1. A means for conducting a preliminary interview with a patient; means for generating a physician's question based on the results of the preliminary interview; A means of integrating doctor and patient questions to create a medical record; A means for linking the medical record to a doctor; A means for extracting keywords and symptoms using natural language processing; Using generative AI models to generate questions for doctors to ask to better understand a patient's medical condition; and A system including:

2. The system according to claim 1, wherein the preliminary interview is conducted by accessing a dedicated URL and by online automatic transcription with an avatar or by form input.

3. The system described in claim 1, characterized in that the questions posed by the doctor are questions that are expected at the time of a medical examination, and the medical record is created by suggesting the expected medical condition, procedure, treatment time, etc. based on prior confirmation, and the doctor uses this as a reference when making an examination.

Images