System

Abstract

A system is provided.SOLUTION: The system includes a means for reading identification information of a patient, a means for performing voice interaction with the patient, a means for converting voice data into text, a means for analyzing the text data and extracting important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing a comparison result and notifying a medical worker, and a means for reflecting summary information on an electronic medical record.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, the medical field is facing a serious labor shortage, which means that medical professionals are unable to devote sufficient time to communicating with patients. However, casual conversations with patients contain important health-related information that cannot be ignored. In particular, non-verbal information such as a patient's behavior and complexion is important in making medical decisions. To solve this problem and improve the efficiency of doctors' consultations, a new system that supports communication with patients is needed. [Means for solving the problem]

[0005] The present invention is a system that includes a means for reading a patient's identification information, a means for engaging in a voice dialogue with the patient, a means for converting voice data to text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, and a means for reflecting the summary information in an electronic medical record. Furthermore, by further including a means for recording the patient's behavior and complexion, medical examinations can be performed taking non-verbal information into consideration. Additionally, by including a means for providing feedback of the extracted health information to the patient via a hologram or a screen display, the system can assist the patient in understanding the situation and the medical professionals in their confirmation process.

[0006] "Patient Identifying Information" is personal data used by a hospital or medical facility to identify a patient, such as a patient card number or ID number.

[0007] "Voice interaction" refers to spoken communication between the patient and the system.

[0008] "Voice data" refers to data that is a digital recording of the sounds made by a patient.

[0009] "Means for converting to text" refers to technology or devices that convert voice data into text information.

[0010] "Text data" refers to data in which voice data is converted into character information.

[0011] "Significant health information" refers to data about a patient's health condition that is considered particularly important for medical decisions and examinations.

[0012] "Past medical examination data" refers to records and data relating to a patient's past medical examinations.

[0013] "Comparative means" refers to techniques or devices that compare current data with past data to identify anomalies and trends.

[0014] A "summarization tool" is a technique or device that concisely summarizes and reports extracted health information.

[0015] "Healthcare workers" refers to doctors, nurses, and other medical staff working in hospitals and healthcare facilities.

[0016] "Means of notification" refers to the techniques and methods used to convey important information to designated parties.

[0017] An "electronic medical record" is a system that manages a patient's medical records in digital format.

[0018] "Means for recording behavior and facial expression" refers to techniques and devices for observing and recording a patient's movements and facial expressions.

[0019] "Holographic or visual feedback" refers to technology or devices that visually communicate extracted health information to patients. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system for efficiently collecting patient health information and providing it to medical professionals. This system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data and extracting important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, and a means for updating an electronic medical record with the summarized information.

[0042] What the program does

[0043] 1. Patient admission process

[0044] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[0045] 2. Start a conversation

[0046] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[0047] 3. Organizing and recording conversation content

[0048] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[0049] 4. Comparison with past medical examination information

[0050] The server compares the extracted information with past examination data to detect new abnormalities or changes in progress, summarizes this information, and notifies the doctor.

[0051] 5. Providing summary information

[0052] The server sends the summarized information to the terminal, which then provides feedback to the user (patient) via a hologram or screen display. The user receives feedback about their physical condition and is instructed to see a doctor if necessary.

[0053] 6. Reflection in medical records

[0054] The server records the final consultation information in an electronic medical record system, making it available for viewing by doctors and other medical professionals.

[0055] Specific examples

[0056] 1. Reception process

[0057] Patient A holds his / her patient card over the reception desk. The server sends Patient A's latest health record to the terminal, which displays Patient A's information.

[0058] 2. Start a conversation

[0059] Terminal: "Hello, Patient A. How are you feeling?"

[0060] Patient A: "I've been feeling a little tired lately, but I'm OK."

[0061] 3. Organizing and recording conversation content

[0062] The device converts Patient A's conversation into text and sends it to the server. The server extracts the information that says "feeling tired" as important health information.

[0063] 4. Comparison with past medical examination information

[0064] The server compares the data with past medical examination data and detects that the patient's tendency to tire easily has increased. The server summarizes this information and notifies the doctor that "Patient A has recently become more prone to fatigue."

[0065] 5. Providing summary information

[0066] The device displays the doctor's feedback to Patient A as a hologram, saying, "We recommend that you come in for your next appointment to investigate the cause of your fatigue."

[0067] 6. Reflection in medical records

[0068] The server records the medical information, such as "Patient A's tendency to fatigue has increased," in the electronic medical record. Doctors and other medical professionals can then refer to this information.

[0069] In this way, the system of the present invention efficiently collects important health information through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[0073] Step 2:

[0074] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[0075] Step 3:

[0076] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[0077] Step 4:

[0078] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[0079] Step 5:

[0080] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache"). It also collects data to observe the user's complexion and behavior.

[0081] Step 6:

[0082] The server compares the extracted health information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0083] Step 7:

[0084] The server sends the summary information it has created to the terminal. The terminal then provides feedback to the user (patient) using a hologram or a screen display of the summary information. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0085] Step 8:

[0086] The user reviews the feedback and provides additional questions or information as needed. After all information has been collected, the server records the final consultation information in the electronic medical record system. The process is complete.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Current healthcare systems require efficient collection of patient health information and prompt, accurate provision of it to healthcare professionals. However, manually collecting patient information requires significant time and effort, and there is a risk of information omissions. Delays in analyzing and providing feedback on the collected information can also lead to problems with the quality of healthcare services. Furthermore, there is a need to provide more accurate health information by incorporating non-verbal information, such as the patient's behavior and complexion.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summarized information in the electronic medical record, means for providing feedback on the summarized information to the patient, and means for processing the patient's voice data in real time. This automates the collection of information from patients and enables timely and accurate provision of information. Furthermore, by analyzing non-verbal information as well, more accurate health information can be provided to medical professionals.

[0092] "Means for reading patient identification information" refers to a device or mechanism for obtaining unique identification information from an ID card, patient registration card, or other item held by a patient.

[0093] "Means for conducting voice dialogue with patients" refers to a system for carrying out voice communication with patients. Specifically, it refers to devices and software that use a microphone and speaker to conduct dialogue.

[0094] "Means for converting voice data into text" refers to software or a device that uses voice recognition technology to convert input voice data into text data.

[0095] The "means for analyzing text data and extracting important health information" is a system that uses natural language processing technology to identify and extract important information about a patient's symptoms and physical condition from converted text data.

[0096] "Means for comparing extracted health information with past medical examination data" refers to algorithms or software that compare newly acquired health information with previously recorded medical examination data to detect abnormalities or changes.

[0097] "Means for summarizing the comparison results and notifying medical professionals" refers to a communication system or device that summarizes the results of the comparison with past data in an easily understandable format and notifies medical professionals such as doctors and nurses.

[0098] The "means for reflecting summary information in electronic medical records" refers to an interface or software for automatically recording the generated summary information in the medical institution's electronic medical record system.

[0099] "Means for providing summarized information to patients" refers to methods for providing summarized diagnostic results and health information to patients, and includes technologies such as holograms and screen displays.

[0100] The "means for processing patient voice data in real time" is a system that instantly converts collected voice data into text and performs the analysis required for subsequent processing in real time.

[0101] The present invention relates to a system for efficiently collecting patient health information and providing it to medical professionals. The system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying the medical professionals, a means for updating an electronic medical record with the summarized information, a means for providing feedback on the summarized information to the patient, and a means for processing the patient's voice data in real time.

[0102] The following hardware and software are used to implement the system of the present invention. The hardware used includes a patient registration card reader terminal, a voice interaction terminal, and a server. The patient registration card reader terminal is a device for reading patients' patient registration cards and has the ability to read barcodes and RFID tags. The voice interaction terminal is a device for conducting voice interactions with patients and is a tablet or dedicated terminal equipped with a microphone and speaker. The server is a central computer for data processing and storage.

[0103] The software used includes voice recognition software, data analysis software, and electronic medical record systems. Examples of voice recognition software include Google Cloud Speech-to-Text API, which allows for real-time conversion of voice data into text data. Data analysis software uses Python natural language processing libraries (NLTK, spaCy, etc.) to analyze text data and extract important health information. Electronic medical record systems, such as Epic and Cerner, are used to record and manage final consultation information.

[0104] Specifically, patient reception is performed as follows: When a user (patient) holds their patient card over the reception terminal, the terminal reads the information on the card and sends it to the server. The server identifies the patient from the information on the card, retrieves the patient's profile information and past medical history from a database, and sends this to the terminal. The terminal then displays the patient information on its screen.

[0105] The conversation begins as follows: The device greets the patient through an AI voice, saying, "Hello, how are you?", and the user (patient) verbally talks about their health and recent situation. The device's microphone collects the patient's voice and sends it to the voice recognition software.

[0106] The process of organizing and recording conversations is as follows: The device converts collected voice data into text data in real time and sends it to the server. The server analyzes the text data and uses natural language processing technology to extract important health information. The extracted information is compared with past medical examination data. The server retrieves the data from the database and analyzes the results of the comparison.

[0107] Here are some examples of prompts:

[0108] Patient A swipes his / her medical card at the reception desk. The AI ​​then greets him / her with a voice message saying, "Hello, how are you feeling?" Patient A responds, "I've been feeling a little tired lately, but I'm fine." Please convert this conversation into text data, extract important health information, compare it with past data, and summarize the results. Please also explain the process for providing feedback on the summary information to the patient.

[0109] By inputting this prompt into a generative AI model, the process of generating summary information and providing feedback can be automated. This system will improve the efficiency of information collection from patients and the quality of information provided to medical professionals.

[0110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0111] Step 1: Patient admission

[0112] Input: The patient holds their patient card over the reception terminal.

[0113] Operation: The RFID reader or barcode reader on the terminal reads the patient card information and sends it to the server.

[0114] Server: Identifies the patient from the information on the patient card and retrieves the patient's profile information and past medical history from the database.

[0115] Output: The server sends the acquired information to the terminal, which displays the patient information on the screen. This allows the reception staff to check the patient's basic information and past medical history.

[0116] Step 2: Start a conversation

[0117] Input: When the terminal initiates the initial interaction with the patient.

[0118] How it works: The device's speaker outputs the AI ​​voice saying, "Hello, how are you feeling?"

[0119] User (patient): The patient speaks verbally to the terminal about their physical condition and recent situation. The microphone on the terminal collects the patient's voice.

[0120] Output: The collected voice data is sent to speech recognition software, which gathers initial information about the patient's condition.

[0121] Step 3: Organize and record the conversation

[0122] Input: User (patient) voice data.

[0123] How it works: Speech recognition software (e.g., Google Cloud Speech-to-Text API) converts voice data into text in real time.

[0124] Terminal: Sends text data to the server.

[0125] Server: Natural language processing software (e.g., spaCy, NLTK) analyzes the text data and extracts keywords and important health information.

[0126] Output: The extracted vital health information is recorded in a database, ready to be used in the next steps.

[0127] Step 4: Compare with previous medical records

[0128] Input: Newly extracted health information and past consultation data.

[0129] How it works: The server retrieves past medical history from a database and applies a comparison algorithm to compare the old and new health information.

[0130] Server: Detects anomalies and changes and evaluates the extent of the change.

[0131] Output: The comparison results are summarized and information is generated to inform healthcare professionals, allowing them to understand changes in the patient's health status.

[0132] Step 5: Provide summary information

[0133] Input: Summary information of the comparison results.

[0134] Operation: The server sends the summarized information to the terminal, which then sends it to the display device.

[0135] Terminal: Provides necessary feedback to the patient through a holographic display or screen display.

[0136] User (Patient): The patient receives the feedback and decides on their next action (e.g., to schedule a doctor's appointment) based on it.

[0137] Output: Patients will have up-to-date information about their health status, providing guidance on taking appropriate next steps.

[0138] Step 6: Reflecting in the medical record

[0139] Input: Final consultation information and summary information.

[0140] Operation: The server sends medical information to the electronic medical record system.

[0141] Server: Information is stored in a database via the electronic medical record system's API.

[0142] Electronic medical record system: Makes recorded information easily accessible and viewable by medical professionals.

[0143] Output: The consultation information is recorded in the electronic medical record, providing healthcare professionals with up-to-date patient information, which assists in the development of ongoing care plans.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Monitoring the health of workers is extremely important in modern factories. Repetitive, simple tasks and heavy labor can place a strain on workers, putting them at risk of developing health problems. Early detection of abnormalities and improvements to the working environment are essential. However, many current systems lack the means to efficiently collect and properly analyze worker health information, making it difficult to respond appropriately and in a timely manner.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes a means for reading the worker's identification information, a means for conducting a voice dialogue with the worker, and a means for converting the voice data into text, thereby enabling the server to efficiently collect and analyze the worker's health information and provide appropriate feedback in a timely manner.

[0149] "Worker identification information" refers to information used to identify individual workers in a factory, and is obtained through cards, ID badges, biometrics, etc.

[0150] "Voice dialogue" is a means of communication between the system and the worker through voice, and health information and work status are obtained in the form of questions and answers.

[0151] "Means for converting voice data to text" refers to the process of using voice recognition technology to convert the worker's voice into text data, which is then stored in a form that the system can understand and analyze.

[0152] The "means for analyzing text data to extract important health information" is a process for analyzing text data converted from speech and identifying and extracting important health information.

[0153] "Past health data" refers to a worker's past health information and records, and serves as reference data for comparison with newly acquired health information.

[0154] "Means for summarizing the comparison results and notifying the manager" refers to the process of briefly summarizing the results of comparing the extracted health information with past data and notifying the factory manager.

[0155] A "database" is an information storage system for storing worker health information and comparison results, which can be searched and referenced as needed.

[0156] "Means for recording movements and facial expressions" refers to the use of cameras and sensors to capture and analyze physical information such as the movements, facial expressions, and facial expressions of workers.

[0157] "Means for providing feedback using a hologram or screen display" refers to a method for visually conveying extracted health information to the worker, and provides feedback by projecting a hologram or displaying it on a display.

[0158] This invention is a system for efficiently collecting and analyzing health information of workers working in a factory and providing it to a manager. The system includes means for reading the worker's identification information, means for conducting a voice dialogue with the worker, means for converting the voice data into text, means for analyzing the text data and extracting important health information, means for comparing the extracted health information with past health data, means for summarizing the comparison results and notifying the manager, and means for updating the summarized information in a database.

[0159] A specific example of the system configuration includes the following elements:

[0160] First, a worker swipes their factory identification card, and the server confirms the worker's information and displays it on the terminal. RFID readers and biometric systems are used to read the identification information. For example, when a worker swipes their ID card, the reader transmits the card information to the server, which then displays the worker's profile and health data based on that information.

[0161] Next, the device engages in a voice dialogue with the worker. The device asks, "Hello, how are you feeling today?" and the worker talks about their physical condition and working environment. A microphone, speaker, and voice recognition and generation technologies are used for the voice dialogue. The voice data acquired in this process is converted into text via voice recognition software (e.g., Google Speech Recognition API).

[0162] After the voice data is converted to text, the server analyzes the text data to extract important health information. Generative AI models (such as Hugging Face's Transformers library) are used to identify health-related keywords and phrases. This extracted health information is then compared to past health data, which is pulled from a database on the server and compared to the worker's current condition.

[0163] The comparison results are summarized and notified to the administrator. The server generates summary information and sends it to the administrator's terminal. Based on this information, the administrator can understand the health status of the workers and take appropriate measures if necessary.

[0164] Finally, the summary information is entered into a database and stored for future reference, allowing for the accumulation and analysis of long-term health data.

[0165] As a specific example, the following scenario can be envisioned.

[0166] A worker holds his / her ID card over the terminal of a factory robot and says, "I've been suffering from severe shoulder stiffness lately." The system converts this voice information into text and extracts the information that "the shoulder stiffness has continued." After comparing it with past data, the system summarizes it as "If the shoulder stiffness has continued for more than three days, please consider improving your work environment," and notifies the manager. This allows for swift improvements to be made to the work environment.

[0167] Example prompt sentence:

[0168] The worker swipes his / her ID card and talks about his / her health condition: "I've been having really bad shoulder pain lately..."

[0169] System: "Do you have persistent shoulder pain? Let's check your past data."

[0170] Result: "If your shoulder pain persists for more than three days, consider improving your work environment."

[0171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0172] Step 1:

[0173] The user swipes their factory identification card. The terminal uses an RFID reader and biometric system to read the card's information. This identification information is sent to a server, which retrieves the worker's profile and past health data from a database and displays them on the terminal.

[0174] Input: Worker identification information

[0175] Data processing: reading identifying information, retrieving and displaying data from databases

[0176] Output: Display worker profile information

[0177] Step 2:

[0178] The device asks the worker, "Hello, how are you feeling today?" The user then begins talking about their physical condition and working environment.

[0179] Input: Voice regarding worker's health condition

[0180] Data processing: Acquisition of audio data

[0181] Output: Audio data

[0182] Step 3:

[0183] The device uses speech recognition software (e.g., Google Speech Recognition API) to convert the captured voice data into text in real time, which is then sent to the server.

[0184] Input: Audio data

[0185] Data processing: speech-to-text conversion

[0186] Output: Text data

[0187] Step 4:

[0188] The server analyzes the received text data using a generative AI model (e.g., Hugging Face's Transformers library) to extract important health information.

[0189] Input: Text data

[0190] Data processing: Analysis of text data and extraction of health information

[0191] Output: Extracted health information

[0192] Step 5:

[0193] The server retrieves past data from the database to compare the extracted health information with past health data, detects new abnormalities or changes, and notifies the administrator in a summarized form.

[0194] Input: Extracted health information, historical health data

[0195] Data processing: Comparing and summarizing health information

[0196] Output: Summarized health information

[0197] Step 6:

[0198] The summarized information is sent to a terminal, which then displays it as feedback to the worker via a hologram or display, such as a message like, "Are you still suffering from stiff shoulders? We'll compare it with past data."

[0199] Input: Abstracted health information

[0200] Data processing: Displaying summary information

[0201] Output: Display feedback information to the worker

[0202] Step 7:

[0203] The server then updates the database with the summarized information and stores it for future reference. It then sends instructions to the administrator, such as, "If your shoulder pain persists for more than three days, please consider improving your work environment."

[0204] Input: Abstracted health information

[0205] Data processing: storing information in a database

[0206] Output: Save the updated health data

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] This invention is a system for efficiently collecting patient health information and emotional states and providing them to healthcare professionals. The system includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, a means for converting voice data to text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying healthcare professionals, and a means for updating the summarized information in an electronic medical record. It also includes a means for recording the patient's behavior and complexion, a means for providing feedback on the extracted health information via a hologram or a screen display, and an emotion engine for recognizing the user's emotions.

[0209] What the program does

[0210] 1. Patient admission process

[0211] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[0212] 2. Start a conversation

[0213] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[0214] 3. Organizing and recording conversation content

[0215] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[0216] 4. Emotional Recognition

[0217] The device uses an emotion engine to recognize emotions from the user's voice and facial expression, for example, identifying emotions such as "stress," "anxiety," and "happiness" from the tone of voice and facial expression.

[0218] 5. Comparison with past medical examination information

[0219] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0220] 6. Providing summary information

[0221] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0222] 7. Reflection in medical records

[0223] The server records the final medical information in an electronic medical record system, which records the user's emotional state as well as their health condition.

[0224] Specific examples

[0225] 1. Reception process

[0226] Patient B holds his / her patient card over the reception desk. The server sends Patient B's latest health record to the terminal, which displays Patient B's information.

[0227] 2. Start a conversation

[0228] Terminal: "Hello, Patient B. How are you feeling?"

[0229] Patient B: "I haven't been sleeping well lately."

[0230] 3. Organizing and recording conversation content

[0231] The device converts Patient B's conversation into text and sends it to the server. The server extracts the information that "I can't sleep well" as important health information.

[0232] 4. Emotional Recognition

[0233] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[0234] 5. Comparison with past medical examination information

[0235] The server compares the data with past medical examination data and detects an increase in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies the doctor.

[0236] 6. Providing summary information

[0237] The device displays the doctor's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[0238] 7. Reflection in medical records

[0239] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[0240] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0241] The processing flow will be explained below.

[0242] Step 1:

[0243] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[0244] Step 2:

[0245] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[0246] Step 3:

[0247] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[0248] Step 4:

[0249] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[0250] Step 5:

[0251] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache").

[0252] Step 6:

[0253] The device uses an emotion engine to recognize emotions from the user's voice and facial expression. For example, it identifies emotions such as "stress," "anxiety," and "joy" from the tone of voice and facial expression.

[0254] Step 7:

[0255] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0256] Step 8:

[0257] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0258] Step 9:

[0259] The user reviews the feedback and provides additional questions or information as needed.

[0260] Step 10:

[0261] The server records the final consultation information and emotional state in the electronic medical record system, where doctors and other medical professionals can access this information. The process is complete.

[0262] Specific examples

[0263] Step 1:

[0264] Patient B holds his / her patient card over the reception desk. The terminal reads the information on the patient card and sends it to the server.

[0265] Step 2:

[0266] The server obtains detailed information and past medical history of Patient B and sends it to the terminal, which displays it on the screen.

[0267] Step 3:

[0268] The device greets the patient with a voice message saying, "Hello, Patient B. How are you feeling?" Patient B responds, "I haven't been sleeping well lately."

[0269] Step 4:

[0270] The terminal converts patient B's voice into text in real time and sends the text data to the server. The server receives the text data.

[0271] Step 5:

[0272] The server analyzes the text data and extracts important health information such as "not sleeping well."

[0273] Step 6:

[0274] The device uses an emotion engine to recognize the emotion "anxiety" from patient B's tone of voice.

[0275] Step 7:

[0276] The server compares the health information (e.g., "not sleeping well") and the emotional information (e.g., "anxiety") with past medical examination data to detect any abnormalities or changes, and creates a summary based on this information.

[0277] Step 8:

[0278] The server sends the summary information it has created to the terminal, which then displays a hologram to Patient B, saying, "You seem to have been having trouble sleeping and feeling anxious lately. I recommend you come in for your next appointment."

[0279] Step 9:

[0280] Patient B reviews the feedback and asks additional questions if necessary.

[0281] Step 10:

[0282] The server records the final medical information, such as "not sleeping well" and "anxiety," in the electronic medical record system. This information can be referenced by doctors and other medical professionals, completing the process.

[0283] Example 2

[0284] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] Conventional medical systems lack mechanisms for efficiently collecting patients' health information and emotional states and providing them appropriately to medical professionals. In particular, it is difficult to analyze a patient's emotional state in detail and compare it with past data to detect abnormalities or changes in progress. This can result in doctors taking a long time to grasp a patient's overall condition, potentially resulting in a decline in the quality of medical services. To solve this issue, a system is needed that can collect patients' health information and emotional states in real time and appropriately analyze and provide them.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0287] In this invention, the server includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, and a means for converting voice data into text. This automates the process from obtaining patient identification information to collecting health information, enabling efficient information collection. The server also includes a means for analyzing text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, a means for updating the summarized information in an electronic medical record, a means for recognizing the patient's emotional state from voice and video, a means for comparing the emotional information with past data to detect abnormalities or changes in the patient's condition, and a means for summarizing the detected emotional information and notifying medical professionals. This allows for real-time and efficient collection of patient health information and emotional state, and allows for appropriate analysis and provision. This allows medical professionals to quickly grasp the patient's overall condition and improve the quality of medical services.

[0288] "Means for reading patient identification information" refers to the function of transmitting a patient's personal information and medical history to the system using a medium such as a patient registration card or IC card.

[0289] "Means for conducting voice dialogue with patients" refers to a function that uses voice recognition technology to enable two-way voice communication between the system and patients.

[0290] "Means for converting voice data into text" refers to technology that converts voice dialogue between the patient and the system into text in real time.

[0291] "Means for analyzing text data to extract important health information" refers to technology that automatically extracts medically important keywords and phrases from converted text data.

[0292] "Means for comparing extracted health information with past medical examination data" refers to the function of checking newly collected health information against past medical examination history to detect abnormalities or changes.

[0293] "Means for summarizing comparison results and notifying healthcare professionals" refers to a function that automatically summarizes the key points of the comparison results and provides them to healthcare professionals in an easy-to-understand format.

[0294] "Means for reflecting summary information in electronic medical records" refers to a function that automatically adds the generated summary information to the patient's electronic medical record.

[0295] "Means for recognizing a patient's emotional state from audio and video" refers to technology that analyzes the patient's tone of voice and facial expression to identify their emotional state at that time.

[0296] "Means for comparing emotional information with past data to detect abnormalities or changes in progress" refers to a function that compares the recognized emotional state with past emotional data to detect abnormalities or changes.

[0297] "Means for summarizing detected emotional information and notifying medical professionals" refers to the function of summarizing the results of emotion analysis and providing them to medical professionals.

[0298] The present invention is a system for efficiently collecting and providing a patient's health information and emotional state to a medical professional. A specific embodiment of this system will be described.

[0299] The system consists of the following main components:

[0300] A means of reading patient identification information

[0301] A means of conducting voice dialogue with patients

[0302] A means of converting voice data into text

[0303] A means of analyzing text data to extract important health information

[0304] A means of comparing extracted health information with past medical examination data

[0305] A means of summarizing comparison results and communicating them to healthcare professionals

[0306] A means of reflecting summary information in electronic medical records

[0307] A means for recognizing a patient's emotional state from audio and video

[0308] A means of comparing emotional information with past data to detect abnormalities and changes in progress

[0309] A means of summarizing detected emotional information and notifying medical professionals

[0310] System configuration and operation

[0311] 1. Means of reading patient identification:

[0312] When a user holds their patient card over the reception desk, the server reads the card and acquires the patient's identification information. For example, the patient card contains a barcode or IC chip, and a barcode reader or IC card reader is used to read this.

[0313] 2. Means of audio communication with the patient:

[0314] Using the device's built-in voice recognition engine (e.g., Google Cloud Speech-to-Text), the device begins a voice dialogue with the patient by asking questions such as "Hello, how are you feeling?"

[0315] 3. Means of converting audio data to text:

[0316] The device records conversations with patients and converts the audio data into text in real time, which is then sent to a server.

[0317] 4. How to analyze text data and extract important health information:

[0318] The server analyzes the received text data and uses natural language processing to extract important health information, such as a statement like, "I haven't been sleeping well lately."

[0319] 5. Means of comparing extracted health information with past medical examination data:

[0320] The server compares the newly extracted health information with past medical records, detecting any abnormalities or changes in progress.

[0321] 6. Means of summarizing comparison results and communicating them to healthcare professionals:

[0322] The server summarises key points from the comparison results and provides them to healthcare professionals via a notification system.

[0323] 7. How to incorporate summary information into the electronic medical record:

[0324] The server automatically records the generated summary information in the patient's electronic medical record, using a commonly used system such as Epic.

[0325] 8. Means for recognizing a patient's emotional state from audio and visual:

[0326] The device uses an emotion recognition engine (e.g., Amazon Rekognition) built into it to analyze the patient's tone of voice and facial expression to recognize their emotional state.

[0327] 9. Comparing emotional information with historical data to detect anomalies and changes in progress:

[0328] The server compares the emotional information with past emotional data to detect any abnormalities or changes in progress.

[0329] 10. Means for summarizing detected emotional information and notifying healthcare professionals:

[0330] The server summarizes changes in emotional information and notifies healthcare professionals, for example, by providing information such as "The patient continues to be anxious."

[0331] Specific example operation procedure

[0332] 1. Reception process:

[0333] When a user (Patient B) holds their patient card over the reception desk, the server sends Patient B's latest health record to the terminal, and the terminal displays Patient B's information.

[0334] 2. Start a conversation:

[0335] Terminal: "Hello, Patient B. How are you feeling?"

[0336] User: "I haven't been sleeping well lately."

[0337] 3. Organizing and recording conversations:

[0338] The device converts Patient B's conversation into text and sends it to the server, which extracts the information that "I can't sleep well" as important health information.

[0339] 4. Emotion Recognition:

[0340] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[0341] 5. Comparison with past medical examination information:

[0342] The server compares data from past visits and detects increases in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies healthcare professionals.

[0343] 6. Providing Summary Information:

[0344] The device displays the medical professional's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[0345] 7. Reflection in the medical record:

[0346] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[0347] Prompt Sentence Examples

[0348] Below is an example of a prompt that can be used as input to a generative AI model.

[0349] Patient B holds up his / her patient card at the reception desk, and the terminal greets him / her with "Hello, how are you feeling?", after which Patient B responds, "I haven't been sleeping well lately." Please convert this conversation into text data, recognize Patient B's emotion (anxiety) from his / her tone of voice and facial expression, and compare it with past medical examination data to provide summary information.

[0350] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0351] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0352] Processing Steps

[0353] Step 1: Patient admission

[0354] Specific behavior:

[0355] The user swipes their patient card.

[0356] The server reads the information on the patient card and checks the patient's profile information and past medical history.

[0357] The server sends the confirmed information to the terminal.

[0358] The terminal displays the patient's profile information and medical history.

[0359] Input / Output:

[0360] Input: Patient card information

[0361] Data processing / calculation: Extracting patient profile information and past medical history from patient card information

[0362] Output: Display patient profile information and medical history on terminal

[0363] Step 2: Start a conversation

[0364] Specific behavior:

[0365] The device uses a voice recognition engine to interactively ask the patient questions such as "Hello, how are you feeling?"

[0366] The user begins talking about their health and recent events.

[0367] Input / Output:

[0368] Input: Patient consultation start flag

[0369] Data processing / calculation: Activating the speech recognition engine and generating interactive questions

[0370] Output: Voice input from the patient

[0371] Step 3: Organize and record the conversation

[0372] Specific behavior:

[0373] The device records conversations with patients and converts the audio data into text data in real time.

[0374] The terminal sends the converted text data to the server.

[0375] The server analyzes the text data and extracts important health information.

[0376] Input / Output:

[0377] Input: Voice conversation with patient

[0378] Data processing / calculation: Converting voice to text and analyzing it using natural language processing technology

[0379] Output: Text data containing extracted health information

[0380] Step 4: Recognize emotions

[0381] Specific behavior:

[0382] The device uses a built-in camera and microphone to analyze the patient's complexion and tone of voice.

[0383] The device uses an emotion engine to recognize emotions such as "stress," "anxiety," and "joy."

[0384] Input / Output:

[0385] Input: Patient audio and video data

[0386] Data processing / calculation: Voice tone and facial color analysis

[0387] Output: Emotional state identification data

[0388] Step 5: Compare with previous medical records

[0389] Specific behavior:

[0390] The server compares the newly extracted health and emotional information with past medical records.

[0391] The server detects anomalies and changes in progress and compiles this information into a summary.

[0392] Input / Output:

[0393] Input: Extracted health and emotion information

[0394] Data processing / calculation: Comparison with past medical examination data

[0395] Output: Summarized consultation results

[0396] Step 6: Provide summary information

[0397] Specific behavior:

[0398] The server sends the created summary information to the terminal.

[0399] The terminal displays summary information on a hologram or screen and provides feedback to the patient.

[0400] Input / Output:

[0401] Input: Summarized consultation results

[0402] Data processing / calculation: Converting summary information into a display format

[0403] Output: Feedback message displayed to the patient

[0404] Step 7: Reflecting in the medical record

[0405] Specific behavior:

[0406] The server records the final medical information and emotional state in the electronic medical record system.

[0407] Input / Output:

[0408] Input: Final consultation information and emotional state

[0409] Data processing / calculation: Reflecting medical information in the electronic medical record system

[0410] Output: Updated electronic medical record

[0411] The above is the flow and operation of the specific processing steps. This system makes it possible to efficiently collect patient health information and emotional state and provide appropriate feedback to medical professionals.

[0412] (Application example 2)

[0413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] Conventional patient management systems have difficulty understanding a patient's health and emotional state in real time, making it difficult for medical professionals to provide accurate examinations and advice. There are also issues with efficiency in patient care, which could lead to a decline in the quality of medical services. In addition, detailed information, including the patient's emotional state, cannot be reflected in the electronic medical record, making it difficult to compare with past medical data or to adequately manage patients continuously. The present invention aims to solve these problems.

[0415] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summary information in the electronic medical record, means for recognizing the emotional state, and means for displaying the summary information including the recognized emotional state on a terminal in a physical store. This allows for efficient collection of patient health information and emotional state and provision of the information to medical professionals in real time, thereby enabling the quality of medical examinations and the overall improvement of medical services.

[0416] "Means for reading patient identification information" refers to devices or systems for verifying the patient's identity and obtaining data from patient cards, NFC tags, etc.

[0417] The "means for conducting a voice dialogue with a patient" refers to a device or system that receives voice input from a patient and automatically generates and provides questions and responses based on the content of the voice input.

[0418] "Means for converting voice data into text" refers to a device or system that uses voice recognition technology to convert what a patient says into text data.

[0419] The "means for analyzing text data and extracting important health information" refers to a device or system that analyzes acquired text data using natural language processing technology and identifies important information related to health checkups.

[0420] The "means for comparing extracted health information with past medical examination data" refers to a device or system that compares a patient's current health information with past medical examination data to detect abnormalities or changes.

[0421] "Means for summarizing the comparison results and notifying medical professionals" refers to a device or system that concisely summarizes the comparison results of health information and promptly notifies medical professionals.

[0422] The "means for reflecting summarized information in the electronic medical record" refers to a device or system that automatically inputs and records summarized health information into the electronic medical record system.

[0423] "Means for recognizing emotional state" refers to a device or system that determines the psychological emotional state of a patient from the tone of voice, facial expression, etc.

[0424] The "means for displaying summary information including the recognized emotional state on a terminal in a physical store" refers to a device or system that displays a summary of health information including the patient's emotional state on a terminal such as a display or hologram installed in a physical store.

[0425] This invention relates to a system that efficiently collects patient health information and emotional states and provides them to medical professionals, aiming to improve the efficiency of patient care, particularly in physical stores, and provide higher quality medical services.

[0426] System configuration

[0427] The system includes the following main elements:

[0428] 1. A means of reading patient identification information

[0429] 2. Means of communicating with patients through voice

[0430] 3. Means of converting audio data into text

[0431] 4. A means of analyzing text data to extract important health information

[0432] 5. A means to compare extracted health information with past medical examination data

[0433] 6. Means of summarizing comparison results and communicating them to healthcare professionals

[0434] 7. Means of reflecting summary information in electronic medical records

[0435] 8. A means of recognizing emotional states

[0436] 9. A means to display summary information including the recognized emotional state on a brick-and-mortar terminal

[0437] What the program does

[0438] This system performs the following processing by the server, terminal, and user.

[0439] Reception process

[0440] The server uses an NFC reader to read the patient's identification information from their medical card, and the identified patient information is automatically displayed on the terminal.

[0441] Voice dialogue

[0442] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) and a speech synthesis API (such as Google Text-to-Speech) to engage in natural voice conversations with patients. As patients talk about their health and symptoms, their speech is converted into text in real time.

[0443] Data analysis and emotion recognition

[0444] The server analyzes the text data using a natural language processing engine (e.g., NLTK, SpaCy) to extract important health information, and also uses an emotion analysis engine to recognize the patient's emotional state from their tone of voice and facial expressions.

[0445] Comparison and Summary

[0446] The server compares the extracted health information and recognized emotion information with past medical examination data, summarizes the results, and notifies medical professionals.

[0447] Information Feedback

[0448] The device provides the patient with summary information, including the patient's recognized emotional state, via a hologram or a screen display, and the summary information is automatically reflected in the electronic medical record system.

[0449] Adding specific examples

[0450] Patient A's reception and health information collection

[0451] 1. Patient A holds his / her smartphone over the NFC reader at the reception desk.

[0452] 2. The device asks, "Hello, Patient A. How are you feeling right now?"

[0453] 3. Patient A answers, "I've been having terrible headaches lately."

[0454] 4. The device converts the keywords "headache" and "recently" into text data and sends it to the server.

[0455] 5. The server performs emotion analysis and recognizes the emotions "pain" and "anxiety."

[0456] 6. The server compares the data with past medical examination data and notifies the doctor of the summary results.

[0457] 7. The device displays a hologram to Patient A saying, "You've been experiencing severe headaches and anxiety recently. I recommend you see a doctor."

[0458] 8. The server records the new medical information, "headache" and "anxiety," in the electronic medical record.

[0459] Example prompts for generative AI models

[0460] 1. Speech Recognition Engine:

[0461] Transcribe the following audio file to text: "A patient is talking about their health. The content is, 'I've been having really bad headaches lately.'"

[0462] 2. Sentiment Analysis Engine:

[0463] Recognize the emotion in the following text: "I've been having terrible headaches lately." Return the emotion tag (pain, anxiety, etc.).

[0464] 3. Natural Language Processing Engine:

[0465] Extract keywords from the following text: "I've been having terrible headaches lately." Return a list of important keywords.

[0466] The entire system efficiently collects patients' health information and emotional state and provides it to medical professionals in real time, thereby improving the quality of consultations and overall medical services.

[0467] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0468] Step 1:

[0469] The user holds their smartphone over the NFC reader at the reception desk. The NFC reader obtains the patient's identification information and sends it to the server. The input is the patient's medical card, and the output is the patient's identification information. Through this data processing, the patient's past medical examination data is obtained from the server and displayed on a terminal at the physical store.

[0470] Step 2:

[0471] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) to initiate a voice dialogue with the patient. The device asks, "Hello, Patient A. How are you feeling right now?" and the user responds. The input is voice data, and the output is text data. This voice data is converted into text in real time and sent to the server.

[0472] Step 3:

[0473] The server analyzes the received text data using a natural language processing engine (NLTK, SpaCy, etc.) to extract important health information. The input is text data, and the output is the extracted health information. This data processing identifies keywords (e.g., "headache" or "lack of sleep").

[0474] Step 4:

[0475] The server uses an emotion analysis engine to analyze the patient's voice tone and facial expression data acquired from the device's camera to recognize their emotional state. The input is voice and image data, and the output is an emotion tag (e.g., "anxiety" or "stress"). This data calculation determines the patient's psychological state.

[0476] Step 5:

[0477] The server compares the extracted health information and recognized emotional information with past medical examination data. The inputs are health information, emotional information, and past medical examination data, and the output is the comparison results. This data calculation detects new abnormalities and changes in the progress.

[0478] Step 6:

[0479] The server summarizes the comparison results and generates summary information, which includes important health information and emotional state. The input is the comparison results, and the output is summary information. This data processing constructs a concise diagnosis and notifies medical professionals.

[0480] Step 7:

[0481] The terminal provides the patient with summary information via a hologram or screen display. The input is the summary information, and the output is the feedback content. This specific operation allows the patient to gain a deeper understanding of their own health condition.

[0482] Step 8:

[0483] The server reflects the summary information in the electronic medical record system. The input is summary information, and the output is an updated electronic medical record. This data processing records information that will be useful for subsequent examinations and treatment.

[0484] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0486] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0487] [Second embodiment]

[0488] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0489] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0491] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0495] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0496] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0497] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0498] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0499] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0500] The present invention is a system for efficiently collecting patient health information and providing it to medical professionals. This system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data and extracting important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, and a means for updating an electronic medical record with the summarized information.

[0501] What the program does

[0502] 1. Patient admission process

[0503] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[0504] 2. Start a conversation

[0505] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[0506] 3. Organizing and recording conversation content

[0507] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[0508] 4. Comparison with past medical examination information

[0509] The server compares the extracted information with past examination data to detect new abnormalities or changes in progress, summarizes this information, and notifies the doctor.

[0510] 5. Providing summary information

[0511] The server sends the summarized information to the terminal, which then provides feedback to the user (patient) via a hologram or screen display. The user receives feedback about their physical condition and is instructed to see a doctor if necessary.

[0512] 6. Reflection in medical records

[0513] The server records the final consultation information in an electronic medical record system, making it available for viewing by doctors and other medical professionals.

[0514] Specific examples

[0515] 1. Reception process

[0516] Patient A holds his / her patient card over the reception desk. The server sends Patient A's latest health record to the terminal, which displays Patient A's information.

[0517] 2. Start a conversation

[0518] Terminal: "Hello, Patient A. How are you feeling?"

[0519] Patient A: "I've been feeling a little tired lately, but I'm OK."

[0520] 3. Organizing and recording conversation content

[0521] The device converts Patient A's conversation into text and sends it to the server. The server extracts the information that says "feeling tired" as important health information.

[0522] 4. Comparison with past medical examination information

[0523] The server compares the data with past medical examination data and detects that the patient's tendency to tire easily has increased. The server summarizes this information and notifies the doctor that "Patient A has recently become more prone to fatigue."

[0524] 5. Providing summary information

[0525] The device displays the doctor's feedback to Patient A as a hologram, saying, "We recommend that you come in for your next appointment to investigate the cause of your fatigue."

[0526] 6. Reflection in medical records

[0527] The server records the medical information, such as "Patient A's tendency to fatigue has increased," in the electronic medical record. Doctors and other medical professionals can then refer to this information.

[0528] In this way, the system of the present invention efficiently collects important health information through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0529] The processing flow will be explained below.

[0530] Step 1:

[0531] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[0532] Step 2:

[0533] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[0534] Step 3:

[0535] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[0536] Step 4:

[0537] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[0538] Step 5:

[0539] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache"). It also collects data to observe the user's complexion and behavior.

[0540] Step 6:

[0541] The server compares the extracted health information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0542] Step 7:

[0543] The server sends the summary information it has created to the terminal. The terminal then provides feedback to the user (patient) using a hologram or a screen display of the summary information. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0544] Step 8:

[0545] The user reviews the feedback and provides additional questions or information as needed. After all information has been collected, the server records the final consultation information in the electronic medical record system. The process is complete.

[0546] Example 1

[0547] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] Current healthcare systems require efficient collection of patient health information and prompt, accurate provision of it to healthcare professionals. However, manually collecting patient information requires significant time and effort, and there is a risk of information omissions. Delays in analyzing and providing feedback on the collected information can also lead to problems with the quality of healthcare services. Furthermore, there is a need to provide more accurate health information by incorporating non-verbal information, such as the patient's behavior and complexion.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0550] In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summarized information in the electronic medical record, means for providing feedback on the summarized information to the patient, and means for processing the patient's voice data in real time. This automates the collection of information from patients and enables timely and accurate provision of information. Furthermore, by analyzing non-verbal information as well, more accurate health information can be provided to medical professionals.

[0551] "Means for reading patient identification information" refers to a device or mechanism for obtaining unique identification information from an ID card, patient registration card, or other item held by a patient.

[0552] "Means for conducting voice dialogue with patients" refers to a system for carrying out voice communication with patients. Specifically, it refers to devices and software that use a microphone and speaker to conduct dialogue.

[0553] "Means for converting voice data into text" refers to software or a device that uses voice recognition technology to convert input voice data into text data.

[0554] The "means for analyzing text data and extracting important health information" is a system that uses natural language processing technology to identify and extract important information about a patient's symptoms and physical condition from converted text data.

[0555] "Means for comparing extracted health information with past medical examination data" refers to algorithms or software that compare newly acquired health information with previously recorded medical examination data to detect abnormalities or changes.

[0556] "Means for summarizing the comparison results and notifying medical professionals" refers to a communication system or device that summarizes the results of the comparison with past data in an easily understandable format and notifies medical professionals such as doctors and nurses.

[0557] The "means for reflecting summary information in electronic medical records" refers to an interface or software for automatically recording the generated summary information in the medical institution's electronic medical record system.

[0558] "Means for providing summarized information to patients" refers to methods for providing summarized diagnostic results and health information to patients, and includes technologies such as holograms and screen displays.

[0559] The "means for processing patient voice data in real time" is a system that instantly converts collected voice data into text and performs the analysis required for subsequent processing in real time.

[0560] The present invention relates to a system for efficiently collecting patient health information and providing it to medical professionals. The system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying the medical professionals, a means for updating an electronic medical record with the summarized information, a means for providing feedback on the summarized information to the patient, and a means for processing the patient's voice data in real time.

[0561] The following hardware and software are used to implement the system of the present invention. The hardware used includes a patient registration card reader terminal, a voice interaction terminal, and a server. The patient registration card reader terminal is a device for reading patients' patient registration cards and has the ability to read barcodes and RFID tags. The voice interaction terminal is a device for conducting voice interactions with patients and is a tablet or dedicated terminal equipped with a microphone and speaker. The server is a central computer for data processing and storage.

[0562] The software used includes voice recognition software, data analysis software, and electronic medical record systems. Examples of voice recognition software include Google Cloud Speech-to-Text API, which allows for real-time conversion of voice data into text data. Data analysis software uses Python natural language processing libraries (NLTK, spaCy, etc.) to analyze text data and extract important health information. Electronic medical record systems, such as Epic and Cerner, are used to record and manage final consultation information.

[0563] Specifically, patient reception is performed as follows: When a user (patient) holds their patient card over the reception terminal, the terminal reads the information on the card and sends it to the server. The server identifies the patient from the information on the card, retrieves the patient's profile information and past medical history from a database, and sends this to the terminal. The terminal then displays the patient information on its screen.

[0564] The conversation begins as follows: The device greets the patient through an AI voice, saying, "Hello, how are you?", and the user (patient) verbally talks about their health and recent situation. The device's microphone collects the patient's voice and sends it to the voice recognition software.

[0565] The process of organizing and recording conversations is as follows: The device converts collected voice data into text data in real time and sends it to the server. The server analyzes the text data and uses natural language processing technology to extract important health information. The extracted information is compared with past medical examination data. The server retrieves the data from the database and analyzes the results of the comparison.

[0566] Here are some examples of prompts:

[0567] Patient A swipes his / her medical card at the reception desk. The AI ​​then greets him / her with a voice message saying, "Hello, how are you feeling?" Patient A responds, "I've been feeling a little tired lately, but I'm fine." Please convert this conversation into text data, extract important health information, compare it with past data, and summarize the results. Please also explain the process for providing feedback on the summary information to the patient.

[0568] By inputting this prompt into a generative AI model, the process of generating summary information and providing feedback can be automated. This system will improve the efficiency of information collection from patients and the quality of information provided to medical professionals.

[0569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0570] Step 1: Patient admission

[0571] Input: The patient holds their patient card over the reception terminal.

[0572] Operation: The RFID reader or barcode reader on the terminal reads the patient card information and sends it to the server.

[0573] Server: Identifies the patient from the information on the patient card and retrieves the patient's profile information and past medical history from the database.

[0574] Output: The server sends the acquired information to the terminal, which displays the patient information on the screen. This allows the reception staff to check the patient's basic information and past medical history.

[0575] Step 2: Start a conversation

[0576] Input: When the terminal initiates the initial interaction with the patient.

[0577] How it works: The device's speaker outputs the AI ​​voice saying, "Hello, how are you feeling?"

[0578] User (patient): The patient speaks verbally to the terminal about their physical condition and recent situation. The microphone on the terminal collects the patient's voice.

[0579] Output: The collected voice data is sent to speech recognition software, which gathers initial information about the patient's condition.

[0580] Step 3: Organize and record the conversation

[0581] Input: User (patient) voice data.

[0582] How it works: Speech recognition software (e.g., Google Cloud Speech-to-Text API) converts voice data into text in real time.

[0583] Terminal: Sends text data to the server.

[0584] Server: Natural language processing software (e.g., spaCy, NLTK) analyzes the text data and extracts keywords and important health information.

[0585] Output: The extracted vital health information is recorded in a database, ready to be used in the next steps.

[0586] Step 4: Compare with previous medical records

[0587] Input: Newly extracted health information and past consultation data.

[0588] How it works: The server retrieves past medical history from a database and applies a comparison algorithm to compare the old and new health information.

[0589] Server: Detects anomalies and changes and evaluates the extent of the change.

[0590] Output: The comparison results are summarized and information is generated to inform healthcare professionals, allowing them to understand changes in the patient's health status.

[0591] Step 5: Provide summary information

[0592] Input: Summary information of the comparison results.

[0593] Operation: The server sends the summarized information to the terminal, which then sends it to the display device.

[0594] Terminal: Provides necessary feedback to the patient through a holographic display or screen display.

[0595] User (Patient): The patient receives the feedback and decides on their next action (e.g., to schedule a doctor's appointment) based on it.

[0596] Output: Patients will have up-to-date information about their health status, providing guidance on taking appropriate next steps.

[0597] Step 6: Reflecting in the medical record

[0598] Input: Final consultation information and summary information.

[0599] Operation: The server sends medical information to the electronic medical record system.

[0600] Server: Information is stored in a database via the electronic medical record system's API.

[0601] Electronic medical record system: Makes recorded information easily accessible and viewable by medical professionals.

[0602] Output: The consultation information is recorded in the electronic medical record, providing healthcare professionals with up-to-date patient information, which assists in the development of ongoing care plans.

[0603] (Application example 1)

[0604] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Monitoring the health of workers is extremely important in modern factories. Repetitive, simple tasks and heavy labor can place a strain on workers, putting them at risk of developing health problems. Early detection of abnormalities and improvements to the working environment are essential. However, many current systems lack the means to efficiently collect and properly analyze worker health information, making it difficult to respond appropriately and in a timely manner.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0607] In this invention, the server includes a means for reading the worker's identification information, a means for conducting a voice dialogue with the worker, and a means for converting the voice data into text, thereby enabling the server to efficiently collect and analyze the worker's health information and provide appropriate feedback in a timely manner.

[0608] "Worker identification information" refers to information used to identify individual workers in a factory, and is obtained through cards, ID badges, biometrics, etc.

[0609] "Voice dialogue" is a means of communication between the system and the worker through voice, and health information and work status are obtained in the form of questions and answers.

[0610] "Means for converting voice data to text" refers to the process of using voice recognition technology to convert the worker's voice into text data, which is then stored in a form that the system can understand and analyze.

[0611] The "means for analyzing text data to extract important health information" is a process for analyzing text data converted from speech and identifying and extracting important health information.

[0612] "Past health data" refers to a worker's past health information and records, and serves as reference data for comparison with newly acquired health information.

[0613] "Means for summarizing the comparison results and notifying the manager" refers to the process of briefly summarizing the results of comparing the extracted health information with past data and notifying the factory manager.

[0614] A "database" is an information storage system for storing worker health information and comparison results, which can be searched and referenced as needed.

[0615] "Means for recording movements and facial expressions" refers to the use of cameras and sensors to capture and analyze physical information such as the movements, facial expressions, and facial expressions of workers.

[0616] "Means for providing feedback using a hologram or screen display" refers to a method for visually conveying extracted health information to the worker, and provides feedback by projecting a hologram or displaying it on a display.

[0617] This invention is a system for efficiently collecting and analyzing health information of workers working in a factory and providing it to a manager. The system includes means for reading the worker's identification information, means for conducting a voice dialogue with the worker, means for converting the voice data into text, means for analyzing the text data and extracting important health information, means for comparing the extracted health information with past health data, means for summarizing the comparison results and notifying the manager, and means for updating the summarized information in a database.

[0618] A specific example of the system configuration includes the following elements:

[0619] First, a worker swipes their factory identification card, and the server confirms the worker's information and displays it on the terminal. RFID readers and biometric systems are used to read the identification information. For example, when a worker swipes their ID card, the reader transmits the card information to the server, which then displays the worker's profile and health data based on that information.

[0620] Next, the device engages in a voice dialogue with the worker. The device asks, "Hello, how are you feeling today?" and the worker talks about their physical condition and working environment. A microphone, speaker, and voice recognition and generation technologies are used for the voice dialogue. The voice data acquired in this process is converted into text via voice recognition software (e.g., Google Speech Recognition API).

[0621] After the voice data is converted to text, the server analyzes the text data to extract important health information. Generative AI models (such as Hugging Face's Transformers library) are used to identify health-related keywords and phrases. This extracted health information is then compared to past health data, which is pulled from a database on the server and compared to the worker's current condition.

[0622] The comparison results are summarized and notified to the administrator. The server generates summary information and sends it to the administrator's terminal. Based on this information, the administrator can understand the health status of the workers and take appropriate measures if necessary.

[0623] Finally, the summary information is entered into a database and stored for future reference, allowing for the accumulation and analysis of long-term health data.

[0624] As a specific example, the following scenario can be envisioned.

[0625] A worker holds his / her ID card over the terminal of a factory robot and says, "I've been suffering from severe shoulder stiffness lately." The system converts this voice information into text and extracts the information that "the shoulder stiffness has continued." After comparing it with past data, the system summarizes it as "If the shoulder stiffness has continued for more than three days, please consider improving your work environment," and notifies the manager. This allows for swift improvements to be made to the work environment.

[0626] Example prompt sentence:

[0627] The worker swipes his / her ID card and talks about his / her health condition: "I've been having really bad shoulder pain lately..."

[0628] System: "Do you have persistent shoulder pain? Let's check your past data."

[0629] Result: "If your shoulder pain persists for more than three days, consider improving your work environment."

[0630] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0631] Step 1:

[0632] The user swipes their factory identification card. The terminal uses an RFID reader and biometric system to read the card's information. This identification information is sent to a server, which retrieves the worker's profile and past health data from a database and displays them on the terminal.

[0633] Input: Worker identification information

[0634] Data processing: reading identifying information, retrieving and displaying data from databases

[0635] Output: Display worker profile information

[0636] Step 2:

[0637] The device asks the worker, "Hello, how are you feeling today?" The user then begins talking about their physical condition and working environment.

[0638] Input: Voice regarding worker's health condition

[0639] Data processing: Acquisition of audio data

[0640] Output: Audio data

[0641] Step 3:

[0642] The device uses speech recognition software (e.g., Google Speech Recognition API) to convert the captured voice data into text in real time, which is then sent to the server.

[0643] Input: Audio data

[0644] Data processing: speech-to-text conversion

[0645] Output: Text data

[0646] Step 4:

[0647] The server analyzes the received text data using a generative AI model (e.g., Hugging Face's Transformers library) to extract important health information.

[0648] Input: Text data

[0649] Data processing: Analysis of text data and extraction of health information

[0650] Output: Extracted health information

[0651] Step 5:

[0652] The server retrieves past data from the database to compare the extracted health information with past health data, detects new abnormalities or changes, and notifies the administrator in a summarized form.

[0653] Input: Extracted health information, historical health data

[0654] Data processing: Comparing and summarizing health information

[0655] Output: Summarized health information

[0656] Step 6:

[0657] The summarized information is sent to a terminal, which then displays it as feedback to the worker via a hologram or display, such as a message like, "Are you still suffering from stiff shoulders? We'll compare it with past data."

[0658] Input: Abstracted health information

[0659] Data processing: Displaying summary information

[0660] Output: Display feedback information to the worker

[0661] Step 7:

[0662] The server then updates the database with the summarized information and stores it for future reference. It then sends instructions to the administrator, such as, "If your shoulder pain persists for more than three days, please consider improving your work environment."

[0663] Input: Abstracted health information

[0664] Data processing: storing information in a database

[0665] Output: Save the updated health data

[0666] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0667] This invention is a system for efficiently collecting patient health information and emotional states and providing them to healthcare professionals. The system includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, a means for converting voice data to text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying healthcare professionals, and a means for updating the summarized information in an electronic medical record. It also includes a means for recording the patient's behavior and complexion, a means for providing feedback on the extracted health information via a hologram or a screen display, and an emotion engine for recognizing the user's emotions.

[0668] What the program does

[0669] 1. Patient admission process

[0670] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[0671] 2. Start a conversation

[0672] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[0673] 3. Organizing and recording conversation content

[0674] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[0675] 4. Emotional Recognition

[0676] The device uses an emotion engine to recognize emotions from the user's voice and facial expression, for example, identifying emotions such as "stress," "anxiety," and "happiness" from the tone of voice and facial expression.

[0677] 5. Comparison with past medical examination information

[0678] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0679] 6. Providing summary information

[0680] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0681] 7. Reflection in medical records

[0682] The server records the final medical information in an electronic medical record system, which records the user's emotional state as well as their health condition.

[0683] Specific examples

[0684] 1. Reception process

[0685] Patient B holds his / her patient card over the reception desk. The server sends Patient B's latest health record to the terminal, which displays Patient B's information.

[0686] 2. Start a conversation

[0687] Terminal: "Hello, Patient B. How are you feeling?"

[0688] Patient B: "I haven't been sleeping well lately."

[0689] 3. Organizing and recording conversation content

[0690] The device converts Patient B's conversation into text and sends it to the server. The server extracts the information that "I can't sleep well" as important health information.

[0691] 4. Emotional Recognition

[0692] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[0693] 5. Comparison with past medical examination information

[0694] The server compares the data with past medical examination data and detects an increase in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies the doctor.

[0695] 6. Providing summary information

[0696] The device displays the doctor's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[0697] 7. Reflection in medical records

[0698] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[0699] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0700] The processing flow will be explained below.

[0701] Step 1:

[0702] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[0703] Step 2:

[0704] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[0705] Step 3:

[0706] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[0707] Step 4:

[0708] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[0709] Step 5:

[0710] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache").

[0711] Step 6:

[0712] The device uses an emotion engine to recognize emotions from the user's voice and facial expression. For example, it identifies emotions such as "stress," "anxiety," and "joy" from the tone of voice and facial expression.

[0713] Step 7:

[0714] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[0715] Step 8:

[0716] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[0717] Step 9:

[0718] The user reviews the feedback and provides additional questions or information as needed.

[0719] Step 10:

[0720] The server records the final consultation information and emotional state in the electronic medical record system, where doctors and other medical professionals can access this information. The process is complete.

[0721] Specific examples

[0722] Step 1:

[0723] Patient B holds his / her patient card over the reception desk. The terminal reads the information on the patient card and sends it to the server.

[0724] Step 2:

[0725] The server obtains detailed information and past medical history of Patient B and sends it to the terminal, which displays it on the screen.

[0726] Step 3:

[0727] The device greets the patient with a voice message saying, "Hello, Patient B. How are you feeling?" Patient B responds, "I haven't been sleeping well lately."

[0728] Step 4:

[0729] The terminal converts patient B's voice into text in real time and sends the text data to the server. The server receives the text data.

[0730] Step 5:

[0731] The server analyzes the text data and extracts important health information such as "not sleeping well."

[0732] Step 6:

[0733] The device uses an emotion engine to recognize the emotion "anxiety" from patient B's tone of voice.

[0734] Step 7:

[0735] The server compares the health information (e.g., "not sleeping well") and the emotional information (e.g., "anxiety") with past medical examination data to detect any abnormalities or changes, and creates a summary based on this information.

[0736] Step 8:

[0737] The server sends the summary information it has created to the terminal, which then displays a hologram to Patient B, saying, "You seem to have been having trouble sleeping and feeling anxious lately. I recommend you come in for your next appointment."

[0738] Step 9:

[0739] Patient B reviews the feedback and asks additional questions if necessary.

[0740] Step 10:

[0741] The server records the final medical information, such as "not sleeping well" and "anxiety," in the electronic medical record system. This information can be referenced by doctors and other medical professionals, completing the process.

[0742] Example 2

[0743] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] Conventional medical systems lack mechanisms for efficiently collecting patients' health information and emotional states and providing them appropriately to medical professionals. In particular, it is difficult to analyze a patient's emotional state in detail and compare it with past data to detect abnormalities or changes in progress. This can result in doctors taking a long time to grasp a patient's overall condition, potentially resulting in a decline in the quality of medical services. To solve this issue, a system is needed that can collect patients' health information and emotional states in real time and appropriately analyze and provide them.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0746] In this invention, the server includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, and a means for converting voice data into text. This automates the process from obtaining patient identification information to collecting health information, enabling efficient information collection. The server also includes a means for analyzing text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, a means for updating the summarized information in an electronic medical record, a means for recognizing the patient's emotional state from voice and video, a means for comparing the emotional information with past data to detect abnormalities or changes in the patient's condition, and a means for summarizing the detected emotional information and notifying medical professionals. This allows for real-time and efficient collection of patient health information and emotional state, and allows for appropriate analysis and provision. This allows medical professionals to quickly grasp the patient's overall condition and improve the quality of medical services.

[0747] "Means for reading patient identification information" refers to the function of transmitting a patient's personal information and medical history to the system using a medium such as a patient registration card or IC card.

[0748] "Means for conducting voice dialogue with patients" refers to a function that uses voice recognition technology to enable two-way voice communication between the system and patients.

[0749] "Means for converting voice data into text" refers to technology that converts voice dialogue between the patient and the system into text in real time.

[0750] "Means for analyzing text data to extract important health information" refers to technology that automatically extracts medically important keywords and phrases from converted text data.

[0751] "Means for comparing extracted health information with past medical examination data" refers to the function of checking newly collected health information against past medical examination history to detect abnormalities or changes.

[0752] "Means for summarizing comparison results and notifying healthcare professionals" refers to a function that automatically summarizes the key points of the comparison results and provides them to healthcare professionals in an easy-to-understand format.

[0753] "Means for reflecting summary information in electronic medical records" refers to a function that automatically adds the generated summary information to the patient's electronic medical record.

[0754] "Means for recognizing a patient's emotional state from audio and video" refers to technology that analyzes the patient's tone of voice and facial expression to identify their emotional state at that time.

[0755] "Means for comparing emotional information with past data to detect abnormalities or changes in progress" refers to a function that compares the recognized emotional state with past emotional data to detect abnormalities or changes.

[0756] "Means for summarizing detected emotional information and notifying medical professionals" refers to the function of summarizing the results of emotion analysis and providing them to medical professionals.

[0757] The present invention is a system for efficiently collecting and providing a patient's health information and emotional state to a medical professional. A specific embodiment of this system will be described.

[0758] The system consists of the following main components:

[0759] A means of reading patient identification information

[0760] A means of conducting voice dialogue with patients

[0761] A means of converting voice data into text

[0762] A means of analyzing text data to extract important health information

[0763] A means of comparing extracted health information with past medical examination data

[0764] A means of summarizing comparison results and communicating them to healthcare professionals

[0765] A means of reflecting summary information in electronic medical records

[0766] A means for recognizing a patient's emotional state from audio and video

[0767] A means of comparing emotional information with past data to detect abnormalities and changes in progress

[0768] A means of summarizing detected emotional information and notifying medical professionals

[0769] System configuration and operation

[0770] 1. Means of reading patient identification:

[0771] When a user holds their patient card over the reception desk, the server reads the card and acquires the patient's identification information. For example, the patient card contains a barcode or IC chip, and a barcode reader or IC card reader is used to read this.

[0772] 2. Means of audio communication with the patient:

[0773] Using the device's built-in voice recognition engine (e.g., Google Cloud Speech-to-Text), the device begins a voice dialogue with the patient by asking questions such as "Hello, how are you feeling?"

[0774] 3. Means of converting audio data to text:

[0775] The device records conversations with patients and converts the audio data into text in real time, which is then sent to a server.

[0776] 4. How to analyze text data and extract important health information:

[0777] The server analyzes the received text data and uses natural language processing to extract important health information, such as a statement like, "I haven't been sleeping well lately."

[0778] 5. Means of comparing extracted health information with past medical examination data:

[0779] The server compares the newly extracted health information with past medical records, detecting any abnormalities or changes in progress.

[0780] 6. Means of summarizing comparison results and communicating them to healthcare professionals:

[0781] The server summarises key points from the comparison results and provides them to healthcare professionals via a notification system.

[0782] 7. How to incorporate summary information into the electronic medical record:

[0783] The server automatically records the generated summary information in the patient's electronic medical record, using a commonly used system such as Epic.

[0784] 8. Means for recognizing a patient's emotional state from audio and visual:

[0785] The device uses an emotion recognition engine (e.g., Amazon Rekognition) built into it to analyze the patient's tone of voice and facial expression to recognize their emotional state.

[0786] 9. Comparing emotional information with historical data to detect anomalies and changes in progress:

[0787] The server compares the emotional information with past emotional data to detect any abnormalities or changes in progress.

[0788] 10. Means for summarizing detected emotional information and notifying healthcare professionals:

[0789] The server summarizes changes in emotional information and notifies healthcare professionals, for example, by providing information such as "The patient continues to be anxious."

[0790] Specific example operation procedure

[0791] 1. Reception process:

[0792] When a user (Patient B) holds their patient card over the reception desk, the server sends Patient B's latest health record to the terminal, and the terminal displays Patient B's information.

[0793] 2. Start a conversation:

[0794] Terminal: "Hello, Patient B. How are you feeling?"

[0795] User: "I haven't been sleeping well lately."

[0796] 3. Organizing and recording conversations:

[0797] The device converts Patient B's conversation into text and sends it to the server, which extracts the information that "I can't sleep well" as important health information.

[0798] 4. Emotion Recognition:

[0799] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[0800] 5. Comparison with past medical examination information:

[0801] The server compares data from past visits and detects increases in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies healthcare professionals.

[0802] 6. Providing Summary Information:

[0803] The device displays the medical professional's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[0804] 7. Reflection in the medical record:

[0805] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[0806] Prompt Sentence Examples

[0807] Below is an example of a prompt that can be used as input to a generative AI model.

[0808] Patient B holds up his / her patient card at the reception desk, and the terminal greets him / her with "Hello, how are you feeling?", after which Patient B responds, "I haven't been sleeping well lately." Please convert this conversation into text data, recognize Patient B's emotion (anxiety) from his / her tone of voice and facial expression, and compare it with past medical examination data to provide summary information.

[0809] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0810] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0811] Processing Steps

[0812] Step 1: Patient admission

[0813] Specific behavior:

[0814] The user swipes their patient card.

[0815] The server reads the information on the patient card and checks the patient's profile information and past medical history.

[0816] The server sends the confirmed information to the terminal.

[0817] The terminal displays the patient's profile information and medical history.

[0818] Input / Output:

[0819] Input: Patient card information

[0820] Data processing / calculation: Extracting patient profile information and past medical history from patient card information

[0821] Output: Display patient profile information and medical history on terminal

[0822] Step 2: Start a conversation

[0823] Specific behavior:

[0824] The device uses a voice recognition engine to interactively ask the patient questions such as "Hello, how are you feeling?"

[0825] The user begins talking about their health and recent events.

[0826] Input / Output:

[0827] Input: Patient consultation start flag

[0828] Data processing / calculation: Activating the speech recognition engine and generating interactive questions

[0829] Output: Voice input from the patient

[0830] Step 3: Organize and record the conversation

[0831] Specific behavior:

[0832] The device records conversations with patients and converts the audio data into text data in real time.

[0833] The terminal sends the converted text data to the server.

[0834] The server analyzes the text data and extracts important health information.

[0835] Input / Output:

[0836] Input: Voice conversation with patient

[0837] Data processing / calculation: Converting voice to text and analyzing it using natural language processing technology

[0838] Output: Text data containing extracted health information

[0839] Step 4: Recognize emotions

[0840] Specific behavior:

[0841] The device uses a built-in camera and microphone to analyze the patient's complexion and tone of voice.

[0842] The device uses an emotion engine to recognize emotions such as "stress," "anxiety," and "joy."

[0843] Input / Output:

[0844] Input: Patient audio and video data

[0845] Data processing / calculation: Voice tone and facial color analysis

[0846] Output: Emotional state identification data

[0847] Step 5: Compare with previous medical records

[0848] Specific behavior:

[0849] The server compares the newly extracted health and emotional information with past medical records.

[0850] The server detects anomalies and changes in progress and compiles this information into a summary.

[0851] Input / Output:

[0852] Input: Extracted health and emotion information

[0853] Data processing / calculation: Comparison with past medical examination data

[0854] Output: Summarized consultation results

[0855] Step 6: Provide summary information

[0856] Specific behavior:

[0857] The server sends the created summary information to the terminal.

[0858] The terminal displays summary information on a hologram or screen and provides feedback to the patient.

[0859] Input / Output:

[0860] Input: Summarized consultation results

[0861] Data processing / calculation: Converting summary information into a display format

[0862] Output: Feedback message displayed to the patient

[0863] Step 7: Reflecting in the medical record

[0864] Specific behavior:

[0865] The server records the final medical information and emotional state in the electronic medical record system.

[0866] Input / Output:

[0867] Input: Final consultation information and emotional state

[0868] Data processing / calculation: Reflecting medical information in the electronic medical record system

[0869] Output: Updated electronic medical record

[0870] The above is the flow and operation of the specific processing steps. This system makes it possible to efficiently collect patient health information and emotional state and provide appropriate feedback to medical professionals.

[0871] (Application example 2)

[0872] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0873] Conventional patient management systems have difficulty understanding a patient's health and emotional state in real time, making it difficult for medical professionals to provide accurate examinations and advice. There are also issues with efficiency in patient care, which could lead to a decline in the quality of medical services. In addition, detailed information, including the patient's emotional state, cannot be reflected in the electronic medical record, making it difficult to compare with past medical data or to adequately manage patients continuously. The present invention aims to solve these problems.

[0874] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summary information in the electronic medical record, means for recognizing the emotional state, and means for displaying the summary information including the recognized emotional state on a terminal in a physical store. This allows for efficient collection of patient health information and emotional state and provision of the information to medical professionals in real time, thereby enabling the quality of medical examinations and the overall improvement of medical services.

[0875] "Means for reading patient identification information" refers to devices or systems for verifying the patient's identity and obtaining data from patient cards, NFC tags, etc.

[0876] The "means for conducting a voice dialogue with a patient" refers to a device or system that receives voice input from a patient and automatically generates and provides questions and responses based on the content of the voice input.

[0877] "Means for converting voice data into text" refers to a device or system that uses voice recognition technology to convert what a patient says into text data.

[0878] The "means for analyzing text data and extracting important health information" refers to a device or system that analyzes acquired text data using natural language processing technology and identifies important information related to health checkups.

[0879] The "means for comparing extracted health information with past medical examination data" refers to a device or system that compares a patient's current health information with past medical examination data to detect abnormalities or changes.

[0880] "Means for summarizing the comparison results and notifying medical professionals" refers to a device or system that concisely summarizes the comparison results of health information and promptly notifies medical professionals.

[0881] The "means for reflecting summarized information in the electronic medical record" refers to a device or system that automatically inputs and records summarized health information into the electronic medical record system.

[0882] "Means for recognizing emotional state" refers to a device or system that determines the psychological emotional state of a patient from the tone of voice, facial expression, etc.

[0883] The "means for displaying summary information including the recognized emotional state on a terminal in a physical store" refers to a device or system that displays a summary of health information including the patient's emotional state on a terminal such as a display or hologram installed in a physical store.

[0884] This invention relates to a system that efficiently collects patient health information and emotional states and provides them to medical professionals, aiming to improve the efficiency of patient care, particularly in physical stores, and provide higher quality medical services.

[0885] System configuration

[0886] The system includes the following main elements:

[0887] 1. A means of reading patient identification information

[0888] 2. Means of communicating with patients through voice

[0889] 3. Means of converting audio data into text

[0890] 4. A means of analyzing text data to extract important health information

[0891] 5. A means to compare extracted health information with past medical examination data

[0892] 6. Means of summarizing comparison results and communicating them to healthcare professionals

[0893] 7. Means of reflecting summary information in electronic medical records

[0894] 8. A means of recognizing emotional states

[0895] 9. A means to display summary information including the recognized emotional state on a brick-and-mortar terminal

[0896] What the program does

[0897] This system performs the following processing by the server, terminal, and user.

[0898] Reception process

[0899] The server uses an NFC reader to read the patient's identification information from their medical card, and the identified patient information is automatically displayed on the terminal.

[0900] Voice dialogue

[0901] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) and a speech synthesis API (such as Google Text-to-Speech) to engage in natural voice conversations with patients. As patients talk about their health and symptoms, their speech is converted into text in real time.

[0902] Data analysis and emotion recognition

[0903] The server analyzes the text data using a natural language processing engine (e.g., NLTK, SpaCy) to extract important health information, and also uses an emotion analysis engine to recognize the patient's emotional state from their tone of voice and facial expressions.

[0904] Comparison and Summary

[0905] The server compares the extracted health information and recognized emotion information with past medical examination data, summarizes the results, and notifies medical professionals.

[0906] Information Feedback

[0907] The device provides the patient with summary information, including the patient's recognized emotional state, via a hologram or a screen display, and the summary information is automatically reflected in the electronic medical record system.

[0908] Adding specific examples

[0909] Patient A's reception and health information collection

[0910] 1. Patient A holds his / her smartphone over the NFC reader at the reception desk.

[0911] 2. The device asks, "Hello, Patient A. How are you feeling right now?"

[0912] 3. Patient A answers, "I've been having terrible headaches lately."

[0913] 4. The device converts the keywords "headache" and "recently" into text data and sends it to the server.

[0914] 5. The server performs emotion analysis and recognizes the emotions "pain" and "anxiety."

[0915] 6. The server compares the data with past medical examination data and notifies the doctor of the summary results.

[0916] 7. The device displays a hologram to Patient A saying, "You've been experiencing severe headaches and anxiety recently. I recommend you see a doctor."

[0917] 8. The server records the new medical information, "headache" and "anxiety," in the electronic medical record.

[0918] Example prompts for generative AI models

[0919] 1. Speech Recognition Engine:

[0920] Transcribe the following audio file to text: "A patient is talking about their health. The content is, 'I've been having really bad headaches lately.'"

[0921] 2. Sentiment Analysis Engine:

[0922] Recognize the emotion in the following text: "I've been having terrible headaches lately." Return the emotion tag (pain, anxiety, etc.).

[0923] 3. Natural Language Processing Engine:

[0924] Extract keywords from the following text: "I've been having terrible headaches lately." Return a list of important keywords.

[0925] The entire system efficiently collects patients' health information and emotional state and provides it to medical professionals in real time, thereby improving the quality of consultations and overall medical services.

[0926] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0927] Step 1:

[0928] The user holds their smartphone over the NFC reader at the reception desk. The NFC reader obtains the patient's identification information and sends it to the server. The input is the patient's medical card, and the output is the patient's identification information. Through this data processing, the patient's past medical examination data is obtained from the server and displayed on a terminal at the physical store.

[0929] Step 2:

[0930] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) to initiate a voice dialogue with the patient. The device asks, "Hello, Patient A. How are you feeling right now?" and the user responds. The input is voice data, and the output is text data. This voice data is converted into text in real time and sent to the server.

[0931] Step 3:

[0932] The server analyzes the received text data using a natural language processing engine (NLTK, SpaCy, etc.) to extract important health information. The input is text data, and the output is the extracted health information. This data processing identifies keywords (e.g., "headache" or "lack of sleep").

[0933] Step 4:

[0934] The server uses an emotion analysis engine to analyze the patient's voice tone and facial expression data acquired from the device's camera to recognize their emotional state. The input is voice and image data, and the output is an emotion tag (e.g., "anxiety" or "stress"). This data calculation determines the patient's psychological state.

[0935] Step 5:

[0936] The server compares the extracted health information and recognized emotional information with past medical examination data. The inputs are health information, emotional information, and past medical examination data, and the output is the comparison results. This data calculation detects new abnormalities and changes in the progress.

[0937] Step 6:

[0938] The server summarizes the comparison results and generates summary information, which includes important health information and emotional state. The input is the comparison results, and the output is summary information. This data processing constructs a concise diagnosis and notifies medical professionals.

[0939] Step 7:

[0940] The terminal provides the patient with summary information via a hologram or screen display. The input is the summary information, and the output is the feedback content. This specific operation allows the patient to gain a deeper understanding of their own health condition.

[0941] Step 8:

[0942] The server reflects the summary information in the electronic medical record system. The input is summary information, and the output is an updated electronic medical record. This data processing records information that will be useful for subsequent examinations and treatment.

[0943] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0944] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0945] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0946] [Third embodiment]

[0947] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0948] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0949] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0950] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0951] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0952] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0953] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0954] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0955] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0956] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0957] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0958] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0959] The present invention is a system for efficiently collecting patient health information and providing it to medical professionals. This system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data and extracting important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, and a means for updating an electronic medical record with the summarized information.

[0960] What the program does

[0961] 1. Patient admission process

[0962] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[0963] 2. Start a conversation

[0964] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[0965] 3. Organizing and recording conversation content

[0966] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[0967] 4. Comparison with past medical examination information

[0968] The server compares the extracted information with past examination data to detect new abnormalities or changes in progress, summarizes this information, and notifies the doctor.

[0969] 5. Providing summary information

[0970] The server sends the summarized information to the terminal, which then provides feedback to the user (patient) via a hologram or screen display. The user receives feedback about their physical condition and is instructed to see a doctor if necessary.

[0971] 6. Reflection in medical records

[0972] The server records the final consultation information in an electronic medical record system, making it available for viewing by doctors and other medical professionals.

[0973] Specific examples

[0974] 1. Reception process

[0975] Patient A holds his / her patient card over the reception desk. The server sends Patient A's latest health record to the terminal, which displays Patient A's information.

[0976] 2. Start a conversation

[0977] Terminal: "Hello, Patient A. How are you feeling?"

[0978] Patient A: "I've been feeling a little tired lately, but I'm OK."

[0979] 3. Organizing and recording conversation content

[0980] The device converts Patient A's conversation into text and sends it to the server. The server extracts the information that says "feeling tired" as important health information.

[0981] 4. Comparison with past medical examination information

[0982] The server compares the data with past medical examination data and detects that the patient's tendency to tire easily has increased. The server summarizes this information and notifies the doctor that "Patient A has recently become more prone to fatigue."

[0983] 5. Providing summary information

[0984] The device displays the doctor's feedback to Patient A as a hologram, saying, "We recommend that you come in for your next appointment to investigate the cause of your fatigue."

[0985] 6. Reflection in medical records

[0986] The server records the medical information, such as "Patient A's tendency to fatigue has increased," in the electronic medical record. Doctors and other medical professionals can then refer to this information.

[0987] In this way, the system of the present invention efficiently collects important health information through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[0988] The processing flow will be explained below.

[0989] Step 1:

[0990] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[0991] Step 2:

[0992] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[0993] Step 3:

[0994] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[0995] Step 4:

[0996] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[0997] Step 5:

[0998] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache"). It also collects data to observe the user's complexion and behavior.

[0999] Step 6:

[1000] The server compares the extracted health information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1001] Step 7:

[1002] The server sends the summary information it has created to the terminal. The terminal then provides feedback to the user (patient) using a hologram or a screen display of the summary information. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1003] Step 8:

[1004] The user reviews the feedback and provides additional questions or information as needed. After all information has been collected, the server records the final consultation information in the electronic medical record system. The process is complete.

[1005] Example 1

[1006] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] Current healthcare systems require efficient collection of patient health information and prompt, accurate provision of it to healthcare professionals. However, manually collecting patient information requires significant time and effort, and there is a risk of information omissions. Delays in analyzing and providing feedback on the collected information can also lead to problems with the quality of healthcare services. Furthermore, there is a need to provide more accurate health information by incorporating non-verbal information, such as the patient's behavior and complexion.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1009] In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summarized information in the electronic medical record, means for providing feedback on the summarized information to the patient, and means for processing the patient's voice data in real time. This automates the collection of information from patients and enables timely and accurate provision of information. Furthermore, by analyzing non-verbal information as well, more accurate health information can be provided to medical professionals.

[1010] "Means for reading patient identification information" refers to a device or mechanism for obtaining unique identification information from an ID card, patient registration card, or other item held by a patient.

[1011] "Means for conducting voice dialogue with patients" refers to a system for carrying out voice communication with patients. Specifically, it refers to devices and software that use a microphone and speaker to conduct dialogue.

[1012] "Means for converting voice data into text" refers to software or a device that uses voice recognition technology to convert input voice data into text data.

[1013] The "means for analyzing text data and extracting important health information" is a system that uses natural language processing technology to identify and extract important information about a patient's symptoms and physical condition from converted text data.

[1014] "Means for comparing extracted health information with past medical examination data" refers to algorithms or software that compare newly acquired health information with previously recorded medical examination data to detect abnormalities or changes.

[1015] "Means for summarizing the comparison results and notifying medical professionals" refers to a communication system or device that summarizes the results of the comparison with past data in an easily understandable format and notifies medical professionals such as doctors and nurses.

[1016] The "means for reflecting summary information in electronic medical records" refers to an interface or software for automatically recording the generated summary information in the medical institution's electronic medical record system.

[1017] "Means for providing summarized information to patients" refers to methods for providing summarized diagnostic results and health information to patients, and includes technologies such as holograms and screen displays.

[1018] The "means for processing patient voice data in real time" is a system that instantly converts collected voice data into text and performs the analysis required for subsequent processing in real time.

[1019] The present invention relates to a system for efficiently collecting patient health information and providing it to medical professionals. The system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying the medical professionals, a means for updating an electronic medical record with the summarized information, a means for providing feedback on the summarized information to the patient, and a means for processing the patient's voice data in real time.

[1020] The following hardware and software are used to implement the system of the present invention. The hardware used includes a patient registration card reader terminal, a voice interaction terminal, and a server. The patient registration card reader terminal is a device for reading patients' patient registration cards and has the ability to read barcodes and RFID tags. The voice interaction terminal is a device for conducting voice interactions with patients and is a tablet or dedicated terminal equipped with a microphone and speaker. The server is a central computer for data processing and storage.

[1021] The software used includes voice recognition software, data analysis software, and electronic medical record systems. Examples of voice recognition software include Google Cloud Speech-to-Text API, which allows for real-time conversion of voice data into text data. Data analysis software uses Python natural language processing libraries (NLTK, spaCy, etc.) to analyze text data and extract important health information. Electronic medical record systems, such as Epic and Cerner, are used to record and manage final consultation information.

[1022] Specifically, patient reception is performed as follows: When a user (patient) holds their patient card over the reception terminal, the terminal reads the information on the card and sends it to the server. The server identifies the patient from the information on the card, retrieves the patient's profile information and past medical history from a database, and sends this to the terminal. The terminal then displays the patient information on its screen.

[1023] The conversation begins as follows: The device greets the patient through an AI voice, saying, "Hello, how are you?", and the user (patient) verbally talks about their health and recent situation. The device's microphone collects the patient's voice and sends it to the voice recognition software.

[1024] The process of organizing and recording conversations is as follows: The device converts collected voice data into text data in real time and sends it to the server. The server analyzes the text data and uses natural language processing technology to extract important health information. The extracted information is compared with past medical examination data. The server retrieves the data from the database and analyzes the results of the comparison.

[1025] Here are some examples of prompts:

[1026] Patient A swipes his / her medical card at the reception desk. The AI ​​then greets him / her with a voice message saying, "Hello, how are you feeling?" Patient A responds, "I've been feeling a little tired lately, but I'm fine." Please convert this conversation into text data, extract important health information, compare it with past data, and summarize the results. Please also explain the process for providing feedback on the summary information to the patient.

[1027] By inputting this prompt into a generative AI model, the process of generating summary information and providing feedback can be automated. This system will improve the efficiency of information collection from patients and the quality of information provided to medical professionals.

[1028] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1029] Step 1: Patient admission

[1030] Input: The patient holds their patient card over the reception terminal.

[1031] Operation: The RFID reader or barcode reader on the terminal reads the patient card information and sends it to the server.

[1032] Server: Identifies the patient from the information on the patient card and retrieves the patient's profile information and past medical history from the database.

[1033] Output: The server sends the acquired information to the terminal, which displays the patient information on the screen. This allows the reception staff to check the patient's basic information and past medical history.

[1034] Step 2: Start a conversation

[1035] Input: When the terminal initiates the initial interaction with the patient.

[1036] How it works: The device's speaker outputs the AI ​​voice saying, "Hello, how are you feeling?"

[1037] User (patient): The patient speaks verbally to the terminal about their physical condition and recent situation. The microphone on the terminal collects the patient's voice.

[1038] Output: The collected voice data is sent to speech recognition software, which gathers initial information about the patient's condition.

[1039] Step 3: Organize and record the conversation

[1040] Input: User (patient) voice data.

[1041] How it works: Speech recognition software (e.g., Google Cloud Speech-to-Text API) converts voice data into text in real time.

[1042] Terminal: Sends text data to the server.

[1043] Server: Natural language processing software (e.g., spaCy, NLTK) analyzes the text data and extracts keywords and important health information.

[1044] Output: The extracted vital health information is recorded in a database, ready to be used in the next steps.

[1045] Step 4: Compare with previous medical records

[1046] Input: Newly extracted health information and past consultation data.

[1047] How it works: The server retrieves past medical history from a database and applies a comparison algorithm to compare the old and new health information.

[1048] Server: Detects anomalies and changes and evaluates the extent of the change.

[1049] Output: The comparison results are summarized and information is generated to inform healthcare professionals, allowing them to understand changes in the patient's health status.

[1050] Step 5: Provide summary information

[1051] Input: Summary information of the comparison results.

[1052] Operation: The server sends the summarized information to the terminal, which then sends it to the display device.

[1053] Terminal: Provides necessary feedback to the patient through a holographic display or screen display.

[1054] User (Patient): The patient receives the feedback and decides on their next action (e.g., to schedule a doctor's appointment) based on it.

[1055] Output: Patients will have up-to-date information about their health status, providing guidance on taking appropriate next steps.

[1056] Step 6: Reflecting in the medical record

[1057] Input: Final consultation information and summary information.

[1058] Operation: The server sends medical information to the electronic medical record system.

[1059] Server: Information is stored in a database via the electronic medical record system's API.

[1060] Electronic medical record system: Makes recorded information easily accessible and viewable by medical professionals.

[1061] Output: The consultation information is recorded in the electronic medical record, providing healthcare professionals with up-to-date patient information, which assists in the development of ongoing care plans.

[1062] (Application example 1)

[1063] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1064] Monitoring the health of workers is extremely important in modern factories. Repetitive, simple tasks and heavy labor can place a strain on workers, putting them at risk of developing health problems. Early detection of abnormalities and improvements to the working environment are essential. However, many current systems lack the means to efficiently collect and properly analyze worker health information, making it difficult to respond appropriately and in a timely manner.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1066] In this invention, the server includes a means for reading the worker's identification information, a means for conducting a voice dialogue with the worker, and a means for converting the voice data into text, thereby enabling the server to efficiently collect and analyze the worker's health information and provide appropriate feedback in a timely manner.

[1067] "Worker identification information" refers to information used to identify individual workers in a factory, and is obtained through cards, ID badges, biometrics, etc.

[1068] "Voice dialogue" is a means of communication between the system and the worker through voice, and health information and work status are obtained in the form of questions and answers.

[1069] "Means for converting voice data to text" refers to the process of using voice recognition technology to convert the worker's voice into text data, which is then stored in a form that the system can understand and analyze.

[1070] The "means for analyzing text data to extract important health information" is a process for analyzing text data converted from speech and identifying and extracting important health information.

[1071] "Past health data" refers to a worker's past health information and records, and serves as reference data for comparison with newly acquired health information.

[1072] "Means for summarizing the comparison results and notifying the manager" refers to the process of briefly summarizing the results of comparing the extracted health information with past data and notifying the factory manager.

[1073] A "database" is an information storage system for storing worker health information and comparison results, which can be searched and referenced as needed.

[1074] "Means for recording movements and facial expressions" refers to the use of cameras and sensors to capture and analyze physical information such as the movements, facial expressions, and facial expressions of workers.

[1075] "Means for providing feedback using a hologram or screen display" refers to a method for visually conveying extracted health information to the worker, and provides feedback by projecting a hologram or displaying it on a display.

[1076] This invention is a system for efficiently collecting and analyzing health information of workers working in a factory and providing it to a manager. The system includes means for reading the worker's identification information, means for conducting a voice dialogue with the worker, means for converting the voice data into text, means for analyzing the text data and extracting important health information, means for comparing the extracted health information with past health data, means for summarizing the comparison results and notifying the manager, and means for updating the summarized information in a database.

[1077] A specific example of the system configuration includes the following elements:

[1078] First, a worker swipes their factory identification card, and the server confirms the worker's information and displays it on the terminal. RFID readers and biometric systems are used to read the identification information. For example, when a worker swipes their ID card, the reader transmits the card information to the server, which then displays the worker's profile and health data based on that information.

[1079] Next, the device engages in a voice dialogue with the worker. The device asks, "Hello, how are you feeling today?" and the worker talks about their physical condition and working environment. A microphone, speaker, and voice recognition and generation technologies are used for the voice dialogue. The voice data acquired in this process is converted into text via voice recognition software (e.g., Google Speech Recognition API).

[1080] After the voice data is converted to text, the server analyzes the text data to extract important health information. Generative AI models (such as Hugging Face's Transformers library) are used to identify health-related keywords and phrases. This extracted health information is then compared to past health data, which is pulled from a database on the server and compared to the worker's current condition.

[1081] The comparison results are summarized and notified to the administrator. The server generates summary information and sends it to the administrator's terminal. Based on this information, the administrator can understand the health status of the workers and take appropriate measures if necessary.

[1082] Finally, the summary information is entered into a database and stored for future reference, allowing for the accumulation and analysis of long-term health data.

[1083] As a specific example, the following scenario can be envisioned.

[1084] A worker holds his / her ID card over the terminal of a factory robot and says, "I've been suffering from severe shoulder stiffness lately." The system converts this voice information into text and extracts the information that "the shoulder stiffness has continued." After comparing it with past data, the system summarizes it as "If the shoulder stiffness has continued for more than three days, please consider improving your work environment," and notifies the manager. This allows for swift improvements to be made to the work environment.

[1085] Example prompt sentence:

[1086] The worker swipes his / her ID card and talks about his / her health condition: "I've been having really bad shoulder pain lately..."

[1087] System: "Do you have persistent shoulder pain? Let's check your past data."

[1088] Result: "If your shoulder pain persists for more than three days, consider improving your work environment."

[1089] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1090] Step 1:

[1091] The user swipes their factory identification card. The terminal uses an RFID reader and biometric system to read the card's information. This identification information is sent to a server, which retrieves the worker's profile and past health data from a database and displays them on the terminal.

[1092] Input: Worker identification information

[1093] Data processing: reading identifying information, retrieving and displaying data from databases

[1094] Output: Display worker profile information

[1095] Step 2:

[1096] The device asks the worker, "Hello, how are you feeling today?" The user then begins talking about their physical condition and working environment.

[1097] Input: Voice regarding worker's health condition

[1098] Data processing: Acquisition of audio data

[1099] Output: Audio data

[1100] Step 3:

[1101] The device uses speech recognition software (e.g., Google Speech Recognition API) to convert the captured voice data into text in real time, which is then sent to the server.

[1102] Input: Audio data

[1103] Data processing: speech-to-text conversion

[1104] Output: Text data

[1105] Step 4:

[1106] The server analyzes the received text data using a generative AI model (e.g., Hugging Face's Transformers library) to extract important health information.

[1107] Input: Text data

[1108] Data processing: Analysis of text data and extraction of health information

[1109] Output: Extracted health information

[1110] Step 5:

[1111] The server retrieves past data from the database to compare the extracted health information with past health data, detects new abnormalities or changes, and notifies the administrator in a summarized form.

[1112] Input: Extracted health information, historical health data

[1113] Data processing: Comparing and summarizing health information

[1114] Output: Summarized health information

[1115] Step 6:

[1116] The summarized information is sent to a terminal, which then displays it as feedback to the worker via a hologram or display, such as a message like, "Are you still suffering from stiff shoulders? We'll compare it with past data."

[1117] Input: Abstracted health information

[1118] Data processing: Displaying summary information

[1119] Output: Display feedback information to the worker

[1120] Step 7:

[1121] The server then updates the database with the summarized information and stores it for future reference. It then sends instructions to the administrator, such as, "If your shoulder pain persists for more than three days, please consider improving your work environment."

[1122] Input: Abstracted health information

[1123] Data processing: storing information in a database

[1124] Output: Save the updated health data

[1125] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1126] This invention is a system for efficiently collecting patient health information and emotional states and providing them to healthcare professionals. The system includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, a means for converting voice data to text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying healthcare professionals, and a means for updating the summarized information in an electronic medical record. It also includes a means for recording the patient's behavior and complexion, a means for providing feedback on the extracted health information via a hologram or a screen display, and an emotion engine for recognizing the user's emotions.

[1127] What the program does

[1128] 1. Patient admission process

[1129] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[1130] 2. Start a conversation

[1131] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[1132] 3. Organizing and recording conversation content

[1133] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[1134] 4. Emotional Recognition

[1135] The device uses an emotion engine to recognize emotions from the user's voice and facial expression, for example, identifying emotions such as "stress," "anxiety," and "happiness" from the tone of voice and facial expression.

[1136] 5. Comparison with past medical examination information

[1137] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1138] 6. Providing summary information

[1139] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1140] 7. Reflection in medical records

[1141] The server records the final medical information in an electronic medical record system, which records the user's emotional state as well as their health condition.

[1142] Specific examples

[1143] 1. Reception process

[1144] Patient B holds his / her patient card over the reception desk. The server sends Patient B's latest health record to the terminal, which displays Patient B's information.

[1145] 2. Start a conversation

[1146] Terminal: "Hello, Patient B. How are you feeling?"

[1147] Patient B: "I haven't been sleeping well lately."

[1148] 3. Organizing and recording conversation content

[1149] The device converts Patient B's conversation into text and sends it to the server. The server extracts the information that "I can't sleep well" as important health information.

[1150] 4. Emotional Recognition

[1151] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[1152] 5. Comparison with past medical examination information

[1153] The server compares the data with past medical examination data and detects an increase in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies the doctor.

[1154] 6. Providing summary information

[1155] The device displays the doctor's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[1156] 7. Reflection in medical records

[1157] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[1158] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[1159] The processing flow will be explained below.

[1160] Step 1:

[1161] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[1162] Step 2:

[1163] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[1164] Step 3:

[1165] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[1166] Step 4:

[1167] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[1168] Step 5:

[1169] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache").

[1170] Step 6:

[1171] The device uses an emotion engine to recognize emotions from the user's voice and facial expression. For example, it identifies emotions such as "stress," "anxiety," and "joy" from the tone of voice and facial expression.

[1172] Step 7:

[1173] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1174] Step 8:

[1175] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1176] Step 9:

[1177] The user reviews the feedback and provides additional questions or information as needed.

[1178] Step 10:

[1179] The server records the final consultation information and emotional state in the electronic medical record system, where doctors and other medical professionals can access this information. The process is complete.

[1180] Specific examples

[1181] Step 1:

[1182] Patient B holds his / her patient card over the reception desk. The terminal reads the information on the patient card and sends it to the server.

[1183] Step 2:

[1184] The server obtains detailed information and past medical history of Patient B and sends it to the terminal, which displays it on the screen.

[1185] Step 3:

[1186] The device greets the patient with a voice message saying, "Hello, Patient B. How are you feeling?" Patient B responds, "I haven't been sleeping well lately."

[1187] Step 4:

[1188] The terminal converts patient B's voice into text in real time and sends the text data to the server. The server receives the text data.

[1189] Step 5:

[1190] The server analyzes the text data and extracts important health information such as "not sleeping well."

[1191] Step 6:

[1192] The device uses an emotion engine to recognize the emotion "anxiety" from patient B's tone of voice.

[1193] Step 7:

[1194] The server compares the health information (e.g., "not sleeping well") and the emotional information (e.g., "anxiety") with past medical examination data to detect any abnormalities or changes, and creates a summary based on this information.

[1195] Step 8:

[1196] The server sends the summary information it has created to the terminal, which then displays a hologram to Patient B, saying, "You seem to have been having trouble sleeping and feeling anxious lately. I recommend you come in for your next appointment."

[1197] Step 9:

[1198] Patient B reviews the feedback and asks additional questions if necessary.

[1199] Step 10:

[1200] The server records the final medical information, such as "not sleeping well" and "anxiety," in the electronic medical record system. This information can be referenced by doctors and other medical professionals, completing the process.

[1201] Example 2

[1202] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1203] Conventional medical systems lack mechanisms for efficiently collecting patients' health information and emotional states and providing them appropriately to medical professionals. In particular, it is difficult to analyze a patient's emotional state in detail and compare it with past data to detect abnormalities or changes in progress. This can result in doctors taking a long time to grasp a patient's overall condition, potentially resulting in a decline in the quality of medical services. To solve this issue, a system is needed that can collect patients' health information and emotional states in real time and appropriately analyze and provide them.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1205] In this invention, the server includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, and a means for converting voice data into text. This automates the process from obtaining patient identification information to collecting health information, enabling efficient information collection. The server also includes a means for analyzing text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, a means for updating the summarized information in an electronic medical record, a means for recognizing the patient's emotional state from voice and video, a means for comparing the emotional information with past data to detect abnormalities or changes in the patient's condition, and a means for summarizing the detected emotional information and notifying medical professionals. This allows for real-time and efficient collection of patient health information and emotional state, and allows for appropriate analysis and provision. This allows medical professionals to quickly grasp the patient's overall condition and improve the quality of medical services.

[1206] "Means for reading patient identification information" refers to the function of transmitting a patient's personal information and medical history to the system using a medium such as a patient registration card or IC card.

[1207] "Means for conducting voice dialogue with patients" refers to a function that uses voice recognition technology to enable two-way voice communication between the system and patients.

[1208] "Means for converting voice data into text" refers to technology that converts voice dialogue between the patient and the system into text in real time.

[1209] "Means for analyzing text data to extract important health information" refers to technology that automatically extracts medically important keywords and phrases from converted text data.

[1210] "Means for comparing extracted health information with past medical examination data" refers to the function of checking newly collected health information against past medical examination history to detect abnormalities or changes.

[1211] "Means for summarizing comparison results and notifying healthcare professionals" refers to a function that automatically summarizes the key points of the comparison results and provides them to healthcare professionals in an easy-to-understand format.

[1212] "Means for reflecting summary information in electronic medical records" refers to a function that automatically adds the generated summary information to the patient's electronic medical record.

[1213] "Means for recognizing a patient's emotional state from audio and video" refers to technology that analyzes the patient's tone of voice and facial expression to identify their emotional state at that time.

[1214] "Means for comparing emotional information with past data to detect abnormalities or changes in progress" refers to a function that compares the recognized emotional state with past emotional data to detect abnormalities or changes.

[1215] "Means for summarizing detected emotional information and notifying medical professionals" refers to the function of summarizing the results of emotion analysis and providing them to medical professionals.

[1216] The present invention is a system for efficiently collecting and providing a patient's health information and emotional state to a medical professional. A specific embodiment of this system will be described.

[1217] The system consists of the following main components:

[1218] A means of reading patient identification information

[1219] A means of conducting voice dialogue with patients

[1220] A means of converting voice data into text

[1221] A means of analyzing text data to extract important health information

[1222] A means of comparing extracted health information with past medical examination data

[1223] A means of summarizing comparison results and communicating them to healthcare professionals

[1224] A means of reflecting summary information in electronic medical records

[1225] A means for recognizing a patient's emotional state from audio and video

[1226] A means of comparing emotional information with past data to detect abnormalities and changes in progress

[1227] A means of summarizing detected emotional information and notifying medical professionals

[1228] System configuration and operation

[1229] 1. Means of reading patient identification:

[1230] When a user holds their patient card over the reception desk, the server reads the card and acquires the patient's identification information. For example, the patient card contains a barcode or IC chip, and a barcode reader or IC card reader is used to read this.

[1231] 2. Means of audio communication with the patient:

[1232] Using the device's built-in voice recognition engine (e.g., Google Cloud Speech-to-Text), the device begins a voice dialogue with the patient by asking questions such as "Hello, how are you feeling?"

[1233] 3. Means of converting audio data to text:

[1234] The device records conversations with patients and converts the audio data into text in real time, which is then sent to a server.

[1235] 4. How to analyze text data and extract important health information:

[1236] The server analyzes the received text data and uses natural language processing to extract important health information, such as a statement like, "I haven't been sleeping well lately."

[1237] 5. Means of comparing extracted health information with past medical examination data:

[1238] The server compares the newly extracted health information with past medical records, detecting any abnormalities or changes in progress.

[1239] 6. Means of summarizing comparison results and communicating them to healthcare professionals:

[1240] The server summarises key points from the comparison results and provides them to healthcare professionals via a notification system.

[1241] 7. How to incorporate summary information into the electronic medical record:

[1242] The server automatically records the generated summary information in the patient's electronic medical record, using a commonly used system such as Epic.

[1243] 8. Means for recognizing a patient's emotional state from audio and visual:

[1244] The device uses an emotion recognition engine (e.g., Amazon Rekognition) built into it to analyze the patient's tone of voice and facial expression to recognize their emotional state.

[1245] 9. Comparing emotional information with historical data to detect anomalies and changes in progress:

[1246] The server compares the emotional information with past emotional data to detect any abnormalities or changes in progress.

[1247] 10. Means for summarizing detected emotional information and notifying healthcare professionals:

[1248] The server summarizes changes in emotional information and notifies healthcare professionals, for example, by providing information such as "The patient continues to be anxious."

[1249] Specific example operation procedure

[1250] 1. Reception process:

[1251] When a user (Patient B) holds their patient card over the reception desk, the server sends Patient B's latest health record to the terminal, and the terminal displays Patient B's information.

[1252] 2. Start a conversation:

[1253] Terminal: "Hello, Patient B. How are you feeling?"

[1254] User: "I haven't been sleeping well lately."

[1255] 3. Organizing and recording conversations:

[1256] The device converts Patient B's conversation into text and sends it to the server, which extracts the information that "I can't sleep well" as important health information.

[1257] 4. Emotion Recognition:

[1258] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[1259] 5. Comparison with past medical examination information:

[1260] The server compares data from past visits and detects increases in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies healthcare professionals.

[1261] 6. Providing Summary Information:

[1262] The device displays the medical professional's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[1263] 7. Reflection in the medical record:

[1264] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[1265] Prompt Sentence Examples

[1266] Below is an example of a prompt that can be used as input to a generative AI model.

[1267] Patient B holds up his / her patient card at the reception desk, and the terminal greets him / her with "Hello, how are you feeling?", after which Patient B responds, "I haven't been sleeping well lately." Please convert this conversation into text data, recognize Patient B's emotion (anxiety) from his / her tone of voice and facial expression, and compare it with past medical examination data to provide summary information.

[1268] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[1269] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1270] Processing Steps

[1271] Step 1: Patient admission

[1272] Specific behavior:

[1273] The user swipes their patient card.

[1274] The server reads the information on the patient card and checks the patient's profile information and past medical history.

[1275] The server sends the confirmed information to the terminal.

[1276] The terminal displays the patient's profile information and medical history.

[1277] Input / Output:

[1278] Input: Patient card information

[1279] Data processing / calculation: Extracting patient profile information and past medical history from patient card information

[1280] Output: Display patient profile information and medical history on terminal

[1281] Step 2: Start a conversation

[1282] Specific behavior:

[1283] The device uses a voice recognition engine to interactively ask the patient questions such as "Hello, how are you feeling?"

[1284] The user begins talking about their health and recent events.

[1285] Input / Output:

[1286] Input: Patient consultation start flag

[1287] Data processing / calculation: Activating the speech recognition engine and generating interactive questions

[1288] Output: Voice input from the patient

[1289] Step 3: Organize and record the conversation

[1290] Specific behavior:

[1291] The device records conversations with patients and converts the audio data into text data in real time.

[1292] The terminal sends the converted text data to the server.

[1293] The server analyzes the text data and extracts important health information.

[1294] Input / Output:

[1295] Input: Voice conversation with patient

[1296] Data processing / calculation: Converting voice to text and analyzing it using natural language processing technology

[1297] Output: Text data containing extracted health information

[1298] Step 4: Recognize emotions

[1299] Specific behavior:

[1300] The device uses a built-in camera and microphone to analyze the patient's complexion and tone of voice.

[1301] The device uses an emotion engine to recognize emotions such as "stress," "anxiety," and "joy."

[1302] Input / Output:

[1303] Input: Patient audio and video data

[1304] Data processing / calculation: Voice tone and facial color analysis

[1305] Output: Emotional state identification data

[1306] Step 5: Compare with previous medical records

[1307] Specific behavior:

[1308] The server compares the newly extracted health and emotional information with past medical records.

[1309] The server detects anomalies and changes in progress and compiles this information into a summary.

[1310] Input / Output:

[1311] Input: Extracted health and emotion information

[1312] Data processing / calculation: Comparison with past medical examination data

[1313] Output: Summarized consultation results

[1314] Step 6: Provide summary information

[1315] Specific behavior:

[1316] The server sends the created summary information to the terminal.

[1317] The terminal displays summary information on a hologram or screen and provides feedback to the patient.

[1318] Input / Output:

[1319] Input: Summarized consultation results

[1320] Data processing / calculation: Converting summary information into a display format

[1321] Output: Feedback message displayed to the patient

[1322] Step 7: Reflecting in the medical record

[1323] Specific behavior:

[1324] The server records the final medical information and emotional state in the electronic medical record system.

[1325] Input / Output:

[1326] Input: Final consultation information and emotional state

[1327] Data processing / calculation: Reflecting medical information in the electronic medical record system

[1328] Output: Updated electronic medical record

[1329] The above is the flow and operation of the specific processing steps. This system makes it possible to efficiently collect patient health information and emotional state and provide appropriate feedback to medical professionals.

[1330] (Application example 2)

[1331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1332] Conventional patient management systems have difficulty understanding a patient's health and emotional state in real time, making it difficult for medical professionals to provide accurate examinations and advice. There are also issues with efficiency in patient care, which could lead to a decline in the quality of medical services. In addition, detailed information, including the patient's emotional state, cannot be reflected in the electronic medical record, making it difficult to compare with past medical data or to adequately manage patients continuously. The present invention aims to solve these problems.

[1333] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summary information in the electronic medical record, means for recognizing the emotional state, and means for displaying the summary information including the recognized emotional state on a terminal in a physical store. This allows for efficient collection of patient health information and emotional state and provision of the information to medical professionals in real time, thereby enabling the quality of medical examinations and the overall improvement of medical services.

[1334] "Means for reading patient identification information" refers to devices or systems for verifying the patient's identity and obtaining data from patient cards, NFC tags, etc.

[1335] The "means for conducting a voice dialogue with a patient" refers to a device or system that receives voice input from a patient and automatically generates and provides questions and responses based on the content of the voice input.

[1336] "Means for converting voice data into text" refers to a device or system that uses voice recognition technology to convert what a patient says into text data.

[1337] The "means for analyzing text data and extracting important health information" refers to a device or system that analyzes acquired text data using natural language processing technology and identifies important information related to health checkups.

[1338] The "means for comparing extracted health information with past medical examination data" refers to a device or system that compares a patient's current health information with past medical examination data to detect abnormalities or changes.

[1339] "Means for summarizing the comparison results and notifying medical professionals" refers to a device or system that concisely summarizes the comparison results of health information and promptly notifies medical professionals.

[1340] The "means for reflecting summarized information in the electronic medical record" refers to a device or system that automatically inputs and records summarized health information into the electronic medical record system.

[1341] "Means for recognizing emotional state" refers to a device or system that determines the psychological emotional state of a patient from the tone of voice, facial expression, etc.

[1342] The "means for displaying summary information including the recognized emotional state on a terminal in a physical store" refers to a device or system that displays a summary of health information including the patient's emotional state on a terminal such as a display or hologram installed in a physical store.

[1343] This invention relates to a system that efficiently collects patient health information and emotional states and provides them to medical professionals, aiming to improve the efficiency of patient care, particularly in physical stores, and provide higher quality medical services.

[1344] System configuration

[1345] The system includes the following main elements:

[1346] 1. A means of reading patient identification information

[1347] 2. Means of communicating with patients through voice

[1348] 3. Means of converting audio data into text

[1349] 4. A means of analyzing text data to extract important health information

[1350] 5. A means to compare extracted health information with past medical examination data

[1351] 6. Means of summarizing comparison results and communicating them to healthcare professionals

[1352] 7. Means of reflecting summary information in electronic medical records

[1353] 8. A means of recognizing emotional states

[1354] 9. A means to display summary information including the recognized emotional state on a brick-and-mortar terminal

[1355] What the program does

[1356] This system performs the following processing by the server, terminal, and user.

[1357] Reception process

[1358] The server uses an NFC reader to read the patient's identification information from their medical card, and the identified patient information is automatically displayed on the terminal.

[1359] Voice dialogue

[1360] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) and a speech synthesis API (such as Google Text-to-Speech) to engage in natural voice conversations with patients. As patients talk about their health and symptoms, their speech is converted into text in real time.

[1361] Data analysis and emotion recognition

[1362] The server analyzes the text data using a natural language processing engine (e.g., NLTK, SpaCy) to extract important health information, and also uses an emotion analysis engine to recognize the patient's emotional state from their tone of voice and facial expressions.

[1363] Comparison and Summary

[1364] The server compares the extracted health information and recognized emotion information with past medical examination data, summarizes the results, and notifies medical professionals.

[1365] Information Feedback

[1366] The device provides the patient with summary information, including the patient's recognized emotional state, via a hologram or a screen display, and the summary information is automatically reflected in the electronic medical record system.

[1367] Adding specific examples

[1368] Patient A's reception and health information collection

[1369] 1. Patient A holds his / her smartphone over the NFC reader at the reception desk.

[1370] 2. The device asks, "Hello, Patient A. How are you feeling right now?"

[1371] 3. Patient A answers, "I've been having terrible headaches lately."

[1372] 4. The device converts the keywords "headache" and "recently" into text data and sends it to the server.

[1373] 5. The server performs emotion analysis and recognizes the emotions "pain" and "anxiety."

[1374] 6. The server compares the data with past medical examination data and notifies the doctor of the summary results.

[1375] 7. The device displays a hologram to Patient A saying, "You've been experiencing severe headaches and anxiety recently. I recommend you see a doctor."

[1376] 8. The server records the new medical information, "headache" and "anxiety," in the electronic medical record.

[1377] Example prompts for generative AI models

[1378] 1. Speech Recognition Engine:

[1379] Transcribe the following audio file to text: "A patient is talking about their health. The content is, 'I've been having really bad headaches lately.'"

[1380] 2. Sentiment Analysis Engine:

[1381] Recognize the emotion in the following text: "I've been having terrible headaches lately." Return the emotion tag (pain, anxiety, etc.).

[1382] 3. Natural Language Processing Engine:

[1383] Extract keywords from the following text: "I've been having terrible headaches lately." Return a list of important keywords.

[1384] The entire system efficiently collects patients' health information and emotional state and provides it to medical professionals in real time, thereby improving the quality of consultations and overall medical services.

[1385] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1386] Step 1:

[1387] The user holds their smartphone over the NFC reader at the reception desk. The NFC reader obtains the patient's identification information and sends it to the server. The input is the patient's medical card, and the output is the patient's identification information. Through this data processing, the patient's past medical examination data is obtained from the server and displayed on a terminal at the physical store.

[1388] Step 2:

[1389] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) to initiate a voice dialogue with the patient. The device asks, "Hello, Patient A. How are you feeling right now?" and the user responds. The input is voice data, and the output is text data. This voice data is converted into text in real time and sent to the server.

[1390] Step 3:

[1391] The server analyzes the received text data using a natural language processing engine (NLTK, SpaCy, etc.) to extract important health information. The input is text data, and the output is the extracted health information. This data processing identifies keywords (e.g., "headache" or "lack of sleep").

[1392] Step 4:

[1393] The server uses an emotion analysis engine to analyze the patient's voice tone and facial expression data acquired from the device's camera to recognize their emotional state. The input is voice and image data, and the output is an emotion tag (e.g., "anxiety" or "stress"). This data calculation determines the patient's psychological state.

[1394] Step 5:

[1395] The server compares the extracted health information and recognized emotional information with past medical examination data. The inputs are health information, emotional information, and past medical examination data, and the output is the comparison results. This data calculation detects new abnormalities and changes in the progress.

[1396] Step 6:

[1397] The server summarizes the comparison results and generates summary information, which includes important health information and emotional state. The input is the comparison results, and the output is summary information. This data processing constructs a concise diagnosis and notifies medical professionals.

[1398] Step 7:

[1399] The terminal provides the patient with summary information via a hologram or screen display. The input is the summary information, and the output is the feedback content. This specific operation allows the patient to gain a deeper understanding of their own health condition.

[1400] Step 8:

[1401] The server reflects the summary information in the electronic medical record system. The input is summary information, and the output is an updated electronic medical record. This data processing records information that will be useful for subsequent examinations and treatment.

[1402] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1404] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1405] [Fourth embodiment]

[1406] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1407] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1409] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1413] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1414] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1415] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1416] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1417] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1418] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] The present invention is a system for efficiently collecting patient health information and providing it to medical professionals. This system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data and extracting important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, and a means for updating an electronic medical record with the summarized information.

[1420] What the program does

[1421] 1. Patient admission process

[1422] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[1423] 2. Start a conversation

[1424] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[1425] 3. Organizing and recording conversation content

[1426] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[1427] 4. Comparison with past medical examination information

[1428] The server compares the extracted information with past examination data to detect new abnormalities or changes in progress, summarizes this information, and notifies the doctor.

[1429] 5. Providing summary information

[1430] The server sends the summarized information to the terminal, which then provides feedback to the user (patient) via a hologram or screen display. The user receives feedback about their physical condition and is instructed to see a doctor if necessary.

[1431] 6. Reflection in medical records

[1432] The server records the final consultation information in an electronic medical record system, making it available for viewing by doctors and other medical professionals.

[1433] Specific examples

[1434] 1. Reception process

[1435] Patient A holds his / her patient card over the reception desk. The server sends Patient A's latest health record to the terminal, which displays Patient A's information.

[1436] 2. Start a conversation

[1437] Terminal: "Hello, Patient A. How are you feeling?"

[1438] Patient A: "I've been feeling a little tired lately, but I'm OK."

[1439] 3. Organizing and recording conversation content

[1440] The device converts Patient A's conversation into text and sends it to the server. The server extracts the information that says "feeling tired" as important health information.

[1441] 4. Comparison with past medical examination information

[1442] The server compares the data with past medical examination data and detects that the patient's tendency to tire easily has increased. The server summarizes this information and notifies the doctor that "Patient A has recently become more prone to fatigue."

[1443] 5. Providing summary information

[1444] The device displays the doctor's feedback to Patient A as a hologram, saying, "We recommend that you come in for your next appointment to investigate the cause of your fatigue."

[1445] 6. Reflection in medical records

[1446] The server records the medical information, such as "Patient A's tendency to fatigue has increased," in the electronic medical record. Doctors and other medical professionals can then refer to this information.

[1447] In this way, the system of the present invention efficiently collects important health information through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[1448] The processing flow will be explained below.

[1449] Step 1:

[1450] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[1451] Step 2:

[1452] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[1453] Step 3:

[1454] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[1455] Step 4:

[1456] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[1457] Step 5:

[1458] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache"). It also collects data to observe the user's complexion and behavior.

[1459] Step 6:

[1460] The server compares the extracted health information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1461] Step 7:

[1462] The server sends the summary information it has created to the terminal. The terminal then provides feedback to the user (patient) using a hologram or a screen display of the summary information. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1463] Step 8:

[1464] The user reviews the feedback and provides additional questions or information as needed. After all information has been collected, the server records the final consultation information in the electronic medical record system. The process is complete.

[1465] Example 1

[1466] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1467] Current healthcare systems require efficient collection of patient health information and prompt, accurate provision of it to healthcare professionals. However, manually collecting patient information requires significant time and effort, and there is a risk of information omissions. Delays in analyzing and providing feedback on the collected information can also lead to problems with the quality of healthcare services. Furthermore, there is a need to provide more accurate health information by incorporating non-verbal information, such as the patient's behavior and complexion.

[1468] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1469] In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summarized information in the electronic medical record, means for providing feedback on the summarized information to the patient, and means for processing the patient's voice data in real time. This automates the collection of information from patients and enables timely and accurate provision of information. Furthermore, by analyzing non-verbal information as well, more accurate health information can be provided to medical professionals.

[1470] "Means for reading patient identification information" refers to a device or mechanism for obtaining unique identification information from an ID card, patient registration card, or other item held by a patient.

[1471] "Means for conducting voice dialogue with patients" refers to a system for carrying out voice communication with patients. Specifically, it refers to devices and software that use a microphone and speaker to conduct dialogue.

[1472] "Means for converting voice data into text" refers to software or a device that uses voice recognition technology to convert input voice data into text data.

[1473] The "means for analyzing text data and extracting important health information" is a system that uses natural language processing technology to identify and extract important information about a patient's symptoms and physical condition from converted text data.

[1474] "Means for comparing extracted health information with past medical examination data" refers to algorithms or software that compare newly acquired health information with previously recorded medical examination data to detect abnormalities or changes.

[1475] "Means for summarizing the comparison results and notifying medical professionals" refers to a communication system or device that summarizes the results of the comparison with past data in an easily understandable format and notifies medical professionals such as doctors and nurses.

[1476] The "means for reflecting summary information in electronic medical records" refers to an interface or software for automatically recording the generated summary information in the medical institution's electronic medical record system.

[1477] "Means for providing summarized information to patients" refers to methods for providing summarized diagnostic results and health information to patients, and includes technologies such as holograms and screen displays.

[1478] The "means for processing patient voice data in real time" is a system that instantly converts collected voice data into text and performs the analysis required for subsequent processing in real time.

[1479] The present invention relates to a system for efficiently collecting patient health information and providing it to medical professionals. The system includes a means for reading patient identification information, a means for conducting a voice dialogue with the patient, a means for converting the voice data into text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying the medical professionals, a means for updating an electronic medical record with the summarized information, a means for providing feedback on the summarized information to the patient, and a means for processing the patient's voice data in real time.

[1480] The following hardware and software are used to implement the system of the present invention. The hardware used includes a patient registration card reader terminal, a voice interaction terminal, and a server. The patient registration card reader terminal is a device for reading patients' patient registration cards and has the ability to read barcodes and RFID tags. The voice interaction terminal is a device for conducting voice interactions with patients and is a tablet or dedicated terminal equipped with a microphone and speaker. The server is a central computer for data processing and storage.

[1481] The software used includes voice recognition software, data analysis software, and electronic medical record systems. Examples of voice recognition software include Google Cloud Speech-to-Text API, which allows for real-time conversion of voice data into text data. Data analysis software uses Python natural language processing libraries (NLTK, spaCy, etc.) to analyze text data and extract important health information. Electronic medical record systems, such as Epic and Cerner, are used to record and manage final consultation information.

[1482] Specifically, patient reception is performed as follows: When a user (patient) holds their patient card over the reception terminal, the terminal reads the information on the card and sends it to the server. The server identifies the patient from the information on the card, retrieves the patient's profile information and past medical history from a database, and sends this to the terminal. The terminal then displays the patient information on its screen.

[1483] The conversation begins as follows: The device greets the patient through an AI voice, saying, "Hello, how are you?", and the user (patient) verbally talks about their health and recent situation. The device's microphone collects the patient's voice and sends it to the voice recognition software.

[1484] The process of organizing and recording conversations is as follows: The device converts collected voice data into text data in real time and sends it to the server. The server analyzes the text data and uses natural language processing technology to extract important health information. The extracted information is compared with past medical examination data. The server retrieves the data from the database and analyzes the results of the comparison.

[1485] Here are some examples of prompts:

[1486] Patient A swipes his / her medical card at the reception desk. The AI ​​then greets him / her with a voice message saying, "Hello, how are you feeling?" Patient A responds, "I've been feeling a little tired lately, but I'm fine." Please convert this conversation into text data, extract important health information, compare it with past data, and summarize the results. Please also explain the process for providing feedback on the summary information to the patient.

[1487] By inputting this prompt into a generative AI model, the process of generating summary information and providing feedback can be automated. This system will improve the efficiency of information collection from patients and the quality of information provided to medical professionals.

[1488] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1489] Step 1: Patient admission

[1490] Input: The patient holds their patient card over the reception terminal.

[1491] Operation: The RFID reader or barcode reader on the terminal reads the patient card information and sends it to the server.

[1492] Server: Identifies the patient from the information on the patient card and retrieves the patient's profile information and past medical history from the database.

[1493] Output: The server sends the acquired information to the terminal, which displays the patient information on the screen. This allows the reception staff to check the patient's basic information and past medical history.

[1494] Step 2: Start a conversation

[1495] Input: When the terminal initiates the initial interaction with the patient.

[1496] How it works: The device's speaker outputs the AI ​​voice saying, "Hello, how are you feeling?"

[1497] User (patient): The patient speaks verbally to the terminal about their physical condition and recent situation. The microphone on the terminal collects the patient's voice.

[1498] Output: The collected voice data is sent to speech recognition software, which gathers initial information about the patient's condition.

[1499] Step 3: Organize and record the conversation

[1500] Input: User (patient) voice data.

[1501] How it works: Speech recognition software (e.g., Google Cloud Speech-to-Text API) converts voice data into text in real time.

[1502] Terminal: Sends text data to the server.

[1503] Server: Natural language processing software (e.g., spaCy, NLTK) analyzes the text data and extracts keywords and important health information.

[1504] Output: The extracted vital health information is recorded in a database, ready to be used in the next steps.

[1505] Step 4: Compare with previous medical records

[1506] Input: Newly extracted health information and past consultation data.

[1507] How it works: The server retrieves past medical history from a database and applies a comparison algorithm to compare the old and new health information.

[1508] Server: Detects anomalies and changes and evaluates the extent of the change.

[1509] Output: The comparison results are summarized and information is generated to inform healthcare professionals, allowing them to understand changes in the patient's health status.

[1510] Step 5: Provide summary information

[1511] Input: Summary information of the comparison results.

[1512] Operation: The server sends the summarized information to the terminal, which then sends it to the display device.

[1513] Terminal: Provides necessary feedback to the patient through a holographic display or screen display.

[1514] User (Patient): The patient receives the feedback and decides on their next action (e.g., to schedule a doctor's appointment) based on it.

[1515] Output: Patients will have up-to-date information about their health status, providing guidance on taking appropriate next steps.

[1516] Step 6: Reflecting in the medical record

[1517] Input: Final consultation information and summary information.

[1518] Operation: The server sends medical information to the electronic medical record system.

[1519] Server: Information is stored in a database via the electronic medical record system's API.

[1520] Electronic medical record system: Makes recorded information easily accessible and viewable by medical professionals.

[1521] Output: The consultation information is recorded in the electronic medical record, providing healthcare professionals with up-to-date patient information, which assists in the development of ongoing care plans.

[1522] (Application example 1)

[1523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1524] Monitoring the health of workers is extremely important in modern factories. Repetitive, simple tasks and heavy labor can place a strain on workers, putting them at risk of developing health problems. Early detection of abnormalities and improvements to the working environment are essential. However, many current systems lack the means to efficiently collect and properly analyze worker health information, making it difficult to respond appropriately and in a timely manner.

[1525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1526] In this invention, the server includes a means for reading the worker's identification information, a means for conducting a voice dialogue with the worker, and a means for converting the voice data into text, thereby enabling the server to efficiently collect and analyze the worker's health information and provide appropriate feedback in a timely manner.

[1527] "Worker identification information" refers to information used to identify individual workers in a factory, and is obtained through cards, ID badges, biometrics, etc.

[1528] "Voice dialogue" is a means of communication between the system and the worker through voice, and health information and work status are obtained in the form of questions and answers.

[1529] "Means for converting voice data to text" refers to the process of using voice recognition technology to convert the worker's voice into text data, which is then stored in a form that the system can understand and analyze.

[1530] The "means for analyzing text data to extract important health information" is a process for analyzing text data converted from speech and identifying and extracting important health information.

[1531] "Past health data" refers to a worker's past health information and records, and serves as reference data for comparison with newly acquired health information.

[1532] "Means for summarizing the comparison results and notifying the manager" refers to the process of briefly summarizing the results of comparing the extracted health information with past data and notifying the factory manager.

[1533] A "database" is an information storage system for storing worker health information and comparison results, which can be searched and referenced as needed.

[1534] "Means for recording movements and facial expressions" refers to the use of cameras and sensors to capture and analyze physical information such as the movements, facial expressions, and facial expressions of workers.

[1535] "Means for providing feedback using a hologram or screen display" refers to a method for visually conveying extracted health information to the worker, and provides feedback by projecting a hologram or displaying it on a display.

[1536] This invention is a system for efficiently collecting and analyzing health information of workers working in a factory and providing it to a manager. The system includes means for reading the worker's identification information, means for conducting a voice dialogue with the worker, means for converting the voice data into text, means for analyzing the text data and extracting important health information, means for comparing the extracted health information with past health data, means for summarizing the comparison results and notifying the manager, and means for updating the summarized information in a database.

[1537] A specific example of the system configuration includes the following elements:

[1538] First, a worker swipes their factory identification card, and the server confirms the worker's information and displays it on the terminal. RFID readers and biometric systems are used to read the identification information. For example, when a worker swipes their ID card, the reader transmits the card information to the server, which then displays the worker's profile and health data based on that information.

[1539] Next, the device engages in a voice dialogue with the worker. The device asks, "Hello, how are you feeling today?" and the worker talks about their physical condition and working environment. A microphone, speaker, and voice recognition and generation technologies are used for the voice dialogue. The voice data acquired in this process is converted into text via voice recognition software (e.g., Google Speech Recognition API).

[1540] After the voice data is converted to text, the server analyzes the text data to extract important health information. Generative AI models (such as Hugging Face's Transformers library) are used to identify health-related keywords and phrases. This extracted health information is then compared to past health data, which is pulled from a database on the server and compared to the worker's current condition.

[1541] The comparison results are summarized and notified to the administrator. The server generates summary information and sends it to the administrator's terminal. Based on this information, the administrator can understand the health status of the workers and take appropriate measures if necessary.

[1542] Finally, the summary information is entered into a database and stored for future reference, allowing for the accumulation and analysis of long-term health data.

[1543] As a specific example, the following scenario can be envisioned.

[1544] A worker holds his / her ID card over the terminal of a factory robot and says, "I've been suffering from severe shoulder stiffness lately." The system converts this voice information into text and extracts the information that "the shoulder stiffness has continued." After comparing it with past data, the system summarizes it as "If the shoulder stiffness has continued for more than three days, please consider improving your work environment," and notifies the manager. This allows for swift improvements to be made to the work environment.

[1545] Example prompt sentence:

[1546] The worker swipes his / her ID card and talks about his / her health condition: "I've been having really bad shoulder pain lately..."

[1547] System: "Do you have persistent shoulder pain? Let's check your past data."

[1548] Result: "If your shoulder pain persists for more than three days, consider improving your work environment."

[1549] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1550] Step 1:

[1551] The user swipes their factory identification card. The terminal uses an RFID reader and biometric system to read the card's information. This identification information is sent to a server, which retrieves the worker's profile and past health data from a database and displays them on the terminal.

[1552] Input: Worker identification information

[1553] Data processing: reading identifying information, retrieving and displaying data from databases

[1554] Output: Display worker profile information

[1555] Step 2:

[1556] The device asks the worker, "Hello, how are you feeling today?" The user then begins talking about their physical condition and working environment.

[1557] Input: Voice regarding worker's health condition

[1558] Data processing: Acquisition of audio data

[1559] Output: Audio data

[1560] Step 3:

[1561] The device uses speech recognition software (e.g., Google Speech Recognition API) to convert the captured voice data into text in real time, which is then sent to the server.

[1562] Input: Audio data

[1563] Data processing: speech-to-text conversion

[1564] Output: Text data

[1565] Step 4:

[1566] The server analyzes the received text data using a generative AI model (e.g., Hugging Face's Transformers library) to extract important health information.

[1567] Input: Text data

[1568] Data processing: Analysis of text data and extraction of health information

[1569] Output: Extracted health information

[1570] Step 5:

[1571] The server retrieves past data from the database to compare the extracted health information with past health data, detects new abnormalities or changes, and notifies the administrator in a summarized form.

[1572] Input: Extracted health information, historical health data

[1573] Data processing: Comparing and summarizing health information

[1574] Output: Summarized health information

[1575] Step 6:

[1576] The summarized information is sent to a terminal, which then displays it as feedback to the worker via a hologram or display, such as a message like, "Are you still suffering from stiff shoulders? We'll compare it with past data."

[1577] Input: Abstracted health information

[1578] Data processing: Displaying summary information

[1579] Output: Display feedback information to the worker

[1580] Step 7:

[1581] The server then updates the database with the summarized information and stores it for future reference. It then sends instructions to the administrator, such as, "If your shoulder pain persists for more than three days, please consider improving your work environment."

[1582] Input: Abstracted health information

[1583] Data processing: storing information in a database

[1584] Output: Save the updated health data

[1585] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1586] This invention is a system for efficiently collecting patient health information and emotional states and providing them to healthcare professionals. The system includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, a means for converting voice data to text, a means for analyzing the text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying healthcare professionals, and a means for updating the summarized information in an electronic medical record. It also includes a means for recording the patient's behavior and complexion, a means for providing feedback on the extracted health information via a hologram or a screen display, and an emotion engine for recognizing the user's emotions.

[1587] What the program does

[1588] 1. Patient admission process

[1589] When the user (patient) holds their patient card over the terminal, the server checks the patient information and displays it on the terminal. The server also sends the patient's profile information and past medical history to the terminal.

[1590] 2. Start a conversation

[1591] The device greets the patient in an AI voice, asking, "Hello, how are you feeling?" The user then begins talking about their health and recent situation.

[1592] 3. Organizing and recording conversation content

[1593] The device converts the conversation with the patient into text data in real time and sends it to a server, which then analyzes the conversation and extracts important health information based on facial expression, behavior, and other factors.

[1594] 4. Emotional Recognition

[1595] The device uses an emotion engine to recognize emotions from the user's voice and facial expression, for example, identifying emotions such as "stress," "anxiety," and "happiness" from the tone of voice and facial expression.

[1596] 5. Comparison with past medical examination information

[1597] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1598] 6. Providing summary information

[1599] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1600] 7. Reflection in medical records

[1601] The server records the final medical information in an electronic medical record system, which records the user's emotional state as well as their health condition.

[1602] Specific examples

[1603] 1. Reception process

[1604] Patient B holds his / her patient card over the reception desk. The server sends Patient B's latest health record to the terminal, which displays Patient B's information.

[1605] 2. Start a conversation

[1606] Terminal: "Hello, Patient B. How are you feeling?"

[1607] Patient B: "I haven't been sleeping well lately."

[1608] 3. Organizing and recording conversation content

[1609] The device converts Patient B's conversation into text and sends it to the server. The server extracts the information that "I can't sleep well" as important health information.

[1610] 4. Emotional Recognition

[1611] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[1612] 5. Comparison with past medical examination information

[1613] The server compares the data with past medical examination data and detects an increase in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies the doctor.

[1614] 6. Providing summary information

[1615] The device displays the doctor's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[1616] 7. Reflection in medical records

[1617] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[1618] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[1619] The processing flow will be explained below.

[1620] Step 1:

[1621] The user (patient) holds their patient card over the terminal reader. The terminal reads the information on the patient card and sends it to the server.

[1622] Step 2:

[1623] Based on the information on the patient card, the server retrieves the patient's profile information and past medical history from the database and sends it to the terminal.

[1624] Step 3:

[1625] The device displays the patient's information on the screen and greets the user with a voice message saying, "Hello, user. How are you feeling?" The user (patient) begins to talk about their health and recent situation.

[1626] Step 4:

[1627] The device converts the user's voice into text in real time and sends the text data to the server, which receives the text data.

[1628] Step 5:

[1629] The server analyzes the text data and extracts keywords and important health information (such as "fatigue" or "headache").

[1630] Step 6:

[1631] The device uses an emotion engine to recognize emotions from the user's voice and facial expression. For example, it identifies emotions such as "stress," "anxiety," and "joy" from the tone of voice and facial expression.

[1632] Step 7:

[1633] The server compares the extracted health information and recognized emotion information with past medical records to detect new abnormalities or changes in progress, and creates a summary based on this comparison.

[1634] Step 8:

[1635] The server sends the summary information it has created to the device, which then provides feedback to the user via a hologram or screen display. For example, it might say, "You seem to be getting tired more easily recently. I recommend you see a doctor."

[1636] Step 9:

[1637] The user reviews the feedback and provides additional questions or information as needed.

[1638] Step 10:

[1639] The server records the final consultation information and emotional state in the electronic medical record system, where doctors and other medical professionals can access this information. The process is complete.

[1640] Specific examples

[1641] Step 1:

[1642] Patient B holds his / her patient card over the reception desk. The terminal reads the information on the patient card and sends it to the server.

[1643] Step 2:

[1644] The server obtains detailed information and past medical history of Patient B and sends it to the terminal, which displays it on the screen.

[1645] Step 3:

[1646] The device greets the patient with a voice message saying, "Hello, Patient B. How are you feeling?" Patient B responds, "I haven't been sleeping well lately."

[1647] Step 4:

[1648] The terminal converts patient B's voice into text in real time and sends the text data to the server. The server receives the text data.

[1649] Step 5:

[1650] The server analyzes the text data and extracts important health information such as "not sleeping well."

[1651] Step 6:

[1652] The device uses an emotion engine to recognize the emotion "anxiety" from patient B's tone of voice.

[1653] Step 7:

[1654] The server compares the health information (e.g., "not sleeping well") and the emotional information (e.g., "anxiety") with past medical examination data to detect any abnormalities or changes, and creates a summary based on this information.

[1655] Step 8:

[1656] The server sends the summary information it has created to the terminal, which then displays a hologram to Patient B, saying, "You seem to have been having trouble sleeping and feeling anxious lately. I recommend you come in for your next appointment."

[1657] Step 9:

[1658] Patient B reviews the feedback and asks additional questions if necessary.

[1659] Step 10:

[1660] The server records the final medical information, such as "not sleeping well" and "anxiety," in the electronic medical record system. This information can be referenced by doctors and other medical professionals, completing the process.

[1661] Example 2

[1662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] Conventional medical systems lack mechanisms for efficiently collecting patients' health information and emotional states and providing them appropriately to medical professionals. In particular, it is difficult to analyze a patient's emotional state in detail and compare it with past data to detect abnormalities or changes in progress. This can result in doctors taking a long time to grasp a patient's overall condition, potentially resulting in a decline in the quality of medical services. To solve this issue, a system is needed that can collect patients' health information and emotional states in real time and appropriately analyze and provide them.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1665] In this invention, the server includes a means for reading patient identification information, a means for engaging in voice dialogue with the patient, and a means for converting voice data into text. This automates the process from obtaining patient identification information to collecting health information, enabling efficient information collection. The server also includes a means for analyzing text data to extract important health information, a means for comparing the extracted health information with past medical examination data, a means for summarizing the comparison results and notifying medical professionals, a means for updating the summarized information in an electronic medical record, a means for recognizing the patient's emotional state from voice and video, a means for comparing the emotional information with past data to detect abnormalities or changes in the patient's condition, and a means for summarizing the detected emotional information and notifying medical professionals. This allows for real-time and efficient collection of patient health information and emotional state, and allows for appropriate analysis and provision. This allows medical professionals to quickly grasp the patient's overall condition and improve the quality of medical services.

[1666] "Means for reading patient identification information" refers to the function of transmitting a patient's personal information and medical history to the system using a medium such as a patient registration card or IC card.

[1667] "Means for conducting voice dialogue with patients" refers to a function that uses voice recognition technology to enable two-way voice communication between the system and patients.

[1668] "Means for converting voice data into text" refers to technology that converts voice dialogue between the patient and the system into text in real time.

[1669] "Means for analyzing text data to extract important health information" refers to technology that automatically extracts medically important keywords and phrases from converted text data.

[1670] "Means for comparing extracted health information with past medical examination data" refers to the function of checking newly collected health information against past medical examination history to detect abnormalities or changes.

[1671] "Means for summarizing comparison results and notifying healthcare professionals" refers to a function that automatically summarizes the key points of the comparison results and provides them to healthcare professionals in an easy-to-understand format.

[1672] "Means for reflecting summary information in electronic medical records" refers to a function that automatically adds the generated summary information to the patient's electronic medical record.

[1673] "Means for recognizing a patient's emotional state from audio and video" refers to technology that analyzes the patient's tone of voice and facial expression to identify their emotional state at that time.

[1674] "Means for comparing emotional information with past data to detect abnormalities or changes in progress" refers to a function that compares the recognized emotional state with past emotional data to detect abnormalities or changes.

[1675] "Means for summarizing detected emotional information and notifying medical professionals" refers to the function of summarizing the results of emotion analysis and providing them to medical professionals.

[1676] The present invention is a system for efficiently collecting and providing a patient's health information and emotional state to a medical professional. A specific embodiment of this system will be described.

[1677] The system consists of the following main components:

[1678] A means of reading patient identification information

[1679] A means of conducting voice dialogue with patients

[1680] A means of converting voice data into text

[1681] A means of analyzing text data to extract important health information

[1682] A means of comparing extracted health information with past medical examination data

[1683] A means of summarizing comparison results and communicating them to healthcare professionals

[1684] A means of reflecting summary information in electronic medical records

[1685] A means for recognizing a patient's emotional state from audio and video

[1686] A means of comparing emotional information with past data to detect abnormalities and changes in progress

[1687] A means of summarizing detected emotional information and notifying medical professionals

[1688] System configuration and operation

[1689] 1. Means of reading patient identification:

[1690] When a user holds their patient card over the reception desk, the server reads the card and acquires the patient's identification information. For example, the patient card contains a barcode or IC chip, and a barcode reader or IC card reader is used to read this.

[1691] 2. Means of audio communication with the patient:

[1692] Using the device's built-in voice recognition engine (e.g., Google Cloud Speech-to-Text), the device begins a voice dialogue with the patient by asking questions such as "Hello, how are you feeling?"

[1693] 3. Means of converting audio data to text:

[1694] The device records conversations with patients and converts the audio data into text in real time, which is then sent to a server.

[1695] 4. How to analyze text data and extract important health information:

[1696] The server analyzes the received text data and uses natural language processing to extract important health information, such as a statement like, "I haven't been sleeping well lately."

[1697] 5. Means of comparing extracted health information with past medical examination data:

[1698] The server compares the newly extracted health information with past medical records, detecting any abnormalities or changes in progress.

[1699] 6. Means of summarizing comparison results and communicating them to healthcare professionals:

[1700] The server summarises key points from the comparison results and provides them to healthcare professionals via a notification system.

[1701] 7. How to incorporate summary information into the electronic medical record:

[1702] The server automatically records the generated summary information in the patient's electronic medical record, using a commonly used system such as Epic.

[1703] 8. Means for recognizing a patient's emotional state from audio and visual:

[1704] The device uses an emotion recognition engine (e.g., Amazon Rekognition) built into it to analyze the patient's tone of voice and facial expression to recognize their emotional state.

[1705] 9. Comparing emotional information with historical data to detect anomalies and changes in progress:

[1706] The server compares the emotional information with past emotional data to detect any abnormalities or changes in progress.

[1707] 10. Means for summarizing detected emotional information and notifying healthcare professionals:

[1708] The server summarizes changes in emotional information and notifies healthcare professionals, for example, by providing information such as "The patient continues to be anxious."

[1709] Specific example operation procedure

[1710] 1. Reception process:

[1711] When a user (Patient B) holds their patient card over the reception desk, the server sends Patient B's latest health record to the terminal, and the terminal displays Patient B's information.

[1712] 2. Start a conversation:

[1713] Terminal: "Hello, Patient B. How are you feeling?"

[1714] User: "I haven't been sleeping well lately."

[1715] 3. Organizing and recording conversations:

[1716] The device converts Patient B's conversation into text and sends it to the server, which extracts the information that "I can't sleep well" as important health information.

[1717] 4. Emotion Recognition:

[1718] The device analyzes patient B's tone of voice and facial expression and recognizes the emotion "anxiety."

[1719] 5. Comparison with past medical examination information:

[1720] The server compares data from past visits and detects increases in symptoms, particularly anxiety and poor sleep. The server summarizes this information and notifies healthcare professionals.

[1721] 6. Providing Summary Information:

[1722] The device displays the medical professional's feedback to Patient B in the form of a hologram, saying, "It appears you are not sleeping well and are continuing to feel anxious. We recommend that you come in for your next appointment."

[1723] 7. Reflection in the medical record:

[1724] The server records the patient's medical information, such as "not sleeping well" and "anxiety," in the electronic medical record, which can then be accessed by doctors and other medical professionals.

[1725] Prompt Sentence Examples

[1726] Below is an example of a prompt that can be used as input to a generative AI model.

[1727] Patient B holds up his / her patient card at the reception desk, and the terminal greets him / her with "Hello, how are you feeling?", after which Patient B responds, "I haven't been sleeping well lately." Please convert this conversation into text data, recognize Patient B's emotion (anxiety) from his / her tone of voice and facial expression, and compare it with past medical examination data to provide summary information.

[1728] In this way, the system of the present invention efficiently collects important health information and emotional state through communication with patients and provides it to medical professionals, thereby improving the quality of medical services.

[1729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1730] Processing Steps

[1731] Step 1: Patient admission

[1732] Specific behavior:

[1733] The user swipes their patient card.

[1734] The server reads the information on the patient card and checks the patient's profile information and past medical history.

[1735] The server sends the confirmed information to the terminal.

[1736] The terminal displays the patient's profile information and medical history.

[1737] Input / Output:

[1738] Input: Patient card information

[1739] Data processing / calculation: Extracting patient profile information and past medical history from patient card information

[1740] Output: Display patient profile information and medical history on terminal

[1741] Step 2: Start a conversation

[1742] Specific behavior:

[1743] The device uses a voice recognition engine to interactively ask the patient questions such as "Hello, how are you feeling?"

[1744] The user begins talking about their health and recent events.

[1745] Input / Output:

[1746] Input: Patient consultation start flag

[1747] Data processing / calculation: Activating the speech recognition engine and generating interactive questions

[1748] Output: Voice input from the patient

[1749] Step 3: Organize and record the conversation

[1750] Specific behavior:

[1751] The device records conversations with patients and converts the audio data into text data in real time.

[1752] The terminal sends the converted text data to the server.

[1753] The server analyzes the text data and extracts important health information.

[1754] Input / Output:

[1755] Input: Voice conversation with patient

[1756] Data processing / calculation: Converting voice to text and analyzing it using natural language processing technology

[1757] Output: Text data containing extracted health information

[1758] Step 4: Recognize emotions

[1759] Specific behavior:

[1760] The device uses a built-in camera and microphone to analyze the patient's complexion and tone of voice.

[1761] The device uses an emotion engine to recognize emotions such as "stress," "anxiety," and "joy."

[1762] Input / Output:

[1763] Input: Patient audio and video data

[1764] Data processing / calculation: Voice tone and facial color analysis

[1765] Output: Emotional state identification data

[1766] Step 5: Compare with previous medical records

[1767] Specific behavior:

[1768] The server compares the newly extracted health and emotional information with past medical records.

[1769] The server detects anomalies and changes in progress and compiles this information into a summary.

[1770] Input / Output:

[1771] Input: Extracted health and emotion information

[1772] Data processing / calculation: Comparison with past medical examination data

[1773] Output: Summarized consultation results

[1774] Step 6: Provide summary information

[1775] Specific behavior:

[1776] The server sends the created summary information to the terminal.

[1777] The terminal displays summary information on a hologram or screen and provides feedback to the patient.

[1778] Input / Output:

[1779] Input: Summarized consultation results

[1780] Data processing / calculation: Converting summary information into a display format

[1781] Output: Feedback message displayed to the patient

[1782] Step 7: Reflecting in the medical record

[1783] Specific behavior:

[1784] The server records the final medical information and emotional state in the electronic medical record system.

[1785] Input / Output:

[1786] Input: Final consultation information and emotional state

[1787] Data processing / calculation: Reflecting medical information in the electronic medical record system

[1788] Output: Updated electronic medical record

[1789] The above is the flow and operation of the specific processing steps. This system makes it possible to efficiently collect patient health information and emotional state and provide appropriate feedback to medical professionals.

[1790] (Application example 2)

[1791] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1792] Conventional patient management systems have difficulty understanding a patient's health and emotional state in real time, making it difficult for medical professionals to provide accurate examinations and advice. There are also issues with efficiency in patient care, which could lead to a decline in the quality of medical services. In addition, detailed information, including the patient's emotional state, cannot be reflected in the electronic medical record, making it difficult to compare with past medical data or to adequately manage patients continuously. The present invention aims to solve these problems.

[1793] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading patient identification information, means for engaging in voice dialogue with the patient, means for converting voice data to text, means for analyzing the text data to extract important health information, means for comparing the extracted health information with past medical examination data, means for summarizing the comparison results and notifying medical professionals, means for updating the summary information in the electronic medical record, means for recognizing the emotional state, and means for displaying the summary information including the recognized emotional state on a terminal in a physical store. This allows for efficient collection of patient health information and emotional state and provision of the information to medical professionals in real time, thereby enabling the quality of medical examinations and the overall improvement of medical services.

[1794] "Means for reading patient identification information" refers to devices or systems for verifying the patient's identity and obtaining data from patient cards, NFC tags, etc.

[1795] The "means for conducting a voice dialogue with a patient" refers to a device or system that receives voice input from a patient and automatically generates and provides questions and responses based on the content of the voice input.

[1796] "Means for converting voice data into text" refers to a device or system that uses voice recognition technology to convert what a patient says into text data.

[1797] The "means for analyzing text data and extracting important health information" refers to a device or system that analyzes acquired text data using natural language processing technology and identifies important information related to health checkups.

[1798] The "means for comparing extracted health information with past medical examination data" refers to a device or system that compares a patient's current health information with past medical examination data to detect abnormalities or changes.

[1799] "Means for summarizing the comparison results and notifying medical professionals" refers to a device or system that concisely summarizes the comparison results of health information and promptly notifies medical professionals.

[1800] The "means for reflecting summarized information in the electronic medical record" refers to a device or system that automatically inputs and records summarized health information into the electronic medical record system.

[1801] "Means for recognizing emotional state" refers to a device or system that determines the psychological emotional state of a patient from the tone of voice, facial expression, etc.

[1802] The "means for displaying summary information including the recognized emotional state on a terminal in a physical store" refers to a device or system that displays a summary of health information including the patient's emotional state on a terminal such as a display or hologram installed in a physical store.

[1803] This invention relates to a system that efficiently collects patient health information and emotional states and provides them to medical professionals, aiming to improve the efficiency of patient care, particularly in physical stores, and provide higher quality medical services.

[1804] System configuration

[1805] The system includes the following main elements:

[1806] 1. A means of reading patient identification information

[1807] 2. Means of communicating with patients through voice

[1808] 3. Means of converting audio data into text

[1809] 4. A means of analyzing text data to extract important health information

[1810] 5. A means to compare extracted health information with past medical examination data

[1811] 6. Means of summarizing comparison results and communicating them to healthcare professionals

[1812] 7. Means of reflecting summary information in electronic medical records

[1813] 8. A means of recognizing emotional states

[1814] 9. A means to display summary information including the recognized emotional state on a brick-and-mortar terminal

[1815] What the program does

[1816] This system performs the following processing by the server, terminal, and user.

[1817] Reception process

[1818] The server uses an NFC reader to read the patient's identification information from their medical card, and the identified patient information is automatically displayed on the terminal.

[1819] Voice dialogue

[1820] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) and a speech synthesis API (such as Google Text-to-Speech) to engage in natural voice conversations with patients. As patients talk about their health and symptoms, their speech is converted into text in real time.

[1821] Data analysis and emotion recognition

[1822] The server analyzes the text data using a natural language processing engine (e.g., NLTK, SpaCy) to extract important health information, and also uses an emotion analysis engine to recognize the patient's emotional state from their tone of voice and facial expressions.

[1823] Comparison and Summary

[1824] The server compares the extracted health information and recognized emotion information with past medical examination data, summarizes the results, and notifies medical professionals.

[1825] Information Feedback

[1826] The device provides the patient with summary information, including the patient's recognized emotional state, via a hologram or a screen display, and the summary information is automatically reflected in the electronic medical record system.

[1827] Adding specific examples

[1828] Patient A's reception and health information collection

[1829] 1. Patient A holds his / her smartphone over the NFC reader at the reception desk.

[1830] 2. The device asks, "Hello, Patient A. How are you feeling right now?"

[1831] 3. Patient A answers, "I've been having terrible headaches lately."

[1832] 4. The device converts the keywords "headache" and "recently" into text data and sends it to the server.

[1833] 5. The server performs emotion analysis and recognizes the emotions "pain" and "anxiety."

[1834] 6. The server compares the data with past medical examination data and notifies the doctor of the summary results.

[1835] 7. The device displays a hologram to Patient A saying, "You've been experiencing severe headaches and anxiety recently. I recommend you see a doctor."

[1836] 8. The server records the new medical information, "headache" and "anxiety," in the electronic medical record.

[1837] Example prompts for generative AI models

[1838] 1. Speech Recognition Engine:

[1839] Transcribe the following audio file to text: "A patient is talking about their health. The content is, 'I've been having really bad headaches lately.'"

[1840] 2. Sentiment Analysis Engine:

[1841] Recognize the emotion in the following text: "I've been having terrible headaches lately." Return the emotion tag (pain, anxiety, etc.).

[1842] 3. Natural Language Processing Engine:

[1843] Extract keywords from the following text: "I've been having terrible headaches lately." Return a list of important keywords.

[1844] The entire system efficiently collects patients' health information and emotional state and provides it to medical professionals in real time, thereby improving the quality of consultations and overall medical services.

[1845] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1846] Step 1:

[1847] The user holds their smartphone over the NFC reader at the reception desk. The NFC reader obtains the patient's identification information and sends it to the server. The input is the patient's medical card, and the output is the patient's identification information. Through this data processing, the patient's past medical examination data is obtained from the server and displayed on a terminal at the physical store.

[1848] Step 2:

[1849] The device uses a speech recognition engine (such as Google Cloud Speech-to-Text) to initiate a voice dialogue with the patient. The device asks, "Hello, Patient A. How are you feeling right now?" and the user responds. The input is voice data, and the output is text data. This voice data is converted into text in real time and sent to the server.

[1850] Step 3:

[1851] The server analyzes the received text data using a natural language processing engine (NLTK, SpaCy, etc.) to extract important health information. The input is text data, and the output is the extracted health information. This data processing identifies keywords (e.g., "headache" or "lack of sleep").

[1852] Step 4:

[1853] The server uses an emotion analysis engine to analyze the patient's voice tone and facial expression data acquired from the device's camera to recognize their emotional state. The input is voice and image data, and the output is an emotion tag (e.g., "anxiety" or "stress"). This data calculation determines the patient's psychological state.

[1854] Step 5:

[1855] The server compares the extracted health information and recognized emotional information with past medical examination data. The inputs are health information, emotional information, and past medical examination data, and the output is the comparison results. This data calculation detects new abnormalities and changes in the progress.

[1856] Step 6:

[1857] The server summarizes the comparison results and generates summary information, which includes important health information and emotional state. The input is the comparison results, and the output is summary information. This data processing constructs a concise diagnosis and notifies medical professionals.

[1858] Step 7:

[1859] The terminal provides the patient with summary information via a hologram or screen display. The input is the summary information, and the output is the feedback content. This specific operation allows the patient to gain a deeper understanding of their own health condition.

[1860] Step 8:

[1861] The server reflects the summary information in the electronic medical record system. The input is summary information, and the output is an updated electronic medical record. This data processing records information that will be useful for subsequent examinations and treatment.

[1862] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1864] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1865] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1866] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1867] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1868] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1869] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1870] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1871] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1872] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1873] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1874] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1875] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1876] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1877] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1878] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs,...

Claims

1. means for reading patient identification; means for engaging in a voice dialogue with the patient; a means for converting the audio data into text; a means for analyzing the text data to extract important health information; a means for comparing the extracted health information with past medical examination data; A means of summarizing the results of the comparison and communicating them to healthcare professionals; A system including a means for reflecting summary information in an electronic medical record.

2. 10. The system of claim 1, further comprising means for recording the patient's movements and complexion.

3. The system of claim 1 , further comprising means for providing feedback of the extracted health information to the patient via a hologram or a screen display.

Images