Program, information processing method, and information processing apparatus

Abstract

We provide programs and other tools that allow doctors and medical institutions to easily find the patient information they need. [Solution] The program acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and examination information obtained during the patient's examination. Using the acquired monitoring data, electronic medical record information, and examination information, the program causes the computer to execute a process that outputs a report to assist in the examination using a language model.

Description

Technical Field

[0001] The present invention relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] Patent Document 1 discloses a system having a function of acquiring an image of a hospital bed where a nurse call has been issued from a surveillance camera, a function of acquiring a prescription or the like of a patient who has issued a nurse call from an electronic medical record, and a function of associating the acquired image with the prescription or the like and displaying it on a monitor.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Not only the examination information of examinations conducted in hospitals and the like, but also vital data such as blood pressure and heart rate measured at living bases such as home may have an important influence on the diagnosis of patients. However, with the spread of wearable devices and the like, it has become possible to acquire many types of data such as vital data throughout the day, but it has been difficult for doctors to check such a large amount of data and use it for diagnosis.

[0005] In one aspect of the present disclosure, there is provided a program or the like that can easily find patient information required by doctors and medical institutions.

Means for Solving the Problems

[0006] (1) one aspect of this disclosure is a program that acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and examination information obtained during the patient's examination, and uses the acquired monitoring data, the electronic medical record information, and the examination information to cause a computer to execute a process that outputs a report to assist in the examination using a language model.

[0007] (2) The program described in (1) above may output the report using the language model, with respect to vital data measured at the living base and monitoring data including multiple meal images or multiple insulin injection site images taken at the living base.

[0008] (3) The program described in (1) or (2) above may use the electronic medical record information, including the patient's medical history and biochemical test data, and the consultation information, including the test results data and the contents of the medical questionnaire during the consultation, to output the report for assisting the consultation using the language model.

[0009] (4) The program described in any of (1) to (3) above may output a graph showing the trend of vital data measured at the living base, and the report including comments on the graph.

[0010] (5) The program described in (4) above may include mealtime information, which is the time of meals at the living base, as monitoring data, and the comments on the graph may include the mealtime information.

[0011] (6) The program described in any of (1) to (5) above may output the report which includes a graph showing the trends in vital data measured at the living base, and comments regarding singularities or missing data in the graph.

[0012] (7) The program described in any of (1) to (6) above may output a graph showing the trend of the patient's biochemical test data included in the electronic medical record information, and a report including comments relating to the graph.

[0013] (8) The program described in any of (1) to (7) above may output the report which includes a determination of whether or not there is induration at the insulin injection site, and the monitoring data includes multiple images of insulin injection sites taken at the living site.

[0014] (9) The program described in any of (1) to (8) above may acquire video data or audio data of the patient during the examination and output the report that includes information about the patient's psychology.

[0015] (10) The program described in any of (1) to (9) above may provide the language model with prompts including an instruction to create the report based on the acquired monitoring data, the electronic medical record information and the consultation information.

[0016] (11) The program described in any of (1) to (10) above may provide the language model with prompts including commands to create a graph based on the acquired monitoring data, and comments relating to singularities in the graph and missing data in the graph.

[0017] One aspect of this disclosure is an information processing method that acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and examination information obtained during the patient's examination, and uses the acquired monitoring data, the electronic medical record information, and the examination information to output a report to assist in the examination using a language model.

[0018] An information processing apparatus according to one aspect of the present disclosure includes a control unit, and the control unit acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and examination information obtained during the examination of the patient, and uses the acquired monitoring data, the electronic medical record information, and the examination information to output a report for assisting examination by a language model.

Effect of the Invention

[0019] In one aspect of the present disclosure, it is possible to easily find the information of a patient required by a doctor or a medical institution.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram of a system according to one aspect of the present disclosure. [Figure 2] It is a block diagram showing a configuration example of a server device. [Figure 3] It is a diagram showing the configuration of a monitoring patient table. [Figure 4] It is a diagram showing the configuration of a monitoring data table. [Figure 5] It is a block diagram showing a configuration example of an examination terminal device. [Figure 6] It is a block diagram showing a configuration example of an electronic medical record server device. [Figure 7] It is a diagram showing the configuration of a medical record patient table. [Figure 8] It is a diagram showing the configuration of a medical record data table. [Figure 9] It is a diagram showing an example of an interview form. [Figure 10] [[ID= forty-two]]It is a block diagram showing a configuration example of a generation server device. [Figure 11] It is a block diagram showing a configuration example of a monitoring device. [Figure 12] It is a block diagram showing a configuration example of a place-of-residence recording device. [Figure 13] It is a block diagram showing a configuration example of a place-of-residence terminal. [Figure 14] This block diagram shows an example configuration for a medical institution terminal. [Figure 15] This is a block diagram showing an example configuration of a medical institution's recording device. [Figure 16] This is a flowchart illustrating the process for generating reports from server devices. [Figure 17] This diagram shows an example of a report displayed on a medical examination terminal device. [Figure 18] This diagram shows an example of a report displayed on a medical examination terminal device. [Figure 19] This is a flowchart illustrating the process for creating a medical consultation report. [Modes for carrying out the invention]

[0021] The following describes in detail, with reference to drawings, a system that is an example of a program, information processing method, and information processing apparatus relating to one aspect of this disclosure. In the description, similar elements are denoted by the same reference numerals, and redundant explanations are omitted as appropriate. In the following, the term "image" includes both moving images and still images unless otherwise specified.

[0022] Figure 1 is an overview diagram of a system relating to one aspect of this disclosure. The system relating to this form of disclosure includes an information processing device 1 and a medical examination terminal device 2, which can send and receive information via a network 3 such as the Internet. In this form of disclosure, the information processing device 1 will be described as a server device 1 to avoid complexity in the explanation. The system may also further include an electronic medical record server device 4, a generation server device 5, monitoring devices 61a, 61b, and 61c (hereinafter simply referred to as "61"), a living base recording device 65, a living base terminal 68, a medical institution terminal 71, and a medical institution recording device 75, each of which can be connected to the network 3.

[0023] The monitoring device 61, the living base recording device 65, and the living base terminal 68 can be devices used at the patient 91's living base 100. Here, the living base 100 can include the patient 91's home, as well as a hospital room or nursing facility. The living base 100 may also include the entire area where the patient 91 moves in their daily life. The monitoring device 61 may include, for example, a blood glucose meter 61a, a wearable device 61b such as a smartwatch with vital data measurement capabilities, or a blood pressure monitor 61c. The monitoring device 61 may also be a device that measures vital data such as body temperature, heart rate, respiratory rate, blood pressure, oxygen saturation, and blood glucose levels, as well as step count and other data related to the patient 91's body at the living base 100.

[0024] The medical examination terminal device 2, the medical institution terminal 71, and the medical institution recording device 75 may be devices installed in medical institutions 200 such as hospitals, clinics, and testing facilities. The installation location is not limited to one; they may be installed in multiple medical institutions 200, such as a medical institution and a testing facility. The medical examination terminal device 2 and the medical institution terminal 71 may be used by physicians 92 and other medical professionals. Furthermore, the network 3 may be part or all of the on-premises network of the medical institution 200. In other words, part or all of the server device 1, the electronic medical record server device 4, and the generation server device 5 may be connected to the on-premises network of the medical institution 200.

[0025] The medical institution terminal 71 can receive input from medical personnel, such as biochemical test data and other test results 791 of patient 91, and the contents of the medical questionnaire 792, and transmit it to the server device 1 or the electronic medical record server device 4. Here, the examination terminal device 2 can display information about patient 91 being examined by doctor 92, and in this case, it may display patient 91 information transmitted from the server device 1. The examination terminal device 2 may have the same functions as the medical institution terminal 71. Also, the examination terminal device 2 may have the same functions as the electronic medical record server device 4.

[0026] Figure 2 is a block diagram showing an example configuration of server device 1. Server device 1 is an information processing device that processes, stores, and transmits various types of information. Server device 1 may be configured as, for example, a server device, a personal computer, a tablet terminal, or a smartphone. Server device 1 may be composed of multiple information processing devices, or it may be configured as one of multiple virtual devices (virtual machines) configured within a single information processing device.

[0027] The server device 1 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The server device 1 has a control unit 11, a communication unit 12, a reading unit 13, and a storage unit 14. The control unit 11, communication unit 12, reading unit 13, and storage unit 14 are connected to each other via a bus 19 or the like. The server device 1 may also have other configurations, such as not having a reading unit 13.

[0028] The control unit 11 can be configured to have one or more processing units such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), and quantum processor. The control unit 11 reads and executes the program (or program product) 16 stored in the storage unit 14.

[0029] The communication unit 12 is a communication module for performing communication-related processing and can send and receive information with the medical examination terminal device 2, etc., via the network 3. The reading unit 13 can read portable storage media 15 such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, and USB memory. The control unit 11 may read the program 16 from the portable storage media 15 via the reading unit 13 and save it to the storage unit 14. Alternatively, the control unit 11 may download the program 16 from another computer via the network 3, etc., and save it to the storage unit 14.

[0030] The storage unit 14 includes volatile storage units such as RAM (Random Access Memory) and non-volatile storage units such as ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory. As shown in Figure 2, the storage unit 14 stores, for example, a program 16. The program 16 is a program executed by the control unit 11. In this embodiment, the program 16 may also be referred to as a program product 16. The storage unit 14 also stores, for example, a monitoring DB (Database) 17. The program 16 executed by the control unit 11 can add, delete, and modify records in the monitoring DB 17. In this embodiment, the monitoring DB 17 is a database, but it may also be stored in spreadsheet, text file, or other file format.

[0031] Figure 3 shows the structure of the monitoring patient table 171. The monitoring patient table 171 is included in the monitoring DB 17 and links the monitoring patient ID with the medical record patient ID. The monitoring patient table 171 has columns for "Monitoring Patient ID," "Medical Record Patient ID," and "Medical Institution Name." The "Monitoring Patient ID" column stores unique identification information for the medical record patient ID in the monitoring patient table 171. For example, each time a new medical record patient ID is registered, it may include a string of numbers obtained by adding 1 to the numerical part of the medical record patient ID of the last added patient. The medical record patient ID will be described in detail in the explanation of the medical record patient table 461 in Figure 7, which will be described later.

[0032] The "Medical Institution Name" column stores the name of the medical institution that manages the patient's medical record ID. This allows server device 1 to identify patients even if there are duplicate patient IDs in different medical institutions when monitoring patients from different institutions, thus enabling management of patients from different medical institutions. Note that the monitoring patient table 171 in Figure 3 is an example, and it may have other columns, or may not have these columns at all.

[0033] Figure 4 shows the structure of the monitoring data table 172. The monitoring data table 172 is included in the monitoring DB 17 and mainly links data obtained at the living base 100 with the medical record patient ID. The monitoring data table 172 has columns for "Monitoring ID", "Monitoring Patient ID", "Monitoring Item", "Date and Time", and "Data".

[0034] The "Monitoring ID" field stores unique identifiers for each record in the monitoring data table 172. For example, each time a new record is registered, it may contain a string of numbers obtained by adding 1 to the numerical part of the Monitoring ID of the last added record.

[0035] The "Monitoring Patient ID" field stores the Monitoring Patient ID from Monitoring Patient Table 171 to indicate which patient the added record is associated with. The "Date and Time" field stores the date and time the record was created. The "Monitoring Item" field stores the type of data registered in the record. The "Data" field stores the data corresponding to the "Monitoring Item". Here, the data field can be saved as a string.

[0036] The monitoring data table 172 in Figure 4 is a table extracted using the monitoring patient ID "MP000106". For example, in the record with monitoring ID "M0010542", the "Monitoring Item" column is set to "Blood Glucose", the "Date and Time" column is set to "2024 / 2 / 24 00:05", and "150 mg / dL" is stored in the "Data" column. The "Data" column may also be stored as a string.

[0037] Furthermore, in the record with monitoring ID "M0011589", the "Monitoring Item" field is set to "Recorded Data", the "Date and Time" field to "2024 / 06 / 14 00:04", and the file path ""C:\RecordingData\MP000106\20240613DAT.zip"" is saved in the "Data" field. This file path is an example, and the file may be saved on an information processing device on network 3. In the monitoring data table 172, by saving the file path in the "Data" field, even data in file format can be saved in association with a file. The monitoring data table 172 in Figure 4 is an example, and it may have other fields, or may not have these fields at all.

[0038] The monitoring device 61 located at the living base 100 transmits the monitoring data to the server device 1 each time it acquires monitoring data for patient 91 through measurements or other means. Upon receiving the monitoring data, the server device 1 adds the new record of the received monitoring data to the monitoring data table 172 and saves it.

[0039] Furthermore, the recorded data for monitoring items may be transmitted in batches from the living base recording device 65, for example, once a day. Here, the recorded data can consist of one or more image data or one or more audio data. In the case of multiple data, they may be combined into a single file using an archive file or the like. The image data may be a still image or a moving image.

[0040] Figure 5 is a block diagram showing an example configuration of the medical examination terminal device 2. The medical examination terminal device 2 can be, for example, an information processing device such as a personal computer, smartphone, mobile phone, wearable device, and tablet. Here, the medical examination terminal device 2 can be an information processing device mainly composed of electronic circuits using semiconductor circuit elements. The medical examination terminal device 2 may have a control unit 21, a communication unit 22, a reading unit 23, a storage unit 24, a display unit 26, and an input unit 27. The components of the control unit 21, communication unit 22, reading unit 23, storage unit 24, display unit 26, and input unit 27 are connected to each other so as to be able to communicate with each other by a bus 29 or the like.

[0041] The control unit 21 can be configured to include one or more processing units such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), and a quantum processor. The control unit 21 reads and executes the program stored in the storage unit 24.

[0042] The communication unit 22 is a communication module for performing communication-related processing. It can send and receive information with the server device 1, etc., via the network 3 using wired or wireless communication, and can also send and receive information directly with other communication devices using Bluetooth® or the like. The reading unit 23 can read portable storage media 25 such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, and USB memory. The control unit 21 can read programs and / or data from the portable storage media 25 via the reading unit 23 and save them to the storage unit 24. The control unit 21 can also download programs from other computers via the network 3 or the like and save them to the storage unit 24.

[0043] The storage unit 24 includes volatile storage devices such as RAM (Random Access Memory), and non-volatile storage devices such as ROM (Read Only Memory), HDD (Hard Disk Drive), and flash memory. The display unit 26 can be a liquid crystal display or an organic EL (Electro Luminescence) display, etc. The input unit 27 can be an input device such as a keyboard, mouse, touch panel, camera, and microphone.

[0044] The physician 92 and other medical professionals can input information obtained from the patient 91 during the examination, such as the examination details and the prescribed medication, via the input unit 27. The examination terminal device 2 may also function as a medical institution terminal 71, in which case the physician 92 and other medical professionals may input biochemical test data or other test results or information from the medical questionnaire for the patient 91 before, during, or at other times.

[0045] Furthermore, the display unit 26 can appropriately acquire and display information from the patient chart table 461 (Figure 7) and the medical chart data table 462 (Figure 8) of the electronic medical record DB 46 stored by the electronic medical record server device 4, or information acquired during the consultation by the medical institution terminal 71, during the consultation. In addition, the display unit 26 can display information entered in the consultation terminal device 2 during the consultation.

[0046] Furthermore, the medical examination terminal device 2 may be used as an online medical examination terminal that communicates with the patient's home base terminal 68 (described later) and allows for face-to-face online consultations with the patient 91. In this case, the input unit 27 includes a camera and a microphone, and the physician 92 using the medical examination terminal device 2 can conduct the examination while communicating with the patient 91 via image and sound through the home base terminal 68 used by the patient 91.

[0047] Figure 6 is a block diagram showing an example configuration of the electronic medical record server device 4. The electronic medical record server device 4 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The electronic medical record server device 4 has a control unit 41, a communication unit 42, a reading unit 43, and a storage unit 44. The control unit 41, communication unit 42, reading unit 43, and storage unit 44 are connected to each other so as to be able to communicate with one another by a bus 49 or the like. These components have the same configuration as the server device 1 in Figure 2, except for the contents stored in the storage unit 44, so redundant explanations are omitted. Note that the electronic medical record server device 4 may have other configurations, such as not having a reading unit 43, as in the case of server device 1.

[0048] The memory unit 44 stores the electronic medical record database 46. The electronic medical record database 46 stores information recorded in the medical records of, for example, 200 patients in medical institutions. The electronic medical record can store all information about the patient 91. For example, the electronic medical record can store monitoring data, biochemical test data, and other test results. In addition, for example, it can store information recorded by the patient 91 on the medical questionnaire 792, information obtained by the doctor 92 from the patient 91 during the examination, the doctor 92's findings, medication prescription information, and other information that the doctor 92 deems necessary.

[0049] The information stored in these electronic medical record DB46 may be information entered on the examination terminal device 2 and the medical institution terminal 71 and received by the electronic medical record server device 4 via the network 3. Alternatively, information transmitted from the medical institution record device 75 may be stored in the electronic medical record DB46. In this configuration, the electronic medical record DB46 is a database, but it may also be stored in spreadsheet, text file, or other file formats.

[0050] Figure 7 shows the structure of the medical record patient table 461. The medical record patient table 461 is included in the electronic medical record DB 46 and mainly links patients with medical record patient IDs. The medical record patient table 461 has the following columns: "Medical Record Patient ID," "Name," "Medical History," "Address," and "Contact Information." The "Medical Record Patient ID" column stores patient-specific identification information in the medical record patient table 461. For example, each time a new patient is registered, it may contain a string of numbers obtained by adding 1 to the numerical part of the medical record patient ID of the last added patient. The "Name" column stores the patient's name. The "Medical History" column stores the patient's past medical history and health status, such as diseases previously diagnosed, surgical history, and allergic reactions. The "Address" and "Contact Information" columns store the patient's address and contact information such as telephone number. Note that the medical record patient table 461 in Figure 7 is an example, and it may have other columns, or may not have these columns at all.

[0051] Figure 8 shows the structure of the medical record data table 462. The medical record data table 462 is included in the electronic medical record DB 46 and mainly links data obtained from medical institutions with medical record patient IDs. The medical record data table 462 has the following columns: "Medical Record Data ID", "Medical Record Patient ID", "Date and Time", "Classification", "Item Name", and "Data". The "Medical Record Data ID" column stores record-specific identification information in the medical record data table 462. For example, each time a new record is registered, it may include a string of numbers obtained by adding 1 to the numerical part of the medical record data ID of the last added record.

[0052] The "Medical Record Patient ID" field stores the Medical Record Patient ID from Medical Record Patient Table 461 to indicate which patient the added record is associated with. The "Date and Time" field stores the date and time the record was registered. The "Classification" field stores the type of data registered in the record. The "Item Name" field stores a more detailed type of data among the types of data registered in the record. The "Data" field stores the data corresponding to the "Item Name". Here, the data field can be stored as a string. In this example, Medical Record Data Table 462 in Figure 8 is a table extracted using Medical Record Patient ID "K000014".

[0053] For example, in a record with medical record data ID "MD000021", the "Date and Time" field will be saved as "2024 / 4 / 15 15:35", the "Category" field will be set to "Medical Questionnaire", and the combination of questions and answers from the questionnaire will be saved in the "Data" field, for example, as "key" and "value" in JSON format, such as "{"Are you worried about diabetes treatment?":"Worried about developing complications (neuropathy, retinopathy, nephropathy)"}". This allows the contents of the medical questionnaire to be saved. In a record with medical record data ID "MD000028", the "Category" field will be set to "Biochemical Test", the "Item Name" will be set to "HbA1c (NGSP)", and "Data" will be saved as "6.10%". In this way, the results of blood tests and other tests can also be saved.

[0054] Furthermore, even if the results of examinations such as X-rays, MRI (Magnetic Resonance Imaging), or X-ray CT (Computed Tomography) are in the form of electronic files, the file path where the electronic file is saved can be saved in text format in the "Data" field (for example, Figure 8, Medical Record Data ID: MD000035). In this way, electronic files can also be saved in association with records in the Medical Record Data Table 462. Note that the Medical Record Data Table 462 in Figure 8 is an example, and it may have other fields, or may not have these fields at all.

[0055] The data stored in the electronic medical record DB46 is data transmitted from the medical examination terminal device 2 or medical institution terminal 71 installed in the medical institution 200. The data stored in the electronic medical record DB46 may be the medical information obtained during the examination. Here, "medical information obtained during the examination" includes information such as the results of tests requested when receiving an examination, entries in the medical questionnaire 792, information obtained by the doctor 92 from the patient 91 during the examination, the doctor 92's findings, and information on prescriptions for medication and other content that the doctor 92 deems necessary. The doctor 92's findings or diagnostic information can be stored as a string in the data field of the medical record data table 462. If the doctor 92 or others change the medical history during the examination, the medical history field in the medical record patient table 461 is updated.

[0056] Figure 9 shows an example of Questionnaire 792. As shown in this figure, this example of Questionnaire 792 asks about anxieties about the illness currently being treated and requests for medical institutions. It also asks about things the patient is careful about in their daily life and any questions they may have, regardless of the illness they are currently being treated for.

[0057] On the other hand, the responses represent the perspective of patient 91, and it is stated that patient 91 is currently concerned about: "anxiety about developing complications (neuropathy, retinopathy, nephropathy)", "lifetime treatment costs", "a reasonable treatment plan", "sometimes forgetting to measure blood glucose levels at home", "not being able to establish an exercise habit", "a balanced diet of carbohydrates, proteins, and fats", "regular breakfast, lunch, and dinner", "avoiding snacks", "a 30-minute walk twice a week", and "prevention of complications". By providing this information from questionnaire 792 to language model 56 (Figure 10), it is possible to output a report that incorporates what patient 91 is currently concerned about.

[0058] The information from the questionnaire 792 described above may be stored in the medical record data table 462 as a combination of "key" and "value" in JSON format, for example, the combination of the questions in questionnaire 792 and the answers of patient 91 (for example, Figure 8, medical record data ID: MD000021).

[0059] Figure 10 is a block diagram showing an example configuration of the generation server device 5. The generation server device 5 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The generation server device 5 has a control unit 51, a communication unit 52, a reading unit 53, and a storage unit 54. The components of the control unit 51, communication unit 52, reading unit 53, and storage unit 54 are connected to each other so as to be able to communicate with one another by a bus 59 or the like. The components of the control unit 51, communication unit 52, and reading unit 53 can be configured the same as the information processing device (server device) 1 in Figure 2, so redundant explanations are omitted.

[0060] The memory unit 54 may store the language model 56. The language model 56 consists of a program and learning data, and the control unit 51 executes the program by referring to input data received from an external source. The language model 56 may be stored in an external storage device or the like outside the generation server device 5. The generation server device 5 may have other configurations, such as not having a read unit 53, similar to the case of server device 1.

[0061] The generation server device 5 is an information processing device that handles language models 56, such as large language models (LLMs) like GPT (Generative Pre-trained Transformer, registered trademark), BERT (Bidirectional Encoder Representations from Transformer), and Gemini (trademark). The language models 56 may be stored in the storage unit 54 within the generation server device 5, or they may be stored in storage or the like that which is communicated with the generation server device 5.

[0062] The language model 56 can be a multimodal language model, and the generation server device 5 inputs input data such as images, audio, or text to the language model 56 and generates a response sentence. The language model 56 is a pre-trained machine learning model and can be a large-scale language model such as GPT (Generative Pre-trained Transformer) or BERT (Bidirectional Encoder Representations from Transformer), but other language models may also be used.

[0063] Server device 1 may be connected to generation server device 5 via network 3, or it may be connected to generation server device 5 directly or via a local network without using network 3. Since generation server device 5 is a device that handles language model 56, from the perspective of server device 1, when server device 1 sends prompts etc. to generation server device 5, it may be expressed as server device 1 sending (inputting) to language model 56. In this embodiment, when server device 1 obtains processing results using the language model 56 of generation server device 5, it may be expressed as server device 1 generating processing results using language model 56.

[0064] Figure 11 is a block diagram showing an example configuration of the monitoring device 61. The monitoring device 61 can be, for example, a blood glucose meter 61a, a wearable device 61b, and a blood pressure monitor 61c. The monitoring device 61 is not limited to these, and can be a device that measures vital data such as body temperature, heart rate, respiratory rate, blood pressure, oxygen saturation, and blood glucose levels, as well as step count and other data related to the patient's body at the living base 100. Here, the monitoring device 61 can be an information processing device mainly composed of electronic circuits using semiconductor circuit elements. The monitoring device 61 may have a control unit 611, a communication unit 612, a reading unit 613, a storage unit 614, a display unit 616, an input unit 617, and a measurement unit 618. The components of the control unit 611, communication unit 612, reading unit 613, storage unit 614, display unit 616, input unit 617, and measurement unit 618 are connected to each other so as to be able to communicate with each other by a bus 619 or the like. The configurations of the control unit 611, communication unit 612, reading unit 613, storage unit 614, display unit 616, and input unit 617 can be the same as those of the medical examination terminal device 2 in Figure 5, so redundant explanations are omitted.

[0065] Here, the communication unit 612 can connect to the network 3 directly or indirectly via wired or wireless communication and send and receive information with the electronic medical record server device 4 and / or server device 1, etc. The communication unit 612 may also be connected to the network 3 via communication with a living base terminal 68 such as a smartphone or personal computer.

[0066] The measurement unit 618 is a measuring instrument provided in the monitoring device 61 and can perform measurements corresponding to each measurement function. In the case of a blood glucose meter 61a, the measurement unit 618 measures blood glucose levels using the patient's 91 blood. The wearable device 61b may also have a measurement unit 618 that measures one or more of the following: heart rate, blood oxygen saturation, and respiratory rate. In the case of a blood pressure monitor 61c, it may have a measurement unit 618 that measures blood pressure or heart rate, etc. The monitoring device 61 is a device that measures data of the patient 91 at the living base 100 and may have a measurement unit 618 corresponding to the measurement items.

[0067] Figure 12 is a block diagram showing an example configuration of the living base recording device 65. The living base recording device 65 can be, for example, a webcam, a surveillance camera, or an audio recorder. The living base recording device 65 is a device that records or videos the condition of patient 91 at the living base 100, with or without the instruction of patient 91. Here, the living base recording device 65 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The living base recording device 65 may have a control unit 651, a communication unit 652, a reading unit 653, a storage unit 654, a display unit 656, an input unit 657, a camera 658, and a microphone 660. Note that the display unit 656 and the input unit 657 may be omitted if operation is performed only via the communication unit 652 by wire or wireless connection.

[0068] The control unit 651, communication unit 652, reading unit 653, storage unit 654, display unit 656, input unit 657, camera 658, and microphone 660 are connected to each other via a bus 659 or the like to enable communication. The configurations of the control unit 651, communication unit 652, reading unit 653, storage unit 654, display unit 656, and input unit 657 can be the same as those of the medical examination terminal device 2 in Figure 5, so redundant explanations are omitted.

[0069] Here, the communication unit 652 can connect to the network 3 directly or indirectly by wired or wireless communication and send and receive information with the server device 1, etc. The communication unit 652 may also be connected to the network 3 via communication with a living base terminal 68 such as a smartphone or personal computer.

[0070] The camera 658 and microphone 660 may be both, or either one or the other. The camera 658 photographs the patient 91. The microphone 660 records the patient 91's voice. Here, the living base recording device 65 can record using the camera 658 and / or microphone 660 along with the time. The recorded images or audio can be temporarily stored in the storage unit 654. The temporarily stored images or audio can be transmitted to the server device 1, etc., via the network 3.

[0071] The operation of the living base recording device 65 may be controlled by operation via the input unit 657 by the patient 91, operation via voice recognition via the microphone 660, or processing via a communication network by an information processing device (not shown). The living base recording device 65 may record, for example, the details of meals or insulin injections, or the time of meals or insulin injections, while the patient 91 is spending time at the living base 100, using images or audio.

[0072] Furthermore, if the recording is of a meal or insulin injection, it may be transmitted to the server device 1 via the network 3, and the time of the meal or insulin injection may be determined by image analysis or the like. In this case, the captured image may be a moving image or a still image. The living base recording device 65 may operate and stop based on operations by, for example, the patient 91 on the living base terminal 68.

[0073] Figure 13 is a block diagram showing an example configuration of the living base terminal 68. The living base terminal 68 can be, for example, an information processing device such as a personal computer, smartphone, mobile phone, wearable device, or tablet. The living base terminal 68 can receive monitoring data measured by the monitoring device 61 or data recorded by the living base recording device 65 at the patient's living base 100 via wired or wireless communication, through a storage medium, or via input via the input unit 687 of the patient 91, and transmit it to the server device 1, etc. In particular, if the monitoring device 61 or the living base recording device 65 is not communicating directly with the network 3, it can connect to the network 3 via the living base terminal 68 and transmit data such as monitoring data to the server device 1, etc.

[0074] Here, the living base terminal 68 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The living base terminal 68 may have a control unit 681, a communication unit 682, a reading unit 683, a storage unit 684, a display unit 686, and an input unit 687. The control unit 681, communication unit 682, reading unit 683, storage unit 684, display unit 686, and input unit 687 are connected to each other so as to be able to communicate with each other by a bus 689 or the like. The configurations of the control unit 681, communication unit 682, reading unit 683, storage unit 684, display unit 686, and input unit 687 can be the same as those of the medical examination terminal device 2 in Figure 5, so redundant explanations are omitted.

[0075] Here, the communication unit 682 can connect to the network 3 directly or indirectly via wired or wireless communication and send and receive information with the server device 1, etc. The living base terminal 68 may also control, for example, the start / stop of recording of the living base recording device 65 via the communication network within the living base 100. The living base terminal 68 can be operated as the living base recording device 65. There is not limited to one living base terminal 68, but there can be multiple.

[0076] Furthermore, if the living base terminal 68 has a function to record images or audio, such as a camera or microphone, it may record images or audio together with or in place of the living base recording device 65 to create recorded data and transmit it to the server device 1. In this case, as described in the explanation of the living base recording device 65, the recorded data may be capable of deriving the time of a meal based on multiple meal images. The recorded data may also include images of the insulin injection site.

[0077] Furthermore, the living base terminal 68 may be used as an online medical consultation terminal that communicates with the medical examination terminal device 2 used by the physician 92, allowing for online face-to-face consultations with the physician 92. In this case, the living base terminal 68 includes a camera and a microphone, and the patient 91 using the living base terminal 68 can receive online medical consultations while communicating with the physician 92 via the medical examination terminal device 2 using images and audio. In addition, during online medical consultations, the living base terminal 68 may be used to input information to be written on the medical questionnaire 792 and transmit it to the medical examination terminal device 2 or the medical institution terminal 71 described later.

[0078] Figure 14 is a block diagram showing an example configuration of a medical institution terminal 71. The medical institution terminal 71 can be, for example, an information processing device such as a personal computer, smartphone, mobile phone, wearable device, and tablet. The medical institution terminal 71 can acquire patient 91 test results 791, medical questionnaires 792, and other information and transmit them to the electronic medical record server device 4 and / or server device 1.

[0079] Here, the medical institution terminal 71 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The medical institution terminal 71 may have a control unit 711, a communication unit 712, a reading unit 713, a storage unit 714, a display unit 716, and an input unit 717. The components of the control unit 711, communication unit 712, reading unit 713, storage unit 714, display unit 716, and input unit 717 are connected to each other so as to be able to communicate with one another by a bus 719 or the like. The components of the control unit 711, communication unit 712, reading unit 713, storage unit 714, display unit 716, and input unit 717 can be the same as the configuration of the medical examination terminal device 2 in Figure 5, so redundant explanations are omitted.

[0080] Here, the communication unit 712 can connect to the network 3 directly or indirectly by wired or wireless communication and send and receive information with the electronic medical record server device 4 and / or server device 1, etc.

[0081] Furthermore, the communication unit 712 can communicate with the testing equipment 793 to receive test results (measurement results), etc. 791. The input unit 717 can receive input of test results, etc. 791 measured by the testing equipment 793, etc., as well as a medical questionnaire 792 and other information related to the patient 91, through operation by medical professionals such as a doctor 92 or a medical technologist. The patient 91 may directly input the information on the medical questionnaire 792 into the medical institution terminal 71, or the patient 91 may input the information to be written on the medical questionnaire 792 into themselves or another information and communication device and transmit it from the information and communication device to the medical institution terminal 71 for reception.

[0082] The received test results, etc. 791 may include, for example, blood test results including cholesterol, HbA1c, and blood glucose levels and other biochemical tests, X-ray examinations, X-ray CT (Computed Tomography) examinations, MRI (Magnetic Resonance Imaging) examinations, etc., test (measurement) results such as body temperature, blood pressure, heart rate, blood oxygen saturation, height and weight, and records such as surgical images and images of the affected area. The medical institution terminal 71 may also acquire measurement results acquired by the monitoring device 61 via the input unit 717, communication unit 712, or reading unit 713. The patient information 91, such as the acquired test results, etc. 791, can be transmitted to the electronic medical record server device 4 and / or server device 1 via the network 3.

[0083] The medical institution terminal 71 is not limited to one, but can be multiple. Furthermore, the medical institution terminal 71 and the examination terminal device 2 may be the same information and communication device. Also, the medical institution terminal 71 may be installed inside the testing device 793. The medical institution terminal 71 is not limited to being located in a hospital, but may be installed in other locations such as testing facilities where tests are performed.

[0084] Figure 15 is a block diagram showing an example configuration of a medical institution recording device 75. The medical institution recording device 75 can be, for example, a webcam, a surveillance camera, or an audio recorder. The medical institution recording device 75 is a device that records or videos the condition of a patient 91 during a medical examination at a medical institution 200. Here, the medical institution recording device 75 can be an information processing device composed mainly of electronic circuits using semiconductor circuit elements. The medical institution recording device 75 may have a control unit 751, a communication unit 752, a reading unit 753, a storage unit 754, a display unit 756, an input unit 757, a camera 758, and a microphone 760. Note that the display unit 756 and the input unit 757 may be omitted if operation is performed only via the communication unit 752 by wire or wireless connection.

[0085] The control unit 751, communication unit 752, reading unit 753, storage unit 754, display unit 756, input unit 757, camera 758, and microphone 760 are connected to each other via a bus 759 or the like so that they can communicate with one another. The configurations of the control unit 751, communication unit 752, reading unit 753, storage unit 754, display unit 756, input unit 757, camera 758, and microphone 760 can be the same as those of the living base recording device 65 in Figure 12, so redundant explanations are omitted.

[0086] Here, the communication unit 752 can connect to the network 3 directly or indirectly via wired or wireless communication and send and receive information with the server device 1, etc. The communication unit 752 may also be connected to the network 3 via communication with information processing devices such as smartphones or personal computers. Note that communication with the network 3 may use either wired or wireless communication.

[0087] The camera 758 and microphone 760 may be both, or either one or the other. The camera 758 captures at least the appearance of the patient 91 during the examination. The microphone 760 records at least the voice of the patient 91 during the examination. The content recorded here may mainly consist of the conversation between the doctor 92 and the patient 91, and the recording can be in the form of images or audio. The recorded images or audio can be temporarily stored in the storage unit 754. During or after the examination, the medical institution recording device 75 transmits the examination recording data to the server device 1, with or without a request from the server device 1.

[0088] Furthermore, if the input unit 27 of the medical examination terminal device 2 has a camera or microphone, the medical examination terminal device 2 may also operate as a medical institution recording device 75 that records images or audio of the patient 91. In addition, during online consultations, the medical examination terminal device 2 can save the images and audio of the patient 91 received during the consultation as consultation recording data. In this case, the medical examination terminal device 2 may transmit this data to the server device 1 as a medical institution recording device 75.

[0089] Figure 16 is a flowchart illustrating the report generation process of server device 1. As shown in the flowchart of Figure 16, first, server device 1 determines whether it has received a report generation start request from the examination terminal device 2 or the medical institution terminal 71 (step S81). Here, the report generation start request is received along with the medical record patient ID that identifies patient 91. The report generation start request is also received along with the "examination information" and "items to be emphasized" before the examination. The report generation start request may also include a flag indicating whether analysis of the examination record data is necessary.

[0090] In step S81, if the server device 1 has not received a report creation start request (step S81: NO), the process in step S81 is repeated. If the server device 1 has received a report creation start request from the medical examination terminal device 2 or the medical institution terminal 71 (step S81: YES), the medical examination information attached to the report creation start request is acquired (step S82).

[0091] Here, "information obtained before the consultation" refers to information obtained by the medical institution, etc. 200 before the consultation of the patient 91, and may include pre-consultation test results data, entries in the medical questionnaire 792, etc. Test results data may include, for example, heart rate, body temperature, blood pressure, and other test results performed before the consultation.

[0092] Furthermore, server device 1 may receive pre-examination consultation information at a different time than the report creation start request. In this case, the transmission of the consultation information can be performed by the consultation terminal device 2 or the medical institution terminal 71. Server device 1 may also receive the test result data and the information from the medical questionnaire 792 separately from the consultation information. The consultation information is stored in the electronic medical record database of electronic medical record server device 4 after the consultation.

[0093] Furthermore, if the server device 1 receives "items to be emphasized" along with the report creation start request, the server device 1 retrieves the items to be emphasized (step S83). The "items to be emphasized" are monitoring data such as vital data that the doctor wants to check during the examination, such as "changes in blood glucose levels at the patient's place of residence." This allows the server device 1 to display a report containing the items to be emphasized on the examination terminal device 2. Here, the items to be emphasized may be received at a different time than the report creation start request. In this case, the items to be emphasized can be sent by the examination terminal device 2 or the medical institution terminal 71.

[0094] Next, server device 1 obtains electronic medical record information corresponding to the medical record patient ID from electronic medical record server device 4 (step S84). The electronic medical record information is information extracted using the medical record patient ID from the medical record patient table 461 of the electronic medical record DB 46 and the medical record data table 462, which contains the results of various tests performed at medical institutions, etc. 200 in the past.

[0095] The information obtained from the medical record data table 462 here may be narrowed down to the past year or the past six months, depending on the type of data to be obtained. Specifically, the electronic medical record information can be the patient 91's medical history and test data including biochemical test data. Furthermore, when the extraction of electronic medical record information is requested, the electronic medical record server device 4 may perform authentication to determine whether the server device 1 has the authority to obtain the patient 91's information.

[0096] Furthermore, if the electronic medical record server device 4 has already received the medical questionnaire or test results for the day of the consultation from the consultation terminal device 2 or the medical institution terminal 71, it may include them in the electronic medical record information in place of or together with the "consultation information" before the consultation.

[0097] Next, Server Device 1 acquires monitoring data corresponding to the medical record patient ID (step S85). Specifically, Server Device 1 refers to the monitoring patient table 171 of the monitoring DB 17 and acquires the monitoring patient ID based on the medical record patient ID. Subsequently, Server Device 1 extracts and acquires monitoring data from the monitoring data table 172 using the monitoring patient ID.

[0098] Here, the server device 1 may extract only the necessary monitoring items and acquire monitoring data. Alternatively, the server device 1 may extract only the necessary period according to the monitoring items and acquire monitoring data. The monitoring data can be, for example, vital data such as body temperature, heart rate, respiratory rate, blood pressure, oxygen saturation, and blood glucose level, data related to the patient's body measured at the living base 100, such as steps taken, and recorded image and sound data.

[0099] Here, the recorded image or audio data acquired as part of the monitoring data may be sent to the generation server device 5 all at once, or it may be processed in advance in the server device 1. For example, if the recorded data includes an image of the insulin injection site, the server device 1 can use a learning model specifically for images that has been previously trained on images of the insulin injection site. The learning model can be a learning model that, upon input of an image of the insulin injection site, classifies it into four types, such as "normal," "induration," "redness or swelling," and "bleeding or internal bleeding," and outputs a judgment result.

[0100] Here, "normal" indicates that there are no abnormalities at the injection site. "Induration" refers to subcutaneous tissue damage caused by repeated injections in the same area, and injecting insulin into such an area may delay insulin absorption. Induration can occur due to inappropriate needle insertion angle, depth, thickness, or length. "Redness or swelling" may indicate a skin infection or allergic reaction. "Bleeding or bruising" may indicate damage to blood vessels or tissues during injection. Note that the learning model may also classify conditions at the insulin injection site other than those listed above.

[0101] When a judgment result is obtained using a dedicated learning model based on an image of the insulin injection site, only the judgment result obtained from the learning model may be provided to the language model 56 of the generation server device 5 along with the prompts described later, or both the image and the judgment result may be provided to the language model 56. The learning model may be stored in the server device 1, or a learning model stored in another information processing device may be used via the network 3. Alternatively, instead of using the image-specific learning model described here, the image of the insulin injection site may be input to the multimodal language model 56 along with the prompts described later.

[0102] Furthermore, for example, if the recorded data includes a video from the living base 100 along with time information, the server device 1 can input the recorded data into a learning model that has been previously trained on videos of meals, and obtain the time of the meal. Specifically, the server device 1 inputs the frame images for each time point included in the video into the learning model, and the learning model outputs a determination result indicating whether or not a meal is being eaten. The server device 1 saves the time when it obtains a determination result indicating that a meal is being eaten. Here, the server device 1 can save one time point per meal, and there may be multiple meal times. Note that the time information of the images in the recorded data can be manually linked to the image and saved by the patient 91 or others at the time of recording or after recording.

[0103] When results are obtained using a dedicated learning model for identifying images during meals, only the time of the meal, based on the results obtained from the learning model, can be provided to the language model 56 of the generation server device 5 along with the prompts described later. This learning model, like the learning model for images of insulin injection sites, may be stored in the server device 1, or a learning model stored in another information processing device may be used via the network 3. Alternatively, instead of using the image-specific learning model described here, a video from the living base 100 may be input to the multimodal language model 56 along with the prompts described later.

[0104] In addition, instead of video from living base 100, audio from living base can be used. In this case, the learning model can be a dedicated learning model for identifying audio during meals and outputting the time when mealtime is detected. Server device 1 can provide the outputted time to the language model 56 of generation server device 5 along with the prompt described later. In this case as well, the learning model may be stored in server device 1, or a learning model stored in another information processing device may be used via network 3. Alternatively, instead of using the dedicated learning model described here, audio from living base 100 may be input to the multimodal language model 56 along with the prompt described later.

[0105] Alternatively, patient 91 may obtain the time of capture by taking a still image of the meal. In this case, by using a learning model specifically for images, it is possible to create a learning model that outputs not only whether or not it is an image of a meal, but also the contents of the meal (menu, etc.). When server device 1 obtains the contents of the meal from the learning model, it can provide the language model 56 of generation server device 5 with the contents of the meal, along with the prompt described later, in addition to the time of capture as the time of the meal.

[0106] In addition, similar to the video and audio described above, still images of the meal may be input to the multimodal language model 56 along with prompts described later, without using a dedicated learning model. Furthermore, even when the recorded data consists of still images of the meal, time information can be manually linked to the image and saved by the patient 91 or others at the time of recording or after recording.

[0107] Next, the server device 1 determines whether analysis of the medical examination record data is necessary (step S86). Whether analysis of the medical examination record data is necessary can be determined, for example, by reading the flag indicating whether analysis of the medical examination record data is necessary in the report creation start request.

[0108] If analysis of the recorded data from the medical examination is required (Step S86: YES), the server device 1 retrieves the recorded data from, for example, the monitoring DB 17 (Step S87). The transfer of the recorded data from the medical institution's recording device 75 to the server device 1 will be described later.

[0109] After acquiring the medical record data, or if analysis of the medical record data is not required (step S86: NO), the server device 1 generates a prompt that includes an instruction to create a report containing graphs and comments related to the graphs (step S88). The prompt includes, for example, instructions, context, input data, and output format.

[0110] The instructions here are commands to create a report based on monitoring data, electronic medical record information, and consultation information. The command to create the report may also include creating a graph based on the acquired monitoring data, and a report on the anomalies and missing data in the graph. Alternatively, the command to create the report may include a graph that uses vital data such as blood glucose levels, classification results obtained from insulin injection images, and meal times.

[0111] Furthermore, the instruction to create a report may involve creating a report using electronic medical record information, including medical history and biochemical test data, as well as information from the time of the consultation, including test results and entries on the medical questionnaire. The instruction to create a report may also involve creating a report that includes a graph showing the trend of biochemical test data and comments on said graph.

[0112] The context includes the role of the language model, the creation of comments when there are singularities or missing data in the graph, and the requirement that the model not make medical judgments. Input data includes monitoring data including vital signs, medical history, electronic medical record information including biochemical test data, test results data from consultations, consultation information including questionnaire entries, meal times, and insulin image classification results. Specific prompts can be, for example, the following:

[0113] "You are an excellent assistant to a doctor who diagnoses lifestyle-related diseases." The following data should be referenced when carrying out the instructions below: • Monitoring data (vital signs (blood pressure, blood glucose level, oxygen saturation, body temperature), meal images, step count information) File name: 20240612-20240615DAT.zip (food images) monitoring.txt (excluding food images) • Electronic medical record information (past medical history, biochemical test data (cholesterol, HbA1c)) File name: karte.txt • Information provided during the consultation (information from the medical questionnaire, pre-consultation test results (heart rate, body temperature, and blood pressure)) File name: today.txt Please organize the following information into a report of no more than two pages: vital signs measured and recorded at home (blood pressure, blood glucose, oxygen saturation, body temperature), meal images, step count, electronic medical record information (medical history, in-hospital test results including cholesterol and HbA1c), measurements taken during hospital visits (pulse, body temperature, blood pressure), and any questions the patient has written to the doctor on the medical questionnaire. Create a graph showing the trend of blood glucose levels, and highlight any notable anomalies or missing measurement periods with comments. If you are concerned that the lack of information may affect the assessment of trends in the numerical data, please comment on that as well. Please present the results of an analysis of the patient's motivation for treatment and the level of credibility of their responses to the doctor, based on the audio data from their previous consultation. Please only compile and organize the recorded data for clarity; do not make any medical judgments based on this information.

[0114] The prompt described above is, for example, a prompt intended for patient 91 with a lifestyle-related disease such as diabetes. Thus, different prompts can be used depending on the patient 91's pre-existing conditions or medical history. As described in this prompt, the prompt requests that a report be created using the acquired information of patient 91, such as vital data, meal images, step count information, electronic medical record information, measurements taken upon arrival (part of the information taken during the examination), and questionnaire 792.

[0115] Server device 1 creates a prompt, as shown in the example prompt creation above, that includes commands to create a graph based on blood glucose levels as acquired monitoring data, and comments regarding singularities and missing data in the graph, and provides it to language model 56. As a result, server device 1 can obtain a report that includes the monitoring data and comments about its singularities and missing data. Note that the prompt does not necessarily have to mention a graph based on the monitoring data.

[0116] The prompts described above are illustrative and not limited to them. For example, the prompt for a meal image may be replaced with audio information. Here, the meal image or audio information can be recorded together with time data to create recorded data from which the time of the meal can be extracted. In addition, the vital information may include not only blood pressure, blood glucose level, oxygen saturation, and body temperature, but also monitoring data including heart rate, respiratory rate, and other vital data, as well as data on the patient's body measured at the living base 100.

[0117] Furthermore, as described above, if the time of the meal has been extracted based on recorded data including meal images or audio information and a dedicated learning model before server device 1 sends a prompt to generation server device 5, server device 1 may not send the recorded data including meal images along with the prompt, but instead send only the time of the meal along with the prompt.

[0118] Step count information can be obtained by the server device 1 as monitoring data related to the patient 91's physical condition measured at the living base 100. Electronic medical record information can include, for example, the patient 91's medical history and biochemical test data, and can include 791 pieces of test data such as test results.

[0119] Measurements taken upon arrival at the hospital may include not only pulse, body temperature, and blood pressure, but also blood tests including cholesterol, HbA1c, blood glucose levels, and other biochemical tests, X-ray examinations, X-ray CT scans, MRI (Magnetic Resonance Imaging) examinations, and other test (measurement) results such as body temperature, blood pressure, heart rate, blood oxygen saturation, height, and weight. If the information from measurements taken upon arrival (test results during examination, etc.) and questionnaire 792 is included in the electronic medical record information, the server device 1 may acquire it as electronic medical record information.

[0120] Server device 1 can transmit the information that patient 91 has written on questionnaire 792 along with prompts. If the server device 1 has received information on items that it considers important, for example, if the "item to consider important" is "changes in blood glucose levels at living base 100", it may add a note to the prompt such as "Please be sure to describe the changes in blood glucose levels at living base 100." Note that server device 1 does not need to have received information on the "item to consider important".

[0121] Furthermore, in the prompt described above, in step S86, the server device 1 has determined that analysis of the medical examination record data is necessary, and has added the following statement: "Please present the analysis results of the patient's motivation for treatment and the level of credibility of their responses to the doctor based on the audio data from the previous examination." This allows the doctor 92, etc., to judge the credibility of the patient 91's responses to the doctor 92. In this prompt, the medical examination record data is assumed to be audio, but it may also be recorded images.

[0122] Furthermore, if the recorded data includes an image of the insulin injection site, server device 1 may add a prompt such as, "If there is an image of the insulin injection site, please describe whether there is any induration at the insulin injection site, or any other possibility of inflammation or damage to blood vessels, etc." Also, if there is an image of a meal prepared by patient 91, server device 1 may add a prompt such as, "Please calculate the amount of carbohydrates from the contents of the meal and provide comments comparing it to the amount of carbohydrates that can be consumed."

[0123] Furthermore, as described above, if, before server device 1 sends a prompt to generating server device 5, an image of the insulin injection site has been extracted from the recorded data including the image of the insulin injection site, and a determination result regarding the insulin injection site has been output through image analysis, then server device 1 may not send the recorded data including the image of the insulin injection site along with the prompt, but may instead send only the determination result regarding the insulin injection site along with the prompt. Also, server device 1 does not need to receive the recorded data including the image of the insulin injection site.

[0124] As described above, the server device 1 can create prompts including commands to create reports based on acquired monitoring data, electronic medical record information, and consultation information, and provide them to the language model 56. This allows the server device 1 to obtain reports based on monitoring data, electronic medical record information, and consultation information.

[0125] Next, the server device 1 inputs the created prompt into the language model (step S89), and displays the report obtained from the language model on the display unit 26 of the examination terminal device 2 (step S90). Examples of reports are described in detail using Figures 17 and 18.

[0126] As described above, the server device 1 can use the electronic medical record information, including the patient's medical history and biochemical test data, as well as the examination information, including the test results and entries in the questionnaire 792, to output a report to assist in the examination using the language model 56. This allows the doctor 92, etc., to conduct the examination by referring to the report, which takes into account the patient's medical history, test results including biochemical test data, and entries in the questionnaire 792.

[0127] Figures 17 and 18 show examples of reports displayed on the medical examination terminal device 2. Figures 17 and 18 show the first and second halves of the same report, respectively. The report is generated by server device 1 using the language model 56 of server device 5. The generated report is sent from server device 1 to medical examination terminal device 2 and displayed on the display unit 26 of medical examination terminal device 2.

[0128] As shown in Figure 17, in this report, the "items to be emphasized" acquired by the server device 1 in step S83 of Figure 16 include, for example, "changes in blood glucose levels at the place of residence." Therefore, the report begins with the item "About blood glucose levels." However, the order in which the items are listed does not have to be from the items related to the "items to be emphasized." In the section about blood glucose levels, a graph showing the changes in blood glucose levels is presented first.

[0129] In this format, a graph of blood glucose levels is used, but it is also possible to output a report that includes graphs of monitoring data values, including other vital data, and comments on the graphs. In the graph of blood glucose levels in Figure 17, the vertical axis represents the SMBG (Self Monitoring of Blood Glucose) blood glucose level, and the horizontal axis represents the date of measurement. Measurements are taken twice a day in principle, and the SMBG blood glucose level is plotted at the position on the horizontal axis corresponding to the measurement date and time. The range of 70-120 mg / dL on the vertical axis of the graph indicates that it is within the blood glucose target range. In the report in Figure 17, as an overall assessment of the graph, a comparison is made between data from different periods of more than one month, and based on whether the blood glucose measurement results fall within the target range, it is stated that there is an upward trend in blood glucose levels.

[0130] Furthermore, regarding singularities (including spikes) in the blood glucose graph, the report indicates the dates and times when large values ​​appeared at times other than after meals (for example, 8:00 AM on May 27th (Sun) and 6:00 PM on June 19th (Sat)). Also, the rise in blood glucose (spike) that occurred at 8:00 AM on March 13th (Sun), etc., is not mentioned in the report because it occurred after a meal. The monitoring data includes mealtime information (including image or audio recordings that can derive mealtimes) at living site 100, and comments on the graph may also include this mealtime information. This allows for reports that take mealtimes into account, for example, when it comes to blood glucose levels.

[0131] Furthermore, Server Device 1 is outputting a statement in the report indicating that blood glucose measurement data is missing. Specifically, it is outputting a statement indicating that blood glucose measurement data is missing for approximately 8:00 AM on May 1st, 6:00 PM on May 1st, 8:00 AM on May 6th, 8:00 AM on May 9th, and 6:00 PM on May 10th. In addition, Server Device 1 is outputting a statement indicating that the absence of this measurement data may affect trend analysis.

[0132] In this way, the server device 1 can output a report that includes a graph showing the trends in vital data measured at the living base 100, and comments regarding anomalies or missing data in the graph. This allows doctors and others who review the report to easily determine, based on the comments regarding anomalies and missing data, whether it is necessary to investigate the reasons for anomalies or to perform an analysis of numerical trends that take missing data into consideration.

[0133] Furthermore, in Figure 17, server device 1 also outputs a report including a graph showing the trend of patient 91's biochemical test data included in the electronic medical record information, and comments related to the graph. Specifically, it generates a graph for the HbA1c item, which is biochemical test data. The vertical axis represents the measured value of HbA1c (NGSP), and the horizontal axis represents the measurement date. The graph includes the results of a blood test taken by patient 91 on the day of the consultation (June 15th), which is included in the pre-consultation consultation information acquired by server device 1 in step S82. It also includes past HbA1c values ​​(April 15th and May 13th) included in the electronic medical record information acquired from electronic medical record server device 4.

[0134] Regarding the graph showing the trend of HbA1c, which is biochemical test data, the comment states, "The value is not in a particularly dangerous range. Analyzing the factors that caused the shift from an upward trend to a downward trend will be useful in determining the effectiveness of treatment." In this way, server device 1 can output a report that includes a graph showing the trend of the biochemical test data of patient 91 included in the electronic medical record information, as well as comments on the graph. This allows physicians 92 and others who review the report to grasp the characteristics of the biochemical test data at a glance.

[0135] As described above, in this embodiment, graphs relating to monitoring data and biochemical test results are shown. However, it is also possible to show images acquired by the living base recording device 65 or the medical institution recording device 75 (including the examination terminal device 2 operating as the medical institution recording device 75), or images acquired from test results or measurement results, and provide a report that includes comments on the images.

[0136] Furthermore, in the report shown in Figure 18, following Figure 17, the server device 1 records the patient's (91) entries in the questionnaire 792 as accurately as possible so that the patient's (91) intentions are conveyed to the physician (92). If the server device 1 finds any information in the monitoring data, electronic medical record information including biochemical test data, or consultation information that relates to the information in the questionnaire 792, it may include that information in the report under the section for monitoring data, electronic medical record information including biochemical test data, or consultation information. Conversely, if the server device 1 has relevant monitoring data, electronic medical record information including biochemical test data, or consultation information related to the content of the questionnaire, it may also refer to that relevant monitoring data, electronic medical record information, or consultation information in the report.

[0137] Furthermore, the server device 1 can record a report on the monitoring data acquired at the living base 100. In the report shown in Figure 18, the pedometer reading is approximately 3000 steps, in conjunction with the statement in questionnaire 792 that "I take a 30-minute walk for exercise," making it easier for the doctor 92 to give their opinion on whether the exercise is sufficient.

[0138] Furthermore, regarding the test data from medical institutions 200, Server Device 1 states that there are no special notes found in the test data, and, in relation to the statement in Questionnaire 792, "I am worried about developing complications (neuropathy, retinopathy, nephropathy)," it also states that there are no special notes found regarding test items related to such complications, such as HbA1c. In this way, Server Device 1 can output a report that takes into account the information in Questionnaire 792, etc. Note that the test data items are not required.

[0139] Furthermore, in the report shown in Figure 18, Server Device 1 outputs a description of the insulin injection site. Regarding the insulin injection site in Figure 18, Server Device 1 indicates that there is no induration in the image of the insulin injection site, but there is inflammation at the injection site, and describes the possibility that the insertion angle, depth, thickness, and length of the injection needle were inappropriate.

[0140] If the monitoring data includes multiple insulin injection site images taken at the living base 100, the server device 1 can output a report that includes a determination of whether or not induration is present at the insulin injection site. This allows physicians 92 and others who review the report to make a diagnosis regarding induration by referring to the determination of whether or not induration is present in the report. The server device 1 does not need to output information about the insulin injection site in the report.

[0141] Furthermore, in the report shown in Figure 18, Server Device 1 outputs a description of the credibility of Patient 91's responses. The description of the credibility of Patient 91's responses will be explained later. Server Device 1 can record the results of an analysis of Patient 91's motivation for treatment and the level of credibility of their responses to Doctor 92, based on consultation record data such as audio data from the previous consultation.

[0142] In this way, the server device 1 can acquire video or audio data of patient 91 during the examination and output a report that includes information about patient 91's psychology. Here, "information about patient 91's psychology" can include, for example, information related to the patient's motivation for treatment and the credibility of their responses to doctor 92, based on an analysis of facial expressions, eye movements, gestures, tone of voice, etc., using the image or audio data.

[0143] In this configuration, the patient's medical record data from the previous consultation is analyzed, but it is also possible to analyze the medical record data from the current day's consultation immediately after the consultation ends. The server device 1 can obtain a report analyzing the medical record data from the current day's consultation by receiving a report creation request from the consultation terminal device 2 or the medical institution terminal 71 that outputs a prompt containing only content that requires analysis of the medical record data, such as "Please present the analysis results of the patient's motivation for treatment and the level of credibility of their responses to the doctor based on the audio data from the consultation." In this case, the prompt does not need to request the output of graphs or other data based on monitoring data or biochemical test data.

[0144] As described above, the server device 1 that executes the program 16 in this form acquires monitoring data from the patient 91's living base 100, the patient 91's electronic medical record information, and examination information obtained during the patient 91's examination. Using the acquired monitoring data, electronic medical record information, and examination information, the language model 56 can output a report to assist in the examination. This makes it easy for doctors 92 and medical institutions to find the necessary information about patient 91.

[0145] Furthermore, as described above, the server device 1 that executes the program 16 of this embodiment can output a report using the language model 56, using the vital data measured at the living base 100 and the monitoring data including multiple meal images or multiple insulin injection site images taken at the living base 100. This allows information to be organized considering meals (or meal times) in relation to changes in vital data. In addition, for diabetic patients, the report can be made considering the timing of meals and insulin injections in relation to changes in blood glucose levels.

[0146] Next, we will explain the process of creating a consultation report, which, based on video or audio recordings of the patient's condition during the consultation by the medical institution recording device 75, obtains a report after the consultation regarding the patient's motivation for treatment and the credibility of the patient's responses to the doctor 92. In this explanation, the term "medical institution recording device 75" should be interpreted as "consultation terminal device 2" which stores consultation record data, in the case of online consultations.

[0147] The medical institution recording device 75 records the details of the examination as examination data during a medical examination in which the doctor 92 is face-to-face with the patient 91. The content recorded here can mainly consist of the conversation between the doctor 92 and the patient 91, and the recording can be done in image or audio format. During or after the examination, the medical institution recording device 75 transmits the examination data to the server device 1.

[0148] Server device 1, upon receiving the medical examination record data, obtains the date and time of receipt and retrieves the monitoring patient ID included in the medical examination record data. Server device 1 then saves the medical examination record data, which may be an image or audio file, as a file in a folder corresponding to the monitoring patient ID, for example.

[0149] Furthermore, Server Device 1 adds a new record to the monitoring data table 172 and saves "Medical Examination Record Data" as the acquired patient ID and monitoring item names in the monitoring patient ID and monitoring item fields of the new record, respectively (for example, Figure 4, monitoring ID "M0012310"). Server Device 1 can also save the file path of the saved medical examination record data in the data field.

[0150] Figure 19 is a flowchart illustrating the process of creating a medical consultation report. As shown in the flowchart of Figure 19, first, the server device 1 determines whether it has received a request to start creating a medical consultation report from the medical consultation terminal device 2 or the medical institution terminal 71 (step S51). Here, the request to start creating a medical consultation report is received along with the medical record patient ID that identifies patient 91.

[0151] In step S51, if the server device 1 has not received a request to start generating a medical examination report (step S51: NO), the process in step S51 is repeated. If the server device 1 has received a request to start generating a medical examination report from the medical examination terminal device 2 or the medical institution terminal 71 (step S51: YES), the server device 1 retrieves the medical examination record data from the medical institution record device 75 stored in the monitoring DB of the server device 1 (step S52).

[0152] Here, the medical examination record data may be sent to the generation server device 5 along with the prompt described later, but it may also be processed in advance on the server device 1. For example, the pre-processing can be done by using a dedicated learning model that has been trained in advance to judge the speaker's motivation and credibility regarding treatment based on video and audio information. Here, the learning model may be a model that judges either motivation or credibility.

[0153] A learning model for motivation towards treatment can be generated by server device 1, etc., by analyzing facial expressions, eye movements, gestures, tone of voice, etc., from image or audio files annotated with, for example, whether the user is positive, neutral, or negative towards treatment, and using supervised learning. This allows for a learning model that can determine motivation towards treatment in three stages—positive, neutral, or negative—and output the determination result.

[0154] A learning model for assessing the credibility level of spoken content can be generated by server device 1, for example, by analyzing facial expressions, eye movements, gestures, tone of voice, etc., from annotated image or audio files that have been evaluated on a 5-point scale for credibility, and then using supervised learning. This allows for a learning model that can determine the credibility level of spoken content on a 5-point scale and output the determination result.

[0155] In this manner, when a judgment result is obtained using a dedicated learning model based on the medical examination record data, only the judgment result obtained from the learning model may be provided to the language model 56 of the generation server device 5 along with the prompts described later, or both the medical examination record data and the judgment result may be provided to the language model 56. The learning model may be stored in the server device 1, or a learning model stored in another information processing device may be used via the network 3. Alternatively, instead of using the image-specific learning model described here, the medical examination record data may be input to the multimodal language model 56 along with the prompts described later.

[0156] After receiving the medical examination record data, the server device 1 generates a prompt that includes an instruction to create a report on the patient's psychology (step S53). The prompt may include, for example, instructions, context, input data, and output format.

[0157] The instructions here may be an order to create a report on the patient's psychology based on the clinical record data. The context includes creating comments on the role of the language model, the patient's motivation for treatment, and the level of credibility of the patient's responses to the doctor. The input data is the clinical record data. Specific prompts may be, for example, the following:

[0158] The prompt created in step S53 is, for example, "You are an excellent assistant to a doctor who diagnoses lifestyle-related diseases." The following data should be referenced when carrying out the instructions below: • Medical examination record data File name: 20240615_CRDAT.zip (Medical examination record data) Please provide a report analyzing patient motivation for treatment and the level of credibility of their responses to the doctor, based on audio data from the consultation. Please keep your report to one page or less and do not make any medical judgments. It can be done in this way.

[0159] Here, as mentioned above, if the judgment result is obtained using a dedicated learning model, the judgment result such as the confidence level may be added to the prompt. Subsequently, the server device 1 inputs the created prompt to the language model (step S54), and displays the report obtained from the language model on the display unit 26 of the examination terminal device 2 (step S55). With the above prompt, the server device 1 can send a report such as the following to the examination terminal device 2.

[0160] [Patient's condition during examination] During the examination, although the patient appeared agitated, they answered the doctor's questions calmly, and their positive responses regarding treatment appear credible.

[0161] In this way, the server device 1 can display the results of an analysis of the patient's motivation for treatment and the level of credibility of their responses to the doctor 92, based on video or audio recordings of the consultation. Furthermore, the server device 1 can acquire video or audio data of the patient 91 during the consultation and output a report that includes information about the patient's psychology.

[0162] Here, "content regarding the patient's psychology" can, for example, be content related to the motivation for treatment and the credibility of responses to the doctor 92, based on analysis of facial expressions, eye movements, gestures, tone of voice, etc., using video, still images, or audio data.

[0163] In this configuration, the flowchart in Figure 19 represents a post-consultation report, which takes place after physician 92 has met with patient 91 during the examination. However, as shown in the flowchart in Figure 16, the pre-consultation report can include comments about the patient's psychology based on the patient's examination record data from the previous consultation. Physician 92 and others reviewing the report can then consider these comments about the patient's psychology when making a diagnosis.

[0164] As described above, the server device 1 that executes the program 16 in this form acquires monitoring data from the patient 91's living base 100, the patient 91's electronic medical record information, and examination information obtained during the patient 91's examination. Using the acquired monitoring data, electronic medical record information, and examination information, the language model 56 can output a report to assist in the examination. This makes it easy for doctors 92 and medical institutions to find the information about patient 91 that they need.

[0165] In the above-described configuration, the medical examination terminal device 2, the medical institution terminal 71, the electronic medical record server device 4, and the generation server device 5 are described as being composed of different information processing devices. However, some or all of these may be composed of a single information processing device.

[0166] In the above configuration, program 16 is executed on server device 1, but part or all of program 16 may be executed on any of the following: examination terminal device 2, medical institution terminal 71, electronic medical record server device 4, or generation server device 5. Furthermore, program 16 can be referred to as a program product, software, or software product, which may be provided on a recording medium or distributed via a communication network.

[0167] The forms of this disclosure are illustrative in all respects and not restrictive. The scope of the invention is not limited to those shown in the above disclosure but is shown by the claims, and all modifications within the meaning and scope equivalent to the claims are intended.

[0168] Furthermore, the sequences shown in each embodiment described above are not limited, and within a reasonable scope, the order of each processing step may be changed, and multiple processes may be executed in parallel. The processing entity for each process is not limited, and within a reasonable scope, the processing of each device may be executed by other devices.

[0169] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, although the claims do not use the form of a multi-claim that further references a multi-claim (multi-multi-claim), it may be a combination that uses the form of a multi-multi-claim that references all higher-level claims. [Explanation of Symbols]

[0170] 1. Information processing device (server device) 11 Control Unit 12 Communications Department 13 Reading Unit 14 Storage section 15 Portable storage media 16 Programs (Program Products) 17. Monitoring DB (Monitoring Database) 171 Monitoring Patient Table 172 Monitoring Data Table 19 bus 2. Examination terminal device 3 Network 4. Electronic medical record server device 46. ​​Electronic Medical Record Database (Electronic Medical Record DB) 461 Medical Record Patient Table 462 Medical Record Data Table 5. Generation Server Device 56 Language Models 61 Monitoring device 61a Blood glucose meter 61b Wearable devices 61c blood pressure monitor 65. Living Location Recording Device 68 Living Base Terminal 71 Medical Institution Terminals 75 Medical Recording Devices 791 Test results, etc. 792 Questionnaire 793 Inspection equipment 100 Living Bases 200 Medical institutions, etc.

Claims

1. The system acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and information obtained during the patient's examination. Using the acquired monitoring data, electronic medical record information, and consultation information, a language model is used to generate a report to assist in the consultation. A program that instructs a computer to perform a process.

2. The language model outputs the report using vital data measured at the aforementioned living location and monitoring data including multiple meal images or multiple insulin injection site images taken at the aforementioned living location. The program according to claim 1.

3. Using the electronic medical record information, including the patient's medical history and biochemical test data, and the consultation information, including test result data and questionnaire entries from the consultation, the language model outputs the report to assist in the consultation. The program according to claim 2.

4. The report outputs a graph showing the trends in vital data measured at the aforementioned living location, and comments related to the graph. The program according to claim 1.

5. The monitoring data includes mealtime information, which is the time of meals at the living base. The comments regarding the aforementioned graph include the aforementioned mealtime information. The program according to claim 4.

6. The report outputs a graph showing the trends in vital data measured at the aforementioned living location, and comments regarding singularities or missing data in the graph. The program according to claim 4.

7. Output the report which includes a graph showing the trend of the patient's biochemical test data included in the electronic medical record information, and comments related to the graph. The program according to claim 1.

8. The monitoring data includes multiple images of insulin injection sites taken at the living location. Output the report, which includes a determination of whether or not there is induration at the insulin injection site. The program according to claim 1.

9. During the examination, video or audio data of the aforementioned patient is acquired. Output the report that includes information about the patient's psychology. The program according to claim 1.

10. The language model is given prompts that include commands to create the report based on the acquired monitoring data, the electronic medical record information, and the consultation information. The program according to claim 1.

11. The language model is given prompts that include commands to create a graph based on the acquired monitoring data, and comments regarding singularities and missing data in the graph. The program according to claim 1.

12. The system acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and information obtained during the patient's examination. Using the acquired monitoring data, electronic medical record information, and consultation information, a language model is used to generate a report to assist in the consultation. Information processing methods.

13. It includes a control unit, and the control unit is The system acquires monitoring data at the patient's place of residence, the patient's electronic medical record information, and information obtained during the patient's examination. Using the acquired monitoring data, electronic medical record information, and consultation information, a language model is used to generate a report to assist in the consultation. Information processing device.

Images