Program, information processing method, and information processing device

Abstract

To provide a program, etc., with which it is possible to present an appropriate surgery plan from medial images in which the affected area of a patient is captured.SOLUTION: The program causes a computer to execute the processing of: acquiring a medical image in which the affected area of a patient is captured; inputting a prompt including the medical image to a language model so as to acquire surgery plan information indicating a surgery plan of surgical operation from the language model; and displaying the surgery plan information on a display unit. Preferably, the surgery plan information is obtained from the language model by acquiring clinical record data of the patient diagnosed by a doctor and inputting the medical image and a prompt including the clinical record data to the language model.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, an information processing device, and an information processing system. [Background technology]

[0002] There are systems that propose optimal plans (procedures, etc.) for surgical operations. For example, Patent Document 1 discloses a surgical procedure proposal device that acquires a radiographic image of a patient's joint, inputs the preoperative radiographic image, the surgical procedure performed on the joint, and the state of the joint after the surgery, and uses a machine learning model that outputs the surgical procedure to be performed, to select and output the surgical procedure to be performed for the joint shown in the acquired radiographic image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-115188 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object of the present invention is to provide a program or the like that can present an appropriate surgical plan from medical images of the affected area of ​​a patient. [Means for solving the problem]

[0005] In one aspect, the program causes a computer to acquire medical images of a patient's affected area, input a prompt including the medical images into a language model, acquire surgical plan information representing a surgical plan from the language model, and display the surgical plan information on a display unit. [Effects of the Invention]

[0006] In one aspect, an appropriate surgical plan can be presented from medical images of the affected area of ​​a patient. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a medical support system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] 1 is an explanatory diagram showing an example of the record layout of a medical fee DB, a conversion table, and a surgery history DB. [Figure 4] FIG. 2 is a block diagram illustrating an example of the configuration of a terminal. [Figure 5] FIG. 1 is an explanatory diagram illustrating an overview of an embodiment. [Figure 6] FIG. 10 is an explanatory diagram regarding masking processing of personal information. [Figure 7] FIG. 10 is a diagram showing a surgical plan creation screen. [Figure 8] FIG. 10 is a diagram showing a budget / actual management screen. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure executed by the medical assistance system. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention will be described in detail below with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a medical support system. In this embodiment, a medical support system (information processing system) will be described that uses an LLM50 (Large Language Model) to generate and present surgical plan information that represents a surgical plan for a surgical operation to be performed on a patient. The medical support system includes an information processing device 1, terminals 2, 2, 2..., and a generation server 3. Each device is communicatively connected via a network N such as the Internet.

[0009] The information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. The server 1 acquires surgical plan information from the LLM 50 by inputting a prompt containing a medical image of the patient's affected area and the patient's medical record data diagnosed by a doctor into the LLM 50. The server 1 displays the acquired surgical plan information on the terminal 2 and presents it to the user (medical professional).

[0010] Terminal 2 is a mobile terminal used by each user, such as a smartphone or tablet terminal. Terminal 2 captures images of the affected area (e.g., X-ray images) and displays an electronic medical record in accordance with user operations. Server 1 acquires each captured image as medical image and medical record data and inputs them into LLM 50 to acquire surgical plan information. As will be described later, terminal 2 displays multiple surgical plan information (tags) generated by LLM 50 as candidates and creates a surgical plan by accepting input to select surgical plan information from the candidates.

[0011] The generation server 3 is a server computer that generates a response sentence using an LLM 50 from input data such as images and text. The LLM 50 is a large-scale language model such as a Generative Pre-trained Transformer (GPT) or a Bidirectional Encoder Representations from Transformer (BERT), but the model is not particularly limited. When the server 1 acquires medical images and medical record data from the terminal 2, it creates a prompt including the medical images and medical record data and sends it to the generation server 3, thereby causing the generation server 3 to generate surgical plan information.

[0012] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc., and performs various information processing, control processing, etc. by reading and executing a program P1 stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

[0013] The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P1 (program product) and other data required for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a medical fee DB 141, a conversion table 142, and a surgical history DB 143. The medical fee DB 141 is a database that stores medical fee information (information about the surgical procedure) for each surgical procedure. The conversion table 142 is a table that is referenced when converting the surgical procedure name output from the LLM 50 as surgical plan information from the official name for medical fee to the commonly used name for medical practice. The surgical history DB 143 is a database that stores the history of surgeries performed on each patient in accordance with the surgical plan.

[0014] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0015] Furthermore, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. Furthermore, the server 1 may be provided with a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute the program P1 from the portable storage medium 1a.

[0016] FIG. 3 is an explanatory diagram showing an example of the record layout of the medical fee DB 141, the conversion table 142, and the surgery history DB 143.

[0017] The medical fee DB 141 includes a medical fee code column, a surgical procedure name column, a body part name column, a vector value column, and a medical fee point column. The medical fee code column stores a medical fee code (e.g., K code). The surgical procedure name column, body part name column, vector value column, and medical fee point column store a surgical procedure name, body part (affected area) name, vector value, and medical fee point in association with the medical fee code, respectively. Vector values ​​will be described later.

[0018] The conversion table 142 includes a formal name column and a common name column. The formal name column stores the formal names of surgical procedures used in medical fees. The common name column stores the common names of surgical procedures used in clinical practice in association with the formal names.

[0019] The surgery history DB143 includes a surgery date column, a patient name column, a surgical procedure name column, a body part name column, and a surgery time column. The surgery date column stores the surgery date. The patient name column, surgical procedure name column, body part name column, and surgery time column store the name of the patient who underwent surgery, the name of the surgical procedure, the name of the body part where the surgery was performed, and the surgery time, respectively, in association with the surgery date.

[0020] 4 is a block diagram showing an example of the configuration of the terminal 2. The terminal 2 includes a control unit 21, a main memory unit 22, a communication unit 23, a display unit 24, an input unit 25, an imaging unit 26, and an auxiliary memory unit 27. The control unit 21 has one or more processors such as CPUs, and performs various information processing by reading and executing the program P2 stored in the auxiliary storage unit 27. The main storage unit 22 is a temporary storage area such as RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The display unit 24 is a display screen such as a liquid crystal display, and displays images. The input unit 25 is an operation interface such as a touch panel, and accepts operation input from the user. The imaging unit 26 is a camera equipped with an imaging element such as a CMOS (Complementary-MOS), and captures images. The auxiliary storage unit 27 is a non-volatile storage area such as a hard disk, and stores the program P2 (program product) and other data necessary for the control unit 21 to execute processing.

[0021] The terminal 2 may be provided with a reading unit for reading a portable storage medium 2b such as a CD-ROM, and may read and execute the program P2 from the portable storage medium 2b.

[0022] Fig. 5 is an explanatory diagram showing an overview of the embodiment. Fig. 5 illustrates how surgical plan information is generated by inputting a prompt including a medical image (e.g., an X-ray image) and medical record data read from an electronic medical record into the LLM 50. The overview of this embodiment will be described based on Fig. 5.

[0023] In this embodiment, X-ray images are used as medical images, but the medical images may also be images of modalities other than X-ray images (for example, ultrasound images, CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, etc.).

[0024] LLM50 is a machine learning model that has been trained on a large amount of training data and generates a response sentence when it receives input data (images, text, etc.). As described above, LLM50 is a large-scale language model such as GPT or BERT, but the model is not particularly limited.

[0025] In this embodiment, the server 1 uses the LLM 50 to create a surgical plan for a surgical operation. Specifically, the server 1 inputs a prompt including a medical image of the patient's affected area and medical record data read from an electronic medical record into the LLM 50, thereby obtaining surgical plan information representing a surgical plan for a surgical operation from the LLM 50.

[0026] For example, the user uses terminal 2 (mobile terminal) to capture an image of the affected area (e.g., an X-ray image) and also captures the display screen of the electronic medical record. Preferably, as shown in Fig. 5, multiple images of the affected area are captured from multiple directions (two directions in Fig. 5). After capturing each image, terminal 2 transmits (outputs) the image to server 1 and requests output of surgical plan information.

[0027] Here, medical images such as X-ray images may contain personal information such as the patient's name, age, and patient ID. If this image is sent to server 1 as is, there is a possibility that the personal information will remain as a log in server 1 or be used for learning LLM50. Therefore, terminal 2 generates an image in which the personal information is masked and sends it to server 1.

[0028] FIG. 6 is an explanatory diagram regarding the masking of personal information. The X-ray image shown on the left side of FIG. 6 contains information such as the patient ID, name, date of birth, and age. Terminal 2 recognizes the location of personal information in the image using optical character recognition (OCR). Terminal 2 then generates a medical image in which the location has been masked, as shown on the right side of FIG. 6. Terminal 2 transmits the generated medical image to server 1.

[0029] In this embodiment, the image masking process is performed in the terminal 2, but the server 1 may also perform the image masking process.

[0030] Returning to Figure 5, the explanation continues. The server 1 acquires a medical image in which the personal information has been masked and an image of the display screen of the electronic medical record from the terminal 2. The server 1 reads text from the image of the display screen of the electronic medical record using character recognition as medical record data.

[0031] In this embodiment, the server 1 reads the medical chart data, but the terminal 2 may read the medical chart data.

[0032] The server 1 creates a prompt including the above-mentioned medical images and medical record data, and outputs the prompt to the generation server 3. Specifically, the server 1 creates a prompt including a fixed phrase that instructs the LLM 50 to generate multiple pieces of surgical plan information, and the medical images and medical record data. The server 1 outputs the prompt to the generation server 3, causing the LLM 50 to generate multiple pieces of surgical plan information.

[0033] In this embodiment, the server 1 uses the medical fee DB 141 to generate surgical plan information using RAG (Retrieval Augmented Generation) technology. The medical fee DB 141 is a database that stores information (medical fee information) about each surgical procedure, such as the procedure name, the part name, and the medical fee points. The medical fee DB 141 stores information about each surgical procedure in association with vector values ​​converted from medical images and chart data.

[0034] The server 1 converts the medical image and medical record data into vector values ​​(features) by inputting a prompt containing the medical image and medical record data into the LLM 50. The server 1 then searches for information about surgical procedures from the medical fee DB 141 using the converted vector value as a search query. For example, the server 1 converts the medical image and medical record data of representative cases corresponding to each surgical procedure in advance using a predetermined converter (an embedding model), and stores them in the medical fee DB 141 in association with the surgical procedure name, body part name, etc. For example, the server 1 obtains information about the top few surgical procedures that are closest to the vector values ​​converted from the medical image and medical record data of the patient currently being processed from the medical fee DB 141.

[0035] In this embodiment, the medical image and medical record data are described as being converted together into a single vector value, but it is also possible to convert the medical image and medical record data separately into vector values ​​and obtain information about the surgical procedure from the medical fee DB 141 using at least one of the vector values.

[0036] The server 1 inputs the acquired information and a prompt including the medical image and medical record data (vector values) used as the search query to the LLM 50. As a result, the LLM 50 generates surgical plan information (response sentence) by referencing information about the surgical procedure acquired from the medical fee DB 141. The LLM 50 generates multiple pieces of surgical plan information by referencing information about each of the top several surgical procedures. The server 1 acquires the surgical plan information generated by the LLM 50.

[0037] The server 1 may output the surgical plan information (response sentence) generated by the LLM 50 directly to the terminal 2 for display, but in this embodiment, words representing the surgical procedure name and the part name are extracted from the surgical plan information generated by the LLM 50, and the extracted words are displayed as "tags."

[0038] 7 is a diagram showing the surgical plan creation screen. This screen includes a surgical plan input field 71 and a candidate display field 72. The candidate display field 72 is a display field that displays, as tags 73, each word (each surgical procedure name and site name) extracted from multiple pieces of surgical plan information generated by the LLM 50. The surgical plan input field 71 is an input field for selecting and inputting tags 73 representing the surgical procedure name to be performed on the patient and the name of the affected site from the tags 73 displayed in the candidate display field 72.

[0039] As described above, the server 1 inputs a prompt to the LLM 50 to instruct the generation of multiple pieces of surgical plan information, thereby acquiring multiple pieces of surgical plan information from the LLM 50. The server 1 extracts the surgical procedure name and the site name from each piece of surgical plan information.

[0040] As described above, the LLM 50 generates surgical plan information based on information about surgical procedures (medical fee information) stored in the medical fee DB 141. However, the name of the surgical procedure used in calculating the medical fee is an official name such as "open reduction and internal fixation," but this name is not a commonly used name used by medical professionals in clinical practice. For example, medical professionals generally call it "ORIF" rather than "open reduction and internal fixation."

[0041] Therefore, the server 1 converts the surgical procedure name extracted from the surgical plan information into a commonly used name by referring to the conversion table 142. The server 1 causes the terminal 2 to display a tag 73 indicating the converted surgical procedure name and the site name.

[0042] Terminal 2 displays the surgical procedure names and site names extracted from each surgical plan information as tags 73, 73, 73... in a candidate display field 72. The user considers the surgical procedure, etc., referring to the tags 73, i.e., the surgical procedure names and site names, displayed in the candidate display field 72. The user then selects one of the tags 73, 73, 73... and enters it in the surgical plan input field 71 to create a surgical plan.

[0043] Of course, the user may create a surgical plan by freely inputting the name of the surgical procedure, etc., without following the surgical plan (tag 73) generated by the LLM 50.

[0044] The created surgical plan is registered in this system as a surgical schedule. After the surgery, the user registers the details of the surgery (for example, the surgery time, etc.) in this system. The registered details of the surgery are saved in the surgery history DB 143 as a surgery history.

[0045] The server 1 generates various dashboards based on the surgery history stored in the surgery history DB 143 and displays them on the terminal 2. For example, the server 1 generates a screen for managing the budget and actual results of the hospital and displays it on the terminal 2.

[0046] Figure 8 is a diagram showing the budget / actual management screen. As shown in Figure 8, the server 1 calculates the budget and actual sales of the hospital, generates various graphs, and displays them on the terminal 2. The server 1 calculates the actual sales figures based on the medical fee points corresponding to each surgical procedure and site stored in the medical fee DB 141 and the surgical procedure name and site name of each surgery stored in the surgery history DB 143, and generates the graphs. This makes it possible to manage the budget / actual sales of the hospital through this system.

[0047] 9 is a flowchart showing an example of a processing procedure executed by the medical assistance system, and the processing content executed by the medical assistance system will be described with reference to FIG. The control unit 21 of the terminal 2 captures an image of the affected area (e.g., an X-ray image) and a display screen of the electronic medical record in accordance with an operation input by the user (step S11). For example, the control unit 21 captures multiple images of the affected area captured from multiple directions and a display screen of the electronic medical record.

[0048] The control unit 21 recognizes the location (text) of personal information from the captured image of the affected area (step S12). The control unit 21 generates an image in which the recognized location is masked as a medical image (step S13). The control unit 21 outputs the medical image and an image of the display screen of the electronic medical record to the server 1 (step S14).

[0049] The control unit 11 of the server 1 reads text from an image captured of the display screen of the electronic medical record as medical record data (step S15). The control unit 11 converts the medical image and medical record data into vector values ​​by inputting a prompt including the medical image and medical record data into the LLM 50 (step S16). The control unit 11 acquires (searches) information about the surgical procedure based on the converted vector value from the medical fee DB 141, which stores information about each surgical procedure (medical fee information) in association with the vector value (step S17).

[0050] The control unit 11 inputs a prompt including the acquired information and (vector values ​​of) the medical image and medical record data used as the search query to the LLM 50, thereby acquiring surgical plan information representing a surgical plan for a surgical operation from the LLM 50 (step S18). Specifically, the control unit 11 inputs a prompt to the LLM 50 instructing the generation of multiple pieces of surgical plan information, thereby acquiring multiple pieces of surgical plan information (response sentences). The control unit 11 extracts words representing surgical procedure names and body part names from each piece of acquired surgical plan information (step S19). The control unit 11 refers to the conversion table 142 and converts the extracted words representing surgical procedure names from official names for medical fees to commonly used names for medical practice (step S20). The control unit 11 outputs the words representing surgical procedure names and body part names to the terminal 2 (step S21).

[0051] The control unit 21 of the terminal 2 displays words (tags 73) representing the names of each surgical procedure and the names of the body parts as candidates for the surgical plan (step S22). The control unit 21 receives an input to select one of the displayed words (step S23). In response, the control unit 21 creates a surgical plan. The control unit 21 ends the series of processes.

[0052] As described above, according to this embodiment, it is possible to present an appropriate surgical plan from medical images of the affected area of ​​a patient.

[0053] (Variation) In the above-described embodiment, the RAG technology is used to obtain the necessary information from the medical fee DB 141 so that the LLM 50 does not need to learn medical images and the like in advance, but the present embodiment is not limited to this. The server 1 may train the LLM 50 so that it can generate surgical plan information without using an external database.

[0054] That is, the server 1 may fine-tune the LLM 50 using training data in which correct surgical plan information (surgical procedure name and body part name) is associated with training medical images and medical record data. The server 1 generates surgical plan information by inputting training medical images and medical record data into the LLM 50, and optimizes parameters such as weights between neurons so that the generated surgical plan information approximates the correct surgical plan information. This constructs an LLM 50 that is trained to generate surgical plan information when it receives input of a prompt including medical images and medical record data. The server 1 inputs a prompt including a patient's medical images and medical record data into the constructed LLM 50, thereby acquiring surgical plan information and displaying it on the terminal 2.

[0055] In this way, it is not essential to configure the LLM 50 to generate surgical plan information using RAG technology, and the LLM 50 may be made to perform learning in advance so that it can generate correct surgical plan information.

[0056] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0057] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multi-claim format), this is not limited to this format. A format in which multiple claims (multi-multi-claims) that reference at least one other multiple claim may also be used. [Explanation of symbols]

[0058] 1. Server (information processing device) 11 Control section 12 Main memory 13 Communications Department 14 Auxiliary storage P1 Program 141 Medical fee DB 142 Conversion Table 143 Surgery History DB 2. Devices (mobile devices) 21 Control section 22 Main memory 23 Communications Department 24 Display section 25 Input section 26 Imaging unit 27 Auxiliary storage P2 Program 3 Generation Server 50 LLM (Language Modeling)

Claims

1. Acquire a medical image of the affected area of ​​the patient; inputting a prompt including the medical image into a language model to obtain surgical plan information representing a surgical plan for a surgical operation from the language model; The surgical plan information is displayed on a display unit. A program that causes a computer to perform a process.

2. Obtaining medical record data of the patient diagnosed by a doctor; The surgical plan information is obtained from the language model by inputting a prompt including the medical image and the medical record data into the language model. The program according to claim 1.

3. An image of the affected area captured by a mobile terminal and an image of the display screen of an electronic medical record captured by the mobile terminal are acquired as the medical image and medical record data, reading text from an image of the display screen of the electronic medical record; The surgical plan information is obtained from the language model by inputting a prompt including the read text and the medical image into the language model. The program according to claim 2.

4. converting the medical image into vector values ​​by inputting a prompt including the medical image into the language model; Obtaining information about surgical procedures based on the converted vector values ​​from a database that stores information about each surgical procedure in association with a vector value; The surgical planning information is obtained from the language model by inputting a prompt including the obtained information into the language model. The program according to claim 1.

5. A plurality of X-ray images of the affected area taken from a plurality of directions are acquired as the medical images; inputting a prompt including the plurality of X-ray images into the language model to obtain the surgical planning information from the language model; The program according to claim 1.

6. Recognizing the location of personal information from the medical image; generating the medical image with the written portion masked; The surgical plan information is obtained from the language model by inputting a prompt including the generated medical image into the language model. The program according to claim 1.

7. extracting words representing the surgical procedure name and the site name from the surgical plan information; Displaying extracted words The program according to claim 1.

8. Converting the extracted words representing the names of the surgical procedures from formal names to common names; Display the converted word The program according to claim 7.

9. A medical image of the affected area of ​​the patient is acquired. inputting a prompt including the medical image into a language model to obtain surgical plan information representing a surgical plan for a surgical operation from the language model; The surgical plan information is displayed on a display unit. An information processing method in which processing is performed by a computer.

10. An information processing device including a control unit, The control unit A medical image of the affected area of ​​the patient is acquired. inputting a prompt including the medical image into a language model to obtain surgical plan information representing a surgical plan for a surgical operation from the language model; The surgical plan information is displayed on a display unit. Information processing device.

11. An information processing system having a mobile terminal and an information processing device, The mobile terminal, A medical image of the affected area of ​​the patient is acquired. Recognizing the location of personal information from the medical image; generating the medical image with the written portion masked; outputting the generated medical image to the information processing device; The information processing device, acquiring the medical image from the mobile terminal; inputting a prompt including the medical image into a language model to obtain surgical plan information representing a surgical plan for a surgical operation from the language model; The surgical plan information is output to the mobile terminal. Information processing system.

Images