Information Processing Method, Program, and Information Processing System

Abstract

Provided is an information processing method, for example, for assisting the sharing of information among staff members participating in a treatment.　The information processing method causes a computer to perform a process for acquiring patient information pertaining to a patient receiving a catheter treatment, successively acquiring actions related to the catheter treatment for the patient in association with the time that the actions were performed, and causing the acquired patient information and time-series information in which the acquired actions are arranged in time-series to be displayed on a first display device.

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing system.

Background Art

[0002] Catheter treatment is carried out with the cooperation of staff of various occupations such as doctors, nurses, clinical laboratory technicians, clinical engineering technicians, and radiological technicians. By sharing in real time information on the progress of the treatment and future prospects, each staff member can efficiently perform their duties. As a result, the treatment can proceed smoothly.

[0003] Based on the activation states of a plurality of machines installed in the operating room, a device has been proposed that notifies staff outside the operating room, etc., that the end time of the operation will be later than scheduled (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the device of Patent Document 1 cannot assist in information sharing among the staff participating in the treatment.

Means for Solving the Problems

[0006] The information processing method acquires patient information regarding a patient undergoing catheter treatment, sequentially acquires an action related to the catheter treatment for the patient and the time at which the action was performed in association with each other, and causes a first display device to display the acquired patient information and time-series information in which the acquired actions are arranged in time series. When the action is an action of acquiring an image, input the acquired image into a first model that outputs a converted image obtained by converting the image when the image is input, and obtain the converted image output from the first model or a reduced converted image obtained by reducing the converted image, and display, in the time series information, a reduced image obtained by reducing the image in association with the time when the image was acquired, and the converted image or the reduced converted image Causes a computer to execute the processing. [Effect of the Invention]

[0007] An information processing method and the like that can assist information sharing among staff members participating in treatment can be provided. [Brief Description of the Drawings]

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Embodiments for Carrying Out the Invention

[0009] [Embodiment 1] FIG. 1 is an explanatory diagram for explaining the configuration of the information processing system 10. The information processing system 10 of the present embodiment is used in a catheter room and an operation room adjacent to the catheter room. A door and a window (not shown) are provided between the catheter room and the operation room. Note that FIG. 1 schematically shows the information processing system 10. The arrangement and number of each component are not limited to those in FIG. 1. Note that the catheter room is an example of a place where the information processing system 10 is used. The information processing system 10 may be used in an operating room or the like.

[0010] The information processing system 10 includes an information processing apparatus 20, a terminal apparatus 30, a display apparatus 12, a catheter control apparatus 13, a microphone 17, a camera 11, a bedside monitor 16, a barcode reader 15, an image diagnostic apparatus 14, and an electronic medical record system 19. Each apparatus or the like is connected by wire or wirelessly via a network such as a HIS (Hospital Information System). Some apparatuses may be directly connected without going through the network.

[0011] The information processing apparatus 20 and the electronic medical record system 19 may be arranged outside the catheter room and the operation room. Each component may be carried into the catheter room and carried out of the catheter room as needed.

[0012] The information processing system 10 of the present embodiment is a system for supporting catheter treatment performed in a catheter room. A patient receiving catheter treatment lies on the examination table 18.

[0013] The terminal device 30 is a device mainly used by staff or observers in charge of the unclean area in the catheter room or the like. The terminal device 30 cooperates with the information processing device 20 to execute the program of the present embodiment. Details of the information processing device 20 and the terminal device 30 will be described later. The terminal device 30 is an example of the first display device.

[0014] The display device 12 is a large display device such as a large liquid crystal display device or an organic EL (Electro Luminescence) display device. A plurality of display devices 12 are installed in the catheter room. For example, three display devices 12 are suspended from the ceiling of the catheter room in a side-by-side arrangement and are adjusted to a position that is easy for the doctor in charge of catheter treatment to view. Further, the display device 12 is also arranged on the wall surface of the catheter room and in the operation room adjacent to the catheter room. The display device 12 is an example of the second display device.

[0015] For example, when performing intravascular catheter treatment, the latest image taken by the X-ray angiography device is constantly displayed on one of the plurality of display devices 12 arranged in a state that is easy for the doctor in charge of catheter treatment to view. Information displayed on the other display devices 12 will be described later.

[0016] An MDU 131 and an image diagnostic catheter 132 are connected to the catheter control device 13. The image diagnostic catheter 132 is inserted into a luminal organ for use. The luminal organ into which the image diagnostic catheter 132 is inserted is, for example, a blood vessel, a pancreatic duct, a bile duct, or a bronchus. The image diagnostic catheter 132 may also be inserted into the interior of the heart, such as into the atrium or ventricle, through a blood vessel.

[0017] The catheter 132 for image diagnosis is, for example, a catheter for IVUS (Intravascular Ultrasound), a catheter for OCT (Optical Coherence Tomography), or a catheter for OFDI (Optical Frequency Domain Imaging), etc. The catheter 132 for image diagnosis may be equipped with a combination of two or three of the sensors for IVUS, OCT, and OFDI.

[0018] Using the catheter 132 for image diagnosis, a tomographic image can be generated that includes reflectors existing inside the lumen organ such as red blood cells, in addition to the lumen organ wall such as the blood vessel wall, and organs existing outside the lumen organ such as the epicardium and the heart.

[0019] The catheter 132 for image diagnosis may be an endoscope for blood vessels that is used when optically observing the inner wall of the lumen organ.

[0020] The microphone 17 receives voice input. The bedside monitor 16 is a device that measures and displays vital signs such as the electrocardiogram, blood pressure, heart rate, and percutaneous arterial oxygen saturation (SPO2: Peripheral Oxygen Saturation) of the patient receiving catheter treatment. Electrodes for electrocardiogram measurement and the like are connected to the bedside monitor 16.

[0021] The camera 11 photographs the situation in the catheter room. The photographed video is displayed, for example, on the display device 12 in the operation room. The photographed video may be recorded in the log DB 65 (see Figure 2) described later. The camera 11 may be used to receive gesture input.

[0022] The barcode reader 15 is used to read barcodes provided on packaging materials for medical devices such as the imaging catheter 132 used in catheter treatment, guiding catheters, guidewires, balloon catheters, and stents. Information regarding the type, serial number, etc. of the medical devices is recorded in the barcode. The barcode may record a URL (Uniform Resource Locator) that enables access to information regarding the detailed specifications, etc. of the medical devices.

[0023] The imaging diagnostic apparatus 14 is, for example, an X-ray angiography apparatus, an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, a PET (Positron Emission Tomography) apparatus, or an ultrasonic diagnostic apparatus.

[0024] In the electronic medical record system 19, information such as the patient's name, date of birth, data of various examinations performed prior to catheter treatment, and medical history is recorded in association with a patient ID (Identifier) uniquely assigned to the patient. Information regarding catheter treatment performed using the information processing system 10 of the present embodiment is recorded in the electronic medical record system 19.

[0025] In the following description, there are cases where medical images are described to include images captured by the catheter control device 13 and images captured by the imaging diagnostic apparatus 14.

[0026] FIG. 2 is an explanatory diagram for explaining the configuration of the information processing apparatus 20. The information processing apparatus 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, and a bus. The control unit 21 is an arithmetic control device that executes the program of the present embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or multi-core CPUs, etc. are used for the control unit 21. The control unit 21 is connected to each hardware part constituting the information processing apparatus 20 via a bus.

[0027] The main storage device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. In the main storage device 22, information necessary during the processing performed by the control unit 21 and the program being executed by the control unit 21 are temporarily stored.

[0028] The auxiliary storage device 23 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. In the auxiliary storage device 23, the program to be executed by the control unit 21, the log DB (Database) 65, the standby device DB 66, the drug accumulation amount DB 67, and various data necessary for program execution are stored. The log DB 65, the standby device DB 66, and the drug accumulation amount DB 67 may be stored in an external large-capacity storage device or the like connected to the information processing apparatus 20. The communication unit 24 is an interface for performing communication between the information processing apparatus 20 and a network.

[0029] The information processing apparatus 20 is a general-purpose personal computer, tablet, mainframe computer, or a virtual machine operating on a mainframe computer. The information processing apparatus 20 may be configured by a plurality of personal computers performing distributed processing, or hardware such as a mainframe computer. The information processing apparatus 20 may be configured by a cloud computing system or a quantum computer.

[0030] FIG. 3 is an explanatory diagram for explaining the configuration of the terminal device 30. The terminal device 30 includes a control unit 31, a main storage device 32, an auxiliary storage device 33, a communication unit 34, a touch panel 35, and a bus. The touch panel 35 includes a display unit 351 such as a liquid crystal panel or an organic EL panel, and an input unit 352 laminated on the surface of the display unit 351.

[0031] The control unit 31 of the terminal device 30 is an arithmetic control device that executes the program of the present embodiment in cooperation with the control unit 21 of the information processing device 20. One or more CPUs, GPUs, or multi-core CPUs, etc. are used for the control unit 31. The control unit 31 is connected to each hardware part constituting the terminal device 30 via a bus.

[0032] The main storage device 32 is a storage device such as SRAM, DRAM, or flash memory. In the main storage device 32, information necessary during the processing performed by the control unit 31 and the program being executed by the control unit 31 are temporarily stored.

[0033] The auxiliary storage device 33 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. In the auxiliary storage device 33, the program to be executed by the control unit 31 and various data necessary for the execution of the program are stored. The communication unit 34 is an interface for performing communication between the terminal device 30 and a network.

[0034] The terminal device 30 is a general-purpose information device such as a tablet, smartphone, or tablet. The terminal device 30 may be an information device dedicated to the information processing system 10 of the present embodiment. The terminal device 30 may be individually carried by a staff member such as a nurse, or may be installed on a wall or the like.

[0035] FIG. 4 is an explanatory diagram for explaining the record layout of the log DB 65. The log DB 65 is a DB that records a catheter treatment log associating a patient ID, the date of catheter treatment, and the progress of the catheter treatment. The log DB 65 has a patient ID field, a date field, and a progress field. The progress field has a time field, an action field, a file field, a patient state field, a vital field, and a strategy field.

[0036] The patient ID is recorded in the patient ID field. The date of catheter treatment is recorded in the date field. The time is recorded in the time field. The action performed during the catheter treatment is recorded in the action field.

[0037] Images acquired from the catheter control device 13 or the imaging diagnostic device 14, files such as voice memos acquired from the microphone 17, etc. are recorded in the file field. A link to these files is recorded in the file field, and the files themselves may be recorded in a file server or the like (not shown). A video acquired from the catheter control device 13 or the imaging diagnostic device 14 may be recorded in the file field.

[0038] The patient state is recorded in the patient state field with symbols such as "A", "B", etc. Details of the patient state will be described later. The vital signs when the doctor checked the vital signs are recorded in the vital field. Note that apart from the recording in the vital field, log data of various vital signs are recorded by the bedside monitor 16.

[0039] A file regarding the treatment strategy formulated during the catheter treatment is recorded in the strategy field. Details of the treatment strategy will be described later. An empty field in the action field and each field indicates that the corresponding information is not recorded in the log DB 65.

[0040] For example, at 10:05, an action of administering "heparin XXXX units" to the patient is recorded. The patient's condition and vital signs at the time of administration are recorded, but the file and strategy are not recorded. At 10:35, the patient's condition is recorded, and actions, etc. are not recorded. Details of the data recording method in the log DB65 will be described later.

[0041] FIG. 5 is an explanatory diagram for explaining the record layout of the standby device DB66. The standby device DB66 is a DB that records the association between the equipment used in the catheterization laboratory and the number of times of use. The standby device DB66 has an equipment information field and a usage times field. The equipment information field has a variety field and a type field.

[0042] In the variety field, the variety of each piece of equipment is recorded. In the type field, the type of the equipment is recorded. For example, "GW" indicates a guide wire, "BDC" indicates a balloon dilatation catheter, and IVUS indicates a catheter for IVUS.

[0043] The variety of the equipment may include information regarding specifications such as the outer diameter and length of the equipment. For example, "*****3.0mm×10mm" recorded in the record of "BDC" indicates a balloon with a diameter of 3.0 mm and a length of 10 mm at the time of expansion. In the usage times field, the number of times the equipment has been inserted into the body during this operation is recorded.

[0044] An overview of the data recording method for the equipment information field will be described. The equipment used for catheter treatment is carried into the catheterization laboratory. The staff operates the barcode reader 15 to read the barcode attached to the packaging material of each piece of equipment. As described above, information such as the variety of the equipment is recorded in the barcode. The barcode reader 15 transmits the information read to the information processing device 20 via the network.

[0045] The control unit 21 creates a new record in the standby device DB66. The control unit 21 determines the type of equipment based on the variety recorded in the barcode read by the barcode reader 15. The control unit 21 may send the variety to the equipment manufacturer's website or the like and receive the type of equipment. The control unit 21 records the variety of the equipment in the variety field and the type of the equipment in the type field respectively. The control unit 21 sets the usage count field to the initial value "0".

[0046] Even when additional equipment is carried in during catheter treatment, a new record is similarly recorded in the standby device DB66. The update of the information recorded in the usage count field will be described later.

[0047] FIG. 6 is an explanatory diagram for explaining the record layout of the drug cumulative amount DB67. The drug cumulative amount DB67 is a DB that records the name of the drug and the cumulative dosage in association with each other. The drug cumulative amount DB67 has a drug field and a dosage field. The name of the drug is recorded in the drug field. The cumulative dosage of the drug is recorded in the dosage field. The initial value of the dosage field is "0".

[0048] FIGS. 7 and 8 are examples of the screens displayed by the terminal device 30. FIG. 7 shows an example of the screen displayed on the touch panel 35 of the terminal device 30 used by the staff in charge of supporting catheter treatment.

[0049] The screen shown in FIG. 7 includes a patient information column 51, a time series information column 52, a device button 53, a medication button 54, a vital button 55, and a patient status input column 56. The date and the current time are displayed in the upper right of the screen.

[0050] Patient information such as the patient's name, date of birth, and an overview of the medical history is displayed in the patient information column 51. When the user selects the patient information column 51 by a tap operation or the like, the control unit 31 accesses the electronic medical record system 19 to obtain more detailed patient information and displays it on the touch panel 35. Example items of the more detailed patient information are shown in Table 1.

[0051]

Table 1

[0052] Table 1 is an example of patient information that the control unit 31 of the terminal device 30 displays on the display unit 35 when the user selects the patient information column 51. The control unit 31 may display data recorded in the electronic medical record system 19, such as an X-ray CT image taken prior to catheter treatment, on the touch panel 35. The items of detailed information may be set to be different depending on the occupation of the user who uses each terminal device 30 and the like.

[0053] In the time series information column 52, various actions related to catheter treatment and the occurrence times of the actions are displayed in time series. Each control unit 31 displays the time series information column 52 in a predetermined format based on the log DB 65. Each control unit 31 sequentially accesses the log DB 65 to update the display of the time series information column 52. The latest state is displayed in the time series information column 52 of any terminal device 30.

[0054] For example, at 10:05, XXXX units of heparin have been administered. The fact that the doctor checked the patient's vital signs at the time of heparin administration is displayed by the vital sign confirmation mark 524. The fact that the patient is in a good state without feeling pain or the like at the time of heparin administration is displayed by the patient state mark 523 showing a smiling face.

[0055] At 10:12, coronary angiography (CAG: Coronary Angiography) is being performed using an X-ray angiography device. A thumbnail image 522 obtained by reducing the image taken by coronary angiography is displayed in the time series information column 52. The thumbnail image is an example of a reduced image.

[0056] Similarly, at 10:30, a tomographic image is taken by an IVUS catheter. A thumbnail image 522 obtained by shrinking the taken tomographic image is displayed in the time series information column 52. An input field 526 for receiving input of the following items is displayed at the lower end of the time series information column 52. The operation method of the information processing system 10 using the screen shown in FIG. 7 will be described later.

[0057] FIG. 8 is an example of a screen displayed by the terminal device 30. FIG. 8 shows an example of a screen displayed on the terminal device 30 used by the staff in the operation room. The control unit 21 may display the screen shown in FIG. 8 on a part of the display device 12.

[0058] The screen shown in FIG. 8 includes a patient information column 51, a time series information column 52, and a thumbnail column 527. The date and the current time are displayed at the upper right of the screen. Thumbnail images 522 of medical images taken by the imaging diagnosis device 14 or the catheter control device 13 are displayed in the thumbnail column 527. Thumbnails of images taken before the day of catheter treatment may be displayed in the thumbnail column 527. The thumbnail images 522 displayed in the thumbnail column 527 are larger than the thumbnail images 522 displayed in the time series information column 52.

[0059] FIG. 9 is a modified example of the time series information column 52. In FIG. 9, the time is displayed at the left end of the time series information column 52. In FIG. 9, the actions at each time are displayed separately in a column of major items such as "drug administration" and a column of minor items such as "XXXX units of heparin". In the column of minor items, the balloon expansion pressure or the stent expansion pressure when the balloon and the stent are expanded and the expansion time are displayed.

[0060] At 14:29, balloon dilation for expanding the stenosis using a balloon dilation catheter is being performed. During or before and after the balloon dilation, the location where the balloon dilation was performed is indicated by a thumbnail image 522 of a medical image taken by coronary angiography.

[0061] At 14:37, stent implantation is being performed. Stent implantation is carried out by expanding the stent from the inside at the target location. During or before and after stent expansion, a thumbnail image 522 of a medical image taken by coronary angiography shows the location where the stent expansion was performed.

[0062] On some terminal devices 30, a time series information column 52 shown in FIG. 7 may be displayed, and on other terminal devices 30, a time series information column 52 shown in FIG. 9 may be displayed. The control unit 21 of the information processing device 20 may cause a part of the display devices 12 to display the same screen as the screen displayed on any of the terminal devices 30.

[0063] When the user taps the thumbnail image 522 displayed on a predetermined terminal device 30, the control unit 31 of the terminal device 30 and the control unit 21 of the information processing device 20 cooperate to cause the display device 12 to display the original image of the thumbnail image 522 for which the tap operation was received. The user can view the image on a large screen.

[0064] Which display device 12 to display the original image on is set in advance. The control unit 21 of the information processing device 20 or the control unit 31 of the terminal device 30 may receive an instruction from the user to select an image diagnostic device 14 on which to display the original image.

[0065] When the user taps the thumbnail image 522 displayed on another terminal device 30, the control unit 31 acquires the original image from the log DB 65 or the electronic medical record system 19 and displays it on the touch panel 35. For example, an observer such as a student can view the original image without affecting the work of the staff actually performing the catheter treatment.

[0066] FIGS. 10 to 20 are examples of screens displayed by the terminal device 30. FIG. 10 shows an example of a screen displayed by the control unit 31 when a selection of the device button 53 is received. An equipment list column 539 is displayed next to the device button 53.

[0067] In the equipment list column 539, device icons 531 corresponding to each piece of equipment recorded in the standby device DB66 are displayed. It is desirable that the device icon 531 has a design that can easily identify the type and main specifications of the equipment.

[0068] Next to the device icon 531, a usage count column 532 is displayed. In the usage count column 532, the usage count recorded in the usage count field of the standby device DB66 is displayed.

[0069] The user can check the equipment that is ready for use in the equipment list column 539 and the number of times that equipment has been used. When a doctor selects and starts using the equipment, the staff drags and drops the device icon 531 corresponding to the selected equipment into the time series information column 52. The staff may perform a double-tap operation or the like instead of the drag-and-drop operation. The drag-and-drop operation and the double-tap operation are examples of selection operations for the user to select equipment from the equipment list column 539.

[0070] The user may input additional information regarding the use of the equipment by voice input or the like. The control unit 21 of the information processing apparatus 20 records the input additional information together with the information regarding the selected equipment in the log DB65.

[0071] The control unit 31 of the terminal device 30 transmits information regarding the device icon 531 that has been drag-and-dropped to the information processing apparatus 20. The control unit 21 of the information processing apparatus 20 creates a new record in the log DB65 and records the used equipment and the time when use started. The control unit 21 extracts the record corresponding to the equipment that has been drag-and-dropped from the standby device DB66 and adds 1 to the usage count field.

[0072] FIG. 11 shows an example of a screen displayed by the control unit 31 of the terminal device 30 when the selection of the dosing button 54 is accepted. Below the dosing button 54, a drug list column 549 is displayed. In the drug list column 549, drug buttons 541 corresponding to combinations of drugs used in the catheter examination and the dosage are displayed. When a doctor administers a drug, the staff taps the corresponding drug button 541. When a drug is administered to a patient using a pump or the like, information regarding the type or settings of the pump may be registered in the log DB 65.

[0073] The control unit 31 of the terminal device 30 transmits information regarding the tapped dosing button 54 to the information processing device 20. The control unit 21 extracts a record corresponding to the tapped drug from the drug cumulative amount DB 67 and adds the dosage to the dosage field. The control unit 21 creates a new record in the log DB 65 and records the administered drug, the dosage, and the cumulative dosage.

[0074] FIG. 12 shows an example of a screen displayed by the control unit 31 of the terminal device 30 when the selection of the vital button 55 is accepted. Below the vital button 55, a vital graph column 551 is displayed. In the vital graph column 551, a graph similar to the screen of the bedside monitor 16, for example, is displayed. The vital graph column 551 is an illustration of the patient's vital information.

[0075] When a doctor checks the vital state, the staff taps the vital graph column 551. The control unit 21 accepts the tap operation of the vital button 55 via the control unit 31 and records the vital in the vital field of the log DB 65.

[0076] The vital confirmation mark 524 in the time series information column 52 indicates that the vital is recorded in the vital field. For example, the staff in the operation room can also grasp that the doctor has checked the vital by the vital confirmation mark 524 and can pay attention to the patient's vital state without waiting for a direct instruction from the doctor.

[0077] The control unit 31 of the terminal device 30 may display the vital signs recorded in the vital field of the log DB 65 in the time series information column 52 instead of the vital confirmation mark 524. The control unit 21 may display a mark that can identify whether the vital signs recorded in the vital field are within the normal range or are abnormal values instead of the vital confirmation mark 524.

[0078] The attention points handwritten or the like on the vital information displayed on the display device 12 in the operation room or the terminal device 30 etc. may be similarly displayed in the vital graph column 551 of the terminal device 30 in the catheterization room. Changes in vital signs that the operating surgeon has not noticed can be pointed out from the operation room to arouse attention. Input from the operation room may be performed via a touch panel, or may be performed via a mouse or a keyboard.

[0079] FIG. 13 shows a screen of a modified example. When the vital button 55 is not selected, a vital value column 552 is displayed on the screen. The value of the vital signs acquired by the bedside monitor 16 is constantly displayed in the vital value column 552.

[0080] When a doctor checks the vital sign state, the staff taps the vital value column 552. The control unit 21 of the information processing device 20 receives the tap operation of the vital button 55 via the control unit 31 of the terminal device 30 and records the vital signs in the vital field of the log DB 65.

[0081] Nurses etc. listen to whether the patient has pain etc. to judge the patient's state. The nurse may observe the patient to judge the patient's state. The nurse taps an appropriate patient state icon from among a plurality of patient state icons displayed in the patient state input column 56.

[0082] The control unit 21 of the information processing device 20 receives the tap operation of the patient state icon via the control unit 31 of the terminal device 30 and records the code corresponding to the selected patient state icon in the patient state field of the log DB 65. The patient state mark 523 in the time series information column 52 indicates the patient state.

[0083] FIG. 14 shows a screen when a doctor or staff inputs a voice memo. The fact that a voice memo was input at 10:34 is indicated by the microphone icon. When a voice memo is input, the control unit 21 creates a new record in the log DB 65 and records the voice memo data in the file field.

[0084] The strategic mark 525 will be described. For example, when a pullback operation is performed using IVUS, the catheter control device 13 generates longitudinal and cross-sectional images. The catheter control device 13 may have a function of detecting a lesion based on the tomographic image and a function of displaying the basis regarding the detected lesion.

[0085] The doctor formulates a treatment strategy based on the tomographic images generated by the catheter control device 13 and information such as past examinations. The doctor may formulate a treatment strategy when performing coronary angiography. The formulated treatment strategy and the strategic file summarizing data such as the referenced images are recorded in the strategic field of the log DB 65. The strategic mark 525 in the time series information column 52 means that a strategic file is recorded in the log DB 65. The strategic file may be recorded in the electronic medical record system 19.

[0086] When the user taps the strategic mark 525, the strategic file is displayed on the touch panel 35 or the display device 12. The user can confirm the doctor's judgment and its basis and make preparations for the next operation.

[0087] A specific example will be described. FIG. 15 shows an example of a strategic file when a doctor formulates a treatment strategy to perform "stent implantation". The catheter control device 13 detects "dissection" based on the tomographic image and outputs the basis for detecting "dissection" by Grad-CAM (Gradient-weighted Class Activation Mapping). The function of automatically detecting a lesion from an image will be described in Embodiment 3.

[0088] Based on the images and the like generated by the catheter control device 13, the doctor determines a treatment strategy to perform "stent placement" and records it in the strategy file. The treatment strategy and the images referred to by the doctor are summarized in the strategy file.

[0089] Note that the catheter control device 13 does not necessarily have to have a function of detecting lesions. The doctor determines that there is a "dissection" based on the tomogram and decides on a policy of performing "stent placement". Based on the doctor's instructions, the staff marks the site that is the basis for the determination.

[0090] FIG. 16 shows an example of a strategy file when the doctor determines the dimensions of the stent to be implanted. Based on the doctor's instructions, the staff measures the length of the stenosis and the blood vessel diameter from the tomogram. The doctor selects the stent to be used based on the measurement results and records it in the strategy file.

[0091] FIG. 17 shows an example of a screen used when the doctor in charge of the catheter treatment records supplementary information in the log DB65 after the catheter treatment is completed. The doctor displays the screen shown in FIG. 17 on the screen of the terminal device 30 or a personal computer. The patient information column 51 and the time series information column 52 are displayed. FIG. 17 shows an example where the patient status mark 523 is not displayed in the time series information column 52. The patient status mark 523 may be displayed in the time series information column 52.

[0092] The name of the user logged on to the information processing system 10 is displayed in the user name column 529 at the bottom of the screen. At the top of the screen, a modification button 571, an approval button 572, and an output button 573 are displayed.

[0093] The doctor records the supplementary information that could not be recorded during the catheter treatment in the additional comment column 528. The doctor can enter any comment in the additional comment column 528 by keyboard operation or voice input. The additional comment column 528 may accept the selection of typical comment sentences, for example, by means of a pull-down menu or the like.

[0094] After the input is completed, the doctor selects the correction button 571. The comment entered in the additional comment field 528 is reflected in the log DB 65. When the doctor selects the output button 573, the screen shown in FIG. 17 is printed. When a predetermined approver logs on and selects the approval button 572, the data recorded in the log DB 65 is recorded in the electronic medical record system 19.

[0095] FIG. 18 shows an example of a screen used by a nurse to record supplementary information in the log DB 65 after the catheter treatment is completed. Instead of the approval button 572 in FIG. 17, a confirmation button 576 is displayed. A patient status mark 523 is displayed in the time series information field 52.

[0096] The nurse can enter any comment in the additional comment field 528 by keyboard operation or voice input. After the input is completed, the nurse selects the correction button 571. The comment entered in the additional comment field 528 is temporarily recorded in the auxiliary storage device 23. When the doctor in charge of the catheter treatment selects the confirmation button 576, the comment entered in the additional comment field 528 is reflected in the log DB 65.

[0097] Clinical laboratory technicians, clinical engineering technicians, etc. also enter additional comments based on their respective assigned tasks. As described above, the records made by the staff of each occupation can be aggregated in the log DB 65. Since the staff of each occupation do not need to enter the same information repeatedly, an information processing system 10 that can reduce the burden on the staff can be provided.

[0098] Since the staff of each occupation can check the information recorded in the log DB 65 and record additional information, an information processing system 10 that can record a catheter treatment log based on a multi-faceted perspective can be provided.

[0099] FIG. 19 and FIG. 20 show examples of screens for performing workflow analysis related to catheter treatment based on a catheter treatment log. The screen shown in FIG. 19 is displayed on a terminal device 30 or a personal computer or the like connected to the HIS. On the screens shown in FIGS. 19 and 20, an analysis result column 578 and a comment input button 575 are displayed.

[0100] The analysis result column 578 in FIG. 19 includes a time series diagram 58 and a legend column 589. The time series diagram 58 shows the passage of time from left to right, and the operations performed at each time are shown by hatching. For convenience of illustration, some types of hatching overlap. The processes indicated by each hatching are displayed in characters in the legend column 589.

[0101] Below the time series diagram 58, the time of X-ray irradiation is displayed based on the radiation irradiation log recorded in the imaging diagnostic apparatus 14. Since the method of creating the time series diagram 58 based on the log data is known, the description thereof is omitted.

[0102] With the screen of FIG. 19, the user can confirm how much time was required for each operation. The user can input a comment by selecting the comment input button 575. The input comment may be displayed, for example, at the bottom of the screen.

[0103] The analysis result column 578 in FIG. 20 includes a first time series diagram 581 and a second time series diagram 582. The first time series diagram 581 shows the time series of an actual case. The second time series diagram 582 shows the time series of the standard procedure time for a similar treatment technique. The standard procedure time is statistically created based on, for example, past cases.

[0104] With the screen of FIG. 20, the user can compare the standard procedure time with the time required for the actual procedure. The user can input a comment considering, for example, the operations that took longer than the standard procedure time.

[0105] The workflow analysis results may be displayed in any format, such as a table format or a graph format. By performing workflow analysis using the catheter treatment logs recorded in the log DB65, for example, methods for shortening the procedure time can be considered. By shortening the procedure time, an information processing system 10 that reduces the burden on the patient's body can be provided.

[0106] Figures 21 to 26 are flowcharts for explaining the processing flow of the program. Figure 21 shows a program for recording equipment in the standby device DB66.

[0107] As described above, the equipment used for catheter treatment is carried into the catheterization laboratory. The staff operates the barcode reader 15 to read the barcode attached to the packaging material of each piece of equipment. The barcode reader 15 transmits the read barcode information to the information processing device 20. Thereafter, the program shown in Figure 21 is executed.

[0108] The control unit 21 of the information processing device 20 receives the barcode information transmitted from the barcode reader 15 (step S501). The control unit 21 acquires equipment information regarding the type and variety of the equipment based on the barcode information (step S502). The equipment information is included in the barcode information, for example. The equipment information may be acquired by accessing the URL indicated in the barcode information.

[0109] The control unit 21 creates a new record in the standby device DB66. The control unit 21 records the type and variety acquired in step S502 in the type field and the variety field. The control unit 21 records 0, which is the initial value, in the usage count field. Thus, the carried-in equipment is recorded in the standby device DB66 (step S503). The control unit 21 ends the process.

[0110] Note that the serial number read from the barcode and the time when the barcode was read may be recorded in the standby device DB66.

[0111] FIG. 22 shows a program that is activated when a medical image is taken by the catheter control device 13 or the imaging diagnostic device 14.

[0112] When the user performs an operation to take a medical image, the catheter control device 13 or the imaging diagnostic device 14 transmits the taken image data or link information indicating the storage destination of the image data to the information processing device 20. Thereafter, the program shown in FIG. 22 is executed.

[0113] The control unit 21 of the information processing device 20 acquires the taken image data (step S511). The control unit 21 reduces the image acquired in step S511 to create a thumbnail image (step S512). The control unit 21 may acquire the thumbnail image created by the catheter control device 13 or the imaging diagnostic device 14.

[0114] The control unit 21 creates a new record in the log DB 65. The control unit 21 records the time when the image was taken in the time field, information about the device that took the image in the action field, and the files of the image acquired in step S511 and the thumbnail image created in step S512 in the file field (step S513).

[0115] Before and after the image is taken, if the patient's condition is input, the control unit 21 records the patient's condition in the patient condition field. Similarly, if vital signs are confirmed, the control unit 21 records the vital signs in the vital signs field. If a treatment strategy is formulated based on the taken image, the control unit 21 records the file containing the treatment strategy in the strategy field. Thereafter, the control unit 21 ends the process.

[0116] FIG. 23 shows a program that is activated when a tap on the thumbnail image 522 displayed on the touch panel 35 is received. The control unit 31 of the terminal device 30 determines whether the terminal device 30 is set to display the original image on the display device 12 when a tap on the thumbnail image 522 is received (step S621).

[0117] If it is determined that the setting is made (YES in step S621), the control unit 31 of the terminal device 30 transmits information regarding the tapped thumbnail image 522 to the information processing device 20 (step S622). The information regarding the thumbnail image 522 is, for example, the time when the thumbnail image was taken. The control unit 31 ends the process.

[0118] The control unit 21 of the information processing device 20 receives information regarding the tapped thumbnail image (step S521). The control unit 21 searches the log DB 65 to acquire the original image (step S522). The control unit 21 transmits the original image to the predetermined display device 12 (step S523). As a result, the original image is displayed on the display device 12.

[0119] If it is determined that the original image is not set to be displayed on the display device 12 (NO in step S621), the control unit 31 of the terminal device 30 transmits information regarding the tapped thumbnail image 522 to the information processing device 20 (step S623).

[0120] The control unit 21 of the information processing device 20 receives information regarding the tapped thumbnail image (step S524). The control unit 21 searches the log DB 65 to acquire the original image (step S525). The control unit 21 transmits the original image to the terminal device 30 that transmitted the information regarding the thumbnail image 522 (step S526).

[0121] The control unit 31 of the terminal device 30 receives the original image (step S624). The control unit 31 displays the original image on the touch panel 35 (step S625). The control unit 31 ends the process.

[0122] FIG. 24 shows a program that is activated when the device button 53 is selected by the user. The control unit 31 of the terminal device 30 transmits information indicating that the device button 53 has been selected to the information processing device 20 (step S631). The control unit 21 of the information processing device 20 receives the information (step S531). The control unit 21 transmits the standby device DB66 to the terminal device 30 (step S532).

[0123] The control unit 31 of the terminal device 30 receives the standby device DB66 (step S632). The control unit 31 displays the equipment list column 539 as described with reference to FIG. 10 (step S633). The control unit 31 accepts a selection operation such as a drag-and-drop operation of the device icon 531 (step S634). The control unit 31 transmits information about the equipment for which the selection operation has been received to the information processing device 20 (step S635).

[0124] The control unit 21 of the information processing device 20 receives information about the equipment for which the selection operation has been received (step S533). The control unit 21 updates the standby device DB66 (step S534). Specifically, the control unit 21 extracts the record related to the equipment received in step S533 from the standby device DB66. The control unit 21 adds 1 to the usage count field.

[0125] The control unit 21 creates a new record in the log DB65 (step S535). The control unit 21 records the time when the selection operation was received in the time field and the information about the equipment for which the selection operation was received in the action field, respectively.

[0126] The control unit 21 determines whether the selected equipment is the image diagnostic catheter 132 (step S536). If it is determined that it is not the image diagnostic catheter 132 (NO in step S536), the control unit 21 enters a standby state waiting for the next input. If it is determined that it is the image diagnostic catheter 132 (YES in step S536), the control unit 21 determines whether the catheter control device 13 has recorded an image (step S537).

[0127] When it is determined that no image has been recorded (NO in step S537), the control unit 21 enters a standby state waiting for the next input. When it is determined that an image has been recorded (YES in step S537), the control unit 21 activates a subroutine for image recording (step S538). The subroutine for image recording is a program that performs the same processing as the program described using FIG. 22. The difference is that in step S513, a new record is not created, but the file is recorded in the file field of the record created in step S535.

[0128] The control unit 31 of the terminal device 30 determines whether to end the display of the equipment list column 539 (step S636). For example, when the user gives an instruction to end by a double-tap operation or the like, the control unit 31 determines to end the display of the equipment list column 539.

[0129] When it is determined not to end (NO in step S636), the control unit 31 returns to step S634. When it is determined to end (YES in step S636), the control unit 31 makes the equipment list column 539 non-displayed (step S637). The control unit 31 ends the process.

[0130] FIG. 25 shows a program that is activated when the dosing button 54 is selected by the user. The control unit 31 of the terminal device 30 displays the drug list column 549 as described using FIG. 11 (step S641). The control unit 31 accepts the selection of the drug button 541 (step S642). The control unit 31 transmits the type and dosage of the selected drug to the information processing device 20 (step S643).

[0131] The control unit 21 of the information processing device 20 receives the type and dosage of the selected drug (step S541). The control unit 21 updates the drug cumulative amount DB67 (step S542). Specifically, the control unit 21 extracts the record related to the drug received in step S541 from the drug cumulative amount DB67. The control unit 31 adds the dosage received in step S541 to the dosage field.

[0132] The control unit 21 creates a new record in the log DB 65 (step S543). The control unit 21 records the time when the selection operation was received in the time field and the information regarding the drug for which the selection operation was received in the action field, respectively.

[0133] The control unit 31 of the terminal device 30 determines whether to end the display of the drug list column 549 (step S644). For example, when the user instructs termination by a double-tap operation or the like, the control unit 31 determines to end the display of the drug list column 549.

[0134] If it is determined not to end (NO in step S644), the control unit 31 returns to step S642. If it is determined to end (YES in step S644), the control unit 31 makes the drug list column 549 non-displayed (step S645). The control unit 31 ends the process.

[0135] FIG. 26 shows a program that is activated when the vital button 55 is selected by the user. The control unit 31 of the terminal device 30 transmits the start time of the transmission instruction of the vital data to the bedside monitor 16 (step S651).

[0136] The bedside monitor 16 receives the transmission start instruction (step S751). The bedside monitor 16 starts transmitting the vital data to the terminal device 30 (step S752). The vital data is, for example, image data showing the screen displayed by the bedside monitor 16. The vital data may be numerical data indicating the value of each item displayed by the bedside monitor 16. After step S752, the bedside monitor 16 sequentially transmits the vital data to the terminal device 30.

[0137] The control unit 31 of the terminal device 30 receives vital data (step S652). The control unit 31 displays the vital graph column 551 as described with reference to FIG. 12 (step S653). The control unit 31 determines whether or not a tap operation or the like on the vital graph column 551 has been received (step S654).

[0138] If it is determined that the operation has been received (YES in step S654), the control unit 31 transmits the vital that was displayed on the touch panel 35 at the time of receiving the tap operation to the information processing device 20 (step S655). The control unit 21 receives the vital (step S551).

[0139] The control unit 21 of the information processing device 20 records the received vital in the log DB65 (step S552). For example, when a vital is confirmed before and after an action such as drug administration, the control unit 21 records the vital in the vital field of the record in which the action is recorded. When no other action is being performed, the control unit 21 creates a new record in the log DB65 and records the vital.

[0140] If it is determined that no operation has been received (NO in step S654), or after the end of step S655, the control unit 31 of the terminal device 30 determines whether or not to end the display of the vital graph column 551 (step S656). For example, when the user instructs termination by a double-tap operation or the like, the control unit 31 determines to end the display of the vital graph column 551.

[0141] If it is determined not to end (NO in step S656), the control unit 31 returns to step S652. If it is determined to end (YES in step S656), the control unit 31 transmits an instruction to end the transmission of the vital to the bedside monitor 16 (step S657). The control unit 21 makes the vital graph column 551 non-displayed (step S658). The control unit 21 ends the process.

[0142] The bedside monitor 16 receives a transmission end instruction (step S753). The bedside monitor 16 ends the transmission of vital data to the terminal device 30 (step S754).

[0143] The control unit 31 of the terminal device 30 may accept an operation by voice input via a microphone built in the terminal device 30 or the microphone 17 instead of the touch panel 35. Doctors and nurses during the cleaning operation can operate the terminal device 30 by themselves without touching it.

[0144] According to the present embodiment, an information processing method or the like for assisting information sharing among the staff participating in the treatment can be provided. According to the present embodiment, an information processing system 10 that can record a catheter treatment log and share it in real time can be provided.

[0145] According to the present embodiment, an information processing system 10 that can easily record the use of equipment and the administration of drugs in the catheter treatment log can be provided. According to the present embodiment, an information processing system 10 that can easily record the patient's condition can be provided. According to the present embodiment, an information processing system 10 that can easily record that a doctor or the like has paid attention to vital signs can be provided.

[0146] According to the present embodiment, an information processing system 10 that can confirm the equipment ready for use and the number of times of use based on the screen described with reference to FIG. 10 can be provided.

[0147] According to the present embodiment, an information processing system 10 that records the treatment strategy determined by a doctor based on new information obtained using the catheter control device 13, the imaging diagnostic device 14, etc. in the catheter treatment log can be provided.

[0148] According to the present embodiment, an information processing system 10 that allows the staff and observers participating in the catheter treatment to view the treatment strategy formulated by the doctor and the images already taken in real time can be provided.

[0149] According to the present embodiment, it is possible to provide an information processing system 10 that allows staff of each profession to add comments, etc. to the catheter treatment log after the catheter treatment is completed. According to the present embodiment, it is possible to provide an information processing system 10 that automatically records a catheter treatment log that can be utilized for workflow analysis.

[0150] [Embodiment 2] This embodiment relates to an information processing system 10 that displays a thumbnail image 522 of an object arrangement image generated based on an original image. Description of parts common to the first embodiment will be omitted.

[0151] Fig. 27 is an example of a screen displayed by the terminal device 30 of the second embodiment. The screen shown in Fig. 27 is a screen displayed in place of the screen described using Fig. 7. A thumbnail image 522 of an object arrangement image is displayed between a thumbnail image 522 of an image captured by IVUS at 10:30 and a strategy mark 525.

[0152] FIG. 28 is an explanatory diagram for explaining the configuration of the first model 61. The first model 61 is a model that receives an original image and outputs an object arrangement image in which the types of multiple objects included in the original image are associated with the ranges of each object and mapped. The first model 61 is generated by machine learning. The object arrangement image is an example of a converted image obtained by converting the original image. The thumbnail of the object arrangement image is an example of a reduced converted image obtained by reducing the converted image.

[0153] In the object arrangement image shown in FIG. 28, vertical hatching indicates a cross section of the diagnostic imaging catheter 132, right-down hatching indicates the inside of a hollow organ, and thick left-down hatching indicates the epicardium.

[0154] The hatching in FIG. 28 schematically shows that each object is colored differently. The coloring of the objects is an example of a method of distinguishing and displaying each object. Each object may be displayed so as to be distinguishable from other objects by any means such as surrounding the outer edge of each object.

[0155] In FIG. 28, a cross-sectional image is displayed in a portion of the object arrangement image other than the objects listed above. The control unit 21 may classify and display the entire surface of the object arrangement image by classifying it into some object such as "lumen organ wall", "plaque", "calcification", "guide wire", etc.

[0156] The first model 61 is, for example, a semantic segmentation model, and includes an input layer, a neural network, and an output layer. The neural network has a U-Net structure that realizes semantic segmentation. The U-Net structure is composed of a multi-layer encoder layer and a multi-layer decoder layer connected behind it. By semantic segmentation, a label indicating the type of object is assigned to each pixel constituting the input image.

[0157] The first model 61 is generated for each apparatus that captures the original image. The first model 61 may be further generated for each part to be imaged. The first model 61 is recorded in the auxiliary storage device 23. The first model 61 may be recorded in a large-capacity device or the like connected to the information processing system 10.

[0158] The object arrangement image may be generated by the catheter control device 13 or the imaging diagnostic device 14 and output together with the original image.

[0159] Returning to FIG. 27, the description will be continued. When the user taps the thumbnail image of the object arrangement image, the object arrangement image of the original size is displayed on the display device 12 or the touch panel 35.

[0160] FIG. 29 is a flowchart for explaining the processing flow of the program according to Embodiment 2. The program in FIG. 29 is executed instead of the program described with reference to FIG. 22 when an image is captured by the catheter control device 13 or the imaging diagnostic device 14.

[0161] When the user performs an operation to capture an image, the catheter control device 13 and the imaging diagnostic device 14 transmit the captured image data or link information indicating the storage destination of the image data to the information processing device 20. Thereafter, the program shown in FIG. 29 is executed.

[0162] The control unit 21 acquires the captured image data (step S561). The control unit 21 reduces the image acquired in step S561 to create a thumbnail image (step S562). When the catheter control device 13 or the imaging diagnostic device 14 generates an object placement image, a thumbnail image of the object placement image is also created in step S562. The control unit 21 may acquire the thumbnail image created by the catheter control device 13 or the imaging diagnostic device 14.

[0163] The control unit 21 determines whether to perform conversion to an object placement image (step S563). Specifically, when the first model 61 corresponding to the conversion of the image data received in step S561 exists, the control unit 21 determines to perform the conversion.

[0164] When it is determined not to perform the conversion (NO in step S563), the control unit 21 creates a new record in the log DB 65. The control unit 21 records the time when the image was captured in the time field, information about the device that captured the image in the action field, and the files of the image acquired in step S561 and the thumbnail image created in step S562 in the file field (step S564). The control unit 21 ends the process.

[0165] When it is determined to perform conversion (YES in step S563), the control unit 21 selects the first model 61 corresponding to the image data received in step S561 (step S565). The control unit 21 inputs the image data acquired in step S561 into the first model 61 selected in step S565 to obtain an output object arrangement image (step S566). The control unit 21 reduces the acquired object arrangement image to create a thumbnail image (step S567).

[0166] The control unit 21 creates a new record in the log DB65. The control unit 21 records the time when the image was taken in the time field, information about the device that took the image in the action field, and the files of the image acquired in step S561, the thumbnail image created in step S562, the object arrangement image acquired in step S566, and the thumbnail image created in step S567 in the file field (step S568). The control unit 21 ends the process.

[0167] The first model 61 is not limited to a model that converts the original image into an object arrangement image. For example, it may be a model that converts the original image into an image obtained by performing any process such as edge enhancement or contrast conversion.

[0168] According to the present embodiment, an information processing system 10 that displays an object arrangement image can be provided. By referring to the object arrangement image, the user can easily read the image taken by the catheter control device 13 or the image diagnostic device 14.

[0169] [Embodiment 3] The present embodiment relates to an information processing system 10 that automatically detects lesions and creates treatment strategies. Descriptions of parts common to Embodiment 1 are omitted.

[0170] FIG. 30 is an explanatory diagram for explaining the configuration of the second model 62. The second model 62 is a model that receives an input of a medical image and outputs the type of lesion. The second model 62 may be a model that receives inputs of a plurality of medical images and outputs the type of lesion. The plurality of medical images are, for example, images taken using the same device at different timings or under different imaging conditions. The plurality of medical images may be images taken using different devices.

[0171] The second model 62 is generated for each device that captured the medical image and each site where the medical image was captured, and is recorded in the auxiliary storage device 23 or an external mass storage device. The second model 62 is a learning model generated using an image classification algorithm such as, for example, CNN (Convolutional Neural Networks) or YOLO (You only look once).

[0172] FIG. 31 is an explanatory diagram for explaining the configuration of the third model 63. The third model 63 is a model that receives a medical image captured by the catheter control device 13 or the imaging diagnosis device 14 and outputs a treatment strategy.

[0173] The treatment strategy output by the third model 63 is, for example, the type of treatment procedure such as "stent placement" described using FIG. 15. The treatment strategy output by the third model 63 may be the specifications of the equipment used in the treatment procedure, such as the dimensions of the stent described using FIG. 16.

[0174] The third model 63 is generated for each device that captured the medical image, the site where the medical image was captured, the lesion to be treated, and the timing when the medical image was captured, and is recorded in the auxiliary storage device 23 or an external mass storage device. The timing when the medical image was captured indicates a timing during a series of catheter treatments, such as immediately after the start of catheter treatment, after balloon dilation, after stent placement, etc. The third model 63 is a model generated using an algorithm such as CNN or YOLO.

[0175] The third model 63 may be a model that, in addition to medical images, accepts the input of patient information, lesion sites, or catheter treatment logs recorded before the medical images are taken, and outputs a treatment strategy.

[0176] FIG. 32 is a flowchart for explaining the processing flow of the program according to the third embodiment. The program in FIG. 32 is executed when a doctor instructs the formulation of a treatment strategy. When instructing the formulation of a treatment strategy, the doctor designates the medical images to be used.

[0177] The control unit 21 of the information processing apparatus 20 selects the second model 62 to be used based on the apparatus that captured the medical images designated by the doctor and the site where the medical images were captured (step S571). The control unit 21 inputs the medical images designated by the doctor into the second model 62 selected in step S571 to obtain the type of lesion (step S572). The control unit 21 obtains the basis region related to the type of lesion obtained in step S572 by the algorithm of Grad-CAM (Gradient-weighted Class Activation Mapping) (step S573).

[0178] For example, in the treatment strategy described using FIG. 15, "with dissociation" under the left cross-sectional image is the type of lesion obtained in step S572, and the heat map displayed on the right cross-sectional image is the basis region obtained in step S572.

[0179] The control unit 21 selects the third model 63 to be used based on the apparatus that captured the medical images designated by the doctor, the site where the medical images were captured, the lesion to be treated, and the time when the medical images were captured (step S574). The control unit 21 inputs the medical images designated by the doctor into the third model selected in step S574 to obtain a treatment strategy (step S575). For example, in the treatment strategy described using FIG. 15, "stent placement" displayed at the bottom is the treatment strategy obtained in step S575.

[0180] The control unit 21 records the acquired treatment strategy in the log DB 65 (step S576). The control unit 21 ends the process.

[0181] According to the present embodiment, an information processing system 10 that determines the type of lesion and formulates a treatment strategy can be provided.

[0182] The second model 62 may be a model generated using an algorithm capable of detecting both the type and position of an object such as R-CNN (Regions with or Mask R-CNN). An information processing system 10 that displays the position of a lesion based on the output of the second model 62 can be provided without using Grad CAM.

[0183] Instead of using the second model 62, the control unit 21 may accept the type of lesion input by the user. Instead of using the third model 63, the control unit 21 may accept the treatment strategy input by the user.

[0184] [Embodiment 4] This embodiment relates to a method for generating the first model described with reference to FIG. 28. Descriptions of parts common to Embodiment 2 are omitted.

[0185] FIG. 33 is an explanatory diagram for explaining the record layout of the training DB of Embodiment 4. The training DB is a database that records by associating an input and a correct label, and is used for training a model by machine learning. The training DB has an input data field and an output data field.

[0186] The input data field stores a medical image. The output data field stores a painted image in which experts such as doctors have painted different objects in different colors. That is, the output data field stores the object corresponding to each pixel constituting the medical image. A large number of combinations of medical images and painted images painted by experts are stored in the training DB. Note that the training DB is prepared for each combination of the device for taking the medical image and the part to be imaged.

[0187] FIG. 34 is a flowchart for explaining the processing flow of the program according to Embodiment 4. A case of performing machine learning of the first model 61 using the information processing apparatus 20 will be described as an example.

[0188] The program in FIG. 34 may be executed on hardware separate from the information processing apparatus 20, and the first model 61 for which the machine learning has been completed may be copied to the auxiliary storage device 23 via the network. The first model 61 learned on one piece of hardware can be used in a plurality of information processing apparatuses 20.

[0189] Prior to the execution of the program in FIG. 34, an unlearned model such as a U-Net structure for realizing semantic segmentation is prepared. As described above, the U-Net structure is composed of a multi-layer encoder layer and a multi-layer decoder layer connected thereto. By the program in FIG. 34, each parameter of the prepared model is adjusted to perform machine learning.

[0190] The control unit 21 of the information processing apparatus 20 acquires a training record used for one-epoch training from the training DB (step S811). When input data is input to the input layer of the model, the control unit 21 adjusts the parameters of the model so that output data is output from the output layer (step S812).

[0191] The control unit 21 determines whether to end the process (step S813). For example, when the control unit 21 finishes learning for a predetermined number of epochs, it determines to end the process. The control unit 21 may also obtain test data from the training DB, input it into the model during machine learning, and determine to end the process when an output with a predetermined accuracy is obtained.

[0192] If it is determined not to end the process (NO in step S813), the control unit 21 returns to step S811. If it is determined to end the process (YES in step S813), the control unit 21 records the parameters of the learned model in the auxiliary storage device 23 (step S814). Thereafter, the control unit 21 ends the process. Through the above process, a learned model is generated.

[0193] According to the present embodiment, the first model 61 can be generated by machine learning.

[0194] [Embodiment 5] This embodiment relates to a method for generating the second model 62 described with reference to FIG. 30. Descriptions of parts common to Embodiment 3 are omitted.

[0195] FIG. 35 is an explanatory diagram for explaining the record layout of the training DB of Embodiment 5. The training DB has an input data field and an output data field.

[0196] Medical images are recorded in the input data field. In the output data field, lesions determined by experts such as doctors after reading the medical images are recorded. A large number of combinations of medical images and lesions determined by experts etc. are recorded in the training DB. Note that the training DB is prepared for each combination of the device for taking medical images and the part to be imaged.

[0197] The outline of the method for generating the second model 62 of the present embodiment will be described. Prior to the generation of the second model 62, for example, an unlearned model having a CNN structure including repetitions of a convolutional layer and a pooling layer, a fully-connected layer, and a softmax layer is prepared. Except that the structure of the prepared model is different, a learned second model 62 is generated by the same processing flow as the processing described using the flowchart of FIG. 34.

[0198] According to the present embodiment, the second model 62 can be generated by machine learning.

[0199] [Embodiment 6] The present embodiment relates to a method for generating the third model 63 described with reference to FIG. 31. Descriptions of parts common to Embodiment 5 will be omitted.

[0200] FIG. 36 is an explanatory diagram for explaining the record layout of the training DB of Embodiment 6. The training DB has an input data field and an output data field. A medical image is recorded in the input data field. In the output data field, treatment strategies such as treatment methods or specifications of equipment to be used, which are determined by experts with sufficient experience in catheter treatment, are recorded. A large number of combinations of medical images and treatment strategies are recorded in the training DB. Note that the training DB is prepared for each combination of the device that captured the medical image, the site where the medical image was captured, the lesion to be treated, and the time when the medical image was captured.

[0201] The outline of the method for generating the third model 63 of the present embodiment will be described. Prior to the generation of the third model 63, for example, an unlearned model having a CNN structure including repetitions of a convolutional layer and a pooling layer, a fully-connected layer, and a softmax layer is prepared. Except that the structure of the prepared model is different, a learned third model 63 is generated by the same processing flow as the processing described using the flowchart of FIG. 34.

[0202] According to the present embodiment, the third model 63 can be generated by machine learning.

[0203] [Embodiment 7] FIG. 37 is a functional block diagram of the information processing system 10 according to Embodiment 7. The information processing system 10 includes a patient information acquisition unit 81, an action acquisition unit 82, and a display unit 83.

[0204] The patient information acquisition unit 81 acquires patient information regarding a patient who undergoes catheter treatment. The action acquisition unit 82 sequentially acquires actions related to catheter treatment for the patient in association with the time when the actions are performed. The display unit 83 causes the first display device to display the patient information acquired by the patient information acquisition unit 81 and time-series information in which the actions acquired by the action acquisition unit 82 are arranged in time series.

[0205] [Embodiment 8] FIG. 38 is an explanatory diagram for explaining the configuration of the information processing apparatus 20 according to Embodiment 8. This embodiment relates to a form in which the information processing apparatus 20 of this embodiment is realized by operating in combination a general-purpose computer 90 and a program 97. For parts common to Embodiment 1, the description is omitted.

[0206] The computer 90 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, a tablet, a smartphone, or a server computer.

[0207] The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. Further, the control unit 21 may read the program 97 stored in a semiconductor memory 98 such as a flash memory mounted in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

[0208] Among the programs 97, the part executed by the computer 90 is installed as a control program of the computer 90, loaded into the main storage device 22, and executed. As a result, the computer 90 functions as the information processing device 20 described above. The part of the program 97 executed by the terminal device 30 is transmitted via a network and installed as a control program of the terminal device 30. As a result, the terminal device 30 cooperates with the information processing device 20 to perform the functions described above.

[0209] The computer 90 is a general-purpose personal computer, tablet, smartphone, mainframe computer, virtual machine operating on a mainframe computer, cloud computing system, or quantum computer. The computer 90 may be a plurality of personal computers or the like that perform distributed processing.

[0210] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above meaning but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Reference Numerals

[0211] 10 Information processing system 11 Camera 12 Display device 13 Catheter control device 131 MDU 132 Image diagnostic catheter 14 Image diagnostic device 15 Barcode reader 16 Bedside monitor 17 Microphone 18 Examination table 19 Electronic medical record system 20 Information processing device 21 Control unit 22 Main memory device 23 Auxiliary memory device 24 Communication unit 29 Reading unit 30 Terminal device 31 Control unit 32 Main memory device 33 Auxiliary memory device 34 Communication unit 35 Touch panel 351 Display unit 352 Input unit 51 Patient information column 52 Time-series information column 522 Thumbnail image 523 Patient status mark 524 Vital sign confirmation mark 525 Strategy mark 526 Input column 527 Thumbnail column 528 Additional comment column 529 User name column 53 Device button 531 Device icon 532 Usage times column 539 Equipment list column 54 Medication button 541 Drug button 549 Drug list column 55 Vital sign button 551 Vital sign graph column 552 Vital sign numerical value column 56 Patient status input column 571 Modify button 572 Approve button 573 Output button 575 Comment input button 576 Confirm button 578 Analysis result column 58 Time-series diagram 581 First time-series diagram 582 Second time-series diagram 589 Legend column 61 First model 62 Second model 63 Third model 65 Log DB 66 Standby Device DB 67 Drug Accumulation Amount DB 81 Patient Information Acquisition Unit 82 Action Acquisition Unit 83 Display Unit 90 Computer 96 Portable Recording Medium 97 Program 98 Semiconductor Memory

Claims

1. Obtain patient information regarding a patient undergoing catheter treatment, sequentially obtain an action related to the catheter treatment for the patient and the time at which the action was performed in association with each other, cause a first display device to display the obtained patient information and time-series information in which the obtained actions are arranged in time series, when the action is an action of obtaining an image, input the obtained image into a first model that outputs a converted image obtained by converting the image when the image is input, and obtain the converted image output from the first model or a reduced converted image obtained by reducing the converted image, cause the time-series information to display a reduced image obtained by reducing the image in association with the time at which the image was obtained, and the converted image or the reduced converted image, An information processing method for causing a computer to execute the process.

2. When receiving a selection of the reduced image, cause a second display device to display the image corresponding to the reduced image The information processing method according to claim 1.

3. Display an equipment list, the action is an action of receiving a selection of equipment from the displayed equipment list, cause the first display device to display the selected equipment and the time at which the selection was received The information processing method according to claim 1 or claim 2.

4. The equipment list is displayed by an array of icons indicating each piece of equipment The information processing method according to claim 3.

5. Obtain the number of times each piece of equipment included in the equipment list is used from a database that records the association between the equipment and the number of times of use, display the number of times each piece of equipment was used in the equipment list The information processing method according to claim 3 or claim 4.

6. Display a drug list, the action is an action of receiving a selection of a drug from the displayed drug list, cause the first display device to display the selected drug and the time at which the selection was received The information processing method according to any one of claims 1 to 5.

7. When the action is an action of performing stent expansion, display the stent expansion pressure, expansion time, and expansion location in association with the time at which the stent expansion was performed The information processing method according to any one of claims 1 to 6.

8. When the action is an action for balloon expansion, the balloon expansion pressure, expansion time, and expansion location are displayed in association with the time when the balloon expansion was performed. The information processing method according to any one of claims 1 to 7.

9. Obtain the vital information of the patient, Cause the obtained vital information to be displayed on the first display device The information processing method according to any one of claims 1 to 8.

10. Record the time-series information associating the action and the time. The information processing method according to any one of claims 1 to 9.

11. Display the time-series information, Accept an addition or correction to the time-series information, Record the time-series information reflecting the accepted addition or correction. The information processing method according to claim 10.

12. Based on the time-series information, output a time-series diagram showing the passage of time and the operations performed at each time. The information processing method according to claim 10 or claim 11.

13. Input the obtained image into a second model that outputs a treatment strategy when an image is input, and display the treatment strategy output from the second model. The information processing method according to any one of claims 1 to 4.

14. Display an icon for displaying the treatment strategy in association with the time when the treatment strategy was output in the time-series information. The information processing method according to claim 13.

15. The converted image is an object arrangement image obtained by associating and mapping the types of objects included in the image and the ranges of the respective objects. The information processing method according to claim 1.

16. The object includes at least one of a lumen organ wall, a plaque, calcification, and a guide wire. The information processing method according to claim 15.

17. Obtain patient information regarding a patient who undergoes catheter treatment, Sequentially obtain the actions related to the catheter treatment for the patient in association with the time when the actions were performed, Cause the obtained patient information and the time-series information in which the obtained actions are arranged in time series to be displayed on the first display device, When the action is an action for acquiring an image, Input the obtained image into a first model that outputs a converted image obtained by converting the image when an image is input, and obtain the converted image output from the first model or a reduced converted image obtained by reducing the converted image. A program that causes a computer to execute a process of displaying, in association with the time when the image was acquired, a reduced image obtained by reducing the image, and the converted image or the reduced converted image in the time-series information. A program for causing a computer to execute the process.

18. A patient information acquisition unit that acquires patient information regarding a patient undergoing catheter treatment, An action acquisition unit that sequentially acquires, in association with an action related to the catheter treatment for the patient and the time when the action was performed, A display unit that displays, on a first display device, the patient information acquired by the patient information acquisition unit and time-series information in which the actions acquired by the action acquisition unit are arranged in time series, A conversion image acquisition unit that inputs the acquired image into a first model that outputs a converted image obtained by converting the image when the image is input, and acquires the converted image output from the first model or a reduced converted image obtained by reducing the converted image, When the action is an action of acquiring an image, the display unit includes, in the time-series information, a reduced image obtained by reducing the image in association with the time when the image was acquired, and the converted image or the reduced converted image. An information processing system.

19. An information processing system having an information processing device and a terminal device, wherein the information processing device records time-series information associating a patient undergoing catheter treatment, an action related to the catheter treatment, and the time when the action was performed, when the action is an action of acquiring an image, inputs the acquired image into a first model that outputs a converted image obtained by converting the image when the image is input, and acquires the converted image output from the first model or a reduced converted image obtained by reducing the converted image, includes, in the time-series information, a reduced image obtained by reducing the image in association with the time when the image was acquired, and the converted image or the reduced converted image, wherein the terminal device acquires patient information regarding the patient, sequentially acquires the time-series information associated with the patient from the information processing device, and displays the acquired patient information and the acquired time-series information. An information processing system.

Images