PROGRAM, INFORMATION PROCESSING METHOD, INFORMATION PROCESSING APPARATUS AND MODEL GENERATION METHOD

Abstract

This program causes a computer to execute a process of acquiring patient information about a patient who is to undergo treatment and a medical image obtained by imaging a hollow organ of the patient, and inputting the acquired patient information and the acquired medical image to a model learned so as to output treatment information for providing assistance in the treatment to be performed on the patient when the patient information and the medical image are inputted thereto, to output the treatment information.

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, an information processing device, and a model generating method. [Background technology]

[0002] Various methods have been proposed to support treatment based on medical images that visualize the inside of the human body, such as ultrasound images, optical coherence tomography (OCT) images, X-ray images, etc. For example, Patent Document 1 discloses a treatment device selection support system that accepts designation of the range of a diseased part in an intravascular image from an operator, sets the shape of a treatment device suitable for the diseased part in the designated range, and searches a database for information on a treatment device having the set shape and displays it. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-75141 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is not easy to read information such as diseased parts from medical images and reflect it in treatment, and it requires a wealth of knowledge and clinical experience. The invention in Patent Document 1 only searches a database for a treatment device suitable for the diseased part within the range specified by the operator, and does not adequately support treatment based on medical images.

[0005] In one aspect, an object of the present invention is to provide a program or the like that can suitably support treatment based on medical images. [Means for solving the problem]

[0006] A program according to one aspect includes acquiring patient information regarding a patient to be treated and a medical image of a hollow organ of the patient, When the medical image is input, the acquired medical image is input to a first model that has been trained to detect a predetermined object in the medical image, and the object is detected. Based on the object detection result, a second medical image is generated in which an image area corresponding to the object in the medical image is displayed in a different display mode from other image areas, and the second medical image is generated. The patient information and A system that has been trained to output treatment information to assist in the treatment of the patient when Second Model, The generated second medical image; The acquired patient information and and having the computer execute a process of inputting the above and outputting the treatment information. Effect of the Invention

[0007] In one aspect, medical image-based treatment can be favorably supported. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of a treatment support system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a server. [Diagram 3] FIG. 2 is an explanatory diagram showing an example of a record layout of a medical treatment DB. [Figure 4] FIG. 1 is an explanatory diagram showing an overview of a learning model. [Diagram 5] FIG. 2 is an explanatory diagram showing details of a learning model. [Figure 6] FIG. 2 is an explanatory diagram showing an example of a display screen of a display device. [Figure 7] 13 is a flowchart showing the steps of a learning model generation process. [Figure 8] 13 is a flowchart showing a procedure for outputting treatment information. [Figure 9] 13 is a flowchart showing a procedure for outputting treatment information according to Modification 1. [Figure 10] FIG. 11 is an explanatory diagram of a learning model according to the second modified example. [Figure 11] FIG. 11 is an explanatory diagram of a learning model according to the second embodiment. [Figure 12A] FIG. 11 is an explanatory diagram showing a display example of a second medical image according to the second embodiment. [Figure 12B] FIG. 11 is an explanatory diagram showing a display example of a second medical image according to the second embodiment. [Figure 13A] FIG. 11 is an explanatory diagram showing a display example of a second medical image according to the second embodiment. [Figure 13B] FIG. 11 is an explanatory diagram showing a display example of a second medical image according to the second embodiment. [Figure 14] 11 is a flowchart showing the steps of a process for generating a learning model according to the second embodiment. [Figure 15] 11 is a flowchart showing a procedure of an output process of treatment information according to the second embodiment. [Figure 16] 13 is a flowchart showing the procedure of an output process of treatment information according to Modification 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a treatment support system. In this embodiment, a treatment support system that presents treatment information for supporting a patient's treatment to a user (medical worker) based on patient information on a patient undergoing intravascular treatment and medical images of the patient's blood vessels will be described. The treatment support system includes an information processing device 1 and an image diagnostic system 2. The information processing device 1 and the image diagnostic system 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

[0010] In this embodiment, intravascular treatment will be described as an example, however, the target hollow organ is not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines.

[0011] The image diagnostic system 2 includes an intravascular imaging device 21, a fluoroscopic imaging device 22, and a display device 23. The intravascular imaging device 21 is a device for imaging a tomographic image of the inside of a patient's blood vessel, and is, for example, an IVUS (Intravascular Ultrasound) device that performs an ultrasonic examination using a catheter 211. The catheter 211 is a medical instrument inserted into the patient's blood vessel, and includes a piezoelectric element that transmits ultrasonic waves and receives reflected waves from inside the blood vessel. The intravascular imaging device 21 generates an ultrasonic tomographic image based on the signal of the reflected wave received by the catheter 211, and displays it on the display device 23.

[0012] In this embodiment, the intravascular diagnostic imaging apparatus 21 generates an ultrasonic tomographic image, but may generate, for example, an optical coherence tomographic image (OCT image).

[0013] The fluoroscopic imaging device 22 is a device unit for capturing a fluoroscopic image by viewing inside the patient's body, for example, an angiography device for performing angiography. The fluoroscopic imaging device 22 includes an X-ray source 221 and an X-ray sensor 222, and captures an X-ray fluoroscopic image of the patient by the X-ray sensor 222 receiving X-rays irradiated from the X-ray source 221. For example, an X-ray opaque marker is attached to the tip of the catheter 211, and the position of the catheter 211 is visualized in the fluoroscopic image. The fluoroscopic imaging device 22 displays the fluoroscopic image in which the position of the catheter 211 is visualized on the display device 23, and presents it together with an intravascular tomographic image.

[0014] Although ultrasonic tomographic images, optical coherence tomographic images, and angiographic images have been given above as examples of medical images, the medical images may also be computed tomography (CT) images, magnetic resonance imaging (MRI) images, etc.

[0015] The information processing device 1 is an information processing device capable of various information processing and transmitting and receiving information, for example, a server computer, a personal computer, etc. In the present embodiment, the information processing device 1 is a server computer, and will be referred to as the server 1 for simplicity in the following. The server 1 may be a local server installed in the same facility (hospital, etc.) as the image diagnostic system 2, or a cloud server connected to the image diagnostic system 2 via the Internet or the like. The server 1 functions as a treatment support device that outputs treatment information for supporting intravascular treatment of a patient based on patient information on the patient to be treated and medical images (tomographic images and fluoroscopic images) of the patient. Specifically, as described later, the server 1 performs machine learning to learn predetermined training data, and prepares a learning model 50 (see FIG. 4) that uses patient information and medical images as inputs and outputs treatment information. The server 1 inputs patient information and medical images of a patient to be treated into the learning model 50, and acquires treatment information from the learning model 50. The server 1 outputs the treatment information to the image diagnostic system 2, and displays it on the display device 23.

[0016] The treatment information is information for supporting the treatment of a patient and is information for assisting a user (medical worker) who treats the patient. In the present embodiment, endovascular treatment is taken as an example, and information for supporting PCI (Percutaneous Coronary Intervention) using a catheter 211 is output as the treatment information. For example, the treatment information includes the position and characteristics of a lesion such as plaque, a treatment device such as a stent or balloon to be used, a recommended dose of contrast agent and imaging time when taking a fluoroscopic image, and progress information after treatment (for example, risk of complications). The server 1 estimates these pieces of information using the learning model 50 and displays them on the display device 23. The server 1 also detects objects such as a lesion or a treatment device from within a medical image, generates a second medical image in which the detected object is processed so as to be identifiable, and displays the second medical image on the display device 23.

[0017] In this embodiment, the processing for generating treatment information using the learning model 50 is performed in the server 1, but the learning model 50 constructed by the server 1 may be installed in the image diagnosis system 2, and the processing for generating treatment information may be performed locally.

[0018] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit . The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an AI chip (AI semiconductor), etc., and performs various information processing by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

[0019] The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P and other data necessary for the control unit 11 to execute processing. The auxiliary storage unit 14 also stores a learning model 50 and a medical DB 141. The learning model 50 is a machine learning model that has learned training data as described above, and is a model that receives patient information and medical images as input and outputs treatment information to support the treatment of patients. The learning model 50 is expected to be used as a program module that constitutes artificial intelligence software. The medical DB 141 is a database that stores the medical data of patients.

[0020] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0021] In the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit for accepting operation input, a display unit for displaying images, etc. The server 1 may also include a reading unit for reading portable storage medium 1a such as a CD (Compact Disk), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, or an external hard disk, and may read and execute the program P from the portable storage medium 1a. Alternatively, the server 1 may read the program P from a semiconductor memory 1b.

[0022] 3 is an explanatory diagram showing an example of a record layout of the medical DB 141. The medical DB 141 includes a medical ID column, a patient information column, a treatment information column, and an image information column. The medical ID column stores a medical ID for identifying each medical data. The patient information column, the treatment information column, and the image information column store, in association with the medical ID, patient information on the patient to be treated, treatment information on the treatment given to the patient, and medical images obtained by examining the patient.

[0023] Patient information is information about patients who have undergone treatment, and is the patient's medical records. Treatment information includes, for example, the patient's age, sex, diagnosis, risk factors (presence or absence of lifestyle-related diseases, etc.), medical history, treatment history, concomitant medications, blood test results, number of affected blood vessels, left ventricular ejection fraction, history of cardiovascular-related sudden events (myocardial infarction, etc.), etc.

[0024] The treatment information is information about an intravascular treatment performed on a patient, such as a record of PCI. The treatment information includes, for example, the date of PCI, the position of the treated lesion (hereinafter referred to as "lesion site"), the nature of the lesion, the puncture site of the catheter 211, the total amount of contrast agent used, as well as the time of fluoroscopic image capture, the presence or absence and content of additional procedures before stent placement, the presence or absence and content of additional procedures after stent placement, the type of placed stent (e.g., product name), diameter, length, total number, type and length of balloons, maximum expansion diameter, maximum expansion pressure, expansion time, attenuation time, treatment method for bifurcation lesion, and progress information after treatment (e.g., the presence or absence of complications). In this embodiment, the diameters of the stent and balloon are expressed as diameters, but it goes without saying that they may be radii.

[0025] The medical images are images of the blood vessels of a patient, and as described above, include tomographic images of the inside of the blood vessels (ultrasonic tomographic images, optical coherence tomographic images, etc.), fluoroscopic images that visualize the inside of the patient's body using X-rays (angiographic images, computed tomographic images, etc.), and magnetic resonance images. The image information also includes physiological functional evaluation results such as FFR (Fractional Flow Reserve). In this embodiment, as an example, ultrasonic tomographic images and angiographic images are used as medical images to input the learning model 50.

[0026] 4 is an explanatory diagram showing an overview of the learning model 50. The learning model 50 is a machine learning model that receives patient information and medical images as input and outputs treatment information for supporting the treatment of the patient. The server 1 performs machine learning to learn from predetermined training data to generate the learning model 50 in advance. The server 1 then inputs the medical images of the patient acquired from the image diagnostic system 2 and the patient information related to the patient into the learning model 50 to generate treatment information.

[0027] Specifically, as shown in Fig. 4, the learning model 50 receives as input patient information stored in the medical DB 141, a tomographic image captured by the intravascular imaging diagnostic apparatus 21, and a fluoroscopic image captured by the fluoroscopic image capturing apparatus 22. The learning model 50 then outputs, as treatment information, the estimation results obtained for each item shown in the upper right corner of Fig. 4.

[0028] For example, the treatment information includes information about a lesion in a blood vessel. Specifically, the information about the lesion includes the site of the lesion (e.g., the type of blood vessel in which the lesion exists) and the characteristics of the lesion. The information about the lesion may include not only the position and characteristics, but also the length of the lesion in the longitudinal direction of the blood vessel, the size (area) of the lesion in the cross section of the blood vessel, and the like.

[0029] The treatment information also includes information on a treatment device to be inserted into a blood vessel. The treatment device is, for example, a stent or a balloon for expanding a blood vessel. The treatment device is not limited to a stent, and may include, for example, a rotablator for resecting a lesion. The learning model 50 outputs stent information related to the stent, such as the type of stent to be used (e.g., product name), shape (diameter and length), total number of stents, presence or absence and content of additional procedures before stent placement, and presence or absence and content of additional procedures after stent placement. The learning model 50 also outputs balloon information related to the balloon, such as the type of balloon to be used (e.g., product name), shape (length), and expansion conditions (maximum expansion pressure, maximum expansion diameter, expansion time, and attenuation time required for the balloon to be deflated after expansion).

[0030] The information on the therapeutic devices may include the order of use of the therapeutic devices, which can favorably assist the user in selecting a therapeutic device when selecting the therapeutic device and performing treatment.

[0031] The treatment information also includes information on the imaging conditions of the fluoroscopic images. The information on the imaging conditions includes, for example, the dose of contrast agent, the imaging time of the fluoroscopic images ("fluoroscopic time" in FIG. 4), etc. The server 1 allows the learning model 50 to learn the dose of contrast agent, the imaging time, etc. when the treatment is performed on the patient, and is capable of outputting the recommended dose of contrast agent, the imaging time, etc.

[0032] The treatment information also includes progress information that estimates the progress of the patient after treatment. The progress information is, for example, an estimation result that estimates the risk of complications. The server 1 allows the learning model 50 to learn complications that have developed after treatment for patients who have already undergone treatment, and is capable of outputting complications that have a high probability of developing after treatment as progress information. As described later, it is preferable to output a probability value that evaluates the degree of development of each complication for each of a number of complications that may develop after treatment (see FIG. 6).

[0033] 4 shows, as an example, a case where a tomographic image in which plaque has occurred in a patient's blood vessel and a fluoroscopic image showing the position of catheter 211 at the time the tomographic image was generated are input together with patient information to learning model 50. In this case, learning model 50 outputs the position and characteristics of the lesion corresponding to plaque, the stent to be used and its expansion diameter and expansion pressure, progress information after treatment, and the like.

[0034] The learning model 50 outputs the above-mentioned various information as treatment information, and also generates a second medical image by processing the input medical image so that a specific object in the image can be identified, and outputs the second medical image as one of the treatment information. The second medical image is an image in which an image area corresponding to an object such as a lesion is processed as described above, and the image area of ​​the object is displayed in a different display mode from other image areas. For example, the learning model 50 generates a second medical image in which an image area corresponding to an object is displayed in a color other than black and white in a tomographic image displayed in black and white. The server 1 outputs the second medical image to the display device 23 together with information such as the lesion site, and displays it.

[0035] Note that the object targeted by the learning model 50 is not limited to a lesion, and may be, for example, a treatment device to be inserted into a patient's blood vessel (for example, a stent already placed in the patient's blood vessel). The learning model 50 only needs to be capable of generating a second medical image that can identify a specific object present in the blood vessel.

[0036] For simplicity, in the following description, an image region that corresponds to an object in a medical image will be called an "object region."

[0037] Fig. 5 is an explanatory diagram showing details of the learning model 50. In Fig. 5, a model consisting of a first model 51 and a second model 52 is shown as an example of the learning model 50. The details of the learning model 50 will be described with reference to Fig. 5.

[0038] The first model 51 is a machine learning model that receives medical images (tomographic images and fluoroscopic images) as input and outputs a detection result of detecting an object in the image. For example, the first model 51 is a neural network model generated by deep learning, and is a CNN (Convolutional Neural Network) that extracts features of the input image using multiple convolution layers. The first model 51 includes an intermediate layer (hidden layer) in which a convolution layer that convolves pixel information of the input image and a pooling layer that maps the convolved pixel information are alternately connected, and extracts features (feature map) of the input image.

[0039] In this embodiment, the first model 51 is described as a CNN, but it may be a model based on other learning algorithms, such as a Generative Adversarial Network (GAN), a Recurrent Neural Network (RNN), a Support Vector Machine (SVM), a decision tree, or the like.

[0040] In this embodiment, the server 1 generates a first model 51 that identifies, on a pixel-by-pixel basis, whether or not each pixel in an input medical image corresponds to an object region. For example, the server 1 generates, as the first model 51, a semantic segmentation model (such as U-net) or MASK R-CNN (Region CNN).

[0041] The semantic segmentation model is a type of CNN, and a type of EncoderDecoder model that generates output data from input data. In addition to the convolution layer that compresses the input image data, the semantic segmentation model also has a deconvolution layer that maps (enlarges) the features obtained by compression to the original image size. The deconvolution layer identifies which object exists in which position in the image on a pixel-by-pixel basis based on the features extracted in the convolution layer, and generates a binarized label image that indicates which object each pixel corresponds to.

[0042] MASK R-CNN is a variant of Faster R-CNN, which is mainly used for object detection, and is a model that connects a deconvolution layer to Faster R-CNN. In MASK R-CNN, the image features extracted by CNN and the coordinate range of the target object extracted by RPN (Region Proposal Network) are input to the deconvolution layer, and a mask image that masks the image area of ​​the object in the input image is finally generated.

[0043] The server 1 generates these models as the first model 51 and uses them for object detection. Note that all of the above models are merely examples, and the first model 51 may be any model capable of identifying the position and shape of an object in a medical image. In the present embodiment, as an example, the first model 51 will be described as a semantic segmentation model.

[0044] 4, object regions in the intravascular tomographic image and fluoroscopic image are conceptually illustrated by hatching. Objects detected by the first model 51 are, for example, lesions such as plaque, or therapeutic devices such as placed stents. The server 1 inputs medical images to the first model 51 and detects various objects.

[0045] Note that plaque is an example of a lesion, and for example, a narrowed portion of a blood vessel, a calcified tissue, a dissection (flap) of a blood vessel wall, a neointima, or other portions may be detected. Also, a stent is an example of a treatment device, and other devices may be detected. Also, a lesion and a treatment device are examples of objects, and other objects may be detected.

[0046] Continuing the explanation by returning to Fig. 5. The server 1 performs learning using training data in which a training medical image is labeled with data indicating an object region as exemplified in Fig. 4. Specifically, in the training data, a coordinate range corresponding to an object region and a label (metadata) indicating the type of object are added to the training medical image.

[0047] The server 1 inputs a training medical image to the first model 51 and obtains the detection result of detecting an object as an output. Specifically, it obtains a labeled image as an output in which each pixel in the object region is labeled with a value indicating the type of object.

[0048] The server 1 compares the detection result output from the first model 51 with the coordinate range of the correct object area and the type of object indicated by the training data, and optimizes parameters such as the weights between neurons so that the two are close to each other. In this way, the server 1 generates the first model 51.

[0049] In this embodiment, two types of images, a tomographic image and a fluoroscopic image, are processed, but for example, the server 1 may prepare two first models 51 corresponding to the respective images, or the first model 51 may be configured as a multimodal model having two networks for processing the tomographic image and the fluoroscopic image, respectively. This makes it possible to detect objects from each image.

[0050] In this embodiment, the first model 51 receives as input a plurality of frames of medical images (video) that are successive in time series, and detects an object from the medical images of each frame. Specifically, the first model 51 receives as input a plurality of frames of medical images (e.g., tomographic images) that are successive along the longitudinal direction of a blood vessel in accordance with the scanning of the catheter 211. The first model 51 detects an object from the medical images of each frame that are successive along the time axis.

[0051] In the following description, for convenience, each successive frame of medical images will simply be referred to as a "frame image."

[0052] The server 1 may input a plurality of frame images one by one into the first model 51 and detect an object from each frame image individually, but it is preferable to input a plurality of consecutive frame images simultaneously so that an object region can be detected from the plurality of frame images simultaneously. For example, the server 1 sets the first model 51 as a 3D-CNN (e.g., 3D U-net) that handles three-dimensional input data. The server 1 handles the first model 51 as three-dimensional data with the coordinates of the two-dimensional frame images as two axes and the time when each frame image is acquired (the time when the frame image is generated) as one axis. The server 1 inputs a plurality of frame images (e.g., 16 frames) for a predetermined unit time as one set into the first model 51, and simultaneously outputs a label image in which an object region is labeled for each of the plurality of frame images. This allows an object to be detected taking into account the frame images before and after the frame images that are consecutive in time series, thereby improving the detection accuracy.

[0053] In the above, the time series of frame images can be processed by treating the data as three-dimensional data including the time axis, but the present embodiment is not limited to this. For example, the server 1 may detect an object from a plurality of consecutive frame images by making the first model 51 a model combining CNN and RNN. In this case, for example, an LSTM (Long-Short Term Memory) layer is inserted after the intermediate layer related to CNN, and object detection is performed by referring to feature amounts extracted from previous and subsequent frame images. Even in this case, processing can be performed in consideration of previous and subsequent frame images in the same manner as above, and detection accuracy can be improved.

[0054] Next, the second model 52 will be described. The second model 52 is a machine learning model that receives patient information and medical images as inputs and outputs treatment information for supporting intravascular treatment of the patient. For example, the second model 52 is a CNN, which includes an intermediate layer in which a convolutional layer and a pooling layer are alternately connected, and extracts features of the input data.

[0055] In this embodiment, the second model 52 is described as being a CNN, but it may be a model based on other learning algorithms, such as GAN, RNN, SVM, or decision tree.

[0056] The server 1 performs learning using training data in which correct treatment information is added to the patient information and medical images for training. The patient information and medical images for training are patient information and medical images of patients who have already undergone treatment, and are the medical records of the patients and medical images such as tomographic images and fluoroscopic images obtained during the treatment. The correct treatment information is a record of the treatment (PCI) performed on the patient. The server 1 provides the patient information and medical images stored in the medical DB 141 and the treatment information to the second model 52 as training data, and performs learning.

[0057] Specifically, the server 1 inputs data such as the patient's age, sex, and diagnosis included in the patient information as categorical variables indicating attributes of the medical image to the second model 52. The server 1 inputs the patient information as categorical variables together with the tomographic image and the fluoroscopic image to the second model 52, and obtains treatment information for each item exemplified in the upper right corner of Fig. 4 as output.

[0058] As shown in the upper right of FIG. 4, the output items of the second model 52 include items that are shown as classification results, such as "lesion site" and "lesion characteristics," and items that are expressed as continuous values, such as "total contrast time" and "fluoroscopy time." In this way, the second model 52 simultaneously handles so-called classification problems and regression problems, and may, for example, treat all items as regression problems and output continuous values, and convert items to be shown as classification results from continuous values ​​to binary values. Alternatively, a separate output layer may be provided for each item, and the feature values ​​extracted in the intermediate layer may be input to each output layer, so that estimation of each item may be performed separately. Alternatively, the second model 52 itself may be prepared for each item, and estimation may be performed separately.

[0059] The server 1 may directly use a medical image of the patient's blood vessels as input to the second model 52, but in this embodiment, a second medical image is generated by processing the object region detected in the first model 51 so as to be identifiable, and inputs the second medical image to the second model 52. The second medical image is an image in which the object region is displayed in a different display mode depending on the type of object, for example, an image in which a label image output from the first model 51 is superimposed on the original medical image.

[0060] For example, the server 1 processes the label image output from the first model 51 into a semi-transparent mask and superimposes it on the original medical image. In this case, the server 1 makes the display mode of each object region different depending on the type of object, for example, by changing the display color of the mask depending on the type of object. In this way, the server 1 generates a medical image in which the object region is displayed in a different display mode from the other regions, and inputs the medical image to the second model 52.

[0061] The server 1 inputs the training patient information and the medical image with the processed object region into the second model 52, and obtains the treatment information as the output. The server 1 compares the output treatment information with the correct treatment information, and optimizes parameters such as the weights between neurons so that the two are close to each other. In this way, the server 1 generates the second model 52.

[0062] In this embodiment, two types of models, the first model 51 for object detection and the second model 52 for generating treatment information, are combined, but object detection may be performed on a rule basis. That is, it is not essential that the learning model 50 includes the first model 51. Objects may be detected on a rule basis as a preprocessing for input to the learning model 50, and the detection result may be input to the learning model 50.

[0063] When generating treatment information for a patient who is to undergo new treatment, the server 1 acquires medical images of the patient from the image diagnostic system 2 and inputs them into the learning model 50, and generates treatment information including the second medical image. This processing may be performed in real time when the images are acquired, or recorded medical images (videos) may be acquired together and processed afterwards.

[0064] The server 1 outputs the generated treatment information to the display device 23 for display. Note that, in the present embodiment, the treatment information is output to the image diagnostic system 2, but it goes without saying that the treatment information may be output to a device (e.g., a personal computer) other than the image diagnostic system 2 for display.

[0065] 6 is an explanatory diagram showing an example of a display screen of the display device 23. The display device 23 displays each piece of treatment information output from the server 1 and presents it to the user. For example, the display device 23 displays a list 71, a progress information column 72, a tomographic image 73, and a fluoroscopic image 74.

[0066] The list 71 is a table showing a list of the estimation results of each item output from the second model 52. The list 71 includes, for example, information on the lesion, information on the treatment device, information on the imaging conditions of the fluoroscopic image, and other information. The display device 23 presents this treatment information to the user in the list 71 to support the intravascular treatment.

[0067] The progress information field 72 is a display field showing the results of estimating the progress of the user after treatment, which are the results of estimating the onset of complications as described above. The server 1 uses the learning model 50 to calculate the probability value of onset for each of a plurality of complications, and displays the probability values ​​in a list in the progress information field 72 in association with the names of the complications.

[0068] The display device 23 also displays a tomographic image 73 generated by the intravascular diagnostic imaging device 21 and a fluoroscopic image 74 captured by the fluoroscopic image capturing device 22. In this case, the display device 23 displays, as the tomographic image 73 and the fluoroscopic image 74, a second medical image obtained by processing the object region of the original medical image.

[0069] Fig. 7 is a flowchart showing the procedure of the generation process of the learning model 50. The process contents when generating the learning model 50 by learning the training data will be described with reference to Fig. 7. The control unit 11 of the server 1 acquires training data in which correct treatment information is assigned to the training patient information and medical images (step S11). The training patient information and medical images are patient information and medical images of patients who have already undergone treatment. The treatment information is information on the treatment performed on the patient, and includes label data indicating the object area in addition to the treatment records stored in the medical DB 141.

[0070] The control unit 11 generates a first model 51 that detects an object when a medical image is input based on the training data (step S12). For example, as described above, the control unit 11 generates a CNN related to semantic segmentation as the first model 51. The control unit 11 inputs a medical image for training to the first model, and obtains a detection result of detecting an object region as an output. The control unit 11 compares the detected object region with correct label data, and optimizes parameters such as weights between neurons so that the two are similar to each other, thereby generating the first model 51.

[0071] Furthermore, the control unit 11 generates a second model 52 based on the training data, which outputs treatment information for supporting the treatment of a patient when patient information and a medical image are input (step S13). Specifically, the control unit 11 generates the second model 52, which outputs treatment information by inputting the patient information and the second medical image whose object region is processed based on the detection result in the first model 51. The control unit 11 inputs the training patient information and the second medical image whose object region is processed based on the correct label data to the second model, and obtains the treatment information as an output. The control unit 11 compares the output treatment information with the correct treatment information, and optimizes parameters such as weights between neurons so that the two are similar to each other, thereby generating the second model 52. The control unit 11 ends the series of processes.

[0072] 8 is a flowchart showing the procedure of the treatment information output process. The process contents when the treatment information of the patient to be treated is output using the learning model 50 will be described with reference to FIG. The control unit 11 acquires patient information on a patient to be treated and medical images of the blood vessels of the patient (step S31). The control unit 11 inputs the acquired medical images to the first model 51 and detects objects (step S32). Specifically, the control unit 11 inputs tomographic images and fluoroscopic images captured by the intravascular imaging device 21 and the fluoroscopic imaging device 22 to the first model 51 and detects objects in the tomographic images and fluoroscopic images.

[0073] The control unit 11 inputs the second medical image with the object region processed based on the detection result in step S32 and the patient information to the second model 52 to generate treatment information (step S33). The control unit 11 outputs the generated treatment information to the display device 23 to display it (step S34). Specifically, the control unit 11 outputs the second medical image with the object region processed as treatment information, in addition to information such as the lesion in the blood vessel, the treatment device to be used, and the imaging conditions of the fluoroscopic image, and causes the display device 23 to display it. The control unit 11 ends the series of processes.

[0074] As described above, according to the first embodiment, by using the learning model 50 that has already learned the training data, it is possible to appropriately support treatment based on medical images.

[0075] Moreover, according to the first embodiment, the position, characteristics, and the like of a lesion in a blood vessel (hollow organ) can be estimated and presented to the user.

[0076] Moreover, according to the first embodiment, the type, shape, number of devices to be used, order of use, and the like can be estimated and presented to the user.

[0077] Moreover, according to the first embodiment, it is possible to estimate and present suitable imaging conditions for a perspective image.

[0078] Moreover, according to the first embodiment, progress information after treatment, such as the onset of complications, can be estimated and presented to the user.

[0079] Furthermore, according to this embodiment 1, by combining a first model 51 that detects objects from medical images with a second model 52 that generates treatment information based on the detection results by the first model 51, the correct position, shape, etc. of the object can be provided to the second model 52, thereby improving the estimation accuracy of the treatment information.

[0080] Furthermore, according to the first embodiment, by generating a second medical image in which the object region is processed, it is possible to favorably assist in identifying a lesion or the like.

[0081] Furthermore, according to the first embodiment, by inputting a plurality of frame images generated in time series to the learning model 50, it is possible to improve the estimation accuracy of treatment information.

[0082] Furthermore, according to this embodiment 1, by using not only cross-sectional images of inside a blood vessel (hollow organ) but also fluoroscopic images as medical images, not only local images but also images of the entire blood vessel can be provided to the learning model 50, thereby improving the estimation accuracy.

[0083] (Variation 1) In the first embodiment, a description is given of a form in which treatment information is output using the learning model 50. In this modified example, a description is given of a form in which a correction input for correcting the output treatment information is received from a user, and re-learning is performed based on the corrected treatment information.

[0084] 9 is a flowchart showing the procedure of the treatment information output process according to Modification 1. After outputting the treatment information (step S34), the server 1 executes the following process. The control unit 11 of the server 1 accepts, from the user, input for correcting the treatment information output to the display device 23 (step S35). For example, the control unit 11 accepts input for correcting information of each item displayed in the list 71 on the display screen exemplified in Fig. 6. Furthermore, the server 1 accepts input of the correct coordinate range, type, etc., for the second medical image displayed as the tomographic image 73, when the coordinate range of the object region, the type of the object, etc. differ from the actual ones.

[0085] When the control unit 11 receives the input for correcting the treatment information, the control unit 11 performs re-learning using the patient information and medical images input to the learning model 50 and the corrected treatment information as training data, and updates the learning model 50 (step S36). That is, the control unit 11 optimizes parameters such as the weights between neurons so that the treatment information output from the learning model 50 approximates the corrected treatment information, and regenerates the learning model 50. The control unit 11 ends the series of processes.

[0086] As described above, according to this modification example 1, the learning model 50 can be optimized through the operation of this system.

[0087] (Variation 2) In the first embodiment, a configuration has been described in which an object is detected by the first model 51, and a second medical image in which the object region has been processed is input to the second model 52. In this modification, a configuration will be described in which the type, dimensions, etc. of the object are identified from the detection result, and object information indicating the type, dimensions, etc. of the identified object is used as input to the second model 52.

[0088] 10 is an explanatory diagram of a learning model 50 according to Modification 2. As in the first embodiment, the learning model 50 according to this modification also includes a first model 51 and a second model 52. In this modification, the server 1 performs image analysis of the object area detected by the first model 51, and identifies the type, dimensions, etc. of the object.

[0089] For example, the server 1 identifies the type, dimensions, etc. of a stent already placed in the patient's blood vessel. In this modified example, the object to be identified is a stent, but it may be another object such as a lesion. The server 1 performs image analysis of the object area detected as a stent, identifies the name of the placed stent, and identifies the diameter, length, etc. of the stent. The server 1 inputs data such as the identified stent type, dimensions, etc. into the second model 52 as object information, and generates treatment information.

[0090] In this case, the server 1 may input not only the object information (text data) but also the original medical image input to the first model 51 to the second model 52. Alternatively, the server 1 may input the second medical image with the object region processed to the second model 52, similarly to the first embodiment.

[0091] As described above, in this modification, image analysis is performed as preprocessing of the second model 52, and object information is identified and input to the second model 52. This provides data such as the type and dimensions of the object to the second model 52, making it possible to improve the estimation accuracy.

[0092] Since this modification is similar to the first embodiment except that image analysis is performed as pre-processing of the second model 52, detailed description of the flowchart and other details will be omitted.

[0093] (Embodiment 2) In this embodiment, an embodiment for assisting in endovascular treatment, particularly in the placement and expansion of a stent, will be described. Note that the same reference numerals will be used to designate the same parts as in the first embodiment, and the description thereof will be omitted.

[0094] Fig. 11 is an explanatory diagram of a learning model 50 according to the second embodiment. As in the first embodiment, the learning model 50 according to this embodiment includes a first model 51 and a second model 52, and generates a second medical image in which an object region is processed, and other treatment information. As shown in the upper right of Fig. 11, the treatment information includes stent information related to a stent to be placed in a patient's blood vessel, and the type, shape (diameter, length), etc. of the stent to be used are output.

[0095] In this embodiment, the learning model 50 further includes a third model 53. The third model 53 is a machine learning model that estimates a target position in a blood vessel where a stent is to be placed and a target expansion diameter when a pre-treatment medical image is input. Specifically, the third model 53 receives a medical image as input and detects an image region in the image where a stent is to be placed and expanded. In addition to the stent information generated by the second model 52, the server 1 detects an image region where a stent is to be placed and expanded by the third model 53, generates a second medical image showing the detected region, and displays it on the display device 23 as one piece of stent information.

[0096] In the following description, for convenience, the area where the stent is to be placed and expanded is referred to as the "target area."

[0097] For example, the server 1 uses Mask R-CNN as the third model 53. As described above, Mask R-CNN is a CNN that detects a target image area from an input image, and is a model that can identify the target image area on a pixel-by-pixel basis. For example, the server 1 generates the third model 53 using, as training data, data in which a label indicating the coordinate range of a stent placed in a blood vessel of a patient who has already undergone treatment is added to a fluoroscopic image of the patient.

[0098] It should be noted that the third model 53 is not limited to Mask R-CNN, and may be other CNNs such as U-net, or other machine learning models such as GAN.

[0099] For example, the third model 53 detects the target region as a rectangular bounding box as shown in Fig. 11. The target medical image may be not only a fluoroscopic image but also a tomographic image. The target region may be any region in which a stent is to be placed and which is to be expanded by a balloon, and the shape of the target region is not limited to a rectangle.

[0100] The server 1 may input the fluoroscopic image acquired from the fluoroscopic image capturing device 22 directly to the third model 53, but it is preferable to input the second medical image in which an object region such as a lesion is processed by the first model 51 as shown in Fig. 11. This allows the third model 53 to determine a target region in consideration of the position, shape, etc. of the lesion to be treated.

[0101] As in the second modification, the type and dimensions of an object such as a lesion may be identified from the second medical image, and the identified object information may be provided to the third model 53.

[0102] For example, the server 1 generates a second medical image showing both the lesion and the range to be expanded by the stent by further superimposing a bounding box indicating the target area detected by the third model 53 on the second medical image obtained by processing the object area corresponding to the lesion. The server 1 may generate an image enclosing only the target area as the second medical image. The server 1 outputs the generated second medical image to the display device 23 as a perspective image 74.

[0103] In this embodiment, a first model 51 for detecting objects such as lesions and a third model 53 for detecting a target area where a stent is to be placed and expanded are prepared separately, but both may be the same model and object detection and target area detection may be performed simultaneously.

[0104] 12 and 13 are explanatory diagrams showing a display example of the second medical image according to the second embodiment. In this embodiment, the display device 23 displays the second medical image showing the target region where the stent is to be placed and expanded, and also displays the stent currently inserted in the blood vessel so as to be identifiable in the second medical image, thereby supporting the placement and expansion of the stent. Figs. 12A to 13B chronologically illustrate how the stent is inserted into the blood vessel, reaches the target position, and is expanded.

[0105] 12A and 12B show a state in which a stent is inserted into a target region. Display device 23 displays object region 741 and target region 742 in fluoroscopic image 74 in a distinguishable manner using a method such as color display. As described above, object region 741 corresponds to the lesion, and target region 742 corresponds to the region in the blood vessel where stent 743 is to be placed.

[0106] Furthermore, the display device 23 displays a stent 743 inserted into the blood vessel and a rectangular stent area 744 indicating the current position of the stent 743. The server 1 detects at least the current position and current expansion diameter of the stent (hereinafter referred to as the "current diameter") from the fluoroscopic image acquired from the fluoroscopic image capturing device 22, and displays them identifiably by a method such as color display.

[0107] The stent 743 may be detected using the first model 51, or may be detected by image recognition using pattern matching.

[0108] Furthermore, the server 1 calculates a difference value between the detected current position of the stent 743 and the target position of the stent 743 indicated by the target region 742, and displays it at the upper left of the fluoroscopic image 74. The current position of the stent 743 may be, for example, the midpoint of the elongated stent 743, or the tip of the stent 743, and any point of the stent 743 may be detected as the current position. The target position of the stent 743 may be, for example, the center of gravity of the rectangular target region 742, or the midpoint of the short side of the target region 742 located on the opposite side to the stent 743, and any point within the target region 742 may be set as the target position. The server 1 sequentially detects the stent 743 from the fluoroscopic image, calculates a difference value between the current position and the target position, and displays it on the display device 23.

[0109] The server 1 determines whether the stent 743 has reached the target position based on the difference between the current position and the target position. If it is determined that the stent 743 has reached the target position, the server 1 calculates an expansion rate by dividing the current diameter by the target expansion diameter based on the current diameter of the stent 743 and the target expansion diameter of the stent 743 indicated by the target region 742, and displays the expansion rate in the upper left of the fluoroscopic image 74. The target expansion diameter is the width of the target region 742 in a direction perpendicular to the blood vessel longitudinal direction, and is the length of the short side of the rectangular target region 742.

[0110] 13A and 13B show the state from when the stent 743 reaches the target position until the expansion is completed. When the stent 743 reaches the target position, the display device 23 switches the display at the upper left of the fluoroscopic image 74 to the expansion rate. The server 1 sequentially detects the current diameter of the stent 743, calculates the expansion rate, and causes the display device 23 to display it.

[0111] FIG. 14 is a flowchart showing the procedure of the generation process of the learning model 50 according to the second embodiment. The control unit 11 of the server 1 acquires training data for generating the learning model 50 (step S201). The training data according to the present embodiment includes a medical image of a patient who has already been treated, label data indicating an object region such as a lesion, and label data indicating an image region (target region) of a stent placed in a blood vessel. The control unit 11 shifts the process to step S12.

[0112] After executing the process of step S13, the control unit 11 generates a third model 53 that detects a target region where a stent should be placed and expanded when a medical image is input (step S202). Specifically, as described above, the control unit 11 generates Mask R-CNN as the third model 53. For example, the control unit 11 inputs a second medical image in which an object region of a training medical image is processed according to label data to the third model 53, and obtains a detection result of detecting the target region as an output. The control unit 11 compares the obtained target region with the correct label data, and optimizes parameters such as weights between neurons so that the two are close to each other, thereby generating the third model 53. The control unit 11 ends a series of processes.

[0113] 15 is a flowchart showing the procedure of the treatment information output process according to embodiment 2. After generating the treatment information using the second model 52 (step S33), the server 1 executes the following process. The control unit 11 inputs a pre-treatment medical image of the patient to the third model 53 and detects a target area where a stent should be placed (step S221). Specifically, as described above, the control unit 11 inputs a second medical image (fluoroscopic image) generated using the first model 51 to the third model 53 and detects the target area.

[0114] The control unit 11 detects the stent inserted into the patient's blood vessel from the medical image (step S222). The control unit 11 generates a second medical image showing the detected stent and the target area, and outputs it to the display device 23 together with the treatment information generated in step S33 (step S223).

[0115] The control unit 11 calculates a difference value between the current position of the stent and the target position of the stent indicated by the target region, and outputs it to the display device 23 (step S224). The control unit 11 determines whether the stent has reached the target position based on the difference value between the current position and the target position (step S225). If it is determined that the target position has not been reached (S225: NO), the control unit 11 returns the process to step S224. If it is determined that the target position has been reached (S225: YES), the control unit 11 calculates an expansion rate from the current diameter of the stent and the target expansion diameter, and outputs it to the display device 23 (step S226).

[0116] The control unit 11 determines whether the expansion is completed to the target expansion diameter (step S227). If the control unit 11 determines that the expansion is not completed (S227: NO), the control unit 11 returns the process to step S226. If the control unit 11 determines that the expansion is completed (S227: YES), the control unit 11 ends the series of processes.

[0117] In the above description, the type and shape of the stent to be used in the learning model 50 are automatically predicted, and the target position and expansion conditions of the stent are predicted, but the present embodiment is not limited to this. For example, the server 1 may receive a specification input from a user specifying the type and shape of a stent to be used in intravascular treatment, input information of the specified stent (first stent information) to the learning model 50, and predict information such as the target position and expansion conditions of the stent (second stent information). For example, the server 1 may input the first stent information specified by the user together with a medical image to the third model 53, and detect a target area when the stent specified by the user is used. This makes it possible to support stent placement and expansion according to the user's request.

[0118] As described above, according to the second embodiment, it is possible to suitably assist the placement and expansion of a stent during intravascular treatment.

[0119] (Variation 3) In the first modification, a description has been given of a form in which re-learning is performed by accepting correction input after outputting treatment information. Similarly, in the second embodiment, re-learning may be performed by accepting correction input after outputting stent information.

[0120] 16 is a flowchart showing the procedure of the treatment information output process according to Modification 3. After outputting the treatment information including the stent information (step S223), the server 1 executes the following processes. The control unit 11 of the server 1 accepts an input to correct the output stent information (step S241). For example, the control unit 11 accepts an input to correct the type, shape, etc. of the stent displayed as the stent information. The control unit 11 also accepts an input to correct the coordinate range of the target region for the second medical image. The control unit 11 transitions the process to step S224.

[0121] When it is determined that the expansion of the stent is completed (S227: YES), the control unit 11 performs re-learning using the original medical image input to the learning model 50 and the corrected stent information as training data, and updates the learning model 50 (step S242). That is, the control unit 11 optimizes parameters such as the weights between neurons so that the stent information output from the learning model 50 approximates the corrected stent information, and regenerates the learning model 50. The control unit 11 ends the series of processes.

[0122] As described above, according to the present modification example 3, the learning model 50 can be optimized through the operation of this system.

[0123] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0124] 1. Server (information processing device) 11 Control section 12 Main memory 13. Communications Department 14 Auxiliary storage P Program 141 Medical DB 50 Learning Model 51 1st Model 52 2nd Model 53 3rd Model 2. Diagnostic Imaging System 21 Intravascular imaging device 22 Fluoroscopic imaging device 23 Display device

Claims

1. Acquiring patient information regarding a patient to be treated and a medical image of a hollow organ of the patient; inputting the acquired medical image into a first model that has been trained to detect a predetermined object in the medical image when the medical image is input, and detecting the object; generating a second medical image based on the detection result of the object, the second medical image being displayed in a different display mode from other image regions in the medical image; The generated second medical image and the acquired patient information are input to a second model that has been trained to output treatment information that supports treatment to be performed on the patient when the second medical image and the patient information are input, and the treatment information is output. A program that causes a computer to carry out processing.

2. The generated second medical image is input to a third model that has been trained to detect a target area to be treated by a treatment device in the second medical image when the second medical image is input, and the target area is detected. The program according to claim 1.

3. Outputting the treatment information regarding the lesion in the hollow organ 3. The program according to claim 1 or 2.

4. and outputting the treatment information regarding a treatment device to be inserted into the hollow organ. The program according to any one of claims 1 to 3.

5. the treatment is an endovascular treatment; Outputting the treatment information regarding a stent to be placed in the blood vessel The program according to any one of claims 1 to 4.

6. Outputting the treatment information regarding a procedure to be performed before or after the placement of the stent. The program according to claim 5.

7. the treatment is an endovascular treatment; Outputting the treatment information regarding a balloon that expands the blood vessel The program according to any one of claims 1 to 6.

8. The medical image includes an X-ray fluoroscopic image of the hollow organ; The treatment information relating to the imaging conditions of the fluoroscopic image is output. The program according to any one of claims 1 to 7.

9. Progress information that estimates the progress of the patient after treatment is output as the treatment information. The program according to any one of claims 1 to 8.

10. Output the progress information regarding complications that may occur after treatment. The program according to claim 9.

11. acquiring a plurality of the medical images generated along a longitudinal direction of the tubular organ; Inputting the plurality of medical images into the first model to detect the object. The program according to any one of claims 1 to 10.

12. acquiring a tomographic image generated based on a signal detected by a catheter inserted into the hollow organ and a fluoroscopic image of the hollow organ; The tomographic and fluoroscopic images are input into the first model to detect the object. The program according to any one of claims 1 to 11.

13. The medical image includes at least one of an ultrasonic tomographic image, an optical coherence tomographic image, and an X-ray fluoroscopic image of the hollow organ. The program according to any one of claims 1 to 12.

14. Acquiring patient information regarding a patient to be treated and a medical image of a hollow organ of the patient; inputting the acquired medical image into a first model that has been trained to detect a predetermined object in the medical image when the medical image is input, and detecting the object; generating a second medical image based on the detection result of the object, the second medical image being displayed in a different display mode from other image regions in the medical image; The generated second medical image and the acquired patient information are input to a second model that has been trained to output treatment information that supports treatment to be performed on the patient when the second medical image and the patient information are input, and the treatment information is output. An information processing method for causing a computer to execute processing.

15. an acquisition unit that acquires patient information related to a patient to be treated and medical images of a hollow organ of the patient; a detection unit that detects a predetermined object in a medical image by inputting the acquired medical image into a first model that has been trained to detect the object when the medical image is input; a generating unit configured to generate a second medical image based on a detection result of the object, the second medical image being displayed in a different display mode from other image regions in the medical image, the second medical image being displayed in a different display mode from other image regions; an output unit that inputs the generated second medical image and the acquired patient information into a second model that has been trained to output treatment information that supports a treatment to be performed on the patient when the second medical image and the patient information are input, and outputs the treatment information; An information processing device comprising:

16. Acquire training data in which patient information on a patient who has already been treated, label data indicating an image area corresponding to a predetermined object in a medical image of a tubular organ of the patient, and treatment information on the treatment that has already been performed on the patient are assigned, generating a first model based on the medical image and the label data, the first model detecting the object when the medical image is input; generating a second medical image based on the label data, the second medical image being displayed in a different display manner from other image areas of the medical image; A second model is generated based on the patient information, the second medical image, and the treatment information, the second model being configured to output the treatment information when the patient information and the second medical image are input. A model generation method in which processing is performed by a computer.

Images