PROGRAM, IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS AND MODEL GENERATION METHOD

Abstract

To provide a program or the like capable of proposing various conditions in flash operation.SOLUTION: A computer acquires an ultrasonic tomographic image of a cross section of a blood vessel by using an ultrasonic signal, according to a program. Further, the computer calculates information on a lumen of the vessel in the ultrasonic tomographic image, based on the acquired ultrasonic tomographic image. Then, the computer identifies operation information on a flash operation in imaging cross-sections of blood vessels by using a light signal, based on the calculated information on the lumen.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a program, an image processing method, an image processing device, and a model generating method. [Background technology]

[0002] Intravascular treatment is performed by inserting a catheter into a patient's blood vessel and expanding a narrowed portion of the blood vessel using a balloon or a stent. In intravascular treatment, a tomographic image of the blood vessel is taken from inside the blood vessel using an optical coherence tomography (OCT) device, and the blood vessel is observed using the obtained tomographic image. During imaging using OCT, diffuse reflection and attenuation of light may occur due to blood containing blood cell components such as red blood cells, so a flush operation is performed to temporarily create a state without blood by injecting a contrast agent, low molecular weight dextran, physiological saline, or the like into the blood vessel. Patent Document 1 discloses a flush liquid used in the flush operation for appropriately taking a tomographic image to be used in image diagnosis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6562902 Summary of the Invention [Problem to be solved by the invention]

[0004] In the flushing operation, it is necessary to appropriately set conditions such as the type and dilution rate of the flushing liquid, the injection amount, and the flow rate during injection. If the flushing operation is not performed under appropriate conditions, the flushing operation will have to be redone and the tomographic images will have to be retaken, which will extend the procedure time. The conditions in the flushing operation must be set taking into consideration the state of the blood vessels to be treated and the size of the catheter to be used, and it is not easy to select appropriate conditions. Patent Document 1 does not mention the selection of various conditions in the flushing operation.

[0005] According to one aspect, an object of the present invention is to provide a program or the like capable of proposing conditions for a flash operation (operation information related to a flash operation). [Means for solving the problem]

[0006] A program according to one aspect causes a computer to execute a process of acquiring an ultrasonic tomographic image of a cross-section of a blood vessel using an ultrasonic signal, calculating information regarding the lumen of the blood vessel based on the acquired ultrasonic tomographic image, and identifying operation information regarding a flash operation when acquiring an image of the cross-section of the blood vessel using an optical signal based on the calculated information regarding the lumen. Effect of the Invention

[0007] In one aspect, conditions for flash operation (operation information regarding flash operation) can be suggested. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an image diagnostic system. [Diagram 2] FIG. 1 is an explanatory diagram illustrating an overview of a guiding catheter. [Diagram 3] FIG. 1 is a block diagram showing an example of the configuration of an image processing device. [Figure 4] FIG. 2 is an explanatory diagram showing an example of the configuration of a medical treatment DB. [Diagram 5] FIG. 1 is an explanatory diagram showing an overview of a learning model. [Figure 6] FIG. 13 is a schematic diagram showing an example of a screen of flash information. [Figure 7] 13 is a flowchart showing an example of a generation process procedure for a second learning model. [Figure 8] 13 is a flowchart showing an example of a procedure for a flash information presentation process. [Figure 9] FIG. 13 is an explanatory diagram showing a modified example of the learning model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the program, image processing method, image processing device, and model generating method disclosed herein will be described in detail with reference to the drawings showing embodiments thereof. In the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example.

[0010] FIG. 1 is an explanatory diagram showing an example of the configuration of an image diagnostic system. The image diagnostic system of this embodiment includes an intravascular inspection device 1, a fluoroscopic image capture device 2, an image processing device 3, a display device 4, and an input device 5. The intravascular inspection device 1 is a device for imaging a tomographic image of a patient's blood vessel, and captures a tomographic image of a cross section of the blood vessel from inside the blood vessel. The intravascular inspection device 1 of this embodiment is an image diagnostic device using a dual-type imaging catheter 10 having both functions of intravascular ultrasound (IVUS) and optical coherence tomography (OCT). The dual-type imaging catheter 10 is provided with an IVUS mode for acquiring an ultrasonic tomographic image only by IVUS, an OCT mode for acquiring an optical coherence tomographic image only by OCT, and a dual mode for acquiring both tomographic images by IVUS and OCT, and these modes can be switched for use. The intravascular inspection device 1 of this embodiment is not limited to a configuration using a dual-type imaging catheter 10, and may be a configuration using an imaging catheter equipped with only IVUS or only OCT. Hereinafter, the ultrasonic tomographic image and the optical coherence tomographic image will be referred to as an IVUS image and an OCT image, respectively, as appropriate.

[0011] The imaging catheter 10 is a medical instrument inserted into a patient's blood vessel, and the intravascular inspection device 1 captures IVUS and OCT images of the blood vessel from within the blood vessel using the imaging catheter 10 inserted into the blood vessel. When capturing an IVUS or OCT image using the imaging catheter 10, a medical professional first inserts a guiding catheter 60 (see FIG. 2) described below into the blood vessel of the patient, and then inserts the imaging catheter 10 into the guiding catheter 60 inserted into the blood vessel. The imaging catheter 10 has an imaging diagnostic probe 11 (see FIG. 2), and the imaging diagnostic probe 11 is provided with an ultrasonic transceiver unit that transmits ultrasonic waves (ultrasonic signals) and receives reflected waves from within the blood vessel, and an optical transceiver unit that transmits near-infrared light (optical signals) and receives reflected light from within the blood vessel. The intravascular inspection device 1 generates an IVUS image based on the signal of the reflected wave of the ultrasonic wave received by the ultrasonic transceiver unit of the imaging catheter 10, and generates an OCT image based on the signal of the reflected light received by the optical transceiver unit. The IVUS images and OCT images captured by the intravascular inspection device 1 using the imaging catheter 10 are displayed on the display device 4 via the image processing device 3.

[0012] When taking an OCT image, the medical staff performs a flush operation in which a flush liquid composed of a contrast agent, low molecular weight dextran, physiological saline, or the like is injected into the blood vessel to temporarily create a state in which there is no blood (a state in which the blood is replaced with the flush liquid). In the flush operation, the medical staff injects the flush liquid into the blood vessel via the guiding catheter 60 inserted into the blood vessel.

[0013] The fluoroscopic imaging device 2 is a device for taking a fluoroscopic image of the inside of a patient's body, and is, for example, an angiography device for taking an image of the blood vessel from outside the patient's body using X-rays while injecting a contrast agent into the blood vessel of the patient, and obtaining an angiographic image, which is a fluoroscopic image of the blood vessel. The fluoroscopic imaging device 2 includes an X-ray source and an X-ray sensor, and the X-ray sensor receives X-rays irradiated from the X-ray source, thereby taking an X-ray fluoroscopic image of the patient. Note that X-ray opaque markers that do not transmit X-rays are attached to the tips of the imaging catheter 10 and the guiding catheter 60, and the positions of the imaging catheter 10 and the guiding catheter 60 (X-ray opaque markers) are visualized in the fluoroscopic image. The fluoroscopic image taken by the fluoroscopic imaging device 2 is displayed on the display device 4 via the image processing device 3. In this embodiment, the image diagnostic system is configured to include the fluoroscopic imaging device 2 for taking an angiographic image, but is not particularly limited as long as it is a device for taking an image of the blood vessel of the patient and the catheters 10, 60 from multiple directions outside the body.

[0014] A display device 4 and an input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and displays medical images such as IVUS images and OCT images captured by the intravascular inspection device 1 and angio images captured by the fluoroscopic image capturing device 2. The input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone, and accepts various operations by the surgeon. The display device 4 and the input device 5 may be integrally stacked to form a touch panel. The input device 5 and the image processing device 3 may also be integrally configured. Furthermore, the input device 5 may be a sensor that accepts gesture input, gaze input, or the like.

[0015] The image processing device 3 outputs the IVUS image and OCT image captured by the intravascular inspection device 1 and the fluoroscopic image captured by the fluoroscopic image capture device 2 to the display device 4 for display. The image processing device 3 also identifies the size (diameter and length in the running direction of the blood vessel) of the blood vessel to be captured based on the IVUS image captured by the intravascular inspection device 1, and generates operation information (hereinafter referred to as flash information) regarding a flash operation to be performed when capturing an OCT image based on the identified blood vessel size, etc. Therefore, in this embodiment, by capturing an IVUS image before capturing an OCT image, the image processing device 3 can generate flash information based on the IVUS image and present it to the surgeon.

[0016] FIG. 2 is an explanatory diagram for explaining an outline of the guiding catheter 60. The guiding catheter 60 has a sheath 61 and a branch hub 62 attached to the end of the sheath 61. In the following description, the side of the guiding catheter 60 far from the branch hub 62 is described as the tip side, and the branch hub 62 side is described as the base side. The sheath 61 forms a tube portion that is continuous from the tip to the opening 64 of the branch hub 62, and the opening 64 of the branch hub 62 is used as an insertion port of the imaging catheter 10 (imaging diagnostic probe 11) connected to the intravascular inspection device 1. In addition, the branch hub 62 is provided with a side port 63, which is used as a connection port for the syringe 70 filled with a flushing liquid 72. The guiding catheter 60 receives a guide wire that has been inserted into a blood vessel beforehand, and is guided to the vicinity of the entrance of the coronary artery by the guide wire. Inside the guiding catheter 60, the imaging diagnostic probe 11 connected to the connector portion 13 is inserted from the opening 64 of the branch hub 62.

[0017] The imaging catheter 10 has a diagnostic imaging probe 11 and a connector section 13 arranged at an end (base end side) of the diagnostic imaging probe 11. The imaging catheter 10 is connected to an MDU (Motor Drive Unit) (not shown) of the intravascular inspection device 1 via the connector section 13. The diagnostic imaging probe 11 has a sheath (not shown) inserted into the guiding catheter 60, a shaft inserted into the sheath, and a sensor section 12 provided at the tip of the shaft. The tip of the sheath is provided with a guidewire insertion section (not shown) through which a guidewire can be inserted. The guidewire insertion section is used to receive a guidewire previously inserted into a blood vessel and to guide the diagnostic imaging probe 11 to an observation site or an affected area by the guidewire. The sensor section 12 has an ultrasonic transmission / reception section and an optical transmission / reception section. An electric signal cable (not shown) connected to the ultrasonic transmission / reception section and an optical fiber cable (not shown) connected to the optical transmission / reception section are inserted into the shaft, and the electric signal cable and the optical fiber cable are connected to the connector section 13. The diagnostic imaging probe 11 can be advanced and retreated inside the guiding catheter 60, and the shaft and the sensor unit 12 can be advanced and retreated inside the sheath of the diagnostic imaging probe 11 and can also rotate in the circumferential direction. The intravascular inspection device 1 has a drive unit (MDU) to which the imaging catheter 10 is detachably attached by a connector unit 13, and controls the operation of the diagnostic imaging probe 11 inserted into the blood vessel 100 by driving a built-in motor in response to an operation by a medical professional. For example, the MDU performs a pull-back operation to rotate the diagnostic imaging probe 11 in the circumferential direction while pulling it at a constant speed. The diagnostic imaging probe 11 rotates while moving from the tip side to the base end by the pull-back operation, and scans the inside of the blood vessel 100 continuously at a predetermined time interval, thereby continuously capturing multiple IVUS images or OCT images approximately perpendicular to the diagnostic imaging probe 11.

[0018] When performing the flushing operation, the medical staff attaches the syringe 70 filled with the flushing liquid 72 to the side port 63 of the branch hub 62, places the tip of the diagnostic imaging probe 11 near the observation position, and then pushes in the plunger 71 of the syringe 70. As a result, the flushing liquid 72 in the syringe 70 passes through the guiding catheter 60 and is ejected from the tip of the guiding catheter 60 into the blood vessel 100. The ejected flushing liquid 72 flushes out the blood in the area, filling the area with the flushing liquid 72. In this state, an OCT image is captured by the diagnostic imaging probe 11 (sensor unit 12), thereby obtaining an OCT image that is not affected by blood. The plunger 71 is pushed in using, for example, an autoinjector.

[0019] 3 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a control unit 31, a main memory unit 32, an input / output I / F 33, an auxiliary memory unit 34, and a reading unit 35. The control unit 31 is configured using one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a tensor processing unit (TPU), etc. The control unit 31 is connected to each hardware unit constituting the image processing device 3 via a bus.

[0020] The main memory unit 32 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data necessary for the control unit 31 to execute arithmetic processing.

[0021] The input / output I / F 33 is an interface to which the intravascular inspection device 1, the fluoroscopic image capturing device 2, the display device 4, and the input device 5 are connected. The control unit 31 acquires an IVUS image or an OCT image from the intravascular inspection device 1, and acquires an angio image from the fluoroscopic image capturing device 2, via the input / output I / F 33. The control unit 31 also displays a medical image on the display device 4 by outputting a medical image signal of the IVUS image, the OCT image, or the angio image to the display device 4 via the input / output I / F 33. Furthermore, the control unit 31 accepts information input to the input device 5 via the input / output I / F 33.

[0022] The auxiliary storage unit 34 is a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The auxiliary storage unit 34 stores the computer program P executed by the control unit 31 and various data required for the processing of the control unit 31. The auxiliary storage unit 34 also stores a first learning model M1, a second learning model M2, and a medical DB 34a, which will be described later. The first learning model M1 is a machine learning model that has learned predetermined training data, and is a model that takes an IVUS image as an input and outputs a region of the vascular lumen in the input IVUS image. The second learning model M2 (second learning model) is a machine learning model that has learned predetermined training data, and is a model that takes blood vessel information, treatment information, and patient information as input and outputs flash information related to a flash operation. The first learning model M1 and the second learning model M2 are expected to be used as program modules that constitute artificial intelligence software. The medical DB 34a is a database that stores medical data of patients. The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P may be written into the auxiliary memory unit 34 during the manufacturing stage of the image processing device 3, or it may be distributed by a remote server device and received by the image processing device 3 via communication and stored in the auxiliary memory unit 34.

[0023] The reading unit 35 reads data stored in a recording medium 30 such as a CD (Compact Disk), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. The computer program P may be recorded in a readable manner on the recording medium 30, or the reading unit 35 may read the computer program P from the recording medium 30 and store it in the auxiliary storage unit 34. The computer program P may also be recorded in a semiconductor memory, and the control unit 31 may read the computer program P from the semiconductor memory and execute it.

[0024] The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 is described as being one computer.

[0025] In the image processing device 3 of this embodiment, the control unit 31 executes a process of detecting the size of the lumen of the captured blood vessel based on the IVUS image captured by the intravascular inspection device 1 by reading and executing the computer program P stored in the auxiliary storage unit 34. The control unit 31 also executes a process of generating flash information regarding the flash operation performed when capturing an OCT image based on the detected size of the blood vessel lumen (blood vessel information), information regarding a treatment device used for intravascular treatment by the intravascular inspection device 1 (treatment information), and patient information (patient information). Therefore, the image processing device 3 of this embodiment can present the flash information regarding the flash operation when capturing an OCT image to the surgeon. Note that, in the image processing device 3 of this embodiment, the first learning model M1 is used when detecting the blood vessel lumen in the IVUS image, and the second learning model M2 is used when generating operation information of the flash operation from the blood vessel information, treatment information, and patient information.

[0026] FIG. 4 is an explanatory diagram showing an example of the configuration of the medical DB 34a. The medical DB 34a includes a medical ID column, a patient information column, a treatment information column, and an image information column. The medical ID column stores a medical ID for identifying each medical data. The patient information column, the treatment information column, and the image information column store, in association with the medical ID, patient information on the patient to be treated, treatment information on the treatment performed on the patient and the treatment to be performed, and medical images obtained by examining the patient. The patient information is information on the patient who has been treated or is to be treated, and is a medical record of the patient. The patient information includes the patient's age, sex, weight, height, renal function level (e.g., a value obtained by evaluating the Cr value on a five-point scale), diagnosis, medical history, etc. The patient information may also include a medical history, a history of taking medicine, blood test results, etc. The treatment information is information on the intravascular treatment performed on the patient or the intravascular treatment to be performed. The treatment information includes, for example, the date of treatment or the date of planned treatment, the position of the treated lesion (lesion site), the size (diameter, length, volume) and shape of the blood vessel lumen at the lesion, the type (e.g., product name) of the guiding catheter 60 used or the guiding catheter 60 to be used, the type of the imaging catheter 10 used or the imaging catheter 10 to be used, the diameter and length, etc. The treatment information also includes flash information such as the type of flash liquid (e.g., contrast agent) used in the flash operation when taking an OCT image, the dilution rate, the dosage (flash amount), and the flow rate (flash flow rate). The treatment information also includes, in association with the flash information, the success probability when the flash operation is performed under each condition included in the flash information. The success probability is a value registered by the surgeon who performed the flash operation and took the OCT image, and is registered by the surgeon after taking the OCT image, for example. The success of the flash operation means that the surgeon judges that an OCT image that can be sufficiently used for image diagnosis has been taken, and the success probability indicates the surgeon's satisfaction level, for example, on a scale of 100 points. The treatment information also includes the pullback speed and travel distance when capturing an IVUS image or an OCT image, and the type, dilution rate, and dosage of the contrast agent used when capturing an angio image with the fluoroscopic image capture device 2 .Furthermore, the treatment information may include the characteristics of the lesion, the puncture site of the guiding catheter 60, the time of taking the fluoroscopic image, the type, diameter, length and total number of placed stents, the type, length, maximum expansion diameter, maximum expansion pressure, expansion time and attenuation time, progress information after treatment (e.g., the presence or absence of complications), etc. The medical images are images of the patient's blood vessels, and include angio images, IVUS images, and OCT images. The medical images may also include computed tomography images (CT images), magnetic resonance images (MRI images), etc.

[0027] FIG. 5 is an explanatory diagram showing an overview of the learning models M1 and M2. FIG. 5A shows the first learning model M1, and FIG. 5B shows the second learning model M2. The first learning model M1 is a model that recognizes a predetermined object included in an IVUS image. The first learning model M1 is a model that can classify objects in an image pixel by pixel, for example, by semantic segmentation. The first learning model M1 of this embodiment is a machine learning model that inputs one IVUS image, and has been trained to recognize the vascular lumen included in the IVUS image, and outputs the recognition result. Specifically, the first learning model M1 classifies each pixel of the input IVUS image into a region of the vascular lumen and other regions, and outputs a classified IVUS image (hereinafter referred to as a label image) in which each pixel is associated with a label for each region. The first learning model M1 can be configured, for example, by U-Net, FCN (Fully Convolutional Network), SegNet, etc.

[0028] The first learning model M1 has an input layer, an intermediate layer, and an output layer (not shown), and the intermediate layer has a convolution layer, a pooling layer, and a deconvolution layer. The convolution layer extracts image features from pixel information of an image input through the input layer to generate a feature map, and the pooling layer compresses the generated feature map. The deconvolution layer enlarges (maps) the feature map generated by the convolution layer and the pooling layer to the original image size. The deconvolution layer identifies which object exists in which position in the image on a pixel basis based on the features extracted by the convolution layer, and generates a label image indicating which object each pixel corresponds to. As shown on the right side of FIG. 5A, the label image output from the first learning model M1 is an image in which each pixel of the IVUS image is classified into a region of the blood vessel lumen and a region other than the region, and a pixel value according to each region is assigned. In FIG. 5A, the pixels classified into the region of the blood vessel lumen are shown by hatching.

[0029] The first learning model M1 having the above-mentioned configuration can be generated by preparing training data including a training IVUS image and a label image in which data indicating an object to be discriminated (here, a blood vessel lumen) is labeled for each pixel in the IVUS image as shown on the right side of FIG. 5A, and using this training data to perform machine learning on an untrained learning model. In the training label image, a coordinate range corresponding to the area of ​​each object and a label indicating the type of each object are assigned to the training IVUS image. When an IVUS image included in the training data is input, the first learning model M1 learns to output the label image included in the training data. Specifically, the first learning model M1 performs calculations in the intermediate layer based on the input IVUS image, and obtains a detection result in which each object (here, a blood vessel lumen) in the IVUS image is detected. More specifically, the first learning model M1 obtains, as an output, a label image in which a value indicating the type of classified object is labeled for each pixel in the IVUS image. The first learning model M1 then compares the obtained detection result (label image) with the coordinate range of the correct object area and the object type indicated by the training data, and optimizes parameters such as the weights (coupling coefficients) between neurons so that the two are similar. There are no particular limitations on the method of optimizing the parameters, but the steepest descent method, backpropagation method, etc. can be used. This allows the first learning model M1 to be obtained that outputs a label image indicating the area of ​​the blood vessel lumen in the IVUS image when an IVUS image is input.

[0030] The image processing device 3 prepares such a first learning model M1 in advance and uses it to detect the vascular lumen in the IVUS image. The first learning model M1 only needs to be able to identify the position and shape of the vascular lumen in the IVUS image. In this embodiment, the first learning model M1 only needs to be able to recognize the vascular lumen in the IVUS image, but may be configured to recognize the vascular wall (tunica media and plaque) and the catheter area in addition to the vascular lumen. The learning of the first learning model M1 may be performed by another learning device. The trained first learning model M1 generated by learning by the other learning device is downloaded from the learning device to the image processing device 3, for example, via a network or via the recording medium 30, and stored in the auxiliary storage unit 34.

[0031] The second learning model M2 receives as input vascular information on the vascular lumen detected from the IVUS image using the first learning model M1, treatment information on the intravascular treatment to be performed on the patient, and patient information stored in the medical DB 34a. If any of the items of vascular information, treatment information, and patient information cannot be prepared, default information set in advance for each item may be input to the second learning model M2. The second learning model M2 outputs the estimation results estimated for each item as shown on the right side of FIG. 5B as flash information. The vascular information includes, for example, the diameter, length, and volume of a lesion (treatment target area) in the blood vessel. The vascular information may also include the position of the tip of the guiding catheter 60 relative to the coronary artery, and the engagement state between the tip of the guiding catheter 60 and the coronary artery. The length of the lesion can be, for example, a pullback distance in a pullback operation when the intravascular inspection device 1 captures an IVUS image. The position of the tip of the guiding catheter 60 and the engagement state with the coronary artery can be detected based on an angio image captured by the fluoroscopic image capture device 2. The pullback distance can be calculated based on the number of IVUS images captured by one pullback operation and the pullback speed.

[0032] The treatment information includes, for example, the position of the lesion (for example, the type of blood vessel). The treatment information also includes catheter information related to the catheter, such as the type (for example, product name) and shape (diameter and length) of the guiding catheter 60 to be used, the position where the tip of the guiding catheter 60 is to be placed during the flushing operation, the type and shape (diameter and length) of the imaging catheter 10 to be used, and the position where the tip of the imaging catheter 10 is to be placed during the flushing operation. Note that the positions of the tips of the guiding catheter 60 and the imaging catheter 10 can be positions specified based on the angio image captured by the fluoroscopic image capture device 2. The treatment information also includes contrast agent information related to the contrast agent, such as the type of contrast agent to be used, the dilution rate, and the internal pressure limit of the syringe 70 used to administer the contrast agent. The treatment information also includes pullback information related to the pullback operation, such as the pullback speed and pullback distance in the pullback operation performed when capturing the IVUS image. The treatment information may also include the angio image of the patient. Furthermore, when an intravascular treatment is being performed, the treatment information may include the type, dilution rate, and dosage of contrast agent that has already been used, and the type, dilution rate, and dosage of contrast agent that will be used in the future. The patient information includes, for example, the patient's age (age group), weight, and renal function level. The treatment information and patient information described above can use each piece of information recorded in the medical DB 34a.

[0033] The flush information output by the second learning model M2 includes the type and dilution rate of the contrast agent used in the flush solution, the flow rate (flush flow rate) and dosage (flush flow rate) when administering the flush solution, and the success probability when the flush operation is performed under these conditions. The second learning model M2 is configured, for example, by a CNN (Convolutional Neural Network), but may also be configured using algorithms such as a decision tree, a random forest, an SVM (Support Vector Machine), or an RNN (Recurrent Neural Network), or may be configured by combining multiple algorithms.

[0034] The second learning model M2 can be generated by preparing training data in which training vascular information, treatment information, and patient information are associated with correct flash information, and using this training data to perform machine learning on an untrained learning model. The training vascular information, treatment information, and patient information can be medical data of patients who have undergone endovascular treatment, and for example, each piece of information stored in the medical DB 34a can be used. The correct flash information can also be flash information included in the medical data of patients. The image processing device 3 inputs the vascular information, treatment information, and patient information included in the training data to the second learning model M2, and trains the second learning model M2 to output the correct flash information included in the training data.

[0035] Specifically, the image processing device 3 inputs training vascular information, treatment information, and patient information to the second learning model M2, and acquires flash information of each item illustrated in FIG. 5B as an output from the second learning model M2. The image processing device 3 compares the output flash information with correct flash information included in the training data, and optimizes parameters such as weights (coupling coefficients) between neurons so that the two are close to each other. There is no particular limitation on the method of optimizing the parameters, but the steepest descent method, backpropagation method, etc. can be used. This allows for the acquisition of the second learning model M2 that outputs flash information when the patient's vascular information, treatment information, and patient information are input.

[0036] In addition, among the items shown in FIG. 5B, "type of contrast agent" is an item shown in the classification result, but the other items are items expressed by continuous values. In this way, the second learning model M2 simultaneously handles so-called classification problems and regression problems, and for example, all items are considered as regression problems and continuous values ​​are output, and items to be shown as classification results are converted from continuous values ​​to binary values. Alternatively, a separate output layer corresponding to each item may be provided, and the feature values ​​extracted in the intermediate layer may be input to each output layer, so that estimation of each item may be performed separately. Alternatively, the second learning model M2 itself may be prepared for each item, and estimation may be performed separately.

[0037] The image processing device 3 prepares such a second learning model M2 in advance and uses it to generate flash information for a patient who is to undergo new treatment. When generating flash information for a patient who is to undergo new treatment, the image processing device 3 captures an IVUS image of the patient with the intravascular inspection device 1 and detects the size of the vascular lumen from the obtained IVUS image using the first learning model M1. The image processing device 3 also generates flash information by adding the contents of the patient's planned treatment (treatment information) and patient information to the detected size of the vascular lumen (vascular information) and inputting the results into the second learning model M2. The image processing device 3 outputs the generated flash information to the display device 4 for display.

[0038] The second learning model M2 may be learned by another learning device. When learning is performed by another learning device, the learned second learning model M2 is downloaded from the learning device to the image processing device 3 via a network or a recording medium 30, for example, and stored in the auxiliary storage unit 34.

[0039] FIG. 6 is a schematic diagram showing an example of a screen of flash information. The screen shown in FIG. 6 displays two patterns of flash information to be presented to the surgeon: one pattern that emphasizes reducing the dosage (flash amount) of a flash liquid (for example, a contrast agent), and the other pattern that emphasizes reducing the flow rate (flash flow rate) of the flash liquid. The flash information shown in FIG. 6 includes the flash flow rate, the flash amount (total amount of contrast agent), and the success probability, and further includes cautions and advice regarding this flash information. The cautions and advice are stored in the auxiliary storage unit 34 in advance, for example, depending on whether the flash information emphasizes reducing the flash amount or the flash flow rate. Therefore, the control unit 31 can present the cautions and advice to the surgeon together with the flash information by reading out the cautions and advice according to the contents of the generated flash information from the auxiliary storage unit 34 and displaying them on the display screen.

[0040] As shown in the advice of pattern A in FIG. 6, by aligning the tip of the guiding catheter 60 with the blood vessel at an appropriate position during the flush operation, the amount of the flush liquid flowing out to other blood vessels can be reduced, and as a result, the amount of the flush can be reduced. As shown in the advice of pattern B in FIG. 6, by increasing the dilution rate of the contrast medium used in the flush liquid, the viscosity of the flush liquid is reduced, and the load on the blood vessel can be reduced. In this way, advice that can further reduce the amount of the flush or the flush flow rate can be presented to the surgeon. The type and dilution rate of the contrast medium may be included in the flash information and displayed. The image processing device 3 may be configured to identify optimal flash information from the patient's information and display only the identified flash information.

[0041] The surgeon identifies the optimal flash information based on each piece of information and precautions for each presented pattern of flash information. For example, the surgeon identifies the optimal pattern of flash information taking into consideration the patient's physical condition. The surgeon also identifies the optimal flash information taking into consideration the advice presented together with the flash information and the surgeon's own procedural skills. The surgeon then performs a flash operation with the identified flash information (flash conditions). Here, the surgeon sets the identified flash conditions (flash flow rate, flash amount, etc.) for, for example, an autoinjector, and performs injection of the flash liquid by the autoinjector. After performing the flash operation, the surgeon then performs OCT image capture.

[0042] The precautions and advice presented to the surgeon together with the flush information may be, for example, information advising the surgeon to reduce the total amount of flush by increasing the pullback speed if the flush information emphasizes reducing the amount of flush. Note that, if the pullback speed is increased, the time required for the pullback operation is shortened, making it possible to perform the flush operation using a smaller amount of flush liquid.

[0043] The process of learning the training data and generating the second learning model M2 will be described below. Fig. 7 is a flowchart showing an example of the process of generating the second learning model M2. The following process is performed by the control unit 31 of the image processing device 3 according to a computer program P stored in the auxiliary storage unit 34, but may be performed by another learning device.

[0044] The control unit 31 of the image processing device 3 acquires training data in which blood vessel information, treatment information, and patient information for one patient are associated with flash information regarding a flash operation performed on the patient (S11). The flash information includes a success probability for the flash operation under each condition included in the flash information. The blood vessel information, treatment information, and patient information for the training data and the flash information for the training data can use each piece of information stored in the medical care DB 34a. Note that the training data may be registered in a training data DB (not shown) by preparing blood vessel information, treatment information, and patient information for the training data, and flash information in advance. In this case, the control unit 31 may acquire the training data from the training data DB.

[0045] The control unit 31 performs a learning process for the second learning model M2 using the acquired training data (S12). Here, the control unit 31 inputs the vascular information, treatment information, and patient information contained in the training data to the second learning model M2, and acquires output values ​​for each item of the flash information. The control unit 31 compares the output values ​​for each item with the information for each item of the correct answer, and optimizes parameters such as weights between neurons so that the two are close to each other.

[0046] The control unit 31 determines whether there is unprocessed data (S13). For example, when training data is registered in advance in the training data DB, the control unit 31 determines whether there is unprocessed training data in the training data stored in the training data DB. If it is determined that there is unprocessed data (S13: YES), the control unit 31 returns to the process of step S11, and the learning process performs the processes of steps S11 to S12 based on the unprocessed training data. If it is determined that there is no unprocessed data (S13: NO), the control unit 31 ends the series of processes.

[0047] The above-mentioned process provides a second learning model M2 that has been trained to output flash information when the patient's vascular information, treatment information, and patient information are input. The second learning model M2 can be further optimized by repeatedly performing the learning process using the training data as described above. The second learning model M2 that has already been trained can also be retrained by performing the above-mentioned process, in which case a second learning model M2 with higher discrimination accuracy can be obtained.

[0048] FIG. 8 is a flowchart showing an example of a procedure for presenting flash information. The control unit 31 of the image processing device 3 performs the following process according to the computer program P stored in the auxiliary storage unit 34. In the image diagnostic system of this embodiment, when a tomographic image of a blood vessel is captured during intravascular treatment, the intravascular inspection device 1 is first used in IVUS mode to capture an IVUS image. Then, as necessary, the intravascular inspection device 1 is used in OCT mode to capture an OCT image. At that time, the image processing device 3 generates conditions (flash information) for a flash operation to be performed when capturing an OCT image based on the previously captured IVUS image, and presents the conditions (flash information) to the surgeon. The surgeon can select the contents of the flash operation to be performed (flash information) in consideration of the presented flash information.

[0049] The control unit 31 (acquisition unit) of the image processing device 3 acquires an IVUS image of a patient's blood vessel captured from inside the blood vessel from the intravascular inspection device 1 (S21). Then, the control unit 31 generates flash information based on the acquired IVUS image. Note that the control unit 31 may execute a process of generating flash information based on the IVUS image when, for example, an instruction to capture an OCT image is received from the operator via the input device 5 after the process of step S21.

[0050] When generating flash information, the control unit 31 first inputs the IVUS image to the first learning model M1, and detects the blood vessel lumen in the IVUS image based on the output image from the first learning model M1 (S22). The control unit 31 (calculation unit) then calculates the diameter of the blood vessel lumen based on the detected blood vessel lumen (S23). The diameter of the blood vessel lumen is, for example, the minimum and maximum values ​​of the blood vessel inner diameter passing through the center of the blood vessel lumen, and the control unit 31 calculates the diameter based on, for example, the number of pixels on the blood vessel inner diameter in the IVUS image. The control unit 31 (calculation unit) also calculates the length of the blood vessel lumen photographed by IVUS based on the detected blood vessel lumen (S24). The length of the blood vessel lumen is the length in the running direction (longitudinal direction) of the blood vessel, and is, for example, the pullback distance due to the pullback operation performed when photographing the IVUS image. The pullback distance can be calculated based on the number of IVUS images photographed by one pullback operation and the pullback speed. Furthermore, the control unit 31 (calculation unit) calculates the volume of the vascular lumen based on the detected vascular lumen (S25). The volume of the vascular lumen is a volume calculated from the cross-sectional area and length of the vascular lumen. The cross-sectional area of ​​the vascular lumen may be calculated based on the calculated diameter of the vascular lumen, or may be calculated based on the number of pixels in the vascular lumen in the IVUS image. For example, the control unit 31 calculates the cross-sectional area of ​​the vascular lumen in one IVUS image, and calculates the volume of the vascular lumen in one IVUS image based on the distance between adjacent IVUS images (the distance between the shooting positions of the IVUS images). Then, the control unit 31 calculates the volume of the vascular lumen photographed by IVUS by adding up the volumes of the vascular lumen in each IVUS image. By such processing, the control unit 31 calculates vascular information related to the vascular lumen, such as the diameter, length, and volume of the vascular lumen photographed by the IVUS image. The calculation processing of each of these pieces of information can be executed by the calculation processing performed by the conventionally used intravascular inspection device 1 or image processing device 3. In addition, the control unit 31 detects the position of the tip of the guiding catheter 60 and the position of the tip of the diagnostic imaging probe 11 at this point based on, for example, an angio image taken by the fluoroscopic imaging device 2, and uses the detected positions as vascular information.

[0051] The control unit 31 (identification unit) reads out the patient's treatment information and patient information from the medical DB 34a, inputs the calculated vascular information and the read out patient's treatment information and patient information to the second learning model M2, and generates flush information based on the output information from the second learning model M2 (S26). The control unit 31 outputs the generated flush information to the display device 4, and displays the flush information on a screen such as that shown in FIG. 6 (S27). For example, a plurality of patterns of flush information are displayed as shown in FIG. 6, and the surgeon selects the flush information to be executed from the displayed flush information. Based on the selected flush information, the surgeon performs a flush operation by, for example, mounting the syringe 70 filled with the selected flush liquid 72 on the autoinjector, setting the flush flow rate and flush amount, etc. The control unit 31 accepts the surgeon's selection of any of the flush information via, for example, the input device 5 (S28), and when the selection is accepted, stores the selected flush information in the medical DB 34a (S29).

[0052] The autoinjector executes a flash operation based on the flash conditions set by the surgeon, and then the surgeon captures an OCT image using the intravascular inspection device 1. The control unit 31 acquires the OCT image from the intravascular inspection device 1 (S30), and outputs the acquired OCT image to the display device 4 for display (S31). The surgeon observes the state of the blood vessel using the displayed OCT image, and performs intravascular treatment as necessary. In intravascular treatment, the surgeon temporarily removes the imaging catheter 10 from the guiding catheter 60, and inserts a treatment catheter such as a balloon catheter into the guiding catheter 60 to perform the procedure.

[0053] When the OCT image captured by the intravascular inspection device 1 is displayed on the display device 4, the control unit 31 acquires, for example, information regarding the evaluation of the OCT image from the surgeon via the input device 5 (S32). The evaluation information on the OCT image is, for example, information indicating whether the OCT image is a clear frame that can be used sufficiently for image diagnosis, and may be, for example, information indicating that the OCT image is a clear frame (OK) or not a clear frame (NO). The evaluation information on the OCT image may also be the surgeon's satisfaction level when using the image diagnosis (for example, a 5-point scale or a 100-point scale). The control unit 31 may also determine whether the OCT image has been retaken by the intravascular inspection device 1, and if the image has been retaken, give a low evaluation (for example, 1 out of 5) to the flash information at that time, and if the image has not been retaken, give a high evaluation (for example, 5 out of 5) to the flash information at that time. Furthermore, the control unit 31 may extract OCT images (clear frames) that have been evaluated by the surgeon as being above a predetermined level, or OCT images (clear frames) in which the image quality or brightness of the lesion is above a predetermined level, from among a plurality of OCT images captured by one pullback operation, calculate the ratio of clear frames, and use the calculated ratio as the evaluation information. The determination of whether the OCT image is a clear frame or not may be performed using a learning model (neural network) constructed by machine learning. For example, a learning model may be used that is composed of CNN and that has been trained to output information indicating whether the OCT image is a clear image that can be used sufficiently for image diagnosis when the OCT image is input. In this case, the image processing device 3 inputs the OCT image into a trained learning model, and can determine whether the OCT image is a clear frame or not based on the output information from the learning model. The evaluation information thus assigned to each flash information may indicate the success probability of the flash operation, and the success probability may be calculated from such evaluation information. When the control unit 31 acquires the evaluation information for the flash information, the control unit 31 stores the acquired evaluation information (probability of success) in the medical DB 34a in association with the flash information.This allows the flash information and evaluation information on the flash information stored in the medical DB 34a to be used as training data for learning the second learning model M2.

[0054] By the above-mentioned process, in this embodiment, the conditions (flash information) for the flash operation to be performed when capturing an OCT image are generated based on the IVUS image, the patient's treatment information, and the patient information. Therefore, the occurrence of a flash failure in which blood cannot be properly removed by the flash operation is suppressed, and the possibility of performing an appropriate flash operation is improved. If an appropriate flash operation can be performed, the occurrence of redoing the flash operation and retaking the OCT image, etc. is suppressed, and an extension of the procedure time can be avoided. In addition, when performing a flash operation using a contrast agent as the flush liquid, an increase in the dosage of the contrast agent due to redoing the flash operation, more precisely, an increase in the dosage of iodine contained in the contrast agent, can be avoided, and the impact on the patient's renal function can be minimized, and the burden on the renal function can be reduced.

[0055] In the intravascular inspection device 1 using the dual type imaging catheter 10 as in this embodiment, after intravascular treatment, for example after placing a stent in a lesion, an OCT image is taken to confirm the placement position of the stent. Even in this case, an IVUS image is taken before taking the OCT image, flash information is generated from the obtained IVUS image, and the generated flash information can be presented to the surgeon. Even in such a case, the surgeon can select the optimal flash conditions by specifying the content of the flash operation to be performed based on the presented flash information.

[0056] FIG. 9 is an explanatory diagram showing a modified example of the learning model. The first learning model M1 and the second learning model M2 described above may be configured as one learning model as shown in FIG. 9. The learning model M3 shown in FIG. 9 has a configuration in which, for example, a plurality of IVUS images taken by one pullback operation, the patient's treatment information, and the patient information are input, and flash information similar to that of the second learning model M2 described above is output. The learning model M3 shown in FIG. 9 has, for example, a CNN, and extracts image features from the input IVUS image by the CNN. The features extracted here correspond to vascular information related to the vascular lumen in the IVUS image. Therefore, the learning model M3 can generate flash information based on the features extracted by the CNN and the input treatment information and patient information. Even when configured as one learning model M3 in this way, flash information can be generated from the IVUS image, the patient's treatment information, and the patient information, and the generated flash information can be presented to the surgeon.

[0057] In this embodiment, the process of generating each piece of information for the flash operation from the patient's vascular information, treatment information, and patient information may be configured to be performed on a rule-based basis instead of using the second learning model M2. In this case, for example, a table is prepared in which optimal flash information is associated with each combination of vascular information, treatment information, and patient information, and the image processing device 3 identifies flash information corresponding to the patient's vascular information, treatment information, and patient information from the contents of the table. In this case, optimal flash information corresponding to the patient can also be presented to the surgeon.

[0058] In this embodiment, the image processing device 3 locally performs a process of generating flash information based on the IVUS image, the patient's treatment information, and the patient information using the first learning model M1 and the second learning model M2, but is not limited to this configuration. For example, a server may be provided that performs a process of generating flash information using the first learning model M1 and the second learning model M2. In this case, the image processing device 3 may be configured to transmit the IVUS image, the treatment information, and the patient information to the server and acquire the flash information generated by the server. In addition, if the server here is a server that stores the medical DB 34a, the image processing device 3 can transmit only the IVUS image to the server and acquire the flash information generated by the server. Even in such a configuration, the same process as in this embodiment is possible and the same effect can be obtained.

[0059] In this embodiment, the second learning model M2 may be prepared for each treatment situation, such as when emphasizing reduction in the amount of administered flush liquid (contrast medium) or reduction in the flow rate of the flush liquid. The second learning model M2 may also be prepared for each site (e.g., type of blood vessel) to be processed by the flush operation, or for each position of the tip of the guiding catheter 60 relative to the blood vessel. For example, a model for the right coronary artery, a model for the left anterior descending artery, a model for the circumflex artery, a model for when the tip of the guiding catheter 60 is sufficiently engaged with the coronary artery, and a model for when the engagement state is shallow may be prepared. In this case, the image processing device 3 may present the flash information generated using each model to the surgeon. The image processing device 3 may also select a model to be used according to the type of blood vessel to be processed or a setting input by the surgeon, and generate flash information using the selected model to present to the surgeon.

[0060] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0061] 1. Intravascular inspection equipment 2. Fluoroscopic imaging device 3. Image Processing Device 4 Display device 5 Input Devices 10 Imaging Catheter 11 Diagnostic imaging probes 60 Guiding Catheter 61 Sheath 62 Branch Hub 70 Syringe 34a Medical DB M1 First learning model M2 Second Learning Model

Claims

1. Calculating information about the lumen of the blood vessel based on an ultrasonic tomographic image obtained by capturing a cross section of the blood vessel using the ultrasonic signal; determining operation information regarding a flash operation when photographing a cross section of the blood vessel using an optical signal based on the calculated information regarding the lumen, and a success probability of the flash operation based on the operation information; The specified operation information and the success probability are output to a display device. A program that causes a computer to carry out processing.

2. The acquired ultrasonic tomographic image is input to a learning model that has been trained to output information indicating a region of a lumen of a blood vessel in the ultrasonic tomographic image when the ultrasonic tomographic image is input, and the region of the lumen of a blood vessel in the ultrasonic tomographic image is output; Calculating information about the lumen based on the output lumen area The program according to claim 1 , which causes the computer to execute a process.

3. The information about the lumen includes a cross-sectional diameter of the lumen and a longitudinal length of the lumen; Obtaining a plurality of ultrasonic tomographic images by photographing a cross section of the blood vessel at a plurality of locations using ultrasonic signals; Based on the number of acquired ultrasonic tomographic images, the length of the lumen of the blood vessel in the long axis direction is calculated.

3. The program according to claim 1, which causes the computer to execute a process.

4. Identifying the operation information including a flow rate and a flow velocity of a flushing liquid that replaces the blood in the blood vessel during the flushing operation. A program according to any one of claims 1 to 3.

5. Inputting the calculated information about the inner cavity into a second learning model that has been trained to output operation information about the flash operation and a success probability for the flash operation based on the operation information when information about the inner cavity is input, and acquiring operation information about the flash operation and a success probability for the flash operation based on the operation information; The acquired operation information and the success probability are output to the display device.

5. The program according to claim 1, which causes the computer to execute a process.

6. The second learning model is trained to output operation information regarding the flush operation and a success probability for the flush operation based on the operation information when information regarding the lumen and patient information regarding a patient are input, Obtaining patient information about the patient; The calculated information about the lumen and the acquired patient information are input to the second learning model, and operation information about the flush operation and a success probability for the flush operation based on the operation information are output. The program according to claim 5, which causes the computer to execute a process.

7. Calculating information about the lumen of the blood vessel based on an ultrasonic tomographic image obtained by capturing a cross section of the blood vessel using the ultrasonic signal; determining operation information regarding a flash operation when photographing a cross section of the blood vessel using an optical signal based on the calculated information regarding the lumen, and a success probability of the flash operation based on the operation information; The specified operation information and the success probability are output to a display device. An image processing method in which the processing is performed by a computer.

8. a calculation unit that calculates information about a lumen of a blood vessel based on an ultrasonic tomographic image obtained by photographing a cross section of the blood vessel using an ultrasonic signal; a determination unit that determines operation information regarding a flash operation when capturing an image of a cross section of the blood vessel using an optical signal based on the calculated information about the lumen, and a success probability of the flash operation based on the operation information; an output unit that outputs the specified operation information and the success probability to a display device; An image processing device comprising:

9. acquiring training data that associates information about the lumen of a patient's blood vessel, operation information about a flash operation when photographing a cross section of the blood vessel using an optical signal, and a success probability of the flash operation based on the operation information; Using the training data, a trained learning model is generated that outputs operation information regarding the flush operation and a success probability for the flush operation based on the operation information when information regarding the lumen of a blood vessel is input. A model generation method in which processing is performed by a computer.

Images