Program, information processing method, and information processing apparatus.

Abstract

The present invention provides a program, etc., with which it is possible to easily establish the direction of a medical image.　In the present invention, the program causes a computer to execute a process of: acquiring a medical image obtained by imaging a lumen organ using a catheter; specifying, on the basis of the acquired medical image, a direction-of-interest indicating the direction of a site-of-interest with respect to the medical image; and displaying information indicating the specified direction-of-interest in association with the medical image or displaying the medical image that has been image-processed on the basis of the information indicating the direction-of-interest.

Description

Technical Field

[0001] The present invention relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, a catheter for image diagnosis that obtains a medical image inside a luminal organ such as a blood vessel using ultrasonic waves or light is known, and an image diagnosis apparatus that displays a medical image for image diagnosis generated using such a catheter is disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The direction of a medical image obtained by an image diagnosis apparatus changes depending on the bending state of the catheter or the like. It is important for a doctor or the like to correctly grasp the direction of the medical image when making a diagnosis. However, in order to grasp the direction of the medical image, the ability to correctly interpret the drawn medical image is required, so long-term training is necessary.

[0005] An object of the present disclosure is to provide a program or the like that can easily grasp the direction of a medical image.

Means for Solving the Problems

[0006] A program according to one aspect of this disclosure acquires a medical image of a tubular organ using a catheter, identifies a direction of interest indicating the direction of a site of interest relative to the medical image based on the acquired medical image, and causes a computer to perform a process of displaying the identified information indicating the direction of interest in association with the medical image, or displaying the medical image processed based on the information indicating the direction of interest. [Effects of the Invention]

[0007] According to this disclosure, the orientation of medical images can be easily determined. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram showing an example of the configuration of an imaging diagnostic device. [Figure 2] This is a block diagram showing an example of the configuration of an image processing device. [Figure 3] This is an explanatory diagram illustrating the overview of the learning model. [Figure 4] This flowchart shows an example of a processing procedure performed by an image processing device. [Figure 5] This is a schematic diagram showing an example of a screen displayed on a display device. [Figure 6] This is an explanatory diagram illustrating the overview of the learning model in the second embodiment. [Figure 7] This flowchart shows an example of a processing procedure performed by the image processing device in the second embodiment. [Modes for carrying out the invention]

[0009] Specific examples of the computer program, image processing method, and image processing apparatus according to the embodiments of this disclosure will be described below with reference to the drawings.

[0010] (First Embodiment) Figure 1 is an explanatory diagram showing an example of the configuration of the diagnostic imaging device 100. The diagnostic imaging device 100 according to this embodiment comprises an intravascular ultrasound device 101, an angiography device 102, an image processing device (information processing device) 3, a display device 4, and an input device 5. The diagnostic imaging device 100 is a device unit for imaging the tubular organs of a patient. In this embodiment, cardiac catheterization, which is an endovascular treatment, will be described as an example, but the tubular organs targeted by catheterization are not limited to blood vessels, and may be other tubular organs such as the bile duct, pancreatic duct, bronchi, or intestines.

[0011] The intravascular ultrasound device 101 is a device for performing intravascular ultrasound examination and diagnosis by generating IVUS images (medical images) including ultrasound tomographic images, which are cross-sectional views of the patient's blood vessels, using, for example, the intravascular ultrasound (IVUS) method. The intravascular ultrasound device 101 is equipped with a catheter 1 and an MDU (Motor Drive Unit) 2.

[0012] Catheter 1 is an imaging diagnostic catheter for obtaining ultrasound images of blood vessels using the IVUS method. The ultrasound image is an example of a catheter image generated using catheter 1. Catheter 1 has a probe 11 and a connector 12 located at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector 12. A shaft 13 is inserted inside the probe 11. A sensor 14 is connected to the tip of the shaft 13.

[0013] The sensor unit 14 is an ultrasonic transducer that transmits ultrasonic waves based on pulse signals within blood vessels and receives reflected waves reflected by the biological tissue of blood vessels or medical equipment. The sensor unit 14 and shaft 13 can move forward and backward within the probe 11 and can also rotate in the circumferential direction. The sensor unit 14 and shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation.

[0014] The tip of the probe 11 is provided with a guidewire insertion section 15 through which a guidewire can be inserted. The guidewire insertion section 15 constitutes a guidewire lumen and is used to receive a guidewire that has been previously inserted into the blood vessel, and to guide the probe 11 to the affected area using the guidewire. The centerline of the tubular section of the guidewire lumen and the centerline of the tubular section of the probe 11 are separated by a predetermined distance.

[0015] Furthermore, the catheter 1 has an X-ray-insensitive marker 16 to determine the positional relationship between the IVUS image obtained by the intravascular ultrasound device 101 and the angiography image obtained by the angiography device 102. In the example shown in Figure 1, the marker 16 is provided at the tip of the probe 11. When the catheter 1 configured in this way is imaged with X-rays, an angiography image, which is an X-ray fluoroscopic image including the image of the marker 16, is obtained. Note that the position of the marker 16 is just an example; the marker 16 may be provided on the shaft 13 or at a location other than the tip of the probe 11.

[0016] The MDU2 is a drive unit to which the catheter 1 is detachably attached. It controls the movement of the catheter 1 inserted into the blood vessel by driving a built-in motor in response to user operation. The MDU2 performs a pullback operation, rotating the shaft 13 and sensor unit 14 circumferentially while pulling them toward the MDU2 at a constant speed. The sensor unit 14 rotates while moving from the tip to the proximal end due to the pullback operation, continuously scanning the inside of the blood vessel at predetermined time intervals, and outputs the detected ultrasonic reflected wave data to the image processing device 3.

[0017] The image processing device 3 is a processing device that generates an IVUS image of a blood vessel based on the reflected wave data output from the probe 11 of the catheter 1. The image processing device 3 generates an image of one frame for each rotation of the sensor unit 14. The generated image is a cross-sectional image centered on the probe 11 and substantially perpendicular to the probe 11. Since the sensor unit 14 scans while moving inside the blood vessel, an image of one frame is generated at each position rotated once within the moving range (the range of one pull-back). That is, a plurality of frames of images are generated within the moving range. The image processing device 3 causes the display device 4 to display the generated ultrasonic tomographic image, and also accepts the input of various setting values when performing an examination via the input device 5.

[0018] The catheter 1 may be a catheter for generating an optical tomographic image, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which uses near-infrared light to generate an optical tomographic image. In this case, the sensor unit 14 is a transceiver unit that irradiates and receives reflected light of near-infrared light. The catheter 1 may have both the ultrasonic transducer and the transceiver unit for OCT or OFDI as the sensor unit 14, and is for generating a medical image including both an ultrasonic tomographic image and an optical tomographic image.

[0019] The angiography device 102 is an imaging device for obtaining an angiogram (medical image), which is a fluoroscopic image of a patient's blood vessel, by imaging the blood vessel using X-rays from outside the patient's body while injecting a contrast agent into the patient's blood vessel. The angiography device 102 includes an X-ray source and an X-ray sensor, and images the X-ray fluoroscopic image of the patient by the X-ray sensor receiving the X-rays irradiated from the X-ray source. The angiography device 102 outputs the obtained angiogram to the image processing device 3.

[0020] In addition, in this embodiment, the angiography apparatus 102 that mainly captures two-dimensional angiographic images is taken as an example for explanation. However, the apparatus is not particularly limited as long as it can capture the patient's luminal organs and the catheter 1 from different directions outside the body. For example, three-dimensional CT angiography, magnetic resonance (MRI; Magnetic Resonance Imaging) images, etc. may also be used.

[0021] The display device 4 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display, etc., and the input device 5 is, for example, a keyboard, a mouse, a trackball or a microphone, etc. The display device 4 and the input device 5 may be integrally laminated to form a touch panel. Also, the input device 5 and the image processing device 3 may be integrally configured. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input, etc.

[0022] FIG. 2 is a block diagram showing a configuration example of the image processing device 3. The image processing device 3 is a computer and includes a control unit 31, a main memory unit 32, an input / output I / F 33, and an auxiliary storage unit 34.

[0023] The image processing device 3 may be a multi-computer configured to include a plurality of computers. Also, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 is described as being a single computer.

[0024] The control unit 31 is configured using one or more computing devices such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), and TPU (Tensor Processing Unit). The control unit 31 is connected to each hardware component of the image processing device 3 via a bus.

[0025] The main memory unit 32 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores the data necessary for the control unit 31 to perform calculation processing.

[0026] The input / output interface 33 is an interface to which the intravascular ultrasound device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images or angiography images via the input / output interface 33. The control unit 31 also outputs the medical image signals of the IVUS images or angiography images to the display device 4 via the input / output interface 33, thereby displaying the medical images on the display device 4. Furthermore, the control unit 31 receives information input to the input device 5 via the input / output interface 33.

[0027] The auxiliary storage unit 34 is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary storage unit 34 stores the program 3P executed by the control unit 31 and various data necessary for the processing of the control unit 31. The auxiliary storage unit 34 also stores the learning model 3M. Details of the learning model 3M will be described later.

[0028] The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The program 3P may be written to the auxiliary storage unit 34 during the manufacturing stage of the image processing device 3, or the image processing device 3 may acquire the program distributed by a remote server device via communication and store it in the auxiliary storage unit 34. The program 3P may also be recorded in a readable form on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, and a reading unit (not shown) may read it from the recording medium 30 and store it in the auxiliary storage unit 34.

[0029] The control unit 31 reads and executes the program 3P stored in the auxiliary storage unit 34 to acquire IVUS images generated by the medical imaging device 100, identify directions of interest in the IVUS images, and perform the process of displaying the IVUS images associated with the information indicating the identified directions of interest.

[0030] The direction of interest indicates the direction of the area of ​​interest that the physician or other medical professional is interested in. Based on the direction of interest in the IVUS image, the physician or other medical professional will determine the imaging direction (orientation) of the IVUS image relative to the patient. Below, we will explain an example where the area of ​​interest is the epicardium and the direction of interest is the direction of the epicardium. For example, in cardiac catheterization, it is important to correctly determine the direction of the patient's epicardium in order to predict cardiac tamponade, a serious complication. In other words, it is important to correctly determine the direction of the epicardium relative to the IVUS image. However, the orientation of the IVUS image changes constantly depending on the insertion state of catheter 1 in the blood vessel. Correctly determining the direction of the epicardium from the information depicted in the IVUS image requires skill and is not easy for inexperienced physicians. Furthermore, depending on the imaging conditions, information that can identify the direction of the epicardium may not be depicted in the IVUS image, making it even more difficult to determine the direction of the epicardium relative to such IVUS images. In this embodiment, the image processing device 3 identifies the direction of the epicardium, and by providing an IVUS image associated with information indicating the identified direction of the epicardium, physicians and others can easily recognize the direction of the epicardium.

[0031] The method for identifying the direction of interest in an IVUS image in this embodiment will now be described. The control unit 31 of the image processing device 3 detects landmarks included in the IVUS image using a learning model 3M and identifies the direction of interest in the IVUS image based on the detection results. A landmark is information that serves as a marker for identifying the direction of interest in an IVUS image. For example, landmarks include the site of interest, the shape of the tubular organ into which the catheter 1 is inserted, the structure of the side branches, organs such as the heart adjacent to the tubular organ, tubular organs surrounding the tubular organ, lesions, etc. If the direction of interest is the direction of the epicardium, landmarks include, for example, the epicardium which directly indicates the direction of the epicardium, and side branch blood vessels, veins, Trance-thoracic sinus, Triangle of Brocq-Mouchet, myocardial bridge, calcified lesions, guidewire, etc. which indirectly indicate the direction of the epicardium.

[0032] Figure 3 is an explanatory diagram illustrating the overview of the learning model 3M. The learning model 3M is a model that takes IVUS images as input and outputs information about predetermined objects (markers) contained in the IVUS images. Specifically, the learning model 3M accepts multiple frames of IVUS images as input, which are consecutive along the longitudinal direction of the blood vessel as the catheter 1 is scanned. The learning model 3M recognizes markers in the IVUS images of each frame that are consecutive along the time axis t.

[0033] The 3M learning model is, for example, a pre-trained convolutional neural network (CNN) using deep learning. The 3M learning model uses image recognition techniques with semantic segmentation to recognize, pixel by pixel, whether each pixel in the input image corresponds to an object region. The 3M learning model has an input layer that receives an IVUS image, an intermediate layer that extracts and reconstructs image features, and an output layer that outputs information indicating the location, range, and type of objects contained in the IVUS image. The 3M learning model is, for example, U-Net.

[0034] The input layer of the 3M learning model has multiple nodes that accept the pixel value of each pixel in the IVUS image as input, and passes the input pixel value to the hidden layer. The hidden layer has a convolutional layer (CONV layer) and a deconvolutional layer (DECONV layer). The convolutional layer is a layer that compresses the dimensions of the image data. Through dimensionality reduction, the features of the objects are extracted. The deconvolutional layer performs deconvolution and restores the original dimensions. The restoration process in the deconvolutional layer generates a label image that indicates whether each pixel in the IVUS image is an object or not, with a pixel value corresponding to the type of object. The output layer has multiple nodes that output the label image. For example, the label image is such that pixels corresponding to the first landmark (epidamidia) are class "1", pixels corresponding to the second landmark (collateral branch) are class "2", pixels corresponding to the third landmark (vein) are class "3", ..., and pixels corresponding to other images are class "0".

[0035] The learning model 3M may also include medical images other than IVUS images as input elements. Other medical images may include, for example, optical tomography images and angiography images taken at the same time as the IVUS images. The learning model 3M may also include information about the tubular organ into which catheter 1 is inserted as input elements. Information about the tubular organ may include, for example, the names of blood vessels such as the right coronary artery, left coronary artery, and LAD, and a blood vessel identification number (AHA classification). By inputting these as input elements into the learning model 3M, it is possible to recognize landmarks while also considering information other than IVUS images, thereby improving recognition accuracy.

[0036] The learning model 3M can be generated by preparing training data in which IVUS images containing objects (markers) are associated with label images indicating the position, range, and type of each object, and then using this training data to machine-learn an untrained neural network. Specifically, the control unit 31 inputs multiple IVUS images included in the training data into the input layer of the untrained neural network model, performs calculations in the hidden layer, and obtains an image output from the output layer. The control unit 31 then compares the image output from the output layer with the label images included in the training data and optimizes the parameters used in the calculations in the hidden layer so that the image output from the output layer approaches the label image. These parameters are, for example, the weights (connection coefficients) between neurons. The method of parameter optimization is not particularly limited, but for example, the control unit 31 uses backpropagation to optimize various parameters. The position of the markers in the training data may be, for example, a judgment made by a physician with specialized knowledge, which can be used as the correct label.

[0037] As shown in Figure 3, by inputting an IVUS image into the trained model 3M, a label image is obtained that shows various landmarks on a pixel-by-pixel basis.

[0038] The above example described a learning model 3M being a CNN, but the configuration of the learning model 3M is not limited; it just needs to be able to recognize the landmarks contained in the IVUS image. The learning model 3M could be, for example, R-CNN, Mask R-CNN, YOLO (You Only Look Once), or a model constructed using other learning algorithms such as support vector machines or regression trees that do not use neural networks.

[0039] The control unit 31 of the image processing device 3 identifies the direction of interest in the IVUS image using the detection result of landmarks. If the detected landmark is the epicardium, i.e., the detected landmark is the site of interest, the control unit 31 directly identifies the direction of the detected landmark as the direction of the epicardium. If the detected landmark is not the epicardium, i.e., the detected landmark is not the site of interest, the control unit 31 indirectly identifies the direction of the epicardium from the direction of other landmarks, based on the positional relationship between the epicardium and landmarks other than the epicardium (hereinafter referred to as "other landmarks") that are stored in advance. The direction of the epicardium is indicated, for example, using an angle with the 0 o'clock direction as the reference (0 degrees) for a roughly circular IVUS image.

[0040] For example, the control unit 31 stores in advance the rotation angle relative to other landmarks based on the positional relationship between the other landmarks and the epicardium. The rotation angle is the angle by which the probe 11 in the IVUS image is rotated circumferentially with respect to its center in order to calculate the direction of the epicardium from the direction of the other landmarks. Based on the detected other landmarks, the control unit 31 identifies the direction rotated by a predetermined angle from the other landmarks as the direction of the epicardium. For example, consider the case where a branch vessel of a vessel branching off from the vessel into which the catheter 1 was inserted is detected as another landmark. Based on the correspondence between the branch vessel stored in advance and the rotation angle determined by the positional relationship between the said vessel and the branch vessel, the control unit 31 calculates the direction of the epicardium by rotating the direction of the branch vessel by a predetermined angle.

[0041] The control unit 31 may use information other than the IVUS image to determine the direction of the epicardium. Information other than the IVUS image may include, for example, optical tomography images, angiography images, and information about the tubular organ into which catheter 1 is inserted, all acquired at the same time as the IVUS image. Information about tubular organs includes, for example, the names of blood vessels such as the right coronary artery, left coronary artery, and LAD, and the identification number of the blood vessel (AHA classification). For example, let's consider the case where a guidewire is detected as another landmark. The angiography image acquired at the same time as the IVUS image includes images of the blood vessel, catheter 1 (marker 16), and guidewire. Based on the direction of the guidewire in the IVUS image and the positions of the blood vessel, catheter 1 (marker 16), and guidewire in the angiography image, the control unit 31 determines the positional relationship between the IVUS image and the angiography image (the position and direction of the IVUS image relative to the angiography image). Based on the determined positional relationship and the imaging direction of the angiography image relative to the subject, the control unit 31 determines the direction of the IVUS image relative to the subject. The control unit 31 determines the direction of the epicardium relative to the IVUS image based on the direction of the identified IVUS image.

[0042] In this way, the control unit 31 directly or indirectly identifies the direction of the epicardium relative to the IVUS image (direction of interest) based on landmarks included in the IVUS image.

[0043] Next, the control unit 31 removes outliers from the identified epicardial directions. Specifically, the control unit 31 detects and removes epicardial directions that deviate from the majority of epicardial directions (outliers) based on the epicardial directions in a series of IVUS images arranged sequentially along the longitudinal direction of the blood vessel. The method for detecting outliers is not limited. As an example, the control unit 31 derives the deviation between the epicardial direction (angle) in each IVUS image (each frame) and the moving average of the time-series data of that epicardial direction (angle), and detects epicardial directions whose absolute value of the derived deviation is greater than or equal to a threshold as outliers. The control unit 31 removes the detected outliers from the identification results.

[0044] Furthermore, the control unit 31 estimates the direction of the epicardium for IVUS images in which no landmarks have been detected. Specifically, based on the time-series data of the direction (angle) of the epicardium from which the outliers have been removed, the control unit 31 interpolates the discontinuous portion in which the direction of the epicardium has not been obtained using a predetermined interpolation method (e.g., spline interpolation, linear interpolation, etc.). In this way, the control unit 31 obtains the direction of the epicardium for IVUS images in which no landmarks have been detected and the direction of the epicardium has not been identified. Since IVUS images acquired using catheter 1 do not necessarily contain landmarks, the direction of the epicardium can be assigned to all IVUS images by performing interpolation processing using preceding and succeeding frames. The above interpolation processing should be performed not only on IVUS images in which no landmarks have been detected, but also on IVUS images from which specific results have been removed as outliers.

[0045] Figure 4 is a flowchart showing an example of a processing procedure performed by the image processing device 3. The control unit 31 of the image processing device 3 performs the following processing according to program 3P. The control unit 31 may perform the following processing in real time whenever a medical image is output via the intravascular ultrasound device 101, for example, or it may perform the processing retrospectively at any time based on the recorded medical image.

[0046] The control unit 31 of the image processing device 3 acquires medical images, including IVUS images, via the intravascular ultrasound device 101 (step S11). Specifically, the control unit 31 acquires IVUS images generated based on the reflected ultrasound signals acquired via the intravascular ultrasound device 101. The control unit 31 functions as an acquisition unit that acquires medical images of blood vessels. In this case, the control unit 31 may also acquire optical tomography images along with the IVUS images, or it may acquire angiography images via the angiography device 102.

[0047] The control unit 31 inputs the acquired IVUS image as input data to the learning model 3M (step S12). The control unit 31 acquires a label image indicating the location, range, and type of landmark output from the learning model 3M (step S13). The control unit 31 determines whether the landmark is the epicardium, i.e., the site of interest (step S14). The control unit 31 may acquire and store sites of interest by, for example, accepting registration of sites of interest in advance.

[0048] If the control unit determines that the landmark is not the epicardium (step S14: NO), the control unit 31 identifies the direction (angle) of the epicardium relative to the IVUS image based on the positional relationship between the detected non-epidural landmark (another landmark) and the epicardium (step S15), and proceeds to step S17. For example, the control unit 31 refers to the correspondence between the direction of other landmarks and the direction of the epicardium, which is stored in advance, and identifies the direction of the epicardium as the direction obtained by rotating the direction of the other landmark by a predetermined angle based on the detected other landmark. In this case, the control unit 31 may also identify the direction of the epicardium using information other than the IVUS image, such as optical tomography, angiography, or information about the tubular organ into which the catheter 1 is inserted.

[0049] If the benchmark is determined to be the epicardium (step S14: YES), the control unit 31 identifies the direction of the epicardium relative to the IVUS image according to the detected epicardium (step S16). Depending on the position and extent of the detected epicardium, the control unit 31 identifies, for example, the angle passing through the circumferential center of the detected epicardium as the direction of the epicardium.

[0050] The control unit 31 determines whether the benchmark detection process for all acquired IVUS images has been completed (step S17). If it determines that the detection process for all IVUS images has not been completed (step S17: NO), the control unit 31 returns to step S12 and waits until the detection process for all IVUS images is completed.

[0051] If the control unit determines that the detection process for all IVUS images has been completed (step S17: YES), the control unit 31 detects outliers in the identified epicardial direction based on each detection result (step S18). Specifically, the control unit 31 derives the deviation between the epicardial direction in each IVUS image (each frame) and the moving average of the time-series data of that epicardial direction. The control unit 31 determines the relationship between the absolute value of each calculated deviation and a preset threshold, thereby detecting epicardial directions (outliers) where the absolute value of the calculated deviation is greater than or equal to the threshold. The control unit 31 removes the detected outliers from the identification results related to the epicardial direction, i.e., the time-series data of the epicardial direction (step S19).

[0052] The control unit 31 interpolates discontinuous portions where the epicardial direction is not obtained based on time-series data of epicardial direction from which outliers have been removed, using a predetermined interpolation method (e.g., spline interpolation) (step S20). The control unit 31 identifies the epicardial direction for IVUS images where the epicardial direction is not obtained, according to the interpolation result (step S21). IVUS images where the epicardial direction is not obtained include IVUS images where no benchmarks have been detected, and IVUS images from which the identified results have been removed as outliers. This identifies the epicardial direction for all IVUS images. The control unit 31 functions as an identification unit that identifies the epicardial direction for IVUS images.

[0053] The control unit 31 generates screen information to display IVUS images corresponding to the direction of each identified epicardium (step S22). Specifically, the control unit 31 generates screen information that displays the direction of each identified epicardium in association with the IVUS image, or screen information that displays IVUS images processed based on the direction of each identified epicardium. The control unit 31 displays a screen 40 based on the generated screen information on the display device 4 (step S23), and ends the series of processes. The control unit 31 functions as a display unit that displays the direction of the epicardium in association with the IVUS image, or displays IVUS images processed based on the direction of the epicardium.

[0054] Figure 5 is a schematic diagram showing an example of a screen 40 displayed on the display device 4. The screen 40 includes a total image display unit 41 that displays multiple IVUS images corresponding to multiple frames in chronological order, and a selected image display unit 42 that displays IVUS images selected by a physician or other person from among the multiple IVUS images. The selected image display unit 42 displays information indicating the direction of the epicardium relative to any of the IVUS images in association with that IVUS image. In the example shown in Figure 5, the selected image display unit 42 overlays a graphic object (arrow) indicating the direction of the epicardium onto the IVUS image. Furthermore, the selected image display unit 42 overlays landmarks detected from the IVUS image onto the IVUS image.

[0055] The control unit 31 of the image processing device 3 acquires, for each of the multiple IVUS images acquired in a single pullback operation, the landmark detection result (label image) by the learning model 3M and the result of identifying the direction of the epicardium, in association with each image. The control unit 31 processes the label image output from the learning model 3M into a semi-transparent mask for each IVUS image and displays it superimposed on the original IVUS image. In this case, the control unit 31 may change the display color of the mask according to the type of landmark, or otherwise make the display manner of each landmark region different according to the type of landmark. In the example shown in Figure 5, different hatching is used depending on the landmark.

[0056] Furthermore, the control unit 31 generates graphic objects that extend from the center of the IVUS image toward the direction of the epicardium, according to the direction (angle) of the epicardium relative to each IVUS image, and displays them superimposed on the original IVUS image. In this case, the control unit 31 may change the line type and display color of the graphic objects according to the specified direction of the epicardium, thereby differentiating the display manner of each graphic object according to the specified direction of the epicardium. The specified direction of the epicardium includes, for example, direct specification using the epicardium as a landmark, indirect specification using something other than the epicardium as a landmark, and indirect specification by interpolation using other frames. In the example shown in Figure 5, the specified direction of the epicardium is displayed in an identifiable manner using three types of line types: solid lines, dashed lines, and dotted lines. In Figure 5, the solid arrows indicate the direction of interest in the IVUS image where the epicardium was detected as a landmark. In Figure 5, the dashed arrows indicate the direction of interest in the IVUS image where something other than the epicardium was detected as a landmark. In Figure 5, the dotted arrows indicate the direction of interest in the IVUS image, where the direction of interest was identified through interpolation.

[0057] The control unit 31 performs the above processing on all acquired IVUS images and displays each IVUS image, associated with the landmark and epicardium direction, in chronological order on the all-image display unit 41. The control unit 31 accepts, for example, the selection of one of the IVUS images displayed on the all-image display unit 41 via the input device 5, and displays the IVUS image corresponding to the selection on the selected image display unit 42.

[0058] In this manner, the image processing device 3 provides a screen 40 showing the direction of the epicardium for each IVUS image to physicians, etc., via the display device 4. Physicians, etc., can easily grasp the direction of the epicardium for each IVUS image from the screen 40. Furthermore, since the landmarks and the specific details of the epicardium direction are displayed in an identifiable manner, physicians, etc., can estimate the accuracy of landmark detection or the accuracy of epicardium direction identification for the IVUS image, and make a diagnosis considering the estimation results.

[0059] The screen 40 shown in Figure 5 is an example, and of course, the displayed content is not limited to the example in Figure 5. Screen 40 may display medical images other than IVUS images, such as OCT images and angiography images. Furthermore, if the control unit 31 uses information other than IVUS images, such as optical tomography images, angiography images, or information about the tubular organ into which the catheter 1 is inserted, it may display this information on screen 40 in association with the IVUS image.

[0060] The method of displaying the direction of the epicardium is not limited. The direction of the epicardium may be displayed, for example, near the IVUS image. The direction of the epicardium may be displayed using a numerical value indicating its angle in place of a graphic object, or in addition to a graphic object.

[0061] The direction of the epicardium is not limited to what is displayed in association with the IVUS image, but may also be shown in a recognizable manner by applying image processing to the IVUS image. The control unit 31 presents the direction of the epicardium by, for example, rotating each IVUS image based on the direction of the epicardium in each IVUS image. The control unit 31 uses one of a plurality of consecutive IVUS images as a reference image and performs image processing to rotate the other IVUS images so that the direction of the epicardium in the reference image is synchronized with the direction of the epicardium in the other IVUS images, based on the direction of the epicardium in the reference image. For example, if the first frame is the reference image, and the direction of the epicardium in the first frame is 1 o'clock and the direction of the epicardium in the second frame is 3 o'clock, the control unit 31 applies image processing to the IVUS image of the second frame, rotating it 60 degrees clockwise. In this case, the control unit 31 may display information related to the image processing in association with the IVUS image, such as by superimposing the rotation angle on the IVUS image. The image processing method is not limited to rotating other IVUS images in accordance with a reference image. The control unit 31 may, for example, rotate each IVUS image based on the direction of the epicardium in each IVUS image so that the direction of the epicardium in each IVUS image matches a preset reference direction (e.g., the 12 o'clock direction).

[0062] The control unit 31 of the image processing device 3 may retrain the learning model 3M based on the detection results of landmarks. The control unit 31 receives input from a physician or other person to correct the detection results of landmarks and retrains based on the input information. Specifically, for example, on the screen 40 illustrated in Figure 5, the control unit 31 receives correction input to determine whether the position and type of landmark displayed as the detection result are correct or not. Furthermore, if the displayed position and type of landmark are incorrect, the control unit 31 receives input for the correct position and type of landmark. When correction input is received, the control unit 31 retrains the learning model 3M using IVUS images labeled with the corrected detection results (position and type of landmark) as training data and updates the learning model 3M. Similarly, the control unit 31 may also receive input from a physician or other person to correct the direction of epicardium identification results and update the identification rules based on the input information. This makes it possible to improve the accuracy of landmark detection and the accuracy of direction of interest identification through the operation of this system.

[0063] The above describes an example in which the direction of interest is identified as the direction of the epicardium, but this embodiment is not limited to this. For example, when catheter 1 is inserted into a blood vessel of the lower limb, the direction of interest may be the direction of the shin or calf, and when catheter 1 is inserted into another tubular organ, the direction of interest may be the direction of the abdomen or back.

[0064] The control unit 31 may store multiple learning models 3M and specific rules corresponding to areas of interest (directions of interest) in the auxiliary storage unit 34. In this case, the control unit 31 acquires areas of interest, for example, by accepting selection registration from a doctor or the like before the examination, selects learning models 3M and specific rules corresponding to the acquired areas of interest, and uses the selected learning models 3M and specific rules to perform landmark detection and direction of interest identification processing.

[0065] According to this embodiment, the learning model 3M is used to accurately estimate landmarks for determining the direction of the epicardium relative to the IVUS image, and the direction of the epicardium is efficiently determined according to the estimation result. The result of determining the direction of the epicardium is displayed on screen 40 in a manner that is easily recognizable visually, so that even physicians who are inexperienced in interpreting IVUS images can easily grasp the direction of the epicardium and easily identify the direction of the epicardium relative to each IVUS image. Physicians can intuitively grasp information acquired using multiple modalities, such as IVUS images from an intravascular ultrasound device 101 and angiography images from an angiography device 102, by integrating them three-dimensionally, thereby effectively supporting the diagnosis of physicians.

[0066] Furthermore, according to this embodiment, since the direction of interest is identified using different procedures depending on whether the landmark is a region of interest or not, the direction of interest can be identified with high accuracy according to the type of landmark. The accuracy of the direction of interest identification result is improved by removing outliers. In addition, by interpolation processing using preceding and succeeding frame images, the direction of interest can be suitably identified even in IVUS images where landmarks are not detected.

[0067] (Second Embodiment) The second embodiment differs from the first embodiment in that the learning model directly identifies the direction of interest. Below, we will mainly explain the differences from the first embodiment, and components common to both embodiments will be denoted by the same reference numerals and their detailed descriptions will be omitted.

[0068] The control unit 31 of the image processing device 3 identifies the direction of interest in the IVUS image using a learning model 3M stored in the auxiliary storage unit 34. Figure 6 is an explanatory diagram illustrating the overview of the learning model 3M in the second embodiment. The learning model 3M in the second embodiment is a model that takes an IVUS image as input and outputs information indicating the direction of interest in the IVUS image (for example, the direction of the epicardium). Specifically, the learning model 3M receives multiple frames of IVUS images as input, which are consecutive along the longitudinal direction of the blood vessel as the catheter 1 is scanned. The learning model 3M identifies the direction of interest in the IVUS image of each frame that is consecutive along the time axis t.

[0069] The learning model 3M is, for example, a pre-trained convolutional neural network using deep learning. The learning model 3M has an input layer into which an IVUS image is input, an intermediate layer that extracts image features, and an output layer that outputs information indicating the direction of interest in the IVUS image. The intermediate layer may include convolutional layers, pooling layers, and fully connected layers, etc.

[0070] The input layer of the 3M learning model has multiple nodes that accept the pixel value of each pixel in the IVUS image as input, and passes the input pixel value to the hidden layer. The hidden layer has multiple nodes that extract features from the input data and outputs the extracted features using various parameters.

[0071] The output layer has multiple nodes corresponding to the angles of the set directions of interest (e.g., 0 degrees, 1 degree, 2 degrees, ...) and outputs the accuracy for each direction of interest as a score. The control unit 31 can set the direction of interest with the highest score, or the direction of interest with a score above a threshold, as the output value of the output layer. Alternatively, instead of having multiple output nodes that output the accuracy for each angle of interest, the output layer may have a single output node that outputs the angle of the direction of interest with the highest accuracy.

[0072] The learning model 3M can be generated by preparing training data in which IVUS images are associated with labels indicating each direction of interest, and then using this training data to machine-learn an untrained neural network. Specifically, the control unit 31 inputs multiple IVUS images included in the training data into the input layer of the untrained neural network model, performs calculations in the hidden layer, and obtains the directions of interest output from the output layer. The control unit 31 then compares the directions of interest output from the output layer with the directions of interest included in the training data, and optimizes the parameters used in the calculations in the hidden layer so that the directions of interest output from the output layer approach the directions of interest included in the training data. These parameters are, for example, the weights (connection coefficients) between neurons. The method of parameter optimization is not particularly limited, but for example, the control unit 31 uses backpropagation to optimize various parameters. The directions of interest in the training data may be, for example, judgments made by a physician with specialized knowledge.

[0073] The learning model 3M may also include medical images other than IVUS images as input elements. Other medical images may include, for example, optical tomography images and angiography images taken at the same time as the IVUS images. The learning model 3M may also include information about the luminal organ into which catheter 1 is inserted as input elements. Information about the luminal organ may include, for example, the names of blood vessels such as the right coronary artery, left coronary artery, and LAD, and a blood vessel identification number (AHA classification). By inputting these as input elements into the learning model 3M, it is possible to identify directions of interest while also considering information other than IVUS images, thereby improving identification accuracy.

[0074] The above example described a learning model 3M being a CNN, but the configuration of learning model 3M is not limited; it just needs to be able to identify the direction of interest in IVUS images. For example, learning model 3M could be a recurrent neural network (RNN) when time-series data is acquired, or it could be a model constructed using other learning algorithms such as a support vector machine or regression tree that does not use a neural network.

[0075] Figure 7 is a flowchart showing an example of a processing procedure performed by the image processing device 3 in the second embodiment. The control unit 31 of the image processing device 3 performs the following processing according to program 3P.

[0076] The control unit 31 of the image processing device 3 acquires medical images, including IVUS images, via the intravascular ultrasound device 101 (step S31). In this case, the control unit 31 may also acquire optical tomography images along with the IVUS images, or it may acquire angiography images via the angiography device 102.

[0077] The control unit 31 inputs the acquired IVUS image as input data to the learning model 3M (step S32). The control unit 31 identifies the direction of the epicardium relative to the IVUS image by acquiring information indicating the direction of the epicardium output from the learning model 3M (step S33). Subsequently, the control unit 31 outputs a screen displaying the IVUS image corresponding to the direction of the epicardium output from the learning model 3M by executing the processes shown in steps S22 to S23 in Figure 4.

[0078] In the above-described process, if the learning model 3M fails to determine the direction of the epicardium for any of the IVUS images, the control unit 31 may execute the interpolation process shown in steps S20 to S21 in Figure 4.

[0079] According to this embodiment, the orientation of the epicardium relative to the IVUS image can be accurately estimated using the learning model 3M.

[0080] In each of the flowcharts described above, some or all of the processing performed by the image processing device 3 may be performed by an external server (not shown) that is connected to the image processing device 3 in a communicative manner. In this case, the external server's memory stores a program and a learning model similar to the program 3P and learning model 3M described above. The external server acquires medical images from the image processing device 3 via a network such as a LAN (Local Area Network) or the Internet. Based on the acquired medical images, the external server performs the same processing as the image processing device 3 in each embodiment and transmits the result of identifying the direction of the epicardium to the image processing device 3. The image processing device 3 acquires the result of identifying the direction of the epicardium transmitted from the external server and displays a graphic object indicating the direction of the epicardium superimposed on the IVUS image on the display device 4, as shown in Figure 5.

[0081] The examples shown in each of the embodiments described above can be combined in whole or in part to realize other embodiments. Furthermore, the sequences shown in each of the embodiments described above are not limiting, and each processing step may be executed in a different order, and multiple processes may be executed in parallel.

[0082] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the claims and equivalents thereof. [Explanation of symbols]

[0083] 100 Imaging diagnostic equipment 101 Intravascular Ultrasound Examination Device 102 Angiography equipment 1 Catheter 2 MDU 3 Image Processing Device 31 Control Unit 32 Main memory 33 Input / Output Interfaces 34 Auxiliary storage section 3P Program 3M Learning Model 30 Recording media 4 Display device 5 Input devices

Claims

1. Medical images of tubular organs are obtained by using a catheter. The acquired medical image contains landmarks that include the region of interest or areas other than the region of interest, Based on the detected landmarks, a direction of interest is identified that indicates the direction of the area of ​​interest relative to the medical image. The identified information indicating the direction of interest is displayed in association with the medical image, or the medical image processed based on the information indicating the direction of interest is displayed. In identifying the direction of interest, Determine whether the detected landmark is the site of interest. If the landmark is the site of interest, the direction of interest is identified based on the landmark. If the landmark is not the site of interest, the direction of interest is determined based on the positional relationship between the landmark and the site of interest. A program that causes a computer to perform a process.

2. A second medical image including the aforementioned landmark is acquired, Based on the landmarks included in the first medical image and the landmarks included in the second medical image, the positional relationship between the first medical image and the second medical image is determined. Based on the identified positional relationship, the direction of interest is determined. The program according to claim 1.

3. Identify the direction of interest for each of the multiple medical images, Remove outliers of the direction of interest identified based on the direction of interest for each of the multiple medical images. The program according to claim 1 or claim 2.

4. Identify the direction of interest for each of the multiple medical images, By interpolating using the directions of interest for each of the identified medical images, the directions of interest for medical images for which the directions of interest have not been identified are determined. The program according to any one of claims 1 to 3.

5. Medical images of tubular organs are obtained by using a catheter. The acquired medical image contains landmarks that include the region of interest or areas other than the region of interest, Based on the detected landmarks, a direction of interest is identified that indicates the direction of the area of ​​interest relative to the medical image. The identified information indicating the direction of interest is displayed in association with the medical image, or the medical image processed based on the information indicating the direction of interest is displayed. In identifying the direction of interest, Determine whether the detected landmark is the site of interest. If the landmark is the site of interest, the direction of interest is identified based on the landmark. If the landmark is not the site of interest, the direction of interest is determined based on the positional relationship between the landmark and the site of interest. An information processing method in which a computer performs the processing.

6. An acquisition unit that acquires medical images of tubular organs using a catheter, A specific unit detects landmarks including the region of interest or other landmarks included in the acquired medical image, and identifies a direction of interest indicating the direction of the region of interest relative to the medical image based on the detected landmarks, A display unit that displays information indicating the direction of interest identified by the specified unit in association with the medical image, or displays the medical image processed based on the information indicating the direction of interest. Equipped with, The aforementioned identifying unit, in identifying the direction of interest, Determine whether the detected landmark is the site of interest. If the landmark is the site of interest, the direction of interest is identified based on the landmark. If the landmark is not the site of interest, the direction of interest is determined based on the positional relationship between the landmark and the site of interest. Information processing device.

Images