Program, image processing method and image processing device

Abstract

The present invention provides a program, etc., with which it is possible to compensate a missing region in a tomographic image in which a part of a luminal organ is missing. In the present invention, a computer acquires a plurality of tomographic images obtained by imaging a cross-section surface of a luminal organ at a plurality of positions using a catheter, in accordance with a program. The computer extracts, from a plurality of tomographic images, a tomographic image in which a part of the luminal organ is missing. The computer performs, on the extracted tomographic image, compensation of the missing region of the luminal organ.

Description

[Technical Field]

[0001] The present invention relates to a program, an image processing method, and an image processing device. [Background technology]

[0002] BACKGROUND ART A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel to capture a tomographic image (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 164071 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a diagnostic imaging catheter, a tomographic image is captured within a circular region with the center of the catheter as the imaging center. The diagnostic imaging catheter is not necessarily positioned at the center of the luminal organ. If the catheter is positioned near the luminal wall, the luminal organ will be imaged at a position offset from the imaging range. In particular, with large blood vessels such as coronary arteries, the cross section of the blood vessel may not fit within the imaging range, resulting in a tomographic image in which a portion of the blood vessel is missing. In such a tomographic image in which a portion of the luminal organ is missing, the condition of the luminal organ cannot be properly determined, making interpretation of the tomographic image cumbersome. Furthermore, while the size of the lumen of a luminal organ and the thickness of the luminal wall are calculated from the tomographic image, there is a problem in that this information cannot be accurately calculated from a tomographic image in which a portion of the luminal organ is missing.

[0005] In one aspect, an object of the present invention is to provide a program or the like that can complement a missing area in a tomographic image in which a part of a hollow organ is missing. [Means for solving the problem]

[0006] A program according to one aspect acquires multiple cross-sectional images of a tubular organ taken at multiple locations using a catheter, extracts from the multiple cross-sectional images those in which a portion of the tubular organ is missing, and causes a computer to perform a process on the extracted cross-sectional images to complement the missing area of ​​the tubular organ. [Effects of the Invention]

[0007] In one aspect, it is possible to complement a missing region in a tomographic image in which a portion of a hollow organ is missing. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of an imaging diagnostic apparatus. [Figure 2A] FIG. 2 is an explanatory diagram showing an example of an IVUS image taken by the intravascular examination device. [Figure 2B] FIG. 2 is an explanatory diagram showing an example of an IVUS image taken by the intravascular examination device. [Figure 3] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing device. [Figure 4A] FIG. 2 is an explanatory diagram showing an overview of a first learning model. [Figure 4B] FIG. 2 is an explanatory diagram showing an overview of a first learning model. [Figure 5] 10 is a flowchart showing an example of a procedure for interpolating an IVUS image. [Figure 6A] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 6B] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 6C] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 6D] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 7] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 8] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 9A] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 9B] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 10] FIG. 10 is an explanatory diagram illustrating a complementation process. [Figure 11] FIG. 10 is an explanatory diagram showing an overview of a second learning model. [Figure 12] 10 is a flowchart showing an example of a complementation processing procedure according to the second embodiment. [Figure 13] FIG. 10 is an explanatory diagram showing a modified example of the second learning model. [Figure 14] 11 is a flowchart showing an example of a complementation processing procedure according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The program, image processing method, and image processing device of the present disclosure will be described in detail below with reference to the drawings illustrating embodiments thereof. In the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example. However, the hollow organs that are the subject of catheterization treatment are not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines.

[0010] (Embodiment 1) FIG. 1 is an explanatory diagram showing an example of the configuration of an imaging diagnostic apparatus 100. The imaging diagnostic apparatus 100 of this embodiment includes an intravascular examination apparatus 1, a fluoroscopic imaging apparatus 2, an image processing apparatus 3, a display apparatus 4, and an input apparatus 5. The intravascular examination apparatus 1 is an apparatus for imaging intravascular tomographic images of a patient's blood vessel, and is, for example, an IVUS (Intravascular Ultrasound) apparatus that performs ultrasound examinations using a catheter 1a. The catheter 1a is a medical instrument inserted into the patient's blood vessel and includes an ultrasound transmitting / receiving unit that transmits ultrasound waves and receives reflected waves from within the blood vessel. The intravascular examination apparatus 1 generates an ultrasound tomographic image (IVUS image) of a cross section of the blood vessel based on a signal of the reflected wave received by the catheter 1a, and displays the image on the display apparatus 4 using an image processing apparatus 3. Note that, although the intravascular examination apparatus 1 generates an ultrasound tomographic image in this embodiment, it may also be configured to generate an optical coherence tomography (OCT) image captured by, for example, OCT. In this case, the catheter 1a includes an optical transmitter / receiver that transmits near-infrared light and receives reflected light from within the blood vessel.

[0011] The fluoroscopic imaging device 2 is a device for capturing fluoroscopic images of the inside of a patient's body. For example, it is an angiography device that captures blood vessels from outside the patient's body using X-rays to obtain an angioimage, which is a fluoroscopic image of the blood vessels. The fluoroscopic imaging device 2 includes an X-ray source and an X-ray sensor, and captures an X-ray fluoroscopic image of the patient by receiving X-rays emitted from the X-ray source with the X-ray sensor. A marker made of an X-ray opaque material that does not transmit X-rays is attached to the tip of the catheter 1a, and the position of the catheter 1a is visualized in the fluoroscopic image. The fluoroscopic image captured by the fluoroscopic imaging device 2 is displayed on the display device 4 by the image processing device 3, and the surgeon is presented with a fluoroscopic image in which the position of the catheter 1a (marker) is visualized together with an intravascular tomographic image. In this embodiment, the imaging diagnostic device 100 is configured to include the fluoroscopic imaging device 2 that captures two-dimensional angioimages. However, the present invention is not particularly limited to such a device as long as it captures images of the patient's luminal organs and the catheter 1a from multiple directions outside the body.

[0012] A display device 4 and an input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and displays medical images such as IVUS images captured by the intravascular inspection device 1 and angio images captured by the fluoroscopic image capturing device 2. The input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone, and accepts various operations by the surgeon. The display device 4 and the input device 5 may be stacked together to form a touch panel. Alternatively, the input device 5 and the image processing device 3 may be configured as an integrated unit. Furthermore, the input device 5 may be a sensor that accepts gesture input, gaze input, or the like.

[0013] 2A and 2B are explanatory diagrams showing examples of IVUS images captured by the intravascular inspection device 1. The intravascular inspection device 1 captures IVUS images (tomographic images) in a circular area with the center of the catheter 1a as the imaging range. The imaging range is the area surrounded by a circle shown by a solid line in FIGS. 2A and 2B, and the circle indicates the contour line (outline) of the IVUS image. Generally, IVUS images are presented to the surgeon as a square image as shown in FIGS. 2A and 2B, and black pixels are added to the area outside the imaging range, i.e., the area outside the circle, to generate and present a square image. Therefore, in this embodiment, the contour line of the IVUS image (tomographic image) refers to the circle in FIGS. 2A and 2B.

[0014] FIG. 2A is an example of an IVUS image captured when the catheter 1a is positioned at the center of the vascular lumen, and FIG. 2B is an example of an IVUS image captured when the catheter 1a is positioned near the vascular wall (lumen wall). The dashed lines in FIGS. 2A and 2B schematically indicate the vascular contour (the outer contour of the vascular wall). As shown in FIG. 2A, when the catheter 1a is positioned at the center of the vascular lumen, the entire blood vessel falls within the imaging range, and a tomographic image (IVUS image) capturing the entire blood vessel is obtained. On the other hand, when the catheter 1a is positioned toward the vascular wall, a portion of the blood vessel does not fall within the imaging range, resulting in a tomographic image (IVUS image) in which a portion of the blood vessel is missing, as shown in FIG. 2B. The image processing device 3 of this embodiment performs a process of complementing a missing region in an IVUS image in which a portion of the blood vessel is missing, as shown in FIG. 2B. Hereinafter, an IVUS image in which a portion of the blood vessel is missing is referred to as an IVUS image in which the blood vessel is framed out, and an IVUS image in which the blood vessel is not missing is referred to as an IVUS image in which the blood vessel is not framed out.

[0015] 3 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer and includes a control unit 31, a main memory unit 32, an input / output I / F 33, an auxiliary memory unit 34, and a reading unit 35. The control unit 31 is configured using one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing unit (GPGPU), or a tensor processing unit (TPU). The control unit 31 is connected to each hardware unit constituting the image processing device 3 via a bus. The main memory unit 32 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 31 to execute arithmetic processing.

[0016] The input / output I / F 33 is an interface to which the intravascular inspection device 1, the fluoroscopic imaging device 2, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images from the intravascular inspection device 1 and acquires angio images from the fluoroscopic imaging device 2 via the input / output I / F 33. The control unit 31 also displays medical images on the display device 4 by outputting medical image signals of the IVUS images and the angio images to the display device 4 via the input / output I / F 33. Furthermore, the control unit 31 receives information input to the input device 5 via the input / output I / F 33.

[0017] The auxiliary storage unit 34 is a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The auxiliary storage unit 34 stores the computer program P executed by the control unit 31 and various data required for the processing of the control unit 31. The auxiliary storage unit 34 also stores a first learning model M1 (described later). The first learning model M1 is a machine learning model that has learned predetermined training data and inputs an IVUS image and outputs the regions of the vascular lumen and vascular wall in the input IVUS image. The first learning model M1 is expected to be used as a program module constituting artificial intelligence software. The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit 34 during manufacturing of the image processing device 3, or the image processing device 3 may acquire the computer program P via communication from a remote server device and store it in the auxiliary storage unit 34.

[0018] The reading unit 35 reads data stored on a recording medium 30 such as a CD (Compact Disk), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. The computer program P may be recorded in a readable manner on the recording medium 30, or the reading unit 35 may read it from the recording medium 30 and store it in the auxiliary storage unit 34. Alternatively, the computer program P may be recorded in a semiconductor memory, and the control unit 31 may read the computer program P from the semiconductor memory and execute it.

[0019] The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer.

[0020] In the image processing device 3 of this embodiment, the control unit 31 reads and executes the computer program P stored in the auxiliary storage unit 34, thereby executing a process of complementing frame-out regions (defective regions) of IVUS images in which blood vessels are framed out of IVUS images captured by the intravascular inspection device 1. Therefore, the image processing device 3 of this embodiment can provide IVUS images in which defective regions of blood vessels have been complemented. Note that the image processing device 3 of this embodiment uses a first learning model M1 when identifying IVUS images in which blood vessels are framed out.

[0021] 4A and 4B are explanatory diagrams showing an overview of the first learning model M1. The first learning model M1 is a model that recognizes a predetermined object contained in an IVUS image. The first learning model M1 is a model that can classify objects in an image pixel by pixel using image recognition technology using, for example, semantic segmentation. The first learning model M1 of this embodiment is a machine learning model that inputs a single IVUS image and is trained to recognize the vascular lumen and vascular wall contained in the IVUS image, and outputs the recognition results. Specifically, the first learning model M1 classifies each pixel of the input IVUS image into a vascular lumen region, a vascular wall region, and other regions, and outputs a classified IVUS image (hereinafter referred to as a labeled image) in which each pixel is associated with a label for each region. The first learning model M1 can be configured, for example, using a U-Net, a fully convolutional network (FCN), a SegNet, or the like.

[0022] The first learning model M1 has an input layer, an intermediate layer, and an output layer (not shown). The intermediate layer includes a convolutional layer, a pooling layer, and a deconvolutional layer. The convolutional layer extracts image features from pixel information of the image input via the input layer to generate a feature map, and the pooling layer compresses the generated feature map. The deconvolutional layer enlarges (maps) the feature map generated by the convolutional layer and pooling layer to the original image size. The deconvolutional layer identifies which object exists at which position in the image on a pixel-by-pixel basis based on the features extracted by the convolutional layer, and generates a label image indicating which object each pixel corresponds to. Figures 4A and 4B show an example in which the IVUS image shown in Figures 2A and 2B is input to the first learning model M1. As shown on the right side of Figures 4A and 4B, the labeled image output from the first learning model M1 is an image in which each pixel of the IVUS image is classified into a vascular lumen region, a vascular wall region, and other regions, and a pixel value corresponding to each region is assigned. In Figures 4A and 4B, the pixel values ​​corresponding to the vascular lumen and vascular wall regions are indicated by different hatching.

[0023] The first learning model M1 configured as described above can be generated by preparing training data including training IVUS images and labeled images in which data indicating the objects to be identified (here, the vascular lumen and vascular wall) is labeled for each pixel in the IVUS image, as shown on the right side of FIGS. 4A and 4B, and then using this training data to train an untrained learning model. The labeled images for training are assigned labels indicating the coordinate ranges corresponding to the regions of each object and the type of each object. When an IVUS image included in the training data is input, the first learning model M1 learns to output the labeled images included in the training data. Specifically, the first learning model M1 performs calculations in the intermediate layer based on the input IVUS image to obtain detection results for each object (here, the vascular lumen and vascular wall) in the IVUS image. More specifically, the first learning model M1 obtains as output a labeled image in which each pixel in the IVUS image is labeled with a value indicating the type of classified object. The first learning model M1 then compares the acquired detection results (labeled image) with the coordinate range and object type of the correct object region indicated by the training data, and optimizes parameters such as the weights (coupling coefficients) between neurons so that the two are similar. The parameter optimization method is not particularly limited, but examples include the steepest descent method and backpropagation algorithm. This results in a first learning model M1 that, when an IVUS image is input, outputs a labeled image indicating the vascular lumen and vascular wall regions in the IVUS image.

[0024] The image processing device 3 prepares such a first learning model M1 in advance and uses it to detect the vascular lumen and vascular walls in IVUS images. The first learning model M1 only needs to be able to identify the position and shape of the vascular lumen and vascular walls in IVUS images. The first learning model M1 may be trained by another learning device. The trained first learning model M1 generated by training on the other learning device is downloaded from the learning device to the image processing device 3, for example, via a network or via the recording medium 30, and stored in the auxiliary storage unit 34.

[0025] The following describes the process of complementing a defective region of a blood vessel in an IVUS image in which the blood vessel is out of frame. FIG. 5 is a flowchart showing an example of the complementation process procedure for an IVUS image, and FIGS. 6A to 10 are explanatory diagrams explaining the complementation process. The control unit 31 of the image processing device 3 performs the following process in accordance with a computer program P stored in the auxiliary storage unit 34. The intravascular inspection device 1 performs an imaging process while performing a pull-back operation in which the ultrasound transmitting / receiving unit provided on the catheter 1a is rotated while moving at a constant speed along the axial length direction (the direction in which the blood vessel runs). The intravascular inspection device 1 also performs imaging process continuously at predetermined time intervals, so that multiple IVUS images are acquired in one pull-back operation. Therefore, the image processing device 3 performs the following process on the multiple IVUS images acquired in one pull-back operation.

[0026] The control unit 31 (acquisition unit) of the image processing device 3 acquires one frame (one piece) of IVUS image captured by the intravascular inspection device 1 (S11). The IVUS image may be an IVUS image that has already been captured by the intravascular inspection device 1 and stored in the main memory unit 32 or the auxiliary memory unit 34, or may be an IVUS image that is sequentially output from the intravascular inspection device 1.

[0027] The control unit 31 performs a process of extracting the vascular lumen and vascular wall from the acquired IVUS image (S12). Here, the control unit 31 inputs the IVUS image to the first learning model M1 and identifies the vascular lumen and vascular wall regions in the IVUS image based on the labeled image output from the first learning model M1. Specifically, when the control unit 31 acquires labeled images such as those shown on the right side of FIGS. 4A and 4B, it identifies the respective contours of the vascular lumen and vascular wall regions indicated by the labeled images. FIGS. 6A to 6D show examples of the contours of the vascular lumen and vascular wall. The dashed lines in FIGS. 6A to 6D indicate the contours of the vascular lumen (the inner surface of the vascular wall), and the dashed-dotted lines in FIGS. 6A to 6D indicate the contours of the vascular wall (the outer surface of the vascular wall). In this way, by identifying the vascular lumen and vascular wall regions in the IVUS image using the first learning model M1 that has learned training data, the vascular lumen and vascular wall can be accurately detected. In step S11, the control unit 31 may acquire an IVUS image from which the regions of the vascular lumen and vascular wall have already been extracted using the first learning model M1, and in this case, the process of step S12 is skipped.

[0028] Based on the extracted regions of the vascular lumen and vascular wall in the IVUS image, the control unit 31 determines whether the IVUS image is a frame-out image in which a portion of the vascular lumen or vascular wall is missing (S13). For example, the control unit 31 determines whether the contour line of the vascular lumen intersects with the contour line of the IVUS image. If so, it determines that a portion of the vascular lumen is missing. If not, it determines that the vascular lumen is not missing. The control unit 31 also determines whether the contour line of the vascular wall intersects with the contour line of the IVUS image. If so, it determines that a portion of the vascular wall is missing. If not, it determines that the vascular wall is not missing. Specifically, in the example shown in FIG. 6A, both the contour line of the vascular lumen and the contour line of the vascular wall intersect with the contour line of the IVUS image, so it is determined that the IVUS image is one in which both the vascular lumen and the vascular wall are frame-out. 6B, the contour line of the vascular lumen does not intersect with the contour line of the IVUS image, but the contour line of the vascular wall intersects with the contour line of the IVUS image, so it is determined that the IVUS image is one in which only the vascular wall is out of frame. Through this processing, the control unit 31 functions as an extraction unit that extracts out-of-frame tomographic images from multiple tomographic images.

[0029] The first learning model M1 detects the vascular lumen region and the vascular wall region, respectively, and thus can identify the contours of the vascular lumen and the vascular wall as shown in FIGS. 6C and 6D. In FIG. 6C, the contour of the vascular lumen is shown by a dashed line, and the dashed portion of the contour of the IVUS image indicates a contour that overlaps (matches) with the contour of the vascular lumen. In FIG. 6C, a portion of the contour of the vascular wall also overlaps with a portion of the contour of the IVUS image, but this is not shown here to avoid cluttering the drawing. In FIG. 6D, the contour of the vascular wall is shown by a dashed line, and the dashed portion of the contour of the IVUS image indicates a contour that overlaps (matches) with the contour of the vascular wall. Thus, when a portion of the vascular lumen or vascular wall is missing, a portion of the contour of the vascular lumen and the vascular wall overlaps with the contour of the IVUS image. Therefore, the control unit 31 may determine whether a portion of the contour line of the vascular lumen overlaps with the contour line of the IVUS image, and if so, determine that a portion of the vascular lumen is missing, and if not, determine that the vascular lumen is not missing. The control unit 31 may also determine whether a portion of the contour line of the vascular wall overlaps with the contour line of the IVUS image, and if so, determine that a portion of the vascular wall is missing, and if not, determine that the vascular wall is not missing. Furthermore, the control unit 31 may first determine whether a vascular wall is missing (whether it is out of frame), and if a vascular wall is missing, determine whether the vascular lumen is missing. In this case, if a vascular wall is not missing, it is possible to skip the determination of whether a vascular lumen is missing.

[0030] If the control unit 31 determines that the IVUS image acquired in step S11 is an image in which the blood vessel is out of frame (S13: YES), it determines whether the neighboring frames are IVUS images (images without a missing area) that are not out of frame (S14). Specifically, if only the vascular wall is out of frame in the IVUS image of the frame to be processed, the control unit 31 determines whether the neighboring frames are IVUS images in which the vascular wall is not out of frame. Furthermore, if the vascular lumen and vascular wall are out of frame in the IVUS image of the frame to be processed, the control unit 31 determines whether the neighboring frames are IVUS images in which the vascular lumen and vascular wall are not out of frame. Note that the IVUS images are captured while the ultrasound transmitting / receiving unit provided on the catheter 1a is pulled from a position (distal portion) far from the intravascular inspection device 1 by a pullback operation. Therefore, the blood vessel is thin (small outer diameter) at the start of imaging, and the blood vessel tends to become thicker as the ultrasound transmitting / receiving unit moves. Therefore, the IVUS image before the IVUS image in which the blood vessel is out of frame is expected to not be out of frame.

[0031] If it is determined that the adjacent frame is an IVUS image that is out of frame (S14: NO), the control unit 31 (complementation unit) performs processing to complement the out-of-frame defect area in the IVUS image of the processing target frame (self frame) using the contour lines that are not out of frame (S16). FIG. 7 illustrates processing to complement the defect area of ​​the vascular lumen based on the contour lines of the vascular lumen and the defect area of ​​the vascular wall based on the contour lines of the vascular wall in an IVUS image in which the vascular lumen and vascular wall are out of frame. The left side of FIG. 7 shows an IVUS image in which the contour lines of the vascular lumen and vascular wall are identified, the upper right side of FIG. 7 shows a state in which the defect contour line of the vascular lumen has been complemented, and the lower right side of FIG. 7 shows a state in which the defect contour line of the vascular wall has been complemented. If the vascular lumen and vascular wall are out of frame, the control unit 31 first complements the defect area of ​​the vascular lumen. For example, the control unit 31 complements the contour lines of the defect area in the processing target frame using splines or the like based on the contour lines of the vascular lumen that are not missing. Specifically, the control unit 31 calculates parameters for a spline function based on the contour line of the non-defective region, and uses the calculated spline function to complement the contour line of the defect region. The non-defective contour line is shown by a dashed line in the upper right of FIG. 7, and the contour line of the defect region, as shown by the dashed line, is complemented by a spline function based on this contour line. Similarly, the control unit 31 complements the contour line of the defect region of the vascular wall based on the contour line of the non-defective vascular wall. As a result, as shown in the lower right of FIG. 7, the contour line of the defect region of the vascular wall, as shown by the dashed line, is complemented by a spline function based on the contour line of the non-defective vascular wall (dashed line portion).

[0032] In addition, when both the vascular lumen and the vascular wall are defective, the interpolation process for the vascular lumen or the interpolation process for the vascular wall may be performed first, provided that the contour of the vascular wall after interpolation is located outside (not inside) the contour of the vascular lumen after interpolation. The interpolation of the contours of the vascular lumen and the vascular wall is not limited to the above-described process. For example, the control unit 31 may calculate the radius of curvature of a circle approximating the contour of the undefected vascular lumen and the center position of the circle corresponding to the radius of curvature, and interpolate the contour of the defective region of the vascular lumen using the calculated center position and radius of curvature. Similarly, the control unit 31 may calculate the radius of curvature and center position of a circle approximating the contour of the undefected vascular wall, and interpolate the contour of the defective region of the vascular wall using the calculated center position and radius of curvature.

[0033] On the other hand, if it is determined that the nearby frame is an IVUS image that is not out of frame (S14: YES), the control unit 31 (complementation unit) performs processing to complement the missing area in the out-of-frame IVUS image using the IVUS image of the nearby frame (S15). FIG. 8 shows processing to complement the missing area of ​​the vascular lumen in the nth frame IVUS image when the vascular lumen in the n-1th frame IVUS image is not out of frame and the vascular lumen in the nth frame IVUS image is out of frame. The control unit 31 first identifies the center position of the vascular lumen based on the contour of the vascular lumen in the n-1th frame IVUS image that is not out of frame. Then, the control unit 31 predicts the center position of the vascular lumen in the nth frame based on the contour and center position of the vascular lumen in the n-1th frame and the contour of the vascular lumen in the nth frame. Because the contours of the vascular lumen are similar in chronologically adjacent IVUS images, the control unit 31 can predict the center position of the vascular lumen in the nth frame by superimposing the contour of the vascular lumen in the n-1th frame with the contour of the vascular lumen in the nth frame. The control unit 31 complements the defective region (defective contour) of the vascular lumen in the nth frame based on the predicted center position. Here, the control unit 31 may calculate the radius of curvature of a circle approximating the contour based on the non-defective contour of the vascular lumen in the nth frame and the predicted center position, and complement the contour of the defective region using the calculated radius of curvature and the predicted center position. Alternatively, the control unit 31 may predict the contour of the vascular lumen in the nth frame from the contour of the vascular lumen in the n-1th frame without predicting the center position of the vascular lumen in the nth frame, and complement the contour of the vascular region. As a result, the contour of the vascular lumen in the nth frame is complemented as shown by the dashed dotted line in FIG. 8.

[0034] Similarly, the control unit 31 complements the contour of the vascular wall in the nth frame IVUS image, in which the vascular wall is out of frame, based on the n-1th frame IVUS image, in which the vascular wall is not out of frame. In the example shown in FIG. 8 , since the vascular wall is also out of frame in the n-1th frame IVUS image, the complementation process of the vascular wall in the nth frame IVUS image is not performed based on the n-1th frame IVUS image. In this case, for example, by the process of step S16, the complementation process of the vascular wall defect region may be performed from the contour of the vascular wall in the current frame (nth frame). Alternatively, for example, the complementation process of the vascular wall in the nth frame IVUS image may be performed based on an IVUS image in which the vascular wall is not out of frame, in an n-2th frame or earlier. Alternatively, if the contour of the vascular wall in the n-1th frame IVUS image is complemented based on the n-2th frame IVUS image (if the center position of the vascular wall is predicted), the complementation process of the vascular wall in the nth frame IVUS image may be performed using the complemented contour (predicted center position).

[0035] If the control unit 31 determines in step S13 that the IVUS image is not an image that has been framed out (S13: NO), it skips steps S14 to S16 and proceeds to step S17. After processing step S15 or step S16, the control unit 31 determines whether or not there is a frame (IVUS image) for which the above-mentioned processing has not yet been performed among the multiple IVUS images acquired in one pullback operation (S17). If it determines that there is an unprocessed frame (S17: YES), the control unit 31 returns to the processing of step S11 and performs the above-mentioned processing of steps S11 to S16 on the unprocessed frame. In this way, it is possible to determine whether or not the vascular lumen or vascular wall has been framed out for the IVUS image of each frame, and if so, to complement the contour of the missing area.

[0036] After interpolating the missing region in the IVUS image that is out of frame as described above, the control unit 31 calculates vascular information about the blood vessel captured in the IVUS image based on the IVUS image that is not out of frame and the interpolated IVUS image (S18). The vascular information includes, for example, information about the vascular lumen, information about the vascular wall, and information about plaque. For example, as shown by the solid arrow in FIG. 9A, the vascular lumen information includes the minimum and maximum values ​​of the vascular inner diameter (Minimum Lumen Diameter: MinLD) and the maximum value (MaxLD) of the vascular inner diameter passing through the center of the vascular lumen, as well as the vascular lumen cross-sectional area (Lumen CSA) of the vascular lumen. The vascular information also includes the eccentricity of the vascular lumen calculated from the minimum and maximum values ​​of the vascular inner diameter (MinLD) and MaxLD. The eccentricity of the vascular lumen is calculated using, for example, the formula {(MaxLD - MinLD) / MaxLD}. The vascular information further includes the cross-sectional area of ​​the reference lumen (CSA) in the section where the ultrasound transmitting / receiving unit was moved by the pullback operation, for example, the cross-sectional area of ​​the blood vessel lumen photographed at the tip side of the catheter 1a (cross-sectional area of ​​the distal part), the cross-sectional area of ​​the blood vessel lumen photographed at the base end side of the catheter 1a (cross-sectional area of ​​the proximal part), the maximum cross-sectional area within the section, and the average cross-sectional area within the section. The vascular information further includes the degree of stenosis of the blood vessel lumen (lumen area stenosis) calculated from the cross-sectional area of ​​each reference section.

[0037] The information on the vascular wall includes the cross-sectional area of ​​the external elastic membrane (EEM) (EEM CSA), which is calculated as the cross-sectional area of ​​the blood vessel, and the cross-sectional area of ​​the plaque and tunica media (plaque plus media CSA), which is calculated from the cross-sectional area of ​​the blood vessel. The cross-sectional area of ​​the plaque and tunica media is calculated, for example, using the formula (EEM CSA - lumen CSA). The information on the vascular wall may also include the minimum and maximum values ​​of the blood vessel outer diameter passing through the center of the blood vessel, as shown by the dashed arrow in FIG. 9A.

[0038] The plaque information includes the minimum and maximum distances (minimum plaque plus media thickness) from the outer edge of the vascular intima to the outer edge of the blood vessel along a line passing through the center of the vascular lumen, as well as the plaque eccentricity. The plaque eccentricity is calculated, for example, using the formula {(maximum plaque plus media thickness - minimum plaque plus media thickness) / maximum plaque plus media thickness}. The plaque information also includes an index of plaque burden (plaque plus media burden), which is calculated, for example, using the formula (plaque plus media CSA / EEM CSA).

[0039] Furthermore, if the IVUS image is taken after a stent has been placed at the stenotic site, the vascular information calculated based on the IVUS image may include information about the stent. The stent information may include the cross-sectional area of ​​the area surrounded by the stent (stent CSA), the minimum stent diameter passing through the center of the stent, and the maximum stent diameter. The stent information may also include stent symmetry, which is calculated using the minimum and maximum stent diameters. The stent symmetry is calculated, for example, using the formula {(maximum diameter - minimum diameter) / maximum diameter}. The stent information may also include the ratio of the minimum stent cross-sectional area to the cross-sectional area of ​​the control portion (stent expansion), which is calculated using the minimum stent cross-sectional area (minimum stent CSA). This ratio is calculated, for example, using the formula (minimum stent cross-sectional area / cross-sectional area of ​​the control portion).

[0040] Furthermore, the vascular information calculated based on the IVUS image may include a measured amount of calcification in the plaque. The measured amount of calcification in the plaque is expressed as a numerical value, for example, less than 1 / 4 circumference, 1 / 4 to 1 / 2 circumference of the blood vessel (one circumference of 360°), with the center of the vascular lumen or the center of the catheter 1a (i.e., the center of the IVUS image) as the measurement center. Note that the operator may be able to select whether to use the center of the vascular lumen or the center of the catheter 1a as the measurement center. The calculation process for each piece of information described above is a process executed by a conventional intravascular inspection device 1 or image processing device 3, so detailed explanation will be omitted. When the control unit 31 calculates the vascular information described above, it displays it on the display device 4 and presents it to the operator.

[0041] In addition to calculating the blood vessel information described above, the control unit 31 may generate a display screen displaying an IVUS image in which the contour lines of the vascular lumen or vascular wall that were previously out of frame are complemented for an IVUS image in which the vascular lumen or vascular wall was previously out of frame. FIG. 9B shows a state in which the contour lines of the defect region (vascular lumen and vascular wall) have been complemented for an IVUS image in which the vascular lumen and vascular wall were previously out of frame. In the screen shown in FIG. 9B, the contour lines of the vascular lumen and vascular wall that were not out of frame are shown in solid lines, and the contour lines of the complemented defect region are shown in dashed dotted lines. As shown in FIG. 9B, the complemented region (contour line) can be explicitly presented by displaying (adding) the complemented contour line in a different manner from the contour line that was not out of frame.

[0042] Furthermore, the control unit 31 may generate a three-dimensional image of the blood vessel captured with IVUS images based on the IVUS images in which the defect region has been complemented. Continuously captured IVUS images can be used to generate a three-dimensional image. Therefore, the control unit 31 may generate a three-dimensional image of the blood vessel captured with IVUS images by stitching together, in order of capture, IVUS images in which the vascular lumen and vascular wall are not framed and IVUS images in which the defect region has been complemented. Three-dimensional images can be generated, for example, using the voxel method. Three-dimensional images are voxel data represented by voxel coordinate values ​​in a predetermined coordinate system and voxel values ​​indicating the type of object. The data format of the three-dimensional image is not particularly limited and may be polygon data or point cloud data. Figure 10 shows a three-dimensional image created by stitching together multiple frames of IVUS images captured with a single pullback operation in order of capture. In the example shown in Figure 10, the vascular lumen and vascular wall are framed out in the IVUS image captured at the proximal end of the catheter 1a, and the framed-out defect region is complemented in the three-dimensional image. The outline of the completed missing area is shown by a dashed line on the screen shown in Fig. 10. Here too, as shown in Fig. 10, the outline of the completed missing area is displayed in a different manner from the outline of the missing area that is not framed out, thereby making it possible to clearly present the framed-out area.

[0043] In this embodiment, when the vascular lumen or vascular wall is out of frame in an IVUS image captured by the intravascular inspection device 1, the out-of-frame area (defective area) can be complemented. Therefore, the state inside the blood vessel can be appropriately observed based on the complemented IVUS image. Furthermore, the size of the vascular lumen and the thickness of the vascular wall can be accurately calculated based on the complemented IVUS image. Note that by complementing the out-of-frame contour based on the contours of the vascular lumen and vascular wall that are not out of frame in the current frame, it is possible to accurately complement the contours of the vascular lumen and vascular wall that are nearly circular. Furthermore, when the contour of the vascular region is complemented based on the contours of the vascular lumen and vascular wall in chronologically adjacent frames, more accurate complementation processing is possible.

[0044] In this embodiment, the image processing device 3 is configured to complement the frame-out defect area when the vascular lumen or vascular wall is out of frame in the IVUS image captured by the intravascular inspection device 1. In addition, when the intravascular inspection device 1 is configured to capture an OCT image, the image processing device 3 may be configured to complement the frame-out defect area (vascular lumen and vascular wall) in the OCT image in which the vascular lumen or vascular wall is out of frame.

[0045] In this embodiment, the process of extracting the vascular lumen and vascular wall in a tomographic image (e.g., an IVUS image) may be performed using the first learning model M1 or may be performed rule-based. In this embodiment, the image processing device 3 locally performs the process of detecting the vascular lumen and vascular wall regions in the IVUS image using the first learning model M1, but this configuration is not limiting. For example, a server may be provided to perform the process of detecting the vascular lumen and vascular wall using the first learning model M1. In this case, the image processing device 3 may be configured to transmit the IVUS image to the server and acquire the vascular lumen and vascular wall regions in the IVUS image identified by the server. Even with such a configuration, the same process as in this embodiment is possible, and the same effects can be obtained.

[0046] (Embodiment 2) The following describes an imaging diagnostic device 100 that uses a learning model to execute processing to complement frame-out regions (defective regions) in IVUS images in which the vascular lumen or vascular wall is framed out. The imaging diagnostic device 100 of this embodiment can be realized by devices similar to those in the imaging diagnostic device 100 of embodiment 1, and therefore a description of similar configurations will be omitted. Note that in the imaging diagnostic device 100 of this embodiment, the image processing device 3 stores a second learning model M2 (second learning model) in the auxiliary storage unit 34 in addition to the configuration of the image processing device 3 of embodiment 1 shown in FIG.

[0047] FIG. 11 is an explanatory diagram showing an overview of the second learning model M2. The second learning model M2 is a machine learning model that has learned predetermined training data. It inputs an IVUS image in which the vascular lumen or vascular wall is framed out, and outputs an IVUS image in which the defective region of the vascular lumen or vascular wall in the input IVUS image is complemented. The second learning model M2 is expected to be used as a program module constituting artificial intelligence software. The second learning model M2 outputs an image showing the contours of the vascular lumen and vascular wall in the input IVUS image, as shown in FIG. 11, for example. The second learning model M2 may also be configured to output an IVUS image in which the contours of the vascular lumen and vascular wall have been added to the input IVUS image. As shown in FIG. 11, the IVUS image input to the second learning model M2 is, for example, an IVUS image that has been subjected to padding processing, which adds black pixels or the like to a predetermined surrounding area of ​​the IVUS image shown in FIGS. 2A and 2B. The areas where black pixels are added are areas where the vascular lumen or vascular wall may be complemented, and by inputting an IVUS image containing such an area into the second learning model M2, the second learning model M2 can output an IVUS image in which the defective area of ​​the vascular lumen or vascular wall has been complemented.

[0048] The second learning model M2 is configured, for example, by a CNN (Convolutional Neural Network), a U-Net, a GAN (Generative Adversarial Network), a CycleGAN, etc. Note that the second learning model M2 may be configured using other algorithms or may be configured by combining multiple algorithms.

[0049] The second learning model M2 is, for example, a model that has been trained to recognize the contours of the vascular lumen and vascular wall contained in the input IVUS image on a pixel-by-pixel basis. Specifically, the second learning model M2 classifies each pixel of the input IVUS image into pixels on the contour of the vascular lumen, pixels on the contour of the vascular wall, and other pixels, and outputs a classified IVUS image (referred to as a labeled image) in which each pixel is associated with a label for each classification. The second learning model M2 has, for example, intermediate layers including a convolutional layer, a pooling layer, and a deconvolutional layer. For the IVUS image input to the second learning model M2, a feature map is generated in the convolutional layer and pooling layer from the pixel information of the image. The deconvolutional layer enlarges (maps) the feature map generated in the convolutional layer and pooling layer to the original image size. In this case, the deconvolution layer identifies each pixel in the image as being on the contour of the vascular lumen, the contour of the vascular wall, or another pixel based on the features extracted in the convolution layer. A predetermined value (pixel value) is assigned to each pixel identified as being on the contour of the vascular lumen or the contour of the vascular wall, and a predetermined value is also assigned to pixels at positions corresponding to the contour of the defect area to generate a labeled image (an IVUS image for output). In FIG. 11, pixels on the contour of the vascular lumen are indicated by dashed lines, and pixels on the contour of the vascular wall are indicated by dashed lines. While FIG. 11 shows a labeled image in which the contours of the vascular lumen and the vascular wall are indicated, a labeled image showing only the contour of the defect area of ​​the vascular lumen or the vascular wall may also be output. Also, while FIG. 11 shows a labeled image in which black pixels are added to the area surrounding the IVUS image containing the completed defect area, this configuration is not limiting.

[0050] The second learning model M2 configured as described above can be generated by preparing training data including IVUS images captured with the vascular lumen or vascular wall out of frame and IVUS images that are not out of frame, and using this training data to train an untrained learning model. The out-of-frame IVUS images may be images created from IVUS images that are not out of frame. The training IVUS images that are not out of frame are labeled images in which data indicating the contours of the vascular lumen or vascular wall is labeled for each pixel on the contours of the vascular lumen and vascular wall in the image. The training labeled images are assigned labels indicating the coordinate range corresponding to the contours of the vascular lumen or vascular wall after interpolation and the type of vascular lumen or vascular wall for the training IVUS image (the out-of-frame IVUS image). The second learning model M2 learns to output the labeled images included in the training data when an IVUS image included in the training data is input. Specifically, the second learning model M2 performs calculations in the intermediate layer based on the input IVUS image and obtains a detection result for the vascular lumen and vascular wall contours in the IVUS image. More specifically, the second learning model M2 obtains as output a labeled image in which each pixel in the IVUS image is labeled with a value indicating the classification result of the pixel being classified as a vascular lumen or a vascular wall contour. The second learning model M2 then compares the obtained labeled image with the correct labeled image included in the training data and optimizes parameters such as the weights (coupling coefficients) between neurons so that the two images are similar. The parameter optimization method is not particularly limited, but examples include the steepest descent method and backpropagation algorithm. This results in a second learning model M2 that, when an IVUS image is input, outputs a labeled image indicating the vascular lumen and vascular wall contours (including the interpolated contours) in the IVUS image.

[0051] The image processing device 3 prepares such a second learning model M2 in advance and uses it to complement defective areas of the vascular lumen or vascular wall in IVUS images. The second learning model M2 need only be capable of complementing the contours of defective areas of the vascular lumen or vascular wall in IVUS images. The second learning model M2 may be trained by another learning device. When training is performed by another learning device, the trained second learning model M2 is downloaded from the training device to the image processing device 3, for example, via a network or via the recording medium 30, and stored in the auxiliary storage unit 34.

[0052] Fig. 12 is a flowchart showing an example of the complementation processing procedure of the second embodiment. The processing shown in Fig. 12 is the processing shown in Fig. 5 with step S21 added instead of steps S14 to S16. Explanation of the same steps as in Fig. 5 will be omitted. In the image processing device 3 of this embodiment, the control unit 31 performs the processing of steps S11 to S13 shown in Fig. 5. This determines whether or not the vascular lumen or vascular wall is out of frame in the IVUS image.

[0053] In this embodiment, when the control unit 31 determines in step S13 that the IVUS image is a frame-out image (S13: YES), it performs a process of complementing a missing region on the IVUS image to be processed using the second learning model M2 (S21). Here, the control unit 31 inputs the IVUS image to be processed to the second learning model M2, and complements a missing region of the vascular lumen or vascular wall in the IVUS image to be processed based on a label image output from the second learning model M2. Note that the control unit 31 may generate an IVUS image in which the missing region has been complemented, for example, by overlaying a label image such as that shown in FIG. 11 on the IVUS image to be processed.

[0054] After the process of step S21, the control unit 31 executes the processes of step S17 and subsequent steps. According to the above-described process, in this embodiment, it is also possible to determine whether the vascular lumen or vascular wall is out of frame for each IVUS image frame, and if so, to generate an IVUS image in which the contour of the defect area is complemented. Therefore, an IVUS image in which the defect area is complemented and the state inside the blood vessel can be easily observed can be presented to the surgeon.

[0055] In this embodiment, the same effects as those of the above-described embodiments can be obtained. Furthermore, in this embodiment, a process for complementing a missing region in an IVUS image in which the vascular lumen or vascular wall is framed out is performed using the second learning model M2. Therefore, by training the second learning model M2 with high accuracy, it is possible to complement a missing region in an IVUS image with high accuracy. Furthermore, in this embodiment, the modified examples described as appropriate in each of the above-described embodiments can also be applied.

[0056] FIG. 13 is an explanatory diagram showing a modified example of the second learning model M2. As shown in FIG. 13, the second learning model M2 may be configured to input, for example, a labeled image (an image showing the vascular lumen and vascular wall regions) output from the first learning model M1, and output an image in which a defective region of the vascular lumen or vascular wall has been complemented for the input labeled image. In this case, for example, in the process shown in FIG. 12, the control unit 31 inputs an IVUS image to the first learning model M1 in step S12, and the resulting labeled image can be input to the second learning model M2 in step S21. In this case, too, an image in which a defective region of the vascular lumen or vascular wall in the IVUS image to be processed has been complemented is obtained based on the labeled image output from the second learning model M2.

[0057] (Embodiment 3) This section describes an imaging diagnostic device 100 that considers whether the imaging position of an IVUS image is a location where the blood vessel lumen or blood vessel wall is likely to be out of frame, based on a fluoroscopic image (e.g., an angio image) captured by a fluoroscopic imaging device 2. The imaging diagnostic device 100 of this embodiment can be realized by devices similar to the devices in the imaging diagnostic device 100 of embodiment 1, and therefore, a description of similar configurations will be omitted.

[0058] In the diagnostic imaging device 100 of this embodiment, the imaging position of the IVUS image captured by the intravascular inspection device 1 corresponds to the position of the blood vessel in the angio image captured by the fluoroscopic imaging device 2. Therefore, the image processing device 3 determines whether or not the blood vessel lumen or blood vessel wall in the IVUS image is likely to be out of frame, taking into account factors such as the thickness of the blood vessel at the position in the angio image corresponding to the position at which the IVUS image to be processed was captured. For example, if the imaging position of the IVUS image is close to where the coronary artery connects to the aorta, it determines that the blood vessel lumen or blood vessel wall is likely to be out of frame. When the image processing device 3 of this embodiment determines that the blood vessel lumen or blood vessel wall is likely to be out of frame based on the imaging position of the IVUS image, it performs a process of determining whether or not the blood vessel is actually out of frame, and, if so, performs a process of complementing the missing area.

[0059] Fig. 14 is a flowchart showing an example of the complementation processing procedure of the third embodiment. The processing shown in Fig. 14 is obtained by adding steps S31 and S32 between steps S11 and S12 in the processing shown in Fig. 5. Explanation of the same steps as in Fig. 5 will be omitted. In the image processing device 3 of this embodiment, the control unit 31 performs the processing of step S11 shown in Fig. 5 to acquire an IVUS image to be processed.

[0060] The control unit 31 identifies a position in the angio image corresponding to the imaging position of the acquired IVUS image (S31). For example, each position in the angio image is assigned image information (e.g., an image number) of the IVUS image, and the control unit 31 identifies the position to which the information of the IVUS image to be processed is assigned. Then, based on the identified position, the control unit 31 determines whether or not the vascular lumen or vascular wall in this IVUS image is likely to be out of frame (S32). Here, for example, a region that should be determined to be likely to be out of frame may be set in advance, and the control unit 31 may determine whether or not the identified position is included in the set region in the angio image, and determine whether or not it is out of frame depending on whether or not it is included.

[0061] If the control unit 31 determines that the IVUS image to be processed may be out of frame (S32: YES), it proceeds to the processing of step S12, and if it determines that there is no possibility that the image is out of frame (S32: NO), it proceeds to the processing of step S17. Therefore, if there is a possibility that the image is out of frame, the control unit 31 executes the processing from step S12 onwards, determines whether the vascular lumen or vascular wall in the IVUS image to be processed is out of frame, and if it is out of frame, performs processing to complement the missing area. Furthermore, if there is no possibility that the image is out of frame, it does not perform the processing to determine whether it is out of frame, thereby suppressing the execution of unnecessary processing and speeding up the processing.

[0062] This embodiment provides the same effects as the above-described embodiments. Furthermore, this embodiment determines whether the imaging position of an IVUS image is a location where the vascular lumen or vascular wall may be out of frame based on an angio image, and therefore, it is possible to extract IVUS images captured at locations where there is no possibility of the vascular lumen or vascular wall being out of frame. Therefore, by not performing the process of determining whether the IVUS image is out of frame for IVUS images captured at locations where there is no possibility of the vascular lumen or vascular wall being out of frame, it is possible to simplify the process.

[0063] In this embodiment, instead of determining whether the imaging position of an IVUS image is a location that may be out of frame based on an angio image, it may be determined whether the imaging position is a location that may be out of frame based on the IVUS image itself. Specifically, it may be configured to determine whether the imaging position of the IVUS image is a location that may be out of frame using the blood vessel diameter obtained from the IVUS image. For example, if the imaging depth in the IVUS image is 6 mm and the blood vessel diameter at the imaging position of the IVUS image is 6 mm or more, at such an imaging position, the catheter 1a is likely to be biased toward the blood vessel wall, increasing the possibility that the blood vessel lumen or blood vessel wall will be out of frame. Therefore, it is possible to determine whether the imaging position is a location that may be out of frame based on the blood vessel diameter obtained from the IVUS image.

[0064] In this embodiment, the modified examples described in the above-described embodiments can also be applied as appropriate. For example, when the vascular lumen or vascular wall is out of frame, a similar process can be performed on an OCT image instead of an IVUS image to complement the missing region.

[0065] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0066] 1. Intravascular inspection equipment 2. Fluoroscopic imaging device 3. Image processing device 4 Display device 5 Input Devices 1a catheter M1 First learning model M2 Second learning model

Claims

1. A catheter is used to capture a plurality of cross-sectional images of the hollow organ at a plurality of locations, and inputting the acquired multiple tomographic images into a learning model that has been trained to output information indicating the regions of the lumen and luminal wall of the luminal organ in the tomographic images when the tomographic images are input, and outputting the lumen and luminal wall of the luminal organ in each of the tomographic images; determining whether a part of the contour line of the lumen or lumen wall of the hollow organ that has been output matches or intersects with the contour of the tomographic images; extracting the tomographic images that are determined to be coincident or intersecting as tomographic images in which a part of the hollow organ is missing; The missing region of the hollow organ is complemented in the extracted tomographic image. A program that causes a computer to perform a process.

2. The tomographic image in which a part of the hollow organ is missing is a cross-sectional image in which a part of the hollow organ is out of frame and does not fit within the imaging range. The program according to claim 1.

3. In the tomographic image in which a part of the hollow organ is missing, the contour line of the missing region of the hollow organ is complemented based on the contour line of the hollow organ that is not missing.

3. The program according to claim 1, which causes the computer to execute processing.

4. predicting a center position of the hollow organ in the cross-sectional image containing the defect based on a tomographic image in which the hollow organ is not missing, the tomographic image being a tomographic image captured in the vicinity of the tomographic image containing the defect; Based on the predicted center position of the hollow organ, a missing region of the hollow organ in the missing tomographic image is complemented.

3. The program according to claim 1, which causes the computer to execute processing.

5. inputting the extracted missing tomographic image into a second learning model that has been trained to output a missing region of the tubular organ in a tomographic image when the tomographic image contains a missing portion of the tubular organ; and outputting the missing region of the tubular organ in the missing tomographic image; The defective region of the hollow organ in the defective tomographic image is complemented based on the output defective region of the hollow organ.

5. The program according to claim 1, which causes the computer to execute a process.

6. A contour line of the missing region of the hollow organ is added to the tomographic image in which a part of the hollow organ is missing, in a manner that indicates the missing portion.

6. The program according to claim 1, which causes the computer to execute a process.

7. The hollow organ is a blood vessel, the lumen is a blood vessel lumen, and the lumen wall is a blood vessel wall.

7. The program according to claim 1.

8. A plurality of cross-sectional images of the hollow organ taken at a plurality of locations using a catheter are acquired from a storage unit, inputting the acquired multiple tomographic images into a learning model that has been trained to output information indicating the regions of the lumen and luminal wall of the luminal organ in the tomographic images when the tomographic images are input, and outputting the lumen and luminal wall of the luminal organ in each of the tomographic images; determining whether a part of the contour line of the lumen or lumen wall of the hollow organ that has been output matches or intersects with the contour of the tomographic images; extracting the tomographic images that are determined to be coincident or intersecting as tomographic images in which a part of the hollow organ is missing; The missing region of the hollow organ is complemented in the extracted tomographic image. An image processing method in which processing is performed by a computer.

9. An image processing device having a control unit, The control unit A catheter is used to capture a plurality of cross-sectional images of the hollow organ at a plurality of locations, and inputting the acquired multiple tomographic images into a learning model that has been trained to output information indicating the regions of the lumen and luminal wall of the luminal organ in the tomographic images when the tomographic images are input, and outputting the lumen and luminal wall of the luminal organ in each of the tomographic images; determining whether a part of the contour line of the lumen or lumen wall of the hollow organ that has been output matches or intersects with the contour of the tomographic images; extracting the tomographic images that are determined to be coincident or intersecting as tomographic images in which a part of the hollow organ is missing; The missing region of the hollow organ is complemented in the extracted tomographic image. Image processing device.

Images