Computer program, image processing method, and image processing apparatus

Abstract

In the present invention, a computer is caused to perform processing in which: a plurality of tomographic images of a hollow organ are acquired while moving a sensor unit, which is for scanning the hollow organ, along the travel direction of the hollow organ; selected from a plurality of tomographic images acquired prior to a procedure with respect to the hollow organ is at least one tomographic image that relates to a target site at which the procedure will be performed on the hollow organ; a tomographic image from after the procedure that corresponds to the selected one tomographic image from before the procedure is identified with image recognition processing with respect to a plurality of tomographic images acquired after the procedure; and, on the basis of the identification result of the tomographic image, the tomographic image acquired prior to the procedure and the tomographic image acquired after the procedure are associated and displayed.

Description

Technical Field

[0001] The present invention relates to a computer program, an image processing method, and an image processing apparatus.

Background Art

[0002] As a minimally invasive treatment for ischemic heart diseases such as angina or myocardial infarction, endovascular treatment represented by percutaneous coronary intervention (PCI) is being performed. For preoperative diagnosis or postoperative result confirmation in endovascular treatment, intravascular imaging diagnostic devices such as an intravascular ultrasound diagnostic device (IVUS) using ultrasonic waves and an optical frequency domain imaging device (OFDI) using near-infrared rays are used.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] When medical staff checks the state of blood vessels using the ultrasonic tomographic image acquired after treatment, they compare it with the ultrasonic tomographic image before treatment to confirm the effect of treatment. When comparing ultrasonic tomographic images, it is necessary to check the ultrasonic tomographic images before and after treatment taken at the same position in the longitudinal direction of the blood vessel. On the other hand, precisely aligning the scanning start position within the blood vessel is difficult, resulting in longitudinal displacement of the ultrasound images acquired before and after the procedure. Currently, medical professionals manually select ultrasound images showing the same blood vessel location, manually adjust the longitudinal position, and then confirm the treatment effect. The same situation applies to optical coherence tomography (OCT) images.

[0005] One aspect of this disclosure aims to provide a computer program, an image processing method, and an image processing apparatus that can display tomographic images of a tubular organ acquired before and after a procedure on the tubular organ, while aligning the longitudinal position of the tubular organ. [Means for solving the problem]

[0006] A computer program relating to one aspect of this disclosure acquires multiple tomographic images of a tubular organ by moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, selects at least one tomographic image from the multiple tomographic images acquired before treatment of the tubular organ that is related to the area of ​​the tubular organ to be treated, identifies a post-treatment tomographic image corresponding to the selected pre-treatment tomographic image by image recognition processing of the multiple tomographic images acquired after treatment, and causes the computer to perform a process of displaying the pre-treatment tomographic image and the post-treatment tomographic image in association with each other based on the identification result of the tomographic images.

[0007] An image processing method relating to one aspect of the present disclosure involves acquiring multiple tomographic images of a tubular organ by moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, selecting at least one tomographic image from among the multiple tomographic images acquired before treatment of the tubular organ that is related to the target area of ​​the tubular organ to be treated, identifying a post-treatment tomographic image corresponding to the selected pre-treatment tomographic image by image recognition processing of the multiple tomographic images acquired after treatment, and displaying the pre-treatment tomographic image and the post-treatment tomographic image in association with each other based on the identification result of the tomographic images.

[0008] An image processing device relating to one aspect of the present disclosure includes an acquisition unit that acquires a plurality of tomographic images of a tubular organ while moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, and a processing unit that performs processing to display the tomographic images acquired by the acquisition unit on a display device, wherein the processing unit selects at least one tomographic image from a plurality of tomographic images acquired before treatment of the tubular organ that is related to the target area of ​​the tubular organ to be treated, identifies a post-treatment tomographic image corresponding to the selected pre-treatment tomographic image by image recognition processing of a plurality of tomographic images acquired after treatment, and displays the pre-treatment tomographic image and the post-treatment tomographic image in association with each other based on the identification result of the tomographic images. [Effects of the Invention]

[0009] One aspect of this disclosure is that when displaying tomographic images of a tubular organ acquired before and after a procedure on the tubular organ, the longitudinal position of the tubular organ can be aligned and displayed. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram showing an example of the configuration of an imaging diagnostic device. [Figure 2] This is an explanatory diagram showing an example of the configuration of a diagnostic imaging catheter. [Figure 3] This is a schematic diagram illustrating a cross-section of a blood vessel through which the sensor is inserted. [Figure 4A] This is an explanatory diagram of a tomographic image. [Figure 4B] This is an explanatory diagram of a tomographic image. [Figure 5] This is a block diagram showing an example of the configuration of an image processing device. [Figure 6A] This is a block diagram showing an example configuration of an IVUS image recognition learning model. [Figure 6B] This is a block diagram showing an example configuration of an IVUS image recognition learning model. [Figure 7] This is a flowchart showing the image processing procedure according to this embodiment 1. [Figure 8] This is a flowchart showing the image processing procedure according to Embodiment 1. [Figure 9] This is an example display of the IVUS image before treatment and the treatment plan screen. [Figure 10] This is an example display of IVUS images, etc. before and after treatment. [Figure 11] This is a flowchart showing the image processing procedure according to Embodiment 2. [Figure 12A] This is an explanatory diagram showing the image processing method according to Embodiment 2. [Figure 12B] This is an explanatory diagram showing the image processing method according to Embodiment 2. [Figure 12C] This is an explanatory diagram showing the image processing method according to Embodiment 2. [Figure 13] This is a flowchart showing the image processing procedure according to Embodiment 3. [Figure 14] This is a flowchart showing the image processing procedure according to Embodiment 4. [Figure 15] This is an explanatory diagram showing a plurality of pre-treatment and post-treatment IVUS images P1 arranged in the longitudinal direction of the blood vessel. [Figure 16] This is a flowchart showing the image processing procedure according to Embodiment 5.

Modes for Carrying Out the Invention

[0011] Hereinafter, the computer program, image processing method, and image processing apparatus of the present disclosure will be described in detail based on the drawings showing their embodiments. In each of the following embodiments, cardiac catheter treatment, which is an endovascular treatment, will be described as an example. However, the luminal organ targeted for catheter treatment is not limited to blood vessels, and may be other luminal organs such as bile ducts, pancreatic ducts, bronchi, intestines, etc. Further, the present invention is not limited to these examples, and is intended to be defined by the claims, and to include all modifications within the meaning and scope equivalent to the claims. At least a part of the embodiments described below may be arbitrarily combined.

[0012] This embodiment describes an imaging diagnostic device using a dual-type catheter equipped with both intravascular ultrasound (IVUS) and optical coherence tomography (OFDI) functions. The dual-type catheter has three modes: one for acquiring ultrasound tomography images using IVUS only, one for acquiring optical coherence tomography images using OFDI only, and one for acquiring both ultrasound and optical coherence tomography images using both IVUS and OFDI. These modes can be switched between. Hereinafter, ultrasound tomography images and optical coherence tomography images will be referred to as IVUS images and OFDI images, respectively. Furthermore, IVUS images and OFDI images will be collectively referred to as tomography images.

[0013] (Embodiment 1) Figure 1 is an explanatory diagram showing an example configuration of the diagnostic imaging device 100. The diagnostic imaging device 100 of this embodiment 1 comprises an intravascular examination device 101, an angiography device 102, an image processing device 3, a display device 4, and an input device 5. The intravascular examination device 101 comprises a diagnostic imaging catheter 1 and an MDU (Motor Drive Unit) 2. The diagnostic imaging catheter 1 is connected to the image processing device 3 via the MDU 2. The display device 4 and the input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, mouse, trackball, or microphone. The display device 4 and the input device 5 may be stacked together to form a touch panel. The input device 5 and the image processing device 3 may also be configured together. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input, etc.

[0014] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that uses X-rays to image blood vessels from outside the patient's body while injecting a contrast agent into the patient's blood vessels, thereby obtaining an angiographic image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and the X-ray sensor receives the X-rays irradiated from the X-ray source to image an X-ray fluoroscopic image of the patient. The angiography device 102 outputs the acquired angiographic image to the image processing device 3, and displays it on the display device 4 via the image processing device 3.

[0015] Figure 2 is an explanatory diagram showing an example of the configuration of the imaging diagnostic catheter 1. Note that the area enclosed by the upper dashed line in Figure 2 is an enlargement of the area enclosed by the lower dashed line. The imaging diagnostic catheter 1 has a probe 11 and a connector portion 15 located at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector portion 15. In the following description, the side of the imaging diagnostic catheter 1 furthest from the connector portion 15 will be referred to as the tip side, and the side with the connector portion 15 will be referred to as the proximal end side. The probe 11 is equipped with a catheter sheath 11a, and its tip is provided with a guidewire insertion portion 14 through which a guidewire can be inserted. The guidewire insertion portion 14 constitutes a guidewire lumen and is used to receive a guidewire that has been previously inserted into the blood vessel and to guide the probe 11 to the affected area by the guidewire. The catheter sheath 11a forms a continuous tube from the guidewire insertion portion 14 to the connector portion 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip of the shaft 13.

[0016] The sensor unit 12 has a housing 12c, and the tip end of the housing 12c is formed in a hemispherical shape to suppress friction and snagging with the inner surface of the catheter sheath 11a. Inside the housing 12c are an ultrasonic transmitting / receiving unit 12a that transmits ultrasonic waves into the blood vessel and receives reflected waves from within the blood vessel, and an optical transmitting / receiving unit 12b that transmits near-infrared light into the blood vessel and receives reflected light from within the blood vessel. In the example shown in Figure 2, the ultrasonic transmitting / receiving unit 12a is provided on the tip end of the probe 11, and the optical transmitting / receiving unit 12b is provided on the proximal end. In other words, the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are arranged inside the housing 12c at a predetermined distance apart along the axial direction on the central axis of the shaft 13 (on the dashed line in Figure 2). Furthermore, the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are arranged such that the transmission and reception directions of ultrasonic waves and near-infrared light are approximately 90 degrees with respect to the axial direction of the shaft 13 (radial direction of the shaft 13). It is desirable that the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b be mounted slightly offset from the radial direction so as not to receive reflected waves and reflected light from the inner surface of the catheter sheath 11a. In this embodiment 1, for example, as shown by the arrow in Figure 2, the ultrasonic transmitting / receiving unit 12a is positioned so that the direction inclined toward the proximal end with respect to the radial direction is the direction of ultrasonic irradiation, and the optical transmitting / receiving unit 12b is positioned so that the direction inclined toward the tip with respect to the radial direction is the direction of near-infrared light irradiation.

[0017] The shaft 13 contains an electrical signal cable (not shown) connected to the ultrasonic transmitting / receiving unit 12a and an optical fiber cable (not shown) connected to the optical transmitting / receiving unit 12b. The probe 11 is inserted into the blood vessel from the tip end. The sensor unit 12 and the shaft 13 can move forward and backward inside the catheter sheath 11a and can also rotate in the circumferential direction. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation.

[0018] The MDU2 is a drive device to which the connector portion 15 of the diagnostic imaging catheter 1 is detachably attached. It drives a built-in motor in response to user (medical professional) operations and controls the movement of the diagnostic imaging catheter 1 inserted into a blood vessel. For example, the MDU2 performs a pullback operation, rotating the sensor portion 12 and shaft 13 inserted into the probe 11 circumferentially while pulling them toward the MDU2 at a constant speed. The sensor portion 12 rotates while moving from the tip to the proximal end due to the pullback operation, and continuously scans the inside of the blood vessel at predetermined time intervals, thereby continuously capturing multiple transverse images approximately perpendicular to the probe 11 at predetermined intervals. The MDU2 outputs the reflected ultrasound wave signal received by the ultrasound transmitting / receiving unit 12a and the reflected light signal received by the optical transmitting / receiving unit 12b to the image processing device 3.

[0019] The image processing device 3 acquires the reflected ultrasound signal output from the ultrasound transmitting / receiving unit 12a via the MDU2 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data. The ultrasound line data is data that shows the reflected ultrasound intensity in the depth direction of the blood vessel as seen from the ultrasound transmitting / receiving unit 12a. Based on the generated ultrasound line data, the image processing device 3 constructs an IVUS image P1 (see Figures 4A and 4B) representing the transverse layer of the blood vessel.

[0020] Furthermore, the image processing device 3 acquires interference light data by interfering the reflected light signal output from the optical transceiver 12b via the MDU2 with the reference light obtained by separating the light from the light source, and generates optical line data based on the acquired interference light data. The optical line data is data that shows the reflection intensity of the reflected light in the depth direction of the blood vessel as seen from the optical transceiver 12b. Based on the generated optical line data, the image processing device 3 constructs an OFDI image P2 (see Figures 4A and 4B) representing the transverse layer of the blood vessel.

[0021] Here, we will explain the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b, and the IVUS image P1 and OFDI image P2 constructed from the ultrasonic line data and optical line data.

[0022] Figure 3 is a schematic diagram showing a cross-section of a blood vessel through which the sensor unit 12 is inserted, and Figures 4A and 4B are explanatory diagrams of tomographic images. First, using Figure 3, we will explain the operation of the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b inside the blood vessel, and the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b.

[0023] When tomographic imaging is started with the sensor unit 12 and shaft 13 inserted into the blood vessel, the sensor unit 12 rotates around the central axis of the shaft 13 in the direction indicated by the arrow. At this time, the ultrasonic transmitting / receiving unit 12a transmits and receives ultrasound at each rotation angle. Lines 1, 2, ... 512 indicate the direction of ultrasound transmission and reception at each rotation angle. In this embodiment 1, the ultrasonic transmitting / receiving unit 12a intermittently transmits and receives ultrasound 512 times while rotating 360 degrees (1 rotation) inside the blood vessel. Since the ultrasonic transmitting / receiving unit 12a acquires data for one line in the transmission / reception direction with each ultrasound transmission and reception, 512 ultrasound line data points extending radially from the center of rotation can be obtained during one rotation. The 512 ultrasound line data points are dense near the center of rotation, but become sparser as they move away from the center of rotation. Therefore, the image processing device 3 can construct a two-dimensional IVUS image P1 as shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process.

[0024] Similarly, the optical transceiver 12b also transmits and receives near-infrared light (measurement light) at each rotation angle. Since the optical transceiver 12b also transmits and receives measurement light 512 times while rotating 360 degrees inside the blood vessel, 512 optical line data lines extending radially from the center of rotation can be obtained during one rotation. With respect to the optical line data, the image processing device 3 can construct the two-dimensional OFDI image P2 shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process.

[0025] A two-dimensional tomographic image constructed from multiple ultrasound line data in this manner is called an IVUS image P1, which is one frame. Similarly, a two-dimensional tomographic image constructed from multiple optical line data is called an OFDI image P2, which is one frame. As the sensor unit 12 scans while moving within the blood vessel, one frame of IVUS image P1 or OFDI image P2 is acquired at each position after one rotation within the movement range. That is, one frame of IVUS image P1 or OFDI image P2 is acquired at each position from the tip to the proximal end of the probe 11 within the movement range, so as shown in Figure 4B, multiple frames of IVUS image P1 or OFDI image P2 are acquired within the movement range.

[0026] The number of ultrasonic and optical transmissions / receptions per rotation is just an example, and the number of transmissions / receptions is not limited to 512. Furthermore, the number of ultrasonic transmissions / receptions and the number of optical transmissions / receptions may be the same or different.

[0027] Figure 5 is a block diagram showing an example configuration of the image processing device 3. The image processing device 3 is a computer and comprises a processing unit 31, a storage unit 32, an ultrasonic line data generation unit 33, an optical line data generation unit 34, an input / output interface 35, and a reading unit 36. The processing unit 31 is configured using one or more computing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), and a TPU (Tensor Processing Unit). The processing unit 31 is connected to each hardware component of the image processing device 3 via a bus.

[0028] The memory unit 32 includes, for example, a main memory unit and an auxiliary memory unit. The main memory unit is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data necessary for the processing unit 31 to perform calculations. The auxiliary memory unit is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary memory unit stores the computer program (program product) P executed by the processing unit 31, the IVUS image recognition learning model 61, the OFDI image recognition learning model 62, and various other data necessary for processing. The memory unit 32 also functions as a tomographic image database 63. The tomographic image database 63 stores ID (identification information), imaging date and time, name of the imaged blood vessel, name of the lesion, and treatment-related information associated with multiple IVUS images P1 acquired by the image processing device 3. Furthermore, the tomographic image DB63 stores multiple OFDI images P2 acquired by the image processing device 3, associating them with ID (identification information), acquisition date and time, name of the vessel being imaged, name of the lesion, and treatment-related information. Treatment-related information includes information indicating the type of treatment performed on the vessel, such as stent placement, balloon dilation, or dilation of an implanted stent, and other information related to the treatment. In other words, treatment-related information is information that serves as a clue to identify a predetermined pre-treatment tomographic image as a tomographic image related to a post-treatment tomographic image for multiple IVUS images P1 acquired at different times. As long as it is possible to identify a post-treatment tomographic image that corresponds to a predetermined pre-treatment tomographic image, the content of the treatment-related information is not particularly limited. The auxiliary storage unit may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit during the manufacturing stage of the image processing device 3, or it may be distributed by a remote server device and acquired by the image processing device 3 via communication and stored in the auxiliary storage unit. The computer program P may also be recorded in a readable form on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, and the reading unit 36 ​​may read it from the recording medium 30 and store it in the auxiliary storage unit.

[0029] The ultrasound line data generation unit 33 acquires the reflected ultrasound wave signal output from the ultrasound transmitting / receiving unit 12a of the intravascular examination device 101 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data.

[0030] The optical line data generation unit 34 acquires interference light data by interfering the reflected light signal output from the optical transmitting / receiving unit 12b of the intravascular examination device 101 with reference light obtained by separating the light from the light source, and generates optical line data based on the acquired interference light data.

[0031] The input / output interface 35 is an interface to which the intravascular examination device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The processing unit 31 acquires angiography images from the angiography device 102 via the input / output interface 35. The processing unit 31 also displays medical images on the display device 4 by outputting medical image signals of IVUS images P1, OFDI images P2, or angiography images to the display device 4 via the input / output interface 35. Furthermore, the processing unit 31 receives information input to the input device 5 via the input / output interface 35.

[0032] The image processing device 3 may be a multi-computer system comprising multiple computers. Alternatively, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer.

[0033] The processing unit 31 of the image processing device 3 reads and executes a computer program P stored in the storage unit 32, thereby generating ultrasound line data in the ultrasound line data generation unit 33 and constructing an IVUS image P1 based on the generated ultrasound line data. The processing unit 31 also reads and executes a computer program P stored in the storage unit 32, thereby generating optical line data in the optical line data generation unit 34 and constructing an OFDI image P2 based on the generated optical line data.

[0034] The processing unit 31 of the image processing device 3 according to this embodiment 1 reads and executes a computer program P stored in the storage unit 32, thereby aligning the longitudinal positions of multiple IVUS images P1 acquired before a predetermined procedure such as stent placement on a blood vessel and multiple IVUS images P1 acquired after the procedure, and displaying them (see Figure 10). By comparing the pre- and post-procedure IVUS images P1 displayed with their longitudinal positions aligned, the user can confirm the condition of the blood vessel and the treatment effect.

[0035] <Image Recognition Learning Model> Figures 6A and 6B are block diagrams showing examples of the configuration of the IVUS image recognition learning model 61. The IVUS image recognition learning model 61 is a model that recognizes predetermined object images contained in the IVUS image P1. The IVUS image recognition learning model 61 can classify objects at the pixel level by using image recognition technology, for example, semantic segmentation, and can recognize various objects contained in the IVUS image P1. Specifically, as shown in Figures 6A and 6B, the IVUS image recognition learning model 61 recognizes the lumen image in the IVUS image P1. The IVUS image recognition learning model 61 also recognizes stent images, plaque images, vascular wall images, etc., as shown in Figure 6A. The vascular wall is the tunica media, and more specifically, the external elastic membrane (EEM). Note that the plaque image etc. shown in Figure 6 are just examples of object images, and the IVUS image recognition learning model 61 may be configured to recognize side branches, epicardium, or veins, etc.

[0036] The IVUS image recognition learning model 61 is, for example, a pre-trained convolutional neural network (CNN) using deep learning. The IVUS image recognition learning model 61 has an input layer 61a into which the IVUS image P1 is input, an intermediate layer 61b that extracts and reconstructs the features of the IVUS image P1, and an output layer 61c that outputs an object-extracted IVUS image P1' that shows the object images contained in the IVUS image P1 on a pixel-by-pixel basis. Examples of IVUS image recognition learning models 61 include U-Net, FCN (Fully Convolutional Network), SegNet, etc.

[0037] The input layer 61a of the IVUS image recognition learning model 61 has multiple neurons that accept input of an IVUS image P1, that is, the pixel values ​​of each pixel that makes up the IVUS image P1, and passes the input pixel values ​​to the hidden layer 61b. The hidden layer 61b has multiple convolutional layers (CONV layers) and multiple deconvolutional layers (DECONV layers). The convolutional layers are layers that compress the dimensions of the IVUS image P1. Through dimensionality reduction, feature quantities of the object image are extracted. The deconvolutional layers perform deconvolution and restore to the original dimensions. The restoration process in the deconvolutional layers generates an object-extracted IVUS image P1' in which each pixel has a pixel value (class data) corresponding to the object class. The output layer 61c has multiple neurons that output the object-extracted IVUS image P1'. As shown in Figures 6A and 6B, the object-extracted IVUS image P1' is an image that is classified, for example, by color according to the type of object, such as lumen image, stent image, vessel wall image, plaque image, side branch, epicardium, or vein.

[0038] The IVUS image recognition learning model 61 can be generated by preparing training data consisting of an IVUS image P1 obtained by the ultrasonic transmitting / receiving unit 12a and an object-extracted IVUS image P1' in which class data corresponding to the type of object is assigned (annotated) to each pixel of the IVUS image P1, and then using this training data to machine-learn an untrained neural network. Specifically, the parameters of the neural network are optimized so that the difference between the object-extracted IVUS image P1' output when the IVUS image P1 of the training data is input to the untrained neural network is small, and the annotated image used as training data. These parameters are, for example, the weights between nodes (connection coefficients). The method of parameter optimization is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method.

[0039] According to the IVUS image recognition learning model 61 trained in this manner, by inputting an IVUS image P1 obtained by imaging a blood vessel in which a stent has been implanted, as shown in Figure 6A, an object-extracted IVUS image P1' is obtained in which each part is classified at the pixel level according to the type of object, such as the lumen image of the blood vessel, the blood vessel wall image, the plaque image, and the stent. Also, as shown in Figure 6B, by inputting an IVUS image P1 obtained by imaging a blood vessel without a stent, an object-extracted IVUS image P1' is obtained in which each part is classified at the pixel level according to the type of object, such as the lumen image of the blood vessel, the blood vessel wall image, and the plaque image.

[0040] The OFDI image recognition learning model 62 has the same configuration as the IVUS image recognition learning model 61 and recognizes lumen images and other object images in the OFDI image P2. By inputting the OFDI image P2 into the OFDI image recognition learning model 62, an object-extracted OFDI image is obtained in which each part is classified at the pixel level according to the type of object, such as a vascular lumen image, stent image, vascular wall image, and plaque image.

[0041] <Image Processing Procedure> Figures 7 and 8 are flowcharts showing the image processing procedure according to this embodiment 1. The following mainly describes the image processing procedure for the IVUS image P1.

[0042] The processing unit 31 of the image processing device 3 acquires multiple IVUS images P1 before a procedure such as stent placement using the ultrasound line data generation unit 33 (step S111). Specifically, the ultrasound line data generation unit 33 of the image processing device 3 generates multiple ultrasound line data based on reflected wave data output from the ultrasound transmitting / receiving unit 12a. The multiple ultrasound line data are assigned line numbers, for example, in chronological order of observation time. The line numbers correspond to the observation time. In other words, the line numbers correspond to the observation position and observation direction. The processing unit 31 constructs multiple frames of IVUS images P1 based on the ultrasound line data. The multiple frames of IVUS images P1 are assigned frame numbers, for example, in chronological order of observation time. The frame numbers correspond to the observation positions. The multiple frames of IVUS images P1 correspond to images obtained by observing blood vessels at multiple observation positions from the tip to the proximal end of the probe 11. The processing unit 31 that executes the process in step S111 functions as an acquisition unit that acquires multiple IVUS images P1 of the blood vessel while moving the sensor unit 12 along the direction of the blood vessel's course.

[0043] Next, the processing unit 31 stores the multiple IVUS images P1 acquired before the procedure on the blood vessel in the storage unit 32, associating them with the ID, acquisition date and time, blood vessel name, lesion name, and procedure-related information (step S112).

[0044] Next, the processing unit 31 creates a treatment plan for stent placement and stores it in the storage unit 32 (step S113).

[0045] Figure 9 shows an example of the display of the IVUS image P1 and treatment plan screen before the procedure. Based on multiple IVUS images P1 acquired before the procedure, the processing unit 31 displays, for example, an IVUS image P1 which is a cross-sectional image approximately perpendicular to the long axis of the blood vessel and a longitudinal section image which is approximately parallel to the central axis of the blood vessel, side by side. In the example shown in Figure 9, IVUS images P1 of the distal reference area (Ref. Distal), proximal reference area (Ref. Proximal), and minimum lumen area (MLA) are displayed. The user uses the longitudinal section image and cross-sectional image (IVUS image P1) of the blood vessel displayed on the display device 4 to confirm the minimum lumen and plans the size and placement of the stent to be placed. The user specifies the stent placement position and size using the input device 5, and the image processing device 3 receives various specification operations from the input device 5, creates treatment plan information including the stent placement position and size, and stores it in the storage unit 32. The white arrows shown in Figure 9 indicate treatment plan information, which includes, for example, information indicating the frames of IVUS image P1 corresponding to both ends and the center position in the longitudinal direction of the stent to be implanted.

[0046] Next, the processing unit 31 acquires multiple IVUS images P1 after the procedure using the ultrasound line data generation unit 33 (step S114). The processing unit 31 that executes the process in step S114 functions as an acquisition unit that acquires multiple IVUS images P1 of the blood vessel while moving the sensor unit 12 along the direction of the blood vessel's course. The processing unit 31 then stores the multiple IVUS images P1 acquired after the procedure on the blood vessel in the storage unit 32, associating them with ID, imaging date and time, blood vessel name, lesion name, and procedure-related information (step S115).

[0047] Next, the processing unit 31 reads the pre-treatment IVUS image P1 to be used for comparison (step S116). The user can select the pre-treatment IVUS image P1 to be used for comparison using the input device 5, and the processing unit 31 reads the selected IVUS image P1 as the pre-treatment IVUS image P1 to be used for comparison.

[0048] Furthermore, the processing unit 31 reads the treatment plan information associated with the pre-treatment IVUS image P1 from the storage unit 32 (step S117). Based on the read treatment plan information, the processing unit 31 identifies a representative frame image of the pre-treatment IVUS image P1 related to the stent placement position (step S118). Specifically, the processing unit 31 identifies the frame images of the IVUS image P1 corresponding to the positions of both ends and the central part of the stent to be placed. Here, the frame number of the IVUS image P1 of the stent end on the distal end of the blood vessel is n, the frame number of the IVUS image P1 of the stent end on the proximal end of the blood vessel is (n+ns), and the frame number of the IVUS image P1 of the central part of the stent is (n+ns / 2).

[0049] Next, the processing unit 31 inputs the multiple IVUS images P1 acquired after the procedure into the IVUS image recognition learning model 61, performs image recognition processing on the IVUS images P1, and recognizes the stent images (step S119). Through the processing in step S119, the processing unit 31 can distinguish and recognize the multiple IVUS images P1 acquired after the procedure into frame images that include the stent images and frame images that do not include the acquired stent images.

[0050] Based on the recognition result of step S119, the processing unit 31 identifies a representative frame of the IVUS image P1 after the procedure relating to the stent placement position (step S120). Specifically, the processing unit 31 identifies the frame images of the IVUS image P1 corresponding to the positions of both ends and the central part of the stent to be placed. The frame images containing the stent image are consecutive. For example, if the frame images from frame number N to frame number N+Ns contain the stent image, the IVUS image P1 of frame number N, the IVUS image P1 of frame number (N+Ns), and the IVUS image P1 of frame number (N+Ns / 2) are identified as representative frame images. The IVUS image P1 of frame number N is the frame image of the stent end on the distal end of the blood vessel, and the IVUS image P1 of frame number (N+Ns) is the frame image of the stent end on the proximal end of the blood vessel.

[0051] The processing unit 31 associates multiple pre- and post-treatment IVUS images P1 with the identified pre-treatment representative frame image and post-treatment representative frame image (step S121). Specifically, it associates the IVUS image P1 with frame number n, which corresponds to one end of the stent before treatment, with the post-treatment IVUS image P1 with frame number N. Similarly, it associates the IVUS image P1 with frame number ns, which corresponds to the other end of the stent before treatment, with the post-treatment IVUS image P1 with frame number Ns. It also associates the IVUS image P1 with frame number (n+ns / 2), which corresponds to the central part of the stent before treatment, with the post-treatment IVUS image P1 with frame number (N+Ns / 2).

[0052] If the longitudinal scale of the IVUS image P1 acquired before treatment does not perfectly match the longitudinal scale of the IVUS image P1 acquired after treatment, it is advisable to correlate the representative frame image before treatment with the representative frame image after treatment by interpolating or downsampling the IVUS image P1 before or after treatment as necessary.

[0053] Next, the processing unit 31 aligns the longitudinal position and displays the IVUS longitudinal section images before and after the procedure (step S122).

[0054] Figure 10 shows an example of displaying IVUS images P1, etc., before and after treatment. The processing unit 31 displays the longitudinal section image of the blood vessel before treatment and the longitudinal section image of the blood vessel after treatment side by side. Specifically, the processing unit 31 generates a first longitudinal section image approximately parallel to the central axis of the blood vessel based on multiple IVUS images P1 acquired before treatment and displays it on the display device 4. Similarly, the processing unit 31 generates a second longitudinal section image approximately parallel to the central axis of the blood vessel based on multiple IVUS images P1 acquired after treatment and displays it on the display device 4. In the example shown in Figure 10, the processing unit 31 displays the first longitudinal section image before treatment on the upper side and the second longitudinal section image after treatment on the lower side. The display of the distal reference area (Ref.Distal), proximal reference area (Ref.Proximal), and minimum lumen area (MLA) is the same as in Figure 9.

[0055] Next, as shown in Figure 10, the processing unit 31 sets and displays a section designation bar B in the peripheral frame containing the pre-treatment and post-treatment IVUS images P1 (step S123). The section designation bar B is a line image that designates any one IVUS image P1 in the pre-treatment IVUS longitudinal section image and the corresponding IVUS image P1 in the post-treatment IVUS longitudinal section image. More specifically, the section designation bar B is a thick line image that penetrates the pre-treatment IVUS longitudinal section image and the post-treatment IVUS longitudinal section image in the longitudinal direction. Through the processing in step S123, the processing unit 31 displays the section designation bar B in the peripheral frame (the leftmost frame image in Figure 10) containing both the pre-treatment and post-treatment IVUS images P1.

[0056] The user can move the section designation bar B in the left-right direction in Figure 10 by operating the input device 5, and the processing unit 31 determines whether or not it has received a change in the position of the section designation bar B (step S124). If it determines that the change in position has been received (step S124: YES), the processing unit 31 changes the position of the section designation bar B and displays it (step S125).

[0057] If the processing in step S125 is completed or if it is determined that the change to the section designation bar B has not been accepted (step S124: NO), the processing unit 31 displays the pre-treatment and post-treatment IVUS images P1 (cross-sectional images) corresponding to the position of the section designation bar B (step S126).

[0058] The pre- and post-treatment IVUS images P1 displayed by the above process correspond in their longitudinal positions, but their circumferential orientations do not necessarily match. The user can rotate the IVUS image P1 by operating the input device 5, and the processing unit 31 accepts the circumferential position correction in the IVUS image P1 (step S127). Then, the processing unit 31 rotates the IVUS image P1 based on the accepted position correction, that is, the angle information for rotating the IVUS image P1, and corrects its circumferential orientation (step S128).

[0059] The processing unit 31 displays an auto-play button, which the user can operate by operating the input device 5. The processing unit 31 determines whether or not it has received an auto-play instruction (step S129). If it determines that it has not received an auto-play instruction (step S129: NO), the processing unit 31 returns to step S123.

[0060] If the processing unit 31 determines that it has received an instruction for automatic playback (step S129: YES), it performs an automatic playback display that sequentially displays each of the multiple IVUS images P1 before and after the procedure, from the IVUS image P1 at the tip to the IVUS image P1 at the proximal end (step S130), and returns the process to step S123.

[0061] As described above, with the image processing device 3 etc. according to this embodiment 1, when displaying IVUS images P1 of a blood vessel acquired before and after treatment of the blood vessel, the position of the blood vessel in the longitudinal direction can be aligned and the IVUS images P1 before and after treatment can be displayed in correspondence.

[0062] Specifically, based on the treatment plan information, multiple IVUS images P1 acquired before and after the procedure can be associated by associating the IVUS images P1 of the ends and central parts of the stent to be implanted with the IVUS images P1 of the ends and central parts of the stent identified by image recognition of the IVUS image P1 after the procedure.

[0063] After aligning the observation positions along the long axis of the pre- and post-treatment IVUS images P1, it becomes possible to compare the pre- and post-treatment IVUS images P1 and extract the differences. For example, if a calcified lesion is removed, the volume removed can be calculated three-dimensionally by comparing the IVUS images P1.

[0064] In this embodiment 1, an example was described in which the IVUS image P1 before and after treatment is displayed by aligning the longitudinal position of the blood vessel. However, the OFDI image P2 may also be configured to be displayed before and after treatment by aligning the longitudinal position of the blood vessel using a similar process.

[0065] Furthermore, in this embodiment 1, we have described an IVUS image recognition learning model 61 that can recognize stent images by classifying the IVUS image P1 on a pixel-by-pixel basis using semantic segmentation. However, the image processing device 3 may be configured using a learning model that determines the presence or absence of stents on a frame-by-frame basis.

[0066] (Embodiment 2) The image diagnostic device 100 according to Embodiment 2 differs from Embodiment 1 in that it displays the pre- and post-treatment IVUS images P1 in correspondence by aligning the longitudinal position of the blood vessel based on the characteristic quantities outside the balloon expansion site. The other components of the image processing device 3 are the same as those of the image processing device 3 according to Embodiment 1, so the same reference numerals are used for the same parts, and detailed explanations are omitted.

[0067] Figure 11 is a flowchart showing the image processing procedure according to this second embodiment, and Figures 12A, 2B, and 12C are explanatory diagrams showing the image processing method according to this second embodiment. The processing unit 31 of the image processing device 3 acquires a plurality of IVUS images P1 (step S211) and stores them in the storage unit 32 (step S212), similar to the first embodiment.

[0068] The user uses the longitudinal and transverse images (IVUS image P1) of the blood vessel displayed on the display device 4 to identify the narrowest part and plan the location for balloon expansion. The user specifies the target area for balloon expansion using the input device 5, and the image processing device 3 receives various specification operations from the input device 5 and creates treatment plan information including the target area for balloon expansion (planned expansion range), as shown in Figure 12A, and stores it in the storage unit 32 (step S213).

[0069] Next, the processing unit 31, similar to Embodiment 1, acquires multiple IVUS images P1 after the procedure using the ultrasound line data generation unit 33 (step S214) and stores them in the storage unit 32 (step S215). The processing unit 31 reads out the IVUS image P1 before the procedure for comparison (step S216). It also reads out the treatment plan information associated with the IVUS image P1 before the procedure from the storage unit 32 (step S217).

[0070] The processing unit 31 inputs multiple IVUS images P1 acquired before and after the procedure into the IVUS image recognition learning model 61, performs image recognition processing on the IVUS images P1, and recognizes the lumen image or the external elastic membrane (step S218).

[0071] Next, the processing unit 31 calculates the lumen diameter or vessel diameter outside the target site before the procedure in which balloon dilation should be performed, based on the image recognition result from step S218 and the treatment plan information (step S219). Specifically, as shown in Figure 12B, the processing unit 31 can calculate the lumen diameter based on the lumen image obtained by image recognition of the IVUS image P1 outside the target site. The processing unit 31 can also calculate the vessel diameter based on the external elastic membrane image obtained by image recognition of the IVUS image P1 outside the target site.

[0072] On the other hand, the processing unit 31 calculates the lumen diameter or vessel diameter outside the target site after the balloon dilation procedure (step S220). Since the longitudinal positions of the IVUS image P1 acquired before the procedure and the IVUS image P1 acquired after the procedure roughly correspond, the processing unit 31 can identify the IVUS image P1 outside the target site where the balloon dilation was performed based on the procedure plan information, as shown in Figure 12C, and calculate the lumen diameter based on the lumen image obtained by image recognition of the IVUS image P1. Similarly, the processing unit 31 can calculate the vessel diameter. The processing unit 31 may also be configured to calculate the lumen diameter or vessel diameter for each of the multiple IVUS images P1 after the procedure.

[0073] Next, the processing unit 31 associates the pre- and post-treatment IVUS images P1 so that the lumen diameter or vessel diameter outside the expansion range matches (step S221). Since the lumen diameter or vessel diameter may not perfectly match before and after treatment due to the effects of pulsation, etc., the processing unit 31 performs a process to associate the pre- and post-treatment IVUS images P1 so that the lumen diameter or vessel diameter outside the expansion range approximately matches (substantially matches). For example, the processing unit 31 calculates the difference between the lumen diameter or vessel diameter based on the pre-treatment IVUS image P1 and the corresponding lumen diameter or vessel diameter based on the post-treatment IVUS image P1, and identifies a correspondence between the pre-treatment IVUS image P1 and the post-treatment IVUS image P1 such that the sum of these differences based on multiple IVUS images P1 is minimized. In other words, the processing unit 31 associates both IVUS images P1 so that the observation position of the IVUS image P1 acquired before treatment matches the observation position of the IVUS image P1 acquired after treatment. By matching the pre-treatment IVUS image P1 with the post-treatment IVUS image P1 in such a way that the sum of the above differences is minimized, it becomes possible to make a correspondence such that the lumen diameter or vessel diameter outside the dilated range substantially matches.

[0074] The details of the process in step S211 are as follows. Before the process in step S221, it is assumed that the IVUS image P1 of the nth frame is associated with the IVUS image P1 of the nth frame. n is an integer greater than or equal to 1. The processing unit 31 calculates the difference in lumen diameter or vessel diameter based on the object-extracted IVUS image P1' of the corresponding frame number, for example. The processing unit 31 calculates the difference in lumen diameter or vessel diameter for a predetermined number of object-extracted IVUS images P1'. Next, the processing unit 31 similarly calculates the difference between the lumen diameter or vessel diameter based on the object-extracted IVUS image P1' of the (n+α)th frame and the lumen diameter or vessel diameter based on the object-extracted IVUS image P1' of the nth frame. α is an integer and corresponds to the amount of translation of the 3D lumen image. The processing unit 31 verifies the correspondence between the object-extracted IVUS images P1' before and after treatment while incrementing the variable α by 1. Then, it identifies the variable α that minimizes the sum of the differences, for example, the sum of the squares of the differences.

[0075] After completing the processing in step S221, the processing unit 31 performs the same processing as in steps S122 to S130 of Embodiment 1.

[0076] As described above, according to the image processing device 3, etc. of this embodiment 2, by associating multiple IVUS images P1 acquired before and after the procedure so that the lumen diameter or vessel diameter outside the target site before the procedure in which balloon expansion is to be performed matches the lumen diameter or vessel diameter of the vessel after the procedure as closely as possible, the longitudinal position of the vessel can be aligned and the IVUS images P1 before and after the procedure can be displayed in association.

[0077] (Embodiment 3) The image diagnostic device 100 according to Embodiment 3 differs from Embodiment 1 in that, when stent expansion (optimization) is performed, it displays the pre- and post-treatment IVUS images P1 in correspondence with the pre- and post-treatment stent images, aligning the longitudinal position of the blood vessel. The other components of the image processing device 3 are the same as those of the image processing device 3 according to Embodiment 1, so the same reference numerals are used for the same parts, and detailed explanations are omitted.

[0078] Figure 13 is a flowchart of the image processing procedure according to this third embodiment. The processing unit 31 of the image processing device 3 performs the same processing as steps S111 to S117 of the first embodiment (steps S311 to S317).

[0079] After completing the processing in step S317, the processing unit 31 inputs the multiple IVUS images P1 acquired before and after the procedure into the IVUS image recognition learning model 61, performs image recognition processing on the IVUS images P1, and recognizes the stent image (step S318).

[0080] Next, the processing unit 31 identifies a representative frame relating to the stent placement position before expansion (step S319). Specifically, the processing unit 31 identifies frame images of IVUS images P1 corresponding to the positions of both ends and the central part of the stent from a plurality of IVUS images P1 acquired before the procedure.

[0081] Furthermore, the processing unit 31 identifies a representative frame relating to the stent placement position after expansion (step S320). Specifically, the processing unit 31 identifies frame images of IVUS images P1 corresponding to the positions of both ends and the central part of the stent from a plurality of IVUS images P1 acquired after the procedure.

[0082] Next, the processing unit 31 associates multiple IVUS images P1 before treatment (before optimization of the stent placement state) and after treatment (after optimization of the stent placement state) based on the identified representative frame image before treatment and representative frame image after treatment (step S321). Specifically, it associates the IVUS image P1 corresponding to one end of the stent before treatment with the IVUS image P1. Similarly, it associates the IVUS image P1 corresponding to the other end of the stent before treatment with the IVUS image P1 after treatment. It also associates the IVUS image P1 corresponding to the central part of the stent before treatment with the IVUS image P1 after treatment.

[0083] After completing the processing in step S321, the processing unit 31 performs the same processing as in steps S122 to S130 of Embodiment 1.

[0084] As described above, according to the image processing device 3 of this embodiment 3, when stent expansion is performed, multiple IVUS images P1 acquired before and after the procedure can be associated by aligning the position of the stent image, and the IVUS images P1 before and after the procedure can be displayed in association with each other by aligning the longitudinal position of the blood vessel.

[0085] (Embodiment 4) The image diagnostic device 100 according to Embodiment 4 differs from Embodiment 1 in that it displays the pre-treatment and post-treatment IVUS images P1 in correspondence with each other by aligning the longitudinal position of the blood vessels based on the image of the calcified area in the IVUS image P1. The other components of the image processing device 3 are the same as those of the image processing device 3 according to Embodiment 1, so the same reference numerals are used for the same parts, and detailed explanations are omitted.

[0086] Figure 14 is a flowchart showing the image processing procedure according to Embodiment 4. The processing unit 31 of the image processing device 3 performs the same processing as steps S111 to S121 of Embodiment 1. The processing in step S121 makes the positions of the pre-treatment IVUS image P1 and the post-treatment IVUS image P1 in the longitudinal direction of the blood vessel approximately coincide. The processing unit 31 according to Embodiment 4 fine-tunes the correspondence between the pre-treatment and post-treatment IVUS images P1 by performing the following processing.

[0087] After completing the processing in step S121, the processing unit 31 inputs multiple IVUS images P1 acquired before and after treatment into the IVUS image recognition learning model 61, performs image recognition processing on the IVUS images P1, and recognizes the calcified area images (step S421). The processing unit 31 identifies the frame image of the IVUS image P1 that contains the calcified area images. The processing unit 31 also identifies the center angle of the calcified area images in the IVUS images P1.

[0088] Figure 15 is an explanatory diagram showing multiple IVUS images P1 before and after treatment, arranged along the longitudinal direction of a blood vessel. The upper figure shows multiple IVUS images P1 before treatment, arranged along the longitudinal direction of a blood vessel. The white areas indicate IVUS images P1 that do not include images of calcified areas, while the hatched areas indicate IVUS images P1 that include images of calcified areas. "20°", "70°", "120°", etc., indicate the central angle of the calcified area in the IVUS image P1 that includes images of calcified areas, that is, the circumferential position of the calcified area in the IVUS image P1. The lower figure shows multiple IVUS images P1 after treatment, arranged along the longitudinal direction of a blood vessel. Comparing the IVUS images P1 before and after treatment, it can be seen that the observation position and observation direction along the longitudinal direction of the blood vessel are different in the two cases.

[0089] Next, the processing unit 31 associates the pre-treatment and post-treatment IVUS images P1 so that the longitudinal positions of multiple calcification sites coincide (step S422). For example, if multiple consecutive IVUS images P1 contain images of calcification sites, the processing unit 31 recognizes the multiple IVUS images P1 as a single group and identifies the IVUS image P1 at the longitudinal center of the blood vessel as the representative frame image. The processing unit 31 treats the pre-treatment representative frame image and the post-treatment representative frame image with the closest longitudinal distance as a pair and calculates the distance between the representative frame images. If there are multiple pairs of representative frame images, the processing unit 31 calculates the distance between each representative frame image, fine-tunes the longitudinal position of the IVUS image P1 so that the sum of the distances is minimized, and associates the pre-treatment IVUS image P1 with the post-treatment IVUS image P1.

[0090] Next, the processing unit 31 corrects the rotation angle of the IVUS image P1 so that the central angles of the calcification site images in the IVUS image P1 match (step S423). For example, the processing unit 31 calculates the difference between the central angle of the calcification site image included in the representative frame image before processing and the central angle of the calcification site image included in the corresponding representative frame image after treatment. The processing unit 31 then similarly calculates the difference in central angles for other representative frame images, calculates a correction amount for the rotation angle so that the sum of the differences is minimized, and corrects the orientation of the IVUS image P1 by rotating the IVUS image P1 by the calculated correction amount. In the example shown in Figure 15, the orientation of the IVUS image P1 before and after treatment can be made to match by subtracting 30 degrees from the circumferential angle of the IVUS image P1 after treatment, or in other words, by rotating it 30 degrees clockwise.

[0091] The processing unit 31 displays the representative frame image before treatment, in which the longitudinal position has been finely adjusted and the center angle has been corrected in steps S422 and S423, and the representative frame image after treatment, in correspondence (step S424).

[0092] As described above, the image processing device 3, etc., according to this fourth embodiment can more accurately align the longitudinal position of the blood vessel and display the IVUS images P1 before and after treatment in correspondence with each other compared to the first embodiment. Furthermore, the rotation angle, i.e., orientation, of the IVUS images P1 before and after treatment can be aligned and displayed.

[0093] In this embodiment 4, an example was described in which the IVUS images P1 before and after the procedure are associated based on the stent placement position, and the relationship and center angle of the IVUS images P1 are corrected based on the position and center angle of the calcification site image. However, the system may be configured to display the associated IVUS images P1 before and after the procedure using only the method of this embodiment 4.

[0094] Furthermore, in this embodiment 4, an example was described in which pre- and post-treatment IVUS images P1 are matched using calcification site images so that the observation position and observation direction in the longitudinal direction of the blood vessel coincide. However, the system may be configured to perform similar processing based on images of other objects such as side branches, attenuated plaque, epicardium, or veins.

[0095] (Embodiment 5) The image diagnostic apparatus 100 according to Embodiment 5 differs from Embodiment 1 in that it automatically performs the process of step S116 according to Embodiment 1. The other components of the image processing apparatus 3 are the same as those of the image processing apparatus 3 according to Embodiment 1, so the same reference numerals are used for the same parts, and detailed descriptions are omitted.

[0096] Figure 16 is a flowchart showing the image processing procedure according to this embodiment 5. In step S116, the processing unit 31 performs the following processing. The processing unit 31 reads the imaging date and time, blood vessel name, lesion name, and treatment-related information related to the post-treatment IVUS image P1 from the storage unit 32 (step S551). Next, the processing unit 31 searches for an IVUS image P1 with an earlier imaging date and time, matching blood vessel name, lesion name, etc., and consistent treatment details (step S552), and reads the pre-treatment IVUS image P1 that matches the search conditions from the tomographic image DB 63 (step S553).

[0097] As described above, according to the image processing device 3, etc. of this embodiment 5, when an IVUS image P1 is acquired after treatment, the data of the IVUS image P1 before treatment, which is used as a comparison target, is automatically read from the storage unit 32, and the longitudinal tomographic image P1 of the blood vessel and the IVUS image P1 based on each IVUS image P1 can be displayed side by side on the display device 4. [Explanation of Symbols]

[0098] 1. Diagnostic imaging catheter 2 MDU 3 Image Processing Device 4 Display device 5 Input devices 11 probes 12 Sensor section 12a Ultrasonic Transceiver Unit 12b Optical Transceiver Unit 12c Housing 13 shafts 14 Guide wire insertion section 15 Connector section 30 Recording media 31 Processing Unit 32 Storage section 33 Ultrasonic line data generation unit 34 Optical line data generation unit 61 IVUS Image Recognition Learning Model 62 OFDI Image Recognition Learning Models P1 IVUS image P1' Object Extraction IVUS Image P2 OFDI image 100 Imaging diagnostic equipment 101 Intravascular Examination Device 102 Angiography equipment

Claims

1. A sensor unit for scanning a tubular organ is moved along the direction of travel of the tubular organ, and multiple tomographic images of the tubular organ are acquired. From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before treatment and the tomographic images acquired after treatment are displayed in correspondence. A computer program that causes a computer to perform a process, The procedure plan information includes information showing tomographic images of the end or central part of the target site where the stent is to be placed, among multiple tomographic images acquired before the procedure, and shows tomographic images of the target site where the procedure on the tubular organ is to be performed. Based on the acquired treatment plan information, select at least one tomographic image related to the target site on the tubular organ where treatment is to be performed. Image recognition processing of multiple tomographic images acquired after the procedure recognizes tomographic images including images of the stent. The post-treatment tomographic image of the end or central portion of the implanted stent is identified as the tomographic image corresponding to the pre-treatment tomographic image of the said end or central portion. A computer program that causes the aforementioned computer to perform a process.

2. A sensor unit for scanning a tubular organ is moved along the direction of travel of the tubular organ while acquiring multiple cross-sectional images of the tubular organ, From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before treatment and the tomographic images acquired after treatment are displayed in correspondence. A computer program that causes a computer to perform a process, The procedure plan information includes information showing the tomographic image of the area to be treated with balloon dilation from among multiple tomographic images acquired before the procedure, and the tomographic image of the area to be treated on the tubular organ. Based on the acquired treatment plan information, select at least one tomographic image related to the target site on the tubular organ where treatment is to be performed. By comparing the feature quantities obtained from image recognition processing of tomographic images outside the target area where balloon expansion is to be performed with the same feature quantities obtained from image recognition processing of multiple tomographic images acquired after the procedure, the post-treatment tomographic image corresponding to the pre-treatment tomographic image outside the target area where balloon expansion is to be performed is identified. A computer program that causes the aforementioned computer to perform a process.

3. A sensor unit for scanning a tubular organ is moved along the direction of travel of the tubular organ while acquiring multiple cross-sectional images of the tubular organ, From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before treatment and the tomographic images acquired after treatment are displayed in correspondence. A computer program that causes a computer to perform a process, When a procedure is performed to optimize the placement of a stent in the tubular organ, the tomographic image including the stent is recognized by image recognition processing of multiple tomographic images acquired before and after the procedure. The tomographic image of the end or central portion of the optimized stent after treatment is identified as the tomographic image corresponding to the pre-treatment tomographic image of the end or central portion of the implanted stent before optimization. A computer program that causes the aforementioned computer to perform a process.

4. A sensor unit for scanning a tubular organ is moved along the direction of travel of the tubular organ while acquiring multiple cross-sectional images of the tubular organ, From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before treatment and the tomographic images acquired after treatment are displayed in correspondence. A computer program that causes a computer to perform a process, If object images are included in the tomographic images before and after treatment, the correspondence between the object images included in the tomographic images before treatment and the object images included in the tomographic images after treatment is identified by matching them. The tomographic images acquired before the procedure and those acquired after the procedure are displayed in correspondence so that the position in the direction of course and the angle in the circumferential direction of the tubular organ match. A computer program that causes the aforementioned computer to perform a process.

5. Recognizing object images contained in multiple tomographic images acquired before and after the procedure, The tomographic image acquired before treatment and the tomographic image acquired after treatment are matched so that the positions of the object images contained in the tomographic images before and after treatment correspond. A computer program according to any one of claims 1 to 4.

6. Recognizing object images contained in multiple tomographic images acquired before and after the procedure, Based on the aforementioned treatment plan information, the correspondence between the tomographic image acquired before treatment and the tomographic image acquired after treatment is identified, and then the correspondence is modified so that the positions of the object images included in the tomographic image before and after treatment correspond. The computer program according to claim 1 or claim 2.

7. When object images are included in the pre- and post-treatment tomographic images corresponding to multiple areas separated in the direction of travel, the correspondence between the pre-treatment tomographic images and the post-treatment tomographic images is identified by associating the pre-treatment tomographic images of the multiple areas with the post-treatment tomographic images of the multiple areas. The tomographic images acquired before the procedure and those acquired after the procedure are displayed in correspondence so that the direction of the tubular organ is aligned. A computer program according to any one of claims 1 to 4.

8. The object images include images of plaque, collateral branches, epicardium, or veins. The computer program according to claim 5.

9. When multiple tomographic images of the tubular organ are obtained, the multiple tomographic images obtained, identification information that identifies the tubular organ related to the tomographic images, and treatment-related information related to the treatment of the tubular organ are stored in the storage unit in association with each other. Based on the identification information and the treatment-related information, multiple tomographic images acquired before the treatment are read from the storage unit to be displayed in association with multiple tomographic images acquired after the treatment. A computer program according to any one of claims 1 to 4.

10. Multiple tomographic images of a tubular organ are obtained by moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ. From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before treatment and the tomographic images acquired after treatment are displayed in correspondence. An image processing method, The procedure plan information includes information showing tomographic images of the end or central part of the target site where the stent is to be placed, among multiple tomographic images acquired before the procedure, and shows tomographic images of the target site where the procedure on the tubular organ is to be performed. Based on the acquired treatment plan information, select at least one tomographic image related to the target site on the tubular organ where treatment is to be performed. Image recognition processing of multiple tomographic images acquired after the procedure recognizes tomographic images including images of the stent. The post-treatment tomographic image of the end or central portion of the implanted stent is identified as the tomographic image corresponding to the pre-treatment tomographic image of the said end or central portion. Image processing methods.

11. An acquisition unit that acquires multiple tomographic images of a tubular organ while moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, A processing unit that performs the process of displaying the tomographic image acquired by the acquisition unit on a display device. Equipped with, The aforementioned processing unit, From among a plurality of tomographic images acquired before the procedure on the tubular organ, at least one tomographic image related to the area on the tubular organ to be treated is selected. Image recognition processing of multiple tomographic images acquired after the procedure identifies the post-treatment tomographic image corresponding to the selected pre-treatment tomographic image. Based on the identification results of the tomographic images, the tomographic images acquired before the procedure and the tomographic images acquired after the procedure are displayed in correspondence. Furthermore, the processing unit, The procedure plan information includes information showing tomographic images of the end or central part of the target site where the stent is to be placed, among multiple tomographic images acquired before the procedure, and shows tomographic images of the target site where the procedure on the tubular organ is to be performed. Based on the acquired treatment plan information, select at least one tomographic image related to the target site on the tubular organ where treatment is to be performed. Image recognition processing of multiple tomographic images acquired after the procedure recognizes tomographic images including images of the stent. The post-treatment tomographic image of the end or central portion of the implanted stent is identified as the tomographic image corresponding to the pre-treatment tomographic image of the said end or central portion. Image processing device.

Images