Computer program, image processing method, and image processing device

Abstract

In the present invention, a computer is caused to perform processing in which: a plurality of tomographic images of a hollow organ are acquired while moving a sensor unit, which is for scanning the hollow organ, along the travel direction of the hollow organ; selected from a plurality of tomographic images acquired prior to a procedure with respect to the hollow organ is at least one tomographic image that relates to a target site at which the procedure will be performed on the hollow organ; a tomographic image from after the procedure that corresponds to the selected one tomographic image from before the procedure is identified with image recognition processing with respect to a plurality of tomographic images acquired after the procedure; and, on the basis of the identification result of the tomographic image, the tomographic image acquired prior to the procedure and the tomographic image acquired after the procedure are associated and displayed.

Description

Computer program, image processing method, and image processing device

[0001] The present invention relates to a computer program, an image processing method, and an image processing device.

[0002] Intravascular treatments, such as percutaneous coronary intervention (PCI), are being performed as minimally invasive treatments for ischemic heart diseases such as angina pectoris and myocardial infarction. For preoperative diagnosis or postoperative confirmation of endovascular treatment, intravascular imaging diagnostic devices, such as intravascular ultrasound (IVUS: Intravascular Ultrasound) and optical coherence tomography (OFDI: Optical Frequency Domain Imaging), are used.

[0003] JP-A-11-56752 Publication Special Publication No. 2010-508973

[0004] When medical professionals check the condition of blood vessels using ultrasound tomographic images acquired after treatment, they compare them with ultrasound tomographic images acquired before treatment to confirm the effectiveness of the treatment. When comparing ultrasound tomographic images, it is necessary to check the ultrasound tomographic images taken before and after treatment at the same longitudinal position of the blood vessel. However, it is difficult to precisely align the scanning start position within the blood vessel, and the ultrasound tomographic images acquired before and after treatment are misaligned in the longitudinal direction. Currently, medical professionals manually select ultrasound tomographic image data showing the blood vessel in the same position, manually align the longitudinal position, and confirm the effectiveness of the treatment. The same situation applies to optical coherence tomographic images.

[0005] One aspect of the present disclosure aims to provide a computer program, an image processing method, and an image processing device that can align and display the longitudinal position of a tubular organ when displaying tomographic images of the tubular organ obtained before and after treatment on the tubular organ.

[0006] A computer program according to one aspect of the present disclosure causes a computer to perform the following process: acquire multiple tomographic images of a tubular organ while moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ; select at least one tomographic image from the multiple tomographic images acquired before treatment of the tubular organ that is related to a target area where treatment of the tubular organ is to be performed; identify a post-treatment tomographic image corresponding to the selected one pre-treatment tomographic image through image recognition processing of the multiple tomographic images acquired after treatment; and, based on the result of identifying the tomographic images, display the tomographic image acquired before treatment in association with the tomographic image acquired after treatment.

[0007] An image processing method according to one aspect of the present disclosure acquires multiple tomographic images of a tubular organ by moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, selects at least one tomographic image from the multiple tomographic images acquired before treatment of the tubular organ that relates to a target area where treatment of the tubular organ is to be performed, identifies a post-treatment tomographic image corresponding to the selected one pre-treatment tomographic image through image recognition processing of the multiple tomographic images acquired after treatment, and displays the tomographic image acquired before treatment and the tomographic image acquired after treatment in association with each other based on the result of tomographic image identification.

[0008] An image processing device according to one aspect of the present disclosure comprises an acquisition unit that acquires multiple tomographic images of a tubular organ while moving a sensor unit for scanning the tubular organ along the direction of travel of the tubular organ, and a processing unit that executes processing to display the tomographic images acquired by the acquisition unit on a display device, wherein the processing unit selects at least one tomographic image related to a target site where treatment is to be performed on the tubular organ from among the multiple tomographic images acquired before treatment on the tubular organ, identifies a post-treatment tomographic image corresponding to the selected one pre-treatment tomographic image by image recognition processing of the multiple tomographic images acquired after treatment, and displays the tomographic image acquired before treatment and the tomographic image acquired after treatment in association with each other based on the result of tomographic image identification.

[0009] In one aspect of the present disclosure, when displaying tomographic images of a hollow organ acquired before and after a treatment on the hollow organ, the longitudinal position of the hollow organ can be aligned and displayed.

[0010] 1 is an explanatory diagram showing an example of the configuration of an image diagnostic device. FIG. 1 is an explanatory diagram showing an example of the configuration of an image diagnostic catheter. FIG. 2 is an explanatory diagram schematically showing a cross section of a blood vessel through which a sensor unit is inserted. FIG. 2 is an explanatory diagram of a tomographic image. FIG. 3 is an explanatory diagram of a tomographic image. FIG. 3 is a block diagram showing an example of the configuration of an image processing device. FIG. 4 is a block diagram showing an example of the configuration of an IVUS image recognition learning model. FIG. 4 is a block diagram showing an example of the configuration of an IVUS image recognition learning model. FIG. 5 is a flowchart showing an image processing procedure according to the first embodiment. FIG. 6 is a flowchart showing an image processing procedure according to the first embodiment. FIG. 7 is a display example of an IVUS image before treatment and a treatment planning screen. FIG. 8 is a display example of IVUS images before and after treatment. FIG. 9 is a flowchart showing an image processing procedure according to the second embodiment. FIG. 10 is an explanatory diagram showing an image processing method according to the second embodiment. FIG. 11 is an explanatory diagram showing an image processing method according to the second embodiment. FIG. 12 is a flowchart showing an image processing procedure according to the third embodiment. FIG. 13 is a flowchart showing an image processing procedure according to the fourth embodiment. FIG. 14 is an explanatory diagram showing a plurality of IVUS images P1 before and after treatment arranged in the longitudinal direction of a blood vessel. FIG. 15 is a flowchart showing an image processing procedure according to the fifth embodiment.

[0011] The computer program, image processing method, and image processing device of the present disclosure will be described in detail below with reference to the drawings illustrating embodiments thereof. In the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example. However, the hollow organs targeted by the catheterization treatment are not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines. Furthermore, the present invention is not limited to these examples, but is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are intended to be included. At least some of the embodiments described below may be combined in any desired manner.

[0012] In this embodiment, an imaging diagnostic device using a dual-type catheter equipped with both intravascular ultrasound (IVUS) and optical coherence tomography (OFDI) functions will be described. The dual-type catheter is provided with a mode for acquiring ultrasound tomographic images using only IVUS, a mode for acquiring optical coherence tomographic images using only OFDI, and a mode for acquiring tomographic images using both IVUS and OFDI, and these modes can be switched between. Hereinafter, ultrasound tomographic images and optical coherence tomographic images will be referred to as IVUS images and OFDI images, respectively, as appropriate. IVUS images and OFDI images will also be collectively referred to as tomographic images.

[0013] (Embodiment 1) FIG. 1 is an explanatory diagram showing an example of the configuration of an imaging diagnostic apparatus 100. The imaging diagnostic apparatus 100 of this embodiment 1 includes an intravascular examination apparatus 101, an angiography apparatus 102, an image processing device 3, a display device 4, and an input device 5. The intravascular examination apparatus 101 includes an imaging diagnostic catheter 1 and an MDU (Motor Drive Unit) 2. The imaging diagnostic catheter 1 is connected to the image processing device 3 via the MDU 2. The image processing device 3 is connected to the display device 4 and the input device 5. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone. The display device 4 and the input device 5 may be stacked together to form a touch panel. Alternatively, the input device 5 and the image processing device 3 may be integrated. Furthermore, the input device 5 may be a sensor that accepts gesture input, gaze input, or the like.

[0014] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that images blood vessels from outside the patient's body using X-rays while injecting a contrast agent into the patient's blood vessels, thereby obtaining an angio image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and images an X-ray fluoroscopic image of the patient by the X-ray sensor receiving X-rays irradiated from the X-ray source. The angiography device 102 outputs the angio image obtained by imaging to the image processing device 3, and the angiography device 102 displays the angio image on the display device 4 via the image processing device 3.

[0015] FIG. 2 is an explanatory diagram showing an example of the configuration of the diagnostic imaging catheter 1. Note that the area enclosed by the dashed-dotted line on the upper side of FIG. 2 is an enlarged version of the area enclosed by the dashed-dotted line on the lower side. The diagnostic imaging catheter 1 includes a probe 11 and a connector 15 disposed at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector 15. In the following description, the side of the diagnostic imaging catheter 1 farther from the connector 15 will be referred to as the distal end, and the connector 15 side will be referred to as the proximal end. The probe 11 includes a catheter sheath 11a, and a guidewire insertion portion 14 at its distal end, through which a guidewire can be inserted. The guidewire insertion portion 14 forms a guidewire lumen and is used to receive a guidewire previously inserted into a blood vessel and guide the probe 11 to the affected area via the guidewire. The catheter sheath 11a forms a continuous tubular portion extending from the guidewire insertion portion 14 to the connector 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip side of the shaft 13.

[0016] The sensor unit 12 has a housing 12c, the distal end of which is hemispherically shaped to reduce friction and snagging with the inner surface of the catheter sheath 11a. The housing 12c contains an ultrasonic transmitter / receiver 12a, which transmits ultrasonic waves into a blood vessel and receives reflected waves from the blood vessel, and an optical transmitter / receiver 12b, which transmits near-infrared light into the blood vessel and receives reflected light from the blood vessel. In the example shown in FIG. 2 , the ultrasonic transmitter / receiver 12a is provided at the distal end of the probe 11, and the optical transmitter / receiver 12b is provided at the proximal end. That is, the ultrasonic transmitter / receiver 12a and the optical transmitter / receiver 12b are disposed within the housing 12c, spaced apart a predetermined distance along the central axis of the shaft 13 (along the two-dot chain line in FIG. 2 ). The ultrasonic transmitter / receiver 12a and the optical transmitter / receiver 12b are disposed such that the transmission and reception directions of the ultrasonic waves and near-infrared light are approximately 90 degrees relative to the axial direction of the shaft 13 (i.e., radially of the shaft 13). It is desirable that the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are mounted slightly offset from the radial direction so as not to receive waves and light reflected by the inner surface of the catheter sheath 11a. In this embodiment 1, for example, as shown by the arrow in Figure 2, the ultrasonic transmitting / receiving unit 12a is disposed in an orientation in which the direction in which it irradiates ultrasonic waves is inclined toward the base end with respect to the radial direction, and the optical transmitting / receiving unit 12b is disposed in an orientation in which the direction in which it irradiates near-infrared light is inclined toward the tip end with respect to the radial direction.

[0017] An electric signal cable (not shown) connected to the ultrasound transmitting / receiving unit 12a and an optical fiber cable (not shown) connected to the optical transmitting / receiving unit 12b are inserted into the shaft 13. The probe 11 is inserted into a blood vessel from the distal end side. The sensor unit 12 and the shaft 13 can move forward and backward within the catheter sheath 11a and can also rotate circumferentially. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13.

[0018] The MDU 2 is a driving device to which the connector 15 of the diagnostic imaging catheter 1 is detachably attached, and which drives a built-in motor in response to operation by a user (medical professional) to control the operation of the diagnostic imaging catheter 1 inserted into a blood vessel. For example, the MDU 2 performs a pull-back operation to rotate the sensor unit 12 and shaft 13 inserted into the probe 11 circumferentially while pulling them toward the MDU 2 at a constant speed. The sensor unit 12 rotates while moving from the distal end to the proximal end by the pull-back operation, thereby continuously scanning the blood vessel at predetermined time intervals to continuously capture multiple transverse cross-sectional images approximately perpendicular to the probe 11 at predetermined intervals. The MDU 2 outputs reflected wave signals of ultrasound received by the ultrasound transmitting / receiving unit 12a and reflected light signals received by the optical transmitting / receiving unit 12b to the image processing device 3.

[0019] The image processing device 3 acquires reflected wave signals of the ultrasound output from the ultrasound transmitting / receiving unit 12a via the MDU 2 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data. The ultrasound line data is data indicating the reflection intensity of ultrasound in the depth direction of the blood vessel as seen from the ultrasound transmitting / receiving unit 12a. The image processing device 3 constructs an IVUS image P1 (see FIGS. 4A and 4B ) representing a transverse layer of the blood vessel based on the generated ultrasound line data.

[0020] The image processing device 3 also acquires interference light data by causing interference between the reflected light signal output from the optical transceiver 12b via the MDU 2 and reference light obtained by separating the light from the light source, and generates optical line data based on the acquired interference light data. The optical line data indicates the reflection intensity of the reflected light in the depth direction of the blood vessel as seen from the optical transceiver 12b. The image processing device 3 constructs an OFDI image P2 (see FIGS. 4A and 4B ) showing the transverse layers of the blood vessel based on the generated optical line data.

[0021] Here, the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b, and the IVUS image P1 and OFDI image P2 constructed from the ultrasonic line data and optical line data will be described.

[0022] Fig. 3 is an explanatory diagram showing a schematic cross section of a blood vessel through which the sensor unit 12 is inserted, and Figs. 4A and 4B are explanatory diagrams of tomographic images. First, the operation of the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b within the blood vessel, and the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b will be described with reference to Fig. 3.

[0023] When imaging of a tomographic image begins with the sensor unit 12 and shaft 13 inserted into a blood vessel, the sensor unit 12 rotates in the direction indicated by the arrow, with the central axis of the shaft 13 serving as the center of rotation. At this time, the ultrasound transmitter / receiver 12a transmits and receives ultrasound waves at each rotation angle. Lines 1, 2, ... 512 indicate the transmission and reception direction of ultrasound waves at each rotation angle. In this embodiment 1, the ultrasound transmitter / receiver 12a intermittently transmits and receives ultrasound waves 512 times during a 360-degree rotation (one rotation) within the blood vessel. Since the ultrasound transmitter / receiver 12a acquires one line of data in the transmission and reception direction by transmitting and receiving ultrasound waves once, 512 pieces of ultrasound line data extending radially from the center of rotation can be obtained during one rotation. The 512 pieces of ultrasound line data are dense near the center of rotation but become sparse as they move away from the center of rotation. Therefore, the image processing device 3 can generate pixels in the empty spaces of each line by a well-known interpolation process, thereby constructing a two-dimensional IVUS image P1 as shown in FIG. 4A.

[0024] Similarly, the optical transceiver 12b also transmits and receives near-infrared light (measurement light) at each rotation angle. The optical transceiver 12b also transmits and receives measurement light 512 times while rotating 360 degrees within the blood vessel, so that 512 optical line data extending radially from the center of rotation can be obtained during one rotation. For the optical line data, the image processing device 3 generates pixels in the empty spaces of each line by well-known interpolation processing, thereby constructing the two-dimensional OFDI image P2 shown in FIG. 4A .

[0025] A two-dimensional tomographic image constructed from a plurality of ultrasound line data is referred to as one frame of an IVUS image P1. A two-dimensional tomographic image constructed from a plurality of optical line data is referred to as one frame of an OFDI image P2. Since the sensor unit 12 scans while moving within a blood vessel, one frame of an IVUS image P1 or an OFDI image P2 is acquired at each position of one rotation within the range of movement. That is, one frame of an IVUS image P1 or an OFDI image P2 is acquired at each position from the distal end to the proximal end of the probe 11 within the range of movement, so that multiple frames of an IVUS image P1 or an OFDI image P2 are acquired within the range of movement, as shown in FIG. 4B .

[0026] Note that the number of times that ultrasonic waves and light are transmitted and received in one rotation is an example, and the number of times that transmission and reception is not limited to 512. Furthermore, the number of times that ultrasonic waves are transmitted and received and the number of times that light is transmitted and received may be the same or different.

[0027] 5 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer and includes a processing unit 31, a storage unit 32, an ultrasound line data generating unit 33, an optical line data generating unit 34, an input / output I / F 35, and a reading unit 36. The processing unit 31 is configured using one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing unit (GPGPU), a tensor processing unit (TPU), or the like. The processing unit 31 is connected to each hardware unit constituting the image processing device 3 via a bus.

[0028] The memory unit 32 includes, for example, a main memory unit and an auxiliary memory unit. The main memory unit is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data necessary for the processing unit 31 to execute arithmetic processing. The auxiliary memory unit is a storage device such as a hard disk, an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The auxiliary memory unit stores a computer program (program product) P executed by the processing unit 31, an IVUS image recognition learning model 61, an OFDI image recognition learning model 62, and various data necessary for other processing. The memory unit 32 also functions as a tomographic image DB 63. The tomographic image DB 63 stores multiple IVUS images P1 acquired by the image processing device 3, in association with IDs (identification information), imaging dates and times, names of blood vessels captured, names of lesions, and treatment-related information. The tomographic image DB 63 also stores multiple OFDI images P2 acquired by the image processing device 3, each associated with an ID (identification information), the date and time of imaging, the name of the blood vessel where the image was taken, the name of the lesion, and treatment-related information. The treatment-related information is information indicating the type of treatment performed on the blood vessel, such as stent placement, balloon dilation, or expansion of the placed stent, and is information related to the treatment. In other words, the treatment-related information is information that provides clues for identifying a pre-treatment tomographic image as a tomographic image related to a post-treatment tomographic image among multiple IVUS images P1 acquired at different times. The content of the treatment-related information is not particularly limited as long as it can identify a post-treatment tomographic image corresponding to a pre-treatment tomographic image. The auxiliary storage unit may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit during manufacturing of the image processing device 3, or the image processing device 3 may acquire the computer program P via communication from a remote server device and store it in the auxiliary storage unit. The computer program P may be recorded in a readable manner on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, or the reading unit 36 ​​may read it from the recording medium 30 and store it in the auxiliary memory unit.

[0029] The ultrasound line data generating unit 33 acquires reflected wave signals of ultrasound output from the ultrasound transmitting / receiving unit 12a of the intravascular inspection device 101 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data.

[0030] The optical line data generating unit 34 obtains interference light data by causing interference between the reflected light signal output from the optical transmitting / receiving unit 12b of the intravascular inspection device 101 and the reference light obtained by separating the light from the light source, and generates optical line data based on the obtained interference light data.

[0031] The input / output I / F 35 is an interface to which the intravascular inspection device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The processing unit 31 acquires an angiography image from the angiography device 102 via the input / output I / F 35. The processing unit 31 also displays a medical image on the display device 4 by outputting a medical image signal of the IVUS image P1, the OFDI image P2, or the angiography image to the display device 4 via the input / output I / F 35. Furthermore, the processing unit 31 receives information input to the input device 5 via the input / output I / F 35.

[0032] The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer.

[0033] The processing unit 31 of the image processing device 3 reads and executes the computer program P stored in the storage unit 32, thereby generating ultrasound line data in the ultrasound line data generation unit 33 and executing a process of constructing an IVUS image P1 based on the generated ultrasound line data. The processing unit 31 also reads and executes the computer program P stored in the storage unit 32, thereby generating optical line data in the optical line data generation unit 34 and executing a process of constructing an OFDI image P2 based on the generated optical line data.

[0034] The processing unit 31 of the image processing device 3 according to the first embodiment reads and executes the computer program P stored in the storage unit 32 to execute a process of displaying a plurality of IVUS images P1 acquired before a predetermined treatment, such as stent placement in a blood vessel, and a plurality of IVUS images P1 acquired after the treatment, with the longitudinal positions of these images aligned (see FIG. 10 ). The user can check the state of the blood vessel and the effect of treatment by comparing the pre-treatment and post-treatment IVUS images P1, which are displayed with the longitudinal positions aligned.

[0035] <Image Recognition Learning Model> Figures 6A and 6B are block diagrams showing an example configuration of an IVUS image recognition learning model 61. The IVUS image recognition learning model 61 is a model that recognizes a predetermined object image included in an IVUS image P1. The IVUS image recognition learning model 61 can classify objects pixel by pixel by using image recognition technology using semantic segmentation, for example, and can recognize various objects included in the IVUS image P1. Specifically, as shown in Figures 6A and 6B, the IVUS image recognition learning model 61 recognizes an image of the lumen in the IVUS image P1. The IVUS image recognition learning model 61 also recognizes, for example, a stent image, a plaque image, a vascular wall image, etc., as shown in Figure 6A. The vascular wall is the tunica media, more specifically, the external elastic membrane (EEM). The plaque image shown in FIG. 6 is an example of an object image, and the IVUS image recognition learning model 61 may be configured to recognize a side branch, an epicardium, a vein, or the like.

[0036] The IVUS image recognition learning model 61 is, for example, a convolutional neural network (CNN) that has been trained using deep learning. The IVUS image recognition learning model 61 has an input layer 61a to which an IVUS image P1 is input, an intermediate layer 61b that extracts and restores features of the IVUS image P1, and an output layer 61c that outputs an object-extracted IVUS image P1' that represents an object image included in the IVUS image P1 in pixel units. The IVUS image recognition learning model 61 is, for example, a U-Net, a fully convolution network (FCN), a SegNet, or the like.

[0037] The input layer 61a of the IVUS image recognition learning model 61 has a plurality of neurons that accept input of the IVUS image P1, i.e., the pixel values ​​of each pixel constituting the IVUS image P1, and passes the input pixel values ​​to the intermediate layer 61b. The intermediate layer 61b has a plurality of convolution layers (CONV layers) and a plurality of deconvolution layers (DECONV layers). The convolution layer is a layer that compresses the dimensionality of the IVUS image P1. Dimensional compression extracts features of the object image. The deconvolution layer performs deconvolution processing to restore the original dimensions. The restoration processing in the deconvolution layer generates an object-extracted IVUS image P1' in which each pixel has a pixel value (class data) corresponding to the object class. The output layer 61c has a plurality of neurons that output the object-extracted IVUS image P1'. The object-extracted IVUS image P1' is an image that is classified, for example, color-coded, by object type, such as a lumen image, a stent image, a vascular wall image, a plaque image, a side branch, an epicardium, or a vein, as shown in Figures 6A and 6B.

[0038] The IVUS image recognition learning model 61 can be generated by preparing training data including an IVUS image P1 obtained by the ultrasound transmission / reception unit 12a and an object-extracted IVUS image P1' in which each pixel of the IVUS image P1 is annotated with class data corresponding to the type of object, and then using the training data to train an untrained neural network. Specifically, the neural network parameters are optimized so as to minimize the difference between the object-extracted IVUS image P1' output when the IVUS image P1 of the training data is input to the untrained neural network and the annotated image as training data. The parameters include, for example, weights (coupling coefficients) between nodes. While the parameter optimization method is not particularly limited, the processing unit 31 optimizes various parameters using, for example, the steepest descent method.

[0039] According to the thus trained IVUS image recognition learning model 61, as shown in Fig. 6A, by inputting an IVUS image P1 obtained by imaging a blood vessel in which a stent has been placed into the IVUS image recognition learning model 61, an object-extracted IVUS image P1' is obtained in which each part is classified pixel by pixel according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, a plaque image, a stent, etc. Also, as shown in Fig. 6B, by inputting an IVUS image P1 obtained by imaging a blood vessel without a stent into the IVUS image recognition learning model 61, an object-extracted IVUS image P1' is obtained in which each part is classified pixel by pixel according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, a plaque image, etc.

[0040] The OFDI image recognition learning model 62 has the same configuration as the IVUS image recognition learning model 61, and recognizes the lumen image and other object images in the OFDI image P2. By inputting the OFDI image P2 into the OFDI image recognition learning model 62, an object-extracted OFDI image is obtained in which each part is classified on a pixel-by-pixel basis according to the type of object, such as a blood vessel lumen image, a stent image, a blood vessel wall image, or a plaque image.

[0041] 7 and 8 are flowcharts showing the image processing procedure according to the present embodiment 1. The image processing procedure for the IVUS image P1 will be mainly described below.

[0042] The processing unit 31 of the image processing device 3 acquires, via the ultrasound line data generation unit 33, multiple IVUS images P1 before treatment, such as stent placement (step S111). Specifically, the ultrasound line data generation unit 33 of the image processing device 3 generates multiple ultrasound line data based on reflected wave data output from the ultrasound transmission / reception unit 12a. The multiple ultrasound line data are assigned line numbers, for example, in chronological order of observation time. The line numbers correspond to observation times. In other words, the line numbers correspond to observation positions and observation directions. The processing unit 31 constructs multiple frames of IVUS images P1 based on the ultrasound line data. The multiple frames of IVUS images P1 are assigned frame numbers, for example, in chronological order of observation time. The frame numbers correspond to observation positions. The multiple frames of IVUS images P1 correspond to images obtained by observing blood vessels at multiple observation positions ranging from the distal end to the proximal end of the probe 11. The processing unit 31 that executes the process of step S111 functions as an acquisition unit that acquires a plurality of IVUS images P1 of the blood vessels while moving the sensor unit 12 along the running direction of the blood vessels.

[0043] Next, the processing unit 31 stores the plurality of IVUS images P1 acquired before the treatment on the blood vessel in the storage unit 32 in association with the ID, imaging date and time, blood vessel name, lesion name, and treatment-related information (step S112).

[0044] Next, the processing unit 31 creates treatment plan information for stent placement and stores it in the storage unit 32 (step S113).

[0045] FIG. 9 shows an example of a display of a pre-treatment IVUS image P1 and a treatment planning screen. Based on multiple IVUS images P1 acquired before treatment, the processing unit 31 displays, for example, an IVUS image P1, which is a cross-sectional image of the blood vessel substantially perpendicular to the long axis, and a longitudinal image of the blood vessel substantially parallel to the central axis, arranged vertically. In the example shown in FIG. 9 , IVUS images P1 of a distal reference portion (Ref. Distal), a proximal reference portion (Ref. Proximal), and the most stenotic portion (MLA: minimum lumen area) are displayed. The user confirms the most stenotic portion using the longitudinal and transverse cross-sectional images (IVUS images P1) of the blood vessel displayed on the display device 4 and plans the size, placement position, etc. of a stent to be placed. The user specifies the placement position and size of the stent using the input device 5. The image processing device 3 accepts various designation operations via the input device 5, creates treatment planning information including the placement position and size of the stent, and stores the information in the storage unit 32. The white arrows in FIG. 9 indicate treatment plan information, which includes, for example, information indicating frames of the IVUS image P1 corresponding to both ends and the center position in the longitudinal direction of the stent to be placed.

[0046] Next, the processing unit 31 acquires multiple post-treatment IVUS images P1 using the ultrasound line data generator 33 (step S114). The processing unit 31 that executes the process of step S114 functions as an acquisition unit that acquires multiple IVUS images P1 of the blood vessel while moving the sensor unit 12 along the blood vessel's running direction. The processing unit 31 then stores the multiple IVUS images P1 acquired after the blood vessel treatment in the storage unit 32 in association with the ID, imaging date and time, blood vessel name, lesion name, and treatment-related information (step S115).

[0047] Next, the processing unit 31 reads out the pre-treatment IVUS image P1 to be used for comparison (step S116). The user can select the pre-treatment IVUS image P1 to be used for comparison using the input device 5, and the processing unit 31 reads out the selected IVUS image P1 as the pre-treatment IVUS image P1 to be used for comparison.

[0048] The processing unit 31 also reads out treatment plan information associated with the pre-treatment IVUS image P1 from the storage unit 32 (step S117). Based on the read out treatment plan information, the processing unit 31 identifies representative frame images of the pre-treatment IVUS image P1 corresponding to the stent placement position (step S118). Specifically, the processing unit 31 identifies frame images of the IVUS image P1 corresponding to the positions of both ends and the center of the stent to be placed. Here, the frame number of the IVUS image P1 of the stent end on the distal side of the blood vessel is set to n, the frame number of the IVUS image P1 of the stent end on the proximal side of the blood vessel is set to (n+ns), and the frame number of the IVUS image P1 of the stent center portion is set to (n+ns / 2).

[0049] Next, the processing unit 31 executes image recognition processing on the IVUS images P1 and recognizes the stent image by inputting the multiple IVUS images P1 acquired after the treatment into the IVUS image recognition learning model 61 (step S119). By the processing of step S119, the processing unit 31 can recognize the multiple IVUS images P1 after the treatment by distinguishing between frame images including stent images and frame images not including stent images.

[0050] Based on the recognition result of step S119, the processing unit 31 identifies representative frames of the post-treatment IVUS images P1 corresponding to the stent placement position (step S120). Specifically, the processing unit 31 identifies frame images of the IVUS images P1 corresponding to the positions of both ends and the center of the stent to be placed. The frame images containing the stent image are consecutive. For example, if a stent image is included in frame images from frame number N to frame number N+Ns, the processing unit 31 identifies the IVUS image P1 with frame number N, the IVUS image P1 with frame number (N+Ns), and the IVUS image P1 with frame number (N+Ns / 2) as representative frame images. The IVUS image P1 with frame number N is a frame image of the stent end on the distal side of the blood vessel, and the IVUS image P1 with frame number (N+Ns) is a frame image of the stent end on the proximal side of the blood vessel.

[0051] The processing unit 31 associates the pre- and post-treatment IVUS images P1 with each other based on the identified pre-treatment representative frame image and post-treatment representative frame image (step S121). Specifically, the IVUS image P1 with frame number n, which corresponds to one end of the pre-treatment stent, is associated with the post-treatment IVUS image P1 with frame number N. Similarly, the IVUS image P1 with frame number ns, which corresponds to the other end of the pre-treatment stent, is associated with the post-treatment IVUS image P1 with frame number Ns. Furthermore, the IVUS image P1 with frame number (n+ns / 2), which corresponds to the center of the pre-treatment stent, is associated with the post-treatment IVUS image P1 with frame number (N+Ns / 2).

[0052] In addition, if the longitudinal scale of the IVUS image P1 acquired before the treatment does not completely match the longitudinal scale of the IVUS image P1 acquired after the treatment, it is advisable to interpolate or thin out the IVUS image P1 before or after the treatment as necessary to match the representative frame image before the treatment with the representative frame image after the treatment.

[0053] Next, the processing unit 31 adjusts the longitudinal position and displays the pre-treatment and post-treatment IVUS longitudinal section images (step S122).

[0054] FIG. 10 shows a display example of IVUS images P1 before and after treatment. The processing unit 31 displays a longitudinal cross-sectional image of the blood vessel before treatment and a longitudinal cross-sectional image of the blood vessel after treatment side by side. Specifically, the processing unit 31 generates a first longitudinal cross-sectional image substantially parallel to the central axis of the blood vessel based on multiple IVUS images P1 acquired before treatment and displays it on the display device 4. Similarly, the processing unit 31 generates a second longitudinal cross-sectional image substantially parallel to the central axis of the blood vessel based on multiple IVUS images P1 acquired after treatment and displays it on the display device 4. In the example shown in FIG. 10, the processing unit 31 displays the first longitudinal cross-sectional image before treatment on the upper side and the second longitudinal cross-sectional image after treatment on the lower side. The display of the distal reference portion (Ref. Distal), proximal reference portion (Ref. Proximal), and most stenotic portion (MLA: minimum lumen area) is the same as in FIG. 9.

[0055] Next, as shown in FIG. 10 , the processing unit 31 sets and displays a cross-section designation bar B in the peripheral frame containing the pre- and post-treatment IVUS images P1 (step S123). The cross-section designation bar B is a line image that designates an arbitrary IVUS image P1 in the pre-treatment IVUS longitudinal images and an IVUS image P1 corresponding to the arbitrary IVUS image P1 in the post-treatment IVUS longitudinal images. More specifically, the cross-section designation bar B is a thick line image that vertically passes through the pre-treatment IVUS longitudinal image and the post-treatment IVUS longitudinal image. By the processing of step S123, the processing unit 31 displays the cross-section designation bar B in the peripheral frame (the leftmost frame image in FIG. 10 ) containing both the pre- and post-treatment IVUS images P1.

[0056] 10 by operating the input device 5, the processing unit 31 determines whether or not a change in the position of the cross-section designation bar B has been accepted (step S124). If it is determined that a change in the position has been accepted (step S124: YES), the processing unit 31 changes the position of the cross-section designation bar B and displays it (step S125).

[0057] When the processing of step S125 is completed or when it is determined that a change to the cross-section designation bar B has not been accepted (step S124: NO), the processing unit 31 displays the pre-treatment and post-treatment IVUS images P1 (cross-sectional images) corresponding to the position of the cross-section designation bar B (step S126).

[0058] The pre- and post-treatment IVUS images P1 displayed by the above process correspond in longitudinal position but do not necessarily have the same circumferential orientation. The user can rotate the IVUS image P1 by operating the input device 5, and the processing unit 31 accepts a circumferential position correction for the IVUS image P1 (step S127). The processing unit 31 then rotates the IVUS image P1 and corrects its circumferential orientation based on the accepted position correction content, i.e., the angle information for rotating the IVUS image P1 (step S128).

[0059] The processing unit 31 displays an automatic playback button, and the user can operate the automatic playback button by operating the input device 5. The processing unit 31 determines whether an automatic playback instruction has been received (step S129). If it determines that an automatic playback instruction has not been received (step S129: NO), the processing unit 31 returns the process to step S123.

[0060] If it is determined that an instruction for automatic playback has been received (step S129: YES), the processing unit 31 executes automatic playback display, sequentially displaying each of the multiple IVUS images P1 before and after treatment, from the IVUS image P1 at the tip end to the IVUS image P1 at the base end (step S130), and returns the processing to step S123.

[0061] As described above, according to the image processing device 3 etc. of this embodiment 1, when displaying IVUS images P1 of a blood vessel obtained before and after treatment on the blood vessel, the longitudinal position of the blood vessel can be aligned and the IVUS images P1 before and after treatment can be displayed in correspondence with each other.

[0062] Specifically, based on the treatment plan information, IVUS images P1 of the ends and center of the stent to be placed are matched with IVUS images P1 of the ends and center of the stent identified by image recognition of the IVUS image P1 after treatment, thereby enabling multiple IVUS images P1 obtained before and after treatment to be matched.

[0063] After aligning the observation positions in the longitudinal direction of the pre- and post-treatment IVUS images P1, it is possible to compare the pre- and post-treatment IVUS images P1 and extract the difference. For example, if a calcified lesion is removed, the volume of the removed area can be calculated three-dimensionally using the comparison image of the IVUS image P1.

[0064] In this embodiment 1, an example has been described in which the longitudinal positions of the blood vessels are aligned to display the IVUS image P1 before and after the treatment. However, the OFDI image P2 may also be configured to be displayed by similar processing, with the longitudinal positions of the blood vessels aligned to display the OFDI image P2 before and after the treatment.

[0065] Furthermore, in this embodiment 1, an IVUS image recognition learning model 61 capable of recognizing stent images by classifying the IVUS image P1 on a pixel-by-pixel basis using semantic segmentation has been described, but the image processing device 3 may also be configured using a learning model that determines the presence or absence of a stent on a frame image-by-frame basis.

[0066] (Embodiment 2) The imaging diagnostic device 100 according to embodiment 2 differs from embodiment 1 in that it aligns the longitudinal position of the blood vessel based on the feature amount of the outside of the balloon expansion site and displays the pre-treatment and post-treatment IVUS images P1 in association with each other. The other configuration of the image processing device 3 is the same as that of the image processing device 3 according to embodiment 1, so the same reference numerals are used for the same parts and detailed description will be omitted.

[0067] 11 is a flowchart showing an image processing procedure according to the second embodiment, and Figures 12A, 2B, and 12C are explanatory diagrams showing an image processing method according to the second embodiment. As in the first embodiment, the processing unit 31 of the image processing device 3 acquires a plurality of IVUS images P1 (step S211) and stores them in the storage unit 32 (step S212).

[0068] The user checks the most stenotic area using the longitudinal and transverse cross-sectional images (IVUS image P1) of the blood vessel displayed on the display device 4, and plans the location of balloon dilation, etc. The user specifies the target area for balloon dilation using the input device 5, and the image processing device 3 accepts various specification operations via the input device 5, and creates treatment plan information including the target area for balloon dilation (planned dilation range), etc., as shown in Fig. 12A, and stores it in the storage unit 32 (step S213).

[0069] Next, as in the first embodiment, the processing unit 31 acquires multiple post-treatment IVUS images P1 using the ultrasound line data generator 33 (step S214) and stores them in the storage unit 32 (step S215). The processing unit 31 then reads out the pre-treatment IVUS image P1 to be used for comparison (step S216). The processing unit 31 also reads out treatment plan information associated with the pre-treatment IVUS image P1 from the storage unit 32 (step S217).

[0070] The processing unit 31 inputs multiple IVUS images P1 acquired before and after the treatment into the IVUS image recognition learning model 61, thereby performing image recognition processing on the IVUS images P1 and recognizing the lumen image or the external elastic membrane (step S218).

[0071] Next, the processing unit 31 calculates the lumen diameter or blood vessel diameter outside the target region before the treatment where balloon inflation is to be performed based on the image recognition result of step S218 and the treatment plan information (step S219). Specifically, as shown in Fig. 12B, the processing unit 31 can calculate the lumen diameter based on the lumen image obtained by image recognition of the IVUS image P1 outside the target region. The processing unit 31 can also calculate the blood vessel diameter based on the external elastic membrane image obtained by image recognition of the IVUS image P1 outside the target region.

[0072] Meanwhile, the processing unit 31 calculates the lumen diameter or vascular diameter outside the target region after the balloon dilation procedure (step S220). Because the longitudinal positions of the IVUS image P1 acquired before the procedure and the IVUS image P1 acquired after the procedure roughly correspond, the processing unit 31 can identify the IVUS image P1 outside the target region after the balloon dilation procedure based on the treatment plan information, as shown in FIG. 12C , and calculate the lumen diameter based on the lumen image obtained by image recognition of the IVUS image P1. The processing unit 31 can also calculate the vascular diameter. The processing unit 31 may also be configured to calculate the lumen diameter or vascular diameter for each of the multiple post-treatment IVUS images P1.

[0073] Next, the processing unit 31 matches the pre- and post-treatment IVUS images P1 so that the lumen diameter or vascular diameter outside the dilation range matches (step S221). Because the lumen diameter or vascular diameter does not always match completely before and after treatment due to the influence of pulsation, etc., the processing unit 31 performs processing to match the pre- and post-treatment IVUS images P1 so that the lumen diameter or vascular diameter outside the dilation range approximately matches (substantially matches). For example, the processing unit 31 calculates the difference between the lumen diameter or vascular diameter based on the pre-treatment IVUS image P1 and the corresponding post-treatment IVUS image P1, and identifies the correspondence between the pre-treatment IVUS image P1 and the post-treatment IVUS image P1 such that the sum of the differences based on the multiple IVUS images P1 is minimized. That is, the processing unit 31 associates the two IVUS images P1 so that the observation position of the IVUS image P1 acquired before the treatment matches the observation position of the IVUS image P1 acquired after the treatment. By associating the IVUS image P1 before the treatment with the IVUS image P1 after the treatment so that the sum of the differences is minimized, it becomes possible to associate the IVUS images P1 so that the lumen diameter or blood vessel diameter outside the dilation range substantially matches.

[0074] Details of the processing of step S211 are as follows. Before the processing of step S221, the IVUS image P1 of the nth frame and the IVUS image P1 of the nth frame are assumed to be associated with each other. n is an integer greater than or equal to 1. The processing unit 31 calculates the difference in lumen diameter or blood vessel diameter based on, for example, the object-extracted IVUS image P1' of the corresponding frame number. The processing unit 31 calculates the difference in lumen diameter or blood vessel diameter for a predetermined number of object-extracted IVUS images P1'. Next, the processing unit 31 similarly calculates the difference between the lumen diameter or blood vessel diameter based on the object-extracted IVUS image P1' of the (n+α)th frame and the lumen diameter or blood vessel diameter based on the object-extracted IVUS image P1' of the nth frame. α is an integer and corresponds to the amount of translation of the three-dimensional lumen image. The processing unit 31 verifies the correspondence between the object-extracted IVUS image P1' before and after treatment while incrementing the variable α by 1. Then, a variable α that minimizes the sum of the differences, for example, the sum of the squares of the differences, is identified.

[0075] After completing the process of step S221, the processing unit 31 executes the same processes as steps S122 to S130 in the first embodiment.

[0076] As described above, according to the image processing device 3 etc. of this embodiment 2, by matching multiple IVUS images P1 acquired before and after the treatment so that the lumen diameter or blood vessel diameter outside the target area before the treatment where balloon expansion is to be performed most closely matches the lumen diameter or blood vessel diameter of the blood vessel after the treatment, it is possible to align the longitudinal positions of the blood vessels and display the IVUS images P1 before and after the treatment in a corresponding manner.

[0077] (Embodiment 3) The imaging diagnostic device 100 according to embodiment 3 differs from embodiment 1 in that, when stent expansion (optimization) is performed, the image diagnostic device 100 aligns the longitudinal position of the blood vessel based on the pre-treatment and post-treatment stent images and displays the pre-treatment and post-treatment IVUS images P1 in association with each other. Since the other configuration of the image processing device 3 is the same as that of the image processing device 3 according to embodiment 1, the same reference numerals are used for the same parts and detailed description will be omitted.

[0078] 13 is a flowchart showing an image processing procedure according to the present embodiment 3. The processing unit 31 of the image processing device 3 executes the same processes as steps S111 to S117 in the first embodiment (steps S311 to S317).

[0079] After completing the processing of step S317, the processing unit 31 inputs multiple IVUS images P1 acquired before and after the treatment into the IVUS image recognition learning model 61, thereby performing image recognition processing of the IVUS image P1 and recognizing the stent image (step S318).

[0080] Next, the processing unit 31 identifies representative frames relating to the stent placement positions before expansion (step S319). Specifically, the processing unit 31 identifies frame images of the IVUS images P1 corresponding to the positions of both ends and the center of the stent from the multiple IVUS images P1 acquired before the treatment.

[0081] The processing unit 31 also identifies representative frames related to the positions of the stent after expansion (step S320). Specifically, the processing unit 31 identifies frame images of the IVUS images P1 corresponding to the positions of both ends and the center of the stent from the multiple IVUS images P1 acquired after the treatment.

[0082] Next, the processing unit 31 associates the multiple IVUS images P1 before the treatment (before the optimization of the stent placement state) and after the treatment (after the optimization of the stent placement state) based on the identified representative frame image before the treatment and the representative frame image after the treatment (step S321). Specifically, the processing unit 31 associates the IVUS image P1 corresponding to one end of the stent before the treatment with the IVUS image P1. Similarly, the processing unit 31 associates the IVUS image P1 corresponding to the other end of the stent before the treatment with the IVUS image P1 after the treatment. Furthermore, the processing unit 31 associates the IVUS image P1 corresponding to the central portion of the stent before the treatment with the IVUS image P1 after the treatment.

[0083] After completing the process of step S321, the processing unit 31 executes the same processes as steps S122 to S130 in the first embodiment.

[0084] As described above, according to the image processing device 3 etc. of this embodiment 3, when a stent is expanded, by aligning the position of the stent image, it is possible to associate multiple IVUS images P1 acquired before and after the treatment, and by aligning the longitudinal position of the blood vessel, it is possible to display the IVUS images P1 before and after the treatment in a corresponding manner.

[0085] (Embodiment 4) The image diagnostic device 100 according to embodiment 4 differs from embodiment 1 in that it aligns the longitudinal positions of blood vessels based on an image of a calcified site in the IVUS image P1 and displays pre-treatment and post-treatment IVUS images P1 in association with each other. Since the other configuration of the image processing device 3 is the same as that of the image processing device 3 according to embodiment 1, the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0086] 14 is a flowchart showing an image processing procedure according to the fourth embodiment. The processing unit 31 of the image processing device 3 executes the same processes as steps S111 to S121 of the first embodiment. The processing of step S121 allows the positions of the pre-treatment IVUS image P1 and the post-treatment IVUS image P1 in the longitudinal direction of the blood vessel to roughly coincide with each other. The processing unit 31 according to the fourth embodiment executes the following process to finely correct the correspondence between the pre-treatment and post-treatment IVUS images P1.

[0087] After completing step S121, the processing unit 31 inputs the multiple IVUS images P1 acquired before and after the treatment into the IVUS image recognition learning model 61 to perform image recognition processing on the IVUS images P1 and recognize the calcification site image (step S421). The processing unit 31 identifies a frame image of the IVUS image P1 that includes the calcification site image. The processing unit 31 also identifies the central angle of the calcification site image in the IVUS image P1.

[0088] FIG. 15 is an explanatory diagram showing multiple pre- and post-treatment IVUS images P1 arranged along the longitudinal direction of a blood vessel. The upper diagram shows multiple pre-treatment IVUS images P1 arranged along the longitudinal direction of a blood vessel. The white areas indicate IVUS images P1 that do not contain calcification images, while the hatched areas indicate IVUS images P1 that include calcification images. "20°," "70°," "120°," and the like indicate the central angle of the calcification in the IVUS image P1 that includes the calcification image, i.e., the circumferential position of the calcification in the IVUS image P1. The lower diagram shows multiple post-treatment IVUS images P1 arranged along the longitudinal direction of the blood vessel. Comparing the pre- and post-treatment IVUS images P1 reveals that the observation position and observation direction along the longitudinal direction of the blood vessel are different between the two.

[0089] Next, the processing unit 31 associates the pre- and post-treatment IVUS images P1 so that the longitudinal positions of the multiple calcification sites coincide (step S422). For example, if multiple consecutive IVUS images P1 contain images of calcification sites, the processing unit 31 recognizes the multiple IVUS images P1 as a group and identifies the IVUS image P1 at the longitudinal center of the blood vessel as a representative frame image. The processing unit 31 treats the pre-treatment representative frame image and the post-treatment representative frame image that are closest in longitudinal distance as a pair, and calculates the distance between the representative frame images. If there are multiple pairs of representative frame images, the processing unit 31 calculates the distance between each pair and fine-tunes the longitudinal positions of the IVUS images P1 so that the sum of the distances is minimized, thereby associating the pre-treatment IVUS image P1 with the post-treatment IVUS image P1.

[0090] Next, the processing unit 31 corrects the rotation angle of the IVUS image P1 so that the central angles of the calcification site images in the IVUS image P1 are aligned (step S423). For example, the processing unit 31 calculates the difference between the central angle of the calcification site image included in the representative frame image before processing and the central angle of the calcification site image included in the corresponding representative frame image after treatment. The processing unit 31 then similarly calculates the central angle differences for the other representative frame images, calculates the amount of correction for the rotation angle so that the sum of the differences is minimized, and corrects the orientation of the IVUS image P1 by rotating the IVUS image P1 by the calculated correction amount. In the example shown in FIG. 15 , the orientation of the pre-treatment IVUS image P1 and the post-treatment IVUS image P1 can be aligned by subtracting 30 degrees from the circumferential angle of the post-treatment IVUS image P1, i.e., by rotating it 30 degrees clockwise.

[0091] The processing unit 31 displays the representative frame image before treatment and the representative frame image after treatment, whose longitudinal position has been finely adjusted and whose central angle has been corrected in the processes of steps S422 and S423, in association with each other (step S424).

[0092] As described above, the image processing device 3 and the like according to the fourth embodiment can more accurately align the longitudinal positions of blood vessels and display the IVUS images P1 before and after treatment in a corresponding manner compared to the first embodiment. Furthermore, the rotation angles, i.e., the orientations, of the IVUS images P1 before and after treatment can be aligned and displayed.

[0093] In this fourth embodiment, an example has been described in which the pre- and post-treatment IVUS images P1 are associated with each other based on the stent placement position, and the correspondence and central angle of the IVUS image P1 are corrected based on the position and central angle of the calcification site image. However, the method of this fourth embodiment alone may be used to associate and display the pre- and post-treatment IVUS images P1.

[0094] Furthermore, in the fourth embodiment, an example has been described in which pre-treatment and post-treatment IVUS images P1 are associated with each other using an image of a calcified site so that the observation position and observation direction in the longitudinal direction of the blood vessel coincide. However, similar processing may be performed based on other object images, such as side branches, attenuated plaque, epicardium, or veins.

[0095] (Embodiment 5) An image diagnostic device 100 according to embodiment 5 differs from embodiment 1 in that it automatically executes the process of step S116 according to embodiment 1. Other configurations of the image processing device 3 are similar to those of the image processing device 3 according to embodiment 1, and therefore similar parts are denoted by the same reference numerals and detailed description thereof will be omitted.

[0096] 16 is a flowchart showing the image processing procedure according to the fifth embodiment. In step S116, the processing unit 31 executes the following process. The processing unit 31 reads the imaging date and time, blood vessel name, lesion name, treatment-related information, etc., associated with the post-treatment IVUS image P1 from the storage unit 32 (step S551). Next, the processing unit 31 searches for an IVUS image P1 having an earlier imaging date and time, matching blood vessel name, lesion name, etc., and consistent treatment details (step S552), and reads a pre-treatment IVUS image P1 that matches the search criteria from the tomographic image DB 63 (step S553).

[0097] As described above, according to the image processing device 3 etc. of this embodiment 5, when an IVUS image P1 is acquired after a treatment, the data of the IVUS image P1 before the treatment, which is used for comparison, is automatically read out from the storage unit 32, and the longitudinal cross-sectional image P1 of the blood vessel and the IVUS image P1 based on each IVUS image P1 can be displayed side by side on the display device 4.

[0098] REFERENCE SIGNS LIST 1 Diagnostic imaging catheter 2 MDU 3 Image processing device 4 Display device 5 Input device 11 Probe 12 Sensor unit 12a Ultrasound transmitting and receiving unit 12b Light transmitting and receiving unit 12c Housing 13 Shaft 14 Guide wire insertion unit 15 Connector unit 30 Recording medium 31 Processing unit 32 Storage unit 33 Ultrasound line data generating unit 34 Light line data generating unit 61 IVUS image recognition learning model 62 OFDI image recognition learning model P1 IVUS image P1' Object-extracted IVUS image P2 OFDI image 100 Diagnostic imaging device 101 Intravascular examination device 102 Angiography device

Claims

1. a sensor unit for scanning a tubular organ is moved along a running direction of the tubular organ to obtain a plurality of tomographic images of the tubular organ; selecting at least one tomographic image related to a target site where the treatment of the hollow organ is to be performed from among a plurality of tomographic images acquired before the treatment of the hollow organ; Identifying a post-treatment tomographic image corresponding to the one pre-treatment tomographic image selected by image recognition processing of the plurality of tomographic images acquired after the treatment; Based on the result of identifying the tomographic images, the tomographic images acquired before the treatment and the tomographic images acquired after the treatment are displayed in association with each other. A computer program for causing a computer to execute a process.

2. obtaining treatment plan information indicating a tomographic image of a target site where a treatment is to be performed on the hollow organ; At least one tomographic image related to a target site where a treatment for the hollow organ is to be performed is selected based on the acquired treatment plan information.

2. The computer program product of claim 1.

3. The treatment plan information includes information indicating a tomographic image of an end or a center of a target site where a stent is to be placed, among a plurality of tomographic images acquired before the treatment, By performing image recognition processing on the multiple tomographic images acquired after the procedure, a tomographic image including an image of a stent is recognized; A post-treatment cross-sectional image of an end or a central portion of the placed stent is identified as a cross-sectional image corresponding to the pre-treatment cross-sectional image of the end or the central portion.

3. A computer program according to claim 2.

4. The treatment plan information includes information indicating a tomographic image of a target site where balloon expansion is to be performed, among a plurality of tomographic images acquired before the treatment; By comparing a feature amount obtained by image recognition processing of a tomographic image of the outside of the target site where balloon expansion is to be performed with the feature amount obtained by image recognition processing of a plurality of tomographic images acquired after the treatment, a post-treatment tomographic image corresponding to the pre-treatment tomographic image of the outside of the target site where balloon expansion is to be performed is identified. A computer program according to claim 2 or 3.

5. When a treatment related to optimizing the placement state of a stent placed in the tubular organ is performed, a tomographic image including an image of the stent is recognized by image recognition processing of a plurality of tomographic images acquired before and after the treatment; Identifying a post-treatment cross-sectional image of an end or central portion of the optimized stent as a cross-sectional image that corresponds to a pre-treatment cross-sectional image of said end or central portion of the deployed stent before optimization. A computer program according to any one of claims 1 to 3.

6. Recognizing object images contained in a plurality of tomographic images acquired before and after the treatment; The tomographic image acquired before the treatment is associated with the tomographic image acquired after the treatment such that the positions of the object images included in the tomographic images before and after the treatment correspond to each other. A computer program according to any one of claims 1 to 3.

7. Recognizing object images contained in a plurality of tomographic images acquired before and after the treatment; A correspondence relationship between a tomographic image acquired before the treatment and a tomographic image acquired after the treatment is specified based on the treatment plan information, and then the correspondence relationship is corrected so that positions of object images included in the tomographic images before and after the treatment correspond to each other. A computer program according to claim 2 or 3.

8. When an object image is included in the pre-treatment and post-treatment tomographic images corresponding to a plurality of sites separated in the traveling direction, the pre-treatment tomographic images of the plurality of sites are associated with the post-treatment tomographic images of the plurality of sites to identify a correspondence relationship between the pre-treatment tomographic images and the post-treatment tomographic images; A tomographic image acquired before the treatment and a tomographic image acquired after the treatment are displayed in association with each other so that the positions of the hollow organs in the running direction are aligned. A computer program according to any one of claims 1 to 3.

9. When an object image is included in the pre-treatment and post-treatment tomographic images, the object image included in the pre-treatment tomographic image is associated with the object image included in the post-treatment tomographic image to determine a correspondence relationship between a circumferential angle of the tubular organ in the multiple tomographic images before the treatment and a circumferential angle of the tubular organ in the multiple tomographic images after the treatment; A tomographic image acquired before the treatment and a tomographic image acquired after the treatment are displayed in association with each other so that the position in the running direction and the angle in the circumferential direction of the hollow organ are aligned. A computer program according to any one of claims 1 to 3.

10. The object image includes an image of a plaque, a side branch, an epicardium, or a vein.

7. A computer program according to claim 6.

11. when a plurality of tomographic images of the hollow organ are acquired, the acquired plurality of tomographic images, identification information for identifying the hollow organ related to the tomographic images, and treatment-related information related to the treatment for the hollow organ are stored in a storage unit in association with each other; Based on the identification information and the treatment-related information, a plurality of tomographic images acquired before the treatment are read out from the storage unit, the plurality of tomographic images being to be displayed in association with a plurality of tomographic images acquired after the treatment. A computer program according to any one of claims 1 to 3.

12. acquiring a plurality of tomographic images of a hollow organ by moving a sensor unit for scanning the hollow organ along a running direction of the hollow organ; selecting at least one tomographic image related to a target site where the treatment of the hollow organ is to be performed from among a plurality of tomographic images acquired before the treatment of the hollow organ; Identifying a post-treatment tomographic image corresponding to the one pre-treatment tomographic image selected by image recognition processing of the plurality of tomographic images acquired after the treatment; Based on the result of identifying the tomographic images, the tomographic images acquired before the treatment and the tomographic images acquired after the treatment are displayed in association with each other. Image processing methods.

13. an acquisition unit that acquires a plurality of tomographic images of a hollow organ while moving a sensor unit for scanning the hollow organ along a running direction of the hollow organ; a processing unit that executes a process of displaying the tomographic image acquired by the acquisition unit on a display device; Equipped with The processing unit includes: selecting at least one tomographic image related to a target site where the treatment of the hollow organ is to be performed from among a plurality of tomographic images acquired before the treatment of the hollow organ; Identifying a post-treatment tomographic image corresponding to the one pre-treatment tomographic image selected by image recognition processing of the plurality of tomographic images acquired after the treatment; Based on the result of identifying the tomographic images, the tomographic images acquired before the treatment and the tomographic images acquired after the treatment are displayed in association with each other. Image processing device.