Computer program, operation method of image processing device, and image processing device

Abstract

To provide an image processing method capable of matching the scales of an ultrasound tomogram and an optical coherence tomogram obtained by a dual-type catheter having IVUS and OCT functions.SOLUTION: An image processing method acquires an ultrasound tomogram and an optical coherence tomogram at a sensor unit having an ultrasound transmitter / receiver that transmits and receives ultrasound and an optical transmitter / receiver that transmits and receives light, facing radially outward from the lumen of a hollow organ, recognizes a first lumen image of the hollow organ contained in the ultrasound tomogram, recognizes a second lumen image of the hollow organ contained in the optical coherence tomogram, and corrects the scale in the radial direction of the ultrasound tomogram or optical coherence tomogram so that the dimensions of the first and second lumen images are adjusted on the basis of the recognized first and second lumen images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a computer program, an image processing method, and an image processing device. [Background technology]

[0002] Conventionally, imaging diagnostic devices have been widely used for the diagnosis of arteriosclerosis, preoperative diagnosis during intravascular treatment using high-performance catheters such as balloon catheters or stents, or postoperative result confirmation. Imaging diagnostic devices include intravascular ultrasound imaging devices (IVUS: IntraVascular Ultrasound), optical coherence tomography imaging devices (OCT: Optical Coherence Tomography), etc., each of which has different characteristics. In addition, an imaging diagnostic device that combines the functions of IVUS and OCT, that is, an imaging diagnostic device that has an ultrasound transceiver unit capable of transmitting and receiving ultrasound, and an optical transceiver unit capable of transmitting and receiving light, has been proposed (for example, Patent Documents 1 and 2). With this type of imaging diagnostic device, it is possible to construct both ultrasound tomographic images, taking advantage of the IVUS's ability to measure deep areas, and optical coherence tomographic images, taking advantage of the OCT's ability to measure at high resolution, in a single scan. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-56752 [Patent Document 2] Special Publication No. 2010-508973 [Patent Document 3] Re-tabled publication 2014 / 162367 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the scales of the constructed ultrasound tomographic images and optical coherence tomographic images do not necessarily match, which is one of the factors that makes it difficult to interpret the tomographic images.

[0005] It is also possible to store the characteristics of the device and match the scales of the ultrasonic tomographic image and the optical coherence tomographic image. However, in order to transmit and receive light by the optical transmitting and receiving unit, it is necessary to replace the blood in the blood vessels using a flushing liquid, and the ultrasonic waves output from the ultrasonic transmitting and receiving unit propagate through the flushing liquid rather than through the blood. Since the propagation speed of the ultrasonic waves varies depending on the type of the flushing liquid, the scales of the ultrasonic tomographic image and the optical coherence tomographic image do not match. The type and concentration of the flushing liquid are not uniquely determined, and may be changed depending on the condition of the patient. For this reason, there is a limit to the correction process of the propagation speed of the ultrasonic waves according to the type of the flushing liquid, and it is not possible to match the scales of the ultrasonic tomographic image and the optical coherence tomographic image. Furthermore, if the propagation speed of the ultrasonic waves is set incorrectly, ultrasonic tomographic images and optical coherence tomographic images with inconsistent scales will be constructed.

[0006] In one aspect, an object of the present invention is to provide a computer program, an image processing method, and an image processing device that can align the scale of ultrasound tomographic images and optical coherence tomographic images obtained using a dual-type catheter equipped with IVUS and OCT functions. [Means for solving the problem]

[0007] In one aspect, the computer program causes a computer to perform a process of acquiring ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves radially outward from the inner cavity of a tubular organ and an optical transceiver unit that transmits and receives light, recognizing a first inner lumen image of the tubular organ contained in the acquired ultrasonic tomographic images, recognizing a second inner lumen image of the tubular organ contained in the acquired optical coherence tomographic images, and correcting the radial scale of the ultrasonic tomographic images or the optical coherence tomographic images based on the recognized first inner lumen image and second inner lumen image so that dimensions of the first inner lumen image and the second inner lumen image are consistent.

[0008] In one aspect, the image processing method acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves radially outward from the lumen of a tubular organ and an optical transceiver unit that transmits and receives light, recognizes a first lumen image of the tubular organ included in the acquired ultrasonic tomographic images, recognizes a second lumen image of the tubular organ included in the acquired optical coherence tomographic images, and corrects the radial scale of the ultrasonic tomographic images or the optical coherence tomographic images based on the recognized first lumen image and second lumen image so that dimensions of the first lumen image and the second lumen image are consistent.

[0009] In one aspect, the image processing device includes an acquisition unit that acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves radially outward from the lumen of a tubular organ and an optical transceiver unit that transmits and receives light, a first recognition unit that recognizes a first lumen image of the tubular organ included in the ultrasonic tomographic image acquired by the acquisition unit, a second recognition unit that recognizes a second lumen image of the tubular organ included in the optical coherence tomographic image acquired by the acquisition unit, and a correction unit that corrects the radial scale of the ultrasonic tomographic image or the optical coherence tomographic image based on the first lumen image and the second lumen image recognized by the first recognition unit and the second recognition unit so that dimensions of the first lumen image and the second lumen image are consistent. Effect of the Invention

[0010] In one aspect, the scales of ultrasound images and optical coherence tomography images obtained using a dual type catheter with IVUS and OCT functions can be matched. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of an image diagnostic apparatus. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the configuration of a catheter for diagnostic imaging. [Diagram 3] 1 is an explanatory diagram illustrating a cross section of a blood vessel through which a sensor portion is inserted; FIG. [Figure 4] FIG. [Diagram 5] FIG. 1 is a block diagram showing an example of the configuration of an image processing device. [Figure 6] A block diagram showing an example configuration of a first learning model. [Figure 7] A block diagram showing an example configuration of a second learning model. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of an image quality improvement learning model. [Figure 9] 1 is a flowchart showing a method for generating an image quality improvement learning model. [Figure 10] 4 is a flowchart showing an image processing procedure according to the first embodiment. [Figure 11] FIG. 2 is an explanatory diagram showing a three-dimensional lumen image constructed based on an object extraction IVUS image and an object extraction OCT image. [Figure 12] FIG. 13 is an explanatory diagram conceptually showing a method for matching three-dimensional lumen images by rigid registration and correcting the observation position, observation direction, and scale. [Figure 13] 1 is an explanatory diagram conceptually showing a correspondence relationship between an IVUS image and an OCT image and a method of correcting the scale. [Figure 14] 13 is an explanatory diagram conceptually showing a scale correction method for an inner cavity portion. FIG. [Figure 15]13 is an explanatory diagram conceptually showing a method of scale correction for the portion outside the lumen. FIG. [Figure 16] 10 is a flowchart showing an image processing procedure according to the second embodiment. [Figure 17] 10 is an explanatory diagram showing a method of correcting the correspondence relationship between ultrasonic line data and optical line data; FIG. [Figure 18] FIG. 11 is an explanatory diagram showing a method of correcting the scale of ultrasonic line data and optical line data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the computer program, image quality improvement learning model, learning model generation method, image processing method, and image processing device of the present disclosure will be described in detail with reference to the drawings showing the embodiments. In each of the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example, but the tubular organs to be treated by catheterization are not limited to blood vessels, and may be other tubular organs such as bile ducts, pancreatic ducts, bronchi, and intestines. The present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. At least some of the embodiments described below may be arbitrarily combined.

[0013] In this embodiment, an imaging diagnostic device using a dual-type catheter having both functions of intravascular ultrasound (IVUS) and optical coherence tomography (OCT) will be described. The dual-type catheter is provided with a mode for acquiring an ultrasonic tomographic image only by IVUS, a mode for acquiring an optical coherence tomographic image only by OCT, and a mode for acquiring both tomographic images by IVUS and OCT, and these modes can be switched for use. Hereinafter, the ultrasonic tomographic image and the optical coherence tomographic image will be appropriately referred to as an IVUS image and an OCT image, respectively. Also, the IVUS image and the OCT image will be collectively referred to as a tomographic image.

[0014] (Embodiment 1) FIG. 1 is an explanatory diagram showing a configuration example of an image diagnostic apparatus 100. The image diagnostic apparatus 100 of this embodiment includes an intravascular inspection apparatus 101, an angiography apparatus 102, an image processing device 3, a display device 4, and an input device 5. The intravascular inspection apparatus 101 includes an image diagnostic catheter 1 and an MDU (Motor Drive Unit) 2. The image diagnostic catheter 1 is connected to an image processing device 3 via the MDU 2. The image processing device 3 is connected to a display device 4 and an input device 5. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone. The display device 4 and the input device 5 may be integrally stacked to form a touch panel. The input device 5 and the image processing device 3 may also be integrally configured. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input.

[0015] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device for imaging blood vessels from outside the patient's body using X-rays while injecting a contrast agent into the patient's blood vessels, and obtaining an angiogram, which is a fluoroscopic image of the blood vessels. The angiography device 102 includes an X-ray source and an X-ray sensor, and images an X-ray fluoroscopic image of the patient by the X-ray sensor receiving X-rays irradiated from the X-ray source. The angiography device 102 outputs the angiogram obtained by imaging to the image processing device 3, and the image is displayed on the display device 4 via the image processing device 3. The display device 4 displays the angiogram and a tomographic image of the blood vessel imaged using the diagnostic imaging catheter 1.

[0016] FIG. 2 is an explanatory diagram showing a configuration example of the diagnostic imaging catheter 1. The area surrounded by the upper dashed line in FIG. 2 is an enlarged view of the area surrounded by the lower dashed line. The diagnostic imaging catheter 1 has a probe 11 and a connector section 15 disposed at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector section 15. In the following description, the side of the diagnostic imaging catheter 1 far from the connector section 15 is described as the tip side, and the connector section 15 side is referred to as the base side. The probe 11 includes a catheter sheath 11a, and a guidewire insertion section 14 through which a guidewire can be inserted is provided at the tip. The guidewire insertion section 14 constitutes a guidewire lumen, and is used to receive a guidewire inserted in advance into a blood vessel and to guide the probe 11 to the affected area by the guidewire. The catheter sheath 11a forms a tube section that is continuous from the guidewire insertion section 14 to the connector section 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip side of the shaft 13.

[0017] The sensor unit 12 has a housing 12c, and the tip side of the housing 12c is formed in a hemispherical shape to suppress friction and catching with the inner surface of the catheter sheath 11a. In the housing 12c, an ultrasonic transmission / reception unit 12a that transmits ultrasonic waves into a blood vessel and receives reflected waves from the inside of the blood vessel, and an optical transmission / reception unit 12b that transmits near-infrared light into the blood vessel and receives reflected light from the inside of the blood vessel are arranged. In the example shown in FIG. 2, the ultrasonic transmission / reception unit 12a is provided on the tip side of the probe 11, and the optical transmission / reception unit 12b is provided on the base end side. That is, the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b are arranged in the housing 12c, separated by a predetermined length along the axial direction on the central axis of the shaft 13 (on the two-dot chain line in FIG. 2). The ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b are arranged so that the transmission / reception directions of the ultrasonic waves and the near-infrared light are at an angle of approximately 90 degrees to the axial direction of the shaft 13 (the radial direction of the shaft 13). It is preferable that the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b are attached slightly offset from the radial direction so as not to receive the reflected waves and light from the inner surface of the catheter sheath 11a. In this embodiment, for example, as shown by the arrow in Fig. 2, the ultrasonic transmission / reception unit 12a is disposed in a position in which the direction of irradiation of ultrasonic waves is inclined toward the base end side with respect to the radial direction, and the optical transmission / reception unit 12b is disposed in a position in which the direction of irradiation of near-infrared light is inclined toward the tip end side with respect to the radial direction.

[0018] An electric signal cable (not shown) connected to the ultrasonic transmission / reception unit 12a and an optical fiber cable (not shown) connected to the optical transmission / reception unit 12b are inserted into the shaft 13. The probe 11 is inserted into a blood vessel from the tip side. The sensor unit 12 and the shaft 13 can advance and retreat inside the catheter sheath 11a and can rotate in the circumferential direction. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13 as the rotation axis.

[0019] The MDU 2 is a drive device to which the connector 15 of the diagnostic imaging catheter 1 is detachably attached, and which drives a built-in motor in response to an operation by a medical professional to control the operation of the diagnostic imaging catheter 1 inserted into a blood vessel. For example, the MDU 2 performs a pull-back operation to rotate the sensor unit 12 and shaft 13 inserted into the probe 11 in a circumferential direction while pulling them toward the MDU 2 side at a constant speed. The sensor unit 12 moves from the distal end side to the proximal end side and rotates while scanning the blood vessel continuously at a predetermined time interval, thereby continuously capturing multiple transverse slice images approximately perpendicular to the probe 11 at a predetermined interval. The MDU 2 outputs the reflected wave signal of the ultrasound received by the ultrasound transmitting / receiving unit 12a and the reflected light signal received by the light transmitting / receiving unit 12b to the image processing device 3. The MDU 2 has a high-speed pullback mode in which the sensor unit 12 is retracted at a high speed, and a low-speed pullback mode in which the sensor unit 12 is retracted at a low speed. In the high-speed pullback mode, the MDU 2 drives the motor at a high speed to retract the sensor unit 12 at a high speed. In the low-speed pullback mode, the MDU 2 drives the motor at a low speed to retract the sensor unit 12 at a low speed. The image processing device 3 recognizes whether it is operating in the high-speed pullback mode or the low-speed pullback mode, and executes processing according to the pullback speed. Details will be described later.

[0020] The image processing device 3 acquires the reflected wave signal of the ultrasonic wave output from the ultrasonic transmission / reception unit 12a via the MDU 2 as reflected wave data, and generates ultrasonic line data based on the acquired reflected wave data. The ultrasonic line data is data indicating the reflection intensity of the ultrasonic wave in the depth direction of the blood vessel as seen from the ultrasonic transmission / reception unit 12a. The image processing device 3 constructs an IVUS image 60 (see FIG. 4) showing a transverse layer of the blood vessel based on the generated ultrasonic line data.

[0021] The image processing device 3 also obtains interference light data by causing interference between the reflected light signal output from the optical transceiver 12b via the MDU 2 and a reference light obtained by separating the light from the light source, and generates optical line data based on the obtained interference light data. The optical line data is data indicating the reflection intensity of the reflected light in the depth direction of the blood vessel as viewed from the optical transceiver 12b. The image processing device 3 constructs an OCT image 70 (see FIG. 4) showing the cross-sectional layers of the blood vessel based on the generated optical line data.

[0022] Here, the ultrasonic line data and optical line data obtained by the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b, and the IVUS image 60 and the OCT image 70 constructed from the ultrasonic line data and the optical line data will be described.

[0023] Fig. 3 is an explanatory diagram showing a schematic cross section of a blood vessel through which the sensor unit 12 is inserted, and Fig. 4 is an explanatory diagram of a tomographic image. First, the operation of the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b in the blood vessel, and the ultrasonic line data and the optical line data obtained by the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b will be described with reference to Fig. 3.

[0024] When the sensor unit 12 and the shaft 13 are inserted into the blood vessel and imaging of a tomographic image is started, the sensor unit 12 rotates in the direction indicated by the arrow with the central axis of the shaft 13 as the center of rotation. At this time, the ultrasonic transmission / reception unit 12a transmits and receives ultrasonic waves at each rotation angle. Lines 1, 2, ... 512 indicate the transmission and reception direction of ultrasonic waves at each rotation angle. In this embodiment, the ultrasonic transmission / reception unit 12a intermittently transmits and receives ultrasonic waves 512 times during a 360-degree rotation (one rotation) in the blood vessel. The ultrasonic transmission / reception unit 12a obtains one line of data in the transmission and reception direction by transmitting and receiving ultrasonic waves once, so that 512 pieces of ultrasonic line data extending radially from the center of rotation can be obtained during one rotation. The 512 pieces of ultrasonic line data are dense near the center of rotation, but become sparse as they move away from the center of rotation. Therefore, the image processing device 3 can construct a two-dimensional IVUS image 60 as shown in FIG. 4A by generating pixels in the empty spaces of each line by a well-known interpolation process.

[0025] Similarly, the optical transceiver 12b also transmits and receives near-infrared light (measurement light) at each rotation angle. The optical transceiver 12b also transmits and receives measurement light 512 times while rotating 360 degrees in the blood vessel, so that data of 512 optical lines extending radially from the center of rotation can be obtained during one rotation. For the optical line data, the image processing device 3 generates pixels in the empty spaces of each line by known interpolation processing, thereby constructing the two-dimensional OCT image 70 shown in FIG. 4A.

[0026] A two-dimensional tomographic image constructed from a plurality of ultrasonic line data in this manner is referred to as one frame of an IVUS image 60. Also, a two-dimensional tomographic image constructed from a plurality of optical line data is referred to as one frame of an OCT image 70. Since the sensor unit 12 scans while moving inside a blood vessel, one frame of an IVUS image 60 or an OCT image 70 is acquired at each position of one rotation within the movement range. That is, one frame of an IVUS image 60 or an OCT image 70 is acquired at each position from the distal end side to the proximal end side of the probe 11 within the movement range, so that multiple frames of an IVUS image 60 or an OCT image 70 are acquired within the movement range as shown in FIG. 4B.

[0027] Note that the number of times that ultrasonic waves and light are transmitted and received in one rotation is an example, and the number of times that transmission and reception is not limited to 512. In addition, the number of times that ultrasonic waves are transmitted and received and the number of times that light is transmitted and received may be the same or different.

[0028] In order to minimize the amount of contrast agent required to obtain the OCT image 70, it is desirable to pull back the sensor unit 12 at a high speed. However, since it takes time for the transmitted ultrasound to be reflected at the target site and return, the intravascular inspection device 101 reduces the number of transmissions and receptions when the pullback speed is increased. In other words, the intravascular inspection device 101 reduces the number of acquired ultrasound line data so that it can receive reflected waves at a required depth in the blood vessel. Note that, since the propagation speed of near-infrared light is sufficiently fast, the number of acquired optical line data can be maintained regardless of the movement speed of the sensor unit 12. For example, when the speed at which the sensor unit 12 is retracted is slow (slow pullback), the image processing device 3 acquires 512 pieces of ultrasonic line data and optical line data during one rotation of the sensor unit 12, whereas when the speed at which the sensor unit 12 is retracted is fast (high-speed pullback), the image processing device 3 acquires 256 pieces of ultrasonic line data and 512 pieces of optical line data during one rotation of the sensor unit 12. As described above, the intravascular inspection device 101 and the image processing device 3 reduce the number of acquired ultrasonic line data by a larger amount than the number of acquired optical line data when the retraction speed of the sensor unit 12 is fast. The number of acquired optical line data may be maintained or reduced.

[0029] Furthermore, the intravascular inspection device 101 may be configured to shorten the transmission and reception time of ultrasonic waves when the pullback speed is increased, in which case the depth direction of the blood vessel to be imaged becomes shorter. When the pullback speed is increased in this way, the image quality of the IVUS image 60 is reduced. Hereinafter, a high-image-quality IVUS image 60 constructed when the sensor unit 12 is retracted at a first speed is referred to as a high-image-quality IVUS image 60b, and a low-image-quality IVUS image 60 constructed when the sensor unit 12 is retracted at a second speed faster than the first speed is referred to as a low-image-quality IVUS image 60a. The high-image-quality IVUS image 60b and the low-image-quality IVUS image 60a mean that the image quality of one IVUS image 60 is higher or lower than the image quality of the other IVUS image 60 when comparing the image quality of both. When describing technical features that do not require a distinction between the high-image-quality IVUS image 60b and the low-image-quality IVUS image 60a, the two are simply referred to as IVUS images 60 without distinction between them.

[0030] 5 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a processing unit 31, a storage unit 32, an ultrasonic line data generating unit 33, an optical line data generating unit 34, an input / output I / F 35, and a reading unit 36. The processing unit 31 is configured using one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a tensor processing unit (TPU), etc. The processing unit 31 is connected to each hardware unit constituting the image processing device 3 via a bus.

[0031] The storage unit 32 includes, for example, a main storage unit and an auxiliary storage unit. The main storage unit is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the processing unit 31 to execute arithmetic processing. The auxiliary storage unit is a storage device such as a hard disk, an electrically erasable programmable ROM (EEPROM), or a flash memory. The auxiliary storage unit stores the computer program P executed by the processing unit 31, the first learning model 6, the second learning model 7, the image quality improvement learning model 8, and various data required for other processing. The auxiliary storage unit may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit during the manufacturing stage of the image processing device 3, or the image processing device 3 may acquire the computer program P distributed by a remote server device through communication and store it in the auxiliary storage unit. The computer program P may be recorded in a readable manner on a recording medium 30 such as a magnetic disk, an optical disk, or a semiconductor memory, or the reading unit 36 ​​may read the computer program P from the recording medium 30 and store it in the auxiliary storage unit.

[0032] The ultrasound line data generating unit 33 acquires reflected wave signals of the ultrasound output from the ultrasound transmitting / receiving unit 12a of the intravascular inspection device 101 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data.

[0033] The optical line data generating unit 34 obtains interference light data by causing interference between the reflected light signal output from the optical transmitting / receiving unit 12b of the intravascular inspection device 101 and the reference light obtained by separating the light from the light source, and generates optical line data based on the obtained interference light data.

[0034] The input / output I / F 35 is an interface to which the intravascular inspection device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The processing unit 31 acquires an angio image from the angiography device 102 via the input / output I / F 35. The processing unit 31 also displays a medical image on the display device 4 by outputting a medical image signal of the IVUS image 60, the OCT image 70, or the angio image to the display device 4 via the input / output I / F 35. Furthermore, the processing unit 31 receives information input to the input device 5 via the input / output I / F 35.

[0035] The image processing device 3 may be a multi-computer including a plurality of computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 is described as being one computer. In this embodiment, an angiography device 102 that captures two-dimensional angio images is connected to the image processing device 3, but is not limited to the angiography device 102 as long as it is a device that captures images of a patient's tubular organ and the diagnostic imaging catheter 1 from multiple directions outside the living body.

[0036] The processing unit 31 of the image processing device 3 reads out and executes the computer program P stored in the storage unit 32, thereby generating ultrasonic line data in the ultrasonic line data generating unit 33, and executing a process of constructing an IVUS image 60 based on the generated ultrasonic line data. The processing unit 31 also reads out and executes the computer program P stored in the storage unit 32, thereby generating optical line data in the optical line data generating unit 34, and executing a process of constructing an OCT image 70 based on the generated optical line data.

[0037] However, since the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b are arranged at different positions in the sensor unit 12 and have different transmission and reception directions of ultrasonic waves and light, even if the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b scan the blood vessel at the same imaging timing, a deviation occurs in the observation position and observation direction of the blood vessel. The deviation in the observation position is a deviation in the longitudinal direction and running direction of the blood vessel. The deviation in the observation direction is a deviation in the circumferential direction of the blood vessel. In addition, in order to transmit and receive light using the optical transmission / reception unit 12b, it is necessary to replace the blood in the blood vessel using a flushing liquid. Since the propagation speed of ultrasonic waves differs depending on the type of flushing liquid, the scales of the ultrasonic tomographic image and the optical coherence tomographic image will no longer match. Therefore, the processing unit 31 reads out and executes the computer program P stored in the storage unit 32, thereby correcting deviations in the observation position and observation direction in the IVUS image 60 and the OCT image 70 when constructing the IVUS image 60 and the OCT image 70, and executing processing to match the scales of the IVUS image 60 and the OCT image 70. In this manner, the image processing device 3 of this embodiment can construct the IVUS image 60 and the OCT image 70 with the same observation position, observation direction, and scale, and can provide images that are easy to interpret.

[0038] As described above, in order to minimize the amount of contrast agent required to obtain the OCT image 70, it is desirable to pull back the sensor unit 12 at high speed, but in this case, a problem occurs in that the image quality of the IVUS image 60 deteriorates. Therefore, the processing unit 31 reads out and executes a computer program P stored in the storage unit 32, thereby executing a process of generating and constructing a high-image-quality IVUS image 60b using the low-image-quality IVUS image 60a and the OCT image 70 whose image quality does not deteriorate.

[0039] Next, a first learning model 6 and a second learning model 7 used for correcting the deviation of the observation position and the observation direction and for correcting the scale of an ultrasonic tomographic image and an optical coherence tomographic image will be described. In addition, an image quality improvement learning model 8 for improving the image quality of a low-image-quality IVUS image 60a will be described.

[0040] FIG. 6 is a block diagram showing an example of the configuration of the first learning model 6. The first learning model 6 is a model that recognizes a predetermined object included in the IVUS image 60. The first learning model 6 can classify objects in pixel units by using an image recognition technology using, for example, semantic segmentation, and can recognize various objects included in the IVUS image 60. Specifically, the first learning model 6 recognizes a lumen image (first lumen image) in the IVUS image 60. The first learning model 6 also recognizes an object image (first object image) outside the lumen in the IVUS image 60, such as a blood vessel wall image, a plaque image, a stent image, etc. The blood vessel wall is the tunica media, and more specifically, the external elastic membrane (EEM). The first learning model 6 may be configured to recognize the center position of the sensor unit 12.

[0041] The first learning model 6 is, for example, a convolutional neural network (CNN) that has been trained by deep learning. The first learning model 6 has an input layer 6a to which an IVUS image 60 is input, an intermediate layer 6b that extracts and restores features of the IVUS image 60, and an output layer 6c that outputs an object-extracted IVUS image 61 that indicates an object image included in the IVUS image 60 in pixel units. The first learning model 6 is, for example, a U-Net, a FCN (Fully Convolution Network), a SegNet, or the like.

[0042] The input layer 6a of the first learning model 6 has a plurality of neurons that accept input of the IVUS image 60, that is, the pixel values ​​of each pixel that constitutes the IVUS image 60, and passes the input pixel values ​​to the intermediate layer 6b. The intermediate layer 6b has a plurality of convolution layers (CONV layers) and a plurality of deconvolution layers (DECONV layers). The convolution layer is a layer that compresses the dimensions of the IVUS image 60. The feature amount of the object image is extracted by the dimensional compression. The deconvolution layer performs a deconvolution process to restore the original dimensions. The restoration process in the deconvolution layer generates an object-extracted IVUS image 61 in which each pixel has a pixel value (class data) corresponding to the class of the object. The output layer 6c has a plurality of neurons that output the object-extracted IVUS image 61. As shown in FIG. 6, the object-extracted IVUS image 61 is an image that is classified, for example, color-coded, according to the type of object, such as an intraluminal image, a vascular wall image, a plaque image, and a stent image.

[0043] The first learning model 6 can be generated by preparing training data having an IVUS image 60 obtained by the ultrasound transmission / reception unit 12a and an object-extracted IVUS image 61 in which each pixel of the IVUS image 60 is annotated with class data according to the type of the corresponding object, and by machine learning an untrained neural network using the training data. Specifically, the parameters of the neural network are optimized so that the difference between the object-extracted IVUS image 61 output when the IVUS image 60 of the training data is input to the untrained neural network and the image annotated as the training data becomes small. The parameters are, for example, weights (coupling coefficients) between nodes. The method of optimizing the parameters is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method or the like.

[0044] According to the first learning model 6 trained in this manner, by inputting an IVUS image 60 into the first learning model 6 as shown in FIG. 6, an object-extracted IVUS image 61 is obtained in which each part is classified on a pixel-by-pixel basis according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, a plaque image, etc.

[0045] FIG. 7 is a block diagram showing an example of the configuration of the second learning model 7. The second learning model 7 is a model that recognizes a predetermined object included in the OCT image 70. Like the first learning model 6, the second learning model 7 can classify objects on a pixel-by-pixel basis by utilizing an image recognition technique using, for example, semantic segmentation, and can recognize various objects included in the OCT image 70. Specifically, the second learning model 7 recognizes a lumen image (second lumen image) in the OCT image 70. In addition, the second learning model 7 recognizes an object image (second object image) outside the lumen in the OCT image 70, such as a blood vessel wall image, a plaque image, a stent image, etc.

[0046] The second learning model 7 is, for example, a convolutional neural network that has been trained by deep learning. The second learning model 7 has an input layer 7a to which an OCT image 70 is input, an intermediate layer 7b that extracts and restores features of the OCT image 70, and an output layer 7c that outputs an object-extracted OCT image 71 that indicates an object image included in the OCT image 70 in pixel units. The second learning model 7 is, for example, a U-Net, a FCN (Fully Convolution Network), a SegNet, or the like.

[0047] The input layer 7a of the second learning model 7 has a plurality of neurons that accept input of the OCT image 70, that is, the pixel values ​​of each pixel that constitutes the OCT image 70, and passes the input pixel values ​​to the intermediate layer 7b. The intermediate layer 7b has a plurality of convolution layers and a plurality of deconvolution layers. The convolution layer is a layer that compresses the dimensions of the OCT image 70. The feature amount of the object image is extracted by the dimensional compression. The deconvolution layer performs a deconvolution process to restore the original dimensions. The restoration process in the deconvolution layer generates an object extraction OCT image 71 in which each pixel has a pixel value (class data) corresponding to the class of the object. The output layer 7c has a plurality of neurons that output the object extraction OCT image 71. The object extraction OCT image 71 is an image that is classified, for example, color-coded, according to the type of object, such as an intraluminal image, a vascular wall image, a plaque image, and a stent image.

[0048] The second learning model 7 can be generated by preparing training data having an OCT image 70 obtained by the optical transmitting and receiving unit 12b and an object-extracted OCT image 71 in which class data according to the type of object corresponding to each pixel of the OCT image 70 is assigned (annotated), and performing machine learning on an untrained neural network using the training data. Specifically, similar to the learning of the first learning model 6, parameters of the neural network are optimized so that the difference between the object-extracted OCT image 71 output when the OCT image 70 of the training data is input to the untrained neural network and the image annotated as the training data becomes small.

[0049] According to the second learning model 7 trained in this manner, by inputting an OCT image 70 into the second learning model 7 as shown in FIG. 7, an object-extracted OCT image 71 is obtained in which each part is classified on a pixel-by-pixel basis according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, a plaque image, etc.

[0050] 8 is a block diagram showing an example of the configuration of the image quality improvement learning model 8. The image quality improvement learning model 8 is a model that generates a high-quality IVUS image 60b based on a low-quality IVUS image 60a and an OCT image 70 captured at approximately the same observation position. Even during high-speed pullback, the image quality improvement learning model 8 can generate a high-quality IVUS image 60b by supplementing and reconstructing the information of the low-quality IVUS image 60a with the OCT image 70, since the image quality of the OCT image 70 does not deteriorate.

[0051] The image quality improvement learning model 8 is, for example, a convolutional neural network that has been trained by deep learning. The image quality improvement learning model 8 has a first input layer 81a to which a low-quality IVUS image 60a is input, a second input layer 82a to which an OCT image 70 is input, an intermediate layer 8b that extracts and restores features of the low-quality IVUS image 60a and the OCT image 70, and an output layer 8c that outputs a high-quality IVUS image 60b restored based on the extracted features. The image quality improvement learning model 8 is, for example, a U-Net.

[0052] The first input layer 81a of the image quality improvement learning model 8 has a plurality of neurons that receive input of the pixel values ​​of each pixel that constitutes the low-quality IVUS image 60a, i.e., the IVUS image 60, and passes the input pixel values ​​to the intermediate layer 8b. The second input layer 82a has a plurality of neurons that receive input of the OCT image 70, i.e., the pixel values ​​of each pixel that constitutes the OCT image 70, and passes the input pixel values ​​to the intermediate layer 8b. The intermediate layer 8b has a plurality of convolution layers and a plurality of deconvolution layers. The convolution layer is a layer that compresses the dimensions of the low-quality IVUS image 60a and the OCT image 70. The feature amounts of the low-quality IVUS image 60a and the OCT image 70 are extracted by the dimensional compression. The deconvolution layer performs deconvolution processing to restore one high-quality IVUS image 60b. The output layer 8c has a plurality of neurons that output the high-quality IVUS image 60b.

[0053] 9 is a flowchart showing a method for generating the image quality improvement learning model 8. Here, as an example, an example will be described in which the image processing device 3 generates the image quality improvement learning model 8, but the type of computer that generates the image quality improvement learning model 8 is not particularly limited.

[0054] First, the image processing device 3 is set to a high-speed pullback mode in which the sensor unit 12 is pulled back at high speed, and acquires a low-quality IVUS image 60a and an OCT image 70 of a predetermined blood vessel region (step S101). The low-quality IVUS image 60a and the OCT image 70 obtained when high-speed pullback is performed are the IVUS image 60 and the OCT image 70 constructed by scanning the blood vessel at approximately the same observation position. Then, the image processing device 3 sets the sensor unit 12 to a low-speed pullback mode in which the sensor unit 12 is pulled back at a low speed, and acquires a high-quality IVUS image 60b of the same vascular region (step S102). The image processing device 3 accumulates the low-quality IVUS image 60a and the OCT image 70 acquired in steps S101 and S102 when a high-speed pullback is performed, and the high-quality IVUS image 60b acquired when a low-speed pullback is performed, as training data (step S103). The image processing device 3 accumulates a large amount of training data by repeating the processes of steps S101 to S103. Of course, the image processing device 3 may be configured to collect training data acquired and accumulated by other image diagnostic devices 100.

[0055] The low-quality IVUS image 60a, the OCT image 70, and the high-quality IVUS image 60b constituting the training data are desirably images obtained from the same observation position and observation direction. A method for matching the observation positions and observation directions of the low-quality IVUS image 60a and the OCT image 70 of multiple frames by comparing intraluminal images will be described later. By using this method, the observation positions and observation directions of the low-quality IVUS image 60a, the OCT image 70, and the high-quality IVUS image 60b, which are the training data, can be matched.

[0056] Then, the image processing device 3 generates an untrained neural network by machine learning using the accumulated training data (step S104). Specifically, the parameters of the neural network are optimized so that the difference between the IVUS image 60 output when the low-quality IVUS image 60a and the OCT image 70 of the training data are input to the untrained neural network is reduced, and the high-quality IVUS image 60b of the training data is reduced. The parameters are, for example, weights (coupling coefficients) between nodes. The method of optimizing the parameters is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method or the like.

[0057] According to the image quality improvement learning model 8 thus learned, as shown in FIG. 8, a high-quality IVUS image 60b is obtained by inputting a low-quality IVUS image 60a and an OCT image 70 captured at approximately the same observation position using the sensor unit 12.

[0058] 10 is a flowchart showing an image processing procedure according to the first embodiment. The ultrasonic line data generating unit 33 and the optical line data generating unit 34 of the image processing device 3 generate a plurality of ultrasonic line data and optical line data based on the reflected wave data output from the ultrasonic transmitting / receiving unit 12a and the interference light data obtained by interference between the reflected light and the reference light output from the optical transmitting / receiving unit 12b (step S111). The ultrasonic line data and optical line data are assigned line numbers in chronological order of the observation time, for example. The line numbers correspond to the observation time points. In other words, the line numbers correspond to the observation positions and the observation directions.

[0059] Next, the processing unit 31 constructs a plurality of frames of IVUS images 60 based on the ultrasonic line data, and constructs a plurality of frames of OCT images 70 based on the optical line data (step S112). The plurality of frames of IVUS images 60 and OCT images 70 are assigned frame numbers, for example, in chronological order of observation time. The frame numbers correspond to observation positions. The plurality of frames of IVUS images 60 and OCT images 70 correspond to images obtained by observing blood vessels at a plurality of observation positions from the distal end to the proximal end of the probe 11. However, since the arrangement of the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b and the transmission / reception directions of ultrasonic waves or light are different, the observation positions of the IVUS images 60 and the OCT images 70 assigned with the same frame numbers do not necessarily match. In other words, the IVUS images 60 and the OCT images 70 are misaligned in the running direction of the blood vessel. For the same reason, the IVUS images 60 and the OCT images 70 are also misaligned in the circumferential direction of the blood vessel. Furthermore, blood is replaced with a flushing liquid in order to obtain the OCT image 70, and a difference in scale occurs between the IVUS image 60 and the OCT image 70 depending on the type of flushing liquid, etc.

[0060] The processing unit 31, the ultrasound line data generating unit 33, and the optical line data generating unit 34 that execute the processing of steps S111 and S112 function as acquisition units that acquire an IVUS image 60 (ultrasonic tomographic image) and an OCT image 70 (optical coherence tomographic image).

[0061] Next, the processing unit 31 recognizes the lumen image and the object image outside the lumen (e.g., a blood vessel wall image, a plaque image, a stent image) in the IVUS image 60 by inputting the IVUS image 60 to the first learning model 6 (step S113). The processing unit 31 executes image recognition processing for each of the multiple frames of the IVUS image 60. Note that the object-extracted IVUS image 61 output from the first learning model 6 is assigned the same frame number as the input IVUS image 60 or a similar frame number. The processing unit 31 executing the processing of step S113 also functions as a first recognition unit that recognizes the lumen image (first lumen image) in the IVUS image 60.

[0062] Moreover, the processing unit 31 recognizes the lumen image and the object image outside the lumen (e.g., a blood vessel wall image, a plaque image, a stent image) in the OCT image 70 by inputting the OCT image 70 to the second learning model 7 (step S114). The processing unit 31 executes image recognition processing for each of the multiple frames of the IVUS image 60. Note that the object-extracted OCT image 71 output from the second learning model 7 is assigned the same frame number as the input OCT image 70 or a similar frame number. Moreover, the processing unit 31 that executes the processing of step S114 functions as a second recognition unit that recognizes the lumen image (second lumen image) in the OCT image 70.

[0063] Although the object images that can be recognized by the first learning model 6 and the second learning model 7 are not necessarily the same, the processing unit 31 can recognize at least the lumen image in the IVUS image 60 and the lumen image in the OCT image 70. In addition, some of the object images, such as the blood vessel wall, plaque, and stent, that are outside the lumen image are recognized as common object images in both the IVUS image 60 and the OCT image 70.

[0064] 11 is an explanatory diagram showing three-dimensional lumen images 65, 75 constructed based on an object-extracted IVUS image 61 and an object-extracted OCT image 71. The object-extracted IVUS images 61 of multiple frames show the lumen region at multiple observation positions in the running direction of the blood vessel, and the processing unit 31 can construct a three-dimensional lumen image 65 based on images of the lumen region included in the object-extracted IVUS images 61 of multiple frames, as shown in the upper diagram of FIG. Similarly, the object extraction OCT images 71 of multiple frames show the lumen region at multiple observation positions in the direction of the blood vessel, and the processing unit 31 can construct a three-dimensional lumen image 75 based on the images of the lumen region contained in the object extraction OCT images 71 of the multiple frames, as shown in the lower diagram of Figure 11.

[0065] Next, the processing unit 31 Rigid Body Registration Thus, the processing unit 31 specifies the amount of translation, rotation, and scale correction coefficient of the three-dimensional lumen image 65 that matches the position, orientation, and dimensions of the three-dimensional lumen images 65, 75 (step S115). That is, the processing unit 31 specifies the correspondence of frame numbers in which the lumen images in the multiple frames of IVUS images 60 match the lumen images in the multiple frames of OCT images 70. The processing unit 31 also specifies the amount of rotation of the IVUS images 60 that matches the orientation of the lumen image in the IVUS images 60 and the orientation of the lumen image in the OCT images 70. The processing unit 31 also specifies the amount of radial scale correction of the lumen portion in the IVUS images 60.

[0066] Figure 12 shows Rigid Body Registration 12 is an explanatory diagram conceptually showing a method of matching three-dimensional lumen images 65, 75 and correcting the observation position, observation direction, and scale by using the above method. In Fig. 12, the upper diagram is a three-dimensional lumen image 65 based on an object-extracted IVUS image 61, and the lower diagram is a three-dimensional lumen image 75 based on an object-extracted OCT image 71. The three-dimensional lumen image 65 and the three-dimensional lumen image 75 are misaligned in the observation position, observation direction, and scale in the radial direction of the blood vessel. The processing unit 31 is, for example, as shown in the upper and center figures. Rigid Body Registration By translating the 3D lumen image 65 based on the object-extracted IVUS image 61, rotating it in the circumferential direction, and expanding and contracting the blood vessel lumen in the radial direction, the amount of translation, the amount of rotation, and the scale correction coefficient of the lumen part are specified so that the position, orientation, and dimensions of the 3D lumen images 65, 75 match. Specifically, the processing unit 31 calculates the similarity or dissimilarity between the 3D lumen image 65 and the 3D lumen image 75 in the 3D coordinate system, and specifies the amount of translation, the amount of rotation, and the scale correction coefficient of the lumen part so that the similarity is maximized or the dissimilarity is minimized. For example, normalized cross-correlation or the like may be used as the similarity. For the dissimilarity, the sum of squared residuals or the like may be used. The processing unit 31 may calculate the similarity or dissimilarity of the contours of the 3D lumen images 65, 75.

[0067] FIG. 13 is an explanatory diagram conceptually showing the correspondence relationship between the IVUS image 60 and the OCT image 70 and a method of correcting the scale. Rigid Body Registration The translation of the three-dimensional lumen image 65 by corresponds to a correction of the correspondence between the frame number of the IVUS image 60 and the frame image of the OCT image 70. Moreover, the rotation of the three-dimensional lumen image 65 corresponds to an image rotation process in the IVUS image 60.

[0068] The processing unit 31 corrects the correspondence between the IVUS image 60 and the OCT image 70 (step S116). That is, the frame of the IVUS image 60 captured at the same observation position is associated with the frame of the OCT image 70. Specifically, the frame number of the IVUS image 60 captured at the same observation position is associated with the frame number of the OCT image 70. Since the translation amount identified in step S115 corresponds to the difference between the frame number of the IVUS image 60 captured at the same observation position and the frame number of the OCT image 70, the IVUS image 60 can be associated with the OCT image 70 by adding or subtracting the number of frames corresponding to the difference. The image processing device 3 stores the correspondence between the translation amount and the difference in frame numbers.

[0069] The processing unit 31 also corrects the orientation of the IVUS image 60 so that the orientation of the object image in the IVUS image 60 and the orientation of the object image in the OCT image 70 match, i.e., so that the observation directions are the same (step S117). The amount of rotation identified in step S115 corresponds to the amount of deviation in the orientations of the IVUS image 60 and the OCT image 70, i.e., the amount of deviation in the rotation angle, so that the orientations of the IVUS image 60 and the OCT image 70 can be aligned by rotating the IVUS image 60 by the amount of deviation.

[0070] In addition, 3D intraluminal images65,75 Rigid Body Registration However, the observation position, observation direction, and scale may be corrected by directly comparing the object-extracted IVUS image 61 and the object-extracted OCT image 71 without constructing the three-dimensional lumen images 65, 75.

[0071] Before the translation, the IVUS image 60 of the nth frame and the OCT image 70 of the nth frame are assumed to correspond to each other. n is an integer equal to or greater than 1. The processing unit 31 calculates the similarity or difference between the lumen region in the object-extracted IVUS image 61 of the corresponding frame number and the lumen region in the object-extracted OCT image 71. The processing unit 31 calculates the similarity or difference between the lumen region for a predetermined number of object-extracted IVUS images 61 and object-extracted OCT images 71. Next, the processing unit 31 similarly calculates the similarity or difference between the lumen region in the object-extracted IVUS image 61 of the (n+α)th frame and the lumen region in the object-extracted OCT image 71 of the nth frame. α is an integer and corresponds to the amount of translation of the three-dimensional lumen image 65. The processing unit 31 verifies the correspondence between the object-extracted IVUS image 61 and the object-extracted OCT image 71 while incrementing the variable α by 1. Furthermore, when comparing the lumen regions, the processing unit 31 rotates the orientation of the lumen region in the object extraction IVUS image 61 in the circumferential direction by the angle correction amount θ. The angle correction amount θ corresponds to the amount of rotation of the three-dimensional lumen image 65. The center of rotation is not the center of the object extraction IVUS image 61, but is rotated around a position corresponding to the center of the sensor unit 12. Furthermore, when comparing the lumen regions, the processing unit 31 expands or contracts the radial dimension of the lumen region in the object extraction IVUS image 61 by the scale correction coefficient k1. The processing unit 31 calculates the similarity or dissimilarity of the lumen region in the object extraction IVUS image 61 and the object extraction OCT image 71 while changing the angle correction amount θ and the scale correction coefficient k by a predetermined amount together with the variable α, and specifies the variable α, the angle correction amount θ, and the scale correction coefficient k1 that maximizes the similarity or minimizes the dissimilarity. The processing unit 31 then stores the specified variable α, the angle correction amount θ, and the scale correction coefficient k1.

[0072] In addition, the amount of deviation of the lumen region generally falls within a certain range depending on the arrangement of the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b and the characteristics of the available flushing liquid. For this reason, it is advisable to set a predetermined search range for each of the variable α, the angle correction amount θ, and the scale correction coefficient k1. It is advisable to configure the processing unit 31 to specify the variable α, the angle correction amount θ, and the scale correction coefficient k1 that maximizes the similarity or minimizes the dissimilarity of the lumen region within the search range. In addition, since the frame numbers of the IVUS image 60 and the OCT image 70, which match the observation position, may differ depending on the arrangement of the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b, the irradiation direction of the ultrasonic wave and the near-infrared light, the size of the blood vessel, etc., it is preferable that the search range of the variable α includes positive and negative values. However, it is preferable that the width of the search range on the positive side of the variable α and the width of the search range on the negative side are asymmetric depending on the arrangement of the ultrasonic transmission / reception unit 12a and the optical transmission / reception unit 12b. This makes it possible to specify the variable α more efficiently.

[0073] Next, the processing unit 31 corrects the scale of the lumen portion in the IVUS image 60 (step S118).

[0074] FIG. 14 is an explanatory diagram conceptually showing a method of correcting the scale of the lumen part. The upper diagram shows an object-extracted IVUS image 61 and an IVUS image 60 before correction. The lower diagram shows an object-extracted OCT image 71 and an IVUS image 60 after scale correction. The processing unit 31 identifies a scale correction coefficient k1 that matches the dimensions of the lumen image in the IVUS image 60 with the dimensions of the lumen image in the OCT image 70 by comparing the dimensions of the lumen region in the object-extracted IVUS image 61 with the dimensions of the lumen region in the object-extracted OCT image 71 through the above-mentioned processing. The correction coefficient k1 is a coefficient that corrects the radial length scale of the lumen image centered on the part corresponding to the sensor unit 12 in the IVUS image 60. The processing unit 31 performs scale correction by linearly converting the radial length of the lumen image in the IVUS image 60 using the correction coefficient k1. More specifically, if the distance between an arbitrary object image in the lumen region and the center of the sensor unit 12 is R1, then the distance R1' between the corrected object image and the center of the sensor unit 12 is expressed as R1 × k1. The processing unit 31 determines the correction coefficient k1 such that the contour when the lumen region of the object extraction IVUS image 61 is corrected with the correction coefficient k1 matches the contour of the lumen region in the object extraction OCT image 71, in other words, such that the similarity between the two contours is maximized or the difference is minimized. Then, in step S118, the processing unit 31 uses the obtained correction coefficient k1 to correct the radial length scale of the portion corresponding to the lumen region in the IVUS image 60. Here, the scale of the outer lumen region is not corrected. If the lumen area is reduced, The outer lumen region is circumferentially expanded by known interpolation processes. If the lumen area expands, The lumen outer region is narrowed in the circumferential direction by thinning processing. Note that the scale correction of the lumen region may be performed using IVUS line data, and the IVUS image 60 may be reconstructed using the scale-corrected IVUS line data.

[0075] Next, the processing unit 31 corrects the scale of the portion outside the lumen in the IVUS image 60 (step S119). The processing unit 31 that executes the processes of steps S115 to S119 functions as a correction unit that performs various correction processes such as observation position, observation direction, and scale correction.

[0076] FIG. 15 is an explanatory diagram conceptually showing a scale correction method for the lumen outside portion. The upper diagram shows an object extraction IVUS image 61 and an IVUS image 60 before correction, and the lower diagram shows an object extraction OCT image 71 and an IVUS image 60 after scale correction. The processing unit 31 identifies a scale correction coefficient k2 that makes the object image outside the lumen in the IVUS image 60 match the size of the object image outside the lumen in the OCT image 70 by comparing an arbitrary object image outside the lumen region in the object extraction IVUS image 61 with the size of the object image outside the lumen region in the object extraction OCT image 71. The correction coefficient k2 is a coefficient that corrects the radial length scale on the lumen outside side centered on the part corresponding to the sensor unit 12 in the IVUS image 60. The processing unit 31 performs scale correction by linearly converting the radial length of the image outside the lumen in the IVUS image 60 using the correction coefficient k2. More specifically, if the distance between an arbitrary object image outside the lumen region and the center of the sensor unit 12 is R2, the distance R2' between the corrected object image and the center of the sensor unit 12 is expressed as R1'+(R2-R1)×k2. The processing unit 31 obtains a correction coefficient k2 that matches the object image outside the lumen when the lumen region of the object extraction IVUS image 61 is corrected with the correction coefficient k1 with the object image outside the lumen in the object extraction OCT image 71, in other words, maximizes the similarity between the two object images or minimizes the dissimilarity between the two object images. The processing unit 31 corrects the radial length scale of the part outside the lumen in the IVUS image 60 using the obtained correction coefficient k2. Note that the scale correction of the lumen outside region may be performed using IVUS line data, and the IVUS image 60 may be reconstructed using the IVUS line data after the scale correction.

[0077] Next, when the IVUS image 60 is a low-quality IVUS image 60a, the processing unit 31 executes image quality improvement processing of the low-quality IVUS image 60a using the image quality improvement learning model 8 (step S120). Specifically, as shown in FIG. 8, the low-quality IVUS image 60a is input to a first input layer 81a of the image quality improvement learning model 8, and the OCT image 70 is input to a second input layer 81b to generate a high-quality IVUS image 60b. Note that, by inputting the low-quality IVUS image 60a and the OCT image 70, which have matching observation positions, observation directions, and scales, to the image quality improvement learning model 8, a more accurate high-quality IVUS image 60b can be obtained. Note that the processing unit 31 that executes the processing of step S120 functions as an image quality improvement processing unit that performs image quality improvement of the IVUS image 60 using the image quality improvement learning model 8.

[0078] Then, the processing unit 31 displays the IVUS image 60 and the OCT image 70, which have been corrected in the observation position, observation direction, and scale, and have had their image quality improved, on the display device 4 (step S121).

[0079] The processing unit 31 may be configured to first display the low-quality IVUS image 60a before image quality improvement on the display device 4, and each time a high-quality IVUS image 60b is generated, display the low-quality IVUS image 60a instead of the high-quality IVUS image 60b. Also, the processing unit 31 may perform image quality improvement processing on the priority droplet of the currently displayed low-quality IVUS image 60a.

[0080] As described above, according to the image processing device 3 etc. of this embodiment 1, it is possible to align the observation position and observation direction of the IVUS image 60 and the OCT image 70 obtained using a dual-type catheter equipped with an IVUS function and an OCT function.

[0081] The scales, particularly the radial scales, of the IVUS and OCT images 60 and 70 obtained using a dual type catheter with IVUS and OCT capabilities can be matched.

[0082] Furthermore, a high-quality IVUS image 60b can be constructed even during high-speed pullback. Specifically, a high-quality IVUS image 60b can be constructed based on a low-quality IVUS image 60a and an OCT image 70. Therefore, it is possible to achieve both high-speed imaging of the IVUS image 60 and the OCT image 70 using a dual-type catheter equipped with an IVUS function and an OCT function, reduction in contrast agent consumption, and high image quality.

[0083] In this embodiment 1, an example has been described in which the first learning model 6 and the second learning model 7 are used to recognize the lumen image in the IVUS image 60 and the OCT image 70, but the contour of the lumen image may be extracted by image processing such as binarization, edge detection, pattern matching, etc.

[0084] Also, an example has been described in which the scale correction is performed using different correction coefficients for the lumen portion and the lumen outer portion in the IVUS image 60, but it may be configured such that the radial scale is corrected only for the lumen portion. Furthermore, although an example has been described in which a single scale correction coefficient k1 is used to perform scale correction of the lumen images in all IVUS images 60, it is also possible to specify a correction coefficient k1 for each predetermined number of consecutive frames and correct the scale of the lumen images in the IVUS images 60.

[0085] Furthermore, although an example has been described in which correction of the observation position, observation direction, and radial scale correction of the IVUS image 60 and the OCT image 70 is performed, the configuration may be such that either one or two of correction of the observation position and observation direction, and radial scale correction are performed.

[0086] Furthermore, in this embodiment, 、3 Although an example in which the translation amount, rotation amount, and scale correction amount of the three-dimensional lumen images 65 and 75 are specified has been described, a case in which the scale correction coefficient is not specified ,flatAlternatively, the apparatus may be configured to specify only the amount of row shift and the amount of rotation. Although the linear transformation for translating, rotating, enlarging, or reducing the three-dimensional lumen images 65, 75 has been mainly described, the apparatus may be configured to calculate the amount of correction for aligning the three-dimensional lumen images 65, 75 by using a high-order polynomial or other nonlinear transformation.

[0087] Furthermore, an example has been described in which the image quality of the IVUS image 60 is improved using a low-image-quality IVUS image 60a in which the observation position and observation direction have been matched and scale correction has been performed, and an OCT image 70. However, it is also possible to configure the system to generate a high-image-quality IVUS image 60b by inputting the low-image-quality IVUS image 60a and the OCT image 70 into the image quality improvement learning model 8 without performing such correction.

[0088] Furthermore, in this embodiment 1, an example has been described in which the orientation and scale of the IVUS image 60 are corrected based on the OCT image 70, but it may also be configured to correct the orientation and scale of the OCT image 70 based on the IVUS image 60. Specifically, in step S115, the processing unit 31 specifies the amount of rotation of the OCT image 70 that matches the orientation of the lumen image in the IVUS image 60 with the orientation of the lumen image in the OCT image 70. The processing unit 31 also specifies the amount of scale correction in the radial direction of the lumen portion in the OCT image 70. The processing unit 31, for example, Rigid Body Registration By translating and rotating the 3D lumen image 75 based on the object-extracted OCT image 71 in the circumferential direction, and by expanding and contracting the vascular lumen in the radial direction, the amount of translation, rotation, and scale correction coefficient of the lumen portion that matches the position, orientation, and dimensions of the 3D lumen images 65, 75 are identified. Then, in step S117, the processing unit 31 corrects the orientation of the OCT image 70 so that the orientation of the object image in the IVUS image 60 and the orientation of the object image in the OCT image 70 match, that is, so that the observation directions are the same. In addition, the processing unit 31 corrects the scale of the inner cavity portion in the OCT image 70 in step S118, and corrects the scale of the outer cavity portion in the OCT image 70 in step S119.

[0089] Furthermore, if there is a third medical image other than the IVUS image 60 and the OCT image 70, the IVUS image 60 and the OCT image 70 may be configured to be corrected based on that medical image so that the observation position, observation direction, and scale of the IVUS image 60 and the OCT image 70 are consistent with each other.

[0090] (Embodiment 2) The image diagnostic device 100 according to the second embodiment differs from the first embodiment in that the ultrasonic line data and the light line database are used to correct the observation position, observation direction, and scale. Since the other configurations of the image processing device 3 are similar to those of the image processing device 3 according to the first embodiment, the same reference numerals are used for the similar parts, and detailed descriptions thereof will be omitted.

[0091] 16 is a flowchart showing an image processing procedure according to embodiment 2. The processing unit 31 generates ultrasonic line data and optical line data (step S211) in the same manner as in embodiment 1. Then, the processing unit 31 performs image recognition of an intraluminal image, a vascular wall image, plaque, a stent, and the like, using the ultrasonic line data and the optical line data (steps S212 and S213). The first learning model 6 and the second learning model 7 learn not from the frame images of the IVUS image 60 and the OCT image 70 constructed as frame images, but from the ultrasonic line data and the optical line data constituting one frame as training data. In other words, an image in which the ultrasonic line data constituting one frame are arranged is an image expressed in polar coordinates with the first axis being the circumferential direction and the second axis being the depth direction. When ultrasonic line data constituting one frame is input to the first learning model 6, object-extracted ultrasonic line data from which an object image is extracted is output. Similarly, when OCT line data constituting one frame is input to the second learning model 7, object-extracted OCT line data from which an object image is extracted is output.

[0092] Next, the processing unit 31 performs the same correction of the observation position, observation direction, and scale on a line data basis as in the first embodiment (steps S214 to S218). The processes of steps S215 and S216 may be performed in a single process. The processing unit 31 constructs an IVUS image 60 and an OCT image 70 based on the corrected ultrasound line data and optical line data (step S219). The processes of steps S220 and S221 are the same as those in the first embodiment.

[0093] FIG. 17 is an explanatory diagram showing a method of correcting the correspondence between the ultrasound line data and the light line data. The upper diagram is a schematic diagram showing the ultrasound line data, and the lower diagram is a schematic diagram showing the light line sensor. As in the first embodiment, the processing unit 31 assumes that the n-th line of ultrasound line data and the n-th line of light line data are associated with each other before translation. n is an integer of 1 or more. The processing unit 31 calculates, for example, the similarity or difference between the lumen area in the corresponding n-th line to m-th line of object-extraction ultrasound line data and the lumen area in the object-extraction light line data. m is an integer greater than n. Next, the processing unit 31 similarly calculates the similarity or difference between the lumen area in the (n+β)th line to (m+β)th line of ultrasound line data and the lumen area in the object-extraction light line data in the n-th line to m-th line of object-extraction light line data. The variable β is an integer of 1 or more, and corresponds to the amount of translation and rotation of the three-dimensional lumen image 65. The corresponding relationship between the ultrasonic line data and the optical line data is verified while incrementing the variable β by 1. Then, the processing unit 31 identifies the variable β that maximizes the similarity or minimizes the dissimilarity. Then, the processing unit 31 stores the identified variable β.

[0094] FIG. 18 is an explanatory diagram showing a method of correcting the scale of ultrasonic line data and light line data. A scale correction coefficient k1 is specified to match the dimensions of the lumen image of the ultrasonic line data so that the contour of the lumen region in the object extraction ultrasonic line data and the contour of the lumen region in the light line data match. Similarly, a scale correction coefficient k2 of the outside of the lumen of the ultrasonic line data is specified so that the dimensions of the object region outside the lumen in the object extraction ultrasonic line data and the dimensions of the object region outside the lumen in the object extraction light line data match. The processing unit 31 corrects the radial length scale of the part corresponding to the lumen region in the ultrasonic line data using the obtained correction coefficient k1. The processing unit 31 also corrects the radial length scale of the part corresponding to the region outside the lumen in the ultrasonic line data using the correction coefficient k2.

[0095] According to the image processing device 3 etc. of this embodiment 2, similarly to the embodiment 1, it is possible to align the observation position and observation direction of the IVUS image 60 and the OCT image 70 obtained using a dual-type catheter, and further to align the scale.

[0096] The observation position, observation direction, and scale are corrected by performing image processing on the ultrasound line data and the ultrasound line data to construct the IVUS image 60 and the OCT image 70, so that corrections can be made more accurately and efficiently to construct the IVUS image 60 and the OCT image 70.

[0097] In the second embodiment, an example has been described in which the scale of the ultrasound line data is corrected with reference to the light line database, but the scale of the light line database may be corrected with reference to the ultrasound line data. [Explanation of symbols]

[0098] 1 Diagnostic imaging catheter 2 MDU 3. Image Processing Device 4 Display device 5 Input Devices 6 First Learning Model 7. Second Learning Model 8. Image quality improvement learning model 11 Probe 11a 12 Sensor section 12a Ultrasonic transmitter / receiver 12b Optical transmitter / receiver 12c Housing 13 Shaft 14 Guide wire insertion part 15 Connector part 30 Recording media 31 Processing section 32 Storage section 33 Ultrasonic line data generator 34 Optical line data generator 60 IVUS images 60a Low quality IVUS image 60b High-quality IVUS image 61 Object Extraction IVUS Images 65 3D Lumen Imaging 70 OCT images 71 Object Extraction OCT Images 75 3D Lumen Imaging 100 Diagnostic imaging equipment 101 Intravascular inspection device 102 Angiography equipment

Claims

1. While blood in a tubular organ, which is a blood vessel, is being replaced with a flushing liquid, an ultrasonic tomographic image and an optical coherence tomographic image are obtained using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of the tubular organ toward the outside in the radial direction and an optical transceiver unit that transmits and receives light, Recognizing a first lumen image of the tubular organ included in the acquired ultrasonic tomographic image; Recognizing a second lumen image of the tubular organ included in the acquired optical coherence tomographic image; Based on the recognized first and second lumen images, a scale of the ultrasonic tomographic image radially inward from the contour of the first lumen image or a scale of the optical coherence tomographic image radially inward from the contour of the second lumen image is corrected so that dimensions of the contour of the first lumen image and the contour of the second lumen image are consistent. A computer program for causing a computer to execute a process.

2. Recognizing a first object image of a predetermined object located radially outward from the first lumen image included in the acquired ultrasonic tomographic image; Recognizing a second object image of the predetermined object located radially outward from the second lumen image included in the acquired optical coherence tomographic image; Based on the recognized first object image and the recognized second object image, a scale of the ultrasonic tomographic image on the radially outer side of the first lumen image or a scale of the optical coherence tomographic image on the radially outer side of the second lumen image is corrected so that the sizes of the first object image and the second object image are consistent.

2. A computer program product according to claim 1, for causing a computer to carry out a process.

3. The first lumen image in the ultrasonic tomographic image is recognized by inputting the acquired ultrasonic tomographic image into a first learning model that recognizes the first lumen image included in the ultrasonic tomographic image; The acquired optical coherence tomographic image is input to a second learning model that recognizes the second lumen image included in the optical coherence tomographic image, thereby recognizing the second lumen image in the optical coherence tomographic image.

3. A computer program product according to claim 1 or 2, for causing a computer to execute a process.

4. The acquired ultrasonic tomographic image is input to a first learning model that recognizes the first lumen image included in the ultrasonic tomographic image and the first object image located radially outward from the first lumen image, thereby recognizing the first lumen image and the first object image in the ultrasonic tomographic image; The acquired optical coherence tomographic image is input to a second learning model that recognizes the second lumen image included in the optical coherence tomographic image and the second object image located radially outward from the second lumen image, thereby recognizing the second lumen image and the second object image in the optical coherence tomographic image.

3. A computer program product according to claim 2, for causing a computer to execute a process.

5. a plurality of the ultrasonic tomographic images and a plurality of the optical coherence tomographic images are acquired while moving the sensor unit along a running direction of the tubular organ; By comparing the first lumen image included in the plurality of ultrasonic tomographic images with the second lumen image included in the plurality of optical coherence tomographic images, the ultrasonic tomographic images and the optical coherence tomographic images which are tomographic images at substantially the same observation position are identified; Based on the first lumen image and the second lumen image included in the identified ultrasonic tomographic image and optical coherence tomographic image, a scale in the radial direction of the ultrasonic tomographic image or the optical coherence tomographic image is corrected so that the dimensions of the first lumen image and the second lumen image are consistent. A computer program product according to any one of claims 1 to 4, for causing a computer to execute a process.

6. By performing rigid body registration of the three-dimensional lumen image based on the recognized first lumen image and the three-dimensional lumen image based on the recognized second lumen image, the scale is corrected so that the dimensions of the three-dimensional lumen image are consistent.

6. A computer program product according to claim 5, for causing a computer to execute a process.

7. 1. A method of operating an image processing device, comprising: The image processing device includes: While the blood in the tubular organ, which is a blood vessel, is being replaced with a flushing liquid, an ultrasonic tomographic image and an optical coherence tomographic image are constructed from data obtained by a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of the tubular organ toward the outside in the radial direction and an optical transceiver unit that transmits and receives light, Recognizing a first lumen image of the tubular organ included in the acquired ultrasonic tomographic image; Recognizing a second lumen image of the tubular organ included in the acquired optical coherence tomographic image; Based on the recognized first and second lumen images, a scale of the ultrasonic tomographic image radially inward from the contour of the first lumen image or a scale of the optical coherence tomographic image radially inward from the contour of the second lumen image is corrected so that dimensions of the contour of the first lumen image and the contour of the second lumen image are consistent. A method for operating an image processing device.

8. An acquisition unit that acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of the tubular organ radially outward while the blood in the tubular organ, which is a blood vessel, is replaced with a flush liquid, and an optical transceiver unit that transmits and receives light; a first recognition unit that recognizes a first lumen image of the tubular organ included in the ultrasonic tomographic image acquired by the acquisition unit; a second recognition unit that recognizes a second lumen image of the tubular organ included in the optical coherence tomographic image acquired by the acquisition unit; a correction unit that corrects a scale of the ultrasonic tomographic image radially inward from the contour of the first lumen image or a scale of the optical coherence tomographic image radially inward from the contour of the second lumen image based on the first lumen image and the second lumen image recognized by the first recognition unit and the second recognition unit so that dimensions of a contour of the first lumen image and a contour of the second lumen image are consistent with each other; An image processing device comprising:

Images