Computer program, method of operating an image processing device, and image processing device.

Abstract

This computer program causes a computer to perform processing in which: an ultrasound tomography image and an optical coherence tomography image of a luminal organ are simultaneously acquired using a sensor unit having an ultrasound transmitting / receiving unit that transmits and receives ultrasound and a light transmitting / receiving unit that transmits and receives light; an image defect in the ultrasound tomography image or the optical coherence tomography image is determined; a normal optical coherence tomography image corresponding to an ultrasound tomography image determined to have an image defect is input to a learning model, which has been trained to output a corresponding ultrasound tomography image when an optical coherence tomography image is inputted thereto, to thereby cause the learning model to output a corresponding normal ultrasound tomography image; and the ultrasound tomography image determined to have an image defect is corrected using the ultrasound tomography image outputted from the learning model.

Description

Technical Field

[0001] The present invention relates to a computer program, an image processing method, and an image processing apparatus.

Background Art

[0002] As a minimally invasive treatment for ischemic heart diseases such as angina pectoris or myocardial infarction, endovascular treatment represented by Percutaneous Coronary Intervention (PCI) is performed. For preoperative diagnosis in endovascular treatment or postoperative result confirmation, intravascular image diagnostic apparatuses such as an intravascular ultrasound tomographic diagnostic apparatus (IVUS: IntraVascular Ultra Sound) using ultrasonic waves and an optical coherence tomographic diagnostic apparatus (OFDI: Optical Frequency Domain Imaging) using near-infrared rays are used. Further, an image diagnostic apparatus combining the function of IVUS and the function of OFDI, that is, an image diagnostic apparatus including an ultrasonic transmitting / receiving unit capable of transmitting and receiving ultrasonic waves and an optical transmitting / receiving unit capable of transmitting and receiving light, has been proposed (for example, Patent Documents 1 and 2). According to such an image diagnostic apparatus, both an ultrasonic tomographic image that takes advantage of the characteristics of IVUS capable of measuring up to a deep region and an optical coherence tomographic image that takes advantage of the characteristics of OFDI capable of performing measurement with high resolution can be constructed in a single scan.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of Invention

Problems to be Solved by the Invention

[0004] By the way, in IVUS, it is necessary to prime the area around the sensor (inside the catheter) with physiological saline solution in order to image the inside of a blood vessel. If priming is not performed correctly, the ultrasound will be attenuated due to the inclusion of air, etc., which can lead to the problem of not being able to obtain accurate vascular tomographic images (ultrasound images). In OFDI, it is necessary to flush the blood inside the blood vessels with a contrast agent in order to image the inside of the vessels. If the flushing is not done properly, there is a problem in that the blood can interfere with the acquisition of accurate vascular tomographic images (optical coherence tomography). While imaging diagnostic devices that have both IVUS and OFDI functions offer the advantage of being able to observe the same blood vessel simultaneously with two sensor units, they also have the potential to cause image defects due to priming failures and flash failures.

[0005] If image defects occur, such as priming failure, the catheter must be removed from the body, repriming performed, and then the image retaken, which increases the workload and burden on the operator. Similarly, if the flash fails, the contrast agent must be used again for flashing and the image retaken, increasing the amount of contrast agent used and thus increasing the burden on the patient.

[0006] One aspect of this disclosure aims to provide a computer program, an image processing method, and an image processing apparatus capable of correcting defects in ultrasound and optical coherence tomography images obtained using a dual-type catheter equipped with IVUS and OFDI functions. [Means for solving the problem]

[0007] A computer program (program product) relating to one aspect of this disclosure simultaneously acquires ultrasound tomography and optical coherence tomography images of a tubular organ using a sensor unit having an ultrasound transmitting / receiving unit for transmitting and receiving ultrasound and an optical transmitting / receiving unit for transmitting and receiving light, determines whether the acquired ultrasound tomography image or optical coherence tomography image is defective, and causes a learning model that has been trained to output a corresponding ultrasound tomography image (or optical coherence tomography image) when an optical coherence tomography image (or the ultrasound tomography image) is input to output a normal optical coherence tomography image (or the ultrasound tomography image) corresponding to the ultrasound tomography image (or the optical coherence tomography image) determined to be defective, thereby causing the computer to perform a process to correct the ultrasound tomography image (or the optical coherence tomography image) determined to be defective using the ultrasound tomography image (or the optical coherence tomography image) output from the learning model.

[0008] An image processing method relating to one aspect of this disclosure simultaneously acquires an ultrasound tomography image and an optical coherence tomography image of a tubular organ using a sensor unit having an ultrasound transmitting / receiving unit for transmitting and receiving ultrasound and an optical transmitting / receiving unit for transmitting and receiving light, determines whether the acquired ultrasound tomography image or optical coherence tomography image is defective, and, when an optical coherence tomography image (or the ultrasound tomography image) is input to a learning model that has been trained to output a corresponding ultrasound tomography image (or optical coherence tomography image), inputs a normal optical coherence tomography image (or the ultrasound tomography image) corresponding to the ultrasound tomography image (or the optical coherence tomography image) determined to be defective, thereby causing the learning model to output the corresponding normal ultrasound tomography image (or the optical coherence tomography image), and corrects the ultrasound tomography image (or the optical coherence tomography image) determined to be defective using the ultrasound tomography image (or the optical coherence tomography image) output from the learning model.

[0009] An image processing device relating to one aspect of the present disclosure includes: an acquisition unit that simultaneously acquires an ultrasound tomography image and an optical coherence tomography image of a tubular organ using a sensor unit having an ultrasound transmitting / receiving unit for transmitting and receiving ultrasound and an optical transmitting / receiving unit for transmitting and receiving light; a determination unit that determines image defects in the ultrasound tomography image or the optical coherence tomography image acquired by the acquisition unit; a learning model that has been trained to output a corresponding ultrasound tomography image (or optical coherence tomography image) when an optical coherence tomography image (or the ultrasound tomography image) is input to the learning model; and a processing unit that corrects the ultrasound tomography image (or the optical coherence tomography image) that was determined to be defective by inputting a normal optical coherence tomography image (or the ultrasound tomography image) that corresponds to the ultrasound tomography image (or the optical coherence tomography image) that was determined to be defective to the learning model. [Effects of the Invention]

[0010] One aspect of this disclosure is the ability to correct defects in ultrasound and optical coherence tomography images obtained using a dual-type catheter equipped with IVUS and OFDI functions. [Brief explanation of the drawing]

[0011] [Figure 1] This is an explanatory diagram showing an example of the configuration of an imaging diagnostic device. [Figure 2] This is an explanatory diagram showing an example of the configuration of a diagnostic imaging catheter. [Figure 3] This is a schematic diagram illustrating a cross-section of a blood vessel through which the sensor is inserted. [Figure 4A] This is an explanatory diagram of a fault image. [Figure 4B] This is an explanatory diagram of a fault image. [Figure 5] This is a block diagram showing an example of the configuration of an image processing device. [Figure 6A] This is a block diagram showing an example configuration of an IVUS fault detection learning model. [Figure 6B]This is a block diagram showing a configuration example of an OFDI defect detection learning model. [Figure 7A] This is a block diagram showing a configuration example of an IVUS image recognition learning model. [Figure 7B] This is a block diagram showing a configuration example of an OFDI image recognition learning model. [Figure 8] This is a block diagram showing a configuration example of a first generator. [Figure 9] This is a conceptual diagram showing a generation method of the first generator. [Figure 10] This is a block diagram showing a configuration example of a second generator. [Figure 11] This is a flowchart showing an image processing procedure according to the present embodiment. [Figure 12] This is a flowchart showing an image processing procedure according to the present embodiment. [Figure 13] This is a conceptual diagram showing a correction method for IVUS images and OFDI images. [Figure 14] This is an explanatory diagram conceptually showing a method of associating IVUS images and OFDI images. [Figure 15] This is a first screen display example of IVUS images and OFDI images. [Figure 16] This is a second screen display example of IVUS images and OFDI images. [Mode for Carrying Out the Invention]

[0012] Hereinafter, the computer program, image processing method, and image processing apparatus of the present disclosure will be described in detail based on the drawings showing the embodiments thereof. In each of the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example. The present invention is not limited to these illustrations, is indicated by the scope of the claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of the claims. At least a part of the embodiments described below may be arbitrarily combined.

[0013] This embodiment describes an imaging diagnostic device using a dual-type catheter equipped with both intravascular ultrasound (IVUS) and optical coherence tomography (OFDI) functions. The dual-type catheter has three modes: one for acquiring ultrasound tomography images using IVUS only, one for acquiring optical coherence tomography images using OFDI only, and one for acquiring both ultrasound and optical coherence tomography images using both IVUS and OFDI. These modes can be switched between. Hereinafter, ultrasound tomography images and optical coherence tomography images will be referred to as IVUS images and OFDI images, respectively. Furthermore, IVUS images and OFDI images will be collectively referred to as tomography images.

[0014] Figure 1 is an explanatory diagram showing an example configuration of the diagnostic imaging apparatus 100. The diagnostic imaging apparatus 100 of this embodiment comprises an intravascular examination device 101, an angiography device 102, an image processing device 3, a display device 4, and an input device 5. The intravascular examination device 101 comprises a diagnostic imaging catheter 1 and an MDU (Motor Drive Unit) 2. The diagnostic imaging catheter 1 is connected to the image processing device 3 via the MDU 2. The display device 4 and the input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, mouse, trackball, or microphone. The display device 4 and the input device 5 may be stacked together to form a touch panel. The input device 5 and the image processing device 3 may also be configured together. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input, etc.

[0015] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that uses X-rays to image blood vessels from outside the patient's body while injecting a contrast agent into the patient's blood vessels, thereby obtaining an angiographic image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and the X-ray sensor receives the X-rays irradiated from the X-ray source to image an X-ray fluoroscopic image of the patient. The angiography device 102 outputs the acquired angiographic image to the image processing device 3, and displays it on the display device 4 via the image processing device 3.

[0016] Figure 2 is an explanatory diagram showing an example of the configuration of the imaging diagnostic catheter 1. Note that the area enclosed by the upper dashed line in Figure 2 is an enlargement of the area enclosed by the lower dashed line. The imaging diagnostic catheter 1 has a probe 11 and a connector portion 15 located at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector portion 15. In the following description, the side of the imaging diagnostic catheter 1 furthest from the connector portion 15 will be referred to as the tip side, and the side with the connector portion 15 will be referred to as the proximal end side. The probe 11 is equipped with a catheter sheath 11a, and its tip is provided with a guidewire insertion portion 14 through which a guidewire can be inserted. The guidewire insertion portion 14 constitutes a guidewire lumen and is used to receive a guidewire that has been previously inserted into the blood vessel and to guide the probe 11 to the affected area by the guidewire. The catheter sheath 11a forms a continuous tube from the guidewire insertion portion 14 to the connector portion 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip of the shaft 13.

[0017] The sensor unit 12 has a housing 12c, and the tip end of the housing 12c is formed in a hemispherical shape to suppress friction and snagging with the inner surface of the catheter sheath 11a. Inside the housing 12c are an ultrasonic transmitting / receiving unit 12a that transmits ultrasonic waves into the blood vessel and receives reflected waves from within the blood vessel, and an optical transmitting / receiving unit 12b that transmits near-infrared light into the blood vessel and receives reflected light from within the blood vessel. In the example shown in Figure 2, the ultrasonic transmitting / receiving unit 12a is provided on the tip end of the probe 11, and the optical transmitting / receiving unit 12b is provided on the proximal end. In other words, the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are arranged inside the housing 12c at a predetermined distance apart along the axial direction on the central axis of the shaft 13 (on the dashed line in Figure 2). Furthermore, the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are arranged such that the transmission and reception directions of ultrasonic waves and near-infrared light are approximately 90 degrees with respect to the axial direction of the shaft 13 (radial direction of the shaft 13). It is desirable that the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b be mounted slightly offset from the radial direction so as not to receive reflected waves and reflected light from the inner surface of the catheter sheath 11a. In this embodiment, for example, as indicated by the arrow in Figure 2, the ultrasonic transmitting / receiving unit 12a is positioned so that the direction inclined toward the proximal end with respect to the radial direction is the direction of ultrasonic irradiation, and the optical transmitting / receiving unit 12b is positioned so that the direction inclined toward the tip with respect to the radial direction is the direction of near-infrared light irradiation.

[0018] The shaft 13 contains an electrical signal cable (not shown) connected to the ultrasonic transmitting / receiving unit 12a and an optical fiber cable (not shown) connected to the optical transmitting / receiving unit 12b. The probe 11 is inserted into the blood vessel from the tip end. The sensor unit 12 and the shaft 13 can move forward and backward inside the catheter sheath 11a and can also rotate in the circumferential direction. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation.

[0019] The MDU2 is a drive device to which the connector portion 15 of the diagnostic imaging catheter 1 is detachably attached. It drives a built-in motor in response to user (medical professional) operations and controls the movement of the diagnostic imaging catheter 1 inserted into a blood vessel. For example, the MDU2 performs a pullback operation, rotating the sensor portion 12 and shaft 13 inserted into the probe 11 circumferentially while pulling them toward the MDU2 at a constant speed. The sensor portion 12 rotates while moving from the tip to the proximal end due to the pullback operation, and continuously scans the inside of the blood vessel at predetermined time intervals, thereby continuously capturing multiple transverse images approximately perpendicular to the probe 11 at predetermined intervals. The MDU2 outputs the reflected ultrasound wave signal received by the ultrasound transmitting / receiving unit 12a and the reflected light signal received by the optical transmitting / receiving unit 12b to the image processing device 3.

[0020] The image processing device 3 acquires the reflected ultrasound signal output from the ultrasound transmitting / receiving unit 12a via the MDU2 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data. The ultrasound line data is data that shows the reflected ultrasound intensity in the depth direction of the blood vessel as seen from the ultrasound transmitting / receiving unit 12a. Based on the generated ultrasound line data, the image processing device 3 constructs an IVUS image P1 (see Figures 4A and 4B) representing the transverse layer of the blood vessel.

[0021] Furthermore, the image processing device 3 acquires interference light data by interfering the reflected light signal output from the optical transceiver 12b via the MDU2 with the reference light obtained by separating the light from the light source, and generates optical line data based on the acquired interference light data. The optical line data is data that shows the reflection intensity of the reflected light in the depth direction of the blood vessel as seen from the optical transceiver 12b. Based on the generated optical line data, the image processing device 3 constructs an OFDI image P2 (see Figures 4A and 4B) representing the transverse layer of the blood vessel.

[0022] Here, we will explain the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b, and the IVUS image P1 and OFDI image P2 constructed from the ultrasonic line data and optical line data.

[0023] Figure 3 is a schematic diagram showing a cross-section of a blood vessel through which the sensor unit 12 is inserted, and Figures 4A and 4B are explanatory diagrams of tomographic images. First, using Figure 3, we will explain the operation of the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b inside the blood vessel, and the ultrasonic line data and optical line data obtained by the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b.

[0024] When tomographic imaging is started with the sensor unit 12 and shaft 13 inserted into the blood vessel, the sensor unit 12 rotates around the central axis of the shaft 13 in the direction indicated by the arrow. At this time, the ultrasonic transmitting / receiving unit 12a transmits and receives ultrasound at each rotation angle. Lines 1, 2, ... 512 indicate the direction of ultrasound transmission and reception at each rotation angle. In this embodiment, the ultrasonic transmitting / receiving unit 12a intermittently transmits and receives ultrasound 512 times while rotating 360 degrees (1 rotation) inside the blood vessel. Since the ultrasonic transmitting / receiving unit 12a acquires data for one line in the transmission / reception direction with each ultrasound transmission and reception, 512 ultrasound line data points extending radially from the rotation center can be obtained during one rotation. The 512 ultrasound line data points are dense near the rotation center, but become sparser as they move away from the rotation center. Therefore, the image processing device 3 can construct a two-dimensional IVUS image P1 as shown in Figure 4A by generating pixels in the empty spaces between each line using a well-known interpolation process.

[0025] Similarly, the optical transceiver 12b also transmits and receives near-infrared light (measurement light) at each rotation angle. Since the optical transceiver 12b also transmits and receives measurement light 512 times while rotating 360 degrees inside the blood vessel, 512 optical line data lines extending radially from the center of rotation can be obtained during one rotation. With respect to the optical line data, the image processing device 3 can construct the two-dimensional OFDI image P2 shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process.

[0026] A two-dimensional tomographic image constructed from multiple ultrasound line data in this manner is called an IVUS image P1, which is one frame. Similarly, a two-dimensional tomographic image constructed from multiple optical line data is called an OFDI image P2, which is one frame. As the sensor unit 12 scans while moving within the blood vessel, one frame of IVUS image P1 or OFDI image P2 is acquired at each position after one rotation within the movement range. That is, one frame of IVUS image P1 or OFDI image P2 is acquired at each position from the tip to the proximal end of the probe 11 within the movement range, so as shown in Figure 4B, multiple frames of IVUS image P1 or OFDI image P2 are acquired within the movement range.

[0027] The number of ultrasonic and optical transmissions / receptions per rotation is just an example, and the number of transmissions / receptions is not limited to 512. Furthermore, the number of ultrasonic transmissions / receptions and the number of optical transmissions / receptions may be the same or different.

[0028] Figure 5 is a block diagram showing an example configuration of the image processing device 3. The image processing device 3 is a computer and comprises a processing unit 31, a storage unit 32, an ultrasonic line data generation unit 33, an optical line data generation unit 34, an input / output interface 35, and a reading unit 36. The processing unit 31 is configured using one or more computing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), and a TPU (Tensor Processing Unit). The processing unit 31 is connected to each hardware component of the image processing device 3 via a bus.

[0029] The memory unit 32 includes, for example, a main memory unit and an auxiliary memory unit. The main memory unit is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data necessary for the processing unit 31 to perform arithmetic processing. The auxiliary memory unit is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary memory unit stores the computer program P executed by the processing unit 31, the IVUS fault detection learning model 61, the OFDI fault detection learning model 62, the IVUS image recognition learning model 71, the OFDI image recognition learning model 72, the first generator (first learning model) 81, the second generator (second learning model) 82, and various other data necessary for processing. The auxiliary memory unit may also be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary memory unit during the manufacturing stage of the image processing device 3, or it may be distributed by a remote server device and acquired by the image processing device 3 via communication and stored in the auxiliary memory unit. The computer program P may be recorded in a readable manner on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, and the reading unit 36 ​​may read it from the recording medium 30 and store it in the auxiliary storage unit.

[0030] The ultrasound line data generation unit 33 acquires the reflected ultrasound wave signal output from the ultrasound transmitting / receiving unit 12a of the intravascular examination device 101 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data.

[0031] The optical line data generation unit 34 acquires interference light data by interfering the reflected light signal output from the optical transmitting / receiving unit 12b of the intravascular examination device 101 with reference light obtained by separating the light from the light source, and generates optical line data based on the acquired interference light data.

[0032] The input / output interface 35 is an interface to which the intravascular examination device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The processing unit 31 acquires angiography images from the angiography device 102 via the input / output interface 35. The processing unit 31 also displays medical images on the display device 4 by outputting medical image signals of IVUS images P1, OFDI images P2, or angiography images to the display device 4 via the input / output interface 35. Furthermore, the processing unit 31 receives information input to the input device 5 via the input / output interface 35.

[0033] The image processing device 3 may be a multi-computer system comprising multiple computers. Alternatively, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer.

[0034] The processing unit 31 of the image processing device 3 reads and executes a computer program P stored in the storage unit 32, thereby generating ultrasound line data in the ultrasound line data generation unit 33 and constructing an IVUS image P1 based on the generated ultrasound line data. The processing unit 31 also reads and executes a computer program P stored in the storage unit 32, thereby generating optical line data in the optical line data generation unit 34 and constructing an OFDI image P2 based on the generated optical line data.

[0035] Incidentally, since the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b are positioned differently in the sensor unit 12 and transmit / receive ultrasound and light in different directions, even if the ultrasonic transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b scan the blood vessel at the same imaging timing, a discrepancy will occur in the observation position and observation direction of the blood vessel. The discrepancy in observation position is a discrepancy in the longitudinal direction and course of the blood vessel. The discrepancy in observation direction is a discrepancy in the circumferential direction of the blood vessel. Therefore, the processing unit 31 reads and executes the computer program P stored in the storage unit 32, thereby correcting the discrepancy in the observation position and direction in the IVUS image P1 and OFDI image P2 when constructing the IVUS image P1 and OFDI image P2. Specifically, the processing unit 31 uses the IVUS image recognition learning model 71 and the OFDI image recognition learning model 72 to recognize the lumen images in the IVUS image P1 and OFDI image P2. Then, for example, the processing unit 31 identifies the amount of translation and rotation that will match the 3D lumen image based on the IVUS image P1 and the 3D lumen image based on the OFDI image P2, and based on the identified amount of translation and rotation, it associates the IVUS image P1 and OFDI image P2 so that the observation position and direction match.

[0036] Furthermore, if priming is not performed properly, the ultrasound signal may be attenuated due to air contamination, which can result in poor image quality in the IVUS image P1. If the procedure of flushing the blood with contrast agent is not performed properly, poor image quality in the OFDI image P2 may occur due to the influence of the blood. Therefore, the processing unit 31 reads and executes the computer program P stored in the storage unit 32 to detect defects in the IVUS image P1 and OFDI image P2. Specifically, the processing unit 31 uses the IVUS defect detection learning model 61 and the OFDI defect detection learning model 62 to detect defects in the IVUS image P1 and OFDI image P2.

[0037] Furthermore, the processing unit 31 reads and executes a computer program P stored in the storage unit 32 to correct the frame image of the defective IVUS image P1 using the corresponding normal OFDI image P2. Specifically, the processing unit 31 uses the first generator 81 to generate the corresponding normal IVUS image P1 from the normal OFDI image P2 and replaces the defective IVUS image P1 with the generated IVUS image P1. Similarly, the processing unit 31 uses the second generator 82 to read and execute a computer program P stored in the storage unit 32 to correct the frame image of the defective OFDI image P2 using the corresponding normal IVUS image P1. Specifically, the processing unit 31 generates the corresponding normal OFDI image P2 from the normal IVUS image P1 and replaces the defective OFDI image P2 with the generated OFDI image P2. The above process can correct image defects in the IVUS image P1 or OFDI image P2 caused by improper priming or flashing.

[0038] <Fault detection learning model> This section describes the IVUS defect detection learning model 61 and the OFDI defect detection learning model 62, which detect image defects in IVUS image P1 and OFDI image P2. Figure 6A is a block diagram showing an example configuration of the IVUS fault detection learning model 61, and Figure 6B is a block diagram showing an example configuration of the OFDI fault detection learning model 62.

[0039] Figure 6A shows an example configuration of the IVUS defect detection learning model 61. The IVUS defect detection learning model 61 is a trained model that detects defects in the IVUS image P1 due to priming failures. The IVUS defect detection learning model 61 is a neural network having, for example, an input layer 61a that accepts the input of the IVUS image P1, an intermediate layer 61b that extracts image features, and an output layer 61c that outputs whether or not the IVUS image P1 has image defects. The IVUS defect detection learning model 61 in this embodiment is a CNN (Convolutional Neural Network) such as ResNet or DenseNet.

[0040] The image processing device 3 generates an IVUS defect detection learning model 61 by performing deep learning on a CNN model to learn the relationship between IVUS images P1 and the presence or absence of image defects. More specifically, it prepares training data that includes the association of normal IVUS images P1 with training data indicating normality, and the association of defective IVUS images P1 with training data indicating image defects. The image processing device 3 then optimizes the parameters of an untrained neural network so that the difference between the output data and the training data is minimized when the IVUS images P1 from the training data are input to the neural network. These parameters include, for example, the weights (connection coefficients) between nodes. The method of parameter optimization is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method.

[0041] The input layer 61a of the neural network has multiple nodes that accept the pixel values ​​of each pixel related to the IVUS image P1 as input, and passes the input pixel values ​​to the intermediate layer 61b. The intermediate layer 61b has multiple layers, each layer having multiple nodes that extract image features, and passes the extracted features to the output layer 61c. For example, if the IVUS defect detection learning model 61 is a CNN, the intermediate layer 61b has a configuration in which multiple convolution layers that convolve the pixel values ​​of each pixel input from the input layer 61a and pooling layers that map the pixel values ​​convolved in the convolution layer, and finally extracts image features while compressing the image pixel information. The output layer 61c has nodes that output data indicating the presence or absence or probability of image defects.

[0042] In this embodiment, the IVUS defect detection learning model 61 is assumed to be a CNN such as DenseNet, but the model configuration is not limited to a CNN. The IVUS defect detection learning model 61 may be a learning model with a configuration such as a neural network other than a CNN, an SVM (Support Vector Machine), a Bayesian network, or a decision tree such as XGBoost. Furthermore, the output layer 61c may include multiple nodes that output data indicating the type of image defect.

[0043] Figure 6B shows an example configuration of the OFDI defect detection learning model 62. The OFDI defect detection learning model 62 is a trained model that detects defects in OFDI images P2 caused by flash failures. The OFDI defect detection learning model 62 is a neural network having, for example, an input layer 62a that accepts the input of OFDI image P2, an intermediate layer 62b that extracts image features, and an output layer 62c that outputs whether or not the OFDI image P2 has image defects. The image processing device 3 generates the OFDI defect detection learning model 62 by performing deep learning on a CNN model to learn the relationship between OFDI image P2 and the presence or absence of image defects. The details of the OFDI fault detection learning model 62's configuration and generation method are the same as those of the IVUS fault detection learning model 61, so the details are omitted.

[0044] <Image Recognition Learning Model> This section describes the IVUS image recognition learning model 71 and the OFDI image recognition learning model 72, which are used to correct for discrepancies in the observation position and direction of the IVUS image P1 and the OFDI image P2. Figure 7A is a block diagram showing an example configuration of the IVUS image recognition learning model 71, and Figure 7B is a block diagram showing an example configuration of the IVUS image recognition learning model 71 and the OFDI image recognition learning model 72.

[0045] Figure 7A is a block diagram showing an example configuration of the IVUS image recognition learning model 71. The IVUS image recognition learning model 71 is a model that recognizes predetermined objects contained in the IVUS image P1. The IVUS image recognition learning model 71 can classify objects on a pixel-by-pixel basis by using image recognition technology, for example, semantic segmentation, and can recognize various objects contained in the IVUS image P1. Specifically, the IVUS image recognition learning model 71 recognizes the lumen image in the IVUS image P1. The IVUS image recognition learning model 71 may also be configured to recognize the vascular wall image. The vascular wall is the tunica media, and more specifically, the external elastic membrane (EEM). The lumen image and vascular wall image in the IVUS image P1 are examples of first object images.

[0046] The IVUS image recognition learning model 71 is, for example, a pre-trained convolutional neural network (CNN) using deep learning. The IVUS image recognition learning model 71 has an input layer 71a into which the IVUS image P1 is input, an intermediate layer 71b that extracts and reconstructs the features of the IVUS image P1, and an output layer 71c that outputs an object-extracted IVUS image P1' that shows the object images contained in the IVUS image P1 on a pixel-by-pixel basis. Examples of IVUS image recognition learning models 71 include U-Net, FCN (Fully Convolutional Network), SegNet, etc.

[0047] The input layer 71a of the IVUS image recognition learning model 71 has multiple neurons that accept input of an IVUS image P1, that is, the pixel values ​​of each pixel that makes up the IVUS image P1, and passes the input pixel values ​​to the intermediate layer 71b. The intermediate layer 71b has multiple convolutional layers (CONV layers) and multiple deconvolutional layers (DECONV layers). The convolutional layers are layers that compress the dimensions of the IVUS image P1. Through dimensionality reduction, feature quantities of the object image are extracted. The deconvolutional layers perform deconvolution and restore to the original dimensions. The restoration process in the deconvolutional layers generates an object-extracted IVUS image P1' in which each pixel has a pixel value (class data) corresponding to the object class. The output layer 71c has multiple neurons that output the object-extracted IVUS image P1'. As shown in Figure 7A, the object-extracted IVUS image P1' is an image that is classified by type of object, such as a lumen image, a blood vessel wall image, or a plaque image, for example, color-coded.

[0048] The IVUS image recognition learning model 71 can be generated by preparing training data consisting of an IVUS image P1 obtained by the ultrasonic transmitting / receiving unit 12a and an object-extracted IVUS image P1' in which class data corresponding to the type of object is attached (annotated) to each pixel of the IVUS image P1, and then using this training data to machine-learn an untrained neural network. Specifically, the parameters of the neural network are optimized so that the difference between the object-extracted IVUS image P1' output when the IVUS image P1 of the training data is input to the untrained neural network is small, and the annotated image used as training data. These parameters are, for example, the weights between nodes (connection coefficients). The method of parameter optimization is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method.

[0049] According to the IVUS image recognition learning model 71 trained in this manner, by inputting an IVUS image P1 into the IVUS image recognition learning model 71, as shown in Figure 7A, an object-extracted IVUS image P1' is obtained in which each part is classified at the pixel level according to the type of object, such as an image of the lumen of a blood vessel, an image of the blood vessel wall, and an image of plaque.

[0050] Figure 7B is a block diagram showing an example configuration of the OFDI image recognition learning model 72. The OFDI image recognition learning model 72 is a model that recognizes predetermined objects contained in the OFDI image P2. Similar to the IVUS image recognition learning model 71, the OFDI image recognition learning model 72 can classify objects on a pixel-by-pixel basis by utilizing image recognition techniques, such as semantic segmentation, and can recognize various objects contained in the OFDI image P2. Specifically, the OFDI image recognition learning model 72 recognizes lumen images, blood vessel images, etc., in the OFDI image P2. Lumen images and blood vessel wall images in the OFDI image P2 are examples of second object images.

[0051] The OFDI image recognition learning model 72 is, for example, a convolutional neural network trained by deep learning. The OFDI image recognition learning model 72 has an input layer 72a into which the OFDI image P2 is input, an intermediate layer 72b that extracts and reconstructs the features of the OFDI image P2, and an output layer 72c that outputs an object-extracted OFDI image P2' that shows the object images contained in the OFDI image P2 on a pixel-by-pixel basis. Examples of OFDI image recognition learning models 72 include U-Net, FCN (Fully Convolutional Network), SegNet, etc.

[0052] The input layer 72a of the OFDI image recognition learning model 72 has multiple neurons that accept input of an OFDI image P2, that is, the pixel values ​​of each pixel that makes up the OFDI image P2, and passes the input pixel values ​​to the hidden layer 72b. The hidden layer 72b has multiple convolutional layers and multiple deconvolutional layers. The convolutional layers are layers that reduce the dimensionality of the OFDI image P2. Through dimensionality reduction, feature quantities of the object image are extracted. The deconvolutional layers perform deconvolution and restore to the original dimensions. The restoration process in the deconvolutional layers generates an object-extracted OFDI image P2' in which each pixel has a pixel value (class data) corresponding to the object class. The output layer 72c has multiple neurons that output the object-extracted OFDI image P2'. The object-extracted OFDI image P2' is an image that is classified by type of object, such as a lumen image, a blood vessel wall image, a plaque image, or a stent image, for example, color-coded.

[0053] The OFDI image recognition learning model 72 can be generated by preparing training data consisting of an OFDI image P2 obtained by the optical transceiver unit 12b and an object-extracted OFDI image P2' in which class data corresponding to the type of object is attached (annotated) to each pixel of the OFDI image P2, and then using this training data to train an untrained neural network. Specifically, similar to the training of the IVUS image recognition learning model 71, the parameters of the neural network are optimized so that the difference between the object-extracted OFDI image P2' output when the OFDI image P2 of the training data is input to the untrained neural network is small, and the annotated image used as training data.

[0054] According to the OFDI image recognition learning model 72 trained in this manner, by inputting an OFDI image P2 into the OFDI image recognition learning model 72, as shown in Figure 7B, an object-extracted OFDI image P2' is obtained in which each part is classified at the pixel level according to the type of object, such as an image of the lumen of a blood vessel, an image of the blood vessel wall, and an image of plaque.

[0055] <Image generation learning model> Figure 8 is a block diagram showing an example configuration of the first generator 81. The first generator 81 is a learning model that generates an IVUS image P1 from an OFDI image P2. The first generator 81 has an input layer 81a into which the OFDI image P2 is input, an intermediate layer 81b that extracts features from the OFDI image P2 and reconstructs the IVUS image P1, and an output layer 81c that outputs the generated IVUS image P1. The first generator 81 can be a GAN such as pix2pix, CycleGAN, or StarGAN, a VAE (Variational Autoencoder), or a neural network such as a CNN (e.g., U-net).

[0056] The input layer 81a of the first generator 81 has multiple neurons that accept input of an OFDI image P2, that is, the pixel values ​​of each pixel that makes up the OFDI image P2, and passes the input pixel values ​​to the intermediate layer 81b. The intermediate layer 81b has multiple convolutional layers (CONV layers) and multiple deconvolutional layers (DECONV layers). The convolutional layers are layers that compress the dimensions of the IVUS image P1. Dimensionality compression extracts the features of the object image. The deconvolutional layers perform deconvolution and restore the original dimensions. The restoration process in the deconvolutional layers generates an IVUS image P1 in which each pixel has the pixel values ​​of the IVUS image P1. The output layer 81c has multiple neurons that output an IVUS image P1 corresponding to the input OFDI image P2. As shown in Figure 8, when a normal OFDI image P2 is input to the first generator 81, a normal IVUS image P1, which is the generated image, is output.

[0057] The first generator 81 is generated by preparing training data that associates OFDI image P2, which is a normal image for training, with IVUS image P1, which is a normal image for training, and training an untrained learning model using this training data. It is desirable that the OFDI image P2 and IVUS image P1 for training include images of lesions that have been determined to require endovascular treatment. The image processing device 3 of this embodiment performs training using the OFDI image P2 and IVUS image P1 prepared for training and generates the first generator 81 that generates IVUS image P1 from OFDI image P2.

[0058] Figure 9 is a conceptual diagram showing a method for generating the first generator 81. The first generator 81 can be generated, for example, by preparing a discriminator 81d and performing adversarial learning between the first generator 81 and the discriminator 81d. For example, as shown in the upper part of Figure 9, the learning of the discriminator 81d and the learning of the first generator 81 are performed alternately.

[0059] During the training of the classifier 81d, the image processing unit 3 inputs a training OFDI image P2 to the first generator 81 with its parameters fixed. The first generator 81, upon receiving the training OFDI image P2, generates and outputs an IVUS image P1. The image processing unit 3 provides the classifier 81d with pairs of OFDI image P2 (training OFDI image P2) and IVUS image P1 (IVUS image P1 generated by the first generator 81), corresponding to the input and output of the first generator 81, as false data. The image processing unit 3 also provides the classifier 81d with pairs of OFDI image P2 and IVUS image P1 included in the training data as true data. The image processing unit 3 optimizes the parameters of the classifier 81d so that it outputs a false value when false data is input and a true value when true data is input.

[0060] On the other hand, during the training of the first generator 81, the image processing device 3 fixes the parameters of the discriminator 81d and trains the first generator 81 as shown in the lower part of Figure 9. The image processing device 3 inputs the training OFDI image P2 to the first generator 81 and optimizes the parameters of the first generator 81 so that when the training OFDI image P2 and the IVUS image P1 generated by the first generator 81 are input to the discriminator 81d, the probability of truth or falsehood approximates 50%. In this way, by performing the training of the first generator 81 and the training of the discriminator 81d in an adversarial relationship, a first generator 81 is generated that can output the corresponding IVUS image P1 when the OFDI image P2 is input, as shown in Figure 8. The parameters to be optimized are, for example, the weights (connection coefficients) between neurons, and methods such as backpropagation can be used to optimize the parameters.

[0061] When the image processing device 3 generates an IVUS image P1 using the first generator 81, it uses only the first generator 81. The first generator 81 may be trained by another training device. A trained first generator 81, which has been trained by another training device, is downloaded from the training device to the image processing device 3, for example, via a network or recording medium 30, and stored in the storage unit 32.

[0062] Figure 10 is a block diagram showing an example configuration of the second generator 82. The second generator 82 is a learning model that generates an OFDI image P2 from an IVUS image P1. The second generator 82 has the same configuration as the first generator 81. The second generator 82 has an input layer 82a into which the IVUS image P1 is input, an intermediate layer 82b that extracts features from the IVUS image P1 and reconstructs the OFDI image P2, and an output layer 82c that outputs the generated OFDI image P2. The second generator 82 can be a GAN such as pix2pix, CycleGAN, or StarGAN, a VAE (Variational Autoencoder), or a neural network such as a CNN (e.g., U-net).

[0063] The second generator 82 is generated by preparing training data that associates an IVUS image P1, which is a normal image for training, with an OFDI image P2, which is also a normal image for training, and then training an untrained learning model using this training data. It is desirable that the IVUS image P1 and OFDI image P2 for training include images of lesions that are judged to require endovascular treatment. The image processing device 3 of this embodiment performs training using the IVUS image P1 and OFDI image P2 prepared for training and generates a second generator 82 that generates an OFDI image P2 from an IVUS image P1.

[0064] <Image Processing Procedure> Figures 11 and 12 are flowcharts showing the image processing procedure according to this embodiment, and Figure 13 is a conceptual diagram showing the correction method for IVUS image P1 and OFDI image P2. The ultrasonic line data generation unit 33 and optical line data generation unit 34 of the image processing apparatus 3 generate multiple ultrasonic line data and optical line data based on the reflected wave data output from the ultrasonic transmitting / receiving unit 12a and the interference light data obtained by the interference of reflected light and reference light output from the optical transmitting / receiving unit 12b (step S111). The multiple ultrasonic line data and optical line data are assigned line numbers, for example, in chronological order of observation time. The line number corresponds to the observation time. In other words, the line number corresponds to the observation position and observation direction.

[0065] Next, the processing unit 31 constructs multiple frames of IVUS images P1 based on ultrasound line data and multiple frames of OFDI images P2 based on optical line data (step S112). The multiple frames of IVUS images P1 and OFDI images P2 are assigned frame numbers, for example, in chronological order of observation time. The frame numbers correspond to the observation positions. The multiple frames of IVUS images P1 and OFDI images P2 correspond to images obtained by observing blood vessels at multiple observation positions from the tip to the proximal end of the probe 11. However, because the arrangement of the ultrasound transmitting / receiving unit 12a and the optical transmitting / receiving unit 12b and the direction of ultrasound or optical transmission and reception are different, the observation positions of IVUS images P1 and OFDI images P2 assigned the same frame number do not necessarily coincide. In other words, IVUS images P1 and OFDI images P2 are misaligned in the direction of the blood vessel's course. Also, for the same reason, IVUS images P1 and OFDI images P2 are misaligned in the circumferential direction of the blood vessel.

[0066] Furthermore, the processing unit 31, the ultrasound line data generation unit 33, and the optical line data generation unit 34, which execute the processes in steps S111 and S112, function as acquisition units that simultaneously acquire IVUS image P1 (ultrasound tomography image) and OFDI image P2 (optical coherence tomography image). The left side of Figure 13 shows multiple acquired IVUS images P1 and OFDI images P2.

[0067] Next, the processing unit 31 recognizes the lumen image in the IVUS image P1 by inputting the IVUS image P1 into the IVUS image recognition learning model 71 (step S113). The processing unit 31 performs image recognition processing for each of the multiple frames of the IVUS image P1. The object-extracted IVUS image P1' output from the IVUS image recognition learning model 71 is assigned the same frame number as the input IVUS image P1 or a similar frame number.

[0068] Furthermore, the processing unit 31 recognizes the lumen image in the OFDI image P2 by inputting the OFDI image P2 into the OFDI image recognition learning model 72 (step S114). The processing unit 31 performs image recognition processing for each of the multiple frames of the OFDI image P2. The object-extracted OFDI image P2' output from the OFDI image recognition learning model 72 is assigned the same frame number as the input OFDI image P2 or a similar frame number.

[0069] Although the object images that can be recognized by the IVUS image recognition learning model 71 and the OFDI image recognition learning model 72 are not necessarily the same, the processing unit 31 can recognize at least the lumen image in the IVUS image P1 and the lumen image in the OFDI image P2.

[0070] Next, the processing unit 31 determines the amount of translation and rotation of the 3D lumen image to match the position and orientation of the 3D lumen image based on the object-extracted IVUS image P1' with the 3D lumen image based on the object-extracted OFDI image P2' by rigid body registration (step S115). In other words, the processing unit 31 determines the correspondence between the frame numbers in which the lumen image in the IVUS image P1 of multiple frames matches the lumen image in the OFDI image P2 of multiple frames. The processing unit 31 also determines the amount of rotation of the IVUS image P1 or OFDI image P2 to match the orientation of the lumen image in the IVUS image P1 with the orientation of the lumen image in the OFDI image P2.

[0071] The object-extracted IVUS image P1' from multiple frames shows the lumen region at multiple observation positions along the direction of the blood vessel's course, and the processing unit 31 can construct a three-dimensional lumen image based on the images of the lumen region included in the object-extracted IVUS image P1' from multiple frames. Similarly, the object-extracted OFDI image P2' from multiple frames shows the lumen region at multiple observation positions along the direction of the blood vessel's course, and the processing unit 31 can construct a three-dimensional lumen image based on the images of the lumen region included in the object-extracted OFDI image P2' from multiple frames. The processing unit 31 then determines the amount of translation and rotation that matches the position and orientation of the three-dimensional lumen image by, for example, translating and rotating the three-dimensional lumen image based on the object-extracted IVUS image P1' in the circumferential direction using rigid body registration. Specifically, the processing unit 31 calculates the similarity or difference between two three-dimensional lumen images in a three-dimensional coordinate system and determines the amount of translation and rotation that maximizes the similarity or minimizes the difference. For similarity, for example, normalized cross-correlation can be used. For difference, for example, the sum of squared residuals can be used. The processing unit 31 should calculate the similarity or difference of the contours of the 3D lumen images.

[0072] Figure 14 is a conceptual diagram illustrating the method for correlating IVUS image P1 and OFDI image P2. The translation of the 3D lumen image by rigid body registration corresponds to correcting the correspondence between the frame number of IVUS image P1 and the frame image of OFDI image P2. The rotation of the 3D lumen image corresponds to the image rotation process in IVUS image P1.

[0073] The processing unit 31 corrects the correspondence between the IVUS image P1 and the OFDI image P2 (step S116). In other words, it associates the frame images of the IVUS image P1 and the OFDI image P2, which were captured at the same observation position. Specifically, it associates the frame numbers of the IVUS image P1 and OFDI image P2, which were captured at the same observation position. The amount of translation identified in step S115 corresponds to the difference between the frame numbers of the IVUS image P1 and OFDI image P2, which were captured at the same observation position. By adding or subtracting the number of frames corresponding to this difference, the IVUS image P1 and OFDI image P2 can be associated. The image processing unit 3 stores the correspondence between the amount of translation and the difference in frame numbers. Furthermore, since the amount of rotation identified in step S115 corresponds to the amount of misalignment in orientation between the IVUS image P1 and the OFDI image P2, that is, the amount of misalignment in rotation angle, the processing unit 31 can align the orientations of the IVUS image P1 and the OFDI image P2 by rotating the IVUS image P1 by the amount of misalignment.

[0074] Although the correction of observation position and direction using rigid body registration of 3D lumen images has been described, the observation position, direction, and scale may also be corrected by directly comparing the object-extracted IVUS image P1' and the object-extracted OFDI image P2' without constructing a 3D lumen image.

[0075] Before translation, the IVUS image P1 of the nth frame and the OFDI image P2 of the nth frame are assumed to be associated. n is an integer greater than or equal to 1. The processing unit 31 calculates the similarity or difference between the lumen region in the object-extracted IVUS image P1' and the lumen region in the object-extracted OFDI image P2' of the corresponding frame number. The processing unit 31 calculates the similarity or difference of the lumen regions for a predetermined number of object-extracted IVUS images P1' and object-extracted OFDI images P2'. Next, the processing unit 31 similarly calculates the similarity or difference between the lumen region in the object-extracted IVUS image P1' of the (n+α)th frame and the lumen region in the object-extracted OFDI image P2' of the nth frame. α is an integer and corresponds to the amount of translation of the 3D lumen image. The processing unit 31 verifies the correspondence between the object-extracted IVUS image P1' and the object-extracted OFDI image P2' while incrementing the variable α by 1. Furthermore, when comparing the lumen regions, the processing unit 31 rotates the orientation of the lumen region in the object-extracted IVUS image P1' in the circumferential direction by an angle correction amount θ. The angle correction amount θ corresponds to the amount of rotation of the 3D lumen image. Note that the rotation center is not the center of the object-extracted IVUS image P1', but rather the center of the sensor unit 12. The processing unit 31 calculates the similarity or difference of the lumen regions in the object-extracted IVUS image P1' and the object-extracted OFDI image P2' while changing the angle correction amount θ by predetermined amounts along with the variable α, and identifies the variable α and angle correction amount θ that maximize the similarity or minimize the difference. The processing unit 31 then stores the identified variable α and angle correction amount θ.

[0076] After completing the processing in step S116, the processing unit 31 inputs the multiple IVUS images P1 acquired in steps S111 and S112 into the IVUS fault detection learning model 61 to determine whether each IVUS image P1 is a faulty image (step S117). Similarly, the processing unit 31 inputs the multiple OFDI images P2 acquired in steps S111 and S112 into the OFDI fault detection learning model 62 to determine whether each OFDI image P2 is a faulty image (step S118). In the central diagram of Figure 13, the frames enclosed in thick borders represent the frames of IVUS image P1 and OFDI image P2, which were determined to be defective images.

[0077] The processing unit 31 determines whether or not there is an image defect in the IVUS image P1 based on the detection result in step S117 (step S119). The processing unit 31 that performs the processing in step S119 functions as a determination unit that determines whether or not there is an image defect in the acquired IVUS image P1. If the processing unit 31 determines that there is a defective IVUS image P1 (step S119: YES), it identifies a normal OFDI image P2 corresponding to the defective IVUS image P1 and inputs the corresponding normal OFDI image P2 to the first generator 81 to generate and output a corresponding normal IVUS image P1 (step S120). The OFDI image P2 corresponding to the defective IVUS image P1 is an OFDI image P2 obtained at approximately the same observation position and direction as the defective IVUS image P1. In other words, it is an OFDI image P2 that corresponds to the IVUS image P1 corrected in step S116. The processing unit 31 corrects the defective IVUS image P1 using the IVUS image P1 output from the first generator 81 (step S121). For example, as shown in the right figure of Figure 13, the processing unit 31 corrects the defective IVUS image P1 by replacing it with the generated IVUS image P1. The processing unit 31 may be configured to correct the image using any known image correction method, such as combining the IVUS image P1 determined to be defective with the generated IVUS image P1. If there are multiple IVUS images P1 determined to be defective, the processing unit 31 performs steps S120 and S121 for each of the multiple IVUS images P1 to correct the IVUS image P1.

[0078] If the processing in step S121 is completed, or if it is determined in step S119 that there are no defective IVUS images P1 (step S119: NO), the processing unit 31 determines whether or not there are defects in the OFDI image P2 based on the detection result in step S118 (step S122). The processing unit 31 that executes the processing in step S122 functions as a determination unit that determines whether or not there are defects in the acquired OFDI image P2. If the processing unit 31 determines that there is a defective OFDI image P2 (step S122: YES), it identifies a normal IVUS image P1 corresponding to the defective OFDI image P2 and inputs the corresponding normal IVUS image P1 to the second generator 82 to generate and output a corresponding normal OFDI image P2 (step S123). The IVUS image P1 corresponding to the defective OFDI image P2 is an IVUS image P1 obtained at approximately the same observation position and direction as the defective OFDI image P2. In other words, it is an IVUS image P1 that has been corrected in step S116 and is in a corresponding relationship with the OFDI image P2. The processing unit 31 corrects the defective OFDI image P2 using the OFDI image P2 output from the second generator 82 (step S124). For example, as shown in the right-hand figure of Figure 13, the processing unit 31 corrects the OFDI image P2 that has been determined to be defective by replacing it with a generated OFDI image P2. The processing unit 31 may also be configured to correct the image using any known image correction method, such as combining the OFDI image P2 that has been determined to be defective with the generated OFDI image P2. If there are multiple OFDI images P2 that have been determined to be defective, the processing unit 31 performs steps S123 and S124 for each of the multiple OFDI images P2 to correct the OFDI image P2.

[0079] If the processing in step S124 is completed, or if it is determined in step S122 that there are no defective OFDI images P2 (step S122: NO), the processing unit 31 outputs a medical image signal to the display device 4 and displays the corrected IVUS image P1 and OFDI image P2 on the display device 4 (step S125).

[0080] The processing unit 31 displays the first frame image F1 on the frame of the corrected IVUS image P1 (step S126) and displays the second frame image F2 on the frame of the corrected OFDI image P2 (step S127). The first frame image F1 and the second frame image F2 may be configured to have different colors, line types, line thicknesses, etc.

[0081] Figure 15 shows a first example of screen display of IVUS image P1 and OFDI image P2. The processing unit 31 displays, for example, IVUS image P1 and OFDI image P2, which are cross-sectional images approximately perpendicular to the long axis of the blood vessel, and a longitudinal section image approximately parallel to the central axis of the blood vessel, side by side. In the example shown in Figure 15, IVUS image P1 and OFDI image P2 for the distal reference area (Ref. Distal), proximal reference area (Ref. Proximal), and lesion area (MLA: minimum lumen area) are displayed. As described above, the corrected IVUS image P1 is surrounded by the first frame image F1, and the corrected OFDI image P2 is surrounded by the second frame image F2. The user can recognize the corrected IVUS image P1 and OFDI image P2 from the displayed IVUS image P1 and OFDI image P2 by the first frame image F1 and the second frame image F2.

[0082] Figure 16 shows a second example of screen display of IVUS image P1 and OFDI image P2. The processing unit 31 displays, for example, IVUS image P1 and OFDI image P2 and a longitudinal section image approximately parallel to the central axis of the blood vessel side by side. In the example shown in Figure 16, the processing unit 31 displays both the IVUS image P1, which is the uncorrected defective image, and the corrected IVUS image P1. Similarly, the processing unit 31 displays both the OFDI image P2, which is the uncorrected defective image, and the corrected OFDI image P2. The processing unit 31 may also be configured to switch between displaying the uncorrected defective IVUS image P1 and OFDI image P2 and the corrected IVUS image P1 and OFDI image P2 when a display switching operation is received from the input device 5. Furthermore, the processing unit 31 may be configured to display the uncorrected IVUS image P1 or OFDI image P2 when it receives a predetermined scan from the input device 5, for example, when the corrected IVUS image P1 or OFDI image P2 is clicked.

[0083] As described above, the image processing device 3, etc., according to this embodiment can correct defects in the IVUS image P1 and OFDI image P2 obtained using a dual-type catheter equipped with IVUS and OFDI functions.

[0084] Furthermore, even if multiple IVUS images P1 and OFDI images P2 are detected, these multiple IVUS images P1 and OFDI images P2 can be corrected.

[0085] If a normal OFDI image P2 exists that corresponds to a defective IVUS image P1, the IVUS image P1 can be corrected using the first generator 81. Similarly, if there is a normal IVUS image P1 corresponding to the defective OFDI image P2, the OFDI image P2 can be corrected using the second generator 82.

[0086] This configuration corrects IVUS image P1 and OFDI image P2 after aligning the observation position and direction of IVUS image P1 and OFDI image P2 obtained using a dual-type catheter equipped with both IVUS and OFDI functions. Therefore, it is possible to correct poorly formatted IVUS image P1 and OFDI image P2 more accurately.

[0087] Furthermore, as shown in Figure 15, the processing unit 31 displays the first frame image F1 on the corrected IVUS image P1, allowing the user to recognize whether the IVUS image P1 displayed on the display device 4 is a corrected image. Similarly, the processing unit 31 displays the second frame image F2 on the corrected OFDI image P2, allowing the user to recognize whether the OFDI image P2 displayed on the display device 4 is a corrected image.

[0088] Furthermore, as shown in Figure 16, the processing unit 31 can display the uncorrected IVUS image P1 along with the corrected IVUS image P1. Similarly, it can display the uncorrected OFDI image P2 along with the corrected OFDI image P2. The user can more accurately recognize the state of the blood vessels using the uncorrected IVUS image P1 and OFDI image P2.

[0089] In this embodiment, an example of detecting image defects in IVUS image P1 and OFDI image P2 was described using the IVUS defect detection learning model 61 and OFDI defect detection learning model 62, which are learning models. However, the system may be configured to detect image defects using rule-based processing. For example, the processing unit 31 may be configured to detect image defects by detecting the presence or absence of a typical pattern image in the IVUS image P1, such as the presence or absence of a double-ring image, through processing such as pattern matching. Alternatively, the processing unit 31 may be configured to detect image defects in OFDI image P2 by comparing the average brightness value of OFDI image P2 with a predetermined threshold. More specifically, OFDI image P2 with an average brightness value less than the predetermined threshold may be detected as a defective image.

[0090] Furthermore, in this embodiment, an example of recognizing the lumen image in IVUS image P1 and OFDI image P2 was described using the IVUS image recognition learning model 71 and the OFDI image recognition learning model 72. However, the contour of the lumen image may be extracted by image processing such as binarization, edge detection, and pattern matching.

[0091] Furthermore, while we have explained how to correct the observation position and direction of the IVUS image P1 and OFDI image P2, you may also perform radial scale correction. Furthermore, although an example of matching IVUS image P1 and OFDI image P2 on a frame-by-frame basis has been described, it is also possible to configure the system to perform corrections to match the observation position and observation direction in the ultrasound line data and optical line database, and then construct IVUS image P1 and OFDI image P2 after matching each line data. [Explanation of Symbols]

[0092] 1. Diagnostic imaging catheter 2 MDU 3 Image Processing Device 4 Display device 5 Input devices 11 probes 12 Sensor section 12a Ultrasonic Transceiver Unit 12b Optical Transceiver Unit 12c Housing 13 shafts 14 Guide wire insertion section 15 Connector section 30 Recording media 31 Processing Unit 32 Storage section 33 Ultrasonic line data generation unit 34 Optical line data generation unit 61 IVUS Fault Detection Learning Model 62 OFDI fault detection learning models 71 IVUS Image Recognition Learning Model 72 OFDI Image Recognition Learning Models 81 1st generator 82 Second generator P1 IVUS image P1' Object Extraction IVUS Image P2 OFDI image P2' Object Extraction OFDI Image 100 Imaging diagnostic equipment 101 Intravascular Examination Device 102 Angiography equipment

Claims

1. Using a sensor unit having an ultrasonic transmitting / receiving unit that transmits and receives ultrasound and an optical transmitting / receiving unit that transmits and receives light, ultrasound tomography images and optical coherence tomography images of tubular organs are acquired simultaneously. The acquired ultrasound tomography image or optical coherence tomography image is then analyzed for image defects. When an optical coherence tomography (or ultrasound) image is input to a trained model that has been trained to output a corresponding ultrasound image (or optical coherence tomography) image, a normal optical coherence tomography image (or ultrasound image) corresponding to the ultrasound image (or optical coherence tomography) image determined to be defective is input to the trained model, causing it to output a corresponding normal ultrasound image (or optical coherence tomography) image. The ultrasound tomography image (or optical coherence tomography image) output from the learning model is used to correct the ultrasound tomography image (or optical coherence tomography image) that has been determined to be defective. A computer program that causes a computer to perform a process.

2. While moving the sensor unit along the direction of travel of the tubular organ, multiple ultrasound tomography images and multiple optical coherence tomography images are acquired. The system determines image defects in each of the multiple ultrasound tomography images and optical coherence tomography images, and corrects the ultrasound tomography images and optical coherence tomography images that have been determined to be defective. The computer program according to claim 1.

3. The aforementioned learning model, A first learning model that has been trained to output the corresponding ultrasonic tomography image when the optical coherence tomography image is input, A second trained model that has been trained to output the corresponding optical coherence tomography image when the aforementioned ultrasound tomography image is input, Includes, The first learning model is made to output a corresponding normal ultrasound image by inputting the optical coherence tomography image corresponding to the ultrasound image determined to be defective, and the second learning model is made to output a corresponding normal ultrasound image by inputting the ultrasound image corresponding to the optical coherence tomography image determined to be defective. The ultrasound tomography images and optical coherence tomography images output from the first and second learning models are used to correct the ultrasound tomography images and optical coherence tomography images that were determined to be defective. The computer program according to claim 2.

4. By comparing a first object image included in a plurality of ultrasound tomography images with a second object image included in a plurality of optical coherence tomography images, the optical coherence tomography image (or ultrasound tomography image) that is a tomographic image of substantially the same observation position as the ultrasound tomography image (or optical coherence tomography image) that was determined to be a defective image is identified. By inputting the identified optical coherence tomography (or ultrasound tomography) image into the learning model, the ultrasound tomography (or optical coherence tomography) image is output. A computer program according to any one of claims 1 to 3.

5. The presence or absence of correction to the aforementioned ultrasonic tomography image or optical coherence tomography image is displayed as an image. A computer program according to any one of claims 1 to 3.

6. Display both the uncorrected ultrasound tomography image or optical coherence tomography image and the corrected ultrasound tomography image or optical coherence tomography image. A computer program according to any one of claims 1 to 3.

7. A method for operating an image processing device, The aforementioned image processing device is A sensor unit having an ultrasonic transmitting / receiving unit that transmits and receives ultrasound and an optical transmitting / receiving unit that transmits and receives light simultaneously acquires ultrasound tomography and optical coherence tomography images of tubular organs, The acquired ultrasound tomography image or optical coherence tomography image is then analyzed for image defects. When an optical coherence tomography (or ultrasound) image is input to a trained model that has been trained to output a corresponding ultrasound image (or optical coherence tomography) image, a normal optical coherence tomography image (or ultrasound image) corresponding to the ultrasound image (or optical coherence tomography) image determined to be defective is input to the trained model, causing it to output a corresponding normal ultrasound image (or optical coherence tomography) image. The ultrasound tomography image (or optical coherence tomography image) output from the learning model is used to correct the ultrasound tomography image (or optical coherence tomography image) that has been determined to be defective. How the image processing device operates.

8. An acquisition unit that simultaneously acquires ultrasound tomography and optical coherence tomography images of a tubular organ using a sensor unit having an ultrasound transmitting / receiving unit for transmitting and receiving ultrasound and an optical transmitting / receiving unit for transmitting and receiving light, A determination unit for determining image defects in the ultrasonic tomography image or optical coherence tomography image acquired by the acquisition unit, A trained model that, when given the aforementioned optical coherence tomography image (or the aforementioned ultrasound tomography image), outputs the corresponding ultrasound tomography image (or optical coherence tomography image), A processing unit that inputs a normal optical coherence tomography (or optical coherence tomography) image (or optical coherence tomography) image corresponding to the ultrasound tomography image (or optical coherence tomography) image determined to be defective into the learning model, outputs the corresponding normal ultrasound tomography image (or optical coherence tomography) image, and corrects the ultrasound tomography image (or optical coherence tomography) image determined to be defective using the ultrasound tomography image (or optical coherence tomography) image output from the learning model. An image processing device equipped with the following features.

Images