Computer program, information processing method, and information processing apparatus

Abstract

To provide a computer program, an information processing method, and an information processing device.SOLUTION: A computer program causes a computer to: acquire a tomographic image of a hollow organ which is acquired from a diagnostic imaging catheter; estimate a stent indwelling section on the basis of the acquired tomographic image; determine a risk of the diagnostic imaging catheter getting stuck on the basis of the tomographic image; and output information on the risk in the estimated indwelling section.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a computer program, an information processing method, and an information processing apparatus.

Background Art

[0002] One of the treatment methods for occlusions in the coronary artery is a method called percutaneous coronary intervention (PCI). PCI is a minimally invasive treatment that dilates an occluded lesion with a balloon catheter and implants a stent to reconstruct the blood vessel.

[0003] During or after PCI, the state of the blood vessel can be observed by intravascular ultrasound diagnosis (IVUS) or optical coherence tomography (OCT) using an imaging diagnostic catheter. Also, by angiography, which takes pictures of the blood vessel using X-rays while injecting a contrast agent into the blood vessel, the course of the blood vessel leading to the occlusion can be observed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The shaft of an imaging catheter is equipped with a GW lumen for inserting a guidewire. In mechanical imaging catheters where the sensor rotates to image the vascular condition around the entire circumference, the GW lumen is located towards the tip of the sensor. When observing the vascular condition after stent placement using IVUS or OCT, it is necessary to advance the GW lumen beyond the tip of the placed stent towards the distal side of the vessel. When moving the advanced imaging catheter in the direction of withdrawal from the vessel, there is a risk that the proximal opening of the GW lumen may catch on the stent strut, causing a jam.

[0006] One aspect of this is the objective to provide a computer program, information processing method, and information processing device that can alert the user when there is a risk of a program getting stuck. [Means for solving the problem]

[0007] (1) A computer program relating to one aspect is a computer program that causes a computer to perform the following processes: acquire tomographic images of a tubular organ obtained from an imaging catheter, estimate the stent placement section based on the acquired tomographic images, determine the risk of the imaging catheter getting stuck based on the tomographic images, and output information on the risk in the estimated placement section.

[0008] (2) In the computer program described in (1) above, it is preferable to identify the positional relationship between the diagnostic imaging catheter and the tubular organ, or the positional relationship between the guide wire that guides the diagnostic imaging catheter and the tubular organ, based on the tomographic image, and to determine the risk of the diagnostic imaging catheter getting stuck based on the identified positional relationship.

[0009] (3) In the computer program described in (2) above, it is preferable to determine the positional relationship by calculating the distance between the diagnostic imaging catheter and the luminal wall of the tubular organ, or the distance between the guide wire and the luminal wall of the tubular organ, from the tomographic image.

[0010] (4) In the computer program described in (1) above, when a tomographic image of a tubular organ is input, it is preferable to input the acquired tomographic image to a learning model that has been trained to output information regarding the risk, perform calculations by the learning model, and determine the risk of the diagnostic imaging catheter getting stuck based on the calculation results by the learning model.

[0011] (5) In any one of the computer programs described in (1) to (4) above, it is preferable to calculate the outer diameter and inner diameter of the tubular organ from the tomographic image and to estimate the stent placement section based on the calculated outer diameter and inner diameter of the tubular organ.

[0012] (6) In the computer program described in (5) above, it is preferable to calculate the outer diameter based on the position of the external elastic plate in the lumen organ.

[0013] (7) In any one of the computer programs described in (1) to (6) above, the implantation section is preferably a section in which the implantation of the stent is predicted.

[0014] (8) In any one of the computer programs described in (1) to (6) above, the implantation section is preferably an implanted section in which the stent is implanted.

[0015] (9) It is preferable that any one of the computer programs described in (1) to (8) above highlights the location on the tomographic image where it is determined that there is a risk of the diagnostic imaging catheter getting stuck.

[0016] (10) In any one of the computer programs (1) to (9) above, it is preferable to obtain an angiogram imaged together with the tomographic image and highlight, on the angiogram, a location determined to have a risk of the imaging diagnostic catheter stacking.

[0017] (11) An information processing method according to one aspect includes obtaining a tomographic image of a luminal organ obtained from an imaging diagnostic catheter, estimating a stent placement interval based on the obtained tomographic image, determining, based on the tomographic image, a risk of the imaging diagnostic catheter stacking, and outputting information on the risk in the estimated placement interval.

[0018] (12) An information processing apparatus according to one aspect includes an acquisition unit that acquires a tomographic image of a luminal organ obtained from an imaging diagnostic catheter, an estimation unit that estimates a stent placement interval based on the acquired tomographic image, a determination unit that determines, based on the tomographic image, a risk of the imaging diagnostic catheter stacking, and an output unit that outputs information on the risk in the estimated placement interval.

Advantages of the Invention

[0019] In one aspect, it is possible to prompt the user to pay attention according to the level of the stacking risk.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram showing a configuration example of an imaging diagnostic apparatus in Embodiment 1. [Figure 2] It is a schematic diagram showing an overview of an imaging diagnostic catheter. [Figure 3] It is an explanatory diagram showing a cross-section of a blood vessel through which a sensor unit is inserted. [Figure 4] It is an explanatory diagram explaining a tomographic image. [Figure 5] It is a block diagram showing a configuration example of an image processing apparatus. [Figure 6] It is a schematic diagram showing a configuration example of a learning model. [Figure 7]It is a schematic diagram showing the recognition result by the learning model. [Figure 8] It is an explanatory diagram for explaining the outline of the process executed by the image processing apparatus according to Embodiment 1. [Figure 9] It is a flowchart for explaining the procedure of the process executed by the image processing apparatus according to Embodiment 1. [Figure 10] It is a schematic diagram showing an output example of an alert. [Figure 11] It is a flowchart for explaining the procedure of the process executed by the image processing apparatus according to Embodiment 2. [Figure 12] It is a schematic diagram showing a configuration example of the learning model used in Embodiment 3. [Figure 13] It is a flowchart for explaining the procedure of the process executed by the image processing apparatus according to Embodiment 3.

Mode for Carrying Out the Invention

[0021] Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments. (Embodiment 1) FIG. 1 is a schematic diagram showing a configuration example of the image diagnostic apparatus 100 in Embodiment 1. In the present embodiment, an image diagnostic apparatus using a dual-type catheter having both functions of intravascular ultrasound diagnosis (IVUS) and optical coherence tomography (OCT) will be described. In the dual-type catheter, there are provided a mode for acquiring an ultrasonic tomographic image only by IVUS, a mode for acquiring an optical coherence tomographic image only by OCT, and a mode for acquiring both tomographic images by IVUS and OCT, and these modes can be switched and used. Hereinafter, the ultrasonic tomographic image and the optical coherence tomographic image will also be referred to as an IVUS image and an OCT image, respectively. When there is no need to distinguish between the IVUS image and the OCT image, it will simply be referred to as a tomographic image.

[0022] The image diagnostic apparatus 100 according to this embodiment comprises an intravascular examination device 101, an angiography device 102, an image processing device 3, a display device 4, and an input device 5. The intravascular examination device 101 comprises an image diagnostic catheter 1 and an MDU (Motor Drive Unit) 2. The image diagnostic catheter 1 is connected to the image processing device 3 via the MDU 2. The display device 4 and the input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, mouse, touch panel, or microphone. The input device 5 and the image processing device 3 may be configured as a single unit. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input, etc.

[0023] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that uses X-rays to image blood vessels from outside the patient's body while injecting a contrast agent into the patient's blood vessels, thereby obtaining an angiographic image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and the X-ray sensor receives the X-rays irradiated from the X-ray source to image an X-ray fluoroscopic image of the patient. The diagnostic imaging catheter 1 is equipped with a marker that does not transmit X-rays, and the position of the diagnostic imaging catheter 1 (marker) is visualized in the angiographic image. The angiography device 102 outputs the acquired angiographic image to the image processing device 3, and the image processing device 3 displays the angiographic image on the display device 4. The display device 4 displays both the angiographic image and the tomographic image acquired using the diagnostic imaging catheter 1.

[0024] In this embodiment, an angiography device 102 that captures two-dimensional angiographic images is connected to the image processing device 3. However, the device is not limited to the angiography device 102, as long as it captures images of the patient's tubular organs and diagnostic imaging catheter 1 from multiple directions outside the body.

[0025] Figure 2 is a schematic diagram showing an overview of the imaging diagnostic catheter 1. Note that the upper dashed-dotted area in Figure 2 is an enlargement of the lower dashed-dotted area. The imaging diagnostic catheter 1 has a probe 11 and a connector portion 15 located at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector portion 15. In the following description, the side of the imaging diagnostic catheter 1 furthest from the connector portion 15 will be referred to as the tip side, and the side with the connector portion 15 will be referred to as the proximal end side. The probe 11 is equipped with a catheter sheath 11a, and its tip is provided with a guidewire insertion portion 14 through which a guidewire GW can be inserted. The guidewire insertion portion 14 constitutes a guidewire lumen and is used to receive the guidewire GW that has been previously inserted into the blood vessel, and to guide the probe 11 to the affected area by the guidewire GW. The catheter sheath 11a forms a continuous tube from the connection portion with the guidewire insertion portion 14 to the connection portion with the connector portion 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip of the shaft 13.

[0026] The sensor unit 12 has a housing 12d, and the tip end of the housing 12d is formed in a hemispherical shape to suppress friction and snagging with the inner surface of the catheter sheath 11a. Inside the housing 12d are an ultrasonic transmitting / receiving unit 12a (hereinafter referred to as IVUS sensor 12a) that transmits ultrasonic waves into the blood vessel and receives reflected waves from within the blood vessel, and an optical transmitting / receiving unit 12b (hereinafter referred to as OCT sensor 12b) that transmits near-infrared light into the blood vessel and receives reflected light from within the blood vessel. In the example shown in Figure 2, the IVUS sensor 12a is provided on the tip end of the probe 11, and the OCT sensor 12b is provided on the proximal end, and they are arranged at a distance x along the axial direction on the central axis of the shaft 13 (on the dashed line in Figure 2). In the diagnostic imaging catheter 1, the IVUS sensor 12a and the OCT sensor 12b are mounted so that the direction at which ultrasonic waves or near-infrared light are transmitted and received is approximately 90 degrees with respect to the axial direction of the shaft 13 (the radial direction of the shaft 13). Furthermore, it is desirable that the IVUS sensor 12a and OCT sensor 12b be mounted slightly offset from the radial direction so as not to receive reflected waves or reflected light on the inner surface of the catheter sheath 11a. In this embodiment, for example, as indicated by the arrow in Figure 2, the IVUS sensor 12a is mounted with the direction inclined toward the proximal end with respect to the radial direction as the direction of ultrasonic irradiation, and the OCT sensor 12b is mounted with the direction inclined toward the tip with respect to the radial direction as the direction of near-infrared light irradiation.

[0027] The shaft 13 contains an electrical signal cable (not shown) connected to the IVUS sensor 12a and an optical fiber cable (not shown) connected to the OCT sensor 12b. The probe 11 is inserted into the blood vessel from the tip end. The sensor unit 12 and shaft 13 can move forward and backward inside the catheter sheath 11a and can also rotate circumferentially. The sensor unit 12 and shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation. The imaging diagnostic device 100 uses an imaging core composed of the sensor unit 12 and shaft 13 to measure the condition inside the blood vessel by taking ultrasound images (IVUS images) or optical coherence tomography images (OCT images) taken from inside the blood vessel.

[0028] The MDU2 is a drive device to which the probe 11 (imaging catheter 1) is detachably attached by a connector 15. It controls the operation of the imaging catheter 1 inserted into a blood vessel by driving a built-in motor in response to the operation of a medical professional. For example, the MDU2 performs a pullback operation, rotating the sensor unit 12 and shaft 13 inserted into the probe 11 circumferentially while pulling them toward the MDU2 at a constant speed. The sensor unit 12 rotates while moving from the tip to the proximal end due to the pullback operation, and continuously scans the inside of the blood vessel at predetermined time intervals, thereby continuously acquiring multiple transverse images approximately perpendicular to the probe 11 at predetermined intervals. The MDU2 outputs the reflected ultrasound wave data received by the IVUS sensor 12a and the reflected light data received by the OCT sensor 12b to the image processing device 3.

[0029] The image processing device 3 acquires a signal dataset, which is the reflected ultrasound wave data received by the IVUS sensor 12a via the MDU 2, and a signal dataset, which is the reflected light data received by the OCT sensor 12b. The image processing device 3 generates ultrasound line data from the ultrasound signal dataset and constructs an ultrasound tomographic image (IVUS image) of the transverse layer of the blood vessel based on the generated ultrasound line data. The image processing device 3 also generates optical line data from the reflected light signal dataset and constructs an optical coherence tomographic image (OCT image) of the transverse layer of the blood vessel based on the generated optical line data. Here, the signal datasets acquired by the IVUS sensor 12a and the OCT sensor 12b, and the tomographic images constructed from the signal datasets will be explained.

[0030] Figure 3 is an explanatory diagram showing a cross-section of a blood vessel through which the sensor unit 12 is inserted, and Figure 4 is an explanatory diagram explaining the tomographic image. First, using Figure 3, the operation of the IVUS sensor 12a and OCT sensor 12b inside the blood vessel and the signal dataset (ultrasound line data and optical line data) acquired by the IVUS sensor 12a and OCT sensor 12b will be explained. When tomographic imaging is started with the imaging core inserted inside the blood vessel, the imaging core rotates around the central axis of the shaft 13 in the direction indicated by the arrow. At this time, the IVUS sensor 12a transmits and receives ultrasound at each rotation angle. Lines 1, 2, ... 512 indicate the direction of transmission and reception of ultrasound at each rotation angle. In this embodiment, the IVUS sensor 12a intermittently transmits and receives ultrasound 512 times while rotating 360 degrees (1 rotation) inside the blood vessel. The IVUS sensor 12a acquires data for one line in the transmission / reception direction with each ultrasonic transmission and reception, so 512 ultrasonic line data points extending radially from the center of rotation can be obtained during one rotation. The 512 ultrasonic line data points are dense near the center of rotation, but become sparser as they move away from the center of rotation. Therefore, the image processing device 3 can generate a two-dimensional ultrasonic tomographic image (IVUS image) as shown in Figure 4A by generating pixels in the empty spaces between each line using a well-known interpolation process.

[0031] Similarly, the OCT sensor 12b also transmits and receives measurement light at each rotation angle. Since the OCT sensor 12b also transmits and receives measurement light 512 times while rotating 360 degrees within the blood vessel, 512 optical line data points extending radially from the center of rotation can be obtained during one rotation. With respect to the optical line data, the image processing device 3 can generate a two-dimensional optical coherence tomography (OCT) image similar to the IVUS image shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process. That is, the image processing device 3 generates optical line data based on interference light generated by interfering reflected light with, for example, reference light obtained by separating light from a light source within the image processing device 3, and constructs an optical coherence tomography (OCT) image of the transverse layer of the blood vessel based on the generated optical line data.

[0032] The two-dimensional tomographic image generated from these 512 line data points is called one frame of an IVUS image or OCT image. Since the sensor unit 12 scans while moving within the blood vessel, one frame of an IVUS image or OCT image is acquired at each position after one rotation within the movement range. That is, one frame of an IVUS image or OCT image is acquired at each position from the tip to the proximal end of the probe 11 within the movement range, so as shown in Figure 4B, multiple frames of IVUS images or OCT images are acquired within the movement range.

[0033] The diagnostic imaging catheter 1 has X-ray-insensitive markers to confirm the positional relationship between the IVUS image obtained by the IVUS sensor 12a or the OCT image obtained by the OCT sensor 12b and the angiography image obtained by the angiography device 102. In the example shown in Figure 2, marker 14a is provided at the tip of the catheter sheath 11a, for example, at the guidewire insertion section 14, and marker 12c is provided on the shaft 13 side of the sensor section 12. When the diagnostic imaging catheter 1 configured in this way is imaged with X-rays, an angiography image is obtained in which markers 14a and 12c are visualized. The positions of markers 14a and 12c are just examples; marker 12c may be provided on the shaft 13 instead of the sensor section 12, and marker 14a may be provided at a location other than the tip of the catheter sheath 11a.

[0034] Figure 5 is a block diagram showing an example configuration of the image processing device 3. The image processing device 3 is a computer (information processing device) and comprises a control unit 31, a main memory unit 32, an input / output unit 33, a communication unit 34, an auxiliary memory unit 35, and a reading unit 36. The image processing device 3 is not limited to a single computer, but may be a multi-computer composed of multiple computers. Furthermore, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer.

[0035] The control unit 31 is configured using one or more computing devices such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), GPGPU (General purpose computing on graphics processing units), and TPU (Tensor Processing Unit). The control unit 31 is connected to each hardware component of the image processing device 3 via a bus.

[0036] The main memory unit 32 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores the data necessary for the control unit 31 to perform calculation processing.

[0037] The input / output unit 33 is equipped with an interface for connecting external devices such as an intravascular examination device 101, angiography device 102, display device 4, and input device 5. The control unit 31 acquires IVUS images and OCT images from the intravascular examination device 101 and angiography images from the angiography device 102 via the input / output unit 33. The control unit 31 also outputs medical image signals of IVUS images, OCT images, or angiography images to the display device 4 via the input / output unit 33, thereby displaying the medical images on the display device 4. Furthermore, the control unit 31 receives information input to the input device 5 via the input / output unit 33.

[0038] The communication unit 34 is equipped with a communication interface compliant with communication standards such as 4G, 5G, and WiFi. The image processing device 3 communicates with an external server, such as a cloud server connected to an external network such as the Internet, via the communication unit 34. The control unit 31 may access the external server via the communication unit 34 and refer to various data stored in the storage of the external server. The control unit 31 may also collaborate with the external server to perform the processing in this embodiment, for example, by performing inter-process communication.

[0039] The auxiliary storage unit 35 is a storage device such as a hard disk or SSD (Solid State Drive). The auxiliary storage unit 35 stores computer programs executed by the control unit 31 and various data necessary for the processing of the control unit 31. The auxiliary storage unit 35 may also be an external storage device connected to the image processing device 3. The computer programs executed by the control unit 31 may be written to the auxiliary storage unit 35 during the manufacturing stage of the image processing device 3, or the image processing device 3 may acquire programs distributed by a remote server device via communication and store them in the auxiliary storage unit 35. The computer programs may be recorded in a readable form on a recording medium RM such as a magnetic disk, optical disk, or semiconductor memory, and the reading unit 36 ​​may read them from the recording medium RM and store them in the auxiliary storage unit 35. An example of a computer program stored in the auxiliary storage unit 35 is a risk determination program PG that determines the risk of the diagnostic imaging catheter 1 getting stuck, and if it is determined that there is a risk of getting stuck in the stent placement section, it causes the computer to execute a process to output information to warn the user.

[0040] Furthermore, various learning models may be stored in the auxiliary storage unit 35. A learning model is described by its definition information. The definition information of a learning model includes information about the layers that make up the learning model, information about the nodes that make up each layer, and internal parameters such as weight coefficients and biases between nodes. The internal parameters are learned by a predetermined learning algorithm. The auxiliary storage unit 35 stores the definition information of the learning model, including the learned internal parameters. An example of a learning model stored in the auxiliary storage unit 35 is the learning model MD1, which is used for processing to recognize vascular regions, lumen regions, guidewire regions, and catheter regions from tomographic images.

[0041] Figure 6 is a schematic diagram showing an example of the configuration of the learning model MD1. The learning model MD1 is a learning model for image segmentation and is constructed using a neural network with convolutional layers, such as SegNet. Alternatively, the learning model MD1 may be any learning model for image segmentation, not limited to SegNet, such as FCN (Fully Convolutional Network), U-Net (U-Shaped Network), or PSPNet (Pyramid Scene Parsing Network). Furthermore, the learning model MD1 may be any learning model for object detection, such as YOLO (You Only Look Once), SSD (Single Shot Multi-Box Detector), or ViT (Vision Transformer).

[0042] In this embodiment, the input image to the learning model MD1 is a tomographic image (IVUS image or OCT image) of a blood vessel obtained from the intravascular examination device 101. The learning model MD1 is trained to output information indicating the recognition results of the blood vessel region, lumen region, guidewire region, and catheter region included in the tomographic image. Here, the blood vessel region is the region in the tomographic image where a blood vessel exists. In IVUS images, the outer surface of the blood vessel is often unclear. For this reason, the region surrounded by the external elastic membrane (EEM) may be recognized as the blood vessel region instead. The lumen region is the region in the tomographic image surrounded by the inner wall of the blood vessel. The guidewire region is the region in the tomographic image where a guidewire GW inserted into the blood vessel exists. In IVUS images, the guidewire region appears as a relatively small, high-brightness region. The catheter region is the region in the tomographic image where the sensor part 12 of the diagnostic imaging catheter 1 inserted into the blood vessel exists. In IVUS images, the catheter area appears as a black circular region near the center of the image.

[0043] The learning model MD1 comprises, for example, an encoder EN, a decoder DE, and a softmax layer SM. The encoder EN is constructed by alternating convolutional layers and pooling layers. The convolutional layers are multilayered into 2 to 3 layers.

[0044] In a convolutional layer, the input data is convolved with a filter of a predetermined size (e.g., 3x3 or 5x5). Specifically, the input value at the position corresponding to each element of the filter is multiplied by a weight coefficient pre-set for the filter, and a linear sum of these element-wise multiplications is calculated. The output of the convolutional layer is obtained by adding a set bias to the calculated linear sum. The result of the convolutional operation may be transformed by an activation function. For example, ReLU (Rectified Linear Unit) can be used as an activation function. The output of the convolutional layer represents a feature map in which the features of the input data have been extracted.

[0045] The pooling layer calculates local statistics of the feature map output from the convolutional layer, which is a higher layer connected to the input. Specifically, a window of a predetermined size (e.g., 2x2, 3x3) corresponding to the position of the higher layer is set, and local statistics are calculated from the input values ​​within the window. For example, the maximum value can be used as the statistics. The size of the feature map output from the pooling layer is reduced (downsampled) according to the size of the window. The example in Figure 6 shows that in encoder EN, the calculations in the convolutional layer and the pooling layer are sequentially repeated to sequentially downsample a 224x224 pixel input image to 112x112, 56x56, 28x28, ..., 1x1 feature maps.

[0046] The output of encoder EN is input to decoder DE. Decoder DE is constructed by alternating inverse convolutional layers and inverse pooling layers. The inverse convolutional layers are stacked in two to three layers.

[0047] In the deconvolution layer, a deconvolution operation is performed on the input feature map. The deconvolution operation is an operation that reconstructs the feature map before the convolution operation, based on the assumption that the input feature map is the result of a convolution operation using a specific filter. In this operation, when the specific filter is represented by a matrix, the output feature map is generated by calculating the product of the transpose of this matrix and the input feature map. The result of the deconvolution layer operation may also be transformed by an activation function such as ReLU, as described above.

[0048] The inverse pooling layer of the decoder DE is individually mapped one-to-one with the pooling layer of the encoder EN, and the mapped pairs have substantially the same size. The inverse pooling layer expands (upsamples) the size of the feature map that has been downsampled in the pooling layer of the encoder EN. The example in Figure 6 shows that the decoder DE sequentially upsamples to 1×1, 7×7, 14×14, ..., 224×224 feature maps by sequentially repeating the operations in the convolutional layer and the pooling layer.

[0049] The output of decoder DE is input to softmax layer SM. The softmax layer SM outputs the probability of a label identifying a region at each position (pixel) by applying a softmax function to the input value from the inverse convolutional layer connected to the input side. In this embodiment, labels identifying vascular regions, lumen regions, guidewire regions, and catheter regions are set. The control unit 31 of the image processing device 3 recognizes vascular regions, lumen regions, guidewire regions, and catheter regions by referring to the probability of the labels output from softmax layer SM.

[0050] Figure 7 is a schematic diagram showing the recognition results by the learning model MD1. The control unit 31 of the image processing device 3 inputs the tomographic image obtained from the intravascular examination device 101 to the learning model MD1 and performs calculations by the learning model MD1 to obtain the recognition results shown in Figure 7. In Figure 7, the area enclosed by boundary line B1 is the vascular region, the area enclosed by boundary line B2 is the lumen region, the area enclosed by boundary line B3 is the guidewire region, and the area enclosed by boundary line B4 is the catheter region. In image segmentation, the area between boundary line B1 and boundary line B2 is recognized as the vascular region, and the area between boundary line B2 and boundary line B4 is recognized as the lumen region. In addition, the areas inside boundary line B3 and boundary line B4 are recognized as the guidewire region and catheter region, respectively. Note that if the tomographic image is an IVUS image, boundary line B1 indicates the location where the external elastic lamina is present.

[0051] The vessel diameter shown in Figure 7 is the diameter of the vessel region, and the lumen diameter is the diameter of the lumen region. The control unit 31 can calculate the actual dimensions of the vessel diameter and lumen diameter based on the scale of the tomographic image. For example, since IVUS images are generated based on dimensional information measured by ultrasound, the number of millimeters corresponding to one pixel is known information in the diagnostic imaging device 100. The control unit 31 can calculate the actual dimensions of the vessel diameter and lumen diameter based on the known dimensional information in the diagnostic imaging device 100. Also, if the tomographic image is in DICOM format, the metadata includes scale information, so the actual dimensions of the vessel diameter and lumen diameter may be calculated by referring to the scale information included in the metadata. On the other hand, if the tomographic image is in an image format that does not have metadata, if the dimensions of the sensor part 12 (i.e., the catheter region) shown in the tomographic image are known, it is possible to calculate the actual dimensions of the vessel diameter and lumen diameter based on that dimensional information.

[0052] In Figure 7, L1 represents the distance between the guidewire GW and the lumen wall, and L2 represents the distance between the imaging catheter 1 and the lumen wall. Similarly, the control unit 31 can calculate the actual dimensions of distances L1 and L2 based on known dimensional information.

[0053] In Figure 7, for the sake of simplicity, the vascular region and the lumen region are depicted as circular regions, but in reality, these regions are rarely observed as perfectly circular regions. For this reason, the control unit 31 may scan circumferentially with respect to the center (or centroid) of each region and calculate the maximum diameter, minimum diameter, and average diameter for the vascular diameter and lumen diameter, and may also calculate the maximum distance, minimum distance, and average distance for distances L1 and L2.

[0054] Figure 8 is an explanatory diagram illustrating the overview of the processing performed by the image processing device 3 according to Embodiment 1. The image processing device 3 according to Embodiment 1 acquires a tomographic image of the blood vessel before stent placement and estimates the section in which stent placement is predicted (hereinafter also referred to as the placement section) based on the acquired tomographic image. When lipid-rich structures called plaques are deposited in the wall of a blood vessel (coronary artery), the blood vessel narrows or becomes occluded, causing ischemic heart diseases such as angina pectoris and myocardial infarction. The image processing device 3 calculates the blood vessel diameter and lumen diameter based on the tomographic image obtained from the intravascular examination device 101, and estimates the placement section by identifying the narrowed or occluded area of ​​the blood vessel based on the calculated blood vessel diameter and lumen diameter. For example, if the ratio of the lumen diameter to the blood vessel diameter is less than a first threshold (e.g., less than 70%), it is determined that a narrowed area exists in the tomographic image. If tomographic images in which narrowed areas exist appear continuously along the long axis of the blood vessel, that section can be estimated as the stent placement section.

[0055] Furthermore, the image processing device 3 determines the risk of the diagnostic imaging catheter 1 getting stuck based on the acquired tomographic images. The closer the guidewire GW or the diagnostic imaging catheter 1 gets to the lumen wall, the higher the risk of the diagnostic imaging catheter 1 getting stuck. For example, if the distance L1 between the guidewire GW and the lumen wall, or the distance L2 between the diagnostic imaging catheter 1 and the lumen wall, is less than the second threshold (e.g., less than 0.5 mm), it can be determined that the risk of getting stuck is high.

[0056] The image processing device 3 outputs an alert to the operator if there are areas in the stent placement section that pose a high risk of stent stacking. Specifically, the image processing device 3 highlights the areas with a high risk of stacking on the longitudinal tomographic image. Alternatively, the corresponding areas on the angiography image may be highlighted.

[0057] The operation of the image processing device 3 will be described below. Figure 9 is a flowchart illustrating the processing procedure performed by the image processing device 3 according to Embodiment 1. In Embodiment 1, the stack risk is determined using tomographic images taken before the stent is placed. The intravascular examination device 101 scans the inside of the blood vessel while moving the sensor part 12 of the diagnostic imaging catheter 1 from the tip (distal) to the proximal end (proximal) to generate a series of tomographic images (transverse images). The control unit 31 of the image processing device 3 acquires the series of tomographic images generated by the intravascular examination device 101 through the input / output unit 33 (step S101).

[0058] The control unit 31 inputs each of the acquired tomographic images into the learning model MD1, performs calculations using the learning model MD1, and performs segmentation for each tomographic image (step S102). Through segmentation using the learning model MD1, the vascular region, lumen region, guidewire region, and catheter region are extracted from the tomographic image.

[0059] The control unit 31 calculates the vessel diameter and lumen diameter in each tomographic image based on the recognition results from the learning model MD1 (step S103). For example, the control unit 31 can calculate the actual dimensions of the vessel diameter and lumen diameter by referring to known dimensional information (information on how many millimeters one pixel corresponds to) in the diagnostic imaging device 100.

[0060] The control unit 31 estimates the section in which stent placement is predicted based on the vessel diameter and lumen diameter calculated in each tomographic image (step S104). For example, the control unit 31 calculates the ratio of the lumen diameter to the vessel diameter based on the vessel diameter and lumen diameter calculated from the tomographic image, and if the calculated ratio is less than a first threshold (e.g., less than 70%), it determines that a stenotic area exists in that tomographic image. If there is a section in which tomographic images containing a stenotic area appear continuously along the long axis of the vessel, the control unit 31 estimates that section as the stent placement section.

[0061] Based on the recognition results from the learning model MD1, the control unit 31 calculates the distance L1 between the guidewire GW and the luminal wall, and the distance L2 between the diagnostic imaging catheter 1 and the luminal wall for each tomographic image (step S105). For example, the control unit 31 can calculate the actual dimensions of distances L1 and L2 by referring to known dimensional information (information on how many millimeters one pixel corresponds to) in the diagnostic imaging device 100.

[0062] The control unit 31 determines the risk of the imaging catheter 1 getting stuck in each tomographic image based on the calculated distances L1 and L2 (step S106). For example, if the distance L1 between the guidewire GW and the luminal wall, or the distance L2 between the imaging catheter 1 and the luminal wall, is less than a second threshold (e.g., less than 0.5 mm), the control unit 31 can determine that there is a high risk of the imaging catheter 1 getting stuck in that tomographic image.

[0063] In this embodiment, a common threshold (second threshold) is set for distances L1 and L2, but individual thresholds may be set for each. Also, in this embodiment, the level of risk (presence or absence) is determined by comparison with the threshold, but multiple thresholds may be set to determine the risk level in stages.

[0064] In this embodiment, for convenience, the procedure is to estimate the stent placement section and then determine the risk of getting stuck. However, the procedure may also be to determine the risk of getting stuck and then estimate the stent placement section, or these procedures may be performed simultaneously.

[0065] The control unit 31 refers to the estimation result from step S104 and the determination result from step S106 to determine whether there are any locations with a high risk of stent stacking in the section where stent placement is predicted (placement section) (step S107). At this time, it may also be determined whether there are locations with a high risk of stacking in a predetermined proportion or more of the tomographic images among the multiple tomographic images corresponding to the placement section.

[0066] If the control unit 31 determines that there are no locations in the stent placement area with a high risk of getting stuck (S107: NO), it terminates the processing according to this flowchart.

[0067] If the control unit 31 determines that there is a high risk of the stent getting stuck in the stent placement area (S107: YES), it outputs an alert to prompt the operator to take precautions (step S108).

[0068] Figure 10 is a schematic diagram showing an example of alert output. Figure 10 shows an example of a screen displayed on the display device 4 by the control unit 31. This display screen 300 shows a transverse image 310, a longitudinal image 320, and an angiography image 330 of a blood vessel. In this example, the transverse image 310 is the 434th frame out of a total of 875 frames. The transverse image 310 also displays the contour lines 311-314 of the blood vessel region, lumen region, guidewire region, and catheter region, respectively, which are extracted by the learning model MD1.

[0069] The longitudinal image 320 shows markers 321-323. Marker 321 indicates the acquisition location of the transverse image 310. Marker 322 indicates the section where stent placement is predicted. Marker 323 indicates a location with a high risk of stent stacking. Information is also shown indicating that a stent with a diameter of 3.5 mm and a length of 21 mm is available. Other information that may be displayed includes vessel diameter, lumen diameter, plaque bathen (ratio of plaque area to cross-sectional area of ​​the vessel), distance L1 from the guidewire GW to the lumen wall, and distance L2 from the imaging catheter 1 to the lumen wall.

[0070] The angiography image 330 shows a marker 331. This marker 331 indicates a location on the angiography image 330 with a high risk of stacking. In other words, marker 331 on the angiography image 330 corresponds to marker 323 on the longitudinal section image 320. The control unit 31 can use known methods to derive the location on the angiography image 330 that corresponds to the location on the longitudinal section image 320.

[0071] In the example shown in Figure 10, the marker 323 can be displayed on the longitudinal tomographic image 320 to alert the operator. In the example shown in Figure 10, the marker 323 is indicated by a hatched rectangular area, but the shape and color of the marker 323 can be set as appropriate. Text information may be displayed instead of the marker 323. Furthermore, if the image processing device 3 is equipped with an audio output means, information about areas with a high risk of getting stuck may be communicated by voice.

[0072] Furthermore, if the operator modifies the stent placement section indicated by marker 322 on the screen, the control unit 31 may determine whether there are any areas with a high risk of jamming within the modified placement section, and may change the position of marker 323 according to the determination result.

[0073] As described above, in Embodiment 1, if a high risk of stent stacking is determined in a section where stent placement is expected, an alert is output to prompt the operator to take precautions. If stacking occurs, the procedure time will be extended to resolve it, increasing the burden on the patient. Furthermore, if unintentional force is applied to the stent due to the effects of stacking, causing deformation of the stent, correction will be necessary, further increasing the burden on the patient. In Embodiment 1, the operator can be notified of areas with a high risk of stacking before the stent is placed, thereby reducing the risk.

[0074] (Embodiment 2) Embodiment 2 describes the process for determining the risk of stent stacking after the stent has been implanted. The overall configuration of the image diagnostic device 100 and the internal configuration of the image processing device 3 are the same as in Embodiment 1, so their explanation will be omitted.

[0075] Figure 11 is a flowchart illustrating the procedure of processing performed by the image processing device 3 according to Embodiment 2. In Embodiment 1, the stack risk is determined using tomographic images after the stent has been placed. The control unit 31 of the image processing device 3 acquires a series of tomographic images generated by the intravascular examination device 101 through the input / output unit 33 (step S201).

[0076] The control unit 31 inputs each of the acquired tomographic images into the learning model MD1, performs calculations using the learning model MD1, and performs segmentation for each tomographic image (step S202). In Embodiment 2, the learning model MD1 is trained to recognize the vascular region, lumen region, guidewire region, and catheter region, as well as the region inside the stent. The control unit 31 extracts the vascular region, lumen region, guidewire region, catheter region, and stent region from the tomographic image by segmentation using the learning model MD1.

[0077] The control unit 31 estimates the section in which the stent is implanted based on the recognition results from the learning model MD1 (step S203). If tomographic images containing the stent region appear continuously along the long axis of the blood vessel, that section can be estimated as the section in which the stent is implanted (hereinafter also referred to as the implantation section).

[0078] Based on the recognition results from the learning model MD1, the control unit 31 calculates the distance L1 between the guidewire GW and the luminal wall, and the distance L2 between the diagnostic imaging catheter 1 and the luminal wall for each tomographic image (step S204). For example, the control unit 31 can calculate the actual dimensions of distances L1 and L2 by referring to known dimensional information (information on how many millimeters one pixel corresponds to) in the diagnostic imaging device 100. In Embodiment 2, for sections where a stent is implanted, instead of distances L1 and L2, the distance between the guidewire GW and the stent, and the distance between the diagnostic imaging catheter 1 and the stent may be calculated.

[0079] The control unit 31 determines the risk of the imaging catheter 1 getting stuck in each tomographic image based on the calculated distances L1 and L2 (step S205). For example, if the distance L1 between the guidewire GW and the luminal wall, or the distance L2 between the imaging catheter 1 and the luminal wall, is less than a second threshold (e.g., less than 0.5 mm), the control unit 31 can determine that there is a high risk of the imaging catheter 1 getting stuck in that tomographic image.

[0080] In this embodiment, for convenience, the procedure is to estimate the stent placement section and then determine the risk of getting stuck. However, the procedure may also be to determine the risk of getting stuck and then estimate the stent placement section, or these procedures may be performed simultaneously.

[0081] The control unit 31 refers to the estimation result from step S203 and the determination result from step S205 to determine whether or not there are any locations with a high risk of getting stuck in the section where stent placement is expected (placement section) (step S206).

[0082] If the control unit 31 determines that there are no locations in the stent placement area with a high risk of getting stuck (S206: NO), it terminates the processing according to this flowchart.

[0083] If the control unit 31 determines that there is a high risk of the stent getting stuck in the stent placement area (S206: YES), it outputs an alert to prompt the operator to take precautions (step S207).

[0084] The method for outputting alerts is the same as in Embodiment 1. That is, the control unit 31 displays a display screen 300 as shown in Figure 10 on the display device 4, and provides a highlighting to prompt the operator to pay attention.

[0085] As described above, in Embodiment 2, if it is determined that there is a high risk of stacking in the section where the stent is placed, an alert is output to prompt the operator to take precautions. When using the imaging diagnostic catheter 1 for postoperative observation, the proximal opening of the GW lumen may get caught on the stent strut, potentially causing a stack. If a stack occurs, the procedure time will be extended to resolve it, increasing the burden on the patient. Furthermore, if the stent is deformed due to unintentional force being applied as a result of the stack, correction will be necessary, further increasing the burden on the patient. In Embodiment 2, since the operator can be notified of areas with a high risk of stacking after the stent has been placed, the risk of stacking can be reduced.

[0086] In Embodiment 1, the state before stent placement was described, and in Embodiment 2, the state after stent placement was described. However, based on the recognition results by the learning model MD1, it may be possible to determine whether or not a stent has been placed, and if a stent has been placed, the stack risk determination process may be performed according to the flowchart shown in Figure 11, and if a stent has not been placed, the stack risk determination process may be performed according to the flowchart shown in Figure 9.

[0087] (Embodiment 3) Embodiment 3 describes a configuration that uses a learning model to determine the risk of getting stuck. The overall configuration of the image diagnostic device 100 and the internal configuration of the image processing device 3 are the same as in Embodiment 1, so their explanation will be omitted.

[0088] Figure 12 is a schematic diagram showing an example configuration of the learning model MD2 used in Embodiment 3. The learning model MD2 comprises, for example, an input layer LY1, an intermediate layer LY2, and an output layer LY3. An example of the learning model MD2 is a learning model based on a CNN (Convolutional Neural Network). Alternatively, the learning model MD2 may be a learning model based on YOLO, SSD, SVM, decision trees, etc.

[0089] The input layer LY1 receives tomographic images of blood vessels (IVUS images or OCT images). The tomographic image data input to input layer LY1 is then passed to the intermediate layer LY2.

[0090] The intermediate layer LY2 consists of a convolutional layer, a pooling layer, and a fully connected layer. Multiple convolutional and pooling layers may be arranged alternately. The convolutional and pooling layers extract features from the tomographic image input from the input layer LY1 through calculations using the nodes of each layer. The fully connected layer combines the data from which the features have been extracted by the convolutional and pooling layers into a single node and outputs a feature variable transformed by an activation function. The feature variable is output to the output layer through the fully connected layer.

[0091] The output layer LY3 has, for example, one node. Based on the feature variables input from the fully connected layer of the hidden layer LY2, the output layer LY3 calculates a probability indicating the stack risk and outputs it from its node.

[0092] The learning model MD2 is generated by collecting tomographic images when the diagnostic imaging catheter 1 is stuck and tomographic images when the diagnostic imaging catheter 1 is not stuck as training data, and then training the model using a predetermined learning algorithm with the collected tomographic images as training data.

[0093] Note that the learning model for calculating stack risk is not limited to the MD2 learning model shown in Figure 12. For example, the output of the encoder of a trained network, which has been trained using semantic segmentation as shown in Figure 6, may be connected to a classifier that determines stack risk. Alternatively, features related to stacking (distances L1, L2, etc.) may be extracted from the inference results of semantic segmentation and input into the classifier that determines stack risk. By extracting features that are closely related to stacking, the accuracy of the determination can be further improved.

[0094] Figure 13 is a flowchart illustrating the procedure of processing performed by the image processing device 3 according to Embodiment 3. The control unit 31 of the image processing device 3 acquires a series of tomographic images generated by the intravascular examination device 101 through the input / output unit 33 (step S301).

[0095] The control unit 31 inputs each of the acquired tomographic images into the learning model MD1, performs calculations using the learning model MD1, and performs segmentation for each tomographic image (step S302). Using segmentation with the learning model MD1, the control unit 31 extracts, for example, vascular regions, lumen regions, guidewire regions, catheter regions, and stent regions from the tomographic images.

[0096] The control unit 31 estimates the stent placement section based on the recognition results from the learning model MD1 (step S303). The control unit 31 may estimate the section in which stent placement is predicted using the same procedure as in Embodiment 1, or it may estimate the section in which the stent is placed using the same procedure as in Embodiment 2.

[0097] The control unit 31 inputs each of the acquired tomographic images into the learning model MD2, performs calculations using the learning model MD2, and determines the risk of the diagnostic imaging catheter 1 getting stuck in each tomographic image (step S304).

[0098] In this embodiment, for convenience, the procedure is to estimate the stent placement section and then determine the risk of getting stuck. However, the procedure may also be to determine the risk of getting stuck and then estimate the stent placement section, or these procedures may be performed simultaneously.

[0099] The control unit 31 refers to the estimation result from step S303 and the determination result from step S304 to determine whether there are any locations with a high risk of stent placement in the predicted stent placement section (placement section) (step S305). That is, the control unit 31 compares the stacking risk (probability) output from the learning model MD2 with a preset threshold and determines whether there are any locations in the placement section where the stacking risk is greater than the threshold.

[0100] If the control unit 31 determines that there are no locations in the stent placement area with a high risk of getting stuck (S305: NO), it terminates the processing according to this flowchart.

[0101] If the control unit 31 determines that there is a high risk of the stent getting stuck in the stent placement area (S305: YES), it outputs an alert to prompt the operator to take precautions (step S306).

[0102] The method for outputting alerts is the same as in Embodiment 1. That is, the control unit 31 displays a display screen 300 as shown in Figure 10 on the display device 4, and provides a highlighting to prompt the operator to pay attention.

[0103] As described above, in Embodiment 3, if it is determined that there is a high risk of stent stacking in the section where the stent is placed, an alert is output to prompt the operator to take precautions. In Embodiment 3, the stacking risk is determined using the learning model MD2, so the stacking risk can be determined with high accuracy.

[0104] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended. [Explanation of symbols]

[0105] 1. Diagnostic imaging catheter 2 MDU 3 Image Processing Device 4 Display device 5 Input devices 31 Control Unit 32 Main memory 33 Input / output section 34 Communications Department 35 Auxiliary storage 36 Reading section 100 Imaging diagnostic equipment 101 Intravascular Examination Device 102 Angiography equipment PG Risk Assessment Program MD1, MD2 learning models

Claims

1. We obtain tomographic images of tubular organs using imaging catheters. Based on the acquired tomographic images, the stent placement section is estimated. Based on the aforementioned tomographic image, the risk of the imaging diagnostic catheter getting stuck is determined. Output information on the risks in the estimated storage section. A computer program that causes a computer to perform a process.

2. Based on the aforementioned tomographic image, the positional relationship between the imaging diagnostic catheter and the tubular organ, or the positional relationship between the guidewire that guides the imaging diagnostic catheter and the tubular organ, Based on the identified positional relationship, the risk of the imaging catheter getting stuck is determined. A computer program according to claim 1 for causing the computer to perform a process.

3. The positional relationship is determined by calculating the distance between the imaging diagnostic catheter and the luminal wall of the tubular organ, or the distance between the guidewire and the luminal wall of the tubular organ, from the aforementioned tomographic image. A computer program according to claim 2 for causing the computer to perform a process.

4. When a tomographic image of a tubular organ is input, the acquired tomographic image is input to a learning model that has been trained to output information about the risk, and the learning model performs calculations. Based on the calculation results from the aforementioned learning model, the risk of the imaging diagnostic catheter getting stuck is determined. A computer program according to claim 1 for causing the computer to perform a process.

5. From the aforementioned tomographic images, the outer diameter and inner diameter of the tubular organ are calculated. Based on the calculated outer and inner diameters of the tubular organ, the placement section of the stent is estimated. A computer program according to claim 1 for causing the computer to perform a process.

6. The outer diameter is calculated based on the position of the external elastic plate in the tubular organ. A computer program according to claim 5 for causing the computer to perform processing.

7. The aforementioned storage section is the section in which the storage of the stent is expected to take place. The computer program according to claim 1.

8. The aforementioned storage section is the section in which the stent has been placed. The computer program according to claim 1.

9. The areas where the imaging diagnostic catheter is determined to be at risk of getting stuck are highlighted on the tomographic image. A computer program according to any one of claims 1 to 8, for causing the computer to perform processing.

10. An angiography image is acquired along with the tomographic image. The areas where the imaging catheter is determined to be at risk of getting stuck are highlighted on the angiography image. A computer program according to any one of claims 1 to 8, for causing the computer to perform processing.

11. We obtain tomographic images of tubular organs using imaging catheters. Based on the acquired tomographic images, the stent placement section is estimated. Based on the aforementioned tomographic image, the risk of the imaging diagnostic catheter getting stuck is determined. Output information on the risks in the estimated storage section. An information processing method in which a computer performs the processing.

12. An acquisition unit that acquires tomographic images of tubular organs obtained from an imaging diagnostic catheter, Based on the acquired tomographic images, an estimation unit estimates the stent placement section, A determination unit that determines the risk of the imaging diagnostic catheter getting stuck based on the tomographic image, An output unit that outputs information on the risk in the estimated storage section. An information processing device equipped with the following features.

Images