Computer program, information processing method, and information processing device

Abstract

This computer program causes a computer to execute processes of: acquiring a plurality of medical images generated along the axial direction of a blood vessel on the basis of a signal detected by a catheter inserted into the blood vessel; inputting the plurality of extracted medical images to a first learning model that outputs, when a medical image is input, the presence or absence of dissection included in the medical image, thereby determining the presence or absence of dissection included in the plurality of medical images; and, when the medical images include dissection, outputting information relating to the dissection.

Description

[Technical Field]

[0001] The present invention relates to a computer program, an information processing method, and an information processing device. [Background technology]

[0002] Medical images including ultrasonic tomographic images of blood vessels are generated by intravascular ultrasound (IVUS) using a catheter, and intravascular ultrasound examinations are performed. Meanwhile, technologies are being developed to add information to medical images using image processing and machine learning to assist doctors in their diagnoses (see, for example, Patent Document 1). The feature detection method for blood vessel images described in Patent Document 1 detects lumen walls, stents, and the like included in the blood vessel images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-525893 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the detection method of Patent Document 1 does not take into consideration the provision of information about dissection contained in the blood vessel image.

[0005] The object of the present disclosure is to provide a computer program, etc., that provides a catheter operator with information regarding dissection contained in a medical image obtained by scanning a blood vessel with a catheter. [Means for solving the problem]

[0006] The computer program of this embodiment causes a computer to acquire multiple medical images generated along the axial direction of a blood vessel based on a signal detected by a catheter inserted into the blood vessel, and input the extracted multiple medical images into a first learning model that outputs the presence or absence of dissection contained in the medical images when a medical image is input, thereby determining the presence or absence of dissection contained in the multiple medical images, and if the medical image contains dissection, outputting information regarding the dissection.

[0007] The information processing method of this aspect causes a computer to acquire multiple medical images generated along the axial direction of a blood vessel based on a signal detected by a catheter inserted into the blood vessel, input the extracted multiple medical images into a first learning model that outputs the presence or absence of dissection contained in the medical images when the medical images are input, thereby determining the presence or absence of dissection contained in the multiple medical images, and if the medical images contain dissection, executing a process to output information regarding the dissection.

[0008] The information processing device of this aspect includes an acquisition unit that acquires multiple medical images generated along the axial direction of the blood vessel based on a signal detected by a catheter inserted into the blood vessel, a determination unit that determines the presence or absence of dissection contained in the multiple medical images by inputting the extracted multiple medical images into a first learning model that outputs the presence or absence of dissection contained in the medical image when the medical image is input, and an output unit that outputs information regarding the dissection when the medical image contains a dissection. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide a computer program or the like that provides a catheter operator with information regarding dissection contained in a medical image obtained by scanning a blood vessel with the catheter. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram showing an example of the configuration of an imaging diagnostic apparatus. [Figure 2] FIG. 1 is an explanatory diagram illustrating an overview of a catheter for diagnostic imaging. [Figure 3] FIG. 2 is an explanatory diagram showing a cross section of a blood vessel through which a sensor portion is inserted. [Figure 4] FIG. 2 is an explanatory diagram illustrating a tomographic image. [Figure 5] FIG. 1 is a block diagram illustrating an example of the configuration of an image processing device. [Figure 6] FIG. 2 is an explanatory diagram showing an example of a first learning model (dissociation presence / absence determination model). [Figure 7] FIG. 10 is an explanatory diagram showing an example of a second learning model (dissociation angle, etc. estimation model). [Figure 8] 10 is a flowchart showing an information processing procedure performed by a control unit. [Figure 9] FIG. 10 is an explanatory diagram showing an example of displaying information related to dissociation. [Figure 10] 10 is a flowchart showing the information processing procedure by the control unit in the second embodiment (classification of dissociation). DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an image processing method, an image processing device, and a program according to the present disclosure will be described in detail with reference to the drawings showing embodiments thereof.

[0012] (Embodiment 1) FIG. 1 is an explanatory diagram showing an example of the configuration of an imaging diagnostic device 100. In this embodiment, an imaging diagnostic device using a dual-type catheter equipped with the functions of both intravascular ultrasound (IVUS) and optical coherence tomography (OCT) will be described. The dual-type catheter is provided with a mode for acquiring ultrasound tomographic images using only IVUS, a mode for acquiring optical coherence tomographic images using only OCT, and a mode for acquiring tomographic images using both IVUS and OCT, and these modes can be switched for use. Hereinafter, ultrasound tomographic images and optical coherence tomographic images will be referred to as IVUS images and OCT images, respectively, as appropriate. IVUS images and OCT images will be collectively referred to as tomographic images, which correspond to medical images.

[0013] The diagnostic imaging apparatus 100 of this embodiment includes an intravascular inspection apparatus 101, an angiography apparatus 102, an image processing device 3, a display device 4, and an input device 5. The intravascular inspection apparatus 101 includes a diagnostic imaging catheter 1 and an MDU (Motor Drive Unit) 2. The diagnostic imaging catheter 1 is connected to the image processing device 3 via the MDU 2. The display device 4 and the input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone. The display device 4 and the input device 5 may be stacked together to form a touch panel. Alternatively, the input device 5 and the image processing device 3 may be integrated. Furthermore, the input device 5 may be a sensor that accepts gesture input, gaze input, or the like.

[0014] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that images blood vessels from outside the patient's body using X-rays while injecting a contrast agent into the patient's blood vessels, thereby obtaining an angio image, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and images an X-ray fluoroscopic image of the patient by the X-ray sensor receiving X-rays irradiated from the X-ray source. Note that the diagnostic imaging catheter 1 is provided with a marker that is opaque to X-rays, and the position of the diagnostic imaging catheter 1 (marker) is visualized in the angio image. The angiography device 102 outputs the angio image obtained by imaging to the image processing device 3, and the angiography image is displayed on the display device 4 via the image processing device 3. Note that the display device 4 displays the angio image and a tomographic image captured using the diagnostic imaging catheter 1.

[0015] FIG. 2 is an explanatory diagram outlining the diagnostic imaging catheter 1. The upper dashed-dotted area in FIG. 2 is an enlarged version of the lower dashed-dotted area. The diagnostic imaging catheter 1 includes a probe 11 and a connector 15 disposed at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector 15. In the following description, the side of the diagnostic imaging catheter 1 farther from the connector 15 will be referred to as the distal end, and the connector 15 side will be referred to as the proximal end. The probe 11 includes a catheter sheath 11a, the distal end of which is provided with a guidewire insertion portion 14 through which a guidewire can be inserted. The guidewire insertion portion 14 forms a guidewire lumen and is used to receive a guidewire previously inserted into a blood vessel and guide the probe 11 to the affected area via the guidewire. The catheter sheath 11a forms a continuous tubular portion extending from the connection portion with the guidewire insertion portion 14 to the connection portion with the connector 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip side of the shaft 13.

[0016] The sensor unit 12 has a housing 12d, the distal end of which is hemispherically shaped to reduce friction and snagging with the inner surface of the catheter sheath 11a. The housing 12d contains an ultrasonic transceiver 12a (hereinafter referred to as the IVUS sensor 12a) that transmits ultrasonic waves into a blood vessel and receives reflected waves from the blood vessel, and an optical transceiver 12b (hereinafter referred to as the OCT sensor 12b) that transmits near-infrared light into the blood vessel and receives reflected light from the blood vessel. In the example shown in FIG. 2, the IVUS sensor 12a is provided at the distal end of the probe 11, and the OCT sensor 12b is provided at the proximal end. The sensors are disposed on the central axis of the shaft 13 (on the two-dot chain line in FIG. 2) at a distance x along the axial direction. In the diagnostic imaging catheter 1, the IVUS sensor 12a and the OCT sensor 12b are attached such that the direction of transmission and reception of ultrasonic waves or near-infrared light is approximately 90 degrees relative to the axial direction of the shaft 13 (the radial direction of the shaft 13). It is desirable that the IVUS sensor 12a and the OCT sensor 12b are mounted slightly offset from the radial direction so as not to receive waves or light reflected from the inner surface of the catheter sheath 11a. In this embodiment, as shown by the arrow in Fig. 2, for example, the IVUS sensor 12a is mounted so that the direction of ultrasound irradiation is inclined toward the proximal end with respect to the radial direction, and the OCT sensor 12b is mounted so that the direction of near-infrared light irradiation is inclined toward the distal end with respect to the radial direction.

[0017] An electric signal cable (not shown) connected to the IVUS sensor 12a and an optical fiber cable (not shown) connected to the OCT sensor 12b are inserted into the shaft 13. The probe 11 is inserted into a blood vessel from the distal end side. The sensor unit 12 and the shaft 13 are movable forward and backward within the catheter sheath 11a and can also rotate circumferentially. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation. The diagnostic imaging device 100 uses an imaging core formed by the sensor unit 12 and the shaft 13 to measure the state inside the blood vessel using ultrasound tomographic images (IVUS images) taken from inside the blood vessel or optical coherence tomographic images (OCT images) taken from inside the blood vessel.

[0018] The MDU 2 is a drive unit to which the probe 11 (diagnostic imaging catheter 1) is detachably attached via a connector unit 15, and controls the operation of the diagnostic imaging catheter 1 inserted into a blood vessel by driving a built-in motor in response to operation by a medical professional. For example, the MDU 2 performs a pull-back operation to rotate the sensor unit 12 and shaft 13 inserted into the probe 11 in a circumferential direction while pulling them toward the MDU 2 at a constant speed. The sensor unit 12 rotates while moving from the distal end to the proximal end due to the pull-back operation, and thereby scans the inside of the blood vessel continuously at predetermined time intervals, thereby continuously capturing multiple transverse cross-sectional images approximately perpendicular to the probe 11 at predetermined intervals. The MDU 2 outputs reflected wave data of ultrasound received by the IVUS sensor 12a and reflected light data received by the OCT sensor 12b to the image processing device 3.

[0019] The image processing device 3 acquires, via the MDU 2, a signal dataset that is reflected wave data of ultrasound received by the IVUS sensor 12a and a signal dataset that is reflected light data received by the OCT sensor 12b. The image processing device 3 generates ultrasound line data from the ultrasound signal dataset and constructs an ultrasound tomographic image (IVUS image) of a transverse layer of the blood vessel based on the generated ultrasound line data. The image processing device 3 also generates optical line data from the reflected light signal dataset and constructs an optical tomographic image (OCT image) of a transverse layer of the blood vessel based on the generated optical line data. Here, the signal datasets acquired by the IVUS sensor 12a and the OCT sensor 12b and the tomographic image constructed from the signal dataset will be described. FIG. 3 is an explanatory diagram showing a cross section of a blood vessel through which the sensor unit 12 has been inserted, and FIG. 4 is an explanatory diagram showing the tomographic image.

[0020] First, referring to FIG. 3 , the operation of the IVUS sensor 12a and the OCT sensor 12b within a blood vessel and the signal data sets (ultrasound line data and optical line data) acquired by the IVUS sensor 12a and the OCT sensor 12b will be described. When imaging of a tomographic image begins with the imaging core inserted into the blood vessel, the imaging core rotates in the direction indicated by the arrow, with the central axis of the shaft 13 as the center of rotation. At this time, the IVUS sensor 12a transmits and receives ultrasound waves at each rotation angle. Lines 1, 2, ... 512 indicate the transmission and reception directions of ultrasound waves at each rotation angle. In this embodiment, the IVUS sensor 12a intermittently transmits and receives ultrasound waves 512 times while rotating 360 degrees (one rotation) within the blood vessel. The IVUS sensor 12a acquires one line of data in the transmission and reception direction by transmitting and receiving ultrasound waves once, so that 512 ultrasound line data extending radially from the center of rotation can be obtained during one rotation. The 512 ultrasound line data are dense near the center of rotation, but become sparser as they move away from the center of rotation. Therefore, the image processing device 3 generates pixels in the empty spaces of each line by well-known interpolation processing, thereby generating a two-dimensional ultrasound tomographic image (IVUS image) as shown in Figure 4A.

[0021] Similarly, the OCT sensor 12b also transmits and receives measurement light at each rotation angle. Since the OCT sensor 12b also transmits and receives measurement light 512 times while rotating 360 degrees within the blood vessel, 512 optical line data extending radially from the center of rotation can be obtained during one rotation. For the optical line data, the image processing device 3 generates pixels in the empty spaces of each line using a well-known interpolation process, thereby generating a two-dimensional optical coherence tomographic image (OCT image) similar to the IVUS image shown in FIG. 4A. That is, the image processing device 3 generates optical line data based on interference light generated by interfering reflected light with reference light obtained by, for example, separating light from a light source within the image processing device 3, and constructs an optical tomographic image (OCT image) capturing a cross-section of the blood vessel based on the generated optical line data.

[0022] A two-dimensional tomographic image generated from 512 lines of data in this manner is called one frame of an IVUS image or an OCT image. Note that since the sensor unit 12 scans while moving inside the blood vessel, one frame of an IVUS image or an OCT image is acquired at each position of one rotation within the range of movement. That is, one frame of an IVUS image or an OCT image is acquired at each position from the distal end to the proximal end of the probe 11 within the range of movement, so that multiple frames of IVUS images or OCT images are acquired within the range of movement, as shown in FIG. 4B.

[0023] The diagnostic imaging catheter 1 has markers that are opaque to X-rays in order to confirm the positional relationship between an IVUS image obtained by the IVUS sensor 12a or an OCT image obtained by the OCT sensor 12b, and an angio image obtained by the angiography device 102. In the example shown in FIG. 2, a marker 14a is provided at the tip of the catheter sheath 11a, for example, at the guidewire insertion portion 14, and a marker 12c is provided on the shaft 13 side of the sensor unit 12. When the diagnostic imaging catheter 1 configured in this manner is imaged with X-rays, an angio image is obtained in which the markers 14a and 12c are visualized. The positions at which the markers 14a and 12c are provided are merely examples; the marker 12c may be provided on the shaft 13 instead of the sensor unit 12, and the marker 14a may be provided at a location other than the tip of the catheter sheath 11a.

[0024] 5 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer (information processing device) and includes a control unit 31, a main memory unit 32, an input / output I / F 33, an auxiliary memory unit , and a reading unit .

[0025] The control unit 31 is configured using one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), a general-purpose computing on graphics processing unit (GPGPU), a tensor processing unit (TPU), etc. The control unit 31 is connected to each hardware component constituting the image processing device 3 via a bus.

[0026] The main memory unit 32 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 31 to execute arithmetic processing.

[0027] The input / output I / F 33 is an interface to which the intravascular inspection device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images and OCT images from the intravascular inspection device 101 and acquires angio images from the angiography device 102 via the input / output I / F 33. The control unit 31 also displays medical images on the display device 4 by outputting medical image signals of the IVUS images, OCT images, or angio images to the display device 4 via the input / output I / F 33. Furthermore, the control unit 31 receives information input to the input device 5 via the input / output I / F 33.

[0028] A communication unit including a wireless terminal such as 4G, 5G, or WiFi is connected to the input / output I / F 33, and the image processing device 3 may be communicably connected to an external server, such as a cloud server, connected to an external network such as the Internet via the communication unit. The control unit 31 may access the external server via the communication unit and the external network, refer to medical data, research paper information, etc. stored (stored) in a storage device included in the external server, and perform processing related to information provision (provision processing for providing support information). Alternatively, the control unit 31 may cooperate with the external server to perform the processing in this embodiment, for example, by performing inter-process communication.

[0029] The auxiliary storage unit 34 is a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The auxiliary storage unit 34 stores a computer program P (program product) executed by the control unit 31 and various data required for processing by the control unit 31. The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P (program product) may be written to the auxiliary storage unit 34 during the manufacturing stage of the image processing device 3, or may be distributed by a remote server device and acquired by the image processing device 3 via communication and stored in the auxiliary storage unit 34. The computer program P (program product) may be readably recorded on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, or may be read from the recording medium 30 by the reading unit 35 and stored in the auxiliary storage unit 34.

[0030] The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 is described as being a single computer. In this embodiment, an angiography device 102 that captures two-dimensional angioimages is connected to the image processing device 3, but the device is not limited to the angiography device 102 as long as it captures images of a patient's hollow organs and the diagnostic imaging catheter 1 from multiple directions outside the body.

[0031] In the image processing device 3 of this embodiment, the control unit 31 reads and executes a computer program P stored in the auxiliary storage unit 34 to construct an IVUS image based on the signal data set received from the IVUS sensor 12a and an OCT image based on the signal data set received from the OCT sensor 12b. Note that, as will be described later, the IVUS sensor 12a and the OCT sensor 12b have different observation positions at the same imaging timing. Therefore, the control unit 31 corrects the difference in the observation positions in the IVUS image and the OCT image. Therefore, the image processing device 3 of this embodiment provides IVUS images and OCT images with the same observation positions, thereby providing images that are easy to interpret.

[0032] In this embodiment, the diagnostic imaging catheter is described as a dual-type catheter having both intravascular ultrasound (IVUS) and optical coherence tomography (OCT) functions, but is not limited to this. The diagnostic imaging catheter may also be a single-type catheter having either intravascular ultrasound (IVUS) or optical coherence tomography (OCT) functions. Hereinafter, in this embodiment, the diagnostic imaging catheter has an intravascular ultrasound (IVUS) function, and the description will be based on IVUS images generated by the IVUS function. However, in the description of this embodiment, the medical image is not limited to an IVUS image, and the processing of this embodiment may also be performed using an OCT image as the medical image.

[0033] 6 is an explanatory diagram showing an example of the first learning model 341. The first learning model 341 is, for example, a neural network (segmentation NN) such as YOLO or R-CNN that performs object detection, semantic segmentation, or instance segmentation. The first learning model 341 may output whether or not a dissociation (object) is included in the IVUS image based on each IVUS image in the input IVUS image group, and if a dissociation (object) is included, output the region of the dissociation in the IVUS image and the estimation accuracy (score).

[0034] The first learning model 341 is configured, for example, by a convolutional neural network (CNN) trained by deep learning. The first learning model 341 includes an input layer 341a to which medical images such as IVUS images are input, an intermediate layer 341b that extracts image features, and an output layer 341c that outputs information indicating the position and type of objects included in the medical image. The input layer 341a of the first learning model 341 includes multiple neurons that receive input of pixel values ​​of each pixel included in the medical image and transfers the input pixel values ​​to the intermediate layer 341b. The intermediate layer 341b includes an alternating convolutional layer that convolves the pixel values ​​of each pixel input to the input layer 341a and a pooling layer that maps the pixel values ​​convolved in the convolutional layer, thereby extracting image features while compressing the pixel information of the medical image. The intermediate layer 341b transfers the extracted features to the output layer 341c. The output layer 341c has one or more neurons that output the position, range, and type of an image region of an object contained in an image. While the first learning model 341 is described as a CNN, the configuration of the first learning model 341 is not limited to a CNN. The first learning model 341 may be a trained model configured, for example, as a neural network other than a CNN, a fully convolution network (FCN) such as a U-net, SegNet, SSD, SPPnet, a support vector machine (SVM), a Bayesian network, or a regression tree. Alternatively, the first learning model 341 may input image features output from the intermediate layer into a support vector machine (SVM) to perform object recognition.

[0035] The first learning model 341 can be generated by preparing training data in which medical images containing objects such as vascular dissections (hereinafter referred to as dissections) are associated with labels indicating the location (region) and type of each object (dissection, etc.), and then using the training data to train an untrained neural network. According to the first learning model 341 configured in this manner, by inputting a medical image such as an IVUS image into the first learning model 341, information indicating the location, etc., of a dissection contained in the medical image can be obtained. If a medical image does not contain a dissection, information indicating the location and type is not output from the first learning model 341. In other words, the first learning model 341 functions as a classifier (classification) that classifies multiple input IVUS images into IVUS images containing dissections and IVUS images not containing dissections, and corresponds to a dissection presence / absence determination model. Based on this information acquired from the first learning model 341, the control unit 31 derives object information regarding the presence or absence of an object (dissection) contained in the IVUS image. Alternatively, the control unit 31 may derive the object information by using the information itself acquired from the first learning model 341 as the object information. The objects included in the medical image are not limited to dissections, and may include, for example, hematomas, thrombi, plaques that have deviated into stents, lipid plaques, fibrous plaques, calcified areas, epicardium, side branches, veins, guide wires, or stents.

[0036] The control unit 31 may input each IVUS image (frame image) one by one into the first learning model 341 for processing, or may input multiple consecutive frame images simultaneously and simultaneously detect dissociation regions (presence or absence of dissociation) from the multiple frame images. For example, the control unit 31 configures the first learning model 341 as a 3D-CNN (e.g., 3D U-net) that handles three-dimensional input data. The control unit 31 then handles the data as three-dimensional data, with the coordinates of the two-dimensional frame images as the two axes and the time (generation time) t at which each frame image was acquired as the first axis. The control unit 31 inputs multiple frame images (e.g., 16 frames) for a predetermined unit time into the first learning model 341 as a set and simultaneously outputs images in which the dissociation regions (presence or absence of dissociation) are labeled for each of the multiple frame images. This allows the dissociation regions (presence or absence of dissociation) to be detected taking into account the frame images that are consecutive in time series, thereby improving detection accuracy.

[0037] 7 is an explanatory diagram showing an example of the second learning model 342 (dissection angle, etc. estimation model). Like the first learning model 341, the second learning model 342 is configured with a neural network such as a CNN, and when a medical image such as an IVUS image containing dissection is input, the second learning model 342 estimates the dissection angle and minor axis length contained in the medical image. The second learning model 342 is trained using training data in which IVUS images containing dissection are associated with the dissection angle and minor axis length. A large amount of data regarding the dissection angle and minor axis length contained in IVUS images is stored in medical institutions, etc., and training data can be generated using this data.

[0038] As shown in the figure in this embodiment, multiple IVUS images generated in one pullback operation (all IVUS images for one pullback) are input to the first learning model 341. During the pullback operation, the sensor unit 12 is pulled back from the distal side to the proximal side by the MDU 2 at a constant speed (e.g., 0.5 mm / sec or 1.0 mm / sec). The spatial resolution of the image core in generating the IVUS images is, for example, 100 to 200 μm. For example, when the pullback speed is 1.0 mm / sec and the spatial resolution is 100 μm, 10 IVUS images are generated per second. In this case, the length of the blood vessel included in the 10 IVUS images is 1 mm. That is, the 10 IVUS images show cross-sectional views (longitudinal views) of the blood vessel at a distance (separation distance) of 0.1 mm. Furthermore, the 10 IVUS images are generated at intervals of 0.1 seconds (generation time intervals). In this way, the time point at which an IVUS image is generated (generation time point) corresponds to the axial position of the blood vessel indicated by the IVUS image (axial position). For example, if the pullback length is 10 cm (100 mm), 1,000 IVUS images are generated in a pullback time of 10 seconds. When the generated IVUS images are aligned along the axial direction of the blood vessel (hollow organ) at the positions indicated by the blood vessel, these multiple IVUS images are positioned according to a separation distance determined based on the pullback speed and spatial resolution. This identifies the axial position of each of the multiple IVUS images. In other words, the position of the generated IVUS image indicates the axial position of the blood vessel included in the IVUS image.

[0039] Multiple IVUS images (all IVUS images from one pullback) are input to a first learning model 341, which classifies them into IVUS images with dissection (dissection: present) and IVUS images without dissection (dissection: absent) depending on whether or not the IVUS images contain dissection. The IVUS images with dissection (dissection: present) are input to a second learning model 342, and the angle and short axis length of the dissection contained in each IVUS image are output as estimation results from the second learning model 342. The angle of dissection is the central angle corresponding to the range (arc) in which the tunica media is flaped into two layers along the artery in a tomographic image of the blood vessel. The short axis length of the dissection is the length of the chord of the arc specified by the angle of dissection (central angle). By using the first learning model 341 and the second learning model 342 connected in series in this manner, the control unit 31 can derive (acquire) the angle and short axis length of the dissection contained in each IVUS image.

[0040] In this embodiment, the control unit 31 uses, but is not limited to, a first learning model 341 and a second learning model 342 connected in series. The control unit 31 may use a single learning model that extracts (segments) a dissection region when an IVUS image is input, and estimates the angle and minor axis length according to the extracted dissection region.

[0041] Instead of using the second learning model 342, the control unit 31 may extract the dissection region by, for example, edge detection or pattern detection, and calculate the angle and minor axis length of the dissection by pixel analysis of the extracted region or arithmetic processing in an image coordinate system. The control unit 31 may, for example, identify pixels in the dissection region segmented by the first learning model 341, derive an approximate curve (arc) based on the pixels, and calculate the length of a straight line connecting both ends of the approximate curve (arc) as the minor axis length. The control unit 31 may determine the angle of dissection as the interior angle of a line segment extending from the both ends toward the center of gravity of the lumen or the center of the vessel.

[0042] The multiple IVUS images detected (classified) as IVUS images with dissection (dissection: present) are located along the axial direction of the blood vessel, and the control unit 31 derives (calculates) the long axis length of the dissection based on the multiple IVUS images (dissection: present). As described above, the IVUS images are arranged in the order of their generation time points from the distal side to the proximal side in the axial direction of the blood vessel based on the spatial resolution (generation time point interval).

[0043] The control unit 31 extracts (specifies) the IVUS image located most distally and the IVUS image located most proximally from the plurality of IVUS images classified as having dissection (dissection: present) based on the time of generation or the frame number corresponding to the time of generation. If the frame numbers are assigned by incrementing based on the time of generation, the IVUS image with the smallest frame number from the plurality of IVUS images classified as having dissection (dissection: present) corresponds to the IVUS image located most distally, and the IVUS image with the largest frame number corresponds to the IVUS image located most proximal.

[0044] The control unit 31 identifies the frame numbers of the two IVUS images located at both the distal and proximal ends, and calculates (derives) the distance between the two IVUS images as the long axis length of the dissection based on the difference between the frame numbers and the spatial resolution used to generate the IVUS images. For example, if the difference between the frame numbers is 4 and the spatial resolution is 100 μm (0.1 mm separation), the long axis length of the dissection is 0.4 mm (0.4 = 4 * 0.1).

[0045] The control unit 31 of the image processing device 3 executes the following processing based on input data and the like output from the input device 5 in response to operations by an operator of the diagnostic imaging catheter 1, such as a doctor.

[0046] 8 is a flowchart showing the information processing procedure by the control unit. The control unit 31 acquires IVUS images (S101). The control unit 31 reads a group of IVUS images obtained by pullback to acquire a medical image consisting of these multiple IVUS images. The multiple IVUS images correspond to all the IVUS images from one pullback, and each IVUS image is assigned a frame number according to, for example, the time of generation. If the frame numbers are incremented based on the time of generation, the frame numbers assigned to each IVUS image will be numbered in a numbering system that increases from the distal side to the proximal side.

[0047] The control unit 31 determines whether or not the IVUS image contains dissection (S102). The control unit 31 inputs the acquired multiple IVUS images, for example, sequentially from the IVUS image located on the distal side, into the first learning model 341 and obtains the determination result from the first learning model 341, thereby determining whether or not the IVUS image contains dissection.

[0048] If the IVUS image contains dissection (S102: YES), the control unit 31 derives the angle and minor axis length of the dissection (S103). If the IVUS image contains dissection, the control unit 31 inputs the IVUS image to the second learning model 342 and obtains the estimation result by the second learning model 342, thereby deriving the angle and minor axis length of the dissection.

[0049] The control unit 31 associates the derived angle and minor axis length with the IVUS image and stores them (S104). The control unit 31 associates the derived angle and minor axis length using the second learning model 342 with, for example, the frame number of the IVUS image, and stores them in an array format in the auxiliary storage unit 34 (stored in an array variable).

[0050] If the IVUS image does not contain dissection (S102: NO), or after the process of S104, the control unit 31 determines whether or not all IVUS images have been processed (S105). If all IVUS images have not been processed (S105: NO), the control unit 31 performs a loop process to execute the process of S102 again. By performing this loop process, all IVUS images are classified based on the presence or absence of dissection.

[0051] If processing has been performed on all IVUS images (S105: YES), the control unit 31 derives the long axis length of the dissection based on the IVUS images containing the dissection (S106). The frame number of the IVUS image containing the dissection is associated with the angle and short axis length of the dissection. If the frame numbers are assigned by incrementing based on the time of generation, the control unit 31 identifies the IVUS image with the smallest frame number among the frame numbers associated with the angle, etc. as the IVUS image located most distally, and identifies the IVUS image with the largest frame number as the IVUS image located most proximally. The control unit 31 calculates (derives) the distance between the two identified IVUS images as the long axis length of the dissection based on the difference between the frame numbers of the two identified IVUS images and the spatial resolution used to generate the IVUS images.

[0052] The control unit 31 acquires an angio image (S107). As described above, the diagnostic imaging catheter 1 is provided with markers (marker 14a, marker 12c) that are opaque to X-rays. Because the position of the diagnostic imaging catheter 1 (marker) is visualized in the angio image, the control unit 31 can identify and acquire an angio image corresponding to a medical image in which dissection is detected, i.e., an angio image including the internal body part indicated by the medical image, based on the marker. The control unit 31 may identify an angio image corresponding to an IVUS image in which dissection is detected using a co-registration function that registers (synchronizes positions of) the IVUS image and the angio image. When displaying the angio image, the control unit 31 may extract a portion of the blood vessel with dissection included in the angio image and output it to a display device for display. The control unit 31 may also extract a vascular portion containing dissection from an angio image using a learning model such as a neural network with a segmentation function similar to the first learning model used for IVUS images, or a judgment-based method such as classification and Grad-CAM, and compare the extracted vascular portion with the location of dissection determined from the IVUS image to ensure the reliability of the support information. The basis for the judgment can be enhanced by combining an angio image corresponding to an IVUS image determined to contain dissection. Since the support information provided to a physician or other such person includes a medical image in which dissection is detected and an angio image corresponding to the medical image, information useful for evaluating dissection can be efficiently displayed. The control unit 31 may set a portion of the blood vessel identified by the IVUS image determined to contain dissection (a range in the axial direction of the blood vessel) as a reference portion and calculate the vessel diameter (EEM) and lumen diameter of the reference portion. The control unit may also display the calculated vessel diameter (EEM) and lumen diameter on the display screen of the display device 4, superimposed on the display screen.

[0053] The control unit 31 outputs information about the dissection (S108). The control unit 31 outputs the information about the dissection to the display device 4 as support information to be provided to a doctor or the like, and causes the support information to be displayed on the display device 4. The information about the dissection (support information) includes the angle and short axis length of the dissection in each IVUS image, and the long axis length in the axial direction of the blood vessel in which the dissection has occurred.

[0054] FIG. 9 is an explanatory diagram showing a display example of information (support information) related to dissection. In this display example, a transverse view, which is a cross-sectional view of the blood vessel in the axial direction, and a longitudinal view, which is a cross-sectional view of the blood vessel in the radial direction, are displayed side by side. On the longitudinal view, figures such as an arrow indicating the region of dissection included in the IVUS image, the angle of dissection, and the minor axis length are superimposed, and the values ​​of the angle and the minor axis length are annotated. As shown in the figure in this embodiment, the angle of dissection corresponds to the range (arc) in which the tunica media is flaped into two layers along the arterial path in the longitudinal view of the blood vessel, and is represented, for example, by the central angle centered on the center of gravity of the lumen. The minor axis length of the dissection is represented by the length of the chord of the arc specified by the angle of dissection (central angle). On the transverse view, a figure such as an arrow indicating the region where dissection has occurred is superimposed, and the value of the major axis length of the dissection is annotated. The frame number of the IVUS image indicating the region where dissection has occurred may also be displayed. The control unit 31 may, for example, display the frame number of the most distal IVUS image, the frame number of the most proximal IVUS image, and the frame number of the currently selected (displayed) longitudinal section in the region where dissection is occurring.

[0055] An angio image including the internal body part indicated by the IVUS image (longitudinal tomogram) including the dissection is displayed adjacent to the IVUS image. An angio image identified based on a marker or the like may be displayed with a line diagram indicating the positional relationship with the blood vessel part indicated by the IVUS image (longitudinal tomogram) (the part where dissection has occurred). The control unit 31 may output screen data for generating a display screen according to the display example to the display device 4 and cause the display device 4 to display the display screen.

[0056] According to this embodiment, the image processing device 3 (controller 31) inputs medical images (IVUS images) into the first learning model 341 and acquires information regarding the presence or absence of dissection detected (estimated) by the first learning model 341. Therefore, the presence or absence of dissection can be efficiently determined for each of multiple medical images generated along the axial direction of the blood vessel, and the multiple medical images can be classified into medical images with dissection and medical images without dissection. The image processing device 3 extracts medical images with dissection and outputs them to the display device 4, thereby displaying information regarding dissection on the display device 4, thereby efficiently providing support information to the operator of the diagnostic imaging catheter 1, such as a physician. The image processing device 3 inputs medical images with dissection into the second learning model 342, thereby efficiently deriving the angle and short axis length of the dissection contained in the medical image. The support information provided to the physician, etc., includes the angle and short axis length of the dissection, so that information useful for evaluating dissection can be efficiently displayed.

[0057] According to this embodiment, the image processing device 3 identifies multiple medical images in which dissection is detected, and derives the long axis length of the blood vessel in which dissection has occurred based on the position in the axial direction of the blood vessel shown in the medical images. The support information provided to a doctor or the like includes the position (range) of dissection in the axial direction of the blood vessel and the long axis length of the dissection, so that information useful for evaluating dissection can be efficiently displayed.

[0058] (Embodiment 2) 10 is a flowchart showing the information processing procedure by the control unit 31 in embodiment 2 (classification of dissection). The control unit 31 of the image processing device 3 executes the following processing based on input data output from the input device 5 in response to operations by an operator of the diagnostic imaging catheter 1, such as a doctor.

[0059] The control unit 31 acquires an IVUS image (S201). The control unit 31 determines whether or not the IVUS image contains dissociation (S202). The control unit 31 performs the processes of S201 and S202 in the same manner as the processes of S101 and S102 in the first embodiment.

[0060] The control unit 31 identifies the classification of the dissection (S203). The first learning model 341 of the second embodiment is trained to estimate the classification (type) of the dissection contained in the input medical image. The first learning model 341 is trained using training data in which IVUS images containing dissections of each classification are associated with labeled images in which the dissection region is identified and the classification (class) of the dissection is assigned. The output layer of the first learning model 341 may be configured, for example, with a softmax layer, and the first learning model 341 may estimate the probability that the dissection contained in the input medical image falls into one of multiple classifications (types) that have been learned (defined) in advance. The classification (type) of the dissection may be, for example, the Stanford classification (classification based on the extent of dissection), the DeBakey classification (classification based on blood flow in the false lumen), or classification based on an angio image. The Stanford classification includes Type A (dissection in the ascending aorta) and Type B (no dissection in the ascending aorta). The DeBakey classification includes Type I (tear in the ascending aorta with dissection extending distally beyond the aortic arch), Type II (dissection limited to the ascending aorta), Type III (tear in the descending aorta), Type IIIa (dissection not extending to the abdominal aorta), and Type IIIb (dissection extending to the abdominal aorta). Classification based on angiographic images includes Type A (luminal haziness), Type B (linear dissection), Type C (extra-luminal contrast), Type D (spiral dissection), Type E (dissection with persistent), and Type F (dissection with total occlusion). Based on the probability of each classification estimated by the first learning model 341, the control unit 31 identifies the classification with the largest probability value as the classification (type) of dissociation contained in the IVUS image.

[0061] The control unit 31 derives the angle and minor axis length of the dissection (S204). The control unit 31 associates the derived classification, angle, and minor axis length with the IVUS image and stores them (S205). The control unit 31 determines whether processing has been performed on all IVUS images (S206). The control unit 31 derives the major axis length of the dissection based on the IVUS image containing the dissection (S207). When multiple IVUS images acquired by a single pullback operation contain dissections of multiple classifications, the control unit 31 may derive the major axis length of the dissection for each classification. The control unit 31 acquires an angioimage (S208). The control unit 31 performs the processes of S204 and S208 in the same manner as the processes of S103 and S107 in the first embodiment.

[0062] The control unit 31 outputs information related to the dissection (S209). The control unit 31 outputs the information related to the dissection to the display device 4 as support information to be provided to a doctor or the like, and causes the display device 4 to display the support information. The information related to the dissection (support information) includes the classification, angle, and short axis length of the dissection in each IVUS image, and the long axis length in the axial direction of the blood vessel in which the dissection has occurred. The control unit 31 may cause the display device 4 to display the information related to the dissection (support information) using a display example (display screen) similar to that of the first embodiment.

[0063] The control unit 31 may acquire information about a dissection treatment method by inputting information about the dissection into a learning model (dissection treatment estimation model) that outputs information about a dissection treatment method when the information about the dissection is input. The learning model (dissection treatment estimation model) has a configuration similar to the first learning model 341 or the second learning model 342, etc., and is trained using teacher data including information about the dissection and information about the treatment method, and is stored in the auxiliary storage unit 34. While the learning model (dissection treatment estimation model) is used to acquire information about the treatment method, this is not limited thereto. Table information in which information about the dissection and information about the treatment method are associated (correlated) may also be used. The information about the dissection treatment method may include, for example, a treatment device, a procedure for using the treatment device, and the type and frequency of complications that may occur as a result of additional treatment using the treatment device, or both. The information about the dissection includes the angle and short-axis length of the dissection and the long-axis length of the blood vessel in the axial direction in which the dissection occurs, i.e., indicates the state of the dissection. Furthermore, the information about the dissection includes a classification according to the state. By inputting information about dissociation into the learning model (dissociation treatment estimation model) in this way, the control unit 31 can acquire information about the state of dissociation or the treatment method for dissociation according to the classification determined by the state. The control unit 31 can provide (present) the acquired information about the treatment method for dissociation to a doctor or the like by outputting the information to the display device 4.

[0064] The control unit 31 may acquire the state of dissociation and the complication information determined by the state by inputting information about dissociation into a learning model (complication estimation model) that outputs the state of dissociation and complication information determined by the state when the information about dissociation is input. The learning model (complication estimation model) has a configuration similar to the first learning model 341 or the second learning model 342, etc., and is trained using teacher data including information about dissociation, the state of dissociation, and complication information determined by the state, and is stored in the auxiliary storage unit 34. While the learning model (complication estimation model) has been described as being used to acquire complication information, acquisition of complication information and the like is not limited to this, and table information in which information about dissociation is associated with the state of dissociation and the complication information determined by the state may also be used. The state of dissociation includes, for example, detailed states such as the progression of dissociation and the period since onset. The complication information may include, for example, the details of the complication and the frequency of occurrence of the complication. By inputting information about dissociation into the learning model (complication estimation model) in this manner, the control unit 31 can acquire detailed states of dissociation and complication information determined by the state. The control unit 31 outputs the acquired complication information and the like to the display device 4, thereby providing (presenting) the information to a doctor or the like.

[0065] According to this embodiment, the image processing device 3 can efficiently identify the classification of dissociation contained in each of a plurality of medical images by using the first learning model 341 that detects the classification of dissociation. Since the support information provided to a doctor or the like includes the identified classification of dissociation, it is possible to efficiently display information useful for evaluating dissociation.

[0066] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the scope of the claims and the scope equivalent to the claims. [Explanation of symbols]

[0067] 1 Diagnostic imaging catheter 11 Probe 11a Catheter sheath 12 Sensor section 12a Ultrasonic transmitter / receiver 12a IVUS sensor 12b Optical transmitter / receiver 12b OCT sensor 12c marker 12d Housing 13 Shaft 14 Guidewire insertion part 14a Marker 15 Connector part 2 MDU 3. Image processing device (information processing device) 30 Recording media 31 Control Unit 32 Main memory 33 Input / Output Interface 34 Auxiliary storage 341 First Learning Model (Dissociation Presence / Absence Judgment Model) 342 Second Learning Model (Estimation Model of Dissociation Angle, etc.) 35 Reading unit 4 Display device 5 Input Devices 100 Diagnostic imaging equipment 101 Intravascular inspection device 102 Angiography equipment

Claims

1. On the computer, A plurality of medical images are acquired along the axial direction of the blood vessel based on signals detected by a catheter inserted into the blood vessel. inputting the extracted medical images into a first learning model that outputs the presence or absence of dissociation contained in the medical images when the medical images are input, thereby determining the presence or absence of dissociation contained in the medical images; If the medical image includes a dissection, output information regarding the dissection; When the medical image includes a dissection, the angle and the minor axis length of the dissection included in the medical image are derived by inputting the medical image into a second learning model that estimates the angle and the minor axis length of the dissection included in the medical image when the medical image is input; outputting the derived angle and the minor axis length together with information relating to the dissociation; The first learning model and the second learning model are connected in series. A computer program for executing a process.

2. A computer, A plurality of medical images are acquired along the axial direction of the blood vessel based on signals detected by a catheter inserted into the blood vessel. inputting the extracted medical images into a first learning model that outputs the presence or absence of dissociation contained in the medical images when the medical images are input, thereby determining the presence or absence of dissociation contained in the medical images; If the medical image includes a dissection, output information regarding the dissection; outputting the dissection state or complication information determined by the state; The complication information includes a complication content and an occurrence frequency of the complication, The information about the dissection is input to a complication estimation model that outputs the state of the dissection and complication information determined by the state when the information about the dissection is input, thereby obtaining the state of the dissection and the complication information determined by the state. A computer program for executing a process.

3. deriving a long axis length in an axial direction of the blood vessel in which the dissection occurs based on the medical image including the dissection; The derived major axis length is output together with information regarding the dissociation.

3. A computer program according to claim 1 or claim 2.

4. extracting a medical image on the most distal side and a medical image on the most proximal side in an axial direction of the blood vessel from among a plurality of medical images including the dissection; Deriving the long axis length in the axial direction of the blood vessel in which the dissection has occurred based on the two extracted medical images and the spatial resolution used when generating the plurality of medical images.

4. A computer program according to claim 3.

5. the first learning model is trained to output a classification of dissociation contained in a medical image when the medical image is input; inputting the extracted medical images into the first learning model to identify the classification of dissections contained in the extracted medical images; The identified classification of the dissociation is output together with information relating to the dissociation. A computer program according to any one of claims 1 to 4.

6. outputting information regarding a method for treating the dissociation according to the state of the dissociation or a classification determined by the state; The information about the treatment method includes a treatment device, a procedure for using the treatment device, and the type or frequency of complications that may occur as a result of the treatment. A computer program according to any one of claims 1 to 5.

7. A method for detecting a blood vessel by a catheter inserted into the blood vessel, the method comprising: acquiring a plurality of medical images generated along the axial direction of the blood vessel and storing the images in a memory unit; determining the presence or absence of dissociation contained in the plurality of medical images by inputting the plurality of extracted medical images stored in the storage unit into a first learning model that outputs the presence or absence of dissociation contained in the medical image when the medical image is input; If the medical image includes a dissection, output information regarding the dissection; When the medical image includes a dissection, the angle and the minor axis length of the dissection included in the medical image are derived by inputting the medical image into a second learning model that estimates the angle and the minor axis length of the dissection included in the medical image when the medical image is input; outputting the derived angle and the minor axis length together with information relating to the dissociation; The first learning model and the second learning model are connected in series. An information processing method in which processing is performed by a computer.

8. A method for detecting a blood vessel by a catheter inserted into the blood vessel, the method comprising: acquiring a plurality of medical images generated along the axial direction of the blood vessel and storing the images in a memory unit; determining the presence or absence of dissociation contained in the plurality of medical images by inputting the plurality of extracted medical images stored in the storage unit into a first learning model that outputs the presence or absence of dissociation contained in the medical image when the medical image is input; If the medical image includes a dissection, output information regarding the dissection; outputting the dissection state or complication information determined by the state; The complication information includes a complication content and an occurrence frequency of the complication, The information about the dissection is input to a complication estimation model that outputs the state of the dissection and complication information determined by the state when the information about the dissection is input, thereby obtaining the state of the dissection and the complication information determined by the state. An information processing method in which processing is performed by a computer.

9. an acquisition unit that acquires a plurality of medical images generated along the axial direction of the blood vessel based on a signal detected by a catheter inserted into the blood vessel; a determination unit that determines the presence or absence of dissociation contained in the plurality of medical images by inputting the extracted plurality of medical images into a first learning model that outputs the presence or absence of dissociation contained in the medical images when the medical images are input; an output unit that outputs information about a dissection when the medical image includes the dissection; When the medical image includes a dissection, the angle and the minor axis length of the dissection included in the medical image are derived by inputting the medical image into a second learning model that estimates the angle and the minor axis length of the dissection included in the medical image when the medical image is input; outputting the derived angle and the minor axis length together with information relating to the dissociation; The first learning model and the second learning model are connected in series. Information processing device.

10. An acquisition unit that acquires a plurality of medical images generated along the axial direction of the blood vessel based on signals detected by a catheter inserted into the blood vessel; a determination unit that determines the presence or absence of dissociation contained in the plurality of medical images by inputting the extracted plurality of medical images into a first learning model that outputs the presence or absence of dissociation contained in the medical images when the medical images are input; an output unit that outputs information about a dissection when the medical image includes the dissection; outputting the dissection state or complication information determined by the state; The complication information includes a complication content and an occurrence frequency of the complication, The information about the dissection is input to a complication estimation model that outputs the state of the dissection and complication information determined by the state when the information about the dissection is input, thereby obtaining the state of the dissection and the complication information determined by the state. Information processing device.

Images