Computer program, information processing method, and information processing apparatus

Abstract

This invention provides a computer program that offers useful information to the operator contained within medical images. [Solution] The computer program acquires a medical image generated based on signals detected by a catheter inserted into a tubular organ, derives object information regarding the types of objects contained in the acquired medical image, and executes a process to provide support information to the catheter operator based on the derived object information.

Description

Technical Field

[0001] The present invention relates to a computer program, an information processing method, and an information processing apparatus.

Background Art

[0002] Medical images including intravascular ultrasound (IVUS) images of blood vessels are generated by using a catheter, and intravascular ultrasound examinations are performed. On the other hand, for the purpose of assisting doctors' diagnoses, technologies for adding information to medical images by image processing and machine learning have been developed (for example, Patent Document 1). The feature detection method in the blood vessel image described in Patent Document 1 detects a lumen wall, a stent, etc. included in the blood vessel image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the detection method of Patent Document 1, information corresponding to the objects included in the blood vessel image is not considered in terms of providing information.

[0005] An object of the present disclosure is to provide a computer program or the like that provides useful information to an operator of the catheter according to an object included in a medical image based on the medical image obtained by scanning a lumen organ with a catheter.

Means for Solving the Problems

[0006] The computer program according to this embodiment acquires a medical image generated based on a signal detected by a catheter inserted into a tubular organ, derives object information relating to the types of objects contained in the acquired medical image, and executes a process to provide support information to the operator of the catheter based on the derived object information.

[0007] The information processing method according to this embodiment involves causing a computer to acquire a medical image generated based on a signal detected by a catheter inserted into a tubular organ, to derive object information relating to the types of objects contained in the acquired medical image, and to perform a process to provide support information to the operator of the catheter based on the derived object information.

[0008] The information processing device according to this embodiment includes an acquisition unit that acquires a medical image generated based on a signal detected by a catheter inserted into a tubular organ, an extraction unit that extracts object information relating to the types of objects included in the acquired medical image, and a processing unit that provides support information to the operator of the catheter based on the extracted object information. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide a computer program, etc., that provides useful information to the operator of a catheter based on medical images obtained by scanning a tubular organ with a catheter, according to the objects contained in the medical images. [Brief explanation of the drawing]

[0010] [Figure 1] This is an explanatory diagram showing an example of the configuration of an imaging diagnostic device. [Figure 2] This is an explanatory diagram illustrating the overview of a diagnostic imaging catheter. [Figure 3] This is an explanatory diagram showing a cross-section of a blood vessel through which the sensor part has been inserted. [Figure 4] This is an explanatory diagram illustrating tomographic images. [Figure 5]It is a block diagram showing a configuration example of an image processing apparatus. [Figure 6] It is an explanatory diagram showing an example of a learning model. [Figure 7] It is an explanatory diagram showing an example of a related table. [Figure 8] It is a flowchart showing an information processing procedure by a control unit. [Figure 9] It is a flowchart showing an information providing procedure for stent placement. [Figure 10] It is an explanatory diagram showing an example of display of information regarding specification of a reference part. [Figure 11] It is an explanatory diagram showing an example of display of information regarding stent placement. [Figure 12] It is a flowchart showing an information providing procedure for endpoint determination. [Figure 13] It is a flowchart showing a processing procedure for MSA calculation. [Figure 14] It is an explanatory diagram showing an example of a visualized display of an expanded state in the vicinity of a stent placement part. [Figure 15] It is an explanatory diagram showing an example of display of information regarding a desired expansion diameter. [Figure 16] It is an explanatory diagram showing an example of display of information regarding endpoint determination. [Figure 17] It is an explanatory diagram showing an example of a related table in Embodiment 2. [Figure 18] It is an explanatory diagram showing an example of a combination table. [Figure 19] It is a flowchart showing an information processing procedure by a control unit.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, the image processing method, image processing apparatus, and program of the present disclosure will be described in detail based on the drawings showing their embodiments. In each of the following embodiments, cardiac catheter treatment, which is an intravascular treatment, will be described as an example. However, the luminal organ targeted for catheter treatment is not limited to blood vessels, and may be other luminal organs such as bile ducts, pancreatic ducts, bronchi, intestines, etc.

[0012] (Embodiment 1) FIG. 1 is an explanatory diagram showing a configuration example of the imaging diagnostic apparatus 100. In the present embodiment, an imaging diagnostic apparatus using a dual-type catheter having both functions of intravascular ultrasound diagnosis (IVUS) and optical coherence tomography (OCT) will be described. In the dual-type catheter, there are provided a mode for acquiring an ultrasonic tomographic image only by IVUS, a mode for acquiring an optical coherence tomographic image only by OCT, and a mode for acquiring both tomographic images by IVUS and OCT, and these modes can be switched and used. Hereinafter, the ultrasonic tomographic image and the optical coherence tomographic image are appropriately referred to as an IVUS image and an OCT image, respectively. Further, the IVUS image and the OCT image are collectively referred to as a tomographic image, which corresponds to a medical image.

[0013] The imaging diagnostic apparatus 100 of the present embodiment includes an intravascular inspection apparatus 101, an angiography apparatus 102, an image processing apparatus 3, a display apparatus 4, and an input apparatus 5. The intravascular inspection apparatus 101 includes an imaging diagnostic catheter 1 and an MDU (Motor Drive Unit) 2. The imaging diagnostic catheter 1 is connected to the image processing apparatus 3 via the MDU 2. The display apparatus 4 and the input apparatus 5 are connected to the image processing apparatus 3. The display apparatus 4 is, for example, a liquid crystal display or an organic EL display, etc., and the input apparatus 5 is, for example, a keyboard, a mouse, a trackball, or a microphone, etc. The display apparatus 4 and the input apparatus 5 may be integrally laminated to form a touch panel. Further, the input apparatus 5 and the image processing apparatus 3 may be integrally configured. Furthermore, the input apparatus 5 may be a sensor that accepts gesture input or gaze input, etc.

[0014] The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that obtains an angiographic image, which is a fluoroscopic image of the blood vessels, by imaging the blood vessels from outside the patient's body using X-rays while injecting a contrast agent into the patient's blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and the X-ray sensor receives the X-rays irradiated from the X-ray source to image an X-ray fluoroscopic image of the patient. The diagnostic imaging catheter 1 is equipped with a marker that does not transmit X-rays, and the position of the diagnostic imaging catheter 1 (marker) is visualized in the angiographic image. The angiography device 102 outputs the acquired angiographic image to the image processing device 3, and displays it on the display device 4 via the image processing device 3. The display device 4 displays the angiographic image and the tomographic image acquired using the diagnostic imaging catheter 1.

[0015] Figure 2 is an explanatory diagram illustrating the overview of the imaging diagnostic catheter 1. Note that the upper dashed-dotted area in Figure 2 is an enlargement of the lower dashed-dotted area. The imaging diagnostic catheter 1 has a probe 11 and a connector portion 15 located at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector portion 15. In the following description, the side of the imaging diagnostic catheter 1 furthest from the connector portion 15 will be referred to as the tip side, and the side with the connector portion 15 will be referred to as the proximal end side. The probe 11 is equipped with a catheter sheath 11a, and its tip is provided with a guidewire insertion portion 14 through which a guidewire can be inserted. The guidewire insertion portion 14 constitutes a guidewire lumen and is used to receive a guidewire that has been previously inserted into the blood vessel and to guide the probe 11 to the affected area by the guidewire. The catheter sheath 11a forms a continuous tube from the connection portion with the guidewire insertion portion 14 to the connection portion with the connector portion 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip of the shaft 13.

[0016] The sensor unit 12 has a housing 12d, and the tip end of the housing 12d is formed in a hemispherical shape to suppress friction and snagging with the inner surface of the catheter sheath 11a. Inside the housing 12d are an ultrasonic transmitting / receiving unit 12a (hereinafter referred to as IVUS sensor 12a) that transmits ultrasonic waves into the blood vessel and receives reflected waves from within the blood vessel, and an optical transmitting / receiving unit 12b (hereinafter referred to as OCT sensor 12b) that transmits near-infrared light into the blood vessel and receives reflected light from within the blood vessel. In the example shown in Figure 2, the IVUS sensor 12a is provided on the tip end of the probe 11, and the OCT sensor 12b is provided on the proximal end, and they are arranged at a distance x along the axial direction on the central axis of the shaft 13 (on the dashed line in Figure 2). In the diagnostic imaging catheter 1, the IVUS sensor 12a and the OCT sensor 12b are mounted so that the direction at which ultrasonic waves or near-infrared light are transmitted and received is approximately 90 degrees with respect to the axial direction of the shaft 13 (the radial direction of the shaft 13). Furthermore, it is desirable that the IVUS sensor 12a and OCT sensor 12b be mounted slightly offset from the radial direction so as not to receive reflected waves or reflected light on the inner surface of the catheter sheath 11a. In this embodiment, for example, as indicated by the arrow in Figure 2, the IVUS sensor 12a is mounted with the direction inclined toward the proximal end with respect to the radial direction as the direction of ultrasonic irradiation, and the OCT sensor 12b is mounted with the direction inclined toward the tip with respect to the radial direction as the direction of near-infrared light irradiation.

[0017] The shaft 13 contains an electrical signal cable (not shown) connected to the IVUS sensor 12a and an optical fiber cable (not shown) connected to the OCT sensor 12b. The probe 11 is inserted into the blood vessel from the tip end. The sensor unit 12 and shaft 13 can move forward and backward inside the catheter sheath 11a and can also rotate circumferentially. The sensor unit 12 and shaft 13 rotate around the central axis of the shaft 13 as the axis of rotation. The imaging diagnostic device 100 uses an imaging core composed of the sensor unit 12 and shaft 13 to measure the condition inside the blood vessel by taking ultrasound images (IVUS images) or optical coherence tomography images (OCT images) taken from inside the blood vessel.

[0018] The MDU2 is a drive device to which the probe 11 (imaging catheter 1) is detachably attached by a connector 15. It controls the operation of the imaging catheter 1 inserted into a blood vessel by driving a built-in motor in response to the operation of a medical professional. For example, the MDU2 performs a pullback operation, rotating the sensor unit 12 and shaft 13 inserted into the probe 11 circumferentially while pulling them toward the MDU2 at a constant speed. The sensor unit 12 rotates while moving from the tip to the proximal end due to the pullback operation, and continuously scans the inside of the blood vessel at predetermined time intervals, thereby continuously acquiring multiple transverse images approximately perpendicular to the probe 11 at predetermined intervals. The MDU2 outputs the reflected ultrasound wave data received by the IVUS sensor 12a and the reflected light data received by the OCT sensor 12b to the image processing device 3.

[0019] The image processing device 3 acquires a signal dataset, which is the reflected ultrasound wave data received by the IVUS sensor 12a via the MDU 2, and a signal dataset, which is the reflected light data received by the OCT sensor 12b. The image processing device 3 generates ultrasound line data from the ultrasound signal dataset and constructs an ultrasound tomographic image (IVUS image) of the transverse layer of the blood vessel based on the generated ultrasound line data. The image processing device 3 also generates optical line data from the reflected light signal dataset and constructs an optical tomographic image (OCT image) of the transverse layer of the blood vessel based on the generated optical line data. Here, the signal datasets acquired by the IVUS sensor 12a and the OCT sensor 12b, and the tomographic images constructed from the signal datasets will be explained. Figure 3 is an explanatory diagram showing a cross-section of a blood vessel through which the sensor unit 12 is inserted, and Figure 4 is an explanatory diagram explaining the tomographic image.

[0020] First, using Figure 3, we will explain the operation of the IVUS sensor 12a and OCT sensor 12b within a blood vessel, and the signal dataset (ultrasound line data and optical line data) acquired by the IVUS sensor 12a and OCT sensor 12b. When tomographic imaging is started with the imaging core inserted into the blood vessel, the imaging core rotates around the central axis of the shaft 13 in the direction indicated by the arrow. At this time, the IVUS sensor 12a transmits and receives ultrasound at each rotation angle. Lines 1, 2, ... 512 indicate the direction of ultrasound transmission and reception at each rotation angle. In this embodiment, the IVUS sensor 12a intermittently transmits and receives ultrasound 512 times while rotating 360 degrees (1 rotation) within the blood vessel. Since the IVUS sensor 12a acquires data for one line in the transmission and reception direction with each ultrasound transmission and reception, 512 ultrasound line data lines extending radially from the center of rotation can be obtained during one rotation. The 512 ultrasound line data are dense near the rotation center, but become sparser as they move away from the rotation center. Therefore, the image processing device 3 can generate a two-dimensional ultrasound tomography image (IVUS image) as shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process.

[0021] Similarly, the OCT sensor 12b also transmits and receives measurement light at each rotation angle. Since the OCT sensor 12b also transmits and receives measurement light 512 times while rotating 360 degrees within the blood vessel, 512 optical line data points extending radially from the center of rotation can be obtained during one rotation. With respect to the optical line data, the image processing device 3 can generate a two-dimensional optical coherence tomography (OCT) image similar to the IVUS image shown in Figure 4A by generating pixels in the empty spaces of each line using a well-known interpolation process. That is, the image processing device 3 generates optical line data based on interference light generated by interfering reflected light with, for example, reference light obtained by separating light from a light source within the image processing device 3, and constructs an optical tomography (OCT) image of the transverse layer of the blood vessel based on the generated optical line data.

[0022] The two-dimensional tomographic image generated from these 512 line data points is called one frame of an IVUS image or OCT image. Since the sensor unit 12 scans while moving within the blood vessel, one frame of an IVUS image or OCT image is acquired at each position after one rotation within the movement range. That is, one frame of an IVUS image or OCT image is acquired at each position from the tip to the proximal end of the probe 11 within the movement range, so as shown in Figure 4B, multiple frames of IVUS images or OCT images are acquired within the movement range.

[0023] The diagnostic imaging catheter 1 has X-ray-insensitive markers to confirm the positional relationship between the IVUS image obtained by the IVUS sensor 12a or the OCT image obtained by the OCT sensor 12b and the angiography image obtained by the angiography device 102. In the example shown in Figure 2, marker 14a is provided at the tip of the catheter sheath 11a, for example, at the guidewire insertion section 14, and marker 12c is provided on the shaft 13 side of the sensor section 12. When the diagnostic imaging catheter 1 configured in this way is imaged with X-rays, an angiography image is obtained in which markers 14a and 12c are visualized. The positions of markers 14a and 12c are just examples; marker 12c may be provided on the shaft 13 instead of the sensor section 12, and marker 14a may be provided at a location other than the tip of the catheter sheath 11a.

[0024] Figure 5 is a block diagram showing an example configuration of the image processing device 3. The image processing device 3 is a computer (information processing device) and includes a control unit 31, a main memory unit 32, an input / output interface 33, an auxiliary memory unit 34, and a reading unit 35.

[0025] The control unit 31 is configured using one or more computing devices such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), and TPU (Tensor Processing Unit). The control unit 31 is connected to each hardware component of the image processing device 3 via a bus.

[0026] The main memory unit 32 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores the data necessary for the control unit 31 to perform calculation processing.

[0027] The input / output interface 33 is an interface to which the endovascular examination device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images and OCT images from the endovascular examination device 101 and angiography images from the angiography device 102 via the input / output interface 33. The control unit 31 also outputs medical image signals of the IVUS images, OCT images, or angiography images to the display device 4 via the input / output interface 33, thereby displaying the medical images on the display device 4. Furthermore, the control unit 31 receives information input to the input device 5 via the input / output interface 33.

[0028] A communication unit consisting of, for example, a wireless slave device such as 4G, 5G, or WiFi is connected to the input / output I / F 33, and the image processing device 3 may be connected to an external server such as a cloud server connected to an external network such as the Internet via the communication unit. The control unit 31 may access the external server via the communication unit and the external network, refer to medical data, research paper information, etc. stored (stored) in the storage device included in the external server, and perform processing related to information provision (processing to provide support information). Alternatively, the control unit 31 may perform the processing in this embodiment in cooperation with the external server, for example, by performing inter-process communication.

[0029] The auxiliary storage unit 34 is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary storage unit 34 stores the computer program P (program product) executed by the control unit 31 and various data necessary for the processing of the control unit 31. The auxiliary storage unit 34 may also be an external storage device connected to the image processing device 3. The computer program P (program product) may be written to the auxiliary storage unit 34 during the manufacturing stage of the image processing device 3, or it may be distributed by a remote server device and acquired by the image processing device 3 via communication and stored in the auxiliary storage unit 34. The computer program P (program product) may also be recorded in a readable form on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, and the reading unit 35 may read it from the recording medium 30 and store it in the auxiliary storage unit 34.

[0030] The image processing device 3 may be a multicomputer comprising multiple computers. Alternatively, the image processing device 3 may be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 will be described as a single computer. In this embodiment, an angiography device 102 that acquires two-dimensional angiography images is connected to the image processing device 3, but it is not limited to the angiography device 102 as long as it is a device that images the patient's tubular organs and the diagnostic imaging catheter 1 from multiple directions outside the body.

[0031] In the image processing apparatus 3 of this embodiment, the control unit 31 reads and executes a computer program P stored in the auxiliary storage unit 34 to construct an IVUS image based on the signal dataset received from the IVUS sensor 12a and an OCT image based on the signal dataset received from the OCT sensor 12b. As will be described later, the IVUS sensor 12a and the OCT sensor 12b have a shift in their observation positions at the same imaging timing, so the control unit 31 performs a process to correct the shift in observation positions in the IVUS image and the OCT image. Therefore, the image processing apparatus 3 of this embodiment provides images that are easy to interpret by providing IVUS images and OCT images with aligned observation positions.

[0032] In this embodiment, the imaging catheter is described as a dual-type catheter equipped with both intravascular ultrasound (IVUS) and optical coherence tomography (OCT) functions, but is not limited to this. The imaging catheter may also be a single-type catheter equipped with either intravascular ultrasound (IVUS) or optical coherence tomography (OCT) functions. Hereafter, in this embodiment, the imaging catheter is assumed to have intravascular ultrasound (IVUS) functionality, and the description will be based on IVUS images generated by this IVUS function. However, in the description of this embodiment, the medical image is not limited to IVUS images, and OCT images may be used as medical images to perform the processing of this embodiment.

[0033] Figure 6 is an explanatory diagram showing an example of the learning model 341. The learning model 341 is a neural network that performs, for example, object detection, semantic segmentation, or instance segmentation. Based on each IVUS image in the input IVUS image set, the learning model 341 outputs whether or not an object such as a stent or plaque is contained in the IVUS image (presence or absence), and if an object is contained (presence or absence), the type (class) of the object, the region in the IVUS image, and the estimation accuracy (score).

[0034] The learning model 341 is composed of, for example, a pre-trained convolutional neural network (CNN) using deep learning. The learning model 341 has, for example, an input layer 341a into which a medical image such as an IVUS image is input, an intermediate layer 341b that extracts image features, and an output layer 341c that outputs information indicating the position and type of objects contained in the medical image. The input layer 341a of the learning model 341 has multiple neurons that accept input of the pixel value of each pixel contained in the medical image, and passes the input pixel value to the intermediate layer 341b. The intermediate layer 341b has a configuration in which a convolutional layer that convolves the pixel value of each pixel input to the input layer 341a and a pooling layer that maps the pixel value convolved in the convolutional layer are alternately connected, and extracts image features while compressing the pixel information of the medical image. The intermediate layer 341b passes the extracted features to the output layer 341c. The output layer 341c has one or more neurons that output the position, range, and type of the image region of an object contained in the image. Although the learning model 341 is assumed to be a CNN, the configuration of the learning model 341 is not limited to a CNN. The learning model 341 may be a pre-trained model with a configuration such as a neural network other than a CNN, an SVM (Support Vector Machine), a Bayesian network, or a regression tree. Alternatively, the learning model 341 may perform object recognition by inputting the image features output from the hidden layer into an SVM (support vector machine).

[0035] The learning model 341 can be generated by preparing training data that associates medical images containing objects such as the epicardium, collateral branches, veins, guidewires, stents, plaques prolapsed into stents, lipid plaques, fibrous plaques, calcified areas, vascular dissections, thrombi, and hematomas with labels indicating the location (region) and type of each object, and then using this training data to machine-learn an untrained neural network. With the learning model 341 configured in this way, by inputting a medical image such as an IVUS image into the learning model 341, information indicating the location and type of objects contained in the medical image can be obtained. If the medical image does not contain objects, the learning model 341 does not output information indicating the location and type. Therefore, by using the learning model 341, the control unit 31 can obtain whether or not objects are contained in the medical image input to the learning model 341, and if so, the type (class), location (region in the medical image), and estimation accuracy (score) of the object.

[0036] Based on this information obtained from the learning model 341, the control unit 31 derives object information regarding the presence and type of objects included in the IVUS image. Alternatively, the control unit 31 may derive the object information by using the information obtained from the learning model 341 itself as object information.

[0037] Figure 7 is an explanatory diagram showing an example of an association table. The auxiliary storage unit 34 of the image processing device 3 stores the association between object types and support information, for example, as an association table in table format. The association table includes, for example, object type, presence / absence determination, and support information (launch application) as management items (fields) of the table.

[0038] The object type management field stores the object type (name), such as stents, calcified areas, plaques, vascular dissections, and bypass surgery scars. The presence / absence determination management field stores whether each object type is present or absent. The support information (launched application) management field stores the content of the support information depending on the presence or absence of the object type stored in the same record, or the name of the application (launched application) that provides the support information.

[0039] The control unit 31 can efficiently determine support information (launched application) corresponding to the object information by comparing the association table stored in the memory unit with the object information derived using the learning model 341. For example, when the object information relates to a stent, if the object information indicates that a stent is present, the control unit 31 performs a provision process (executes the endpoint determination application) to provide support information related to endpoint determination. If the object information indicates that a stent is not present, the control unit 31 performs a provision process (executes the stent placement application) to provide support information related to stent placement.

[0040] Figure 8 is a flowchart showing the information processing procedure by the control unit 31. The control unit 31 of the image processing device 3 performs the following processing based on the input data output from the input device 5 in response to the operation of the operator of the diagnostic imaging catheter 1, such as a physician.

[0041] The control unit 31 acquires IVUS images (S11). The control unit 31 acquires a medical image consisting of these IVUS images by reading the group of IVUS images obtained by pullback.

[0042] The control unit 31 derives object information regarding the presence and type of objects contained in the IVUS image (S12). For example, the control unit 31 inputs the acquired IVUS image set to the learning model 341 and derives object information based on the presence and type of objects estimated by the learning model 341. The learning model 341 is composed of, for example, a neural network that performs object detection, semantic segmentation, or instance segmentation. Based on each IVUS image in the input IVUS image set, the learning model 341 outputs whether or not an object such as a stent or plaque is contained in the IVUS image (presence or absence), and if an object is contained (presence or absence), the type (class) of the object, the region in the IVUS image, and the estimation accuracy (score).

[0043] The control unit 31 derives object information in the IVUS image based on the estimation results (presence and type of objects) output from the learning model 341. As a result, the object information includes the presence and type of objects contained in the IVUS image, which is the source data for the object information. The object information may be generated as, for example, an XML file, and the presence or absence of each individual object type may be attached (tagged) to all object types that are the target of estimation by the learning model 341. As a result, the control unit 31 can determine, for example, whether or not a stent is present in the IVUS image, that is, whether or not a stent is placed in the blood vessel. In this embodiment, the control unit 31 derives object information in the IVUS image using the learning model 341, but is not limited to this. The control unit 31 may also determine the presence and type of objects contained in the IVUS image using image analysis means such as edge detection and pattern matching, and derive object information based on the determination result.

[0044] The control unit 31 receives input from the operator regarding the situation assessment (S13). The control unit 31 receives input from the operator of the diagnostic imaging catheter 1, such as a physician, regarding the progress of the surgery or the patient's condition, etc. The control unit 31 determines the support information to be provided based on the object information, etc., and performs the processing to provide the support information (S14). The control unit 31 determines the support information to be provided based on the derived object information and the received information regarding the situation assessment, and performs the processing to provide the support information.

[0045] For example, if the input situation determination relates to a stent, the control unit 31 determines, based on the object information derived from the IVUS image, whether the stent is present or absent, i.e., whether it is before or after stent placement, and decides to provide support information according to the determination result. The provision of such support information includes a form of provision in which the support information itself is superimposed on the screen of the display device 4, and the execution of an application (startup application) that performs calculation processing, etc., to generate and present the support information.

[0046] Before stent placement, the control unit 31 determines support information related to stent placement as the support information to be provided, and for example, launches an application (stent placement app) to assist in determining the stent size and predicting complications. After stent placement, the control unit 31 determines support information related to endpoint determination as the support information to be provided, and for example, launches an application (endpoint determination app) to assist in endpoint determination and predicting complications. The control unit 31 may also refer to the related table stored in the auxiliary storage unit 34 and determine the support information (launched app) according to the presence or absence of each type of individual object included in the object information.

[0047] In the illustration of this embodiment, the flow of processing related to the provision of support information (startup application) for each case where the presence or absence and type of object indicated in the object information is the presence or absence of a stent (after implantation, before implantation) will be explained as an example. This flow is illustrative, and the control unit 31 performs a provision process (execution of the startup application) to provide predefined support information according to the presence or absence of each type of object exemplified in the related table, such as the presence or absence of calcification, the presence or absence of plaque, the presence or absence of vascular dissection, the presence or absence of bypass surgery scars, etc. In performing the provision process (execution of the startup application) to provide support information according to the presence or absence and type of object indicated in the object information, the program executed by the control unit 31 may perform branching processing according to the presence or absence and type of object, for example, using a case statement.

[0048] In the example of the related table, the support information (startup application) defined according to the presence or absence of each type of object is not limited to a single instance, but may also have multiple instances of support information (startup applications) defined. In this case, the provision process (execution of the startup application) may be performed for all of the multiple instances of support information (startup applications), or the names of these multiple instances of support information (startup applications) may be displayed in a list format on the display device 4, and the selection of any of the support information (startup applications) may be accepted, after which the provision process (execution of the startup application) of the selected instance of support information (startup application) may be performed.

[0049] In this embodiment, the control unit 31 determines support information based on object information and information related to situation judgment, but it is not limited to this, and the control unit 31 may determine support information based only on object information. That is, it may be possible to eliminate the need to process input regarding situation judgment by the operator, determine the support information to be provided based only on object information derived from the IVUS image, and perform provisioning processing such as launching an application to provide said support information.

[0050] Figure 9 is a flowchart illustrating the procedure for providing information regarding stent placement. This flowchart explains the process of providing support information related to stent placement (launched application: stent placement app) based on Figure 9.

[0051] The control unit 31 acquires an IVUS image before stent placement (S101). The control unit 31 acquires multiple IVUS images before stent placement, which constitute one pullback. These IVUS images may be the same IVUS images used to derive object information. If multiple IVUS images (a group of IVUS images) are used to derive object information, the control unit 31 may acquire any of the IVUS images included in that group of IVUS images.

[0052] The control unit 31 calculates the plaque bathen (S102). The control unit 31, for example, uses a learning model 341 to segment the acquired IVUS image into lumen and vessels and calculates the plaque bathen. Alternatively, by segmenting the lumen and vessels, the area of ​​each (cross-sectional area in the tomographic diagram) may be calculated, and the plaque bathen or plaque area may be calculated by dividing or subtracting the area of ​​the vessel from the area of ​​the lumen.

[0053] The control unit 31 determines whether the plaque bathen is above a predetermined threshold (S103). By determining whether the plaque bathen is above a predetermined threshold, the control unit 31 classifies the plaque bathen based on the threshold. The control unit 31 classifies the calculated area of ​​the plaque bathen based on a predetermined threshold, such as 40%, 50%, or 60%. The threshold may be configured to allow for multiple settings.

[0054] If the threshold is above a predetermined threshold (S103: YES), the control unit 31 groups the frames (IVUS images) above the threshold (S104). The control unit 31 groups the frames (IVUS images) above the plaque-burden threshold as lesions. If the lesions are spaced apart and scattered, they may be grouped individually (L1, L2, L3, ...). However, if the spacing between the groups is between 0.1 and 3 mm or less, they may be grouped together.

[0055] The control unit 31 identifies the group containing the maximum value of the plaque bathen as the lesion (S105). The control unit 31 identifies the group containing the maximum value of the plaque bathen, i.e., the area where the lumen diameter is at its minimum, as the lesion.

[0056] If the value is not above a predetermined threshold (S103:NO), i.e., below the predetermined threshold, the control unit 31 groups the frames (IVUS images) below the threshold (S1031). If the value is below the predetermined threshold, the control unit 31 groups the frames (IVUS images) below the threshold as references. If the plaque bathens that become references are scattered at intervals, they may be grouped individually (R1, R2, R3, ...). However, if the interval (spacing distance) between the groups is 0.1 to 3 mm or less, they may be grouped together.

[0057] The control unit 31 identifies distal and proximal groups as reference units for the lesion (S106). For example, the control unit 31 classifies all IVUS images according to the determination result of whether or not they are above the plaque barden threshold, and then identifies distal and proximal groups as reference units for the lesion. The control unit 31 identifies the groups located distal and proximal to the identified lesion from among the grouped reference units as reference units for comparison with the lesion.

[0058] Figure 10 is an explanatory diagram showing an example of how information for identifying a reference area can be displayed. In this example, a graph of the average lumen diameter and a graph of plaque bathen (PB) are displayed side by side, one above the other. The horizontal axis represents the length of the blood vessel (axial length). When the threshold for plaque bathen (PB) is 50%, the area exceeding this threshold is identified as a lesion. For this lesion, the locations where the average lumen diameter is maximum in areas within 10 mm distal and proximal to the lesion are identified as the distal reference area and the proximal reference area, respectively. By displaying this information, support (assistance) can be provided to the operator in identifying a reference area. As shown in the figure in this embodiment, the lesion may be, for example, a group of 3 mm or more that have a plaque bathen (PB) of 50% or more. The reference area may be the part with the maximum average lumen diameter within 10 mm before and after the lesion. If there are large side branches within the blood vessel and the vessel diameter changes significantly, a reference point may be identified between the lesion and the side branch. In identifying this reference point, the image shown in the figure may be displayed on the display device 4, and the operator may be allowed to make corrections. Furthermore, when displaying the image on the display device 4, the location of the large side branch may be indicated.

[0059] The control unit 31 calculates the vessel diameter, lumen diameter, and area of ​​the distal and proximal reference sections (S107). The control unit 31 calculates the vessel diameter (EEM), lumen diameter, and area of ​​the distal and proximal reference sections. In this case, the length between the reference sections, i.e., the length from the distal reference section to the proximal reference section, may be set to a maximum of 10 mm, for example.

[0060] The control unit 31 displays support information (S108). As an example, the control unit 31 outputs support information regarding stent placement to the display device 4, as shown in the figure in this embodiment, and displays the support information on the display device 4. Figure 11 is an explanatory diagram showing an example of the display of information regarding stent placement. In this display example, a transverse view, which is a tomographic view of the blood vessel in the axial direction, and a longitudinal view, which is a tomographic view of the blood vessel in the radial direction, are displayed side by side. That is, the support information regarding stent placement includes multiple longitudinal views (radial cross-sectional area of ​​the blood vessel) from IVUS images and a transverse view (axial cross-sectional area of ​​the blood vessel) formed by linking these longitudinal views. In the transverse view, the distal reference area (Ref. Distal) and the proximal reference area (Ref. Proximal) are shown, and the MLA (minimum lumen area) located between these reference areas is shown. By displaying such information, support (assistance) regarding stent placement can be provided to the operator.

[0061] Figure 12 is a flowchart showing the information provision procedure for endpoint determination. Figure 13 is a flowchart showing the processing procedure for calculating the MSA. In this flow, the provision process for providing support information regarding endpoint determination (launched application: endpoint determination APP) will be explained based on Figure 12.

[0062] The control unit 31 acquires IVUS images after stent placement (S111). The control unit 31 acquires multiple IVUS images after stent placement, which constitute one pullback. The control unit 31 determines the presence or absence of a stent for each of the acquired multiple IVUS images (S112). The control unit 31 determines the presence or absence of a stent for multiple IVUS images (one pullback) by, for example, using a learning model 341 with object detection capabilities or image analysis processing such as edge detection and pattern patching.

[0063] If no stent is used (S112: YES), the control unit 31 performs segmentation of the lumen and vessels in the IVUS image without a stent (S113). The control unit 31 performs segmentation of the lumen and vessels in the IVUS image without a stent, for example, using a learning model 341 that has a segmentation function. The control unit 31 calculates a representative value of the diameter or area of ​​the vessel or lumen (S114). The control unit 31 calculates a representative value of the diameter or area based on the segmented lumen and vessels.

[0064] If a stent is present (S112:NO), the control unit 31 performs stent segmentation on the IVUS image with the stent (S115). The control unit 31 performs stent segmentation on the IVUS image with the stent, for example, using a learning model 341 that has a segmentation function. The control unit 31 calculates a representative value of the diameter or area of ​​the stent lumen (S116). The control unit 31 calculates a representative value of the diameter or area of ​​the stent lumen based on the segmented stent.

[0065] The control unit 31 derives the expansion state near the stent implantation site (S117). Based on the calculated representative values ​​of the diameter or area of ​​the blood vessel or lumen, and the representative values ​​of the diameter or area of ​​the stent lumen, the control unit 31 derives the expansion state near the stent implantation site and visualizes it by outputting and displaying the expansion state on the display device 4. As shown in the figure in this embodiment, the expansion state near the stent implantation site displayed (visualized) on the display device 4 may, for example, be displayed by coloring the area where the stent is provided in a cross-sectional view.

[0066] Figure 14 is an explanatory diagram showing an example of a visualized display of the expansion state near the stent placement site. In this display example, graphs of the vessel, lumen, and diameter and area of ​​the stent in the stent-placed state are displayed side by side. The horizontal axis represents the length of the vessel (length in the axial direction). The position of the Minimum Stent Area (MAS) is indicated in these graphs. By displaying this information, it is possible to provide support (assistance) to the operator in understanding the expansion state near the stent placement site. In deriving the expansion state near the stent placement site, the control unit 31 calculates the Minimum Stent Area (MSA) at the stent placement site. The processing procedure for calculating the MSA will be described later.

[0067] The control unit 31 derives the planned expansion diameter (S118). The control unit 31, for example, refers to a pre-planned plan stored in the auxiliary storage unit 34 and derives the desired planned expansion diameter based on the stent expansion diameter included in the pre-planned plan. The control unit 31 may accept input from the operator when deriving the planned expansion diameter. The control unit 31 may display a graph showing the derived desired (planned) expansion diameter superimposed on an image showing the expansion state near the stent implantation area.

[0068] The control unit 31 derives an expansion diameter based on evidence information (S119). The control unit 31 derives a desirable (desired) expansion diameter by referring to evidence information such as research paper information pre-stored in the auxiliary storage unit 34, for example. The control unit 31 may accept input of the operator's own index when deriving the desirable (desired) expansion diameter. The control unit 31 may display a graph showing the derived desirable (desired) expansion diameter superimposed on an image showing the expansion state near the stent implantation site.

[0069] The control unit 31 displays information corresponding to the derived expansion diameter (S120). The control unit 31 displays information in a different manner depending on whether the derived expansion diameter is less than or equal to the desired diameter or area, for example, by changing the display color. Figure 15 is an explanatory diagram showing an example of displaying information regarding the desired expansion diameter. In this display example, a graph of the desired stent diameter and area in a vessel with the stent in place is displayed side by side. The horizontal axis represents the length of the vessel (length in the axial direction). By displaying such information, the operator can be provided with support (assistance) in understanding the desired stent diameter and area.

[0070] The control unit 31 receives the operator's decision on whether or not post-dilation is necessary (S121). The control unit 31 receives input regarding the operator's decision on whether or not post-dilation is necessary. The control unit 31 derives the recommended dilation pressure based on the dilation diameter during post-dilation (S122). Based on the dilation diameter during post-dilation, the control unit 31 derives the recommended dilation pressure by, for example, referring to a compliance chart stored in the auxiliary storage unit 34 and identifying the recommended dilation pressure included in the compliance chart. The control unit 31 displays the derived recommended dilation pressure by superimposing it on, for example, an image showing the dilation state near the stent implantation site. Figure 16 is an explanatory diagram showing an example of the display of information related to endpoint determination. In this display example, a transverse tomography, which is a tomography of the blood vessel in the axial direction, and a longitudinal tomography, which is a tomography of the blood vessel in the radial direction, are displayed side by side. The transverse tomography shows the position of the MAS. By displaying such information, support (assistance) can be provided to the operator regarding endpoint determination.

[0071] The procedure for calculating the Minimum Stent Area (MSA) of the stent implantation site will be explained with reference to Figure 13. This procedure for calculating the MSA may be performed, for example, as a subroutine in the process of S117 (derivation of the expanded state near the stent implantation site).

[0072] The control unit 31 acquires the IVUS image (M001). The control unit 31 acquires the IVUS image by reading one pullback portion. The control unit 31 determines the presence or absence of a stent (M002). The control unit 31 determines the presence or absence of a stent in each frame (IVUS image) and stores the processing result for each frame (IVUS image) in an array (array type variable), for example.

[0073] The control unit 31 accepts corrections regarding the presence or absence of a stent (M003). The control unit 31 accepts corrections regarding the presence or absence of a stent based, for example, on the operator's actions. The control unit 31 calculates the stent area (M004). The control unit 31 calculates (identifies) the stent area by processing each frame (IVUS image) that contains the stent. In performing this processing, the control unit 31 may calculate the lumen diameter and area using a learning model 341 with segmentation functionality. Alternatively, the control unit 31 may calculate the minor and major diameters of the lumen diameter and derive the eccentricity (minor diameter / major diameter) by dividing the minor diameter by the major diameter.

[0074] The control unit 31 calculates the MSA (M005). The control unit 31 calculates the MSA in the identified stent region based on the calculated lumen diameter, area, and eccentricity. The control unit 31 determines the risk of stent thrombosis (M006). The control unit 31 may, for example, function as an MSA detector and determine whether it is greater than 5.5 square mm (MSA > 5.5 [mm^2]), and if it is greater than 5.5 square mm, it may determine that there is no risk of stent thrombosis.

[0075] According to this embodiment, the image processing device 3 derives object information regarding the presence and type of objects contained in a medical image, such as an IVUS image, acquired using the diagnostic imaging catheter 1. Based on the derived object information, the image processing device 3 performs a provision process to provide support information to the operator of the diagnostic imaging catheter 1, such as a physician. Thus, it can provide the operator with appropriate support information according to the presence and type of objects contained in the medical image generated using the diagnostic imaging catheter 1.

[0076] According to this embodiment, the image processing device 3 inputs a medical image to the learning model 341 and derives object information using the type of object estimated by the learning model 341. Since the learning model 341 is trained to estimate objects contained in a medical image by inputting a medical image, the image processing device 3 can efficiently obtain whether or not an object is contained in the medical image, and if an object is contained, the type of that object.

[0077] According to this embodiment, the types of objects identified as being included in the medical image include at least one of the following: epicardium, collateral branches, veins, guidewires, stents, plaques prolapsed into stents, lipid plaques, fibrous plaques, calcified areas, vascular dissections, thrombi, and hematomas. Therefore, appropriate support information can be provided to the operator depending on the object which may be a region of interest, such as a lesion in a tubular organ.

[0078] According to this embodiment, the support information provision process performed according to object information includes a provision process for providing support information related to stent placement and endpoint determination. Therefore, if the type of object in question is a stent, appropriate support information can be provided to the operator according to the presence or absence of the stent.

[0079] (Embodiment 2) Figure 17 is an explanatory diagram showing an example of an association table in Embodiment 2. Figure 18 is an explanatory diagram showing an example of a combination table in Embodiment 2. The association table in Embodiment 2, like Embodiment 1, includes, for example, object type and existence / absence determination as management items (fields) of the table, and further includes determination flag values.

[0080] The object type management item (field) stores the object type (name), such as a stent, similar to Embodiment 1. The presence / absence determination management item (field) stores the presence or absence of each object type, similar to Embodiment 1.

[0081] The management item (field) for the determination flag value stores a determination flag value corresponding to the presence or absence of an object type stored in the same record. For example, this determination flag value includes a type flag indicating the object type and a presence / absence flag indicating the presence or absence of the object, and is composed of a value that concatenates these type flag and presence / absence flag. In this embodiment, letters such as A (stent) and B (calcified part) indicate the object type (type flag), and the numbers 1 (present) and 0 (absent) indicate the presence or absence of the object (present / absence flag). By using such determination flag values, it is possible to uniquely determine the value indicating the presence or absence of each object type.

[0082] The combination table includes, for example, a combination code, support information (launched application), and the number of support information items as its management items (fields). The management item (field) for the combination code stores information (combination code) indicating the combination of judgment flag values ​​shown in the related table.

[0083] The combination code stores a string formed by concatenating judgment flag values ​​indicating the presence (1) or absence (0) of each object type. For example, if the combination code is A0:B0:C0:D0:E0, it indicates that all objects represented by A through E are absent in the IVUS image. If the combination code is A1:B0:C0:D0:E0, it indicates that only object A (stent) is present. If the combination code is A1:B1:C0:D0:E0, it indicates that only A (stent) and B (calcified area) are present. In this way, even if the IVUS image contains multiple types of objects, the combination code can be used to uniquely determine the combination of presence or absence for each object type.

[0084] The management items (fields) for support information (launched applications) may store, for example, the content of the support information corresponding to the combination code stored in the same record, or the name of the application (name of the launched application) for providing the support information. The stored support information (launched applications) are not limited to one, but may be two or more. Alternatively, depending on the combination code, information indicating that there is no support information (launched application) to store may be stored. For example, if the combination code is A0:B0:C0:D0:E0, it can be said that the IVUS image does not contain any type of object, and the blood vessels shown in the IVUS image are healthy, and the control unit 31 may not perform any processing to provide support information (launched applications). For example, if the combination code is A0:B0:C0:D1:E1, it can be shown that the IVUS image contains multiple objects, and multiple pieces of support information (launched applications) corresponding to these multiple objects may be stored.

[0085] The control unit 31 may perform provision processing (execution of the startup application) for all of the multiple support information (startup applications). Alternatively, the control unit 31 may accept the selection of one of the multiple support information (startup applications) and perform provision processing (execution of the startup application) for the selected support information (startup application). For example, the control unit 31 can efficiently determine the support information (startup application) by deriving object information in accordance with the format of the combination code, and then comparing the derived object information with the combination table.

[0086] The management item (field) for the number of support information items stores, for example, the number of support information items (launched applications) (number of types) stored in the same record. The control unit 31 may also change the display manner when executing the support information (launched application) provision process depending on the number stored in the management item (field) for the number of support information items.

[0087] Figure 19 is a flowchart showing the information processing procedure by the control unit 31. The control unit 31 of the image processing device 3 performs the following processing based on the input data output from the input device 5 in response to the operation of the operator of the diagnostic imaging catheter 1, such as a physician.

[0088] The control unit 31 acquires an IVUS image (S21). The control unit 31 derives object information regarding the presence and type of objects included in the IVUS image (S22). The control unit 31 performs the processing from S21 to S22 in the same manner as S11 to S12 in Embodiment 1.

[0089] The control unit 31 determines the support information to be provided based on object information, etc. (S23). The control unit 31 may, for example, derive object information based on the presence or absence of all types of objects defined in the association table stored in the auxiliary storage unit 34 by referring to the association table stored in the auxiliary storage unit 34. The learning model 341 has been trained on all of these types of objects, and by inputting IVUS images into the learning model 341, the control unit 31 can obtain the presence or absence of all types of objects defined in the association table. The control unit 31 determines the support information to be provided by comparing the object information thus derived with, for example, the combination table stored in the auxiliary storage unit 34, and thereby the number of pieces of support information is determined.

[0090] The control unit 31 determines whether there are multiple types of support information to be provided (S24). The control unit 31 determines whether there are multiple types of support information determined according to the object information by referring to a combination table stored in the auxiliary storage unit 34, for example.

[0091] If there are multiple types of support information to provide (S24:YES), the control unit 31 displays the names of the multiple support information on the display device 4 (S25). The control unit 31 accepts the selection of any of the support information (S26). The control unit 31 displays the names of these multiple support information (launch applications) on the display device 4, for example in a list format, and accepts the selection of any of the support information (launch applications) by the user in response to the user's operation using the touch panel function or input device 5 provided on the display device 4.

[0092] The control unit 31 performs the support information provision process (S27). After the processing in S26, or if there are no more than one type of support information to provide (S24:NO), the control unit 31 performs the support information provision process (execution of the startup application) selected in the S26 process. If there are no more than one type of support information to provide (S24:NO), i.e., if there is only one type of support information to provide, the control unit 31 performs the support information provision process (execution of the startup application) determined in the S23 process. The control unit 31 performs the support information provision process (execution of the startup application) for the selected or determined support information (startup application), for example, a stent placement application or an endpoint determination application, similar to Embodiment 1.

[0093] According to this embodiment, the association table, which links object types with support information, is stored in a predetermined storage area accessible by the control unit 31 of the image processing device 3, such as a memory unit. Therefore, the control unit 31 can efficiently determine the support information corresponding to the object type by referring to the association table stored in the memory unit. The association table includes not only support information corresponding to the presence or absence of a specific type of object, but also support information corresponding to combinations of the presence or absence of multiple types of objects. Therefore, appropriate support information can be provided to the operator not only according to the presence or absence of a single type of object, but also according to combinations of the presence or absence of multiple types of objects.

[0094] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The technical features described in each embodiment can be combined with each other, and the scope of the invention is intended to include all modifications within the claims and equivalents thereof. [Explanation of Symbols]

[0095] 1. Diagnostic imaging catheter 11 probes 11a Catheter Sheath 12 Sensor section 12a Ultrasonic Transceiver Unit 12a IVUS sensor 12b Optical Transceiver Unit 12b OCT sensor 12c marker 12d Housing 13 shafts 14 Guide wire insertion section 14a Marker 15 Connector section 2 MDU 3. Image Processing Device (Information Processing Device) 30 Recording media 31 Control Unit 32 Main memory 33 Input / Output Interfaces 34 Auxiliary storage 341 Learning Models 35 Reading section 4 Display device 5 Input devices 100 Imaging diagnostic equipment 101 Intravascular Examination Device 102 Angiography equipment

Claims

1. On the computer, Medical images are obtained based on signals detected by a catheter inserted into a tubular organ. Object information regarding the types of objects contained in the acquired medical image is derived, Based on the derived object information, support information is provided to the operator of the catheter. A computer program designed to execute a process.

2. Based on the derived object information, the application to be launched is determined. By executing the determined application, support information corresponding to the object information is provided. The computer program according to claim 1.

3. By inputting the acquired medical image into a learning model that estimates the objects contained in a medical image, the type of object contained in the medical image is identified. The computer program according to claim 1 or claim 2.

4. The types of objects include at least one of the following: epicardium, side branches, veins, guidewires, stents, plaques prolapsed into the stent, lipid plaques, fibrous plaques, calcified areas, vascular dissections, thrombi, and hematomas. A computer program according to any one of claims 1 to 3.

5. If a medical image contains a stent, run an application that provides support information for endpoint determination. If the medical image does not include a stent, run an application that provides support information regarding stent placement. A computer program according to any one of claims 1 to 4.

6. By referring to a related table that associates the types of objects included in the medical image with supporting information, the supporting information corresponding to the derived object information is determined. A computer program according to any one of claims 1 to 5.

7. Based on the derived object information, multiple candidate support information items are output. The system accepts the selection of one of the multiple candidate support information items output. Provide selected support information. A computer program according to any one of claims 1 to 6.

8. The aforementioned tubular organ is a blood vessel. A computer program according to any one of claims 1 to 7.

9. On the computer, Medical images are obtained based on signals detected by a catheter inserted into a tubular organ. Object information regarding the types of objects contained in the acquired medical image is derived, Based on the derived object information, support information is provided to the operator of the catheter. An information processing method that executes a process.

10. An acquisition unit that acquires medical images generated based on signals detected by a catheter inserted into a tubular organ, A derivation unit that derives object information relating to the types of objects contained in the acquired medical image, A processing unit that provides support information to the operator of the catheter based on the derived object information. An information processing device equipped with the following features.

Images