IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND IMAGE PROCESSING SYSTEM

Abstract

To provide a technology that enables medical personnel to concentrate on work in an area of ​​interest and supports the determination of the area of ​​interest during catheter examination or treatment of blood vessels. [Solution] In an image processing device (1), an image acquisition unit (110) acquires an X-ray image created based on X-ray absorption rates, with the image including at least blood vessels and intravascular inspection or treatment devices as subjects. A region of interest acquisition unit (111) accepts one or more regions including at least a portion of the device included in the X-ray image as regions of interest. A tracking unit (112) tracks each region of interest in the X-ray image. A notification unit (113) notifies the user when at least one of the regions of interest satisfies a condition defined for each region of interest. The system also includes functions to record actions during surgery and display a list of instruments to be used.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and an image processing system, and more particularly to an image processing technique for use in the inspection or treatment of blood vessels. [Background technology]

[0002] In recent years, catheter surgery has become common for the examination and treatment of blood vessels throughout the body, such as the brain and heart, in which a medical professional such as a doctor passes a catheter through the blood vessels of a subject and performs various procedures while displaying the position of the catheter on an X-ray image. In this catheter surgery, a medical professional generally passes a medical device such as a catheter through the subject's groin or upper arm, and advances the device to the area of ​​interest.

[0003] For example, during examination or treatment of cerebrovascular disease, medical personnel advance a catheter from the subject's groin or brachial artery, through the aortic arch and carotid artery, and into the cerebral artery. Since medical personnel need a lot of experience and training to perform these procedures appropriately, for example, Patent Document 1 discloses a simulation system for improving the efficiency of surgical procedures performed using a catheter system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 139894 Summary of the Invention [Problem to be solved by the invention]

[0005] In vascular examination or treatment, a medical professional passes a guiding catheter to a target site, and then passes multiple devices for examination or treatment, such as a guidewire, a stent, a balloon, a microcatheter, a liquid embolic material, a filter, and a delivery wire of an embolization coil for embolizing an aneurysm, through the guiding catheter. That is, in actual examination or treatment, a medical professional needs to pay attention to the movements and positions of multiple devices not only when passing a guiding catheter to a target site, but also during treatment. On the other hand, in order to achieve the original purpose of the examination or treatment, a medical professional needs to concentrate on the work at the target site.

[0006] The present invention has been made in consideration of these points, and aims to provide a technology that enables medical professionals to concentrate on working on areas of interest and supports their judgment on areas of interest during vascular catheter examination or treatment. [Means for solving the problem]

[0007] A first aspect of the present invention is an image processing device comprising: an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest.

[0008] When a region including the tip of a catheter such as a guiding catheter or the tip of a guidewire is set as the region of interest, the notification unit may notify the user on the condition that the region of interest disappears from the image.

[0009] The notification unit may notify the user on condition that when a region including the tip of a catheter such as a guiding catheter or the tip of a guidewire is set as the region of interest, the distance between the region of interest and the edge of the image is less than a predetermined threshold distance.

[0010] The notification unit may notify the user on condition that at least one of a moving distance, a moving speed, and an acceleration of the region of interest in the image exceeds a predetermined threshold.

[0011] The notification unit may cause a display device that displays the image to display the distance between the region of interest and an edge of the image.

[0012] The notification unit may change a manner in which the distance is displayed on the display device depending on a distance between the region of interest and an edge of the image.

[0013] The notification unit may notify the user on condition that a value obtained by dividing the distance between the region of interest and an edge of the image by a moving speed of the region of interest within the image is less than a predetermined threshold.

[0014] The image processing device may further include a marker detection unit that detects a marker provided on a delivery wire of the embolization coil approaching a region of interest set on a part of a guiding catheter that guides the delivery wire, and the tracking unit may further track the detected marker, and the notification unit may notify the user of the timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap.

[0015] The notification unit may notify the user when the marker passes through the region of interest.

[0016] The notification unit may cause a display device to display a distance that the marker must move before the embolization coil is disconnected from the delivery wire.

[0017] When a feature amount indicating a shape of the device included in the region of interest satisfies a predetermined condition, the notification unit may notify a user of the image processing device of this fact.

[0018] The feature may be curvature, and the notification unit may notify the user when the curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or when the curvature is changing but the tip does not move.

[0019] The notification unit may notify the user on the condition that the value obtained by subtracting the length of the center line of the blood vessel contained in the image or region of interest from the length of the device contained in the image or region of interest exceeds a predetermined threshold length.

[0020] The notification unit may have functions such as displaying the area of ​​interest in a different color from the image, changing the font, size or color of the displayed text, changing the color of the entire screen of the display device or a portion of it, displaying a graphic on the entire screen of the display device or outside the frame or a portion of it, enlarging the area of ​​interest, or notifying the user by changing the color or size of a mark added to the area of ​​interest.

[0021] The notification unit may use sound or vibration for notification.

[0022] A second aspect of the present invention is an image processing method, in which a processor of an image processing device executes the steps of acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, tracking each of the regions of interest in the image, and, when at least one of the regions of interest satisfies a condition defined for each of the regions of interest, notifying a user of the image processing device of this fact.

[0023] A third aspect of the present invention is a program that causes a computer to perform the following functions: acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the X-ray image as regions of interest, tracking each of the regions of interest in the X-ray image, and notifying a user of the computer when at least one of the regions of interest satisfies a condition defined for each of the regions of interest.

[0024] In order to provide this program or to update a part of the program, a computer-readable recording medium having this program recorded thereon may be provided, or this program may be transmitted over a communication line.

[0025] A fourth aspect of the present invention is an image processing system, which includes the above-mentioned image processing device and an imaging device that captures an image of a person having an intravascular inspection or treatment device inserted therein (an image of a surgical field) and transmits the image to the image processing device.

[0026] Any combination of the components disclosed in this specification, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also effective as aspects of the present invention. Effect of the Invention

[0027] According to the present invention, in vascular catheter examination or treatment, a technique can be provided that enables medical personnel to focus on work in areas of interest and prevents oversights and delayed judgment in areas other than those of interest. [Brief description of the drawings]

[0028] [Figure 1] 1 is a diagram illustrating an external view of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an image processing device according to an embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a region of interest. [Figure 4] FIG. 13 is a diagram for explaining an example of a condition set for a region of interest. [Diagram 5] FIG. 11 is a diagram illustrating an example of a message notified by a notification unit. [Figure 6] FIG. 13 is a diagram for explaining another example of a condition set for a region of interest. [Figure 7] FIG. 13 is a diagram for explaining another example of a condition set for a region of interest. [Figure 8] FIG. 13 is a diagram for explaining the timing of detachment of the embolization coil. [Figure 9] 13 is a diagram for explaining conditions related to the shape of a device that are set in a region of interest. FIG. [Figure 10] 4 is a flowchart illustrating a flow of an image analysis process executed by the image processing device according to the embodiment. [Figure 11] FIG. 1 is a schematic diagram of a typical neural network. [Figure 12] 1 is a diagram illustrating a functional configuration of an image processing device having a replay function according to an embodiment of the present invention; [Figure 13] FIG. 13 is a diagram for explaining an example of a replay display when a notification occurs. [Figure 14] FIG. 13 is a diagram for explaining an example of displaying a replay playback window on a real-time display screen. [Figure 15]1 is a diagram illustrating a functional configuration of an image processing device including a state estimation unit according to an embodiment. [Figure 16] 11A and 11B are diagrams for explaining an example of a display of an estimation result of a state estimation unit according to an embodiment. [Figure 17] 13 is a diagram for explaining an example of estimating a catheter tip position using a 2nd marker in the embodiment. FIG. [Figure 18] FIG. 13 is a diagram for explaining an embodiment in which the results of image analysis are displayed on two screens of different sizes. [Figure 19] 11A and 11B are diagrams for explaining an embodiment in which a screen to be notified is recognized by highlighting the frame of the screen. [Figure 20] 1 is a diagram for explaining the display of a product list of devices for intravascular inspection or treatment (for example, various catheters and coils) according to an embodiment. FIG. [Figure 21] 5A to 5C are diagrams for explaining a process of storing the position and shape of a device according to an embodiment. [Figure 22] 13 is a flowchart for explaining the flow of a catheter tip position estimation process using a second marker, which is executed by an image processing device of an embodiment. [Diagram 23] 11 is a flowchart for explaining a process flow of a replay function executed by the image processing device according to the embodiment. [Figure 24] 10 is a flowchart for explaining the flow of processing for estimating the position of a region of interest outside a frame, which is executed by an image processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] <Outline of the embodiment> 1 is a diagram showing a schematic external view of an image processing system S according to an embodiment. The image processing system S includes an image processing device 1, a display device 2, and an X-ray imaging device 3. Hereinafter, an overview of the embodiment will be described with reference to FIG.

[0030] The X-ray imaging device 3 is a device for capturing an X-ray image of a subject P, who is a person having a device (hereinafter, sometimes simply referred to as a "device") inserted therein for inspecting or treating blood vessels, and transmitting the X-ray image to the image processing device 1. For this reason, the X-ray imaging device 3 includes an X-ray irradiator 30 (a first X-ray irradiator 30a and a second X-ray irradiator 30b) for irradiating X-rays to the subject P, an X-ray detector 31 for detecting the X-rays irradiated by the X-ray irradiator 30, and a bed 32 for supporting the subject P.

[0031] As shown in FIG. 1, the first X-ray irradiator 30a and the second X-ray irradiator 30b can irradiate X-rays to the head of the subject P at different angles of incidence. Here, the X-rays irradiated by the first X-ray irradiator 30a are detected by the first X-ray detector 31a and converted into an X-ray image based on the absorption rate of the X-rays. The X-rays irradiated by the second X-ray irradiator 30b are detected by a second X-ray detector (not shown) and converted into an X-ray image. These X-ray images are displayed on the display device 2. During a treatment of the cerebrovascular region, the positions of the first and second X-ray irradiators are generally fixed, and the area of ​​the image displayed on the display device 2 is fixed.

[0032] The image generated from the X-rays detected by the X-ray detector 31 shows the subject P's blood vessels (which become visible by injecting a contrast agent) and tissues such as bones, as well as various devices used in examining and treating blood vessels (e.g., catheters such as guiding catheters, guide wires, embolization coils, delivery wires for transporting embolization coils to the area of ​​interest, etc.).

[0033] The image processing device 1 according to the embodiment is a device for assisting a user performing a vascular catheter examination or treatment (hereinafter, simply referred to as a "catheter surgery" except when distinguishing between a vascular catheter examination and a vascular catheter treatment). The image processing device 1 recognizes and / or tracks one or more regions of interest that are set in advance in an X-ray image generated based on X-rays detected by an X-ray imaging device 3. The image processing device 1 analyzes the X-ray image, and when the state of any of the regions of interest meets a condition set for each region of interest, notifies a medical professional (hereinafter, simply referred to as a "user") who is a user of the image processing device 1. By setting a region of interest that requires attention other than the region of interest for achieving the purpose of the examination or treatment, the user can concentrate on the work at the site of interest (for example, work such as guiding a microcatheter into an aneurysm, inserting a coil into an aneurysm, expanding a balloon, placing a stent, etc.).

[0034] <Functional configuration of image processing device 1 according to the embodiment> FIG. 2 is a diagram showing a schematic functional configuration of an image processing device 1 according to an embodiment. The image processing device 1 includes a storage unit 10 and a control unit 11. In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

[0035] The memory unit 10 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the image processing device 1, a RAM (Random Access Memory) that serves as the working area of ​​the image processing device 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referenced when the application programs are executed.

[0036] The control unit 11 is a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the image processing device 1, and functions as an image acquisition unit 110, a region of interest acquisition unit 111, a tracking unit 112, a notification unit 113, a marker detection unit 114, and a distance measurement unit 115 by executing programs stored in the memory unit 10.

[0037] 2 shows an example in which the image processing device 1 is configured as a single device. However, the image processing device 1 may be realized by a plurality of computing resources such as processors and memories, for example, as in a cloud computing system. In this case, each unit constituting the control unit 11 is realized by at least one of a plurality of different processors executing a program.

[0038] The image acquisition unit 110 acquires an X-ray image created based on the absorption rate of X-rays, the image including at least the blood vessel of the subject P and an examination or treatment device in the blood vessel as a subject. For example, the X-ray image may include an aneurysm occurring in the blood vessel of the subject P, a stenosis or an infarction of the blood vessel that is of interest to the medical staff. The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device included in the X-ray image as a region of interest. For example, the tip of a guidewire (GW), the tip of a guiding catheter (GC), a catheter marker, a coil, etc. may be set as the region of interest, but multiple regions may be set at the same time, such as the tip of a GC and a GW, the tips of two GWs, or the tip of a GW and a coil. In some aspects, the region of interest may include a blood vessel (a vascular lesion such as a cerebral aneurysm or a stenosis) or a bone.

[0039] 3(a)-(d) are diagrams for explaining a region of interest. Specifically, Fig. 3(a) is a diagram that shows a schematic example of an X-ray image of a blood vessel V. In the example shown in Fig. 3(a), a guidewire, which is a type of device D, is present in the blood vessel V.

[0040] FIG. 3(b) is a diagram showing an example of a candidate region C that is a candidate for the region of interest. The region of interest acquisition unit 111 may include a candidate region detector generated using a known machine learning method such as a neural network (FIG. 11). In FIG. 3(b), a first candidate region C1 (tip of the guiding catheter) and a second candidate region C2 (tip of the guidewire) are shown as candidate regions for the region of interest. This candidate region C is a result obtained when the region of interest acquisition unit 111 inputs a frame image of an X-ray image into the candidate region detector. As a non-limiting example, the candidate region detector is trained to detect the tip of the guidewire, the tip of the guiding catheter, a catheter marker, and the like.

[0041] Detection generally means identifying the position and shape of an object in a still image of one frame in a video. For example, to detect the tip of a guidewire, since the guidewire is thin, it may be detected as a point coordinate (x, y) on the screen. To detect the entire guidewire, the curved, thin, thread-like shape can be detected as a one-dimensional curve or as a two-dimensional fill (segmentation).

[0042] Examples of object detection algorithms that can be used in the candidate region detector include, but are not limited to, Faster R-CNN, YOLO, SSD, U-Net, ResNet, etc. Note that, when implementing an object recognition algorithm, an image processing library such as OpenCV (Open Source Computer Vision Library) may be used.

[0043] Faster R-CNN is a CNN (Convolutional Neural Network) that simultaneously performs region extraction and recognition. A convolutional neural network (CNN) is a neural network with many deep layers that is composed of layers with several characteristic functions such as "convolutional layers" and "pooling layers", and has demonstrated excellent performance especially in the field of image recognition. Faster R-CNN can extract and recognize regions of interest from input images in almost real time (about 10 to 20 frames per second). Faster R-CNN allows end-to-end learning from image input to object detection.

[0044] YOLO (You Only Look Once) is also a CNN that simultaneously extracts and recognizes regions. In YOLO, the entire image is divided into grids and a bounding box is calculated for each region. YOLO's CNN architecture enables high-speed object detection.

[0045] SSD (Single Shot MultiBox Detector) is also a CNN that simultaneously extracts and recognizes regions. SSD can output multi-scale detection frames from output layers at various levels. SSD places approximately 9,000 rectangular frames of different sizes and shapes called default boxes on the image and calculates a prediction value for each of them. SSD aims to increase speed by reducing the filter size.

[0046] U-Net is a CNN that recognizes objects pixel by pixel, called segmentation. It is composed of convolutional layers and has an almost symmetrical Encoder-Decoder structure, where the feature map downsampled through pooling in the Encoder is upsampled in the Decoder.

[0047] FIG. 3(c) is a diagram showing an example of a method for setting the region of interest R. In the example shown in FIG. 3(c), a state in which the user selects the second candidate region C2 as the region of interest R is shown. Specifically, the user selects the region of interest R by moving the mouse cursor M to the second candidate region C2 using a pointing device (not shown) such as a mouse. Note that the presentation of the candidate region C by the region of interest acquisition unit 111 is optional, and the user may directly set the region of interest R in an X-ray image in a state in which the candidate region C is not presented. For example, the region of interest R can be set by drawing a rectangle in the image using a pointing device such as a mouse to specify the region. Alternatively, the region of interest acquisition unit 111 may acquire, as the region of interest R, the output of a candidate region detector generated using a known machine learning method or the like.

[0048] Fig. 3(d) is a diagram showing a region of interest R set by the user. In the example shown in Fig. 3(d), the region of interest R is shown as a black pentagon, and is a region including the tip portion of the device D. In Fig. 3(d), only one region of interest R is set, but the user can set two or more regions of interest R.

[0049] Returning to the description of FIG. 2, the tracking unit 112 tracks one or more regions of interest R in the X-ray image. The tracking unit 112 can realize tracking of the regions of interest R by using a known image tracking technique. Known image tracking techniques that can be used include, but are not limited to, algorithms such as Boosting, MIL, TLD, MedianFlow, KCF, GOTURN, MOSSE, and CSRT. The tracking algorithm may be used in combination with the object detection algorithm described above. For implementation of the algorithm, a library such as OpenCV (Open Source Computer Vision Library) may be used. In the context of the present invention, tracking of the region of interest R also includes intermittently detecting the region of interest R and identifying its state. Therefore, in some aspects of the present invention, it is not required that the region of interest R be recognized in all frames of the X-ray images acquired over time. In some aspects of the present invention, tracking of the region of interest R may be performed using a tracking algorithm or an object detection algorithm, or a combination of them.

[0050] The BOOSTING tracker is a tracker based on an online version of AdaBoost (the algorithm used internally by the HAAR cascade-based face detector). The classifier is trained at runtime using ground truth and false truth examples of the object. The first bounding box identified by the user (or another object detection algorithm) is treated as the ground truth example of the object, and any image outside the bounding box is treated as background. Given a new frame, the classifier is applied to all pixels in the neighborhood of the previous position and the score is recorded. The new position of the object is the one with the maximum score. This gives us another ground truth example for the classifier. Further frames are fed in and the classifier is updated with this additional data.

[0051] The MIL tracker is based on a similar concept to the BOOSTING tracker described above. The main difference is that instead of considering only the current position of the object as the ground truth, it looks at a small neighborhood around the current position to generate several potential ground truths. Instead of identifying ground truths and false truths, MIL specifies a ground truth and false truth "bag". The ground truth bag does not contain all the images that are ground truths. Only one image in the ground truth bag should be the ground truth. The ground truth bag contains an image centered on the object's current position and a small neighborhood around it. Even if the tracked object's current position is not accurate, if the sample near the current position is in the ground truth bag, this bag is likely to contain at least one image with the object properly centered. The MIL tracker performs well, does not drift as much as the BOOSTING tracker, and performs reasonably well in the presence of partial occlusion.

[0052] KFC stands for Kernelized Correlation Filter. This tracker is based on the concepts proposed in the above two trackers. This tracker exploits the fact that the ground truth samples used in the MIL tracker have large overlapping regions. Such overlapping data provides some good mathematical properties, making tracking faster and more accurate at the same time. It outperforms MIL in both accuracy and speed, and has better reporting of tracking failures.

[0053] TLD stands for Track, Learn, Detect. As the name suggests, this tracker breaks down the long-term tracking task into three components: (short-term) tracking, learning, and detection. The tracker tracks the object frame by frame. The detector identifies all previously observed appearances and calibrates the tracker if necessary. Learning estimates the detector's errors and updates it to avoid future errors. The output of this tracker tends to be a bit unstable, for example, if you are tracking a pedestrian and there are other pedestrians in the scene, the tracker may temporarily track a different pedestrian than the one you want to track. The advantage is that it works best under occlusion across multiple frames. The disadvantage is that it has a high number of false positives.

[0054] The MEDIANFLOW tracker tracks objects both forward and backward in time and measures the discrepancy between these two trajectories. By minimizing this forward / backward error, it is able to reliably detect tracking failures and select reliable trajectories in videos. This tracker performs best when motion is predictable and small, and there are no occlusions. Unlike other trackers that continue even when tracking clearly fails, this tracker can recognize when tracking has failed.

[0055] The GOTURN tracker is an algorithm based on a convolutional neural network (CNN). This tracker is robust to viewpoint changes, lighting changes, and deformations, but may not handle occlusions well.

[0056] The MOSSE (Minimum Output Sum of Squared Error) tracker uses adaptive correlation for object tracking. The MOSSE tracker is robust to changes in lighting, scale, pose, and non-rigid deformations. The tracker can also detect occlusion based on peak-to-sidelobe ratio and resume tracking where it left off when the object reappears. The MOSSE tracker also works at high frame rates (450fps and above). Its advantages are that it is very simple to implement, is as accurate as other complex trackers, and is much faster.

[0057] The CSRT tracker uses a spatial confidence map for tracking. The CSRT tracker operates at a relatively low frame rate (25 fps) but gives higher accuracy in object tracking.

[0058] When at least one of the regions of interest R satisfies the condition defined for each region of interest, the notification unit 113 notifies the user of the condition. Specifically, when at least one of the regions of interest R satisfies the condition defined for each region of interest, the notification unit 113 notifies the user by displaying a message indicating the condition on the display device 2 or by sounding a notification sound from a speaker of the image processing device 1 (not shown). Alternatively, when the user is wearing a device equipped with a vibrating member such as a smartphone, the notification unit 113 may notify the user by vibrating the vibrating member. The type, amplitude, and frequency of the notification sound may be changed depending on the conditions. The period and amplitude of the vibration may be changed depending on the conditions.

[0059] In this way, by setting a region of interest R in an X-ray image in advance, the user of the image processing device 1 can have the image processing device 1 track the behavior of the set region of interest R, allowing the user to concentrate on the work at the target site. When displaying the region of interest, the notification unit 113 may display the region of interest in a color different from the color of the acquired image (X-ray image). Generally, angiography images are monochrome, so it is difficult to recognize devices included in the image. However, by displaying the region of interest in a color, it is possible to easily recognize the devices and provide more accurate information to the user. Furthermore, when multiple regions of interest are set, the regions of interest may be colored using different colors.

[0060] [Example of conditions set for region of interest R] Next, a specific example of the conditions set for the region of interest R will be described.

[0061] In vascular catheter surgery, the user first passes a guiding catheter through the blood vessel of the subject P to the vicinity of the area of ​​interest. Next, the user passes other devices such as a guidewire, a delivery wire, or a balloon catheter through the guiding catheter to observe (diagnose) or treat the area of ​​interest, for example, coil embolization of a cerebral aneurysm.

[0062] A catheter is a type of device that is a long, thin tube with a lumen. This catheter is passed through a blood vessel, and guided, for example, into an aneurysm, and a coil is inserted through the lumen of the catheter to prevent the aneurysm from rupturing.

[0063] There are several types of catheters. First, the guiding catheter is a slightly thicker catheter with a diameter of about 2-4 mm, and serves to connect the puncture site to just before the target site. Smaller devices are inserted into this guiding catheter and guided to the target site. The advantage of the guiding catheter is that various devices can be carried from the puncture site of the blood vessel in the groin or arm to the proximal part of the target site through the guiding catheter, eliminating the need to check the path each time. This allows, for example, efficient treatment to be performed without moving a fixed screen.

[0064] Some guiding catheters have a balloon at the tip (balloon-equipped guiding catheter), which can be used to stop blood flow in the blood vessel or stabilize the guiding catheter.

[0065] An intermediate catheter is placed inside a guiding catheter to deliver a thinner device to the target site, closer to the target site where the blood vessel is narrower. Multiple intermediate catheters may be used (gradually thinner).

[0066] Microcatheters are the thinnest catheters, and are soft, making them easy to insert into small blood vessels. Coils and stents are placed inside these microcatheters and transported to the target site. Embolic materials can also be flushed out of the microcatheter.

[0067] A balloon catheter is a microcatheter with a balloon near the tip, which is inflated near the target site. This can be used, for example, to prevent coils from coming out of an aneurysm, to expand a narrowed area, or to stop blood flow when a blood vessel is perforated, thus stopping bleeding. When the word "balloon" is used, it usually refers only to the balloon part of the balloon catheter.

[0068] When guiding these catheters, a guidewire of appropriate size is often used inside. In rare cases, they may be guided along the bloodstream.

[0069] A guidewire is a long, thin wire. In general, a guidewire is used to select a branched blood vessel and guide a soft catheter to a target site. Other examples include a delivery wire for carrying a stent and a device with a balloon attached to the guidewire that can stop blood flow. In this specification, the term "guidewire" is used in a broad sense to include these. The reason is that the tip of any of these guidewires may perforate a blood vessel, causing serious complications. One of the purposes of the present invention is to assist in preventing blood vessel perforation by the tip of such a guidewire, so the term "guidewire" refers to a thin wire that may cause blood vessel perforation at its tip.

[0070] During the procedure, the blood vessels of the subject P are present inside the subject P's body, so the user cannot directly observe the various devices inserted into the blood vessels. Therefore, as described above, the user operates the device D while viewing the blood vessel V and the device D (e.g., a guiding catheter, a guide wire, a coil, a catheter, a balloon, a stent, etc.) captured in the X-ray image generated from the X-rays that have passed through the body of the subject P. As shown in FIG. 1, the X-ray imaging device 3 used in the catheter surgery is a device for generating an X-ray image of a specific area including a site of interest (e.g., an aneurysm, a stenosis site, an infarction site, etc. that require treatment) in the body of the subject P. In many cases during the procedure, particularly in the cerebrovascular area, the area acquired as an X-ray image is fixed. Therefore, the entire device D is not always captured in the X-ray image that the user can observe.

[0071] Therefore, if the user is concentrating on a task at the site of interest, such as inserting a coil into an aneurysm or placing a stent, the tip of the guidewire or the tip of the guiding catheter may be out of the viewing angle of the X-ray image and disappear from the X-ray image, making it impossible for the user to observe it. In such a case, the tip of the device may not be noticed even if it has perforated a blood vessel. Perforation of a blood vessel is a serious complication that can be life-threatening. Perforation of a blood vessel is likely to occur at the tip of a device, especially at the tip of a guidewire or catheter. Therefore, in the present invention, the tip is particularly focused on. There may be cases where there are multiple tips in one screen, or where the tip of the device may perforate a blood vessel in a location other than the site of interest of the surgeon. In addition, in endovascular treatment of the brain, it is necessary to treat while viewing up to four multiple screens at the same time, and it is impossible for the surgeon to always pay attention to multiple tips on multiple screens. This support system compensates for this problem with technology such as AI. Therefore, an example of a condition set for the region of interest R used in the present invention is a condition regarding the distance between the tip of the guiding catheter or the tip of the guidewire and the edge of the X-ray image. Alternatively, the condition may be that the region of interest R has exceeded a specific range (for example, a boundary line specified by a user using a pointing device such as a mouse) specified on the X-ray image, or that it has simply moved. That is, in some aspects of the present invention, the range of any region in the image may be treated as being similar to the edge, not limited to the edge of the image. The boundary line may be a straight line, a curve, a circle, a rectangle, or another polygon. In some aspects of the present invention, the distance between the region of interest and the edge of the specific range may be displayed. The display mode of the distance may be changed depending on the magnitude of the distance between the region of interest and the edge of the specific range. In some aspects of the present invention, the notification unit may notify the user on the condition that the value obtained by dividing the distance between the region of interest and the edge of the specific range by the moving speed of the region of interest in the image is less than a predetermined threshold. Some aspects of the present invention also include a function that makes it easier for the surgeon to make a decision by making the specific range easier to see by superimposing it on the display. The method of displaying the range may be not only superimposed but also arrows, but is not limited to these.It should be noted that in some aspects of the invention, the distance may be determined either by a straight line distance or by a distance along a blood vessel.

[0072] Furthermore, instead of the region of interest, the position of an intravascular inspection or treatment device (also called a device of interest) at a certain time may be automatically or by the user (surgeon or assistant) to superimpose all or part of the device detected at that time on the subsequent real-time image. This allows the user (surgeon or assistant) to recognize the movement (deviation) of the device from a specific time point by superimposing it on the real-time image (Figure 21). The shape of the device may be detected automatically by image analysis, or by the surgeon marking it with a mouse or touch panel. Specific examples of devices to be superimposed include the tip of a guidewire, a marker, a stent, a guiding catheter, etc., but are not limited to these.

[0073] For example, assume that the position and shape of the guidewire tip (black part) at the time shown in the left diagram of Fig. 21 are stored. In this case, for example, when the user designates the guidewire tip with a pointing device such as a touch panel or a mouse, the guidewire tip is recognized and the stored guidewire tip is superimposed on an X-ray image obtained in real time thereafter, as shown in the center diagram of Fig. 21. This makes it possible to visualize and easily grasp how far the guidewire tip has moved since then, as shown in the right diagram of Fig. 21.

[0074] Therefore, in some aspects of the present invention, the image processing device includes a memory unit that acquires and stores the position and / or shape of an inspection or treatment device within a blood vessel at any point in time, and has a function of superimposing the stored position and / or shape of the device on images acquired thereafter.

[0075] For example, in carotid artery stenting, it is important to prevent distal embolism (cerebral infarction) that the filter or balloon placed in the distal part of the internal carotid artery is stable and does not move too much. A notification can be issued if the tip of the guide wire attached to the end of the guide wire or the like goes outside a certain range specified by the surgeon or automatically. In cerebral aneurysm embolization, if the tip of the guide wire inside the balloon catheter goes outside a specified range, there is a risk of the balloon slipping off, becoming unable to return it, or perforating the distal blood vessel, so a notification can be issued. Similarly, when inserting a coil, if it goes outside a certain area, there is a risk of blocking an important blood vessel, so a notification can be issued. If the coil deviates from the mask image (area) of the aneurysm, a notification can be issued because there is a possibility of aneurysm perforation due to the coil or guide wire catheter. It is important for the guiding catheter to be stable in all endovascular treatments, but there are times when you want to correct it before it disappears outside the frame, and if it goes out of a certain area, a notification can be issued. In tumor embolization, cerebral arteriovenous malformation embolization, dural arteriovenous fistula embolization, etc., embolization is performed using an embolic substance such as a liquid or particles. If the embolic substance exceeds a designated area, it may cause cerebral infarction in an important area, so a notification can be issued. Therefore, in some aspects of the present invention, the device includes a liquid embolic substance, a particle embolic substance, etc. In addition to the above examples, in endovascular surgery, care is taken to keep any device or embolic substance within a designated area, and this can be supported.

[0076] 4(a)-(b) are diagrams for explaining an example of conditions set for a region of interest R. In the example shown in FIG. 4(a), the region of interest R is set at the tip of a device D. In the example shown in FIG. 4(a), information W indicating the speed and acceleration of the region of interest R and the distance (number of pixels) between the region of interest R and an edge F of the X-ray image is displayed superimposed on the X-ray image. Note that FIG. 4(a) shows an example in which the device D is a guidewire. In some aspects of the present invention, a specific region designated in the X-ray image may be regarded as the same as the edge F of the X-ray image.

[0077] When a region including the tip of the guidewire is set as the region of interest R, the notification unit 113 notifies the user on condition that the region of interest R disappears from the X-ray image. In some embodiments, the X-ray image is an image showing a fixed region including the site of interest. Usually, during a procedure, the region displayed as the X-ray image is fixed. In addition, when a region including the tip of the guidewire is set as the region of interest R, the notification unit 113 notifies the user on condition that the distance between the region of interest R and the edge F of the X-ray image is less than a predetermined threshold distance.

[0078] Here, the "predetermined threshold distance" is a "reference distance for determining protrusion" provided for the notification unit 113 to determine whether or not there is a high probability that the tip of the device will deviate from the angle of view of the X-ray image. A specific value of the predetermined threshold distance may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, and the like, and is, for example, 5% of either the number of vertical pixels or the number of horizontal pixels of the X-ray image. This allows the user to receive a notification when the region of interest R, which is a region including the tip of the guiding catheter or the tip of the guidewire, approaches the edge F of the X-ray image, both before and after the region of interest R deviates from the edge F.

[0079] In addition, the magnitude of the movement of the device D in the blood vessel V can be a useful index for the user to predict the time until the tip of the device D reaches the edge F of the X-ray image. Specifically, the greater the speed or acceleration of the tip of D, the shorter the time until the tip of D reaches the edge F of the X-ray image. Therefore, when a region including the tip of the guiding catheter or the tip of the guidewire is set as the region of interest R, the notification unit 113 may notify the user on the condition that at least one of the moving speed and the acceleration of the region of interest R in the X-ray image exceeds a predetermined threshold. In this way, when the moving speed or the acceleration of the region of interest R is greater than the threshold, the user's attention can be drawn before the region of interest R approaches the edge F of the X-ray image.

[0080] Furthermore, the distance between the tip of the device D in the blood vessel V and the edge F of the X-ray image is a useful index for the user to determine the likelihood that the tip of the device D will reach the edge F of the X-ray image. Therefore, as shown in information W in Fig. 4, the notification unit 113 may cause the display device 2 that displays the X-ray image to display the distance between the region of interest R and the edge F of the X-ray image.

[0081] Here, the "distance between the region of interest R and the edge F of the X-ray image" may be a distance when the region of interest R moves to the edge F of the X-ray image along the blood vessel V into which the device D in which the region of interest R is set is inserted. This can be realized by the distance measuring unit 115 extracting the blood vessel V using a blood vessel recognition engine generated in advance using a known machine learning method or the like, and measuring the distance from the region of interest R to the edge F along the blood vessel V. Alternatively, the distance measuring unit 115 may measure the above-mentioned distance based on the trajectory of the device D that has progressed through the blood vessel V. Specifically, when the user advances the guiding catheter through the blood vessel V, the tracking unit 112 tracks the tip of the guiding catheter and stores the trajectory in the storage unit 10. The distance measuring unit 115 may determine the length of the trajectory included in the region of interest R among the trajectories stored in the storage unit 10 as the above-mentioned length.

[0082] By having the notification unit 113 display the distance between the tip of the device D and the edge F of the X-ray image on the display device 2, the user can objectively grasp at a glance how far the device D needs to move before it reaches the edge F of the X-ray image. Furthermore, the notification unit 113 may change the display mode of the distance depending on the distance between the region of interest R and the edge F. For example, the shorter the distance, the larger the font size, the different color depending on the distance (blue → yellow → red), the region of interest may be enlarged depending on the distance, or the color or size of the mark added to the region of interest may be changed depending on the distance. By devising the display mode as described above, it is possible to make it easier for the user to notice the change in distance.

[0083] In the example shown in Fig. 4(b), the region of interest R set at the tip of the device D is closer to the edge F than in the example shown in Fig. 4(a). Therefore, the font of the information W indicating the speed and acceleration of the region of interest R and the distance (number of pixels) between the region of interest R and the edge F of the X-ray image is larger than in the example shown in Fig. 4(a).

[0084] The "distance between the region of interest R and the edge F of the X-ray image" may be the shortest distance between the region of interest R and the edge F of the X-ray image, or may be the length measured to the edge F of the X-ray image along the direction of movement of the region of interest R. In this case, the blood vessel extraction process by the distance measuring unit 115 can be omitted, which is advantageous in that the processing speed can be increased.

[0085] The notification unit 113 may notify the user on condition that the value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R in the X-ray image is less than a predetermined threshold. The value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R in the X-ray image can be said to be, so to speak, the expected time until the region of interest R reaches the edge F of the X-ray image. Therefore, the value of the "predetermined threshold" is a "grace period for determining whether or not the region of interest R is likely to fall outside the angle of view of the X-ray image" provided for the notification unit 113 to determine whether or not there is a high probability that the region of interest R will fall outside the angle of view of the X-ray image. The specific value of the grace period may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, and the like, and is, for example, 3 seconds.

[0086] Fig. 5 is a diagram showing an example of a message Ms notified by the notification unit 113. In the example shown in Fig. 5, the notification unit 113 displays a message indicating that the region of interest R will disappear from the X-ray image in three seconds (so-called frame-out) by superimposing it on the X-ray image. Furthermore, as in the example shown in Fig. 5, when the region of interest R approaches the edge F of the X-ray image, the notification unit 113 may change the shape of the region of interest R (changed to a circle in Fig. 5), increase the size of the region of interest R, or change the color of the region of interest R. This allows the notification unit 113 to easily recognize that the region of interest R is about to frame out.

[0087] FIG. 6 is a diagram for explaining another example of conditions set for the region of interest R. Specifically, FIG. 6 shows an example in which the device D is a guiding catheter. Unlike the guidewire shown in FIG. 4(a)-(b), the guiding catheter is different in that not only its tip portion but the entire device D may disappear from the X-ray image. However, whether the device D is a guiding catheter or a guidewire, it is the same in that its tip portion disappears from the X-ray image. Therefore, in the case of a guiding catheter, the region of interest R is set at the tip portion, as in the case of a guidewire.

[0088] In the above example, the threshold value is determined based on the distance between the region of interest R and the edge F of the X-ray image as the "predetermined threshold distance" mainly from the viewpoint of preventing the surgical tool from going out of the frame. However, the moving distance of the region of interest R itself may be focused on, and a notification may be issued to the user when the moving distance of the region of interest R is equal to or greater than a predetermined threshold. For example, when a stent is to be deployed with the tip of the catheter in the aneurysm, it is necessary to take care that the tip of the catheter does not go out of the aneurysm. However, since the surgeon is watching the stent at this time, he or she cannot follow the movement of the tip of the catheter with his or her eyes. In such a case, by configuring the system to notify the user when the moving distance of the region of interest set at the tip of the catheter exceeds a predetermined moving distance, the user can immediately notice the abnormality when the tip of the catheter is about to go out of the aneurysm. The threshold value of the moving distance may be set based on the position of the region of interest (e.g., the tip of the catheter) after the region of interest reaches a predetermined position (e.g., inside the aneurysm).

[0089] (Induction of embolization coils) Next, as another example of the condition set for the region of interest R, a condition related to a guiding catheter that guides a delivery wire of an embolization coil will be described.

[0090] Embolization, which fills the cerebral aneurysm with an embolization coil, is known as a catheter treatment for cerebral aneurysms. This treatment is a surgical procedure in which a guiding catheter is passed to the vicinity of the cerebral aneurysm, which is the target area, a delivery wire for an embolization coil is passed through the guiding catheter, and the embolization coil is separated and filled into the cerebral aneurysm, thereby blocking the inflow of blood into the cerebral aneurysm.

[0091] Figures 7(a)-(b) are schematic diagrams showing an example of an X-ray image of a guiding catheter that guides a delivery wire of an embolization coil, and are diagrams for explaining another example of conditions that are set for the region of interest R.

[0092] As described above, the X-ray imaging device 3 can irradiate the head of the subject P with X-rays at different angles of incidence. Figures 7(a) and 7(b) show examples of X-ray images taken by irradiating the head of the subject P with X-rays at different angles of incidence. The user performs embolization while viewing the two images shown in Figures 7(a) and 7(b).

[0093] In Fig. 7(b), the aneurysm is indicated by the symbol A. The user embolizes the aneurysm by placing multiple embolization coils E in the aneurysm A. Here, since the aneurysm A can be observed in the X-ray image shown in Fig. 7(b), the user focuses on one of the two X-ray images.

[0094] The delivery wire for carrying the embolization coil E to the aneurysm A is connected to the embolization coil E at its tip, and the user carries out the task of detaching the embolization coil E from the delivery wire after carrying the embolization coil E to the aneurysm A. As shown in Fig. 7(a)-(b), when the embolization coil E reaches the aneurysm A, the position of the embolization coil E being carried becomes unclear in the X-ray image due to other embolization coils E already placed in the aneurysm A. For this reason, the delivery wire for the embolization coil E is provided with a marker L for timing the detachment of the embolization coil E. The user determines the timing of detaching the embolization coil E using the marker L provided on the delivery wire as a guide, not the embolization coil E itself in the X-ray image.

[0095] However, as shown in FIG. 7(a)-(b), an X-ray image in which the aneurysm A can be observed may differ from an X-ray image in which the marker L can be observed. Furthermore, even within the same screen, the aneurysm A and the marker L may be located far apart, making it difficult to observe the two locations at the same time. Therefore, a user of the image processing device 1 first sets a region of interest R in a part of the guiding catheter that guides the delivery wire of the embolization coil E. Specifically, the user sets the region of interest R at a position where the marker L and the region of interest R overlap at the timing when the embolization coil E is detached from the delivery wire.

[0096] The region of interest acquisition unit 111 receives a region of interest R that is set in a part of the guiding catheter that guides the delivery wire. The region of interest R can be specified using, for example, a pointing device such as a mouse, a touch panel, or the like. In FIG. 7(a), the region of interest R is indicated by a white star. The marker detection unit 114 detects markers L approaching the region of interest R among markers L provided on a delivery wire of an embolization coil E for embolizing an aneurysm. In addition, the tracking unit 112 tracks the markers L detected by the marker detection unit 114.

[0097] When the marker L and the region of interest R overlap, the notification unit 113 notifies the user of the timing to cut and detach the embolization coil E from the delivery wire based on the positional relationship between the marker L and the region of interest R.

[0098] Figures 8(a)-(d) are diagrams for explaining the timing of detaching the embolization coil E. The device D shown by the dashed line in Figures 8(a)-(d) is a delivery wire. As shown in Figures 8(a)-(d), a marker L is attached to a certain section of the delivery wire. Note that Figures 8(a)-(d) show a typical example of the marker L. Other examples include a dashed dotted line "- - -" or a long straight line.

[0099] When the user moves the delivery wire, the marker L also moves in conjunction with the movement of the delivery wire in the X-ray image. On the other hand, the guiding catheter for guiding the delivery wire may move slightly due to friction or the like in response to the movement of the delivery wire, but the amount of movement is small compared to the movement of the delivery wire. Therefore, the user sets in advance the region of interest R at the position of the guiding catheter that corresponds to the position where the marker L should be when the embolization coil E reaches the aneurysm A.

[0100] As shown in Figures 8(a)-(d), the marker L is present over a certain section on the wire. When the user moves the delivery wire, the tip of the marker L comes into contact with the region of interest R, as shown in Figure 8(b). This indicates that the embolization coil E has approached the aneurysm A.

[0101] 8(c), the marker L and the region of interest R overlap each other. When the marker L and the region of interest R overlap each other, the notification unit 113 starts an operation of notifying the user of the timing to disconnect and detach the embolization coil E from the delivery wire.

[0102] Specifically, as shown by symbol W2 in Figure 7(a), when the marker L and the region of interest R overlap, the notification unit 113 causes the display device 2 to display information W2 indicating the distance the marker L must move before the embolization coil E is disconnected from the delivery wire.

[0103] 8(d), the user sets the region of interest R so that the end of the marker L passes through the region of interest R just when the embolization coil E reaches the aneurysm A. In this case, the information W2 displayed on the display device 2 by the notification unit 113 indicates the distance until the end of the marker L passes through the region of interest R.

[0104] When the marker L passes through the region of interest R, the notification unit 113 further notifies the user of that fact. This allows the user to notice the timing to disconnect the embolization coil E from the delivery wire even when the aneurysm A is focused on the captured image as shown in FIG. 7(b).

[0105] (Device D shape) Next, as yet another example of a condition set for the region of interest R, a condition regarding the shape of a device D (for example, a guidewire) will be described.

[0106] When a feature amount indicating the shape of device D included in region of interest R satisfies a predetermined condition, notification unit 113 notifies the user of image processing device 1. Specifically, notification unit 113 notifies based on a feature amount indicating the curvature of device D included in region of interest R or the "deflection" of device D included in region of interest R.

[0107] When the user attempts to advance device D with the tip of device D caught on a blood vessel wall or the like, the tip of device D is bent. In general, when the tip of device D is bent, elastic energy is accumulated in that portion. The more the tip of device D is bent, that is, the greater the curvature of device D becomes (the smaller the radius of curvature becomes), the more elastic energy is accumulated. When the amount of accumulated elastic energy becomes large, the elastic force of device D releases the tip from being caught, and the tip may move at high speed. As a result, the tip of device D may suddenly disappear from the X-ray image.

[0108] Therefore, the notification unit 113 notifies the user on the condition that the curvature of the device D included in the region of interest exceeds a predetermined threshold curvature, or the tip does not move even though the curvature changes. It may also be considered that the tip of the device D is immobile, or that the moving distance is lower than a certain threshold. Here, the "predetermined threshold curvature" is a "reference curvature for notification determination" provided for the notification unit 113 to determine whether or not the tip of the device D is likely to move at high speed. A specific value of the predetermined threshold curvature may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, the material, size, elasticity coefficient, etc. of the device D.

[0109] 9(a)-(c) are diagrams for explaining conditions related to the shape of the device D set in the region of interest R. Specifically, FIG. 9(a)-(b) are diagrams for explaining notification based on the curvature of the device D. In FIG. 9(a), the region of interest R is indicated by a dashed circle. As a non-limiting example, the region of interest R in FIG. 9(a) is a circle with a radius of 1 cm centered on the tip of the device D. Also, FIG. 9(b) is a histogram showing the distribution of the curvature of the device D included in the region of interest R. Specifically, FIG. 9(b) shows the distribution of the radii of curvature obtained by dividing the device D included in the region of interest R into a plurality of small regions and calculating the radius of curvature of the device D in each small region.

[0110] The notification unit 113 notifies the user when a feature amount (for example, a statistic such as the average value, mode, or median value of the curvature) calculated from the distribution of the curvature in a histogram showing the distribution of the curvature of the device D exceeds a predetermined threshold curvature, or when the tip does not move despite a change in the curvature. This allows the image processing device 1 to provide an opportunity for the user to notice that an elastic force is being accumulated in the device D.

[0111] Furthermore, the notification unit 113 may notify the user on the condition that the value obtained by subtracting the length of the center line of the blood vessel V included in the region of interest R from the length of the device D included in the region of interest R exceeds a predetermined threshold length (threshold value). Fig. 9(c) is a schematic diagram showing the relationship between the length of the device D included in the region of interest R and the length of the center line of the blood vessel V included in the region of interest R. Although the device D and the blood vessel are curved inside the body of the subject P, for convenience of explanation, the device D and the blood vessel V are displayed as straight lines in Fig. 9(c). Also, in Fig. 9(c), the center line of the blood vessel V is shown by a dashed line.

[0112] The distance measuring unit 115 uses a blood vessel recognition engine to extract a blood vessel V in the region of interest R and traces its centerline to obtain a length D1. Similarly, the distance measuring unit 115 uses a device recognition engine generated using a known machine learning method or the like to extract a device D and obtains its length D2.

[0113] Generally, when a user advances device D into blood vessel V, device D will advance in a serpentine manner along the wall of blood vessel V. Therefore, length D2 of device D in blood vessel V will be longer than length D1 of blood vessel V (length of center line of blood vessel V), and device D will bend in blood vessel V, which means that elastic energy is accumulated in device D. If the amount of this bending becomes large, some trigger may release the elastic energy, causing device D to move significantly. As a result, the tip of device D may disappear from the X-ray image.

[0114] The difference length B, which is the length obtained by subtracting the length D1 from the length D2 calculated by the distance measuring unit 115, can be an index showing the amount of deflection of the device D in the blood vessel V. Therefore, the notification unit 113 displays the length B obtained by subtracting the length D1 from the length D2 calculated by the distance measuring unit 115, and notifies the user when the length B is longer than a predetermined threshold length. This allows the image processing device 1 to provide an opportunity for the user to notice that an elastic force is being accumulated in the device D.

[0115] (Estimation of catheter tip position using 2nd marker) Aneurysm embolization catheters are equipped with a tip marker (1st marker) and a second marker (usually located 3 cm from the tip). It is important for the surgeon to know where the tip of the catheter is located within the aneurysm in order to perform the surgery safely, but if a coil enters the aneurysm, the position of the tip marker becomes difficult to determine, reducing safety (see Figure 17). For example, if the tip marker moves to the back of the aneurysm, the tip or the coil coming out of the tip may perforate the aneurysm, leading to a serious complication of subarachnoid hemorrhage. Conversely, if the tip marker is about to leave the aneurysm, the catheter or coil may deviate from the aneurysm and must be reinserted into the aneurysm, which carries the risk of perforating the aneurysm wall.

[0116] For example, in FIG. 17, the left diagram shows a state in which a microcatheter is inserted into an aneurysm. The size of the aneurysm can be, for example, 10 mm. The distance between the 1st marker and the 2nd marker attached to the microcatheter is constant (usually 30 mm). For example, assume that the positions of the 1st marker and the 2nd marker are recorded in this state (stored positions). The center diagram in FIG. 17 shows a state in which the position of the microcatheter has moved. When a coil is inserted into the aneurysm, the position of the 1st marker becomes invisible or difficult to see. Therefore, the position of the 1st marker is estimated from the difference between the position of the 2nd marker at this time and the position of the 2nd marker previously stored. In this case, the 1st marker is predicted to be approximately 0 mm from the neck of the aneurysm (predicted position A). The right diagram also shows a state in which the position of the microcatheter has moved. Similarly, the position of the 1st marker is estimated based on the moving distance of the 2nd marker. In this case, it is estimated to be 8 mm from the neck of the aneurysm.

[0117] Some aspects of the present invention relate to a method for estimating a tip position from the movement of a second marker when the position of a catheter is unknown because the tip of the catheter is inside a coil (aneurysm). More specifically, the method includes a step of storing a positional relationship between the tip of the catheter and the second marker (e.g., 3 cm apart), a step of storing a distance a between the neckline of the aneurysm and the first marker and the position of the second marker at time t1, a step of calculating a moving distance b from the position of the second marker at time t2, a step of estimating a distance ab from the neckline of the aneurysm of the catheter tip, and a step of notifying a user of the estimated distance. Here, the tip of the catheter, the second marker, and the neckline of the aneurysm may be automatically detected by image recognition by a computer. Alternatively, they may be manually specified by a pointing device such as a mouse or a touch panel. The estimated distance ab may be displayed on a display device. Furthermore, the position of the catheter tip estimated based on the estimated distance ab may be displayed on a display device. An arbitrary threshold may be set and a notification may be made if the distance deviates from the threshold, or the speed and acceleration may be calculated and notified based on the value, in addition to the distance (rapid or large movement is likely to be dangerous). This moving distance is a straight line distance or a distance along the curve of the catheter. Also, since the shape of the curve of the catheter may change, the length of the curve may be used. Furthermore, the distances a, b, and ab may be a probability distribution due to changes in the shape of the catheter. For example, if there are multiple times of initial memorization, a distribution of the distance a can be obtained, so the location can be estimated using the average or variance, and it can be predicted as the most likely point, or it can be displayed as a probability distribution using a heat map, etc. Therefore, the estimated distance may be represented by a probability distribution, and the estimated position of the catheter tip may be displayed based on the probability distribution by coloring it like a heat map.

[0118] In addition to measuring the distance traveled, the position of the second marker can be specified by the surgeon or recognized by a computer, and the position can be indicated on the display device by a semi-transparent overlay or an arrow, etc. From this fixed display, the surgeon and assistant can visually recognize whether the tip marker has advanced or been pulled back by recognizing how far the current second marker has shifted.

[0119] Some aspects of the present invention relate to a program for executing the above method on a computer. Some aspects of the present invention relate to an image processing device for executing the above method and its operating method. Such an image processing device may include, for example, a positional relationship storage unit that stores the positional relationship between the tip of the catheter and the 2nd marker (e.g., 3 cm apart), a position storage unit that stores the distance a between the neckline of the aneurysm and the 1st marker and the position of the 2nd marker at time t1, a distance estimation unit that calculates the moving distance b from the position of the 2nd marker at time t2 and estimates the distance ab from the neckline of the aneurysm of the tip of the catheter, and a notification unit that notifies a user of the estimated distance. Furthermore, some aspects of the present invention relate to a cerebral aneurysm coil embolization assistance system including the above image processing device and an image capturing device that captures an X-ray image of a patient (or subject P) with a guiding catheter and a delivery wire of an embolization coil inserted in a blood vessel and transmits the image to the image processing device.

[0120] <Processing flow for estimating catheter tip position using the 2nd marker> 22 is a flowchart for explaining the flow of the catheter tip position estimation process using the second marker executed by the image processing device according to the embodiment. The process in this flowchart is started, for example, when the image processing device 1 is started, or when the user or the image processing device 1 determines that it is necessary to start the process.

[0121] First, the image acquiring unit 110, the region of interest acquiring unit 111, and the tracking unit 112 function to detect and track the inspection or treatment device D in the blood vessel V by video analysis. Then, the positions of the first marker and the second marker at that time (T=t1) are stored (S102) automatically by image analysis or by designation by the user (operator).

[0122] Furthermore, the aneurysm neck line is determined automatically or by user (operator) designation (S104). Then, the distance A between the aneurysm neck line and the first marker is calculated (S106). At this time, the straight or curved distance A between the aneurysm neck line and the first marker (usually inside the aneurysm) can be measured.

[0123] Next, the second marker is continuously tracked to measure the moving distance b (S108). The second marker is continuously tracked by image analysis, and the moving distance b (straight line or curved distance) of the second marker at T=t2 is measured. This moving distance may have a directional property (plus / minus).

[0124] Then, based on the distances A and b, the distance of the first marker from the aneurysm neck is estimated (S110). For example, the distance of the first marker from the aneurysm neck is estimated to be Ab. Alternatively, it may be estimated that the first marker is located at a position that has moved b in a direction perpendicular to the aneurysm neck from the position of the first marker at T=t1.

[0125] Then, the estimated position of the first marker is displayed (S112). For example, the estimated position of the first marker at T=t2 or the distance from the aneurysm neck can be displayed as a superimposed display or a numerical value. If the first marker is likely to deviate from the aneurysm near the aneurysm neck or is likely to hit the aneurysm wall deep inside the aneurysm, the user (surgeon / assistant) may be additionally notified of this. If the analysis is difficult or if the user (surgeon) specifies otherwise, this function can be temporarily stopped. After this process is completed, the process in this flowchart may be restarted as necessary.

[0126] (Replay function) Since medical personnel pay attention to a specific location during treatment, if the medical personnel is not paying attention to the device that issued the warning or is looking at another screen when the system according to the present disclosure issues a notification, the medical personnel will have to try to understand the situation after the warning is issued. However, since real-time images are updated every moment, it is often difficult to confirm the details around the time the warning was issued. In addition, depending on the nature of the warning, it is necessary to understand the details of the device's operation itself, the difference between the time the warning was issued and the current time, the elapsed time since the warning was issued, etc.

[0127] For example, for the tip of a guidewire, it is desirable to obtain detailed information on the movement of the tip. For example, if the tip moves rapidly and significantly, the risk of blood vessel perforation increases. In the case of the disappearance of a guiding catheter, it is desirable to obtain information on the position and elapsed time at the time when it went outside the image. In the case of a coil detector, it is desirable to obtain information on its deviation from the optimal position.

[0128] In some embodiments of the present invention, the image processing device has a function of storing video and recognition information and providing information as necessary so that the surgeon can review it when necessary (i.e., a replay function). FIG. 12 shows the configuration of an image processing device further including a video recording unit (which can be integrated with 10) that stores images obtained from the image acquisition unit 110, and a video extraction unit that extracts a part of the video before and after the time when the notification occurred. FIG. 13 shows an example in which the replay video when the notification occurred is cropped and enlarged around the area of ​​interest. FIG. 14 shows an example in which a (enlarged) replay playback window is displayed on a real-time display screen. In this example, the replay is enlarged and displayed in a part that is as far removed from the area of ​​interest as possible, which makes it possible to determine what happened in the part where the warning was issued within the limited range of one screen.

[0129] In some aspects of the present invention, the image processing device further includes a video storage unit that stores images (including videos) obtained from the image acquisition unit over time (continuously). In some aspects of the present invention, the image processing device further includes a video extraction unit that extracts video from the video storage unit for a certain period before and after the notification unit issues a notification. The video extraction period and playback speed may be automatically determined based on at least one of the moving distance, moving speed, and acceleration of the region of interest when the notification is generated. Note that, when extracting, information obtained by the region of interest acquisition unit, tracking unit, notification unit, marker detection unit, and / or distance measurement unit may also be used.

[0130] In some aspects of the present invention, the image processing device can display the extracted image on the display device. The extracted image may be automatically and repeatedly displayed a predetermined number of times. The extracted image may also be displayed based on any operation including play, stop, fast forward, rewind, frame by frame, slow playback, and double speed playback. This allows the user to easily check the image. The display device may further display the elapsed time from the time the notification occurred, a comparison of the position of the region of interest after the time the notification occurred and an arbitrary time has elapsed (comparative display includes, for example, the relevant region, the difference in the detected position, and the alignment of the image itself), or the trajectory of the region of interest acquired by the tracking unit, superimposed on the extracted image.

[0131] In some aspects of the present invention, the image processing device can cut out a part of the image near the region of interest from the extracted image and display it. The extracted image can be displayed at a position that does not obstruct the display of the region of interest. The extracted image may be displayed in an enlarged form.

[0132] In some aspects of the present invention, the image processing device can display the extracted image simultaneously with the occurrence of the notification or after a predetermined time has elapsed since the occurrence of the notification. In some aspects of the present invention, the image processing device can display images captured from multiple directions simultaneously.

[0133] The replay display may be used at a time other than when the notification is generated. In other words, the video extraction unit may extract from the video storage unit not only video for a certain period before and after the notification unit issues a notification, but also video for any time or period. For example, when the user (surgeon) feels it is necessary, he or she can specify any region of interest and view the previous scene in a replay display. This allows the user to understand and compare what happened in the region of interest while viewing the real-time display.

[0134] Some aspects of the present invention relate to a program for executing the above method on a computer, and some aspects of the present invention relate to an image processing device that executes the above method and a method of operating the image processing device.

[0135] <Replay display function processing flow> 23 is a flowchart for explaining the flow of processing of the replay function executed by the image processing device according to the embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0136] First, the image acquiring unit 110, the region of interest acquiring unit 111, and the tracking unit 112 function to detect and track an inspection or treatment device D within a blood vessel V by video analysis (S202).

[0137] Next, it is determined whether or not the notification condition (such as the movement distance exceeding a threshold) is satisfied (S204), and if the condition is satisfied, a notification is issued and a replay video before and after the notification condition is satisfied is displayed in part of the display screen (S206). At this time, the real-time image is displayed as usual, and the replay video may be displayed several times so as not to overlap with the area of ​​the notification condition, or may be displayed repeatedly until the user (surgeon / assistant) desires (S208). When the repetition is completed, the replay video screen is closed. After completion, the processing in this flowchart may be restarted.

[0138] (Estimating the location of the region of interest outside the frame) It is dangerous for a region of interest such as the tip of a guidewire or a guiding catheter to move out of the range of the X-ray field angle (frame-out), but the degree of danger varies depending on the amount of movement. Specifically, for example, if the tip of the guidewire is only slightly (e.g., within 5 mm) out of the frame, the possibility of blood vessel perforation is low, but if it is significantly out of the frame (e.g., 20 mm or more), the risk of blood vessel perforation is high. When the region of interest is out of the frame, it is necessary to pull it back into the field angle of the X-ray image, but depending on the situation, it may not be possible to deal with it immediately. In such a case, it is important to know how far the out-of-frame region of interest has moved out of the range of the X-ray field angle and how dangerous the movement is. Therefore, some aspects of the present invention relate to an apparatus for estimating and displaying the position, speed, and acceleration state of the out-of-frame region of interest.

[0139] The estimation of the position of such a region of interest outside the frame can be performed by, for example, an image processing device, which includes an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each region of interest, and a notification unit that notifies the user on the condition that the region of interest disappears from the image when a region including the tip of the catheter or the tip of the guidewire is set as the region of interest, and a state estimation unit that estimates the current position and / or speed of the tip of the catheter or the tip of the guidewire that has disappeared from the image based on the position, speed and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappears from the image (FIG. 15). The current position and / or speed of the region of interest estimated by the state estimation unit can be displayed on a display device.

[0140] Here, in order to estimate the position of the frame-out region of interest, the state estimation unit stores outputs from the region of interest acquisition unit, marker detection unit, and tracking unit in chronological order from before the frame-out, and can calculate the state of the region of interest, such as its position, speed, and acceleration, using the stored outputs. Then, when the region of interest is framed out and cannot be tracked on the screen, the state estimation unit estimates the position, speed, and other information of the region of interest from the state before the frame-out and notifies the user. The estimation method in this case includes, but is not limited to, a learning-based method using deep learning (CNN, RNN, WaveNet, etc.) and a Bayesian estimation method (Kalman filter, extended Kalman filter, ensemble Kalman filter, particle filter, etc.).

[0141] In addition, some aspects of the present invention also relate to a device that calculates a risk level from an estimated state and issues a notification to a user according to the risk level. The position, speed, and risk level of the region of interest estimated by the state estimation unit may be displayed on a display device. The display method may be, but is not limited to, a display using a point, an arrow, a heat map, or the like. For example, if the position of the estimated region of interest is a predetermined distance or more away from the edge of the image, the risk level may be determined to be high (FIG. 16). For example, in FIG. 16, the position of the estimated region of interest is shown by a circle, and the risk level is determined to be higher as the position is estimated to be farther away from the edge of the screen. In this case, for example, the risk level may be shown by the color of the circle (for example, green indicates low risk and red indicates high risk). If the calculated risk level exceeds a certain threshold, an alert can be displayed on the screen of the display device. Alternatively, a notification by voice may be given.

[0142] Therefore, some aspects of the present invention also relate to an image processing device comprising: an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as regions of interest; a tracking unit that tracks each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest, the notification unit notifying the user of this when a region including a tip of a catheter or a tip of a guidewire is set as the region of interest, on the condition that the region of interest disappears from the image; and a state estimation unit that estimates a current position and / or velocity of the tip of the catheter or the tip of the guidewire that has disappeared from the image based on the position, velocity and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappeared from the image, and a warning is notified to the user when the current position and / or velocity of the region of interest estimated by the state estimation unit exceeds a predetermined threshold.

[0143] Some aspects of the present invention relate to a program for executing the above method on a computer, and some aspects of the present invention relate to an image processing device that executes the above method and a method of operating the image processing device.

[0144] <Process flow for estimating the position of a region of interest outside a frame> 24 is a flowchart for explaining the flow of processing for estimating the position of a region of interest outside a frame, which is executed by an image processing device according to an embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0145] First, the image acquiring unit 110, the region of interest acquiring unit 111, and the tracking unit 112 function to detect and track an inspection or treatment device D within a blood vessel V by video analysis (S302).

[0146] Next, it is determined whether the part of device D to be notified has gone outside the frame of the screen (S304), and if the condition is met, the position of the device outside the frame is estimated (S306). For example, the position outside the frame of device D is estimated from the front side of the black part of the guide wire connected to device D that is visible on the screen, the filter, the balloon, etc. For estimation, distance difference, machine learning, etc. can be used.

[0147] Next, the estimated device position is displayed (S308). The estimated position of device D that has gone outside the frame is displayed on the display screen as the actual estimated position or the distance from the frame. The further outside the frame the device is, the stronger the display or notification may be.

[0148] Finally, it is determined whether the guess is no longer necessary because the device D has returned to the screen frame, or the user (surgeon / assistant) has pressed a button that does not require display (S310). After this is completed, the process in this flowchart may be restarted.

[0149] (Layout / Notifications) In some aspects of the present invention, the results of image analysis can be displayed on two screens of different sizes. As described above, in endovascular surgery, surgery is generally performed while viewing multiple screens (for example, four screens), and the surgeon grasps three-dimensional information by viewing at least two screens (generally the front (AP direction or F direction: Anterior-Posterior, Frontal) and the side (RL direction or LAT direction: Right to Left, Lateral)). Therefore, it is important to display images from the front and the side, and it is very important to display them easily on a monitor of a physically limited size. In actual surgery, the monitor is often placed more than 1m away from the patient's bed, and instructions are often given to move the monitor closer to the surgeon even by 1cm. The angles of the front and the side move three-dimensionally, and for example, the front tube is always used to view from an oblique angle, with an angle of 15 degrees to the right and 10 degrees to the head. As a result, even when three-dimensional images are projected into two dimensions, the images are viewed at an angle that makes them easy to understand even in two dimensions. Furthermore, in the case of four screens, the images are Live and Mask on the front, and Live and Mask on the side. Live is a normal fluoroscopic image, similar to a general X-ray, and is viewed in real time. Mask takes the difference (subtraction) between the image and any past Live image selected by the surgeon. As a result, bones visible in Live disappear, and only blood vessels and devices visible with contrast agent are visible, providing an image that is easy for the surgeon to understand.

[0150] The two screens of different sizes can be switched automatically or by the user (surgeon) selection (see Figure 18). Since there is a physical limit to the screen size and it may be desired to view one of them larger, visibility can be improved by displaying the two screens at different sizes as shown in Figure 18.

[0151] Furthermore, the screen to be notified of can be made known to the user by lighting up the frame of the screen, changing the color, or highlighting it (see FIG. 19). As shown in FIG. 19, by notifying the user with a colored frame or the like, it becomes easier for the user to know which screen to pay attention to. When a notification is issued on the smaller screen, the two screens may be automatically swapped (so that the part to pay attention to is switched to the larger, easier-to-see screen). Thus, in some aspects of the present invention, the display device may have a function of calling the user's attention by lighting up, changing the color, or highlighting the frame of one of the two screens.

[0152] (Probability-based display) In some aspects of the present invention, since the area of ​​interest within the screen is output as a probability distribution, it is also possible to express the existence of the area of ​​interest in terms of probability. The probability distribution of the existing area may be displayed as a numerical value, a color, a bar, or the like. Furthermore, it is also possible to express whether or not a scene requires notification as a probability. Whether or not a scene requires notification may be displayed as a numerical value, a color, a bar, or the like according to the probability. Furthermore, which part of the screen is the responsible part may be displayed as a heat map or the like as a probability distribution. The probability distribution may be displayed in a more easily understandable way. For example, 0-30% may be displayed as low, 30-70% as middle, and 70-100% as high, or it may be displayed in three colors. As another example, the area below 70% may be slightly darkened so that the area of ​​interest or the area to be notified appears bright like a spotlight.

[0153] Therefore, in some aspects of the present invention, the notification unit can display, on the display device that displays the image, a numerical value, color, bar, or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a numerical value, color, bar, or heat map based on a value obtained by applying an arbitrary transformation to the probability distribution. The notification unit may also color the regions of interest with a color or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a color or heat map based on a value obtained by arbitrarily transforming the probability distribution, and display the color or heat map on the display device that displays the image, or replace the probability of satisfying the condition with a numerical value or color and display the color or heat map on the display device that displays the image.

[0154] (Device selection display and recording) There are various devices used in endovascular treatment, and each of them has many types. Examples of devices include various catheters, balloon catheters, guidewires, stents, flow diverter stents (fine mesh stents), coils, embolic materials (liquids, particles, etc.), and other embolic devices (WEB, etc.). There are also various types of each device. For example, catheters have standards for tip shape, length, inner lumen, outer lumen, and hardness. Coils have standards for manufacturer, thickness, total length, diameter, hardness, and there are hundreds of types. It is impossible to remember all of these, and it is not known whether they are in stock, so the surgeon checks with the vendor during the operation and proceeds with the treatment. Combinations are also important, and the size of the coil is determined by looking at the image. For an aneurysm, about 5 to 15 coils are usually used, so it is necessary to consider which one to use next, but it is difficult for the surgeon and assistant to remember the inventory management and lineup. It is difficult to grasp the situation because new products are released and old products are discontinued, and the lineup varies depending on the facility. Currently, devices are selected by communicating with providers during treatment, but the process is not smooth.

[0155] First, create a database of the lineup and specifications of devices currently available in the country. It is also possible to enter stock information at each facility. A system is constructed to display this information on a monitor. For example, when a microcatheter is selected, a list of the lineup, specifications, and stock information of available microcatheters is displayed. Additional information, such as whether the device or multiple devices can be placed in the guiding catheter, can also be searched. For example, when the lumen of the guiding catheter is R and the outer diameters of the two devices are r1 and r2, if R>r1+r2, the two devices can be placed in the guiding catheter. A similar approach can be used for one device or three or more devices.

[0156] As another example, in the case of coils, the coil lineup (length, diameter, hardness, shape, etc.) is selected depending on the size of the aneurysm and the behavior of the coil previously inserted. There are hundreds of coil lineups, but it is easy to list them by specifying the length, etc. By displaying them on the monitor, it becomes easier to select during treatment. Since most coil selections involve the same or smaller diameter or length, it is possible to present a lineup of coils that are likely to be used based on information on coils used up to that point. It is also possible to present coils that take stock into consideration. In addition, it is possible to include preferences according to facilities and surgeons (users) in the lineup candidates. Image analysis may be used to suggest options according to the aneurysm and the coil winding method. The surgeon selects the desired device from such a list and performs the surgery. This information can be recorded and automatically created as a surgical record by combining it with snapshots of the treatment video.

[0157] In some aspects of the present invention, a display device that displays images can display a product list of intravascular inspection or treatment devices (e.g., various catheters and coils) (see FIG. 20). The display device may also display a product list filtered by size or availability. Furthermore, the display device may display a list of recommended products based on image analysis results, facility information, or user preference information.

[0158] In some aspects of the invention, the image processing device can automatically or based on user selection create a surgical record that includes information on the devices used, the images acquired, and the results of image analysis.

[0159] Therefore, some aspects of the present invention also relate to an endovascular surgery support system that includes a memory unit that stores a product list of intravascular inspection or treatment devices (e.g., various catheters and coils), a recommendation unit that recommends products to be used based on image analysis results, facility information, or user preference information, and a display unit that displays the recommended products.

[0160] (Example System) Some aspects of the present invention relate to a vascular catheter surgery support system, for example, a catheter surgery support system for the blood vessels of the brain, heart, peripheral limbs, and abdomen, particularly for the blood vessels of the brain. Such a system includes an image processing device and an image capturing device that captures X-ray images of a patient with one or more devices inserted into the blood vessel and transmits them to the image processing device, and the image processing device includes an image acquisition unit that acquires X-ray images of a region (e.g., a fixed region) including at least a portion of the device inserted into the blood vessel over time, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, and a tracking unit that tracks at least one of the regions of interest. and a notification unit that notifies a user of the image processing device when one of the devices satisfies a condition defined for each region of interest, and when the one or more devices are a catheter, a guidewire, a stent, and / or a balloon, and an area including the tip of a catheter or the tip of a guidewire, both ends of a stent, or both ends of a balloon is set as the region of interest, the system may notify the user under the conditions that the region of interest disappears from the image, or the distance between the region of interest and the edge of the image becomes less than a predetermined threshold distance, or the region of interest has shifted a certain distance.

[0161] Here, the notification unit may cause the display device that displays the image to display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest, and the notification unit may change the display mode of the distance on the display device according to the magnitude of the distance, and the change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a figure on the entire screen of the display device or outside the frame or a part of the screen, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of a mark added to the region of interest according to the magnitude of the distance. Furthermore, the notification unit may sound a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0162] Some aspects of the present invention relate to an aneurysm coil embolization assist system, particularly a cerebral aneurysm coil embolization assist system. Such a system may include an image processing device, an image capturing device that captures X-ray images of a patient with a guiding catheter and a delivery wire of an embolization coil inserted in a blood vessel and transmits the images to the image processing device, and the image processing device includes an image capture unit that time-sequentially captures X-ray images of a fixed region including at least an aneurysm generated in the patient's blood vessel, the catheter inserted in the blood vessel, and the delivery wire of the embolization coil, a region of interest capture unit that captures one or more regions including at least a part of the guiding catheter included in the images as regions of interest, a marker detection unit that detects a marker provided on the delivery wire that approaches one or more regions of interest set on a part of the catheter that guides the delivery wire, a tracking unit that tracks the region of interest and each of the markers in the images, and a notification unit that notifies a user of a timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap.

[0163] Here, the notification unit may cause the display device that displays the image to display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest, and the notification unit may change the display mode of the distance on the display device according to the magnitude of the distance, and the change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a figure on the entire screen of the display device or outside the frame or a part of the screen, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of a mark added to the region of interest according to the magnitude of the distance. Furthermore, the notification unit may sound a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0164] <Processing flow of the image processing method executed by the image processing device 1> 10 is a flowchart for explaining the flow of image analysis processing executed by the image processing device 1 according to the embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0165] The image acquisition unit 110 acquires an X-ray image created based on the X-ray absorption rate, the X-ray image including at least a blood vessel V and an inspection or treatment device D inside the blood vessel V as subjects (S2). The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device D included in the X-ray image as a region of interest R (S4).

[0166] The tracking unit 112 tracks each region of interest R in the X-ray image (S6). If at least one of the regions of interest R satisfies the condition defined for each region of interest R (Yes in S8), the notification unit 113 notifies the user of the image processing device 1 of this (S10). If none of the regions of interest R satisfies the defined condition (No in S8), the notification unit 113 skips the notification process.

[0167] Until the image processing is completed (No in S12), the image processing device 1 returns to the process of step S6 and repeats the processes from step S6 to step S10. If the image processing is completed (Yes in S12), the process in this flowchart ends.

[0168] <Diagnosis and comparison of angiography> In angiography examinations and treatments, blood vessels are photographed by contrast to diagnose lesions, but lesions may be overlooked or may need to be judged by comparing the images with images taken earlier on the same day or on a different day, which may take time or be difficult to compare. Also, because the images are projected in two dimensions, it may be difficult to judge. Therefore, some aspects of the present invention relate to an imaging diagnostic device that uses deep learning or the like in angiography using contrast agents to indicate lesions or sites, including but not limited to cerebral aneurysms, stenosis, occlusion, thrombus formation, vascular perforation (contrast agent leakage), shunt disease, tumor vessel nutrient / tumor staining, venous thrombosis, avascular areas in the capillary phase (findings of vascular occlusion), collateral circulation, and the like.Therefore, in some aspects of the present invention, the image processing device detects in the image aneurysms, stenosis, vasospasm, dissection, occlusion, recanalization, thrombus formation, the site and both ends of a thrombus, vascular perforation, extravascular leakage of contrast medium, vascular calcification, arteriosclerosis, shunt disease and its nutrient blood vessels and outflow blood vessels, blood (contrast medium) reflux, cerebral arteriovenous malformation, dural arteriovenous fistula, avascular area, bone mark (internal auditory canal, fundus, supraorbital rim, petrous part, foramen magnum, cervical vertebrae, , clavicle, rib / vertebral number, femoral head, pelvis), tumor blood vessels feeding the tumor / tumor staining, venous occlusion, venous sinus thrombosis, avascular area in the capillary phase, vascular occlusion, shape and distribution of the coil within the aneurysm, position / bulge / shape of the balloon, deviation of the coil into normal blood vessels, insufficient expansion of the stent, degree of adhesion to the blood vessel or twisting, movement of the stent, position of both ends of the stent, positional relationship between the puncture site and the blood vessel (is there a stenosis or is it near a bifurcation?), The present invention may further include a lesion recognition unit that recognizes lesions selected from the group consisting of vascular tortuosity, aortic arch type (how far below the top of the aortic arch the right brachiocephalic artery is), penetration range of liquid embolic material, delay or stagnation of blood (contrast medium) flow, vascular variation (anterior communicating artery, anterior cerebral artery A1, posterior communicating artery, posterior cerebral artery P1, posterior inferior cerebellar artery, anterior inferior cerebellar artery, superior cerebellar artery, superficial temporal artery, each venous sinus, presence or absence and degree of development of each vein), moyamoya vessels (stenosis or occlusion at the tip of the internal carotid artery and development of collateral blood flow beyond that), position of arterial branching and segment (petrosis of the internal carotid artery, cavernous sinus, ophthalmic artery, branching of middle cerebral artery M1), traces of previous surgery (clips, coils, plates, shunt tubes / valves, ventricular tubes, cisternal tubes), position and degree of opening of the WEB device, foreign bodies (dentures, plates), and collateral blood flow. It is also possible to notify the doctor when an abnormality is suspected, or to indicate the area where an abnormality may occur, without indicating the abnormality. In this case, the doctor can make the final decision. Similarly, some aspects of the present invention relate to an image diagnostic device that indicates changes in angiography by comparing the angiography with a previous image or an image taken on a different day. Thus, in some aspects of the present invention, the image processing device may further include an image recognition unit that compares an angiography image in an image with an angiography image previously obtained and stored, and notifies the doctor of changes.For example, changes in the degree of vasospasm, changes in thrombus formation (appearance, disappearance, enlargement, reduction, etc.), release of blocked blood vessels, blockage of blood vessels, displacement of coils, migration of stents, etc. can be indicated.

[0169] <Advantages of the Image Processing Device 1 According to the Embodiment> As described above, the image processing device 1 according to the embodiment can provide a technology that enables a medical professional user to focus on the work in the area of ​​interest during vascular catheter examination or treatment, and supports the user in determining the area of ​​interest.

[0170] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

[0171] <First Modification> Although the above description has been focused on the examination or treatment of cerebrovascular disease, the present invention is not limited to the application of the present invention, and can be applied to the examination and treatment of blood vessels in the circulatory system, such as the heart, peripheral limbs, and abdomen.

[0172] <Second Modification> In the above, an example of a device with two screens as shown in Figs. 7(a)-(b) has been described, but the number of screens is not limited to this, and may be, for example, one screen or three or more screens.

[0173] <Third Modification> In the above, a case has been described in which the X-ray imaging device 3 captures an image of the operative part of the subject P. However, the imaging device that captures the image of the operative part of the subject P is not limited to the X-ray imaging device 3. In addition, the image of the operative part may be captured using a modality such as MRI (Magnetic Resonance Imaging) or an ultrasonic imaging device.

[0174] <Example of embodiment> A part or all of the above embodiments can be described as follows, but the disclosure of the present application is not limited to the following notes.

[0175] (Appendix 1) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; An image processing device comprising: (Appendix 2) the notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, on condition that the region of interest disappears from the image. 2. An image processing device as described in claim 1. (Appendix 3) The notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, on condition that a distance between the region of interest and an edge of the image is less than a predetermined threshold distance. 2. An image processing device as described in claim 1. (Appendix 4) The notification unit notifies the user on condition that at least one of a moving distance, a moving speed, and an acceleration of the region of interest in the image exceeds a predetermined threshold. 4. The image processing device according to claim 1 , (Appendix 5) The notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image. 5. An image processing device according to claim 1 . (Appendix 6) The notification unit changes a display manner of the distance on the display device depending on a distance between the region of interest and an edge of the image. 6. The image processing device according to claim 5. (Appendix 7) The notification unit notifies the user on condition that a value obtained by dividing a distance between the region of interest and an edge of the image by a moving speed of the region of interest within the image is less than a predetermined threshold. 7. An image processing device according to any one of claims 1 to 6. (Appendix 8) The method further includes a marker detector that detects a marker provided on a delivery wire of the embolization coil approaching a region of interest set on a part of a microcatheter that guides the delivery wire, The tracking unit further tracks the detected marker; and the notification unit notifies the user of a timing at which the embolization coil may be disconnected from the delivery wire, when the marker and the region of interest overlap. 8. An image processing device according to claim 1 . (Appendix 9) The notification unit notifies the user when the marker passes through the region of interest. 9. The image processing device according to claim 8. (Appendix 10) the notification unit causes a display device to display a distance that the marker should move until the embolization coil is disconnected from the delivery wire. 10. The image processing device according to claim 8 or 9. (Appendix 11) and when a feature amount indicating a shape of the device included in the region of interest satisfies a predetermined condition, the notification unit notifies a user of the image processing device of that fact. 11. An image processing device according to any one of claims 1 to 10. (Appendix 12) the feature value is a curvature, The notification unit notifies the user when a curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or when the tip does not move despite a change in curvature. 12. The image processing device according to claim 11. (Appendix 13) The notification unit notifies the user on condition that a value obtained by subtracting a length of a center line of the blood vessel included in the image or the region of interest from a length of the device included in the image or the region of interest exceeds a predetermined threshold length. 13. The image processing device according to claim 11 or 12. (Appendix 14) The image processing device of any one of appendices 1 to 13, wherein the notification unit notifies the user by displaying the area of ​​interest in a color different from that of the image, changing the font, size or color of the displayed characters, changing the color of the entire screen of the display device or a portion of it, displaying a graphic on the entire screen of the display device or outside the frame or a portion of it, enlarging the area of ​​interest, or changing the color or size of a mark added to the area of ​​interest.

[0176] (Appendix 15) The notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest and the region of interest has moved or has exceeded a specific range specified on the image. 15. An image processing device according to any one of claims 1 to 14. (Appendix 16) 16. The image processing device according to any one of claims 1 to 15, further comprising a video storage unit that stores images or videos obtained from the image acquisition unit over time. (Appendix 17) 17. The image processing device of claim 16, further comprising a video extraction unit that extracts from the video storage unit video of a certain period before and after the notification unit issues a notification, or video of any time or period specified by the user. (Appendix 18) 18. The image processing device according to claim 17, wherein the extraction period of the image is automatically determined based on at least one of the moving distance, the moving speed, and the acceleration of the area of ​​interest when the notification occurs. (Appendix 19) 19. The image processing device according to claim 17 or 18, wherein the extracted image is displayed on a display device. (Appendix 20) 20. The image processing device of claim 19, wherein the extracted image is automatically displayed repeatedly a predetermined number of times. (Appendix 21) 21. The image processing device of claim 20, wherein the extracted image is displayed based on any operation including play, stop, fast forward, rewind, frame forward, slow playback, and double speed playback. (Appendix 22) 22. The image processing device according to any one of appendices 17 to 21, wherein the time elapsed since the occurrence of the notification, a comparison between the position of the area of ​​interest after the occurrence of the notification and an arbitrary time has elapsed, or a trajectory of the area of ​​interest acquired by the tracking unit is further displayed superimposed on the extracted image. (Appendix 23) 23. The image processing device according to any one of appendices 17 to 22, wherein the extracted image is displayed by cutting out a partial area in the vicinity of the region of interest. (Appendix 24) 24. The image processing device according to any one of appendices 17 to 23, wherein the extracted image is displayed at a position that does not interfere with the display of the region of interest. (Appendix 25) 25. The image processing device according to any one of appendices 17 to 24, wherein the extracted image is enlarged and displayed. (Appendix 26) 26. The image processing device of any one of appendices 17 to 25, wherein the extracted image is displayed simultaneously with the occurrence of a notification or after a predetermined time has elapsed since the occurrence of the notification. (Appendix 27) 27. The image processing device according to any one of appendix 17 to 26, wherein images captured from multiple directions are displayed simultaneously. (Appendix 28) 28. The image processing device according to any one of appendix 2 to 27, further comprising a state estimation unit that estimates a current position and / or velocity of the tip of the catheter or the tip of the guidewire that has disappeared from the image, based on the position, velocity and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappeared from the image. (Appendix 29) 29. The image processing device of claim 28, wherein a warning is notified to a user when a current position and / or velocity of the region of interest estimated by the state estimation unit exceeds a predetermined threshold. (Appendix 30) 30. The image processing device according to any one of claims 1 to 29, wherein the display device that displays the images displays the two images on two screens of different sizes. (Appendix 31) 31. The image processing device according to claim 30, wherein the display device alerts the user by making a frame portion of one of the two screens glow, change a color, or highlight it. (Appendix 32) 32. The image processing device according to any one of claims 1 to 31, wherein the display device for displaying the image displays a product list of devices for intravascular inspection or treatment. (Appendix 33) 33. The image processing device of claim 32, wherein the display device displays a list of products narrowed down by size or stock. (Appendix 34) 34. The image processing device of claim 32 or 33, wherein the display device displays a list of recommended products based on image analysis results, facility information, or user preference information. (Appendix 35) An image processing device according to any one of claims 1 to 34, which creates a surgical record including information on the device used, information on the images acquired, and image analysis results, automatically or based on user selection. (Appendix 36) An image processing device as described in any one of Appendix 1 to 35, wherein the notification unit displays, on a display device that displays the image, a numerical value, color, bar, or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition set for each of the regions of interest, or a numerical value, color, bar, or heat map based on a value obtained by applying an arbitrary transformation to a probability distribution. (Appendix 37) The image processing device according to any one of appendixes 1 to 35, wherein the notification unit colors the region of interest with a color or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition set for each region of interest, or with a color or heat map based on a value obtained by arbitrarily converting a probability distribution, and displays the color or heat map on the display device that displays the image, or replaces the probability of satisfying a condition with a numerical value or color and displays the result on the display device that displays the image. (Appendix 38) The image processing device according to any one of appendices 1 to 37, wherein the notification unit notifies the user when, when a region including the tip of a catheter or the tip of a guidewire is set as the region of interest, the region of interest has moved or has exceeded a specific range specified on an image. (Appendix 39) 39. The image processing device of claim 38, wherein the boundary line of the specific range is represented by a straight line, a curve, a circle, a rectangle, or another polygon. (Appendix 40) 40. The image processing device of claim 38, wherein the specific range is superimposed on an X-ray image. (Appendix 41) 41. The image processing device according to any one of claims 38 to 40, wherein the notification unit causes a display device that displays the image to display the distance between the region of interest and an edge of the specific range. (Appendix 42) The image processing device according to claim 41, wherein the notification unit changes a manner in which the distance is displayed on the display device depending on the distance between the region of interest and an edge of the specific range. (Appendix 43) 43. The image processing device according to any one of appendices 38 to 42, wherein the notification unit notifies the user on condition that a value obtained by dividing the distance between the area of ​​interest and the edge of the specific range by a moving speed of the area of ​​interest within the image is less than a predetermined threshold. (Appendix 44) 44. The image processing device of any one of claims 3 to 43, wherein the distance is determined by either a straight-line distance or a distance along a blood vessel. (Appendix 45) 45. An image processing device according to any one of claims 1 to 44, comprising a memory unit that acquires and stores the position and / or shape of an intravascular inspection or treatment device at any time, and the stored position and / or shape of the device is superimposed on subsequent images. (Appendix 46) 43. The image processing device according to any one of appendix 1 to 42, further comprising a lesion recognition unit that recognizes lesions in the image selected from the group consisting of cerebral aneurysm, stenosis, occlusion, thrombus formation, vascular perforation, extravascular leakage of contrast agent, shunt disease, nutrient blood vessels of tumor blood vessels / tumor staining, venous thrombosis, avascular areas in the capillary phase, vascular occlusion, and collateral circulation. (Appendix 47) 44. The image processing device of any one of claims 1 to 43, further comprising an image recognition unit for comparing angiograms in the image with previously acquired and stored angiograms and notifying changes.

[0177] (Appendix 48) A processor of the image processing device acquiring an image including at least an intravascular inspection or treatment device as a subject; acquiring one or more regions including at least a portion of the device included in the image as a region of interest; tracking each of the regions of interest in the image; If at least one of the regions of interest satisfies a condition defined for each of the regions of interest, notifying a user of the image processing device of that fact; An image processing method that performs

[0178] (Appendix 49) On the computer, A function of acquiring an image including at least an intravascular inspection or treatment device as a subject; a function of acquiring one or more regions including at least a portion of the device included in the X-ray image as a region of interest; tracking each of said regions of interest in said x-ray image; a function of notifying a user of the computer when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; A program to achieve this.

[0179] (Appendix 50) An image processing device according to any one of appendix 1 to 44, an imaging device that captures an image of a person having a device for intravascular inspection or treatment inserted therein and transmits the image to the image processing device; An image processing system comprising:

[0180] (Appendix 51) A cerebrovascular catheter surgery support system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with one or more devices inserted into a blood vessel and transmits the image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed area including at least a target area for achieving the purpose of surgery and a device inserted into a blood vessel over time; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with the one or more devices are catheters, guidewires, stents, and / or balloons; When a region including a tip end of a catheter or a tip end of a guidewire, both ends of a stent, and both ends of a balloon is set as the region of interest, the user is notified on the condition that the region of interest disappears from the image or the distance between the region of interest and an edge of the image becomes less than a predetermined threshold distance. system. (Appendix 52) A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the X-ray image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed area including at least an aneurysm occurring in a patient's blood vessel, a catheter inserted into the blood vessel, and a delivery wire of an embolization coil over time; a region of interest acquisition unit configured to acquire one or more regions including at least a part of the guiding catheter included in the image as a region of interest; a marker detection unit that detects a marker provided on the delivery wire approaching one or more regions of interest set in a part of the catheter that guides the delivery wire; and a tracking unit for tracking each of the region of interest and the marker in the image; a notification unit that notifies a user of a timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap; A system comprising: (Appendix 53) the notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image or a distance between the marker and the region of interest; Here, the notification unit can change the display manner of the distance on the display device depending on the distance, and the change in display manner includes changing the font, size, or color of characters displayed depending on the distance, changing the color of the entire screen of the display device or a portion of it depending on the distance, displaying a figure on the entire screen of the display device, outside the frame, or a portion of it, enlarging and displaying a region of interest depending on the distance, or changing the color or size of a mark added to the region of interest depending on the distance. (Appendix 54) 54. The system of any one of appendixes 51 to 53, wherein the notification unit can emit a notification sound or transmit vibration depending on the magnitude of the distance. (Appendix 55) 55. The system of any one of claims 51 to 54, wherein the distance is determined either by a straight-line distance or a distance along a blood vessel. (Appendix 56) A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the X-ray image to the image processing device; Equipped with The image processing device, a positional relationship storage unit that stores a positional relationship between the tip of the catheter and the second marker; a position storage unit that stores the distance a between the neckline of the aneurysm and the first marker and the position of the second marker at the time t1; a distance estimating unit that calculates a moving distance b from the position of the second marker at time t2 and estimates a distance ab from the aneurysm neckline to the tip of the catheter; a notification unit that notifies a user of the estimated distance; A system comprising: (Appendix 57) 547. The system of claim 546, wherein the estimated distance is represented by a probability distribution. (Appendix 58) 58. The system of claim 56 or 57, wherein the estimated catheter tip position is displayed colored based on a probability distribution.

[0181] (Appendix 59) An endovascular surgery support system comprising: a memory unit that stores a product list of devices for intravascular examination or treatment; a recommendation unit that recommends products to use based on image analysis results, facility information, or user preference information; and a display unit that displays the recommended products. [Explanation of symbols]

[0182] 1. Image processing device 10...Storage section 11. Control section 110...Image acquisition unit 111... Region of interest acquisition unit 112 Tracking Unit 113... Notification Department 114 Marker detection unit 115... Distance measurement section 2...Display device 3. X-ray imaging device 30...X-ray irradiator 31. X-ray detector 32...Bed D...Device E···Embolization coil P...Subject S...Image Processing System

Claims

1. An image processing device for supporting vascular catheter surgery, comprising: an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; a recognition unit that recognizes a lesion or a site in the image, or an image recognition unit that compares an angiographic image in the image with an angiographic image that has been previously acquired and stored to recognize changes; Equipped with The image acquisition unit is an image processing device that acquires X-ray images over time of an area fixed to a lesion of a subject, the area including at least a target area for achieving the purpose of vascular catheter surgery and a device inserted into a blood vessel.

2. 2 . The image processing device according to claim 1 , wherein a lesion or site recognized by the recognition unit is notified, or a suspected abnormal finding is notified, or a possibly abnormal area is notified.

3. The lesion or site in the image may be an aneurysm, stenosis, vasospasm, dissection, occlusion, recanalization, thrombus formation, the site and both ends of a thrombus, vascular perforation, extravascular leakage of contrast medium, vascular calcification, arteriosclerosis, shunt disease and nutrient and outflow vessels of shunt disease, reflux of blood (contrast medium), cerebral arteriovenous malformation, dural arteriovenous fistula, avascular area, bone mark (internal auditory canal, fundus, supraorbital rim, petrous part, foramen magnum, cervical vertebrae, clavicle, rib - Vertebral number, femoral head, pelvis), tumor blood vessels feeding the tumor, tumor staining, venous occlusion, venous sinus thrombosis, avascular area in the capillary phase, vascular occlusion, shape and distribution of the coil within the aneurysm, position, inflation and shape of the balloon, deviation of the coil into normal blood vessels, insufficient expansion of the stent, degree of adhesion to the blood vessel, twisting, movement of the stent, position of both ends of the stent, positional relationship between the puncture site and the blood vessel (is there a stenosis, is it near a branching area), blood vessels 3. The image processing device according to claim 1 or 2, wherein the tortuosity of the aortic arch is selected from the group consisting of tortuosity of the aortic arch, type of aortic arch (how far the right brachiocephalic artery is below the top of the aortic arch), penetration range of liquid embolic material, delay or stagnation of blood (contrast medium) flow, variation of blood vessels (anterior communicating artery, anterior cerebral artery A1, posterior communicating artery, posterior cerebral artery P1, posterior inferior cerebellar artery, anterior inferior cerebellar artery, superior cerebellar artery, superficial temporal artery, presence or absence and degree of development of each venous sinus and each vein), moyamoya vessels (stenosis or occlusion at the tip of the internal carotid artery and development of collateral blood circulation beyond that), position of arterial branching and segment (petrosal portion of the internal carotid artery, cavernous sinus portion, ophthalmic artery portion, branching portion of middle cerebral artery M1), traces of previous surgery (clips, coils, plates, shunt tubes / valves, ventricular tubes, cisternal tubes), position and degree of opening of a WEB device, foreign bodies (dentures, plates), and collateral blood circulation.

4. 3. The image processing device according to claim 1, wherein the lesion or site in the image is selected from the group consisting of cerebral aneurysm, stenosis, occlusion, thrombus formation, vascular perforation, contrast agent leakage, shunt disease, tumor vasculature, tumor staining, venous thrombosis, avascular area in the capillary phase, findings of vascular occlusion, and collateral circulation.

5. 5. The image processing device according to claim 1, wherein a change in the degree of vasospasm, a change in thrombus formation, release of an occluded blood vessel, occlusion of a blood vessel, deviation of a coil, or movement of a stent recognized by the image recognition unit is notified.

6. The image processing device according to any one of claims 1 to 5, wherein the notification unit notifies the user when, when a region including the tip of a catheter or the tip of a guidewire is set as the region of interest, the region of interest has moved or has exceeded a specific range specified on an image.

Images