Image processing device, image processing method, program, and image processing system

Abstract

To provide technique for supporting a healthcare worker during a catheter examination or medical treatment of a blood vessel so as to facilitate concentration on work in an area of attention and provide support in making determination regarding the area of attention.SOLUTION: In an image processing device 1, an image acquisition unit 110 acquires an X-ray image, created on the basis of an absorption rate of X-rays, where at least a blood vessel and a device for examining or medically treating the inside of the blood vessel are included in a photographic subject. A region-of-interest acquisition unit 111 receives, as a region-of-interest, one or a plurality of regions including at least a portion of the device included in the X-ray image. A tracing unit 112 traces each region-of-interest in the X-ray image. If at least any one region-of-interest satisfies a condition established for each region-of-interest, a notification unit 113 notifies a user of the image processing device 1 of the fact. The present disclosure also includes recording actions during a surgery and displaying a list of tools that are used.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a program, and an image processing system, and particularly to an image processing technique for use in vascular examination or treatment.

Background Art

[0002] In recent years, for the examination and treatment of blood vessels throughout the body such as the brain and heart, catheter surgery has been performed in which medical personnel such as doctors pass a catheter through the blood vessels of a subject and perform various procedures while displaying the position of the catheter in an X-ray image. In this catheter surgery, generally, medical personnel pass a medical device such as a catheter through the groin or upper arm of the subject and advance the device to the target area.

[0003] For example, during cerebrovascular examination or treatment, medical personnel advance a catheter from the groin or brachial artery of the subject through the aortic arch and carotid artery to reach the cerebral artery. Since a lot of experience and training are required for medical personnel to perform these procedures appropriately, for example, Patent Document 1 discloses a simulation system for improving the efficiency of surgical procedures performed using a catheter system.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In vascular examinations and treatments, after a medical practitioner passes a guiding catheter to the target site, a plurality of examination or treatment devices such as a guide wire, a stent, a balloon, a microcatheter, a liquid embolization substance, a filter, and a delivery wire for an embolization coil for embolizing an aneurysm are passed through the guiding catheter. That is, in actual examinations and treatments, a medical practitioner needs to pay attention not only when passing the guiding catheter to the target site but also to the movement and position of a plurality of devices during the procedure. On the other hand, in order to achieve the original purpose of the examination or treatment, a medical practitioner must concentrate on the work at the target area.

[0006] The present invention has been made in view of these points, and an object thereof is to provide a technique for concentrating a medical practitioner on the work at the target area and supporting the determination of the target area in catheter examinations or treatments of blood vessels.

Means for Solving the Problems

[0007] A first aspect of the present invention is an image processing apparatus. This apparatus includes an image acquisition unit that acquires an image including at least a subject, which is an examination or treatment device in a blood vessel, a region-of-interest acquisition unit that acquires one or a plurality of regions including at least a part of the device included in the image as regions of interest, a tracking unit that tracks each of the regions of interest in the image, and a notification unit that notifies a user of the image processing apparatus when at least one of the regions of interest satisfies a condition determined for each region of interest.

[0008] When a region including the tip of a catheter such as a guiding catheter or the tip of a guide wire is set as the region of interest, the notification unit may notify the user on the condition that the region of interest disappears from the image.

[0009] When the region of interest includes the tip of a catheter such as a guiding catheter or the tip of a guide wire, the notification unit may notify the user on the condition that the distance between the region of interest and the edge of the image is less than a predetermined threshold distance.

[0010] The notification unit may notify the user on the condition that at least any one of the moving distance, moving speed, and acceleration of the region of interest in the image exceeds a predetermined threshold value.

[0011] The notification unit may cause the display device that displays the image to display the distance between the region of interest and the edge of the image.

[0012] The notification unit may change the display mode of the distance on the display device according to the magnitude of the distance between the region of interest and the edge of the image.

[0013] The notification unit may notify the user on the condition that the value obtained by dividing the distance between the region of interest and the edge of the image by the moving speed of the region of interest in the image is less than a predetermined threshold value.

[0014] The image processing apparatus may further include a marker detection unit that detects a marker provided on a delivery wire of a coil for embolization and approaches a region of interest set in a part of a guiding catheter that guides the delivery wire. The tracking unit may further track the detected marker. The notification unit may notify the user of the timing at which the coil for embolization may be cut from the delivery wire when the marker and the region of interest overlap.

[0015] When the marker passes through the region of interest, the notification unit may notify the user of that fact.

[0016] The notification unit may cause a display device to display the distance that the marker should move before the plug coil is cut from the delivery wire.

[0017] When a feature amount indicating the shape of the device included in the region of interest satisfies a predetermined condition, the notification unit may notify the user of the image processing apparatus of this fact.

[0018] The feature amount may be curvature, and the notification unit may notify the user on the condition that the curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or that the tip does not move even though the curvature is changing.

[0019] The notification unit may notify the user on the condition that a value obtained by subtracting the length of the center line of the blood vessel included in the image or region of interest from the length of the device included in the image or region of interest exceeds a predetermined threshold length.

[0020] The notification unit may have functions such as coloring the region of interest in a color different from that of the image and displaying it, changing the font, size, or color of the displayed characters, changing the color of the entire screen or a part of the display device, displaying a figure on the entire screen, outside the frame, or a part of the display device, enlarging the display of the region of interest, or changing the color or size of the mark attached to the region of interest to notify the user.

[0021] The notification unit may use sound or vibration for notification.

[0022] A second aspect of the present invention is an image processing method. In this method, a processor of an image processing apparatus executes steps of: acquiring an image including at least a subject and a device for inspection or treatment inside a blood vessel; acquiring, as a region of interest, one or a plurality of regions including at least a part of the device included in the image; tracking each of the regions of interest in the image; and notifying a user of the image processing apparatus when at least any one of the regions of interest satisfies a condition defined for each of the regions of interest.

[0023] A third aspect of the present invention is a program. This program causes a computer to realize functions of: acquiring an image including at least a subject and a device for inspection or treatment inside a blood vessel; acquiring, as a region of interest, one or a plurality of regions including at least a part of the device included in the X-ray image; tracking each of the regions of interest in the X-ray image; and notifying a user of the computer when at least any one of the regions of interest satisfies a condition defined for each of the regions of interest.

[0024] To provide this program or to update a part of the program, a computer-readable recording medium recording this program may be provided, or this program may be transmitted via a communication line.

[0025] A fourth aspect of the present invention is an image processing system. This system includes the above-described image processing apparatus and an imaging apparatus that captures an image of a person with a device for inspection or treatment inside a blood vessel (an image of an operative field) inserted therein and transmits the image to the image processing apparatus.

[0026] Note that any combination of the components disclosed in this specification, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. are also effective as aspects of the present invention.

Advantages of the Invention

[0027] According to the present invention, in catheter examination or treatment of blood vessels, it is possible to provide a technique for concentrating medical staff on the work at the attention area and supporting oversight and judgment delays outside the attention area.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Embodiments for Carrying Out the Invention

[0029] <Overview of the Embodiment> FIG. 1 is a diagram schematically showing the appearance of an image processing system S according to an embodiment. The image processing system S includes an image processing apparatus 1, a display apparatus 2, and an X-ray imaging apparatus 3. Hereinafter, the overview of the embodiment will be described with reference to FIG. 1.

[0030] The X-ray imaging device 3 is a device for imaging an X-ray image of a subject P who is a person with a device for blood vessel examination or treatment (hereinafter sometimes simply referred to as "device") inserted, and transmitting the X-ray image to the image processing device 1. For this reason, the X-ray imaging device 3 includes an X-ray irradiator 30 (the first X-ray irradiator 30a and the second X-ray irradiator 30b) for irradiating the subject P with X-rays, an X-ray detector 31 for detecting the X-rays irradiated by the X-ray irradiator 30, and a bed 32 for supporting the subject P.

[0031] As shown in FIG. 1, the first X-ray irradiator 30a and the second X-ray irradiator 30b can irradiate the head of the subject P with X-rays at different incident angles. Here, the X-rays irradiated by the first X-ray irradiator 30a are detected by the first X-ray detector 31a and converted into an X-ray image based on the X-ray absorption rate. Note that the X-rays irradiated by the second X-ray irradiator 30b are detected by a second X-ray detector (not shown) and converted into an X-ray image. These X-ray images are displayed on the display device 2. During the operation in the cerebrovascular region, generally the positions of the first and second X-ray irradiators are fixed, and the region of the image displayed on the display device 2 is fixed.

[0032] In the image generated from the X-rays detected by the X-ray detector 31, various devices used for blood vessel examination and treatment (for example, catheters such as guiding catheters, guide wires, embolization coils, delivery wires for transporting the embolization coil to the target site, etc.) are imaged together with tissues such as blood vessels (which become visible by flowing a contrast agent) and bones of the subject P.

[0033] The image processing apparatus 1 according to the embodiment is a device for assisting a user who performs a catheter examination or treatment of blood vessels (hereinafter, simply referred to as "catheter surgery" unless distinguishing between a catheter examination of blood vessels and a catheter treatment of blood vessels). The image processing apparatus 1 recognizes and / or tracks one or more regions of interest preset in an X-ray image generated based on the X-rays detected by the X-ray imaging apparatus 3. By analyzing the X-ray image, when the state of any region of interest satisfies the conditions defined for each region of interest, the image processing apparatus 1 notifies a medical staff member (hereinafter simply referred to as the "user"), who is the user of the image processing apparatus 1. The user can concentrate on the work at the target site (for example, work such as guiding a micro catheter into an aneurysm, inserting a coil into an aneurysm, expanding a balloon, and implanting a stent) by setting, as regions of interest, areas that should be paid attention to other than the target area for achieving the purpose of the examination or treatment.

[0034] <Functional Configuration of Image Processing Apparatus 1 According to the Embodiment> FIG. 2 is a diagram schematically showing the functional configuration of the image processing apparatus 1 according to the embodiment. The image processing apparatus 1 includes a storage unit 10 and a control unit 11. In FIG. 2, the arrows indicate the main data flow, and there may be a data flow not shown in FIG. 2. In FIG. 2, each functional block shows a configuration in terms of functional units, not in terms of hardware (device) units. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be divided and implemented in a plurality of devices. The exchange of data between the functional blocks may be performed via any means such as a data bus, a network, a portable storage medium, or the like.

[0035] The memory unit 10 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) etc. of the computer that realizes the image processing apparatus 1, a RAM (Random Access Memory) that serves as the working area of the image processing apparatus 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referred to during the execution of the application programs.

[0036] The control unit 11 is a processor such as the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the image processing apparatus 1, and functions as an image acquisition unit 110, an area of interest acquisition unit 111, a tracking unit 112, a notification unit 113, a marker detection unit 114, and a distance measurement unit 115 by executing the programs stored in the memory unit 10.

[0037] Note that FIG. 2 shows an example when the image processing apparatus 1 is configured as a single device. However, the image processing apparatus 1 may be realized by a plurality of computing resources such as a plurality of processors and memories, such as in a cloud computing system. In this case, each part constituting the control unit 11 is realized by at least one of the plurality of different processors executing a program.

[0038] The image acquisition unit 110 acquires an X-ray image created based on the X-ray absorption rate, including at least the blood vessels of the subject P and a device for examination or treatment within the blood vessels as subjects. For example, the X-ray image may include an aneurysm that has occurred in the blood vessels of the subject P that medical staff are interested in, or a stenosis or infarction of the blood vessels. The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device included in the X-ray image as the region of interest. As the region of interest, for example, the tip of a guide wire (GW), the tip of a guiding catheter (GC), a catheter marker, a coil, etc. can be set, but multiple regions, for example, the tips of the GC and GW, the tips of two GWs, the tip of the GW and the coil, etc., multiple regions of interest may be set simultaneously. In some embodiments, the region of interest may include blood vessels (vascular lesion parts such as cerebral aneurysms and stenoses) or bones.

[0039] Figs. 3(a)-(d) are diagrams for explaining the region of interest. Specifically, Fig. 3(a) is a diagram schematically showing an example of an X-ray image of the blood vessel V. In the example shown in Fig. 3(a), a guide wire, which is a type of device D, exists in the blood vessel V.

[0040] Fig. 3(b) is a diagram showing an example of a candidate region C that is a candidate for the region of interest. The region of interest acquisition unit 111 may include a candidate region detector generated using a known machine learning method such as a neural network (Fig. 11). In Fig. 3(b), the first candidate region C1 (the tip of the guiding catheter) and the second candidate region C2 (the tip of the guide wire) are shown as candidate regions for the region of interest. This candidate region C is the result obtained when the region of interest acquisition unit 111 inputs the frame image of the X-ray image to the candidate region detector. Without limitation, as an example, the candidate region detector is trained to detect the tip of the guide wire, the tip of the guiding catheter, the catheter marker, etc.

[0041] Detection generally means identifying the position and shape of an object in a still image of a single frame in a video. For example, in detecting the tip of a guide wire, since the guide wire is thin, it may be detected as point coordinates (x, y) on the screen. Detecting the entire guide wire can be done by detecting a shape like a thin, curved thread as a one-dimensional curve or as a two-dimensional segmentation (segmentation).

[0042] As object detection algorithms that can be used in a candidate region detector, for example, algorithms such as Faster R-CNN, YOLO, SSD, U-Net, ResNet, etc. can be utilized, but it is not limited to these. In implementing an object recognition algorithm, for example, an image processing library such as OpenCV (Open Source Computer Vision Library) may be used.

[0043] Faster R-CNN is a CNN (Convolutional Neural Network) that simultaneously performs region extraction and recognition. A convolutional neural network (CNN) is a neural network with multiple deep layers composed by stacking layers with several characteristic functions such as "convolutional layers" and "pooling layers", and it exhibits excellent performance particularly in the field of image recognition. Faster R-CNN can extract and recognize regions of interest from an input image in almost real time (about 10 to 20 frames per second). In Faster R-CNN, end-to-end learning from image input to object detection is possible.

[0044] YOLO (You Only Look Once) is also a CNN that simultaneously performs region extraction and recognition. In YOLO, the entire image is divided into grids, and a bounding box is obtained for each region. The CNN architecture of YOLO enables fast object detection.

[0045] SSD (Single Shot MultiBox Detector) is also a CNN that performs region extraction and recognition simultaneously. SSD can output multi-scale detection frames from output layers of various hierarchies. SSD places approximately 9000 rectangular frames called default boxes of different sizes and shapes on an image and calculates prediction values for each frame. In SSD, the speed is increased by reducing the filter size.

[0046] U-Net is a CNN that recognizes objects pixel by pixel for segmentation. It is composed of convolutional layers and has an almost symmetric Encoder-Decoder structure. The feature map downsampled through pooling in the Encoder is upsampled by the Decoder.

[0047] Figure 3(c) is a diagram showing an example of a method for setting the region of interest R. In the example shown in Figure 3(c), it shows a state where the user has selected the second candidate region C2 as the region of interest R. Specifically, the user selects the region of interest R by moving the mouse cursor M to the second candidate region C2 using a pointing device (not shown) such as a mouse. Note that the presentation of the candidate region C by the region of interest acquisition unit 111 is arbitrary, and the user may directly set the region of interest R in the X-ray image in a state where the candidate region C is not presented. For example, the region of interest R can be set by drawing a rectangle in the image using a pointing device such as a mouse to specify the region. Alternatively, the region of interest acquisition unit 111 may acquire the output of a candidate region detector generated using a known machine learning method or the like as the region of interest R.

[0048] Figure 3(d) is a diagram showing the region of interest R set by the user. In the example shown in Figure 3(d), the region of interest R is shown as a black pentagon and is the region including the tip portion of the device D. In Figure 3(d), only one region of interest R is set, but the user can set two or more regions of interest R.

[0049] Return to the description of FIG. 2. The tracking unit 112 tracks each of one or more regions of interest R in the X-ray image. The tracking unit 112 can realize the tracking of the region of interest R by using known image tracking techniques. Examples of available known image tracking techniques include those using algorithms such as Boosting, MIL, TLD, MedianFlow, KCF, GOTURN, MOSSE, CSRT, etc., but are not limited thereto. The tracking algorithm may be used in combination with the above-described object detection algorithm. In implementing the algorithm, for example, a library such as OpenCV (Open Source Computer Vision Library) may be used. In the context of the present invention, the tracking of the region of interest R also includes intermittently detecting the region of interest R and identifying its state. Therefore, in some aspects of the present invention, it is not required that the region of interest R be recognized in all frames of the X-ray images acquired over time. In some aspects of the present invention, the tracking of the region of interest R can be performed using a tracking algorithm or an object detection algorithm, or a combination thereof.

[0050] The BOOSTING tracker is a tracker based on an online version of AdaBoost (the algorithm used internally by the HAAR cascade-based face detector). This classifier is trained at runtime using positive and negative examples of the object. The first bounding box specified by the user (or another object detection algorithm) is treated as a positive example of the object, and the image outside the bounding box is treated as the background. When a new frame is given, the classifier is applied to all pixels in the vicinity of the previous position, and the scores are recorded. The new position of the object is the position where the score is maximized. In this way, another positive example of the classifier is obtained. Further frames are input, and the classifier is updated with this additional data.

[0051] The MIL tracker is based on a similar concept to the above BOOSTING tracker. The big difference is that instead of considering only the current position of the object as a positive example, it examines a small neighborhood around the current position to generate several potential positive examples. In MIL, instead of specifying positive and negative examples, it specifies "bags" of positive and negative. A group of images in the positive bag are not all positive examples. Only one image in the positive bag needs to be a positive example. The positive bag contains images centered on the current position of the object and images of its small surrounding neighborhood. Even if the current position of the tracked object is not accurate, if samples in the neighborhood of the current position are in the positive bag, this bag is likely to contain at least one image with the object properly centered. The performance of the MIL tracker is high, it does not drift as much as the BOOSTING tracker, and reasonable performance can be obtained even when partial occlusion occurs.

[0052] KFC means the kernelized correlation filter. This tracker is based on the concepts proposed by the above two trackers. This tracker utilizes the fact that multiple positive samples used in the MIL tracker have large overlapping regions. Such overlapping data brings several excellent mathematical properties and at the same time makes the tracking faster and more accurate. It outperforms MIL in both accuracy and speed and is also excellent in reporting tracking failures.

[0053] TLD means Tracking, Learning, and Detection. As its name indicates, this tracker decomposes the long-term tracking task into three components: (short-term) tracking, learning, and detection. This tracker tracks objects frame by frame. The detector identifies all the appearances observed in the past and calibrates the tracker if necessary. Learning estimates the detector's errors and performs updates to avoid future error occurrences. The output of this tracker tends to be somewhat unstable. For example, when tracking a pedestrian and there are other pedestrians in the scene, this tracker may temporarily track a pedestrian other than the one it intends to track. Its advantages include functioning optimally even under occlusion over multiple frames. Its disadvantages include a high rate of false detections.

[0054] The MEDIANFLOW tracker tracks objects both forward and backward in time and measures the discrepancy between these two trajectories. By minimizing this Forward / Backward error, it can reliably detect tracking failures and enable the selection of reliable trajectories in the video. This tracker performs best when the motion is predictable and small and there is no occlusion. Unlike other trackers that continue even when tracking clearly fails, this tracker can recognize when tracking has failed.

[0055] The GOTURN tracker is an algorithm based on a convolutional neural network (CNN). This tracker is robust to viewpoint changes, lighting changes, and deformations, but may not handle occlusion well in some cases.

[0056] The MOSSE (Minimum Output Sum of Squared Error) tracker uses adaptive correlation for object tracking. The MOSSE tracker is robust to illumination, scale, pose changes, and non-rigid deformations. Also, this tracker detects occlusion based on the peak-to-side-lobe ratio and can resume tracking from where it was interrupted when the object reappears. The MOSSE tracker operates even at a high frame rate (above 450 fps). Advantages include that the implementation is very simple, it is as accurate as other complex trackers, and it is much faster.

[0057] The CSRT tracker uses a spatial reliability map for tracking. The CSRT tracker operates at a relatively low frame rate (25 fps) but provides higher accuracy in object tracking.

[0058] When at least one of the regions of interest R satisfies the conditions defined for each region of interest, the notification unit 113 notifies the user of this. Specifically, for example, when at least one of the regions of interest R satisfies the conditions defined for each region of interest, the notification unit 113 causes a message indicating this to be displayed on the display device 2, or makes a notification sound through a speaker of an image processing device 1 (not shown) to notify the user. Alternatively, when the user is wearing a device equipped with a vibration member such as a smartphone, the user may be notified by vibrating the vibration member. Note that the notification sound may vary in type, amplitude, and frequency according to the conditions. Also, the vibration may vary in period and amplitude according to the conditions.

[0059] In this way, the user of the image processing apparatus 1 can cause the image processing apparatus 1 to track the behavior of the set region of interest R by setting the region of interest R in the X-ray image in advance, so that the user can concentrate on the work at the site of attention. When displaying the region of interest, the notification unit 113 may color and display the region of interest in a color different from the color of the acquired image (X-ray image). Generally, since the images of angiography are monochrome, the difficulty of recognizing the devices included in the images is high. However, by coloring and showing the region of interest, it becomes possible to facilitate the recognition of the devices and provide more accurate information to the user. Further, when a plurality of regions of interest are set, the regions of interest may be colored using different colors.

[0060] [Specific examples of conditions set for the region of interest R] Subsequently, specific examples of the conditions set for the region of interest R will be described.

[0061] In the catheter surgery of blood vessels, the user first passes a guiding catheter into the blood vessel of the subject P up to the vicinity of the site of attention. Subsequently, the user passes other devices such as a guide wire, a delivery wire, and a balloon catheter through the guiding catheter to observe (diagnose) and treat the site of attention, for example, to perform coil embolization of a cerebral aneurysm.

[0062] A catheter is one of the devices and is an elongated tube with a lumen. This catheter is passed through a blood vessel and, for example, guided into an aneurysm, and a coil is inserted from the lumen of the catheter to place the coil in the aneurysm to prevent rupture.

[0063] There are several types of catheters. First, the guiding catheter is slightly thick with a diameter of about 2 - 4 mm and serves to connect from the puncture site to the area in front of the target site. A smaller device is inserted into this guiding catheter to be guided to the target site. The advantage of the guiding catheter is that various devices can be transported through the guiding catheter from the puncture sites in the groin or the blood vessels in the arm to the proximal part of the target site, eliminating the need to check the passage each time. Thus, for example, treatment can be carried out efficiently without moving the fixed screen.

[0064] Some guiding catheters have a balloon at the tip (guiding catheter with balloon), which can stop the blood flow in the blood vessel or stabilize the guiding catheter.

[0065] The intermediate catheter is placed inside the guiding catheter to send a thinner device to the target site and is indwelled in a more proximal part of the target site where the blood vessels are thinner. Multiple intermediate catheters may be used (gradually becoming thinner).

[0066] The microcatheter is the thinnest catheter, soft, and can be inserted into thin blood vessels. A coil or stent is placed inside this microcatheter and transported to the target site. Embolic substances may also be flowed from the microcatheter.

[0067] The balloon catheter has a balloon near the tip of the microcatheter, and the balloon is inflated near the target site. Thereby, for example, it can prevent the coil from coming out of the aneurysm, expand the stenotic site, or stop the blood flow and attempt to stop bleeding when there is a vascular perforation. When referring to the balloon, it usually refers only to the balloon part of the balloon catheter.

[0068] When guiding these catheters, an appropriately sized guide wire is often used inside. Rarely, it may also be guided by riding on the blood flow.

[0069] A guide wire is an elongated wire. Generally, a guide wire is used to guide a flexible catheter through a vascular bifurcation to a target site. In addition, there are delivery wires for carrying stents, devices with a balloon attached to the guide wire that can stop blood flow, etc. In this specification, when referring to a guide wire, it shall have a broad definition including these. The reason is that any of them may pierce a blood vessel with the tip of the guide wire, causing serious complications. One of the objectives of the present invention is to provide assistance for preventing blood vessel perforation by such a guide wire tip. Therefore, anything that includes a thin wire that may cause blood vessel perforation at the tip shall be referred to as a guide wire.

[0070] During the procedure, since the blood vessels of the subject P are inside the body of the subject P, the user cannot directly observe the various devices inserted into the blood vessels. Therefore, as described above, the user operates the device D while observing the blood vessel V and the device D (for example, guiding catheter, guide wire, coil, catheter, balloon, stent, etc.) imaged in the X-ray image generated from the X-rays transmitted through the body of the subject P. As shown in FIG. 1, the X-ray imaging device 3 used in catheter surgery is a device for generating an X-ray image of a specific region including a target site (for example, an aneurysm, a stenosis site, an infarction site, etc.) that requires treatment in the body of the subject P. During the procedure, in many cases, especially in the cerebrovascular region, the region obtained as the X-ray image is fixed. Therefore, the entire device D is not always necessarily imaged in the X-ray image that the user can observe.

[0071] Therefore, if the user focuses on the operation at the target site, for example, operations such as inserting a coil into an aneurysm or implanting a stent, the tip of the guide wire or the tip of the guiding catheter may deviate from the field of view of the X-ray image and disappear from the X-ray image, making it impossible for the user to observe. In such a case, there is a risk that the user may not notice even if the tip of the device has perforated the blood vessel. Vascular perforation is a life-threatening serious complication. Vascular perforation is likely to occur at the tip of the device, especially at the tip of the guide wire or catheter. Therefore, in the present invention, particular attention is paid to the tip. When there are multiple tips within one screen, or when vascular perforation may occur at the tip of the device at a location different from the target site of the operator. Also, in endovascular treatment of the cerebrovascular system, it is necessary to perform treatment while simultaneously viewing up to four multiple screens, and it is impossible for the operator to constantly pay attention to the multiple tips on the multiple screens. This support system compensates for this problem with technologies such as AI. Therefore, as an example of the conditions set for the region of interest R used in the present invention, there is a condition regarding the distance between the tip of the guiding catheter or the tip of the guide wire and the edge of the X-ray image. Alternatively, the condition may be that the region of interest R has exceeded a specific range (for example, a boundary line specified by the user using a pointing device such as a mouse) or has simply moved on the X-ray image. That is, in some aspects of the present invention, not limited to the edge of the image, the range of any region within the image may be treated in the same manner as the edge. The boundary line can be a straight line, a curve, a circle, a rectangle, or other polygons. In some aspects of the present invention, the distance between the region of interest and the edge of the specific range may be displayed. The display mode of the distance may be changed according to the magnitude of the distance between the region of interest and the edge of the specific range. In some aspects of the present invention, the notification unit can also notify the user on the condition that the value obtained by dividing the distance between the region of interest and the edge of the specific range by the moving speed of the region of interest within the image is less than a predetermined threshold value. The present invention also includes in some aspects a function to facilitate the operator's judgment by making the specific range easier to see, such as by superimposed display. The display method of the range is not limited to superimposed display, and arrows or the like may be used.In some aspects of the present invention, the distance can be determined by either the straight-line distance or the distance along the blood vessel.

[0072] Furthermore, instead of the region of interest, the position of a device for examination or treatment within the blood vessel (also referred to as the device of interest) at a certain point in time can be automatically or specified by the user (surgeon / assistant), and all or part of the device detected at that time can be superimposed and displayed on the subsequent real-time image. As a result, the user (surgeon / assistant) can recognize the movement (shift) of the device from a specific point in time by means of the superimposed display on the real-time image (Fig. 21). The shape detection of the device may be automatically obtained by image analysis or performed by the surgeon marking it with a mouse, touch panel, etc. Specific examples of the device to be superimposed and displayed include, but are not limited to, the tip of a guide wire, a marker, a stent, a guiding catheter, etc.

[0073] For example, assume that the position and shape of the tip of the guide wire (black part) in the left figure of Fig. 21 are memorized. In that case, for example, when the user designates the tip of the guide wire with a pointing device such as a touch panel or a mouse, the tip of the guide wire is recognized, and as shown in the middle figure of Fig. 21, the memorized tip of the guide wire is superimposed and displayed on the X-ray image obtained in real time thereafter. As a result, as shown in the right figure of Fig. 21, how much the tip of the guide wire has moved thereafter is visualized and can be easily grasped.

[0074] Therefore, in some aspects of the present invention, the image processing apparatus includes a storage unit that acquires and stores the position and / or shape of a device for examination or treatment within the blood vessel at an arbitrary point in time, and has a function of superimposing and displaying the stored position and / or shape of the device on the image after acquisition.

[0075] For example, in carotid artery stenting, it is important to prevent distal embolism (cerebral infarction) by ensuring that filters or balloons placed in the distal internal carotid artery do not move excessively or unstably. Alarms can be issued when the tips of these devices or the guidewires attached to them move outside a certain range specified by the operator or automatically. In cerebral aneurysm embolization, if the tip of a guidewire inserted into a balloon catheter moves outside the specified range, there is a risk of balloon slippage, inability to retract, or perforation of the distal blood vessel, so an alarm can be issued. Similarly, when inserting a coil, if it moves outside a certain area, it may occlude an important blood vessel, so an alarm can be issued. If the coil deviates from the mask image (area) of the aneurysm, an alarm can be issued because there is a possibility of perforating the aneurysm with the coil, guidewire, catheter, etc. Although it is important for the guiding catheter to be stable in all endovascular treatments, there may be times when correction is desired before it disappears outside the frame, and an alarm can be issued if it appears to move out of a certain area. In tumor embolization, cerebral arteriovenous malformation embolization, dural arteriovenous fistula embolization, etc., embolization is performed using embolizing substances such as liquids and particles. At this time, if the embolizing substance exceeds the specified area, there is a possibility of causing cerebral infarction at an important site, so an alarm can be issued. Therefore, in some aspects of the present invention, the device also includes liquid embolizing substances, particle embolizing substances, etc. In addition to the above examples, in endovascular surgery, care is taken to keep any device or embolizing substance within the specified area, and this can be supported.

[0076] Figures 4(a)-(b) are diagrams for explaining an example of the conditions set for the region of interest R. In the example shown in Figure 4(a), the region of interest R is set at the tip of the device D. Also, in the example shown in Figure 4(a), information W indicating the velocity, acceleration of the region of interest R, and the distance (number of pixels) between the region of interest R and the edge F of the X-ray image is superimposed and displayed on the X-ray image. Note that Figure 4(a) shows an example where the device D is a guidewire. In some aspects of the present invention, a specific region specified within the X-ray image may be regarded as the same as the edge F of the X-ray image.

[0077] When the region including the tip of the guide wire is set as the region of interest R, the notification unit 113 notifies the user on the condition that the region of interest R disappears from the X-ray image. In some embodiments, the X-ray image is an image showing a fixed region including the site of interest. Usually, during the procedure, the region displayed as the X-ray image is fixed. Further, when the region including the tip of the guide wire is set as the region of interest R, the notification unit 113 notifies the user on the condition that the distance between the region of interest R and the edge F of the X-ray image becomes less than a predetermined threshold distance.

[0078] Here, the "predetermined threshold distance" is a "reference distance for determining protrusion" provided for the notification unit 113 to determine whether the tip of the device is likely to deviate from the field of view of the X-ray image. The specific value of the predetermined threshold distance may be determined by experiment in consideration of the frequency of notification by the notification unit 113 and usability of the user, etc. For example, it is 5% of the number of pixels in either the vertical or horizontal direction of the X-ray image. Thereby, when the region of interest R, which is a region including the tip of the guiding catheter or the tip of the guide wire, approaches the edge F of the X-ray image, the user can receive a notification before and after the region of interest R deviates from the edge F.

[0079] Also, the magnitude of the movement of the device D in the blood vessel V can be a useful index for the user to predict the time until the tip of the device D reaches the edge F of the X-ray image. Specifically, the greater the speed and acceleration of the tip of D, the shorter the time until the tip of D reaches the edge F of the X-ray image. Therefore, when the region including the tip of the guiding catheter or the tip of the guide wire is set as the region of interest R, the notification unit 113 may notify the user on the condition that at least one of the moving speed and acceleration of the region of interest R within the X-ray image exceeds a predetermined threshold value. Thereby, when the moving speed and acceleration of the region of interest R are greater than the threshold value, the user's attention can be drawn before the region of interest R approaches the edge F of the X-ray image.

[0080] Furthermore, the distance between the tip of the device D in the blood vessel V and the edge F of the X-ray image is a useful indicator for the user to judge the probability of the tip of the device D reaching the edge F of the X-ray image. Therefore, as shown in the information W in FIG. 4, the notification unit 113 may cause the display device 2 that displays the X-ray image to display the distance between the region of interest R and the edge F of the X-ray image.

[0081] Here, the "distance between the region of interest R and the edge F of the X-ray image" may be the distance when the region of interest R moves to the edge F of the X-ray image along the blood vessel V into which the device D in which the region of interest R is set is inserted. This can be realized by the distance measuring unit 115 extracting the blood vessel V using a blood vessel recognition engine generated in advance using a known machine learning method or the like and measuring the distance from the region of interest R to the edge F along the blood vessel V. Alternatively, the distance measuring unit 115 may measure the above-mentioned distance based on the trajectory of the device D that has advanced in the blood vessel V. Specifically, when the user advances the guiding catheter in the blood vessel V, the tracking unit 112 tracks the tip of the guiding catheter and stores its trajectory in the storage unit 10. The distance measuring unit 115 may use the length of the trajectory included in the region of interest R among the trajectories stored in the storage unit 10 as the above-mentioned length.

[0082] By causing the notification unit 113 to display the distance between the tip of the device D and the edge F of the X-ray image on the display device 2, the user can objectively grasp at a glance how much the device D needs to move to reach the edge F of the X-ray image. Furthermore, the notification unit 113 may change the display mode of the distance according to the magnitude of the distance between the region of interest R and the edge F. For example, increasing the font size as the distance decreases, changing the color according to the distance (blue → yellow → red), enlarging the display of the region of interest according to the magnitude of the distance, or changing the color or size of the mark attached to the region of interest according to the magnitude of the distance. By devising the above-mentioned display mode, it is possible to make it easier for the user to notice the change in the distance.

[0083] In the example shown in FIG. 4(b), the region of interest R set at the tip of the device D is closer to the edge F compared to the example shown in FIG. 4(a). For this reason, the font of the information W indicating the speed, acceleration of the region of interest R, and the distance (number of pixels) between the region of interest R and the edge F of the X-ray image is larger compared to the example shown in FIG. 4(a).

[0084] Note that the "distance between the region of interest R and the edge F of the X-ray image" may be the shortest distance between the region of interest R and the edge F of the X-ray image, or may be the length measured along the moving direction of the region of interest R to the edge F of the X-ray image. In this case, since the blood vessel extraction process by the distance measurement unit 115 can be omitted, it is advantageous in terms of speeding up the process.

[0085] The notification unit 113 may notify the user on the condition that the value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R within the X-ray image is less than a predetermined threshold value. The value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R within the X-ray image can be said to be, so to speak, the predicted time until the region of interest R reaches the edge F of the X-ray image. For this reason, the value of the "predetermined threshold value" is the "margin time for pop-out determination" provided for the notification unit 113 to determine whether there is a high probability that the region of interest R will deviate from the field of view of the X-ray image. The specific value of the margin time may be determined by experiment in consideration of the notification frequency by the notification unit 113 and usability for the user, etc., but for example, it is 3 seconds.

[0086] FIG. 5 is a diagram showing an example of the message Ms notified by the notification unit 113. In the example shown in FIG. 5, the notification unit 113 superimposes and displays on the X-ray image a message indicating that the region of interest R will go out of the X-ray image and disappear after 3 seconds (so-called frame out). Further, as in the example shown in FIG. 5, when the region of interest R approaches the edge F of the X-ray image, the notification unit 113 may change the shape of the region of interest R (changed to a circular shape in FIG. 5), increase the size of the region of interest R, or change the color of the region of interest R. Thereby, the notification unit 113 can make it easier for the operator to recognize that the region of interest R is likely to frame out.

[0087] FIG. 6 is a diagram for explaining another example of the conditions set for the region of interest R. Specifically, FIG. 6 shows an example where the device D is a guiding catheter. Different from the case of the guide wire shown in FIGS. 4(a)-(b), the guiding catheter is different in that not only its tip but also the entire device D can disappear from the X-ray image. However, whether the device D is a guiding catheter or a guide wire, it is the same in that its tip portion disappears from the X-ray image. Therefore, also in the case of the guiding catheter, similar to the guide wire, the region of interest R is set at the tip portion.

[0088] In the above example, mainly from the viewpoint of preventing the frame out of the surgical instrument, the threshold value was determined based on the distance between the region of interest R and the edge F of the X-ray image as the "predetermined threshold distance". However, focusing on the moving distance of the region of interest R itself, it may be configured to notify the user when the moving distance of the region of interest R exceeds a predetermined threshold value. For example, when trying to deploy a stent in a state where the tip of the catheter is inside an aneurysm, it is necessary to pay attention so that the tip of the catheter does not come out of the aneurysm. At this time, since the surgeon is looking at the stent, it is impossible to visually follow the movement of the tip of the catheter. In such a case, by configuring to notify the user when the moving distance of the region of interest set at the tip of the catheter exceeds a predetermined moving distance, when the tip of the catheter is about to come out of the aneurysm, the user can immediately notice the abnormality. Note that the threshold value of the moving distance may be set based on the reached position after the region of interest (for example, the tip of the catheter) reaches a predetermined position (for example, inside the aneurysm).

[0089] (Induction of Embolization Coil) Subsequently, as another example of the conditions set for the region of interest R, the conditions regarding a guiding catheter for guiding a delivery wire of an embolization coil will be described.

[0090] As a catheter treatment for cerebral aneurysms, embolization, which involves filling the cerebral aneurysm with embolization coils, is known. This treatment aims to block the inflow of blood into the cerebral aneurysm by passing a guiding catheter up to the vicinity of the cerebral aneurysm, which is the site of interest, passing a delivery wire for the embolization coil through the guiding catheter, and detaching and filling the embolization coil into the cerebral aneurysm.

[0091] Figures 7(a)-(b) schematically show an example of an X-ray image in which a guiding catheter that guides a delivery wire for an embolization coil is imaged, and is a diagram for explaining another example of the conditions set for the region of interest R.

[0092] As described above, the X-ray imaging device 3 can irradiate the head of the subject P with X-rays at different incident angles. Figures 7(a) and 7(b) illustrate X-ray images taken by irradiating the head of the subject P with X-rays at different incident angles. The user performs the embolization while looking at the two images shown in Figures 7(a) and 7(b).

[0093] In Figure 7(b), the aneurysm is indicated by the reference numeral A. The user embolizes the aneurysm by placing a plurality of embolization coils E into the aneurysm A. Here, since the aneurysm A can be observed in the X-ray image shown in Figure 7(b), the user will focus on one of the two X-ray images.

[0094] The delivery wire for transporting the coil E for embolization is coupled to the coil E for embolization at its tip, and the user performs an operation to disconnect the coil E for embolization from the delivery wire after transporting the coil E for embolization to the aneurysm A. As shown in FIGS. 7(a)-(b), when the coil E for embolization reaches the aneurysm A, the position of the coil E for embolization during transportation becomes unclear in the X-ray image due to other coils E for embolization already placed in the aneurysm A. Therefore, the delivery wire of the coil E for embolization is provided in advance with a marker L for measuring the timing of disconnecting the coil E for embolization. The user determines the timing of disconnecting the coil E for embolization by using, as a reference, the marker L provided on the delivery wire rather than the coil E for embolization itself in the X-ray image.

[0095] However, as shown in FIGS. 7(a)-(b), it may also happen that the X-ray image capable of observing the aneurysm A is different from the X-ray image capable of observing the marker L. Also, even within the same screen, the positions of the aneurysm A and the marker L may be separated, and it may be difficult to observe two locations simultaneously. Therefore, the user of the image processing apparatus 1 first sets a region of interest R in a part of the guiding catheter that guides the delivery wire of the coil E for embolization. Specifically, the user sets the region of interest R at a position where the marker L and the region of interest R overlap at the timing of disconnecting the coil E for embolization from the delivery wire.

[0096] The region-of-interest acquisition unit 111 receives the region of interest R set in a part of the guiding catheter that guides the delivery wire. The designation of the region of interest R can be performed, for example, using a pointing device such as a mouse or a touch panel. In FIG. 7(a), the region of interest R is indicated by an open star. The marker detection unit 114 detects the marker L that approaches the region of interest R among the markers L provided on the delivery wire of the coil E for embolizing the aneurysm. Also, the tracking unit 112 tracks the marker L detected by the marker detection unit 114.

[0097] When the marker L and the region of interest R overlap, the notification unit 113 notifies the user of the timing to cut and separate the embolization coil E from the delivery wire based on the positional relationship between the marker L and the region of interest R.

[0098] Figures 8(a)-(d) are diagrams for explaining the timing of separating the embolization coil E. The device D shown by the dashed line in Figures 8(a)-(d) is the delivery wire. As shown in Figures 8(a)-(d), markers L are attached to a certain section on the wire. Note that Figures 8(a)-(d) show a general example of the marker L. In addition, it may be a one-dot chain line "―·―" or a long straight line, etc.

[0099] When the user moves the delivery wire, the marker L also moves in the X-ray image in conjunction with the movement of the delivery wire. On the other hand, although the guiding catheter for guiding the delivery wire moves slightly due to friction or the like in response to the movement of the delivery wire, the amount of movement is small compared to the movement of the delivery wire. Therefore, the user sets the region of interest R in advance at the position of the guiding catheter corresponding to the position where the marker L should be when the embolization coil E reaches the aneurysm A.

[0100] As shown in Figures 8(a)-(d), the marker L exists over a certain section on the wire. When the user moves the delivery wire, as shown in Figure 8(b), the tip of the marker L contacts the region of interest R. This indicates that the embolization coil E has approached the aneurysm A.

[0101] When the user further moves the delivery wire, as shown in Figure 8(c), the marker L and the region of interest R overlap. The notification unit 113 starts the operation of notifying the user of the timing to cut and separate the embolization coil E from the delivery wire when the marker L and the region of interest R overlap.

[0102] Specifically, as indicated by reference numeral W2 in FIG. 7(a), when the marker L and the region of interest R overlap, the notification unit 113 causes the display device 2 to display information W2 indicating the distance by which the marker L should move before the embolization coil E is cut from the delivery wire.

[0103] More specifically, as shown in FIG. 8(d), the user sets the region of interest R such that when the embolization coil E reaches the aneurysm A, the terminal end of the marker L just passes through the region of interest R. In this case, the information W2 that the notification unit 113 causes the display device 2 to display will show the distance until the terminal end of the marker L passes through the region of interest R.

[0104] When the marker L passes through the region of interest R, the notification unit 113 further notifies the user of this fact. As a result, even when the user is concentrating on an image in which the aneurysm A as shown in FIG. 7(b) is imaged, the user can be aware of the timing to cut the embolization coil E from the delivery wire.

[0105] (Shape of the device D) Subsequently, as yet another example of the conditions set for the region of interest R, conditions regarding the shape of the device D (for example, a guide wire) will be described.

[0106] When a feature amount indicating the shape of the device D included in the region of interest R satisfies a predetermined condition, the notification unit 113 notifies the user of the image processing apparatus 1 of this fact. Specifically, the notification unit 113 notifies based on the curvature of the device D included in the region of interest R or a feature amount indicating the "deflection" of the device D included in the region of interest.

[0107] When the user attempts to advance the device D with the tip of the device D caught on a blood vessel wall or the like, the tip portion of the device D is bent. Generally, when the tip of the device D is bent, elastic energy is accumulated in that portion. This elastic energy increases in amount as the tip portion of the device D bends more, that is, as the curvature of the device D increases (the radius of curvature decreases). When the amount of accumulated elastic energy increases, the catch at the tip may be released by the elastic force of the device D, and the tip portion may move at high speed. As a result, the tip portion of the device D may suddenly disappear from the X-ray image.

[0108] Therefore, the notification unit 113 notifies the user on the condition that the curvature of the device D included in the region of interest exceeds a predetermined threshold curvature or that the tip does not move while the curvature is changing. The fact that the tip of the device D is stationary or that the moving distance is lower than a certain threshold value may also be taken into account. Here, the "predetermined threshold curvature" is a "reference curvature for notification determination" provided for the notification unit 113 to determine whether there is a high probability that the tip of the device D moves at high speed. The specific value of the predetermined threshold curvature may be determined by experiment in consideration of the notification frequency by the notification unit 113, usability, the material and size of the device D, the elastic coefficient, and the like.

[0109] Figs. 9(a)-(c) are diagrams for explaining the conditions regarding the shape of the device D set in the region of interest R. Specifically, Figs. 9(a)-(b) are diagrams for explaining the notification based on the curvature of the device D. In Fig. 9(a), the region of interest R is indicated by a dashed circle. Without limitation, as an example, the region of interest R in Fig. 9(a) is a circle with a radius of 1 centimeter centered on the tip of the device D. Fig. 9(b) is a histogram showing the distribution of the curvature of the device D included in the region of interest R. Specifically, Fig. 9(b) divides the device D included in the region of interest R into a plurality of small regions, obtains the radius of curvature of the device D in each small region, and shows the distribution of the radius of curvature.

[0110] The notification unit 113 notifies the user on the condition that, in the histogram showing the curvature distribution of the device D, a feature amount calculated from the curvature distribution (for example, a statistic such as the average value, the mode value, or the median value of the curvature) exceeds a predetermined threshold curvature, or that the tip does not move even though the curvature is changing. Thereby, the image processing apparatus 1 can provide the user with an opportunity to notice that the elastic force is accumulated in the device D.

[0111] Further, the notification unit 113 may notify the user on the condition that a value obtained by subtracting the length of the center line of the blood vessel V included in the region of interest R from the length of the device D included in the region of interest R exceeds a predetermined threshold length (threshold value). FIG. 9(c) is a schematic diagram showing the relationship between the length of the device D included in the region of interest R and the length of the center line of the blood vessel V included in the region of interest R. Although the device D and the blood vessel are bent in the body of the subject P, for convenience of explanation, the device D and the blood vessel V are shown as straight lines in FIG. 9(c). Also, in FIG. 9(c), the center line of the blood vessel V is indicated by a dashed line.

[0112] The distance measurement unit 115 extracts the blood vessel V in the region of interest R using the blood vessel recognition engine, traces its center line, and obtains the length D1. Similarly, the distance measurement unit 115 extracts the device D using the device recognition engine generated using a known machine learning method or the like, and obtains the length D2 thereof.

[0113] Generally, when the user advances the device D into the blood vessel V, the device D advances while meandering along the wall of the blood vessel V. Therefore, the length D2 of the device D in the blood vessel V is longer than the length of the blood vessel V (the length of the center line of the blood vessel V) D1, which means that the device D is bent in the blood vessel V and elastic energy is accumulated in the device D. If the amount of this bending becomes large, the elastic energy may be released due to some trigger, and the device D may move greatly. As a result, the tip of the device D may disappear from the X-ray image.

[0114] The differential length B, which is the length obtained by subtracting the length D1 from the length D2 calculated by the distance measurement unit 115, can be an indicator indicating the amount of deflection of the device D in the blood vessel V. Therefore, the notification unit 113 displays the length B obtained by subtracting the length D1 from the length D2 calculated by the distance measurement unit 115, and when it becomes longer than a predetermined threshold length, notifies the user of this fact. Thereby, the image processing apparatus 1 can provide the user with an opportunity to notice that the elastic force is accumulated in the device D.

[0115] (Estimation of the catheter tip position using the 2nd marker) The catheter for aneurysm embolization is provided with a tip marker (1st marker) and a 2nd marker (usually at a position 3 cm from the tip). It is important for the surgeon to know where the tip of the catheter is in the aneurysm in order to perform the surgery safely. However, when a coil enters the aneurysm, the position of the tip marker becomes difficult to see and the safety decreases (see Fig. 17). For example, when the tip marker moves to the back of the aneurysm, the tip or the coil coming out of the tip may puncture the aneurysm, leading to a serious complication such as subarachnoid hemorrhage. Conversely, if the tip marker is about to come out of the aneurysm, the catheter or coil may deviate outside the aneurysm and must be inserted into the aneurysm again, and this operation is accompanied by the risk of perforating the aneurysm wall.

[0116] For example, in FIG. 17, the left figure shows a state where a microcatheter is inserted into an aneurysm. The size of the aneurysm can be, for example, 10 mm. The distance between the 1st marker and the 2nd marker attached to the microcatheter is constant (usually 30 mm). For example, assume that the positions of the 1st marker and the 2nd marker are recorded respectively in this state (memory positions). The middle figure in FIG. 17 shows a state where the position of the microcatheter has moved. When a coil enters the aneurysm, the position of the 1st marker becomes invisible or difficult to see. Therefore, the position of the 1st marker is estimated from the difference between the position of the 2nd marker at this time point and the previously memorized position of the 2nd marker. In this case, the 1st marker is predicted to be approximately at 0 mm from the aneurysm neck (predicted position A). Also, the right figure shows a state where the position of the microcatheter has moved. Similarly, the position of the 1st marker is estimated based on the movement distance of the 2nd marker. In this case, it is estimated to be at 8 mm from the aneurysm neck.

[0117] Some aspects of the present invention relate to a method for estimating the tip position of a catheter when the position is unknown because the tip of the catheter is inside a coil (aneurysm). More specifically, a step of memorizing the positional relationship between the tip of the catheter and the second marker (for example, being 3 cm apart), a step of memorizing the distance a between the neck line of the aneurysm and the first marker and the position of the second marker at that time t1, a step of calculating the moving distance b from the position of the second marker at time t2, and estimating the distance a−b from the aneurysm neck line of the tip of the catheter, and a step of notifying the user of the estimated distance. Here, the tip of the catheter, the second marker, and the neck line of the aneurysm can be automatically detected by computer-aided image recognition. Alternatively, it may be manually specified using a pointing device such as a mouse or a touch panel. The estimated distance a−b can be displayed on a display device. Further, the position of the tip of the catheter estimated based on the estimated distance a−b may be displayed on the display device. An arbitrary threshold value may be set, and notification may be given when it deviates therefrom, or not only the distance but also the speed and acceleration may be obtained and notification may be given based on the values (it is highly likely to be dangerous when moving rapidly or greatly). This moving distance is a straight-line distance or a distance along the curve of the catheter. Also, since the curved shape of the catheter may change, the length of the curve may be used. Further, due to the shape change of the catheter, the distances a, b, and a−b may also form a probability distribution. For example, if there are multiple timings for memorizing at first, a distribution of the distance a can be formed. Therefore, the location may be estimated using the mean, variance, etc., and predicted as the most likely single point, or may be displayed as a heat map or the like as a probability distribution. Thus, the estimated distance may be represented by a probability distribution, and the position of the estimated tip of the catheter may be displayed based on the probability distribution by coloring it like a heat map.

[0118] In addition to measuring the movement distance, the operator may specify the position of the second marker or the computer may recognize it, and it may be semi-transparently superimposed or indicated by an arrow or the like in the display device. The operator and the assistant visually recognize whether the tip marker has advanced or been pulled back by recognizing how much the second marker at the current time has deviated from this fixed display.

[0119] Some aspects of the present invention relate to a program for executing the above method on a computer. Further, some aspects of the present invention relate to an image processing apparatus for executing the above method and an operation method thereof. Such an image processing apparatus may include, for example, a positional relationship storage unit that stores the positional relationship between the tip of the catheter and the second marker (for example, 3 cm apart), a positional storage unit that stores the distance a between the neck line of the aneurysm and the first marker and the position of the second marker at that time point t1, a distance estimation unit that calculates the movement distance b from the position of the second marker at time point t2 and estimates the distance a−b from the aneurysm neck line at the tip of the catheter, and a notification unit that notifies the user of the estimated distance. Furthermore, some aspects of the present invention relate to a cerebral aneurysm coil embolization assistance system including the above-described image processing apparatus and an image pickup apparatus that picks up an X-ray image of a patient (or subject P) in a state where a guiding catheter and a delivery wire of an embolization coil are inserted into a blood vessel and transmits the X-ray image to the image processing apparatus.

[0120] <Processing flow for estimating the catheter tip position using the second marker> FIG. 22 is a flowchart for explaining the flow of the catheter tip position estimation process using the second marker executed by the image processing apparatus according to the embodiment. The process in this flowchart starts, for example, when the image processing apparatus 1 is activated or when it is determined by the user or the image processing apparatus 1 that the start of the process is necessary.

[0121] First, when the image acquisition unit 110, the region of interest acquisition unit 111, and the tracking unit 112 function, the device D for inspection or treatment in the blood vessel V is detected and tracked by video analysis. Then, the positions of the 1st marker and the 2nd marker at that time (T = t1) are stored automatically by image analysis or according to the designation of the user (surgeon) (S102).

[0122] Then, the line of the aneurysm neck is determined automatically or according to the designation of the user (surgeon) (S104). Then, the distance A between the line of the aneurysm neck and the 1st marker is calculated (S106). At this time, the linear or curved distance A between the line of the aneurysm neck and the 1st marker (usually inside the aneurysm) can be measured.

[0123] Next, the 2nd marker is continuously tracked to measure the moving distance b (S108). The 2nd marker is continuously tracked by image analysis, and the moving distance b (linear or curved distance) of the 2nd marker at T = t2 is measured. This moving distance may have a direction (plus / minus).

[0124] Then, based on the distances A and b, the distance of the 1st marker from the aneurysm neck is estimated (S110). For example, the distance of the 1st marker from the aneurysm neck is estimated as A - b. Alternatively, it may be estimated that the 1st marker is at a location that has moved b in the direction perpendicular to the aneurysm neck from the position of the 1st marker at T = t1.

[0125] Then, the estimated position of the 1st marker is displayed (S112). For example, the estimated position of the 1st marker at T = t2 or the distance from the aneurysm neck can be displayed in a superimposed manner or numerically. If the 1st marker is near the aneurysm neck and is likely to deviate from the aneurysm, or if it is likely to hit the wall of the aneurysm deep inside the aneurysm, the user (surgeon / assistant) may be additionally notified to that effect. This function can be temporarily stopped in cases where the analysis is difficult or at the request of the user (surgeon). After the end of this process, the process in this flowchart may be restarted as necessary.

[0126] (Replay function) During the operation, medical staff focus on a specific location. Therefore, when the system according to the present disclosure issues a notification, if they do not focus on the device that issued the warning or are looking at another screen, they will try to grasp the situation after the warning occurs. However, since the real-time video is updated moment by moment, it is often difficult to check the details around the time when the warning was issued. Also, depending on the nature of the warning, it is necessary to grasp details such as the details of the operation of the device itself, the difference between the time when the warning occurred and the current time, and the elapsed time since the warning occurred.

[0127] For example, regarding the tip of the guide wire, it is desirable to obtain information on the details of the operation of the tip portion. For example, when it moves rapidly and greatly, the risk of blood vessel perforation increases. In the case of the disappearance of the guiding catheter, it is desirable to obtain information on the position and elapsed time at the moment it went out of the image. In the case of the coil detector, it is desirable to obtain information on the deviation from the optimal point.

[0128] In some aspects of the present invention, the image processing apparatus has a function (i.e., replay function) that stores video and recognition information and can be viewed back by the operator when necessary by providing the necessary information. FIG. 12 shows the configuration of an image processing apparatus further including a video recording unit (integratable with 10) that stores the image obtained from the image acquisition unit 110 and a video extraction unit that extracts a part of the video before and after the notification generation time. FIG. 13 shows an example in which the replay video when the notification occurred is cut out centering on the region of interest and enlarged for display. FIG. 14 shows an example in which an (enlarged) replay playback window is displayed on the real-time display screen. In this example, the replay is enlarged and displayed in a part where the region of interest is removed as much as possible, so that it is possible to determine what happened at the part where the warning was issued within the limited range of one screen.

[0129] In some aspects of the present invention, the image processing apparatus further includes a video storage unit that stores the images (including videos) obtained from the image acquisition unit over time (continuously). Also, in some aspects of the present invention, the image processing apparatus further includes a video extraction unit that extracts videos of a certain period before and after the notification unit issues a notification from the video storage unit. The extraction period and playback speed of the video may be automatically determined based on at least one of the moving distance, moving speed, and acceleration of the region of interest when the notification occurs. Note that, at the time of extraction, the information obtained by the region of interest acquisition unit, tracking unit, notification unit, marker detection unit, and / or distance measurement unit can also be used.

[0130] In some aspects of the present invention, the image processing apparatus can display the extracted video on a display device. The extracted video may be automatically repeated a predetermined number of times. Also, the extracted video may be displayed based on any operation including play, stop, fast forward, rewind, frame advance, slow play, and double speed play. Thereby, the user can easily check the video. Also, on the display device, the elapsed time from the time when the notification occurred, the comparison of the position of the region of interest at the time when the notification occurred and after the elapse of an arbitrary time (the comparison display includes, for example, the corresponding region, the difference in the detection position, and the alignment of the image itself), or the trajectory of the region of interest acquired by the tracking unit may be further displayed superimposed on the extracted video.

[0131] In some aspects of the present invention, the image processing apparatus can cut out and display a partial region near the region of interest from the extracted video. The extracted video can be displayed at a position that does not obstruct the display of the region of interest. The extracted video may be displayed enlarged.

[0132] In some aspects of the present invention, the image processing apparatus can display the extracted video simultaneously with the occurrence of the notification or after the elapse of a predetermined time from the occurrence of the notification. Also, in some aspects of the present invention, the image processing apparatus can display videos taken from a plurality of directions simultaneously.

[0133] Note that the replay display described above may be used at times other than when a notification occurs. That is, the video extraction unit may extract videos of an arbitrary time or period from the video storage unit, in addition to videos of a certain period before and after the notification unit issues a notification. For example, when the user (operator) feels the need, by designating an arbitrary region of interest, the previous scene can be viewed in a replay display. Thereby, the user can grasp and compare what happened in the region of interest while viewing the real-time display.

[0134] Some aspects of the present invention relate to a program for executing the above method on a computer. Further, some aspects of the present invention relate to an image processing apparatus for executing the above method and an operation method thereof.

[0135] <Processing Flow of Replay Display Function> FIG. 23 is a flowchart for explaining the processing flow of the replay function executed by the image processing apparatus according to the embodiment. The processing in this flowchart starts, for example, when the image processing apparatus 1 is activated.

[0136] First, by the functions of the image acquisition unit 110, the region of interest acquisition unit 111, and the tracking unit 112, the device D for inspection or treatment in the blood vessel V is detected and tracked by video analysis (S202).

[0137] Next, it is determined whether a notification condition (such as the movement distance exceeding a threshold) is satisfied (S204). If the condition is satisfied, a notification is issued, and a replay video before and after satisfying the notification condition is displayed on a part of the display screen (S206). At this time, the real-time image may be displayed as usual, and the replay video may be repeatedly displayed about several times or until the user (operator / assistant) desires so as not to overlap the region of the notification condition (S208). When the repetition ends, the screen of the replay video is closed. After the end, the processing in this flowchart may be started again.

[0138] (Estimation of the Position of the Region of Interest outside the Frame) It is dangerous for the region of interest such as the tip of the guide wire or the tip of the guiding catheter to move outside the range of the X-ray imaging angle (frame out), but the degree of danger varies depending on the amount of movement. Specifically, for example, if the tip of the guide wire only protrudes slightly (within 5 mm or the like) outside the frame, the possibility of blood vessel perforation is low, but when it protrudes significantly outside the frame (20 mm or more or the like), the risk of blood vessel perforation is high. When the region of interest frame-outs, it is necessary to pull it back into the imaging angle of the X-ray image, but in some cases, it may not be possible to deal with it immediately depending on the situation. In such a case, it is important to know how far the frame-out region of interest has moved outside the range of the X-ray imaging angle and how dangerous the movement is. Therefore, some aspects of the present invention relate to an apparatus for estimating and displaying the position, velocity, and acceleration states of a frame-out region of interest.

[0139] The estimation of the position of such a region of interest outside the frame can be performed, for example, by an image processing apparatus including an image acquisition unit that acquires an image including at least a subject of an in-vessel inspection or treatment device, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, and a notification unit that notifies a user of the image processing apparatus when at least one of the regions of interest satisfies a condition defined for each region of interest. When a region including the tip of the catheter or the tip of the guide wire is set as the region of interest, the notification unit notifies the user on the condition that the region of interest disappears from the image, and based on the position, velocity, and / or acceleration of the tip of the catheter or the tip of the guide wire immediately before the region of interest disappears from the image, a state estimation unit that estimates the current position and / or velocity of the tip of the catheter or the tip of the guide wire that has disappeared from the image. It can be performed by an image processing apparatus (Fig. 15). The current position and / or velocity of the region of interest estimated by the state estimation unit can be displayed on a display device.

[0140] Here, in order to estimate the position of the region of interest that has timed out, the state estimation unit stores the outputs from the region of interest acquisition unit, the marker detection unit, and the tracking unit in chronological order before the timeout, and can calculate the state such as the position, velocity, and acceleration of the region of interest using the stored outputs. Then, when the region of interest times out and tracking on the screen becomes impossible, the position, velocity, etc. of the region of interest are estimated from the state before the timeout and notified to the user. The estimation methods at that time include learning-based methods using deep learning (such as CNN, RNN, WaveNet, etc.) and methods using Bayesian estimation (such as Kalman filter, extended Kalman filter, ensemble Kalman filter, particle filter, etc.), but are not limited thereto.

[0141] In addition, some aspects of the present invention also relate to a device that calculates a risk level from the estimated state and issues a notification to the user according to the risk level. The position, velocity, and risk level of the region of interest estimated by the state estimation unit can be displayed on a display device. The display method can be, for example, display by points, arrows, heat maps, etc., but is not limited thereto. For example, when the position of the estimated region of interest is separated from the edge of the image by a predetermined distance or more, it can be determined that the risk level is high (FIG. 16). For example, in FIG. 16, the position of the estimated region of interest is indicated by a circle, and it is determined that the risk level is higher as the position is estimated to be farther from the edge of the screen. At this time, for example, the risk level may be indicated by the color of the circle (for example, green indicates low risk level, and red indicates high risk level). When the calculated risk level exceeds a certain threshold, an alert can be displayed on the screen of the display device. Or in addition, a voice notification may be issued.

[0142] Therefore, some aspects of the present invention relate to an image processing apparatus, comprising: an image acquisition unit that acquires an image including at least a device for inspection or treatment within a blood vessel as a subject; a region of interest acquisition unit that acquires, as a region of interest, one or a plurality of regions including at least a part of the device included in the image; a tracking unit that tracks each of the regions of interest in the image; and a notification unit that notifies a user of the image processing apparatus when at least any one of the regions of interest satisfies a condition defined for each of the regions of interest, wherein when a region including the tip of a catheter or the tip of a guide wire is set as the region of interest, the notification unit notifies the user on the condition that the region of interest disappears from the image, and a state estimation unit that estimates the current position and / or velocity of the tip of the catheter or the tip of the guide wire that has disappeared from the image based on the position, velocity, and / or acceleration of the tip of the catheter or the tip of the guide wire immediately before the region of interest disappears from the image, and a warning is notified to the user when the current position and / or velocity of the region of interest estimated by the state estimation unit exceeds a predetermined threshold value, also relates to an image processing apparatus.

[0143] Some aspects of the present invention relate to a program for executing the above method on a computer. Further, some aspects of the present invention relate to an image processing apparatus for executing the above method and an operation method thereof.

[0144] <Processing flow for estimating the position of a region of interest outside the frame> FIG. 24 is a flowchart for explaining the flow of a process for estimating the position of a region of interest outside the frame executed by the image processing apparatus according to the embodiment. The process in this flowchart starts, for example, when the image processing apparatus 1 is activated.

[0145] First, by the functions of the image acquisition unit 110, the region of interest acquisition unit 111, and the tracking unit 112, the device D for inspection or treatment within the blood vessel V is detected and tracked by video analysis (S302).

[0146] Next, it is determined whether the part of the device D to be notified has gone outside the frame of the screen (S304). If the condition is satisfied, the position of the device outside the frame is estimated (S306). For example, the position of the device D outside the frame is estimated from the front side of the black part of the guide wire visible within the screen connected to the device D, the filter, the balloon, etc. Distance difference, machine learning, etc. can be used for the estimation.

[0147] Next, the estimated position of the device is displayed (S308). The estimated position of the device D that has gone outside the frame is displayed on the display screen with the actual estimated position or the distance from the frame, etc. The display and notification may be made stronger as it moves further away from the frame.

[0148] Finally, it is determined whether the estimation is no longer required because the device D has returned within the frame of the screen, or for reasons such as the user (operator / assistant) pressing a button indicating that display is not required (S310). After this ends, the processing in this flowchart may be restarted.

[0149] (Layout / Notification) In some aspects of the present invention, the results of image analysis can be displayed on two screens of different sizes. As described above, in endovascular surgery, generally, surgery is performed while looking at multiple screens (for example, four screens), and the surgeon grasps three-dimensional information by looking at at least two screens (generally the front (AP direction or F direction: Anterior-Posterior, Frontal) and the side (RL direction or LAT direction: Right to Left, Lateral)). Therefore, it is important to display the front and side images, and it is very important to display them clearly on a monitor with a physically limited size. In actual surgery, the monitor is often placed more than 1 m away across the patient's bed, and there are often instructions to move the monitor closer to the surgeon by even 1 cm. Note that the front and side move three-dimensionally, and for example, using the tube ball in the front, an angle of 15 degrees to the right and 10 degrees cephalad is always used to view obliquely. Thus, even when three dimensions are projected onto two dimensions, the angle is adjusted to view clearly in two dimensions. Further, in the case of four screens, there are Live and Mask for the front and Live and Mask for the side. Live is a normal fluoroscopic image similar to a general X-ray photograph, and it is viewed in real time. Mask takes the difference (subtraction) from an arbitrary past Live image selected by the surgeon. Thereby, the bones visible in Live disappear, and for example, only the blood vessels and devices imaged with the contrast agent are visible, and an image that is easy for the surgeon to understand is obtained.

[0150] The two screens of different sizes can be switched automatically or by the selection of the user (surgeon) (see FIG. 18). Since there are physical limitations to the screen size and there may be a desire to view one of them larger, as shown in FIG. 18, the visibility can be improved by displaying the two screens in different sizes.

[0151] Furthermore, the screen to be notified can also be made recognizable to the user by making the frame part of the screen glow, changing its color, or highlighting it (see Fig. 19). As shown in Fig. 19, by notifying with a colored frame or the like, it becomes easier for the user to understand which screen should be focused on. When a notification is issued on the smaller screen side, the two screens may be automatically switched (thereby, the part to be focused on is switched to the larger and more visible screen). Thus, in some aspects of the present invention, the display device may have a function of prompting the user's attention by making the frame part of one of the two screens glow, changing its color, or highlighting it.

[0152] (Display based on probability) In some aspects of the present invention, since the region of interest within the screen is output as a probability distribution, it is also possible to express the existence of the region of interest in terms of probability. The probability distribution of the existing region may be displayed by numerical values, colors, bars, etc. Furthermore, it is also possible to express whether the scene to be notified is a certain probability or not. Whether the scene to be notified is a certain probability or not may be displayed by numerical values, colors, bars, etc. corresponding to the probability. Also, which part within the screen is the responsible part may be displayed by a heat map or the like as a probability distribution. The probability distribution may be converted and displayed so as to be easy to understand. For example, it may be displayed as text such that 0 to 30% is low, 30 to 70% is middle, and 70 to 100% is high, or it may be displayed in three colors. As another example, the region less than 70% may be made slightly darker so that the region of interest or the region to be notified looks bright like a spotlight.

[0153] Therefore, in some aspects of the present invention, the notification unit can display on a display device that displays an image a numerical value, color, bar, or heat map according to a probability that at least any one of the regions of interest satisfies a condition defined for each region of interest, or a numerical value, color, bar, or heat map based on a value obtained by applying an arbitrary conversion to a probability distribution. Further, the notification unit can display on a display device that displays an image by coloring the region of interest with a color or heat map according to a probability that at least any one of the regions of interest satisfies a condition defined for each region of interest, or a color or heat map based on a value obtained by arbitrarily converting a probability distribution, or can display on a display device that displays an image by replacing the probability of satisfying the condition with a numerical value or color.

[0154] (Device Selection Display / Recording) There are various devices used in endovascular treatment, and there are many types of each. Examples of devices include various catheters, balloon catheters, guidewires, stents, flow diverting stents (stents with fine meshes), coils, embolization substances (substances such as liquids and particles), and other embolization devices (such as WEB). Also, there are various types of each device. For example, catheters have specifications such as tip shape, length, inner lumen, outer lumen, and hardness. Coils have specifications such as manufacturer, thickness, total length, diameter, and hardness, and there are hundreds of types. It is impossible to remember all of these, and since it is not known whether there is inventory, during the operation, the surgeon proceeds with the treatment while confirming them with the vendor. Combinations are also important, and the size of the coil, etc., is judged while looking at the image. Since usually about 5 to 15 coils are used for aneurysms, it is necessary to consider which one to use next, but it is difficult for the surgeon and assistant to remember inventory management and lineups, etc. Since new products come out and old products become unavailable, it is also difficult to grasp the situation, and the lineups available at each facility vary. Currently, during the treatment, the device is selected by communicating with the vendor, but it is not smooth.

[0155] First, database the lineup and specifications of the devices currently available in the country. It is also possible to input inventory information for each facility. Construct a system to display that information on a monitor. For example, when a microcatheter is selected, the lineup, specifications, and inventory information of the available microcatheters are listed and displayed. As additional information, it is also possible to search for information on whether that device or multiple devices can fit into a guiding catheter. For example, when the inner diameter of the guiding catheter is R and the outer diameters of two devices are r1 and r2, if R > r1 + r2, the two devices can fit into the guiding catheter. The same concept can be applied to the case of one device or three or more devices.

[0156] As another example, in the case of coils, select the coil lineup (length, diameter, hardness, shape, etc.) based on the size of the aneurysm and the behavior of the previously inserted coil. Although there are hundreds of types of coil lineups, listing them becomes easier by defining the length, etc. By displaying it on the monitor, it becomes easier to make a selection during treatment. Since most coil selections involve making the diameter and length the same or smaller, it is possible to present a lineup of coils that are likely to be used based on the information of the coils used so far. It is also possible to make a presentation considering inventory. In addition, the preferences according to the facility or the operator (user) can be included in the lineup candidates. It may also be proposed according to the aneurysm and the way the coil is wound by image analysis. The operator selects the desired device from such a list and performs the surgery. By recording this information and combining it with a snapshot of the treatment video, etc., a surgical record can be automatically created.

[0157] In some aspects of the present invention, a display device for displaying an image can display a product list of intravascular examination or treatment devices (e.g., various catheters and coils) (see FIG. 20). Further, the display device may display a product list narrowed down by size or inventory. Furthermore, the display device may display a list of recommended products based on the image analysis results, facility information, or user preference information.

[0158] In some aspects of the present invention, the image processing apparatus can automatically or based on the user's selection create a surgical record including information on the devices used, information on the acquired images, and the image analysis results.

[0159] Thus, some aspects of the present invention relate to an intravascular surgery support system, including a storage unit storing a product list of intravascular inspection or treatment devices (e.g., various catheters and coils), a recommendation unit recommending a product to be used based on the image analysis results, facility information, or user preference information, and a display unit displaying the recommended product.

[0160] (Exemplary System) Some aspects of the present invention relate to an intravascular catheter surgery support system, for example, for catheter surgery support in the brain, heart, peripheral arteries, and abdominal blood vessels, particularly cerebral blood vessels. Such a system includes an image processing apparatus and an image capturing apparatus that captures an X-ray image of a patient with one or more devices inserted into the blood vessel and transmits the image to the image processing apparatus. The image processing apparatus includes an image acquisition unit that acquires over time an X-ray image of a region (e.g., a fixed region) including at least a region of interest for achieving the purpose of the surgery and a device inserted into the blood vessel, a region of interest acquisition unit that acquires as a region of interest one or more regions including at least a part of the device included in the image, a tracking unit that tracks each of the regions of interest in the image, and a notification unit that notifies the user of the image processing apparatus when at least any one of the regions of interest satisfies a condition determined for each region of interest. The one or more devices are a catheter, a guide wire, a stent, and / or a balloon. When a region including the tip of the catheter or the tip of the guide wire, both ends of the stent, and both ends of the balloon is set as the region of interest, the system can notify the user on the condition that the region of interest disappears from the image, or the distance between the region of interest and the edge of the image becomes less than a predetermined threshold distance, or the region of interest is displaced by a certain distance.

[0161] Here, the notification unit may cause a display device for displaying an image to display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest. The notification unit can change the display mode of the distance on the display device according to the magnitude of the distance. The change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a graphic on the entire screen, outside the frame, or a part of the display device, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of the mark attached to the region of interest according to the magnitude of the distance. Further, the notification unit may emit a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0162] In addition, some aspects of the present invention relate to an aneurysm coil embolization assistance system, particularly a cerebral aneurysm coil embolization assistance system. Such a system includes an image processing device and an image capturing device that captures an X-ray image of a patient in a state where a guiding catheter and a delivery wire for an embolization coil are inserted into a blood vessel and transmits the image to the image processing device. The image processing device includes an image acquisition unit that acquires an X-ray image of a fixed region including at least an aneurysm formed in the patient's blood vessel, a catheter inserted into the blood vessel, and a delivery wire for an embolization coil over time, a region of interest acquisition unit that acquires one or more regions including at least a part of the guiding catheter included in the image as a region of interest, a marker detection unit that detects a marker provided on the delivery wire and approaching one or more regions of interest set in a part of the catheter that guides the delivery wire, a tracking unit that tracks each of the region of interest and the marker in the image, and a notification unit that notifies a user of a timing at which an embolization coil may be cut from the delivery wire when the marker and the region of interest overlap.

[0163] Here, the notification unit may cause a display device for displaying an image to display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest, and the notification unit can change the display mode of the distance on the display device according to the magnitude of the distance. The change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a graphic on the entire screen, outside the frame, or a part of the screen of the display device, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of the mark attached to the region of interest according to the magnitude of the distance. Further, the notification unit may emit a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0164] <Processing flow of the image processing method executed by the image processing apparatus 1> FIG. 10 is a flowchart for explaining the flow of the image analysis process executed by the image processing apparatus 1 according to the embodiment. The process in this flowchart starts, for example, when the image processing apparatus 1 is activated.

[0165] The image acquisition unit 110 acquires an X-ray image created based on the X-ray absorption rate, including at least the subject of the blood vessel V and the device D for inspection or treatment within the blood vessel V (S2). The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device D included in the X-ray image as the region of interest R (S4).

[0166] The tracking unit 112 tracks each region of interest R in the X-ray image (S6). When at least one of the regions of interest R satisfies the condition determined for each region of interest R (Yes in S8), the notification unit 113 notifies the user of the image processing apparatus 1 of this (S10). When all the regions of interest R do not satisfy the determined conditions (No in S8), the notification unit 113 skips the notification process.

[0167] Until the image processing is completed (No in S12), the image processing apparatus 1 returns to the process of step S6 and repeats the processes from step S6 to step S10. When the image processing is completed (Yes in S12), the processing in this flowchart ends.

[0168] <Diagnosis and comparison of angiography> In angiography examinations and treatments, blood vessels are imaged by contrast agents to diagnose lesions. However, there may be omissions, and it is necessary to compare with images taken on the same day or on different days, which can be time-consuming and difficult. Also, since it is projected two-dimensionally, it may be difficult to make a judgment. Therefore, some aspects of the present invention relate to an image diagnostic apparatus that points out lesions or sites including, but not limited to, cerebral aneurysms, stenosis, occlusion, thrombosis, vascular perforation (contrast agent leakage), shunt diseases, feeding vessels of tumor vessels and tumor hyperenhancement, venous thrombosis, avascular areas in the capillary phase (findings of vascular occlusion), collateral circulation, etc. in angiography using contrast agents by using deep learning or the like.Therefore, in some aspects of the present invention, the image processing apparatus can further include a lesion recognition unit that recognizes lesions selected from the group consisting of aneurysms, stenoses, vasospasms, dissections, occlusions, recanalizations, thromboses, the sites and both ends of thrombi, vascular perforations, extravasations of contrast agents outside the blood vessels, vascular calcifications, arteriosclerosis, shunt diseases and their nutrient vessels and outflow vessels, retrograde flow of blood (contrast agent), arteriovenous malformations, dural arteriovenous fistulas, avascular regions, bone markings (internal auditory canal, fundus of the eye, supraorbital margin, pyramidal part, foramen magnum, cervical vertebrae, clavicle, rib and vertebra numbers, femoral head, pelvis), nutrient vessels of tumor vessels and tumor enhancement, venous occlusions, cavernous sinus thrombosis, avascular regions in the capillary phase, vascular occlusions, the shape and distribution of coils within aneurysms, the position, inflation, and shape of balloons, the deviation of coils into normal blood vessels, insufficient expansion of stents, the degree of adhesion to blood vessels and torsion, the movement of stents, the positions of both ends of stents, the positional relationship between the puncture site and the blood vessel (no stenosis, not near a bifurcation), tortuosity of blood vessels, the type of aortic arch (how far the right brachiocephalic artery is below the top of the aortic arch), the penetration range of liquid embolic substances, delays and stagnations in the flow of blood (contrast agent), variations in blood vessels (the presence or absence and degree of development of the anterior communicating artery, anterior cerebral artery A1, posterior communicating artery, posterior cerebral artery P1, posterior inferior cerebellar artery, anterior inferior cerebellar artery, superior cerebellar artery, superficial temporal artery, each cavernous sinus, each vein), moyamoya vessels (stenosis and occlusion at the tip of the internal carotid artery and the development of collateral blood vessels beyond it), the positions of arterial bifurcation parts and segments (the pyramidal part of the internal carotid artery, the cavernous sinus part, the ophthalmic artery part, the bifurcation part of the middle cerebral artery M1), past surgical scars (clips, coils, plates, shunt tubes and valves, ventricular tubes, cranial cavity tubes), the position and opening degree of WEB devices, foreign bodies (dental prostheses, plates), and collateral blood vessels. Without making a diagnosis, it may notify when abnormal findings are suspected or point out the area where there may be abnormalities. In that case, ultimately, the doctor can make a judgment. Similarly, some aspects of the present invention relate to an image diagnostic apparatus that points out changes when performing angiography compared to a previous angiography or compared to an angiography performed on another day. Therefore, in some aspects of the present invention, the image processing apparatus can further include an image recognition unit that notifies changes by comparing the angiographic image in the image with the previously acquired and stored angiographic image.For example, it is possible to point out changes in the degree of vasospasm, changes in thrombosis (appearance, disappearance, enlargement, shrinkage, etc.), release of occluded blood vessels, occlusion of blood vessels, deviation of coils, movement of stents, and the like.

[0169] <Effects achieved by the image processing apparatus 1 according to the embodiment> As described above, according to the image processing apparatus 1 according to the embodiment, in catheter examination or treatment of blood vessels, it is possible to provide a technique for concentrating a user who is a medical worker on the work in the region of interest and supporting the determination of the region of interest.

[0170] As described above, the present invention has been described using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, all or part of the device can be configured by being functionally or physically dispersed and integrated in an arbitrary unit. Also, new embodiments resulting from an arbitrary combination of a plurality of embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined.

[0171] <First modification example> In the above, the examination or treatment of cerebral blood vessels has been mainly described. However, the application target of the present invention is not limited to cerebral blood vessels, and it can be applied to examinations and treatments within blood vessels including the circulatory system regions such as the heart, peripheral vessels, and abdomen.

[0172] <Second modification example> In the above, the apparatus with two screens as shown in FIGS. 7(a)-(b) has been described as an example. However, the number of screens is not limited to this, and for example, it may be one screen or three or more screens.

[0173] <Third modification example> In the above description, the case where the X-ray imaging apparatus 3 captures an image of the surgical site of the subject P has been described. However, the imaging apparatus for capturing an image of the surgical site of the subject P is not limited to the X-ray imaging apparatus 3. In addition, for example, modalities such as MRI (Magnetic Resonance Imaging) or an ultrasonic imaging apparatus may be used to capture an image of the surgical site.

[0174] <Exemplification of Embodiment> Some or all of the above embodiments may also be described as follows in the following supplementary notes, but the disclosure of the present application is not limited to the following supplementary notes.

[0175] (Supplementary Note 1) An image processing apparatus, an image acquisition unit that acquires an image including at least a subject and a device for intravascular examination or treatment, an interested region acquisition unit that acquires one or more regions including at least a part of the device included in the image as interested regions, a tracking unit that tracks each of the interested regions in the image, a notification unit that notifies a user of the image processing apparatus when at least one of the interested regions satisfies a condition determined for each of the interested regions; An image processing apparatus comprising the above. (Supplementary Note 2) When a region including the tip of a catheter or the tip of a guide wire is set as the interested region, the notification unit notifies the user on the condition that the interested region disappears from the image. The image processing apparatus according to Supplementary Note 1. (Supplementary Note 3) When a region including the tip of a catheter or the tip of a guide wire is set as the interested region, the notification unit notifies the user on the condition that the distance between the interested region and the edge of the image becomes less than a predetermined threshold distance. The image processing apparatus according to Supplementary Note 1. (Supplementary Note 4) The notification unit notifies the user on the condition that at least any one of the moving distance, moving speed, and acceleration of the region of interest in the image exceeds a preset threshold value. The image processing apparatus according to any one of Appendices 1 to 3. (Appendix 5) The notification unit causes the distance between the region of interest and the edge of the image to be displayed on a display device that displays the image. The image processing apparatus according to any one of Appendices 1 to 4. (Appendix 6) The notification unit changes the display mode of the distance on the display device according to the magnitude of the distance between the region of interest and the edge of the image. The image processing apparatus according to Appendix 5. (Appendix 7) The notification unit notifies the user on the condition that a value obtained by dividing the distance between the region of interest and the edge of the image by the moving speed of the region of interest in the image is less than a preset threshold value. The image processing apparatus according to any one of Appendices 1 to 6. (Appendix 8) A marker provided on a delivery wire of a coil for a plug, further comprising a marker detection unit that detects a marker approaching a region of interest set in a part of a microcatheter that guides the delivery wire. The tracking unit further tracks the detected marker. The notification unit notifies the user of the timing at which the coil for the plug may be cut from the delivery wire when the marker and the region of interest overlap. The image processing apparatus according to any one of Appendices 1 to 7. (Appendix 9) When the marker passes through the region of interest, the notification unit notifies the user of that fact. The image processing apparatus according to Appendix 8. (Appendix 10) The notification unit causes the display device to display the distance that the marker should move before the coil for the plug is cut from the delivery wire. The image processing apparatus according to Supplementary Note 8 or 9. (Supplementary Note 11) When the feature amount indicating the shape of the device included in the region of interest satisfies a predetermined condition, the notification unit notifies the user of the image processing apparatus of this fact. The image processing apparatus according to any one of Supplementary Notes 1 to 10. (Supplementary Note 12) The feature amount is curvature. The notification unit notifies the user on the condition that the curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or that the tip does not move while the curvature is changing. The image processing apparatus according to Supplementary Note 11. (Supplementary Note 13) The notification unit notifies the user on the condition that the value obtained by subtracting the length of the center line of the blood vessel included in the image or the region of interest from the length of the device included in the image or the region of interest exceeds a predetermined threshold length. The image processing apparatus according to Supplementary Note 11 or 12. (Supplementary Note 14) The notification unit notifies the user by coloring the region of interest in a color different from that of the image, changing the font, size, or color of the displayed characters, changing the color of the entire screen or a part of the display device, displaying a figure on the entire screen, outside the frame, or a part of the display device, enlarging the display of the region of interest, or changing the color or size of the mark attached to the region of interest. The image processing apparatus according to any one of Supplementary Notes 1 to 13.

[0176] (Supplementary Note 15) When the region including the tip of the catheter or the tip of the guide wire is set as the region of interest, the notification unit notifies the user on the condition that the region of interest has moved or has exceeded a specific range specified on the image. The image processing apparatus according to any one of Supplementary Notes 1 to 14. (Supplementary Note 16) The image processing apparatus according to any one of Appendices 1 to 15, further including a video storage unit that stores the image or video obtained from the image acquisition unit over time. (Appendix 17) The image processing apparatus according to Appendix 16, further including a video extraction unit that extracts video of a certain period before and after the notification unit issues a notification or video of an arbitrary time or period specified by the user from the video storage unit. (Appendix 18) The image processing apparatus according to Appendix 17, characterized in that the extraction period of the video is automatically determined based on at least any one of the moving distance, moving speed, and acceleration of the region of interest when the notification occurs. (Appendix 19) The image processing apparatus according to Appendix 17 or 18, which causes the display device to display the extracted video. (Appendix 20) The image processing apparatus according to Appendix 19, characterized in that the extracted video is automatically repeatedly displayed a predetermined number of times. (Appendix 21) The image processing apparatus according to Appendix 20, characterized in that the extracted video is displayed based on any operation including playback, stop, fast forward, rewind, frame advance, slow playback, and double-speed playback. (Appendix 22) The image processing apparatus according to any one of Appendices 17 to 21, characterized in that the elapsed time from the time when the notification occurs, the comparison of the position of the region of interest after the elapsed time between the time when the notification occurs and an arbitrary time, or the trajectory of the region of interest acquired by the tracking unit is superimposed on and further displayed in the extracted video. (Appendix 23) The image processing apparatus according to any one of Appendices 17 to 22, characterized in that the extracted video is displayed by cutting out a partial region near the region of interest. (Appendix 24) The image processing apparatus according to any one of Appendices 17 to 23, characterized in that the extracted video is displayed at a position that does not obstruct the display of the region of interest. (Appendix 25) The image processing apparatus according to any one of Appendices 17 to 24, characterized in that the extracted video is displayed in an enlarged manner. (Appendix 26) The image processing apparatus according to any one of Appendices 17 to 25, wherein the extracted video is displayed simultaneously with the occurrence of the notification or after a lapse of a predetermined time from the occurrence of the notification. (Appendix 27) The image processing apparatus according to any one of Appendices 17 to 26, wherein videos taken from a plurality of directions are displayed simultaneously. (Appendix 28) The image processing apparatus according to any one of Appendices 2 to 27, further comprising a state estimation unit that estimates the current position and / or velocity of the tip of the catheter or the tip of the guide wire that has disappeared from the image based on the position, velocity, and / or acceleration of the tip of the catheter or the tip of the guide wire immediately before the region of interest disappears from the image. (Appendix 29) The image processing apparatus according to Appendix 28, wherein a warning is notified to the user when the current position and / or velocity of the region of interest estimated by the state estimation unit exceeds a predetermined threshold. (Appendix 30) The image processing apparatus according to any one of Appendices 1 to 29, wherein the display device that displays the image displays two images on two screens with different sizes. (Appendix 31) The image processing apparatus according to Appendix 30, wherein the display device prompts the user's attention by lighting, changing the color, or highlighting the frame portion of one of the two screens. (Appendix 32) The image processing apparatus according to any one of Appendices 1 to 31, wherein the display device that displays the image displays a list of products of devices for intravascular examination or treatment. (Appendix 33) The image processing apparatus according to Appendix 32, wherein the display device displays a list of products narrowed down by size or inventory. (Appendix 34) The image processing apparatus according to Appendix 32 or 33, wherein the display device displays a list of recommended products based on the image analysis result, facility information, or user preference information. (Appendix 35) The image processing apparatus according to any one of Appendices 1 to 34, which automatically or based on a user's selection creates a surgical record including information on a used device, information on an acquired image, and an image analysis result. (Appendix 36) The notification unit displays, on a display device that displays the image, a numerical value, color, bar, or heat map according to a probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a numerical value, color, bar, or heat map based on a value obtained by arbitrarily transforming a probability distribution, the image processing apparatus according to any one of Appendices 1 to 35. (Appendix 37) The notification unit colors the region of interest with a color or heat map according to a color or heat map based on a probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a value obtained by arbitrarily transforming a probability distribution, and displays it on a display device that displays the image, or replaces the probability of satisfying the condition with a numerical value or color and displays it on a display device that displays the image, the image processing apparatus according to any one of Appendices 1 to 35. (Appendix 38) When a region including the tip of a catheter or the tip of a guide wire is set as the region of interest, the notification unit notifies the user on the condition that the region of interest has moved or the region of interest has exceeded a specific range specified on the image, the image processing apparatus according to any one of Appendices 1 to 37. (Appendix 39) The boundary line of the specific range is represented by a straight line, curve, circle, rectangle, or other polygon, the image processing apparatus according to Appendix 38. (Appendix 40) The specific range is superimposed and displayed on an X-ray image, the image processing apparatus according to Appendix 38 or 39. (Appendix 41) The notification unit causes a distance between the region of interest and an edge of the specific range to be displayed on a display device that displays the image, the image processing apparatus according to any one of Appendices 38 to 40. (Appendix 42) The image processing apparatus according to supplementary note 41, wherein the notification unit changes a display mode of the distance on the display device according to a magnitude of a distance between the region of interest and an edge of the specific range. (Supplementary note 43) The image processing apparatus according to any one of supplementary notes 38 to 42, wherein the notification unit notifies the user on condition that a value obtained by dividing a distance between the region of interest and an edge of the specific range by a moving speed of the region of interest in the image is less than a predetermined threshold value. (Supplementary note 44) The image processing apparatus according to any one of supplementary notes 3 to 43, wherein the distance is determined by either a straight-line distance or a distance along a blood vessel. (Supplementary note 45) The image processing apparatus according to any one of supplementary notes 1 to 44, further including a storage unit that acquires and stores a position and / or a shape of a device for inspection or treatment within a blood vessel at an arbitrary point in time, and the stored position and / or shape of the device is superimposed and displayed on an image after acquisition. (Supplementary note 46) The image processing apparatus according to any one of supplementary notes 1 to 42, further including a lesion recognition unit that recognizes a lesion selected from the group consisting of an intracranial aneurysm, stenosis, occlusion, thrombosis, vascular perforation, extravascular leakage of a contrast agent, shunt disease, feeding vessels and tumor enhancement of tumor vessels, venous thrombosis, avascular regions in the capillary phase, vascular occlusion, and collateral circulation in the image. (Supplementary note 47) The image processing apparatus according to any one of supplementary notes 1 to 43, further including an image recognition unit that compares a vascular image in the image with a previously acquired and stored vascular image and notifies of a change.

[0177] (Supplementary note 48) A processor of an image processing apparatus acquires an image including at least a subject of a device for inspection or treatment within a blood vessel; acquires, as a region of interest, one or a plurality of regions including at least a part of the device included in the image; tracks each of the regions of interest in the image; When at least one of the regions of interest satisfies the conditions defined for each region of interest, notifying the user of the image processing apparatus of this fact; An image processing method for executing.

[0178] (Appendix 49) Causing a computer to Have a function of acquiring an image including at least a subject with a device for inspection or treatment inside a blood vessel; Have a function of acquiring, as regions of interest, one or more regions including at least a part of the device included in the X-ray image; Have a function of tracking each of the regions of interest in the X-ray image; Have a function of notifying the user of the computer when at least one of the regions of interest satisfies the conditions defined for each region of interest; A program for realizing.

[0179] (Appendix 50) An image processing apparatus according to any one of Appendices 1 to 44; An imaging apparatus that images a person with a device for inspection or treatment inside a blood vessel inserted therein and transmits the image to the image processing apparatus; An image processing system comprising.

[0180] (Appendix 51) A cerebrovascular catheter surgery support system, comprising: An image processing apparatus; An image imaging apparatus that images an X-ray image of a patient with one or more devices inserted into a blood vessel and transmits the image to the image processing apparatus; Comprising; The image processing apparatus includes: An image acquisition unit that acquires, over time, an X-ray image of a fixed region including at least a part to be noted for achieving the purpose of the surgery and a device inserted into a blood vessel; An interest region acquisition unit that acquires, as interest regions, one or more regions including at least a part of the device included in the image; A tracking unit that tracks each of the regions of interest in the image; A notification unit that notifies the user of the image processing apparatus when at least one of the regions of interest satisfies a condition defined for each region of interest; Comprising; The one or more devices are a catheter, a guide wire, a stent, and / or a balloon, When a region including the tip of the catheter or the tip of the guide wire, both ends of the stent, and both ends of the balloon is set as the region of interest, the region of interest disappears from the image, or the distance between the region of interest and the edge of the image is less than a predetermined threshold distance. A system that notifies the user on the condition that System. (Appendix 52) A cerebral aneurysm coil embolization assistance system, An image processing apparatus, An image imaging apparatus that images an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the image to the image processing apparatus; Comprising; The image processing apparatus, An image acquisition unit that acquires an X-ray image of a fixed region including at least an aneurysm formed in a patient's blood vessel, a catheter inserted into the blood vessel, and a delivery wire for an embolization coil over time; An area of interest acquisition unit that acquires, as an area of interest, one or more areas including at least a part of the guiding catheter included in the image; A marker detection unit that detects a marker provided on the delivery wire and approaching one or more areas of interest set in a part of the catheter that guides the delivery wire; A tracking unit that tracks each of the area of interest and the marker in the image; A notification unit that notifies the user of the timing at which the embolization coil may be cut from the delivery wire when the marker and the area of interest overlap; A system comprising. (Appendix 53) The notification unit causes a display device that displays the image to display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest. Here, the notification unit can change the display mode of the distance on the display device according to the magnitude of the distance. The change in the display mode includes changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a figure on the entire screen, outside the frame, or a part of the display device, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of the mark attached to the region of interest according to the magnitude of the distance. The system according to Supplementary Note 51 or 52. (Supplementary Note 54) The system according to any one of Supplementary Notes 51 to 53, wherein the notification unit can emit a notification sound or transmit a vibration according to the magnitude of the distance. (Supplementary Note 55) The system according to any one of Supplementary Notes 51 to 54, wherein the distance is determined by either a straight-line distance or a distance along a blood vessel. (Supplementary Note 56) A cerebral aneurysm coil embolization assistance system, an image processing device, an imaging device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the image to the image processing device and comprising: wherein the image processing device comprises a positional relationship storage unit that stores the positional relationship between the tip of the catheter and the 2nd marker, a position storage unit that stores the distance a between the neck line of the aneurysm and the 1st marker and the position of the 2nd marker at that time t1, a distance estimation unit that calculates the movement distance b from the position of the 2nd marker at time t2 and estimates the distance a - b from the tip of the catheter to the aneurysm neck line, and a notification unit that notifies the user of the estimated distance. A system. (Appendix 57) The system according to Appendix 546, wherein the estimated distance is represented by a probability distribution. (Appendix 58) The system according to Appendix 56 or 57, wherein the position of the catheter tip to be estimated is colored and displayed based on a probability distribution.

[0181] (Appendix 59) An intravascular surgery support system, comprising: a storage unit that stores a product list of inspection or treatment devices in the blood vessel; a recommendation unit that recommends a product to be used based on image analysis results, facility information, or user preference information; and a display unit that displays the recommended product.

Explanation of Signs

[0182] 1 ··· Image processing device 10 ··· Storage unit 11 ··· Control unit 110 ··· Image acquisition unit 111 ··· Region of interest acquisition unit 112 ··· Tracking unit 113 ··· Notification unit 114 ··· Marker detection unit 115 ··· Distance measurement unit 2 ··· Display device 3 ··· X-ray imaging device 30 ··· X-ray irradiator 31 ··· X-ray detector 32 ··· Hospital bed D ··· Device E ··· Embolization coil P ··· Subject S ··· Image processing system

Claims

1. An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; An image processing device comprising:

2. the notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, on condition that the region of interest disappears from the image.

2. The image processing device according to claim 1.

3. The notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, on condition that a distance between the region of interest and an edge of the image is less than a predetermined threshold distance.

2. The image processing device according to claim 1.

4. The notification unit notifies the user on condition that at least one of a moving distance, a moving speed, and an acceleration of the region of interest in the image exceeds a predetermined threshold.

4. The image processing device according to claim 1 .

5. The notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image.

5. The image processing device according to claim 1.

6. The notification unit changes a display manner of the distance on the display device depending on a distance between the region of interest and an edge of the image. The image processing device according to claim 5.

7. The notification unit notifies the user on condition that a value obtained by dividing a distance between the region of interest and an edge of the image by a moving speed of the region of interest within the image is less than a predetermined threshold.

7. The image processing device according to claim 1.

8. The method further includes a marker detector that detects a marker provided on a delivery wire of the embolization coil approaching a region of interest set on a part of a microcatheter that guides the delivery wire, The tracking unit further tracks the detected marker; and the notification unit notifies the user of a timing at which the embolization coil may be disconnected from the delivery wire, when the marker and the region of interest overlap.

8. The image processing device according to claim 1 .

9. The notification unit notifies the user when the marker passes through the region of interest. The image processing device according to claim 8.

10. the notification unit causes a display device to display a distance that the marker should move until the embolization coil is disconnected from the delivery wire.

10. The image processing device according to claim 8 or 9.

11. and when a feature amount indicating a shape of the device included in the region of interest satisfies a predetermined condition, the notification unit notifies a user of the image processing device of that fact.

11. The image processing device according to claim 1.

12. the feature value is a curvature, The notification unit notifies the user when a curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or when the tip does not move despite a change in curvature. The image processing device according to claim 11.

13. The notification unit notifies the user on condition that a value obtained by subtracting a length of a center line of the blood vessel included in the image or the region of interest from a length of the device included in the image or the region of interest exceeds a predetermined threshold length.

13. The image processing device according to claim 11 or 12.

14. 14. The image processing device of claim 1, wherein the notification unit notifies the user by displaying the region of interest in a color different from that of the image, changing the font, size or color of the displayed characters, changing the color of the entire screen of the display device or a portion of it, displaying a graphic on the entire screen of the display device, outside the frame or a portion of it, enlarging the region of interest, or changing the color or size of a mark added to the region of interest.

15. The notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest and the region of interest has moved or has exceeded a specific range specified on the image.

15. The image processing device according to claim 1.

16. a video storage unit that stores images or videos acquired from the image acquisition unit over time; The video information storage device further includes a video extraction unit that extracts from the video storage unit a video of a certain period before and after the notification unit issues a notification, or a video of an arbitrary time or period designated by a user, 16. The image processing device according to claim 1, further comprising a display device that displays the extracted image.

17. 17. The image processing device according to claim 16, wherein the extraction period of the video is automatically determined based on at least one of a moving distance, a moving speed, and an acceleration of the area of ​​interest when a notification is generated.

18. 18. The image processing device according to claim 16, wherein the extracted image is automatically and repeatedly displayed a predetermined number of times.

19. 19. The image processing device according to claim 16, wherein the extracted video is displayed based on any operation including play, stop, fast forward, rewind, frame by frame, slow playback, and double-speed playback.

20. The image processing device according to any one of claims 16 to 19, further comprising: a time elapsed from the time the notification occurred, a comparison between the position of the area of ​​interest after an arbitrary time has elapsed between the time the notification occurred and the time the notification occurred, or a trajectory of the area of ​​interest acquired by the tracking unit, which is further displayed superimposed on the extracted image.

21. 21. The image processing device according to claim 16, wherein the extracted image is displayed by cutting out a part of the image near the region of interest.

22. 22. The image processing device according to claim 16, wherein the extracted image is displayed at a position that does not obstruct the display of the region of interest and / or is displayed in an enlarged form.

23. 23. The image processing device according to claim 16, wherein the extracted image is displayed simultaneously with generation of the notification or after a predetermined time has elapsed since generation of the notification.

24. 24. The image processing device according to claim 16, wherein images captured from a plurality of directions are simultaneously displayed.

25. 25. The image processing device according to claim 2, further comprising a state estimation unit that estimates a current position and / or velocity of the tip of the catheter or the tip of the guidewire that has disappeared from the image, based on the position, velocity and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappeared from the image.

26. 26. The image processing device according to claim 25, wherein a warning is issued to a user when a current position and / or a speed of the region of interest estimated by the state estimation unit exceeds a predetermined threshold.

27. 27. The image processing device according to claim 1, wherein the display device that displays the images displays the two images on two screens having different sizes.

28. 28. The image processing device according to claim 27, wherein the display device draws the user's attention by making a frame portion of one of the two screens shine, change a color, or highlight it.

29. 29. The image processing device according to claim 1, wherein the display device for displaying the image displays a product list of devices for intravascular inspection or treatment.

30. 30. The image processing device of claim 29, wherein the display device displays a list of products filtered by size or stock.

31. 31. The image processing device according to claim 29 or 30, wherein the display device displays a list of recommended products based on image analysis results, facility information, or user preference information.

32. 32. The image processing device according to claim 1, which creates a surgical record including information on the device used, information on the acquired images, and the results of image analysis automatically or based on the user's selection.

33. 33. The image processing device according to claim 1, wherein the notification unit displays, on a display device that displays the image, a numerical value, a color, a bar, or a heat map corresponding to a probability that at least one of the regions of interest satisfies a condition determined for each of the regions of interest, or a numerical value, a color, a bar, or a heat map based on a value obtained by applying an arbitrary transformation to a probability distribution.

34. 34. The image processing device according to claim 1, wherein the notification unit colors the region of interest with a color or heat map corresponding to a probability that at least one of the regions of interest satisfies a condition set for each region of interest, or with a color or heat map based on a value obtained by arbitrarily converting a probability distribution, and displays the color or heat map on a display device that displays the image, or replaces the probability of satisfying a condition with a numerical value or color and displays the result on a display device that displays the image.

35. 35. The image processing device according to claim 1, wherein the notification unit notifies the user when, when a region including a tip of a catheter or a tip of a guidewire is set as the region of interest, the region of interest has moved or has exceeded a specific range specified on an image.

36. 36. The image processing device according to claim 35, wherein the boundary line of the specific range is represented by a straight line, a curve, a circle, a rectangle, or another polygon.

37. 37. The image processing device according to claim 35, wherein the specific range is displayed superimposed on an X-ray image.

38. 38. The image processing device according to claim 35, wherein the notification unit causes a display device that displays the image to display the distance between the region of interest and an edge of the specific range.

39. 39. The image processing device according to claim 38, wherein the notification unit changes a manner in which the distance is displayed on the display device depending on a distance between the region of interest and an edge of the specific range.

40. 40. The image processing device according to claim 35, wherein the notification unit notifies the user on condition that a value obtained by dividing a distance between the region of interest and an edge of the specific range by a moving speed of the region of interest within the image is less than a predetermined threshold.

41. 41. An image processing apparatus according to any one of claims 3 to 40, wherein the distance is determined either by a straight line distance or a distance along a blood vessel.

42. 42. An image processing device as claimed in any one of claims 1 to 41, further comprising a memory unit which acquires and stores the position and / or shape of an intravascular inspection or treatment device at any time, and the stored position and / or shape of the device is superimposed on subsequent images.

43. Aneurysms, stenosis, vasospasm, dissection, occlusion, recanalization, thrombus formation, location of thrombus and positions of both ends, vascular perforation, extravascular leakage of contrast medium, vascular calcification, arteriosclerosis, shunt disease and its feeding vessels and outflow vessels, blood (contrast medium) reflux, cerebral arteriovenous malformation, dural arteriovenous fistula, avascular area, bone markings (internal auditory canal, fundus, supraorbital rim, petrous part, foramen magnum, cervical vertebrae, clavicle, rib and vertebral numbers, femoral head, pelvis), tumors in the above images Blood vessels feeding vessels / tumor staining, venous occlusion, venous sinus thrombosis, avascular area in the capillary phase, vascular occlusion, shape and distribution of coils within the aneurysm, position, expansion and shape of the balloon, deviation of the coil into normal blood vessels, insufficient expansion of the stent, degree of adhesion to the blood vessel and twisting, movement of the stent, position of both ends of the stent, positional relationship between the puncture site and the blood vessel (is there a stenosis or is it near a bifurcation?), tortuosity of the blood vessel, type of aortic arch (is the right brachiocephalic artery the major one?) 43. The image processing device according to claim 1, further comprising a lesion recognition unit that recognizes lesions selected from the group consisting of: how far below the top of the aortic arch), the penetration range of liquid embolic material, delay or stagnation of blood (contrast medium) flow, variations in blood vessels (anterior communicating artery, anterior cerebral artery A1, posterior communicating artery, posterior cerebral artery P1, posterior inferior cerebellar artery, anterior inferior cerebellar artery, superior cerebellar artery, superficial temporal artery, each venous sinus, and the presence or absence and degree of development of each vein), moyamoya blood vessels (stenosis or occlusion at the tip of the internal carotid artery and the development of collateral blood circulation beyond that), the position of arterial bifurcation and segments (petroma of the internal carotid artery, cavernous sinus, ophthalmic artery, and bifurcation of the middle cerebral artery M1), traces of previous surgery (clips, coils, plates, shunt tubes / valves, ventricular tubes, cisternal tubes), the position and degree of opening of a WEB device, foreign bodies (dentures, plates), and collateral blood circulation.

44. 44. An image processing device according to any preceding claim, further comprising an image recognition unit for comparing angiograms in the image with previously acquired and stored angiograms and indicating changes.

45. A processor of the image processing device acquiring an image including at least an intravascular inspection or treatment device as a subject; acquiring one or more regions including at least a portion of the device included in the image as a region of interest; tracking each of the regions of interest in the image; If at least one of the regions of interest satisfies a condition defined for each of the regions of interest, notifying a user of the image processing device of that fact; An image processing method that performs

46. On the computer, A function of acquiring an image including at least an intravascular inspection or treatment device as a subject; a function of acquiring one or more regions including at least a portion of the device included in the X-ray image as a region of interest; tracking each of said regions of interest in said x-ray image; a function of notifying a user of the computer when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; A program to achieve this.

47. An image processing device according to any one of claims 1 to 42; an imaging device that captures an image of a person having a device for intravascular inspection or treatment inserted therein and transmits the image to the image processing device; An image processing system comprising:

48. A cerebrovascular catheter surgery support system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with one or more devices inserted into a blood vessel and transmits the image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed area including at least a target area for achieving the purpose of surgery and a device inserted into a blood vessel over time; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with the one or more devices are catheters, guidewires, stents, and / or balloons; When a region including a tip end of a catheter or a tip end of a guidewire, both ends of a stent, and both ends of a balloon is set as the region of interest, the user is notified on the condition that the region of interest disappears from the image or the distance between the region of interest and an edge of the image becomes less than a predetermined threshold distance. system.

49. A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the X-ray image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed area including at least an aneurysm occurring in a patient's blood vessel, a catheter inserted into the blood vessel, and a delivery wire of an embolization coil over time; a region of interest acquisition unit configured to acquire one or more regions including at least a part of the guiding catheter included in the image as a region of interest; a marker detection unit that detects a marker provided on the delivery wire approaching one or more regions of interest set in a part of the catheter that guides the delivery wire; and a tracking unit for tracking each of the region of interest and the marker in the image; a notification unit that notifies a user of a timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap; A system comprising:

50. the notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image or a distance between the marker and the region of interest; Here, the notification unit is capable of changing the display manner of the distance on the display device according to the distance, and the change in the display manner includes changing the font, size, or color of characters displayed according to the distance, changing the color of the entire screen or a portion of the display device according to the distance, enlarging and displaying a region of interest according to the distance, or changing the color or size of a mark added to the region of interest according to the distance.

51. 51. The system according to claim 48, wherein the notification unit can emit a notification sound or transmit a vibration depending on the magnitude of the distance.

52. 52. The system of any one of claims 48 to 51, wherein the distance is determined either by a straight line distance or a distance along a blood vessel.

53. A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the X-ray image to the image processing device; Equipped with The image processing device, a positional relationship storage unit that stores a positional relationship between the tip of the catheter and the second marker; a position storage unit that stores the distance a between the neckline of the aneurysm and the first marker and the position of the second marker at the time t1; a distance estimation unit that calculates a moving distance b from the position of the second marker at time t2 and estimates a distance a-b from the aneurysm neckline of the tip of the catheter; a notification unit that notifies a user of the estimated distance; A system comprising:

54. 54. The system of claim 53, wherein the estimated distance is represented by a probability distribution.

55. 55. The system of claim 53 or 54, wherein the estimated catheter tip position is displayed in color based on a probability distribution.

56. An endovascular surgery support system comprising: a memory unit that stores a product list of devices for intravascular examination or treatment; a recommendation unit that recommends products to use based on image analysis results, facility information, or user preference information; and a display unit that displays the recommended products.

Images