Image processing device, image processing method, program, and image processing system

Abstract

To provide a technology for causing a medical worker to concentrate on his or her work in an attention part and supporting determination on the attention part in catheter examination or treatment of a blood vessel.SOLUTION: An image acquisition part 110 of an image processing device 1 acquires an X-ray image created on the basis of an absorption rate of an X-ray, including, in a subject, at least a blood vessel and a device for examination or treatment in the blood vessel. A region-of-interest acquisition part 111 receives one or a plurality of regions of interest including at least part of the device included in the X-ray image as regions of interest. A tracking part 112 tracks each of the regions of interest in the X-ray image. When any one of the regions of interest satisfies a condition stipulated for each region of interest, a notification part 113 notifies a user of the image processing device 1. The present invention also includes recording of operations in surgery and creating of the summary.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and an image processing system, and more particularly to an image processing technique for use in the inspection or treatment of blood vessels. [Background technology]

[0002] In recent years, catheter surgery has become common for the examination and treatment of blood vessels throughout the body, such as the brain and heart, in which a medical professional such as a doctor passes a catheter through the blood vessels of a subject and performs various procedures while displaying the position of the catheter on an X-ray image. In this catheter surgery, a medical professional generally passes a medical device such as a catheter through the subject's groin or upper arm, and advances the device to the area of ​​interest.

[0003] For example, during examination or treatment of cerebrovascular disease, medical personnel advance a catheter from the subject's groin or brachial artery, through the aortic arch and carotid artery, and into the cerebral artery. Since medical personnel need a lot of experience and training to perform these procedures appropriately, for example, Patent Document 1 discloses a simulation system for improving the efficiency of surgical procedures performed using a catheter system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 139894 Summary of the Invention [Problem to be solved by the invention]

[0005] In vascular examination or treatment, a medical professional passes a guiding catheter to a target site, and then passes multiple devices for examination or treatment, such as a guide wire, a stent, a balloon, a microcatheter, a liquid embolic material, a filter, and a delivery wire of an embolization coil for embolizing an aneurysm, through the guiding catheter. That is, in actual examination or treatment, a medical professional needs to pay attention to the movements and positions of multiple devices not only when passing a guiding catheter to a target site, but also during treatment. On the other hand, in order to achieve the original purpose of the examination or treatment, a medical professional needs to concentrate on the work at the target site.

[0006] The present invention has been made in consideration of these points, and aims to provide a technology that enables medical professionals to concentrate on working on areas of interest and supports their judgment on areas of interest during vascular catheter examination or treatment. [Means for solving the problem]

[0007] A first aspect of the present invention is an image processing device. The device includes an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest. The image processing device may further include a tracking unit that tracks each of the regions of interest in the image.

[0008] As a variation of the first aspect, the image processing device may include an output unit that outputs information to a device connected to the image processing device when at least one of the regions of interest satisfies a condition defined for each region of interest, instead of or in addition to a notification unit that notifies a user of the image processing device of the fact that at least one of the regions of interest satisfies a condition defined for each region of interest. The device connected to the image processing device may be connected wirelessly or by wire. The device may be, for example, a surgical robot, a VR / AR / MR device, smart glasses, a device used as a metaverse, or the like. When the output is received, for example, the surgical robot may sound a notification, issue a warning, automatically stop operation, or automatically pull the device. In the case of AR or smart glasses, an operation may be considered such as superimposing a region of interest where a condition is satisfied on the image being viewed and displaying the region of interest to notify the user. This is one example, and the operation of the device receiving the output is not limited to this.

[0009] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0010] When a region including the tip of a catheter such as a guiding catheter or the tip of a guidewire is set as the region of interest, the notification unit or output unit may notify the user or output that information to an apparatus, on the condition that the region of interest disappears from the image.

[0011] When a region including the tip of a catheter such as a guiding catheter or the tip of a guidewire is set as the region of interest, the notification unit or output unit may notify the user or output the information to an apparatus, on the condition that the distance between the region of interest and the edge of the image is less than a predetermined threshold distance.

[0012] The notification unit or output unit may notify the user or output the information to a device if at least one of the movement distance, movement speed, and acceleration of the area of ​​interest within the image exceeds a predetermined threshold.

[0013] The notification unit may cause a display device that displays the image to display the distance between the region of interest and an edge of the image.

[0014] The notification unit may change a manner in which the distance is displayed on the display device depending on a distance between the region of interest and an edge of the image.

[0015] The notification unit or output unit may notify the user or output the information to an apparatus, on condition that the value obtained by dividing the distance between the area of ​​interest and the edge of the image by the speed of movement of the area of ​​interest within the image is less than a predetermined threshold.

[0016] The image processing device may further include a marker detection unit that detects a marker provided on a delivery wire of the embolization coil approaching a region of interest set on a part of a guiding catheter that guides the delivery wire, and the tracking unit may further track the detected marker, and the notification unit or output unit may notify the user of the timing at which the embolization coil may be disconnected from the delivery wire, or output that information to an apparatus, when the marker and the region of interest overlap.

[0017] When the marker passes through the region of interest, the notification unit or output unit may notify the user of this, or output the information to an apparatus.

[0018] The notification unit may cause a display device to display a distance that the marker must move before the embolization coil is disconnected from the delivery wire.

[0019] When a feature amount indicating a shape of the device included in the region of interest satisfies a predetermined condition, the notification unit may notify a user of the image processing device of this fact.

[0020] The feature may be curvature, and the notification or output unit may notify the user or output the information to an equipment when the curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or when the curvature is changing but the tip does not move.

[0021] The feature amount may be the distance of the guidewire or the area of ​​segmentation within the region including the tip of the guidewire. If this exceeds a certain threshold, a notification is issued (FIG. 43). A condition that the movement of the tip is within a certain range may be added to this condition. The range of the region may be changed depending on the magnification rate of the screen, the contents of the surgery, and the location of the blood vessel. The threshold may be determined in advance, or may be based on the surgeon's preference, or a time-dependent value during the surgery may be accumulated, and a notification may be issued when the distance exceeds 2×standard deviation, for example. However, this is an example, and the method of determining the threshold is not limited to this.

[0022] The notification unit or output unit may notify the user or output the information to an equipment on the condition that the value obtained by subtracting the length of the center line of the blood vessel contained in the image or region of interest from the length of the device contained in the image or region of interest exceeds a predetermined threshold length.

[0023] The notification unit may have functions such as displaying the area of ​​interest in a different color from the image, changing the font, size or color of the displayed text, changing the color of the entire screen of the display device or a portion of it, displaying a graphic on the entire screen of the display device or outside the frame or a portion of it, enlarging the area of ​​interest, or notifying the user by changing the color or size of a mark added to the area of ​​interest.

[0024] The notification unit may use sound or vibration for notification.

[0025] A second aspect of the present invention is an image processing method. In this method, a processor of an image processing device executes the steps of acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, tracking each of the regions of interest in the image, and notifying a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest. The method according to the present disclosure may be a method implemented in a computer.

[0026] A third aspect of the present invention is a program. The program causes a computer to realize a function of acquiring an image including at least an intravascular inspection or treatment device as a subject, a function of acquiring one or more regions including at least a part of the device included in the X-ray image as regions of interest, a function of tracking each of the regions of interest in the X-ray image, and a function of notifying a user of the computer when at least one of the regions of interest satisfies a condition defined for each region of interest. The program may include instructions that cause the computer to execute a specific step when the program is executed by the computer.

[0027] In order to provide this program or to update a part of the program, a computer-readable recording medium having this program recorded thereon may be provided, or the program may be transmitted over a communication line. A third aspect of the present invention includes a computer-implemented method, which may be performed by a computer including a processor and a memory capable of storing instructions for executing the method. The instructions or program for executing the method on a computer may be recorded on a non-transitory computer-readable medium.

[0028] A fourth aspect of the present invention is an image processing system, which includes the above-mentioned image processing device and an imaging device that captures an image of a person having an intravascular inspection or treatment device inserted therein (an image of a surgical field) and transmits the image to the image processing device.

[0029] Any combination of the components disclosed in this specification, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also effective as aspects of the present invention.

[0030] A further aspect of the present invention relates to an image processing device including a storage device such as a memory and a processor connected to the storage device, the processor performing processing such as acquiring an image including at least an intravascular inspection or treatment device as a subject. Effect of the Invention

[0031] According to the present invention, in vascular catheter examination or treatment, a technique can be provided that enables medical personnel to focus on work in areas of interest and prevents oversights and delayed judgment in areas other than those of interest. [Brief description of the drawings]

[0032] [Figure 1] 1 is a diagram illustrating an external view of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an image processing device according to an embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a region of interest. [Figure 4] FIG. 13 is a diagram for explaining an example of a condition set for a region of interest. [Diagram 5] FIG. 11 is a diagram illustrating an example of a message notified by a notification unit. [Figure 6] FIG. 13 is a diagram for explaining another example of a condition set for a region of interest. [Figure 7] FIG. 13 is a diagram for explaining another example of a condition set for a region of interest. [Figure 8] FIG. 13 is a diagram for explaining the timing of detachment of the embolization coil. [Figure 9] 13 is a diagram for explaining conditions related to the shape of a device that are set in a region of interest. FIG. [Figure 10] 10 is a flowchart illustrating a flow of an image analysis process executed by the image processing device according to the embodiment. [Figure 11] FIG. 1 is a schematic diagram of a typical neural network. [Figure 12] 1 is a diagram illustrating a functional configuration of an image processing device having a replay function according to an embodiment of the present invention; [Figure 13] FIG. 13 is a diagram for explaining an example of a replay display when a notification occurs. [Figure 14] FIG. 13 is a diagram for explaining an example of displaying a replay playback window on a real-time display screen. [Figure 15] 1 is a diagram illustrating a functional configuration of an image processing device including a state estimation unit according to an embodiment. [Figure 16] 11A and 11B are diagrams for explaining an example of a display of an estimation result of a state estimation unit according to an embodiment. [Figure 17] 13 is a diagram for explaining an example of estimating the catheter tip position using a second marker in the embodiment. FIG. [Figure 18] FIG. 13 is a diagram for explaining an embodiment in which the results of image analysis are displayed on two screens of different sizes. [Figure 19] 11A and 11B are diagrams for explaining an embodiment in which a screen to be notified is recognized by highlighting the frame of the screen. [Figure 20] 1 is a diagram for explaining the display of a product list of devices for intravascular inspection or treatment (for example, various catheters and coils) according to an embodiment. FIG. [Figure 21] 5A to 5C are diagrams for explaining a process of storing the position and shape of a device according to an embodiment. [Figure 22]13 is a flowchart for explaining the flow of a catheter tip position estimation process using a second marker, which is executed by an image processing device according to an embodiment. [Figure 23] 11 is a flowchart for explaining a process flow of a replay function executed by the image processing device according to the embodiment. [Figure 24] 10 is a flowchart for explaining the flow of processing for estimating the position of a region of interest outside a frame, which is executed by an image processing device according to an embodiment. [Diagram 25] 11A to 11C are diagrams for explaining detection of a device at an edge of an image in the embodiment. [Figure 26] 11 is a flowchart for explaining detection of a device at an edge of an image according to an embodiment. [Figure 27] 11A and 11B are diagrams for explaining detection of a device that has passed a specified boundary line in the embodiment. [Figure 28] 11A and 11B are diagrams for explaining detection of a device that has passed a specified boundary line in the embodiment. [Figure 29] 13A and 13B are diagrams for explaining display of a device position on a sub-screen in the embodiment. [Diagram 30] 11A and 11B are diagrams for explaining an example of recognition of a surgical situation in the embodiment; [Diagram 31] FIG. 1 illustrates an example of a configuration of a computer according to the present disclosure. [Diagram 32] Figure 1 shows an example of blood vessel extraction: the left figure shows the recognition result. [Diagram 33] FIG. 2 is a diagram illustrating a functional configuration of an image processing device according to an embodiment. [Diagram 34] FIG. 2 is a diagram for explaining a notification by voice (words) according to an embodiment. [Diagram 35] 11A and 11B are diagrams for explaining automatic setting of a specific boundary line according to an embodiment. [Diagram 36] FIG. 13 is a diagram for explaining notification when a liquid embolic material (such as Onyx or NBCA) crosses a specified boundary. [Figure 37] 11 is a diagram for explaining variations in the method of setting a specific area according to an embodiment. There are various ways to draw a specific area (boundary) (four examples A to B are shown). A: Specification using a rectangle. B: Specification using an ellipse. C: Specification using a horizontal line. D: Specification using a vertical line. [Figure 38] This is a diagram showing a specific example of blood vessel recognition, where the right figure shows the recognition result. [Figure 39] FIG. 13 is a diagram for explaining a notification when the outside of a blood vessel (outside a recognized blood vessel) is specified as a specific region. [Diagram 40] FIG. 13 is a diagram for explaining a notification when a filter is moved. [Diagram 41] This figure shows how the edge of the X-ray portion is automatically detected and the boundary is set accordingly. [Diagram 42] This is a diagram for explaining an embodiment in which an area moves or disappears in response to the enlargement / reduction movement of the screen or a scene change. It is also possible to set it automatically or to suggest it to the user so that the user presses OK or NO accordingly. [Diagram 43] This is a diagram to explain the notification when the tip of the guide wire is suddenly bent. Specifically, for example, it is possible to notify when the length of the GW within the boundary including the tip becomes a certain distance or more. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] <Outline of the embodiment> 1 is a diagram showing a schematic external view of an image processing system S according to an embodiment. The image processing system S includes an image processing device 1, a display device 2, and an X-ray imaging device 3. Hereinafter, an overview of the embodiment will be described with reference to FIG.

[0034] The X-ray imaging device 3 is a device for capturing an X-ray image of a subject P, who is a person having a device (hereinafter, sometimes simply referred to as a "device") inserted therein for inspecting or treating blood vessels, and transmitting the X-ray image to the image processing device 1. For this reason, the X-ray imaging device 3 includes an X-ray irradiator 30 (a first X-ray irradiator 30a and a second X-ray irradiator 30b) for irradiating X-rays to the subject P, an X-ray detector 31 for detecting the X-rays irradiated by the X-ray irradiator 30, and a bed 32 for supporting the subject P.

[0035] As shown in FIG. 1, the first X-ray irradiator 30a and the second X-ray irradiator 30b can irradiate X-rays to the head of the subject P at different angles of incidence. Here, the X-rays irradiated by the first X-ray irradiator 30a are detected by the first X-ray detector 31a and converted into an X-ray image based on the absorption rate of the X-rays. The X-rays irradiated by the second X-ray irradiator 30b are detected by a second X-ray detector (not shown) and converted into an X-ray image. These X-ray images are displayed on the display device 2. During a treatment of the cerebrovascular region, the positions of the first and second X-ray irradiators are generally fixed, and the area of ​​the image displayed on the display device 2 is fixed.

[0036] The image generated from the X-rays detected by the X-ray detector 31 shows the subject P's blood vessels (which become visible by injecting a contrast agent) and tissues such as bones, as well as various devices used in the examination and treatment of blood vessels (e.g., catheters such as guiding catheters, guide wires, embolization coils, delivery wires for transporting embolization coils to the area of ​​interest, etc.).

[0037] The image processing device 1 according to the embodiment is a device for assisting a user performing a vascular catheter examination or treatment (hereinafter, simply referred to as a "catheter surgery" except when distinguishing between a vascular catheter examination and a vascular catheter treatment). The image processing device 1 recognizes and / or tracks one or more regions of interest that are set in advance in an X-ray image generated based on X-rays detected by an X-ray imaging device 3. The image processing device 1 analyzes the X-ray image, and when the state of any of the regions of interest meets a condition set for each region of interest, notifies a medical professional (hereinafter, simply referred to as a "user") who is a user of the image processing device 1. By setting a region of interest that requires attention other than the region of interest for achieving the purpose of the examination or treatment, the user can concentrate on the work at the site of interest (for example, work such as guiding a microcatheter into an aneurysm, inserting a coil into an aneurysm, expanding a balloon, placing a stent, etc.).

[0038] <Functional configuration of image processing device 1 according to the embodiment> FIG. 2 is a diagram showing a schematic functional configuration of an image processing device 1 according to an embodiment. The image processing device 1 includes a storage unit 10 and a control unit 11. In FIG. 2, arrows indicate main data flows, and data flows not shown in FIG. 2 may exist. In FIG. 2, each functional block shows a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data exchange between the functional blocks may be performed via any means, such as a data bus, a network, or a portable storage medium. FIG. 33 is a diagram showing a schematic functional configuration of an image processing device 1 according to another embodiment. In FIG. 33, it is shown that output is performed from the output unit 116 to an external device 4.

[0039] FIG. 31 is a diagram showing an example of the configuration of a computer according to the present disclosure. As shown in FIG. 31, the configuration of the computer may include a CPU 20, which is a processor capable of performing arithmetic processing, a ROM 21, which is a memory connected to the processor and capable of storing BIOS, a RAM 22, which is a memory that may be a working area, and a storage 23, which may store programs, and may further include an input unit 25, an output unit 26, and a storage medium 27 via an input / output interface 24. The input unit 25 may include an input device such as a keyboard. The output unit 26 may include an output device such as a display. Data may be transmitted and received via the input unit 25 and the output unit 26.

[0040] In FIG. 2, the storage unit 10 is a large-capacity storage device (also referred to as storage) such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the image processing device 1, a RAM (Random Access Memory) that serves as the working area of ​​the image processing device 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores an OS (Operating System), application programs, and various information referenced when the application programs are executed.

[0041] The control unit 11 is a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the image processing device 1, and functions as an image acquisition unit 110, a region of interest acquisition unit 111, a tracking unit 112, a notification unit 113, a marker detection unit 114, and a distance measurement unit 115, etc., by executing programs stored in the memory unit 10.

[0042] 2 shows an example in which the image processing device 1 is configured as a single device. However, the image processing device 1 may be realized by a plurality of computing resources such as processors and memories, for example, as in a cloud computing system. In this case, each unit constituting the control unit 11 is realized by at least one of a plurality of different processors executing a program.

[0043] The image acquisition unit 110 acquires an X-ray image created based on the absorption rate of X-rays, the image including at least the blood vessel of the subject P and an examination or treatment device in the blood vessel as a subject. For example, the X-ray image may include an aneurysm occurring in the blood vessel of the subject P, a stenosis or an infarction of the blood vessel that is of interest to the medical staff. The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device included in the X-ray image as a region of interest. For example, the tip of a guidewire (GW), the tip of a guiding catheter (GC), a catheter marker, a coil, etc. may be set as the region of interest, but multiple regions may be set at the same time, such as the tip of a GC and a GW, the tips of two GWs, or the tip of a GW and a coil. In some aspects, the region of interest may include a blood vessel (a vascular lesion such as a cerebral aneurysm or a stenosis) or a bone.

[0044] 3(a)-(d) are diagrams for explaining a region of interest. Specifically, Fig. 3(a) is a diagram that shows a schematic example of an X-ray image of a blood vessel V. In the example shown in Fig. 3(a), a guide wire, which is a type of device D, is present in the blood vessel V.

[0045] FIG. 3(b) is a diagram showing an example of a candidate region C that is a candidate for the region of interest. The region of interest acquisition unit 111 may include a candidate region detector generated using a known machine learning method such as a neural network (FIG. 11). In FIG. 3(b), a first candidate region C1 (tip of the guiding catheter) and a second candidate region C2 (tip of the guidewire) are shown as candidate regions for the region of interest. This candidate region C is a result obtained when the region of interest acquisition unit 111 inputs a frame image of an X-ray image into the candidate region detector. As a non-limiting example, the candidate region detector is trained to detect the tip of the guidewire, the tip of the guiding catheter, a catheter marker, and the like.

[0046] Detection generally means identifying the position and shape of an object in a still image of one frame in a video. For example, to detect the tip of a guidewire, since the guidewire is thin, it may be detected as a point coordinate (x, y) on the screen. To detect the entire guidewire, the curved, thin, thread-like shape can be detected as a one-dimensional curve or as a two-dimensional fill (segmentation).

[0047] Examples of object detection algorithms that can be used in the candidate region detector include, but are not limited to, Faster R-CNN, YOLO, SSD, U-Net, ResNet, etc. Note that, when implementing an object recognition algorithm, an image processing library such as OpenCV (Open Source Computer Vision Library) may be used.

[0048] (Input and output for learning detection) The detection method is not limited to machine learning, but for example, in the case of machine learning, actual surgery videos, examination videos, videos from an angiography device using a phantom, virtual surgery videos, etc. are used as input data for learning. Still images are also acceptable instead of videos. Virtual surgery videos can be created manually, images similar to surgery can be generated using an algorithm, or "generative adversarial networks" (GAN) can be used.

[0049] For learning, videos of endovascular surgery and examinations of any part of the body can be used (e.g., brain, heart, peripheral limbs, abdomen, pelvis). Any medical images such as X-ray images, CT images, and MRI images can also be used.

[0050] A video may be appropriately divided into still images, and one or more still images may be used as input images for learning. The input images may be subjected to any transformation (e.g., normalization, noise removal, etc.). Multiple images may be input as a time-series image. Front and side images may be input simultaneously, or mask and live images may be input. A combination of these may also be used as input images, for example, a time-series of front live, side live, front mask, and side mask images may be input. By inputting these multiple related images, it is expected that accuracy will improve.

[0051] Annotations can be made based on any endovascular treatment device, blood vessels, bones, lesion sites (aneurysm, stenosis, vascular malformation, etc.), vascular anatomy (internal carotid artery, Sylvian vein, etc.), magnification, site (head, chest, etc.), image quality (degree of noise), image type (mask image, live image, during contrast, during 3D imaging, etc.), surgical procedure, device used in surgery, patient information (age, sex, disease, etc.), etc. For endovascular treatment devices, blood vessels, bones, lesion sites, and vascular anatomy, annotations are made on the image with points, lines, circles, ellipses, curves, closed curves, fill (segmentation), bounding boxes, etc. An ID may also be assigned as an instance.

[0052] This is learned using machine learning etc. The learned model is then used to infer the device position and shape (e.g., guidewire curve and segmentation, guidewire tip and tip periphery, balloon inflation, stent stump, stent shape, etc.), blood vessel position, shape and properties (e.g., stenosis), image quality (e.g., high noise), and image type (e.g., during angiography, masked image, etc.). At this time, the inference results of the live image and the inference results of the masked image may be combined to make an inference. Alternatively, the inference results of the front image and the inference results of the side image may be combined to make an inference. This is expected to improve accuracy.

[0053] This model can be used for object detection in images, as well as scene classification for still and video images, and can be used to make inferences on any data input above.

[0054] (Vessel recognition / display) The image processing device 1 according to the embodiment can recognize not only the region of interest including the device, but also blood vessels. The method is the same as the device recognition described above. Machine learning / deep learning (AI) or rule-based methods are used, but the method is not limited to these. For example, annotation of blood vessels (such as filling in by segmentation) is performed in a live or masked video or still image, and learning is performed by deep learning. Alternatively, since the image is grayscale (monochrome), a threshold may be set to extract only white and black parts. Blood vessels may be extracted by contour extraction. If it is a video, an arithmetic average may be taken. Blood vessels are extracted by a combination of these. Blood vessels may be recognized from one or multiple mask images over time, or may be recognized using one or multiple images during contrast enhancement. Furthermore, a front image and a side image may be recognized together. By recognizing images from two directions, it is expected that the recognition accuracy will be improved.

[0055] Recognition may be performed in real time, but is not limited to real time. Images may include intraoperative DSA images, mask images, and 3D images (3DRA), preoperative CTA, MRA, and 3DRA, and other images. Blood vessels may be recognized by filling (segmentation), segmentation boundaries, curve sets, and curve tree structures, but are not limited to these.

[0056] (Recognition and display of vascular anatomy) When recognizing blood vessels, anatomical information can be inferred at the same time. For example, blood vessels can be recognized by identifying left and right, or by anatomical classification such as internal carotid artery, external carotid artery, middle cerebral artery, posterior cerebral artery, posterior communicating artery, and anterior communicating artery. The internal carotid artery can be recognized by dividing it into C1 to C5, and the middle cerebral artery can be recognized by dividing it into M1, M2 (superior trunk, inferior trunk, anterior temporal artery, etc.), M3, etc. Furthermore, blood vessels can be converted into a tree-like shape (linear, 1D) by taking a center line from it, rather than just segmenting it (filling, 2D). The recognition method and representation method are the same as those described above.

[0057] (Recognition and display of vascular lesions) When recognizing blood vessels, lesions may also be recognized. For example, cerebral aneurysms, stenosis, cerebral arteriovenous malformations, occlusions, etc. may be recognized. The recognition method and expression method are the same as those described above.

[0058] (Identification and display of important blood vessels in surgery) Important blood vessels in surgery may be extracted manually or automatically, and the blood vessels may be highlighted or displayed alone. For example, automatic extraction may involve, but is not limited to, selecting blood vessels whose diameters are greater than a threshold, extracting main blood vessels such as the internal carotid artery from the above-mentioned vascular anatomy recognition, or identifying aneurysms or stenosis from the above-mentioned lesion recognition and extracting only blood vessels along them.

[0059] (Example of blood vessel extraction) A specific example of blood vessel extraction is shown in Figure 32. This type of blood vessel extraction can be performed automatically.

[0060] (Learning specific to (neuro)endovascular therapy) A distinctive feature of endovascular treatment is that it is performed by viewing up to four X-ray images (two directions, frontal and lateral images, plus a mask and live image). Therefore, instead of using one of the four images as the input image, two to four images may be used as input for learning. Alternatively, an image generated by appropriately combining the four images may be used as the input image (taking the difference, for example). Alternatively, since the source of the mask image is the difference between a previously taken DSA image (digital subtraction angiography, which is the difference between the live images before and after the contrast agent is injected) and a real-time DA image (digital angiography), the original image of these may be used as the input image.

[0061] In addition, after performing inference on 1 to 4 screens using the model, the inference results can be combined to improve accuracy. For example, if there is a catheter marker in the same location on two screens, the live image and the mask image, there is a high possibility that the recognition is correct, but if there is only one of them, the possibility of it being correct decreases (the confidence level decreases). This can be used to improve the recognition accuracy.

[0062] In some aspects, the present disclosure relates to a method for creating a machine learning model, the method comprising training the model using cerebrovascular images acquired from multiple directions as input.

[0063] (Example Detection Algorithm) Faster R-CNN is a CNN (Convolutional Neural Network) that simultaneously performs region extraction and recognition. A convolutional neural network (CNN) is a neural network with many deep layers that is composed of layers with several characteristic functions such as "convolutional layers" and "pooling layers", and has demonstrated excellent performance especially in the field of image recognition. Faster R-CNN can extract and recognize regions of interest from input images in almost real time (about 10 to 20 frames per second). Faster R-CNN allows end-to-end learning from image input to object detection.

[0064] YOLO (You Only Look Once) is also a CNN that simultaneously extracts and recognizes regions. In YOLO, the entire image is divided into grids and a bounding box is calculated for each region. YOLO's CNN architecture enables high-speed object detection.

[0065] SSD (Single Shot MultiBox Detector) is also a CNN that simultaneously extracts and recognizes regions. SSD can output multi-scale detection frames from output layers at various levels. SSD places approximately 9,000 rectangular frames of different sizes and shapes called default boxes on the image and calculates a prediction value for each of them. SSD aims to increase speed by reducing the filter size.

[0066] U-Net is a CNN that recognizes objects pixel by pixel, called segmentation. It is composed of convolutional layers and has an almost symmetrical Encoder-Decoder structure, where the feature map downsampled through pooling in the Encoder is upsampled in the Decoder.

[0067] FIG. 3(c) is a diagram showing an example of a method for setting the region of interest R. In the example shown in FIG. 3(c), a state in which the user selects the second candidate region C2 as the region of interest R is shown. Specifically, the user selects the region of interest R by moving the mouse cursor M to the second candidate region C2 using a pointing device (not shown) such as a mouse. Note that the presentation of the candidate region C by the region of interest acquisition unit 111 is optional, and the user may directly set the region of interest R in an X-ray image in a state in which the candidate region C is not presented. For example, the region of interest R can be set by drawing a rectangle in the image using a pointing device such as a mouse to specify the region. Alternatively, the region of interest acquisition unit 111 may acquire, as the region of interest R, the output of a candidate region detector generated using a known machine learning method or the like.

[0068] Fig. 3(d) is a diagram showing a region of interest R set by the user. In the example shown in Fig. 3(d), the region of interest R is shown as a black pentagon, and is a region including the tip portion of the device D. In Fig. 3(d), only one region of interest R is set, but the user can set two or more regions of interest R.

[0069] Returning to the description of FIG. 2, the tracking unit 112 tracks one or more regions of interest R in the X-ray image. The tracking unit 112 can realize tracking of the regions of interest R by using a known image tracking technique. Known image tracking techniques that can be used include, but are not limited to, algorithms such as Boosting, MIL, TLD, MedianFlow, KCF, GOTURN, MOSSE, and CSRT. The tracking algorithm may be used in combination with the object detection algorithm described above. For implementation of the algorithm, a library such as OpenCV (Open Source Computer Vision Library) may be used. In the context of the present invention, tracking of the region of interest R also includes intermittently detecting the region of interest R and identifying its state. Therefore, in some aspects of the present invention, it is not required that the region of interest R be recognized in all frames of the X-ray images acquired over time. In some aspects of the present invention, tracking of the region of interest R may be performed using a tracking algorithm or an object detection algorithm, or a combination of them.

[0070] The BOOSTING tracker is a tracker based on an online version of AdaBoost (the algorithm used internally by the HAAR cascade-based face detector). The classifier is trained at runtime using ground truth and false truth examples of the object. The first bounding box identified by the user (or another object detection algorithm) is treated as the ground truth example of the object, and any image outside the bounding box is treated as background. Given a new frame, the classifier is applied to all pixels in the neighborhood of the previous position and the score is recorded. The new position of the object is the one with the maximum score. This gives us another ground truth example for the classifier. Further frames are fed in and the classifier is updated with this additional data.

[0071] The MIL tracker is based on a similar concept to the BOOSTING tracker described above. The main difference is that instead of considering only the current position of the object as the ground truth, it looks at a small neighborhood around the current position to generate several potential ground truths. Instead of identifying ground truths and false truths, MIL specifies a ground truth and false truth "bag". The ground truth bag does not contain all the images that are ground truths. Only one image in the ground truth bag should be the ground truth. The ground truth bag contains an image centered on the object's current position and a small neighborhood around it. Even if the tracked object's current position is not accurate, if the sample near the current position is in the ground truth bag, this bag is likely to contain at least one image with the object properly centered. The MIL tracker performs well, does not drift as much as the BOOSTING tracker, and performs reasonably well in the presence of partial occlusion.

[0072] KFC stands for Kernelized Correlation Filter. This tracker is based on the concepts proposed in the above two trackers. This tracker exploits the fact that the ground truth samples used in the MIL tracker have large overlapping regions. Such overlapping data provides some good mathematical properties, making tracking faster and more accurate at the same time. It outperforms MIL in both accuracy and speed, and has better reporting of tracking failures.

[0073] TLD stands for Track, Learn, Detect. As the name suggests, this tracker breaks down the long-term tracking task into three components: (short-term) tracking, learning, and detection. The tracker tracks the object frame by frame. The detector identifies all previously observed appearances and calibrates the tracker if necessary. Learning estimates the detector's errors and updates it to avoid future errors. The output of this tracker tends to be a bit unstable, for example, if you are tracking a pedestrian and there are other pedestrians in the scene, the tracker may temporarily track a different pedestrian than the one you want to track. The advantage is that it works best under occlusion across multiple frames. The disadvantage is that it has a high number of false positives.

[0074] The MEDIANFLOW tracker tracks objects both forward and backward in time and measures the discrepancy between these two trajectories. By minimizing this forward / backward error, it is able to reliably detect tracking failures and select reliable trajectories in videos. This tracker performs best when motion is predictable and small, and there are no occlusions. Unlike other trackers that continue even when tracking clearly fails, this tracker can recognize when tracking has failed.

[0075] The GOTURN tracker is an algorithm based on a convolutional neural network (CNN). This tracker is robust to viewpoint changes, lighting changes, and deformations, but may not handle occlusions well.

[0076] The MOSSE (Minimum Output Sum of Squared Error) tracker uses adaptive correlation for object tracking. The MOSSE tracker is robust to changes in lighting, scale, pose, and non-rigid deformations. The tracker can also detect occlusion based on peak-to-sidelobe ratio and resume tracking where it left off when the object reappears. The MOSSE tracker also works at high frame rates (450fps and above). Its advantages are that it is very simple to implement, is as accurate as other complex trackers, and is much faster.

[0077] The CSRT tracker uses a spatial confidence map for tracking. The CSRT tracker operates at a relatively low frame rate (25 fps) but gives higher accuracy in object tracking.

[0078] When at least one of the regions of interest R satisfies a condition set for each region of interest, the notification unit 113 notifies the user of the fact. Specifically, when at least one of the regions of interest R satisfies a condition set for each region of interest, the notification unit 113 notifies the user by displaying a message indicating the fact on the display device 2 or by sounding a notification sound from a speaker of the image processing device 1 (not shown). Alternatively, when the user is wearing a device equipped with a vibrating member such as a smartphone, the notification may be made to the user by vibrating the vibrating member. The type, amplitude, and frequency of the notification sound may be changed depending on the conditions. Furthermore, the notification sound may be combined with a voice (FIG. 34). As a problem, the surgeon or assistant may be concentrating on the surgery and may not be able to see the notification screen or may be late in seeing it. In this case, if the notification status can be determined by voice alone, this problem can be solved. For example, sounds such as "beep, coil approaching in mask image on the front" (Figure 34), "beep, guide wire off screen in live image on the side", "beep, guiding catheter off screen in live image on the front", "beep, balloon on side (on B plane) appears on screen", "beep, filter moved on front (on A plane)" allow the surgeon and assistant to understand what happened without taking their eyes off the main screen of the surgery. It is sufficient to communicate by voice what happened on which screen and which device, and how the voice can be communicated is not limited to the above example. In addition, the period and amplitude of the vibration may be changed depending on the conditions.

[0079] In this way, by setting a region of interest R in an X-ray image in advance, the user of the image processing device 1 can have the image processing device 1 track the behavior of the set region of interest R, allowing the user to concentrate on the work at the target site. When displaying the region of interest, the notification unit 113 may display the region of interest in a color different from the color of the acquired image (X-ray image). Generally, angiography images are monochrome, so it is difficult to recognize devices included in the image. However, by displaying the region of interest in a color, it is possible to easily recognize the devices and provide more accurate information to the user. Furthermore, when multiple regions of interest are set, the regions of interest may be colored using different colors.

[0080] [Example of conditions set for region of interest R] Next, a specific example of the conditions set for the region of interest R will be described.

[0081] In vascular catheter surgery, the user first passes a guiding catheter into the blood vessel of the subject P up to the vicinity of the area of ​​interest. Next, the user passes other devices such as a guidewire, a delivery wire, or a balloon catheter through the guiding catheter to observe (diagnose) or treat the area of ​​interest, for example, coil embolization of a cerebral aneurysm.

[0082] A catheter is a type of device that is a long, thin tube with a lumen. This catheter is passed through a blood vessel, and guided, for example, into an aneurysm, and a coil is inserted through the lumen of the catheter to prevent the aneurysm from rupturing.

[0083] There are several types of catheters. First, the guiding catheter is a slightly thicker catheter with a diameter of about 2-4 mm, and serves to connect the puncture site to just before the target site. Smaller devices are inserted into this guiding catheter and guided to the target site. The advantage of the guiding catheter is that various devices can be carried from the puncture site of the blood vessel in the groin or arm to the proximal part of the target site through the guiding catheter, eliminating the need to check the path each time. This allows, for example, efficient treatment to be performed without moving a fixed screen.

[0084] Some guiding catheters have a balloon at the tip (balloon-equipped guiding catheter), which can be used to stop blood flow in the blood vessel or stabilize the guiding catheter.

[0085] An intermediate catheter is placed inside a guiding catheter to deliver a thinner device to the target site, closer to the target site where the blood vessel is narrower. Multiple intermediate catheters may be used (gradually thinner).

[0086] Microcatheters are the thinnest catheters, and are soft, making them easy to insert into small blood vessels. Coils and stents are placed inside these microcatheters and transported to the target site. Embolic materials can also be flushed out of the microcatheter.

[0087] A balloon catheter is a microcatheter with a balloon near the tip, which is inflated near the target site. This can be used, for example, to prevent coils from coming out of an aneurysm, to expand a narrowed area, or to stop blood flow when a blood vessel is perforated, thus stopping bleeding. When the word "balloon" is used, it usually refers only to the balloon part of the balloon catheter.

[0088] When guiding these catheters, a guide wire of appropriate size is often used inside. In rare cases, they may be guided along the bloodstream.

[0089] A guidewire is a long, thin wire. In general, a guidewire is used to select a branched blood vessel and guide a soft catheter to a target site. Other examples include a delivery wire for carrying a stent and a device with a balloon attached to the guidewire that can stop blood flow. In this specification, the term "guidewire" is used in a broad sense to include these. The reason is that the tip of any of these guidewires may perforate a blood vessel, causing serious complications. One of the purposes of the present invention is to assist in preventing blood vessel perforation by the tip of such a guidewire, so the term "guidewire" refers to a thin wire that may cause blood vessel perforation at its tip.

[0090] During the procedure, the blood vessels of the subject P are present inside the subject P's body, so the user cannot directly observe the various devices inserted into the blood vessels. Therefore, as described above, the user operates the device D while viewing the blood vessel V and the device D (e.g., a guiding catheter, a guide wire, a coil, a catheter, a balloon, a stent, etc.) captured in the X-ray image generated from the X-rays that have passed through the body of the subject P. As shown in FIG. 1, the X-ray imaging device 3 used in the catheter surgery is a device for generating an X-ray image of a specific area including a site of interest (e.g., an aneurysm, a stenosis site, an infarction site, etc. that require treatment) in the body of the subject P. In many cases during the procedure, particularly in the cerebrovascular area, the area acquired as an X-ray image is fixed. Therefore, the entire device D is not always captured in the X-ray image that the user can observe.

[0091] Therefore, if the user is concentrating on a task at the site of interest, such as inserting a coil into an aneurysm or placing a stent, the tip of the guidewire or the tip of the guiding catheter may be out of the viewing angle of the X-ray image and disappear from the X-ray image, making it impossible for the user to observe it. In such a case, the tip of the device may not be noticed even if it has perforated a blood vessel. Perforation of a blood vessel is a serious complication that can be life-threatening. Perforation of a blood vessel is likely to occur at the tip of a device, especially at the tip of a guidewire or catheter. Therefore, in the present invention, the tip is particularly focused on. There may be cases where there are multiple tips in one screen, or where the tip of the device may perforate a blood vessel in a location other than the site of interest of the surgeon. In addition, in endovascular treatment of the brain, it is necessary to treat while viewing up to four multiple screens at the same time, and it is impossible for the surgeon to always pay attention to multiple tips on multiple screens. This support system compensates for this problem with technology such as AI. Therefore, an example of a condition set for the region of interest R used in the present invention is a condition regarding the distance between the tip of the guiding catheter or the tip of the guidewire and the edge of the X-ray image. Alternatively, the condition may be that the region of interest R has exceeded a specific range (for example, a boundary line specified by a user using a pointing device such as a mouse) specified on the X-ray image, or that the region of interest R has simply moved. That is, in some aspects of the present invention, the range of any region in the image may be treated as being similar to the edge, not limited to the edge of the image. The boundary line may be a straight line, a curved line, a circle, a rectangle, or another polygon. The boundary line may be moved, deformed, enlarged, or reduced by a user's operation. In some aspects of the present invention, the distance between the region of interest and the edge of the specific range may be displayed. The display mode of the distance may be changed depending on the size of the distance between the region of interest and the edge of the specific range. In some aspects of the present invention, the notification unit may notify the user on the condition that the value obtained by dividing the distance between the region of interest and the edge of the specific range by the moving speed of the region of interest in the image is less than a predetermined threshold. Some aspects of the present invention also include a function of making the specific range easier to see by superimposing it on the display, making it easier for the surgeon to make a decision.The range may be displayed not only by superimposition but also by arrows, but is not limited thereto. In some aspects of the present invention, the distance may be determined by either a straight-line distance or a distance along a blood vessel. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0092] Furthermore, instead of the region of interest, the position of an intravascular inspection or treatment device (also called a device of interest) at a certain time may be automatically or by the user (surgeon or assistant) to superimpose all or part of the device detected at that time on the subsequent real-time image. This allows the user (surgeon or assistant) to recognize the movement (deviation) of the device from a specific time point by superimposing it on the real-time image (Figure 21). The shape of the device may be detected automatically by image analysis, or by the surgeon marking it with a mouse or touch panel. Specific examples of devices to be superimposed include the tip of a guidewire, a marker, a stent, a guiding catheter, etc., but are not limited to these.

[0093] For example, assume that the position and shape of the guidewire tip (black part) at the time shown in the left diagram of Fig. 21 are stored. In this case, for example, when the user designates the guidewire tip with a pointing device such as a touch panel or a mouse, the guidewire tip is recognized and the stored guidewire tip is superimposed on an X-ray image obtained in real time thereafter, as shown in the center diagram of Fig. 21. This makes it possible to visualize and easily grasp how far the guidewire tip has moved since then, as shown in the right diagram of Fig. 21.

[0094] Therefore, in some aspects of the present invention, the image processing device includes a memory unit that acquires and stores the position and / or shape of an inspection or treatment device within a blood vessel at any point in time, and has a function of superimposing the stored position and / or shape of the device on images acquired thereafter.

[0095] For example, in carotid artery stenting, it is important to prevent distal embolism (cerebral infarction) that the filter or balloon placed in the distal part of the internal carotid artery is stable and does not move too much. If they or the tip of the guide wire attached to them go outside a certain range specified by the surgeon or automatically, a notification can be issued. In cerebral aneurysm embolization, if the tip of the guide wire inside the balloon catheter goes outside the specified range, there is a risk of the balloon slipping off, becoming unable to return it, or perforating the distal blood vessel, so a notification can be issued. Similarly, when inserting a coil, if it goes outside a certain area, it may occlude an important blood vessel, so a notification can be issued. If the coil deviates from the mask image (area) of the aneurysm, a notification can be issued because there is a possibility of aneurysm perforation due to the coil or guide wire catheter. It is important for the guiding catheter to be stable in all endovascular treatments, but there are times when you want to correct it before it disappears outside the frame, and if it goes out of a certain area, a notification can be issued. In tumor embolization, cerebral arteriovenous malformation embolization, dural arteriovenous fistula embolization, etc., embolization is performed using an embolic substance such as a liquid or particles. If the embolic substance exceeds a designated area, it may cause cerebral infarction in an important area, so a notification can be issued. Therefore, in some aspects of the present invention, the device includes a liquid embolic substance, a particle embolic substance, etc. In addition to the above examples, in endovascular surgery, care is taken to keep any device or embolic substance within a designated area, and this can be supported.

[0096] 4(a)-(b) are diagrams for explaining an example of conditions set for a region of interest R. In the example shown in FIG. 4(a), the region of interest R is set at the tip of a device D. In the example shown in FIG. 4(a), information W indicating the speed and acceleration of the region of interest R and the distance (number of pixels) between the region of interest R and an edge F of the X-ray image is displayed superimposed on the X-ray image. Note that FIG. 4(a) shows an example in which the device D is a guidewire. In some aspects of the present invention, a specific region designated in the X-ray image may be regarded as the same as the edge F of the X-ray image.

[0097] When a region including the tip of the guidewire is set as the region of interest R, the notification unit 113 notifies the user on condition that the region of interest R disappears from the X-ray image. In some embodiments, the X-ray image is an image showing a fixed region including the site of interest. Normally, during a procedure, the region displayed as the X-ray image is fixed. In addition, when a region including the tip of the guidewire is set as the region of interest R, the notification unit 113 notifies the user on condition that the distance between the region of interest R and the edge F of the X-ray image becomes less than a predetermined threshold distance.

[0098] Here, the "predetermined threshold distance" is a "reference distance for determining protrusion" provided for the notification unit 113 to determine whether or not there is a high probability that the tip of the device will deviate from the angle of view of the X-ray image. A specific value of the predetermined threshold distance may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, and the like, and is, for example, 5% of either the number of vertical pixels or the number of horizontal pixels of the X-ray image. This allows the user to receive a notification when the region of interest R, which is a region including the tip of the guiding catheter or the tip of the guidewire, approaches the edge F of the X-ray image, both before and after the region of interest R deviates from the edge F.

[0099] In addition, the magnitude of the movement of the device D in the blood vessel V can be a useful index for the user to predict the time until the tip of the device D reaches the edge F of the X-ray image. Specifically, the greater the speed or acceleration of the tip of D, the shorter the time until the tip of D reaches the edge F of the X-ray image. Therefore, when a region including the tip of the guiding catheter or the tip of the guidewire is set as the region of interest R, the notification unit 113 may notify the user on condition that at least one of the moving speed and the acceleration of the region of interest R in the X-ray image exceeds a predetermined threshold. In this way, when the moving speed or the acceleration of the region of interest R is greater than the threshold, the user's attention can be drawn before the region of interest R approaches the edge F of the X-ray image.

[0100] Furthermore, the distance between the tip of the device D in the blood vessel V and the edge F of the X-ray image is a useful index for the user to determine the likelihood that the tip of the device D will reach the edge F of the X-ray image. Therefore, as shown in information W in Fig. 4, the notification unit 113 may cause the display device 2 that displays the X-ray image to display the distance between the region of interest R and the edge F of the X-ray image.

[0101] Here, the "distance between the region of interest R and the edge F of the X-ray image" may be a distance when the region of interest R moves to the edge F of the X-ray image along the blood vessel V into which the device D in which the region of interest R is set is inserted. This can be realized by the distance measuring unit 115 extracting the blood vessel V using a blood vessel recognition engine generated in advance using a known machine learning method or the like, and measuring the distance from the region of interest R to the edge F along the blood vessel V. Alternatively, the distance measuring unit 115 may measure the above-mentioned distance based on the trajectory of the device D that has progressed through the blood vessel V. Specifically, when the user advances the guiding catheter through the blood vessel V, the tracking unit 112 tracks the tip of the guiding catheter and stores the trajectory in the storage unit 10. The distance measuring unit 115 may determine the length of the trajectory included in the region of interest R among the trajectories stored in the storage unit 10 as the above-mentioned length.

[0102] By having the notification unit 113 display the distance between the tip of the device D and the edge F of the X-ray image on the display device 2, the user can objectively grasp at a glance how far the device D must move before it reaches the edge F of the X-ray image. Furthermore, the notification unit 113 may change the display mode of the distance depending on the distance between the region of interest R and the edge F. For example, the shorter the distance, the larger the font size, the different color depending on the distance (blue → yellow → red), the region of interest may be enlarged depending on the distance, or the color or size of the mark added to the region of interest may be changed depending on the distance. By devising the display mode as described above, it is possible to make it easier for the user to notice the change in distance.

[0103] In the example shown in Fig. 4(b), the region of interest R set at the tip of the device D is closer to the edge F than in the example shown in Fig. 4(a). Therefore, the font of the information W indicating the speed and acceleration of the region of interest R and the distance (number of pixels) between the region of interest R and the edge F of the X-ray image is larger than in the example shown in Fig. 4(a).

[0104] The "distance between the region of interest R and the edge F of the X-ray image" may be the shortest distance between the region of interest R and the edge F of the X-ray image, or may be the length measured to the edge F of the X-ray image along the direction of movement of the region of interest R. In this case, the blood vessel extraction process by the distance measuring unit 115 can be omitted, which is advantageous in that the processing speed can be increased.

[0105] The notification unit 113 may notify the user on condition that the value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R in the X-ray image is less than a predetermined threshold. The value obtained by dividing the distance between the region of interest R and the edge F of the X-ray image by the moving speed of the region of interest R in the X-ray image can be said to be, so to speak, the expected time until the region of interest R reaches the edge F of the X-ray image. Therefore, the value of the "predetermined threshold" is a "grace period for determining whether or not the region of interest R is likely to fall outside the angle of view of the X-ray image" provided for the notification unit 113 to determine whether or not there is a high probability that the region of interest R will fall outside the angle of view of the X-ray image. The specific value of the grace period may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, and the like, and is, for example, 3 seconds.

[0106] Fig. 5 is a diagram showing an example of a message Ms notified by the notification unit 113. In the example shown in Fig. 5, the notification unit 113 displays a message indicating that the region of interest R will disappear from the X-ray image in three seconds (so-called frame-out) by superimposing it on the X-ray image. Furthermore, as in the example shown in Fig. 5, when the region of interest R approaches the edge F of the X-ray image, the notification unit 113 may change the shape of the region of interest R (changed to a circle in Fig. 5), increase the size of the region of interest R, or change the color of the region of interest R. This allows the notification unit 113 to easily recognize that the region of interest R is about to frame out.

[0107] FIG. 6 is a diagram for explaining another example of conditions set for the region of interest R. Specifically, FIG. 6 shows an example in which the device D is a guiding catheter. Unlike the guidewire shown in FIG. 4(a)-(b), the guiding catheter is different in that not only its tip portion but the entire device D may disappear from the X-ray image. However, whether the device D is a guiding catheter or a guidewire, it is the same in that its tip portion disappears from the X-ray image. Therefore, in the case of a guiding catheter, the region of interest R is set at the tip portion, as in the case of a guidewire.

[0108] In the above example, the threshold value is determined based on the distance between the region of interest R and the edge F of the X-ray image as the "predetermined threshold distance" mainly from the viewpoint of preventing the surgical tool from going out of the frame. However, the moving distance of the region of interest R itself may be focused on, and a notification may be issued to the user when the moving distance of the region of interest R is equal to or greater than a predetermined threshold. For example, when a stent is to be deployed with the tip of the catheter in the aneurysm, it is necessary to take care that the tip of the catheter does not go out of the aneurysm. However, since the surgeon is watching the stent at this time, he or she cannot follow the movement of the tip of the catheter with his or her eyes. In such a case, by configuring the system to notify the user when the moving distance of the region of interest set at the tip of the catheter exceeds a predetermined moving distance, the user can immediately notice the abnormality when the tip of the catheter is about to go out of the aneurysm. The threshold value of the moving distance may be set based on the position of the region of interest (e.g., the tip of the catheter) after the region of interest reaches a predetermined position (e.g., inside the aneurysm).

[0109] (Induction of embolization coils) Next, as another example of the condition set for the region of interest R, a condition related to a guiding catheter that guides a delivery wire of an embolization coil will be described.

[0110] Embolization, which fills the cerebral aneurysm with an embolization coil, is known as a catheter treatment for cerebral aneurysms. This treatment is a surgical procedure in which a guiding catheter is passed to the vicinity of the cerebral aneurysm, which is the target area, a delivery wire for an embolization coil is passed through the guiding catheter, and the embolization coil is separated and filled into the cerebral aneurysm, thereby blocking the inflow of blood into the cerebral aneurysm.

[0111] Figures 7(a)-(b) are schematic diagrams showing an example of an X-ray image of a guiding catheter that guides a delivery wire of an embolization coil, and are diagrams for explaining another example of conditions that are set for the region of interest R.

[0112] As described above, the X-ray imaging device 3 can irradiate the head of the subject P with X-rays at different angles of incidence. Figures 7(a) and 7(b) show examples of X-ray images taken by irradiating the head of the subject P with X-rays at different angles of incidence. The user performs embolization while viewing the two images shown in Figures 7(a) and 7(b).

[0113] In Fig. 7(b), the aneurysm is indicated by the symbol A. The user embolizes the aneurysm by placing multiple embolization coils E in the aneurysm A. Here, since the aneurysm A can be observed in the X-ray image shown in Fig. 7(b), the user focuses on one of the two X-ray images.

[0114] The delivery wire for carrying the embolization coil E to the aneurysm A is connected to the embolization coil E at its tip, and the user carries out the task of detaching the embolization coil E from the delivery wire after carrying the embolization coil E to the aneurysm A. As shown in Fig. 7(a)-(b), when the embolization coil E reaches the aneurysm A, the position of the embolization coil E being carried becomes unclear in the X-ray image due to other embolization coils E already placed in the aneurysm A. For this reason, the delivery wire for the embolization coil E is provided in advance with a marker L for timing the detachment of the embolization coil E. The user determines the timing of detaching the embolization coil E using the marker L provided on the delivery wire as a guide, not the embolization coil E itself in the X-ray image.

[0115] However, as shown in FIG. 7(a)-(b), an X-ray image in which the aneurysm A can be observed may differ from an X-ray image in which the marker L can be observed. Furthermore, even within the same screen, the aneurysm A and the marker L may be located far apart, making it difficult to observe the two locations at the same time. Therefore, a user of the image processing device 1 first sets a region of interest R in a part of the guiding catheter that guides the delivery wire of the embolization coil E. Specifically, the user sets the region of interest R at a position where the marker L and the region of interest R overlap at the timing when the embolization coil E is detached from the delivery wire.

[0116] The region of interest acquisition unit 111 receives a region of interest R that is set in a part of the guiding catheter that guides the delivery wire. The region of interest R can be specified using, for example, a pointing device such as a mouse, a touch panel, or the like. In FIG. 7(a), the region of interest R is indicated by a white star. The marker detection unit 114 detects markers L approaching the region of interest R among the markers L provided on the delivery wire of the embolization coil E for embolizing the aneurysm. In addition, the tracking unit 112 tracks the markers L detected by the marker detection unit 114.

[0117] When the marker L and the region of interest R overlap, the notification unit 113 notifies the user of the timing to cut and detach the embolization coil E from the delivery wire based on the positional relationship between the marker L and the region of interest R.

[0118] Figures 8(a)-(d) are diagrams for explaining the timing of detaching the embolization coil E. The device D shown by the dashed line in Figures 8(a)-(d) is a delivery wire. As shown in Figures 8(a)-(d), a marker L is attached to a certain section of the delivery wire. Note that Figures 8(a)-(d) show a typical example of the marker L. Other examples include a dashed dotted line "- - -" or a long straight line.

[0119] When the user moves the delivery wire, the marker L also moves in conjunction with the movement of the delivery wire in the X-ray image. On the other hand, the guiding catheter for guiding the delivery wire may move slightly due to friction or the like in response to the movement of the delivery wire, but the amount of movement is small compared to the movement of the delivery wire. Therefore, the user sets in advance the region of interest R at the position of the guiding catheter that corresponds to the position where the marker L should be when the embolization coil E reaches the aneurysm A.

[0120] As shown in Figures 8(a)-(d), the marker L is present over a certain section on the wire. When the user moves the delivery wire, the tip of the marker L comes into contact with the region of interest R, as shown in Figure 8(b). This indicates that the embolization coil E has approached the aneurysm A.

[0121] 8(c), the marker L and the region of interest R overlap each other. When the marker L and the region of interest R overlap each other, the notification unit 113 starts an operation of notifying the user of the timing to disconnect and detach the embolization coil E from the delivery wire.

[0122] Specifically, as shown by symbol W2 in Figure 7(a), when the marker L and the region of interest R overlap, the notification unit 113 causes the display device 2 to display information W2 indicating the distance the marker L must move before the embolization coil E is disconnected from the delivery wire.

[0123] 8(d), the user sets the region of interest R so that the end of the marker L passes through the region of interest R just when the embolization coil E reaches the aneurysm A. In this case, the information W2 displayed on the display device 2 by the notification unit 113 indicates the distance until the end of the marker L passes through the region of interest R.

[0124] When the marker L passes through the region of interest R, the notification unit 113 further notifies the user of that fact. This allows the user to notice the timing to disconnect the embolization coil E from the delivery wire even when the aneurysm A is focused on the captured image as shown in FIG. 7(b).

[0125] (Device D shape) Next, as yet another example of a condition set for the region of interest R, a condition regarding the shape of a device D (for example, a guidewire) will be described.

[0126] When a feature amount indicating the shape of device D included in region of interest R satisfies a predetermined condition, notification unit 113 notifies the user of image processing device 1. Specifically, notification unit 113 notifies based on a feature amount indicating the curvature of device D included in region of interest R or the "deflection" of device D included in region of interest R.

[0127] When the user attempts to advance device D with the tip of device D caught on a blood vessel wall or the like, the tip of device D is bent. In general, when the tip of device D is bent, elastic energy is accumulated in that portion. The more the tip of device D is bent, that is, the greater the curvature of device D becomes (the smaller the radius of curvature becomes), the more elastic energy is accumulated. When the amount of accumulated elastic energy becomes large, the elastic force of device D releases the tip from being caught, and the tip may move at high speed. As a result, the tip of device D may suddenly disappear from the X-ray image.

[0128] Therefore, the notification unit 113 notifies the user on the condition that the curvature of the device D included in the region of interest exceeds a predetermined threshold curvature, or the tip does not move even though the curvature changes. It may also be considered that the tip of the device D is immobile, or that the moving distance is lower than a certain threshold. Here, the "predetermined threshold curvature" is a "reference curvature for notification determination" provided for the notification unit 113 to determine whether or not the tip of the device D is likely to move at high speed. A specific value of the predetermined threshold curvature may be determined by experiment in consideration of the frequency of notification by the notification unit 113, usability, the material, size, elasticity coefficient, etc. of the device D.

[0129] 9(a)-(c) are diagrams for explaining conditions related to the shape of the device D set in the region of interest R. Specifically, FIG. 9(a)-(b) are diagrams for explaining notification based on the curvature of the device D. In FIG. 9(a), the region of interest R is indicated by a dashed circle. As a non-limiting example, the region of interest R in FIG. 9(a) is a circle with a radius of 1 cm centered on the tip of the device D. Also, FIG. 9(b) is a histogram showing the distribution of the curvature of the device D included in the region of interest R. Specifically, FIG. 9(b) shows the distribution of the radii of curvature obtained by dividing the device D included in the region of interest R into a plurality of small regions and calculating the radius of curvature of the device D in each small region.

[0130] The notification unit 113 notifies the user when a feature amount (for example, a statistic such as the average value, mode, or median value of the curvature) calculated from the distribution of the curvature in a histogram showing the distribution of the curvature of the device D exceeds a predetermined threshold curvature, or when the tip does not move despite a change in the curvature. This allows the image processing device 1 to provide an opportunity for the user to notice that an elastic force is being accumulated in the device D.

[0131] Furthermore, the notification unit 113 may notify the user on the condition that the value obtained by subtracting the length of the center line of the blood vessel V included in the region of interest R from the length of the device D included in the region of interest R exceeds a predetermined threshold length (threshold value). Fig. 9(c) is a schematic diagram showing the relationship between the length of the device D included in the region of interest R and the length of the center line of the blood vessel V included in the region of interest R. Although the device D and the blood vessel are curved inside the body of the subject P, for convenience of explanation, the device D and the blood vessel V are displayed as straight lines in Fig. 9(c). Also, in Fig. 9(c), the center line of the blood vessel V is shown by a dashed line.

[0132] The distance measuring unit 115 uses a blood vessel recognition engine to extract a blood vessel V in the region of interest R and traces its centerline to obtain a length D1. Similarly, the distance measuring unit 115 uses a device recognition engine generated using a known machine learning method or the like to extract a device D and obtains its length D2.

[0133] Generally, when a user advances device D into blood vessel V, device D will advance in a serpentine manner along the wall of blood vessel V. Therefore, length D2 of device D in blood vessel V will be longer than length D1 of blood vessel V (length of center line of blood vessel V), and device D will bend in blood vessel V, which means that elastic energy is accumulated in device D. If the amount of this bending becomes large, some trigger may release the elastic energy, causing device D to move significantly. As a result, the tip of device D may disappear from the X-ray image.

[0134] The difference length B, which is the length obtained by subtracting the length D1 from the length D2 calculated by the distance measuring unit 115, can be an index showing the amount of deflection of the device D in the blood vessel V. Therefore, the notification unit 113 displays the length B obtained by subtracting the length D1 from the length D2 calculated by the distance measuring unit 115, and notifies the user when the length B is longer than a predetermined threshold length. This allows the image processing device 1 to provide an opportunity for the user to notice that an elastic force is being accumulated in the device D.

[0135] (Estimation of catheter tip position using 2nd marker) Aneurysm embolization catheters are marked with a tip marker (1st marker) and a second marker (usually 3 cm from the tip). It is important for the surgeon to know where the tip of the catheter is inside the aneurysm in order to perform the surgery safely, but if a coil enters the aneurysm, it becomes difficult to determine the location of the tip marker, reducing safety (see Figure 17). For example, if the tip marker moves to the back of the aneurysm, the tip or the coil coming out of the tip may perforate the aneurysm, leading to a serious complication called subarachnoid hemorrhage. Conversely, if the tip marker is about to come out of the aneurysm, the catheter or coil may deviate from the aneurysm and must be reinserted into the aneurysm, which carries the risk of perforating the aneurysm wall.

[0136] For example, in FIG. 17, the left diagram shows a state in which a microcatheter is inserted into an aneurysm. The size of the aneurysm can be, for example, 10 mm. The distance between the 1st marker and the 2nd marker attached to the microcatheter is constant (usually 30 mm). For example, assume that the positions of the 1st marker and the 2nd marker are recorded in this state (stored positions). The center diagram in FIG. 17 shows a state in which the position of the microcatheter has moved. When a coil is inserted into the aneurysm, the position of the 1st marker becomes invisible or difficult to see. Therefore, the position of the 1st marker is estimated from the difference between the position of the 2nd marker at this time and the position of the 2nd marker previously stored. In this case, the 1st marker is predicted to be approximately 0 mm from the neck of the aneurysm (predicted position A). The right diagram also shows a state in which the position of the microcatheter has moved. Similarly, the position of the 1st marker is estimated based on the movement distance of the 2nd marker. In this case, it is estimated to be 8 mm from the neck of the aneurysm.

[0137] Some aspects of the present invention relate to a method for estimating a tip position from the movement of a second marker when the position of a catheter tip is unknown because it is inside a coil (aneurysm). More specifically, the method includes a step of storing a positional relationship between the catheter tip and the second marker (e.g., 3 cm apart), a step of storing a distance a between the neckline of the aneurysm and the first marker and the position of the second marker at time t1, a step of calculating a moving distance b from the position of the second marker at time t2, a step of estimating a distance ab from the neckline of the aneurysm of the catheter tip, and a step of notifying a user of the estimated distance. Here, the catheter tip, the second marker, and the neckline of the aneurysm may be automatically detected by image recognition by a computer. Alternatively, they may be manually specified by a pointing device such as a mouse or a touch panel. The estimated distance ab may be displayed on a display device. Furthermore, the position of the catheter tip estimated based on the estimated distance ab may be displayed on a display device. An arbitrary threshold may be set and a notification may be made if the distance deviates from the threshold, or the speed and acceleration may be calculated and notified based on the value, in addition to the distance (rapid or large movement is likely to be dangerous). This moving distance is a straight line distance or a distance along the curve of the catheter. Also, since the shape of the curve of the catheter may change, the length of the curve may be used. Furthermore, the distances a, b, and ab may be a probability distribution due to changes in the shape of the catheter. For example, if there are multiple times of initial memorization, a distribution of the distance a can be obtained, so the location can be estimated using the average or variance, and it can be predicted as the most likely point, or it can be displayed as a probability distribution using a heat map, etc. Therefore, the estimated distance may be represented by a probability distribution, and the estimated position of the catheter tip may be displayed based on the probability distribution by coloring it like a heat map.

[0138] In addition to measuring the distance traveled, the position of the second marker can be specified by the surgeon or recognized by a computer, and the position can be indicated on the display device by a semi-transparent overlay or an arrow, etc. From this fixed display, the surgeon and assistant can visually recognize whether the tip marker has advanced or been pulled back by recognizing how far the current second marker has shifted.

[0139] Some aspects of the present invention relate to a program for executing the above method on a computer. Some aspects of the present invention relate to an image processing device for executing the above method and its operating method. Such an image processing device may include, for example, a positional relationship storage unit for storing the positional relationship between the tip of the catheter and the 2nd marker (e.g., 3 cm apart), a position storage unit for storing the distance a between the neckline of the aneurysm and the 1st marker and the position of the 2nd marker at time t1, a distance estimation unit for calculating the moving distance b from the position of the 2nd marker at time t2 and estimating the distance ab from the neckline of the aneurysm of the tip of the catheter, and a notification unit for notifying a user of the estimated distance. Furthermore, some aspects of the present invention relate to a cerebral aneurysm coil embolization assistance system including the above image processing device and an image capturing device for capturing an X-ray image of a patient (or subject P) with a guiding catheter and a delivery wire of an embolization coil inserted in a blood vessel and transmitting the image to the image processing device. Note that, in some aspects of the present invention, instead of or in addition to the notification unit, an output unit for outputting information to a device connected to the image processing device may be present.

[0140] <Process flow for estimating catheter tip position using the 2nd marker> 22 is a flowchart for explaining the flow of the catheter tip position estimation process using the second marker executed by the image processing device according to the embodiment. The process in this flowchart is started, for example, when the image processing device 1 is started, or when the user or the image processing device 1 determines that it is necessary to start the process.

[0141] First, the image acquiring unit 110, the region of interest acquiring unit 111, and possibly the tracking unit 112 function to detect and track an inspection or treatment device D in a blood vessel V by video analysis. Then, the positions of the first marker and the second marker at that time (T=t1) are stored (S102) automatically by image analysis or by designation by the user (operator).

[0142] Furthermore, the aneurysm neck line is determined automatically or by user (operator) designation (S104). Then, the distance A between the aneurysm neck line and the first marker is calculated (S106). At this time, the straight or curved distance A between the aneurysm neck line and the first marker (usually inside the aneurysm) can be measured.

[0143] Next, the second marker is continuously tracked and the moving distance b is measured (S108). The second marker is continuously tracked by image analysis, and the moving distance b (straight line or curved distance) of the second marker at T=t2 is measured. This moving distance may have a directional property (plus / minus).

[0144] Then, based on the distances A and b, the distance of the first marker from the neck of the aneurysm is estimated (S110). For example, the distance of the first marker from the neck of the aneurysm is estimated to be Ab. Alternatively, it may be estimated that the first marker is located at a position that is moved by b in the direction perpendicular to the neck of the aneurysm from the position of the first marker at T=t1.

[0145] Then, the estimated position of the first marker is displayed (S112). For example, the estimated position of the first marker at T=t2 or the distance from the aneurysm neck can be displayed as a superimposed display or a numerical value. If the first marker is likely to deviate from the aneurysm near the aneurysm neck or is likely to hit the aneurysm wall deep inside the aneurysm, the user (surgeon / assistant) may be additionally notified of this. If the analysis is difficult or if the user (surgeon) specifies otherwise, this function can be temporarily stopped. After this process is completed, the process in this flowchart may be restarted as necessary.

[0146] (Replay function) Since medical personnel pay attention to a specific location during treatment, if the medical personnel is not paying attention to the device that issued the warning or is looking at another screen when the system according to the present disclosure issues a notification, the medical personnel will have to try to understand the situation after the warning is issued. However, since real-time images are updated every moment, it is often difficult to confirm the details around the time the warning was issued. In addition, depending on the nature of the warning, it is necessary to understand the details of the device's operation itself, the difference between the time the warning was issued and the current time, the elapsed time since the warning was issued, etc.

[0147] For example, for the tip of a guidewire, it is desirable to obtain detailed information on the movement of the tip. For example, rapid large movements increase the risk of blood vessel perforation. In the case of loss of a guiding catheter, it is desirable to obtain information on the position and elapsed time at the time when it went outside the image. In the case of a coil detector, it is desirable to obtain information on its deviation from the optimal position.

[0148] In some embodiments of the present invention, the image processing device has a function of storing video and recognition information and providing information as necessary so that the surgeon can review it when necessary (i.e., a replay function). FIG. 12 shows the configuration of an image processing device further including a video recording unit (which can be integrated with 10) that stores images obtained from an image acquisition unit 110, and a video extraction unit 118 that extracts a part of the video before and after the time when the notification occurred. FIG. 13 shows an example in which the replay video when the notification occurred is cropped and enlarged around the area of ​​interest. FIG. 14 shows an example in which a (enlarged) replay playback window is displayed on a real-time display screen. In this example, the replay is enlarged and displayed in a part that is as far removed from the area of ​​interest as possible, and this makes it possible to determine what happened in the part where the warning was issued within the limited range of one screen.

[0149] In some embodiments of the present invention, the image processing device further includes a video storage unit 117 that stores images (including video) obtained from the image acquisition unit over time (continuously). In some embodiments of the present invention, the image processing device further includes a video extraction unit 118 that extracts video from the video storage unit 117 for a certain period before and after the notification unit issues a notification. The video extraction period and playback speed may be automatically determined based on at least one of the moving distance, moving speed, and acceleration of the region of interest when the notification occurs. Note that, when extracting, information obtained by the region of interest acquisition unit, tracking unit, notification unit, marker detection unit, and / or distance measurement unit may also be used. Note that, in some embodiments of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0150] In some aspects of the present invention, the image processing device can display the extracted image on the display device. The extracted image may be automatically and repeatedly displayed a predetermined number of times. The extracted image may also be displayed based on any operation including play, stop, fast forward, rewind, frame by frame, slow playback, and double speed playback. This allows the user to easily check the image. The display device may further display the elapsed time from the time the notification occurred, a comparison of the position of the region of interest after the time the notification occurred and an arbitrary time has elapsed (comparative display includes, for example, the relevant region, the difference in the detected position, and the alignment of the image itself), or the trajectory of the region of interest acquired by the tracking unit, superimposed on the extracted image.

[0151] In some aspects of the present invention, the image processing device can cut out a part of the image near the region of interest from the extracted image and display it. The extracted image can be displayed at a position that does not obstruct the display of the region of interest. The extracted image may be displayed in an enlarged form.

[0152] In some aspects of the present invention, the image processing device can display the extracted image simultaneously with the occurrence of the notification or after a predetermined time has elapsed since the occurrence of the notification. In some aspects of the present invention, the image processing device can display images captured from multiple directions simultaneously.

[0153] The replay display may be used at a time other than when a notification is generated. In other words, the video extraction unit 118 may extract from the video storage unit 117 not only video for a certain period before and after the notification unit issues a notification, but also video for any time or period. For example, when the user (surgeon) feels it is necessary, by specifying any region of interest, the previous scene can be viewed in a replay display. This allows the user to grasp and compare what happened in the region of interest while viewing the real-time display.

[0154] Some aspects of the present invention relate to a program for executing the above method on a computer, and also relate to an image processing device for executing the above method and a method of operating the same.

[0155] <Replay display function processing flow> 23 is a flowchart for explaining the flow of processing of the replay function executed by the image processing device according to the embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0156] First, the image acquiring unit 110, the region of interest acquiring unit 111, and possibly the tracking unit 112 function to detect and track an inspection or treatment device D within a blood vessel V by video analysis (S202).

[0157] Next, it is determined whether or not the notification condition (such as the movement distance exceeding a threshold) is satisfied (S204), and if the condition is satisfied, a notification is issued and a replay video before and after the notification condition is satisfied is displayed in part of the display screen (S206). At this time, the real-time image is displayed as usual, and the replay video may be displayed several times so as not to overlap with the area of ​​the notification condition, or may be displayed repeatedly until the user (surgeon / assistant) desires (S208). When the repetition is completed, the replay video screen is closed. After completion, the processing in this flowchart may be restarted.

[0158] (Estimating the location of the region of interest outside the frame) It is dangerous for a region of interest such as the tip of a guidewire or a guiding catheter to move out of the range of the X-ray field angle (frame-out), but the degree of danger varies depending on the amount of movement. Specifically, for example, if the tip of the guidewire is only slightly (e.g., within 5 mm) out of the frame, the possibility of blood vessel perforation is low, but if it is significantly out of the frame (e.g., 20 mm or more), the risk of blood vessel perforation is high. When the region of interest is out of the frame, it is necessary to pull it back into the field angle of the X-ray image, but depending on the situation, it may not be possible to deal with it immediately. In such a case, it is important to know how far the out-of-frame region of interest has moved out of the range of the X-ray field angle and how dangerous the movement is. Therefore, some aspects of the present invention relate to an apparatus for estimating and displaying the position, speed, and acceleration state of the out-of-frame region of interest.

[0159] The estimation of the position of such a region of interest outside the frame can be performed by, for example, an image processing device, which includes an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each region of interest, and a notification unit that notifies the user on the condition that the region of interest disappears from the image when a region including the tip of the catheter or the tip of the guidewire is set as the region of interest, and a state estimation unit 119 that estimates the current position and / or speed of the tip of the catheter or the tip of the guidewire that has disappeared from the image based on the position, speed and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappears from the image (FIG. 15). The current position and / or speed of the region of interest estimated by the state estimation unit 119 can be displayed on a display device. In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0160] Here, in order to estimate the position of the frame-out region of interest, the state estimation unit 119 stores the outputs from the region of interest acquisition unit, the marker detection unit, and the tracking unit in chronological order from before the frame-out, and can calculate the state of the region of interest, such as the position, speed, and acceleration, using the stored outputs. Then, when the region of interest is framed out and cannot be tracked on the screen, the state estimation unit 119 estimates the position, speed, and the like of the region of interest from the state before the frame-out and notifies the user. The estimation method at that time includes, but is not limited to, a learning-based method using deep learning (CNN, RNN, WaveNet, etc.) and a Bayesian estimation method (Kalman filter, extended Kalman filter, ensemble Kalman filter, particle filter, etc.).

[0161] In addition, some aspects of the present invention relate to a device that calculates a risk level from an estimated state and issues a notification to a user according to the risk level. The position, speed, and risk level of the region of interest estimated by the state estimation unit 119 may be displayed on a display device. The display method may be, but is not limited to, a display using a point, an arrow, a heat map, or the like. For example, if the position of the estimated region of interest is a predetermined distance or more away from the edge of the image, the risk level may be determined to be high (FIG. 16). For example, in FIG. 16, the position of the estimated region of interest is shown by a circle, and the risk level is determined to be higher as the position is estimated to be farther away from the edge of the screen. In this case, for example, the risk level may be shown by the color of the circle (for example, green indicates low risk and red indicates high risk). If the calculated risk level exceeds a certain threshold, an alert may be displayed on the screen of the display device. Alternatively, a notification may be given by voice.

[0162] Therefore, some aspects of the present invention also relate to an image processing device, comprising: an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit for acquiring one or more regions including at least a part of the device included in the image as a region of interest; a notification unit for notifying a user of the image processing device when at least one of the regions of interest satisfies a condition determined for each region of interest, the notification unit for notifying the user of the region of interest when a region including a tip of a catheter or a tip of a guidewire is set as the region of interest, on the condition that the region of interest disappears from the image; and a state estimation unit 119 for estimating a current position and / or speed of the tip of the catheter or the tip of the guidewire that has disappeared from the image based on the position, speed and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappears from the image, and a warning is notified to the user when the current position and / or speed of the region of interest estimated by the state estimation unit 119 exceeds a predetermined threshold. This image processing device may further comprise a tracking unit for tracking each of the regions of interest in the image. In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0163] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0164] Some aspects of the present invention relate to a program for executing the above method on a computer, and some aspects of the present invention relate to an image processing device that executes the above method and a method of operating the image processing device.

[0165] <Process flow for estimating the position of a region of interest outside a frame> 24 is a flowchart for explaining the flow of processing for estimating the position of a region of interest outside a frame, which is executed by an image processing device according to an embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0166] First, the image acquiring unit 110, the region of interest acquiring unit 111, and possibly the tracking unit 112 function to detect and track an inspection or treatment device D within a blood vessel V by video analysis (S302).

[0167] Next, it is determined whether the part of device D to be notified has gone outside the frame of the screen (S304), and if the condition is met, the position of the device outside the frame is estimated (S306). For example, the position outside the frame of device D is estimated from the front side of the black part of the guide wire visible on the screen connected to device D, the filter, the balloon, etc. For estimation, distance difference, machine learning, etc. can be used.

[0168] Next, the estimated device position is displayed (S308). The estimated position of device D that has gone outside the frame is displayed on the display screen as the actual estimated position or the distance from the frame. The further outside the frame the device is, the stronger the display or notification may be.

[0169] Finally, it is determined whether the guess is no longer necessary because the device D has returned to the screen frame, or the user (surgeon / assistant) has pressed a button for not needing to display, etc. (S310). After this is completed, the process in this flowchart may be restarted.

[0170] (Layout / Notifications) In some aspects of the present invention, the results of image analysis can be displayed on two screens of different sizes. As described above, in endovascular surgery, surgery is generally performed while viewing multiple screens (for example, four screens), and the surgeon grasps three-dimensional information by viewing at least two screens (generally the front (AP direction or F direction: Anterior-Posterior, Frontal) and the side (RL direction or LAT direction: Right to Left, Lateral)). Therefore, it is important to display images from the front and the side, and it is very important to display them easily on a monitor of a physically limited size. In actual surgery, the monitor is often placed more than 1m away from the patient's bed, and instructions are often given to move the monitor closer to the surgeon even by 1cm. The angles of the front and the side move three-dimensionally, and for example, the front tube is always used to view from an oblique angle, with an angle of 15 degrees to the right and 10 degrees to the head. As a result, even when three-dimensional images are projected into two dimensions, the images are viewed at an angle that makes them easy to understand even in two dimensions. Furthermore, in the case of four screens, the images are Live and Mask on the front, and Live and Mask on the side. Live is a normal fluoroscopic image, similar to a general X-ray, and is viewed in real time. Mask takes the difference (subtraction) between the image and any past Live image selected by the surgeon. As a result, bones visible in Live disappear, and only blood vessels and devices visible with contrast agent are visible, providing an image that is easy for the surgeon to understand.

[0171] The two screens of different sizes can be switched automatically or by the user (surgeon) selection (see Figure 18). Since there is a physical limit to the screen size and it may be desired to view one of them larger, visibility can be improved by displaying the two screens at different sizes as shown in Figure 18.

[0172] Furthermore, the screen to be notified of can be made known to the user by lighting up the frame of the screen, changing the color, or highlighting it (see FIG. 19). As shown in FIG. 19, by notifying the user with a colored frame or the like, it becomes easier for the user to know which screen to pay attention to. When a notification is issued on the smaller screen, the two screens may be automatically swapped (so that the part to pay attention to is switched to the larger, easier-to-see screen). Thus, in some aspects of the present invention, the display device may have a function of calling the user's attention by lighting up, changing the color, or highlighting the frame of one of the two screens.

[0173] (Probability-based display) In some aspects of the present invention, since the area of ​​interest within the screen is output as a probability distribution, it is also possible to express the existence of the area of ​​interest in terms of probability. The probability distribution of the existing area may be displayed as a numerical value, a color, a bar, or the like. Furthermore, it is also possible to express whether or not a scene requires notification as a probability. Whether or not a scene requires notification may be displayed as a numerical value, a color, a bar, or the like according to the probability. Furthermore, which part of the screen is the responsible part may be displayed as a heat map or the like as a probability distribution. The probability distribution may be displayed in a more easily understandable way. For example, 0-30% may be displayed as low, 30-70% as middle, and 70-100% as high, or it may be displayed in three colors. As another example, the area below 70% may be slightly darkened so that the area of ​​interest or the area to be notified appears bright like a spotlight.

[0174] Therefore, in some aspects of the present invention, the notification unit can display, on the display device that displays the image, a numerical value, color, bar, or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a numerical value, color, bar, or heat map based on a value obtained by applying an arbitrary transformation to the probability distribution. The notification unit may also color the regions of interest with a color or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition defined for each region of interest, or a color or heat map based on a value obtained by arbitrarily transforming the probability distribution, and display the color or heat map on the display device that displays the image, or replace the probability of satisfying the condition with a numerical value or color and display the color or heat map on the display device that displays the image.

[0175] (Device selection display and recording) There are various devices used in endovascular treatment, and each of them has many types. Examples of devices include various catheters, balloon catheters, guidewires, stents, flow diverter stents (fine mesh stents), coils, embolic materials (liquids, particles, etc.), and other embolic devices (WEB, etc.). There are also various types of each device. For example, catheters have standards for tip shape, length, inner lumen, outer lumen, and hardness. Coils have standards for manufacturer, thickness, total length, diameter, hardness, and there are hundreds of types. It is impossible to remember all of these, and it is not known whether they are in stock, so the surgeon checks with the vendor during the operation and proceeds with the treatment. Combinations are also important, and the size of the coil is determined by looking at the image. For an aneurysm, about 5 to 15 coils are usually used, so it is necessary to consider which one to use next, but it is difficult for the surgeon and assistant to remember the inventory management and lineup. It is difficult to grasp the situation because new products are released and old products are discontinued, and the lineup varies depending on the facility. Currently, devices are selected by communicating with providers during treatment, but the process is not smooth.

[0176] First, create a database of the lineup and specifications of devices currently available in the country. It is also possible to enter stock information at each facility. A system is constructed to display this information on a monitor. For example, when a microcatheter is selected, a list of the lineup, specifications, and stock information of available microcatheters is displayed. Additional information, such as whether the device or multiple devices can be placed in the guiding catheter, can also be searched. For example, when the lumen of the guiding catheter is R and the outer diameters of the two devices are r1 and r2, if R>r1+r2, the two devices can be placed in the guiding catheter. A similar approach can be used for one device or three or more devices.

[0177] As another example, in the case of coils, the coil lineup (length, diameter, hardness, shape, etc.) is selected depending on the size of the aneurysm and the behavior of the coil previously inserted. There are hundreds of coil lineups, but it is easy to list them by specifying the length, etc. By displaying them on the monitor, it becomes easier to select during treatment. Since most coil selections involve the same or smaller diameter or length, it is possible to present a lineup of coils that are likely to be used based on information on coils used up to that point. It is also possible to present coils that take stock into consideration. In addition, it is possible to include preferences according to facilities and surgeons (users) in the lineup candidates. Image analysis may be used to suggest options according to the aneurysm and the coil winding method. The surgeon selects the desired device from such a list and performs the surgery. This information can be recorded and automatically created as a surgical record by combining it with snapshots of the treatment video.

[0178] In some aspects of the present invention, a display device that displays images can display a product list of intravascular inspection or treatment devices (e.g., various catheters and coils) (see FIG. 20). The display device may also display a product list filtered by size or availability. Furthermore, the display device may display a list of recommended products based on image analysis results, facility information, or user preference information.

[0179] In some aspects of the invention, the image processing device can automatically or based on user selection create a surgical record that includes information on the devices used, the images acquired, and the results of image analysis.

[0180] Therefore, some aspects of the present invention also relate to an endovascular surgery support system that includes a memory unit that stores a product list of intravascular inspection or treatment devices (e.g., various catheters and coils), a recommendation unit that recommends products to be used based on image analysis results, facility information, or user preference information, and a display unit that displays the recommended products.

[0181] (Recognizing a new device on the screen) For example, when inserting a device for the treatment of a cerebral aneurysm, the surgeon does not know when the device will arrive within the X-ray screen, so he or she must continue to emit X-rays and watch the screen while moving the device. Usually, the surgeon moves the device quickly until the device becomes visible, and then slowly after it becomes visible. In this case, if the appearance of the device is overlooked and the speed cannot be reduced, there is a risk of blood vessel perforation or dissection. Until the device is visible, X-rays are emitted in a situation where the screen does not change, which leads to increased exposure of the patient, surgeon, assistant, and medical staff. Therefore, in some aspects of the present invention, the image processing device can notify the user (surgeon) on the condition that a newly appeared device in the image, for example, within the edge or within a certain distance from the edge or within a specific area in the image, is newly detected as a region of interest (FIG. 25). More specifically, some aspects of the present invention relate to an image processing device including an image acquisition unit that acquires an image for intravascular inspection or treatment, a region of interest acquisition unit that acquires one or more regions including at least a part of a device for intravascular inspection or treatment as a region of interest when the image includes the device, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each region of interest, the notification unit notifying the user on condition that a newly appeared device or a part of the device in the image or a specific region is newly detected as the region of interest, thereby reducing the burden on the surgeon when performing intravascular treatment.

[0182] In addition, the device appears on a maximum of four screens, the front live image, the side live image, the front mask image, and the side live image. The notification may be made on all of them, or on the first one. For example, the timing at which the device appears from the bottom is often different in the front image and the side image. In this case, if the notification is made on both, the notification will be made continuously, which is uncomfortable for the user. Therefore, the notification may be made when the device appears on one screen, and the notification may not be made even if the same device appears on the other screen after that. The front or side live image and the mask image are often the same size, but may be enlarged in the mask image. In this case, since the device first appears in the live image, the notification may be made on the live image, and then the notification may not be made even if the device appears in the mask image. In this way, it may be possible to set whether to notify the device appearing on the screen only once or multiple times for multiple screens according to the surgeon's preference. Furthermore, since the edge may change due to narrowing the X-ray, for example, when the edge button is pressed, the border may be automatically set to the X-ray range (Figure 41). If the scene temporarily changes due to contrast imaging or the like, the change may be detected and a notification may not be issued. Notification may also be suppressed depending on the image quality. For example, if the mask image is shifted and difficult to see, or if the entire screen is moving due to the patient's body movement, notification may be temporarily suppressed.

[0183] In addition, in some aspects of the present invention, the image processing device may further include a tracking unit that tracks the region of interest in the image. Here, the notification unit can notify the user on the condition that a device that has newly appeared within an edge or within a certain distance from the edge in the image or within a specific region is newly detected as a region of interest. The edge or within a certain distance from the edge in the image refers to, for example, a region within 10%, 8%, 5%, or 3% of the overall length in the vertical or horizontal direction as viewed from the edge of the image, but is not limited thereto. The specific region may be specified by the user or automatically specified by the image processing device. The user can specify the region using a pointing device such as a mouse or a touch panel. The specific region may also be a region that the image processing device automatically determines to be a region to be confirmed. The area to be confirmed may be determined based on, but is not limited to, for example, that the tip of the guidewire is within a certain range, that the tip of the guiding catheter is within a certain range, that the tip of the stent delivery wire is within a certain range, that the coil marker is within a certain range, that the 2nd marker is within a certain range, the range in which the 2nd marker of the catheter moves to see when it is OK to cut the coil, that the filter is within a certain range, that the tip of the catheter is within a certain range, that the embolic material is within a certain range, the range in which the device remains inside a large blood vessel, an aneurysm, or an important blood vessel. For example, the automatic determination of the area to be confirmed may be performed based on, but is not limited to, a model created by machine learning. The area may be manually specified by the user, or may be specified automatically or semi-automatically. For example, when the 2nd marker of the catheter is recognized and the coil enters the aneurysm or the coil marker is detected, the area is automatically set to surround the 2nd marker, or the area is proposed, and the area is confirmed when the user clicks OK (Figure 35). Alternatively, for example, a rectangular area of ​​constant a × constant b is automatically specified at the tip of the guidewire so that the guidewire is in the center.It may be determined automatically, or it may be determined by the user clicking OK, and the length and width may be corrected as necessary, or may be corrected by translation, etc., and then the constant a / b may be changed according to the case, magnification rate, and the surgeon's preference. Furthermore, the region is not limited to a rectangle, but may be a circle, an ellipse, an arbitrary closed curve, a straight line, or an arbitrary curve. In the case of a straight line or an arbitrary curve, it is determined that the region is divided there. In the case of a straight line, the screen is extended and divided into two regions. In addition, the region may be divided along the blood vessel by a straight line or an arbitrary curve. In other words, the region may be divided according to whether it is distal or proximal to the region drawn along the blood vessel by a straight line or a curve. These are specific examples and are not limited to these. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present. Specifically, for example, the output may be sent to a catheter surgery robot via the output unit, and the surgeon may be notified via the robot, or the device may be automatically stopped from moving forward, or the device may be inserted and automatically sent out until it appears on the screen, and when the device appears according to the output of the output unit, the sending may be stopped or the speed may be slowed down. As another example, the device may be connected to AR / VR / MR, and when the output of the device's appearance is received from the output unit, it may be displayed or notified on the AR / VR / MR. Of course, this is not limited to this example, and all output to any other device is included.

[0184] (Recognizing devices that enter a specific area) For example, when inserting a coil into an aneurysm, the guiding catheter or intermediate catheter may move, the stent may move, the balloon may move, and the tip of the guide wire that guides the balloon may move, but the surgeon does not notice these movements or changes because he or she is mainly looking at the aneurysm. In some aspects of the present invention, the image processing device can notify the user (surgeon) on the condition that a device that has newly appeared in an image, for example, within a specific region, is newly detected as a region of interest (FIG. 37). More specifically, some aspects of the present invention relate to an image processing device that includes an image acquisition unit that acquires an image for intravascular inspection or treatment, a region of interest acquisition unit that acquires one or more regions including at least a part of the device as a region of interest when the image includes a device for intravascular inspection or treatment, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each of the regions of interest, and the notification unit notifies the user on the condition that a newly appeared device or a part of the device in the image or within a specific region is newly detected as a region of interest. This reduces the burden on the surgeon when performing endovascular treatment. In addition, in some aspects of the present invention, the image processing device may further include a tracking unit that tracks the region of interest in the image. Here, the notification unit can notify the user on the condition that a device that has newly appeared within an edge of the image, within a certain distance from the edge, or within a specific region is newly detected as the region of interest.

[0185] The specific region may be specified by a user or automatically by an image processing device. The user may specify the region using a pointing device such as a mouse or a touch panel. The specific region may be a region that the image processing device automatically determines to be a region to be confirmed. The region to be confirmed may be determined based on, for example, the tip of the guidewire being within a certain range, the tip of the guiding catheter being within a certain range, the tip of the delivery wire of the stent being within a certain range, the coil marker being within a certain range, the 2nd marker being within a certain range, the range in which the 2nd marker of the catheter moves to see when it is appropriate to cut the coil, the filter being within a certain range, the tip of the catheter being within a certain range, the embolic material being within a certain range, the range in which the device remains inside a large blood vessel, an aneurysm, or an important blood vessel, but is not limited thereto. For example, the automatic determination of the region to be confirmed may be performed based on a model created by machine learning, but is not limited thereto. The region may be manually specified by a user, or may be automatically or semi-automatically specified. For example, when the second marker of the catheter is recognized and a coil enters the aneurysm or a coil marker is detected, the region is automatically set to surround the second marker, or the region is proposed, and the region is confirmed when the user clicks OK (Fig. 35). Or, for example, at the tip of the guidewire, a rectangular region of constant a × constant b is automatically specified so that the guidewire is in the center. It may be automatically determined, or it may be confirmed when the user clicks OK, and the length and width may be corrected as necessary, or it may be confirmed after correction by translation, etc. Note that the constant a / b may be changed depending on the case, magnification rate, and the surgeon's preference. Furthermore, the region is not limited to a rectangle, and may be a circle, an ellipse, an arbitrary closed curve, a straight line, or an arbitrary curve. In the case of a straight line or an arbitrary curve, it is determined that the region is divided there. In the case of a straight line, the screen is divided into two regions by extending it. In addition, the region may be divided along the blood vessel with a straight line or an arbitrary curve. In other words, the region may be divided according to whether it is distal or proximal to the region drawn along the blood vessel with a straight line or curve.

[0186] The specific area may be inside or outside the blood vessel. For example, if it is outside the blood vessel (Figure 39), the notification will be issued when the tip of the guidewire enters the specific area, i.e., when it goes outside the blood vessel. In clinical practice, the guidewire is often in a small blood vessel (perforator branch), but if you are only looking at the front or side, you may be late in noticing and damage the small blood vessel, so by notifying the patient, it is possible to notify the patient early. Of course, the blood vessel may be perforated, and in that case, early notification can be issued to minimize complications.

[0187] Thin blood vessels such as perforator branches may be too thin to be seen in X-ray images. In this case, the automatic segmentation of blood vessels does not include the perforator branches, so by setting the outside of this segmentation as a specific region, it is possible to notify, for example, when the guidewire enters a perforator branch. If perforator branches or relatively thin blood vessels (such as the ophthalmic artery and the thin posterior communicating artery) are visible in X-ray images, a certain threshold can be set and blood vessels with diameters smaller than this threshold can be excluded from the segmentation, making it possible to notify when the guidewire strays into these blood vessels. Furthermore, if there is a blood vessel that the user does not want to enter, the user can specify it and set a specific region to notify when the guidewire enters that blood vessel. The specific region may be an area containing a lesion such as an aneurysm or stenosis, or an area behind it (an inverted region). For example, if an aneurysm is specified (this can be specified by the user or automatically recognized by segmenting the aneurysm), a notification will sound when the guidewire enters the aneurysm. This notifies the user that an aneurysm has been entered, and allows subsequent operations to be performed with caution. When attempting to bring the guidewire distal to the parent vessel of the aneurysm, a notification can be generated to inform the user that the guidewire is in a dangerous location (where it is not desired). These are examples, but are not limited to these.

[0188] Also, the specific area may be automatically surrounded from all of the maximum four screens, the front live image, the side live image, the front mask image, and the side live image, or may be selected and surrounded. For example, in the case of the tip of the guiding catheter, it is sufficient that it is visible in either the front image or the side image, so the specific area may be set by selecting the guiding catheter that is located higher in the two screens. For example, in the case of a notification of a coil marker surrounding a 2nd marker (FIG. 34), the specific area may be selected only in the front or side where the coil marker is easily visible. The visibility of the coil marker may be, for example, selected when detecting the coil marker by selecting the coil marker with a longer length, or by selecting the coil marker with a higher degree of confidence by deep learning, but is not limited to these methods.

[0189] When the scene is temporarily switched due to contrast, the notification may be detected and no notification may be issued. Notification may be suppressed according to the image quality. For example, when the mask image is shifted and becomes difficult to see, or when the entire screen is moving due to the patient's body movement, notification may be temporarily suppressed. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present. Specifically, for example, the output may be output to a catheter surgery robot via the output unit, and the surgeon may be notified through the robot, or the device may be automatically stopped from advancing once, or the device may be inserted and automatically sent out until it appears on the screen, and when the device appears due to the output of the output unit, the sending out may be stopped or the speed may be reduced. As another example, the device may be connected to AR / VR / MR, and when the output of the device's appearance is received from the output unit, it may be displayed or notified on the AR / VR / MR. Of course, this is not limited to this example, and all output to any other device is included.

[0190] When the user sets a specific region, if there is only one device, the relationship between the specific region and the device region of interest is automatically created by setting it as the region of interest. When multiple devices are included, the user may specify it, or the region of interest of the specific device may be automatically set based on the importance of the device, the position of the specific region in the screen, the scene of the surgery, etc. For example, in the case of a guidewire and a guiding catheter, the guidewire may be selected as the guidewire is more important. Or, when the guiding catheter and the guidewire enter the specific region at the same time, the guidewire is usually advanced further, so the guiding catheter may be selected. Also, if the specific region is at the bottom of the screen, it is more likely that the specific region is set with the intention of the guiding catheter rather than the guidewire, so the guiding catheter may be set as the region of interest. Also, in a surgery, when a coil is about to be inserted in a cerebral aneurysm embolization, if the guiding catheter and the guidewire are included in the specific region, the guiding catheter may be specified as the region of interest from the scene of the surgery. These are examples, and can be determined based on the type of device, the position of the specific region, the surgery scene, the user's preference, and the user's previous method.

[0191] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0192] For example, the region of interest may be set to include at least a part of a device for inspection or treatment in a blood vessel, selected from the group consisting of a guiding catheter, a guidewire, an intermediate catheter, a microcatheter, a suction catheter for thrombectomy, a marker, a coil, a stent, a filter, an embolic material, an aneurysm embolization device, and a balloon. For example, the region including at least a part of the device is the tip of a device for inspection or treatment in a blood vessel. Figure 40 notifies when a filter enters a specific region (outside a square) in carotid artery stent placement. In other words, it notifies when the filter moves by a certain amount.

[0193] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as a region of interest, and notifying the user on condition that a device that has newly appeared on an edge of the image is newly detected as a region of interest, and a program for executing the above method on a computer. Some aspects of the present invention also relate to an image processing device that executes the above method and a method of operating the same.

[0194] <Processing flow for recognizing a new device on the screen> 26 is a flowchart for explaining the flow of processing for recognizing a device that has newly entered into a screen, which is executed by the image processing device according to the embodiment. The processing in this flowchart starts, for example, when the image processing device is started.

[0195] First, the image acquisition unit and the region of interest acquisition unit function to detect an examination or treatment device in a blood vessel by image analysis. Then, it is determined whether a notification condition or an output condition (a device newly appearing at the edge of an image is newly detected as a region of interest) is satisfied, and if the condition is satisfied, notification or output is performed.

[0196] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image for intravascular inspection or treatment, and if the image includes an intravascular inspection or treatment device, acquires one or more regions including at least a portion of the device as regions of interest, and if at least one of the regions of interest satisfies a condition defined for each region of interest, notifies a user of the image processing device of this, wherein the notification is given to the user on the condition that a newly appeared device or a portion of the newly appeared device in the image or within a specific region is newly detected as a region of interest.

[0197] Furthermore, some aspects of the present invention relate to an image processing apparatus including a storage device and a processor connected to the storage device, wherein the processor acquires an image for intravascular inspection or treatment, and if the image includes an intravascular inspection or treatment device, acquires one or more regions including at least a portion of the device as regions of interest, and if at least one of the regions of interest satisfies a condition defined for each region of interest, outputs this to an external device, wherein the output is made to the external device on the condition that a newly appeared device or a portion thereof in the image or within a specific region is newly detected as a region of interest. (Notification based on the designation of specific boundaries) In some aspects of the present invention, the image processing device can notify a user (operator) on the condition that a region of interest passes through or is predicted to pass a specific boundary line specified on an image. More specifically, some aspects of the present invention relate to an image processing device that includes an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each of the regions of interest, and the notification unit notifies the user on the condition that the region of interest passes through or is predicted to pass a specific boundary line specified on an image. This allows the conditions for the image processing device to be set more flexibly (FIGS. 27 and 28). The prediction of the passage can be performed based on the distance between the region of interest and the boundary line, or the distance / speed of the region of interest. The image processing device may further include a tracking unit that tracks each of the regions of interest in the image. In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0198] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0199] In some aspects of the present invention, the specific boundary line may be specified by a user or automatically by an image processing device. When specified by a user, various pointing devices such as a touch panel or a mouse may be used. When specified automatically, the conditions and positions for such specification may be set in advance. The specific boundary line may be represented by a straight line, a curve, a circle, a rectangle, or any polygon or closed curve. For example, when the boundary line is a closed curve, "passing" may mean either entering or leaving the area specified by the closed curve. Such a boundary line may be used, for example, to detect the intrusion of a device into a blood vessel where the intrusion of the device is undesirable. For example, by setting a boundary line at the entrance of a branch point to a specific blood vessel, the intrusion of the device into such a blood vessel can be detected.

[0200] The particular boundary may be automatically assigned if the region of interest has not moved for a certain period of time. The particular boundary may be automatically assigned depending on the surgical context, or may be automatically proposed and the user may accept or reject it. The particular boundary may be automatically removed depending on the surgical context, or may be automatically proposed and the user may accept or reject it.

[0201] When the display device that displays the image has a plurality of screens, the particular boundary line specified on one screen may be automatically specified at a corresponding position on the other screens.

[0202] In some aspects, the notification unit notifies the user only when the condition is satisfied for the first time for one or more regions of interest. In other words, if device A and device B are set as regions of interest, notification is performed when device A passes the boundary line for the first time and when device B passes the boundary line for the first time. In some aspects, the notification unit notifies the user only when the condition is satisfied for the first time for one or more regions of interest after a specific time point. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0203] In some embodiments, the notification unit notifies the user only when the condition is satisfied for the first time in one or more regions in the entire region of interest. In other words, if device A and device B are set as regions of interest, notification is only performed when either device A or device B passes the boundary line for the first time, and notification is not performed when the other passes the boundary line. In some embodiments, the notification unit notifies the user only when the condition is satisfied for the first time after a specific time point in one or more regions in the entire region of interest. In some embodiments of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0204] In some aspects, the notification unit notifies the user of one or more specific areas designated by the user as a region of interest only when the conditions are satisfied. In other words, if device A and device B are set as regions of interest and the user specifies notification only for device A, notification is only made when device A passes the boundary line for the first time, and notification is not made when device B passes the boundary line. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0205] Furthermore, in some aspects, the notification unit notifies the user of one or more automatically identified regions of interest only when the conditions are satisfied. In other words, if device A and device B are set as regions of interest and the image processing device automatically specifies notification only for device A, notification is only made when device A passes the boundary line for the first time, and notification is not made when device B passes the boundary line. Note that, in some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0206] In some embodiments, the boundary line is superimposed on the X-ray image. The boundary line may be redrawn according to changes in the range or magnification of the X-ray image. This may be done automatically, or may be suggested to the user, and the boundary line may be redrawn when the user presses OK, or may remain as it is when the user presses No. For example, when the range of the X-ray image is moved up, down, left, right, or rotated, the boundary line is redrawn so as to move or rotate accordingly. Also, for example, when the range of the X-ray image is enlarged, reduced, or moved, the boundary line is redrawn so as to move or rotate accordingly (FIG. 42). In some embodiments, a particular boundary line is redrawn after the X-ray image is interrupted and reacquired. The notification may be made by issuing a warning sound or by changing the display style of the boundary line. Furthermore, different notification methods may be used depending on the direction in which the region of interest passes through a particular boundary line specified on the image. For example, when a boundary line is set at the entrance of a branch point to a particular blood vessel in order to detect the intrusion of a device into a blood vessel where the intrusion of the device is undesirable, different notifications (such as different sounds or colors) may be used when the intrusion of the device into such a blood vessel is detected and when the device is returned outside such a blood vessel. In some embodiments, the orientation at which the region of interest passes a particular designated boundary on the image can be automatically recognized or automatically suggested for the user to accept or reject depending on the surgical situation. Furthermore, different notification methods may be used depending on the speed at which the region of interest crosses the boundary.

[0207] In some aspects, the boundary line may be automatically erased due to a change in scene, or may be suggested to the user to be erased. If suggested, the boundary line is erased if the user clicks OK. For example, when the working angle is changed, the blood vessel shape changes on the 2D screen of the X-ray image, so maintaining the boundary line may become meaningless, and the boundary line is erased. Alternatively, when the platform is moved and the screen moves significantly, the boundary line is erased. This is an example, and the erasure of the boundary line is suggested when the scene is changed, when there is a lot of noise, when the patient's body movement is large, etc.

[0208] In some aspects, the notification unit may cause a display device that displays the image to display the distance between the region of interest and the specific boundary line, and may change a display mode of the distance on the display device depending on the magnitude of the distance between the region of interest and the specific boundary line. The notification unit may notify the user on condition that a value obtained by dividing the distance between the region of interest and the specific boundary line by a moving speed of the region of interest in the image is less than a predetermined threshold. Note that in some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0209] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, tracking each of the regions of interest in the image, and notifying the user or outputting to an external device on the condition that the region of interest passes through or is predicted to pass through a specific boundary line specified on the image, and a program for executing the above method on a computer. The prediction of the passage may be based on the distance between the region of interest and the boundary line, or the distance / velocity of the region of interest. Some aspects of the present invention also relate to an image processing device for executing the above method and an operation method thereof.

[0210] Some aspects of the present invention relate to an image processing apparatus including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more regions including at least a portion of the device included in the image as regions of interest, tracks each of the regions of interest in the image, and notifies the user when the region of interest passes or is predicted to pass a specific boundary line specified on the image.

[0211] Furthermore, some aspects of the present invention relate to an image processing apparatus including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more regions including at least a portion of the device included in the image as regions of interest, tracks each of the regions of interest in the image, and outputs to an external device on the condition that the region of interest passes, or is predicted to pass, a specific boundary line specified on the image.

[0212] (Specifying the area of ​​interest) For example, when a user (operator) needs to specify a device on a screen using a pointing device such as a mouse or a touch panel during surgery, it is not easy to pinpoint a moving device during surgery. For example, a device may move unintentionally during catheter treatment. Therefore, if a device can be selected by enclosing a certain area on the screen with an arbitrary figure such as a rectangle, it is difficult to miss the device even if it is moving. Therefore, in some aspects of the present invention, an image processing device can acquire a region of interest included in a region specified by a user on an image as a region of interest. More specifically, some aspects of the present invention relate to an image processing device including an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest, and the region of interest included in the region specified by the user on the image is acquired as the region of interest. This image processing device may further include a tracking unit that tracks each of the regions of interest in the image. In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0213] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0214] In some embodiments, the area on the image designated by the user is represented by a rectangle, a circle, or an arbitrary closed curve. Candidates for areas that the user can designate may be automatically displayed. In addition, in order to facilitate designation by the user on the screen, the image may be temporarily made a still image when the user designates an area. Alternatively, the image may be temporarily played back in delayed mode (slow playback) when the user designates an area. Furthermore, a portion of the image may be enlarged when the user designates an area. In some embodiments, candidates for areas that the user can designate may be automatically displayed according to the situation of the surgery.

[0215] In some embodiments, when a region including multiple regions of interest is specified, only the region of interest to be confirmed may be automatically selected. In other words, even if the specified region includes multiple regions of interest, the regions of interest that are not important are excluded from the specification. The conditions for determining the region of interest to be confirmed may be set in advance by the user, for example. Alternatively, the conditions may be set automatically by machine learning. For example, the region of interest to be confirmed may be determined according to the situation of the surgery. The situation of the surgery may also be automatically recognized by the image processing device. The region to be confirmed may be determined based on the fact that the tip of the guide wire is within a certain range, the tip of the delivery wire of the stent is within a certain range, the range in which the 2nd marker of the catheter moves to see the timing when the coil can be cut, the filter is within a certain range, the tip of the catheter is within a certain range, the embolic material is within a certain range, the range in which the device remains inside the large blood vessel, the aneurysm, or an important blood vessel.

[0216] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, tracking each of the regions of interest in the image, and notifying a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest, wherein the region of interest acquired is one included in a region specified by the user on the image, and a program for executing the above method on a computer.Furthermore, some aspects of the present invention relate to an image processing device that executes the above method and an operating method thereof.

[0217] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more areas including at least a portion of the device included in the image as areas of interest, tracks each of the areas of interest in the image, and, if at least one of the areas of interest satisfies a condition defined for each area of ​​interest, notifies a user of the image processing device, wherein the area of ​​interest acquired is an area included within an area specified by the user on the image.

[0218] Furthermore, some aspects of the present invention relate to an image processing apparatus including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more areas including at least a part of the device included in the image as areas of interest, tracks each of the areas of interest in the image, and, if at least one of the areas of interest satisfies a condition defined for each of the areas of interest, outputs this to an external device, wherein the area of ​​interest acquired is an area included within an area specified by a user on the image.

[0219] (Awareness of surgical situation) In the case of device recognition in real time during surgery, inference may be made in situations such as angiography, puncture, and noise generation, resulting in erroneous recognition results. Therefore, it is desirable to automatically recognize the situation of a specific surgery and take measures such as stopping device recognition depending on the situation. Therefore, some aspects of the present invention relate to an image processing device including an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, and a situation recognition unit that recognizes the situation of a specific surgery based on the image.

[0220] For example, the specific surgical situation may be at least one situation selected from the group consisting of surgical content, disease name, vascular leakage, vascular perforation, thrombus occurrence, peripheral blood vessel disappearance, guiding of guidewires, catheters, and balloons, catheter aspiration, stent placement, balloon expansion, coil insertion, insertion of an aneurysm embolization device (e.g., WEB (Terumo) which is a bag-shaped embolization device, and PulsRider (Cerenovus) which is a self-expanding implant), timing of coil disconnection, guiding, placement, and retrieval of a filter, injection of a liquid embolic material, injection of a contrast agent, continuation of a still image for a certain period of time or more, discrimination between a mask image and a live image, discrimination of an imaging region and angle, switching of images, and occurrence of noise. The specific surgical situation may be recognized by pattern matching, image recognition, time-series image recognition, time-series difference, or object detection algorithm. For example, the scene change can be recognized as a change in the scene (surgical situation) by taking a time-series difference and if it is greater than a certain threshold value, it is possible to recognize that the scene (surgical situation) has changed. Noise can be classified into no noise, a little noise, and noise, and can be recognized by classification learning using machine learning CNN. Balloon expansion can be determined by recognizing the presence of an expanded balloon through object recognition such as SSD in machine learning and segmentation learning using U-Net (Fig. 30). Fig. 30 shows an example of recognizing, displaying, and recording X-ray images and videos of the hands of a specific operation such as puncture, balloon guidance, and coil insertion. It is also possible to display events such as photography (contrast injection), the guiding catheter or guide wire going out of the screen, and the time period during which the balloon was expanded. For example, clicking on this will allow you to jump to that event and quickly review the operation. If necessary, two or four screens will be displayed as thumbnails. When an event occurs, the screen on which it occurred will be clearly indicated by surrounding it with a different color frame. By recording the time course and the event type and number, it is possible to record important elements of the operation (e.g., the time until the guiding catheter is raised, the number of times the guiding catheter fell off, the number of balloon expansions, and the average expansion time). This will provide feedback to the surgeon and assistants, and is expected to lead to improvements in subsequent procedures.In addition, as shown in Figure 30, the changes in the guidewire speed and coil speed may be recorded together (if the guidewire and coil can be recognized and tracked by deep learning, this can be displayed). This, or by converting it into a histogram or mean / variance, can express the characteristics of the operation or the surgeon's technique. For example, if the guidewire is moved slowly near the aneurysm but relatively quickly in other areas, this suggests that the technique is good. Comparing these between different surgeons, it can capture the characteristics of the technique and serve as an educational indicator of what can be done to become a skilled surgeon. Also, by tracking the progress of the same surgeon for, for example, three years, it can be shown how much he or she has improved (a record of the surgeon's own technical growth). In addition, the hands may be photographed with a surgical camera, and by recording this on the same timeline, it is possible to see, for example, what the hands were like when a certain event occurred. Another advantage is that it allows skilled surgeons to easily review the movements of their hands when inserting a catheter into an aneurysm (with conventional technology, it is difficult to confirm how the hands actually moved in relation to the operations that can be confirmed in the X-ray image because the timeline of the X-ray image and the surgical camera do not match). Furthermore, as shown in the lower right of Figure 30, it is possible to create a list of time passage and events (puncture, guiding catheter guidance, etc.). This can be used to assist in recording the operation. Furthermore, it is possible to record the time when the video is not moving. This means that X-ray fluoroscopy is being performed but there is no movement, indicating that there is unnecessary fluoroscopy. It is difficult to always determine whether unnecessary X-ray fluoroscopy is occurring during surgery, but this system later provides feedback to the surgeon that the fluoroscopy time should be shortened if the unnecessary X-ray fluoroscopy is relatively long. This gradually shortens the X-ray fluoroscopy time, which is expected to reduce radiation exposure to patients and medical staff.

[0221] In some aspects, the image processing device may further include a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each region of interest. In some aspects, the device to be recognized can be specified according to the situation of the recognized specific surgery. For example, if a carotid artery stenting is being performed, a coil usually does not appear, so even if it is mistakenly recognized as a coil, the recognition accuracy can be improved by ignoring it. In addition, if an aneurysm is recognized and it is known which process is currently being performed, it is possible to limit where and what kind of device should be, improving accuracy. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0222] In some aspects, depending on the particular surgical situation recognized, a particular boundary line may be automatically specified or suggested around the region of interest to notify the user if the region of interest passes through the particular boundary line specified on the image.

[0223] For example, when the situation recognition unit recognizes the occurrence of noise, the acquisition of the region of interest can be stopped. Also, when the continuation of a still image for a certain period of time or more is recognized, the user can be notified of the recommendation to reduce exposure.

[0224] In some embodiments, if an injection of contrast agent is detected, or if a 3D image is acquired, or if a cone beam computed tomography (CBCT) is acquired, acquisition of the region of interest may be stopped, or may be acquired but without notification.

[0225] In some aspects, the imaging device further includes a recorder that records the particular surgical situation that is recognized.

[0226] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject and recognizing a specific surgical situation based on the image, and a program for executing the above method on a computer. Some aspects of the present invention also relate to an image processing device for executing the above method and an operating method thereof.

[0227] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, the processor acquiring images including at least an intravascular inspection or treatment device as a subject, and recognizing a specific surgical situation based on the images.

[0228] (Surgery record) Surgical videos are important for recording, educating medical staff, and explaining to patients and their families. However, for example, in the case of cerebrovascular surgery, there are videos corresponding to up to four screens, each of which is 1 to 2 hours long, so viewing and editing takes time. Therefore, it is desirable to automatically detect events and classify scenes using AI, save videos for surgical recording, education, and explanation, and save still images of important images. Therefore, some aspects of the present invention relate to an image processing device comprising an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a recording unit that records the status of the surgery corresponding to the condition when at least one of the regions of interest satisfies a condition defined for each of the regions of interest.

[0229] The conditions for recording include at least one procedure selected from the group consisting of, for example, puncture, arrival of the guiding catheter at the treatment site, balloon guidance, balloon expansion, stent deployment, catheter guidance, first coil insertion, second or subsequent coil insertion, localization of the coil within the aneurysm, deviation of the coil into the parent vessel, catheter removal, coil removal, and end of treatment. The recorded surgical situation may include still images, videos, and / or text information explaining the surgical situation. The video is, for example, a video of a certain period before and after the time when the condition is satisfied. The recorded information may be modifiable by the user.

[0230] In some embodiments, the image processing device further includes a report generator configured to generate a report based on the recorded information, for example, a summary of the surgery based on the recorded still images, video, and / or text information describing the surgical situation.

[0231] In some embodiments, the image processing device further includes a notification unit that notifies the user when the status of a specific surgery is recorded. In some embodiments, the image processing device further includes a notification unit that notifies the user according to the time lapse after the status of a specific surgery is recorded. The time lapse after the status of a specific surgery is recorded may be determined based on the balloon expansion time, the guide wire movement time, speed, and acceleration, or the coil movement time. For example, if there is no movement in the screen even though X-rays are being emitted, that is, if the image is still, this increases unnecessary X-ray exposure for patients and medical staff. By notifying this in real time or as a post-operative record, it leads to reducing the radiation exposure dose throughout the entire surgery. For example, if the time and scene of this unnecessary X-ray exposure are recorded in the post-operative record, the doctor can make an effort to eliminate the unnecessary X-ray exposure from the next time onwards. The time of each procedure is also important. For example, when the balloon is being expanded, it means that the blood vessel is being blocked, and it is said that it is better not to continue the expansion for more than 3 to 5 minutes, depending on the part of the blood vessel (continuing it can lead to cerebral ischemia and cerebral infarction). By graphing the number of balloon inflations and the respective inflation times in real time or after surgery, feedback is obtained that it would be better to deflate the balloon sooner. As another example of the time required for each procedure, in aneurysm coil embolization, by measuring the time required for each procedure, such as balloon guidance, catheter guidance into the aneurysm, and coil insertion, it is possible to visualize where the time was spent in that case. This allows us to see how much time was spent compared to the average or compared to an experienced doctor, which leads to shortening the surgery time and reducing radiation exposure. In addition, by tracking the tip of the guidewire / catheter, for example, it is expected that an experienced doctor will operate carefully (with low speed and acceleration) near the aneurysm and quickly in other places. Conversely, if this balance is lost, there is something that needs to be improved. Furthermore, by visualizing these various parameters for each surgeon, it becomes possible to evaluate the surgeon and understand where each surgeon should improve. Therefore, it becomes possible to visualize the improvement in the skill of an individual surgeon, and also to visualize whether the skill is being maintained.In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0232] In some aspects, the image processing device further includes a notification unit that notifies a user according to the number of times a specific surgical situation is recorded. The specific surgical situation is, for example, the insertion of a coil. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0233] In some aspects, the images are obtained from real-time or previously recorded video.

[0234] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more areas including at least a portion of the device included in the image as areas of interest, and, if at least one of the areas of interest satisfies a condition defined for each area of ​​interest, records the status of a surgery corresponding to that condition.

[0235] (Scene Classification Application) In the case of real-time surgical support, when setting the ROI, the boundary (ROI) may be proposed, appeared, erased, moved, enlarged, or reduced depending on the scene classification. In addition, the presence or absence and type of recognition and notification may be changed. For example, if the scene is classified as being in contrast injection, recognition and notification may be stopped. This is because, generally, the purpose of contrast injection is to see blood vessels, and the device rarely moves at that time, so there is little need for notification. On the other hand, notification may be given. This is because, although rare, the device may move due to contrast injection.

[0236] In addition, when creating a mask image, the surgeon or assistant must select a frame with an appropriate time phase after contrast imaging, which is a time-consuming and tedious task. By applying the scene classification function, it is possible to automatically select a frame that is deemed appropriate for creating a mask image, such as the frame in which the artery is best seen in the case of contrast imaging, or any frame before or after that, or a frame in which the lesion is best seen, or a frame close to the timing used for the mask image during the surgery. The surgeon, assistant, or medical staff involved in the surgery can specify this as the base for the mask image, or they can select an appropriate frame by moving frames before or after it. In addition, it is not necessary to limit it to the arterial phase, but it is possible to obtain a frame with a time phase that is deemed appropriate for the surgery at that time, such as the capillary phase or venous phase.

[0237] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, and, if at least one of the regions of interest satisfies a condition defined for each of the regions of interest, recording a surgical situation corresponding to that condition, and a program for executing the above method on a computer.Furthermore, some aspects of the present invention relate to an image processing device for executing the above method and a method of operating the same.

[0238] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more areas including at least a portion of the device included in the image as areas of interest, and, if at least one of the areas of interest satisfies a condition defined for each area of ​​interest, records the status of a surgery corresponding to that condition.

[0239] (Display of mark corresponding to Y coordinate) In an image including multiple devices, the increase in various information on the screen may make it difficult to grasp the situation at a moment. To solve this problem, a sub-screen that displays only the vertical position of the device is provided next to the main surgery screen (FIG. 29). Therefore, some aspects of the present invention relate to an image processing device that includes an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each of the regions of interest, and further includes a display unit that displays the image including the region of interest and a mark corresponding only to the vertical coordinate of the region of interest next to the image on a display device connected to the image processing device. This image processing device may further include a tracking unit that tracks each of the regions of interest in the image. In some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0240] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0241] In some embodiments, the trajectory of the mark may be displayed for a certain period of time. The mark may be displayed in a different manner for each of the multiple regions of interest. Here, the display manner may be a color, a shape, a pattern, a symbol, a character, a caption, or an animation.

[0242] In some aspects, the image processing device can automatically recognize and highlight the region of interest to be confirmed. The conditions for determining the region of interest to be confirmed may be set in advance by a user, for example. Alternatively, the conditions may be set automatically by machine learning. For example, the region of interest to be confirmed may be determined according to the surgical situation. The surgical situation may also be automatically recognized by the image processing device.

[0243] In some aspects, only a region of interest specified by a user is displayed on the sub-screen, and in some aspects, only a region of interest selected automatically is displayed on the sub-screen.

[0244] Furthermore, in some aspects, the image processing device can notify the user when the mark exceeds a boundary value designated by the user. The boundary value can be designated by the user by, for example, setting a specific range on the sub-screen using a pointing device such as a mouse or a touch panel.

[0245] Furthermore, some aspects of the present invention relate to a method for acquiring an image including at least an intravascular inspection or treatment device as a subject, acquiring one or more regions including at least a part of the device included in the image as regions of interest, tracking each of the regions of interest in the image, and notifying a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest, the method including displaying the image including the region of interest and a mark corresponding only to the vertical coordinate of the region of interest next to the image on a display device connected to the image processing device, and a program for executing the above method on a computer. Some aspects of the present invention also relate to an image processing device for executing the above method and a method of operating the image processing device.

[0246] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more areas including at least a portion of the device included in the image as areas of interest, tracks each of the areas of interest in the image, and, if at least one of the areas of interest satisfies a condition defined for each area of ​​interest, notifies a user of the image processing device and displays, on a display device connected to the image processing device, the image including the area of ​​interest and a mark corresponding only to the vertical coordinate of the area of ​​interest next to the image.

[0247] (Multiple regions of interest tracking display) Some aspects of the present invention relate to an image processing device including an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject, a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest, a tracking unit that tracks each of the regions of interest in the image, and a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition set for each of the regions of interest, and further includes a display unit that displays the multiple regions of interest in different display formats when multiple regions of interest are tracked. In such an aspect, the multiple regions of interest are displayed in different display formats, making it easy to identify individual devices. The display format is, for example, a color, a shape, a pattern, a symbol, a character, a caption, or an animation.

[0248] In some aspects, the region of interest to be confirmed is automatically recognized and highlighted. In some aspects, the region of interest to be confirmed is determined according to the surgical situation. The conditions for determining the region of interest to be confirmed may be preset by a user, for example. Alternatively, the conditions may be automatically set by machine learning. For example, the region of interest to be confirmed may be determined according to the surgical situation. The surgical situation may also be automatically recognized by the image processing device.

[0249] Furthermore, in some aspects, the image processing device may display only the region of interest specified by the user. Also, the image processing device may display only the region of interest selected automatically. In the case of automatic selection, the conditions for such selection may be preset.

[0250] Some aspects of the present invention relate to an image processing device including a storage device and a processor connected to the storage device, wherein the processor acquires an image including at least an intravascular inspection or treatment device as a subject, acquires one or more regions including at least a portion of the device included in the image as regions of interest, tracks each of the regions of interest in the image, and if at least one of the regions of interest satisfies a condition defined for each of the regions of interest, notifies a user of the image processing device of this, and wherein if multiple regions of interest are tracked, the multiple regions of interest are displayed in different display formats.

[0251] (Example System) Some aspects of the present invention relate to a vascular catheter surgery support system, for example, a catheter surgery support system for blood vessels in the brain, heart, peripheral limbs, and abdomen, particularly for blood vessels in the brain. Such a system may include an image processing device and an image capturing device that captures X-ray images of a patient with one or more devices inserted into a blood vessel and transmits the images to the image processing device, wherein the image processing device includes an image acquisition unit that acquires X-ray images over time of a region (e.g., a fixed region) including at least a portion of the device inserted into the blood vessel and a focus area for achieving the purpose of surgery, an area of ​​interest acquisition unit that acquires one or more regions including at least a portion of the device included in the image as an area of ​​interest, and a notification unit that notifies a user of the image processing device when at least one of the areas of interest satisfies a condition defined for each area of ​​interest, and the system may be configured such that when the one or more devices are a catheter, a guidewire, a stent, and / or a balloon, and an area including the tip of a catheter or the tip of a guidewire, both ends of a stent, or both ends of a balloon is set as the area of ​​interest, the system notifies the user when the area of ​​interest disappears from the image, or when the distance between the area of ​​interest and the edge of the image becomes less than a predetermined threshold distance, or when the area of ​​interest shifts by a certain distance. The system may further include a tracking unit that tracks each of the regions of interest in the image. Note that in some aspects of the present invention, instead of or in addition to the notification unit, an output unit that outputs information to a device connected to the image processing device may be present.

[0252] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0253] Here, the notification unit may display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest on the display device that displays the image, and the notification unit may change the display mode of the distance on the display device according to the magnitude of the distance, and the change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a figure on the entire screen of the display device or outside the frame or a part of the screen, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of a mark added to the region of interest according to the magnitude of the distance. Furthermore, the notification unit may sound a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0254] Some aspects of the present invention relate to an aneurysm coil embolization assist system, particularly a cerebral aneurysm coil embolization assist system. Such a system may include an image processing device, an image capturing device that captures X-ray images of a patient with a guiding catheter and a delivery wire of an embolization coil inserted in a blood vessel and transmits the images to the image processing device, and the image processing device includes an image capture unit that captures X-ray images of a fixed region including at least an aneurysm generated in the patient's blood vessel, the catheter inserted in the blood vessel, and the delivery wire of the embolization coil over time, a region of interest capture unit that captures one or more regions including at least a part of the guiding catheter included in the images as regions of interest, a marker detection unit that detects a marker provided on the delivery wire and that approaches one or more regions of interest set on a part of the catheter that guides the delivery wire, and a notification unit that notifies a user of the timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap. This system may further include a tracking unit that tracks each of the region of interest and the marker in the images. In some aspects of the present invention, instead of or in addition to the notification section, there may be an output section that outputs information to a device connected to the image processing device.

[0255] The image processing device may further include a blood vessel recognition unit that recognizes at least a part of a blood vessel in the image. When the region of interest is within a blood vessel having a diameter equal to or smaller than a threshold, or a manually or automatically designated blood vessel, a notification or output may be provided.

[0256] Here, the notification unit may display the distance between the region of interest and the edge of the image or the distance between the marker and the region of interest on the display device that displays the image, and the notification unit may change the display mode of the distance on the display device according to the magnitude of the distance, and the change in the display mode may include changing the font, size, or color of the characters displayed according to the magnitude of the distance, changing the color of the entire screen or a part of the screen of the display device according to the magnitude of the distance, displaying a figure on the entire screen of the display device or outside the frame or a part of the screen, enlarging and displaying the region of interest according to the magnitude of the distance, or changing the color or size of a mark added to the region of interest according to the magnitude of the distance. Furthermore, the notification unit may sound a notification sound according to the magnitude of the distance. Furthermore, the distance may be determined by either a straight-line distance or a distance along a blood vessel.

[0257] <Processing flow of the image processing method executed by the image processing device 1> 10 is a flowchart for explaining the flow of image analysis processing executed by the image processing device 1 according to the embodiment. The processing in this flowchart starts, for example, when the image processing device 1 is started.

[0258] The image acquisition unit 110 acquires an X-ray image created based on the X-ray absorption rate, the X-ray image including at least a blood vessel V and an inspection or treatment device D inside the blood vessel V as subjects (S2). The region of interest acquisition unit 111 acquires one or more regions including at least a part of the device D included in the X-ray image as a region of interest R (S4).

[0259] The tracking unit 112 tracks each region of interest R in the X-ray image (S6). If at least one of the regions of interest R satisfies the condition defined for each region of interest R (Yes in S8), the notification unit 113 notifies the user of the image processing device 1 of this (S10). If none of the regions of interest R satisfies the defined condition (No in S8), the notification unit 113 skips the notification process.

[0260] Until the image processing is completed (No in S12), the image processing device 1 returns to the process of step S6 and repeats the processes from step S6 to step S10. If the image processing is completed (Yes in S12), the process in this flowchart ends.

[0261] <Diagnosis and comparison of angiography> In angiography examinations and treatments, blood vessels are photographed by contrast to diagnose lesions, but lesions may be overlooked or may need to be judged by comparing the images with images taken earlier on the same day or on a different day, which may take time or be difficult to compare. Also, because the images are projected in two dimensions, it may be difficult to judge. Therefore, some aspects of the present invention relate to an imaging diagnostic device that uses deep learning or the like in angiography using contrast agents to indicate lesions or sites, including but not limited to cerebral aneurysms, stenosis, occlusion, thrombus formation, vascular perforation (contrast agent leakage), shunt disease, tumor vessel nutrient / tumor staining, venous thrombosis, avascular areas in the capillary phase (findings of vascular occlusion), collateral circulation, and the like.Therefore, in some aspects of the present invention, the image processing device detects in the image aneurysms, stenosis, vasospasm, dissection, occlusion, recanalization, thrombus formation, the site and both ends of a thrombus, vascular perforation, extravascular leakage of contrast medium, vascular calcification, arteriosclerosis, shunt disease and its nutrient blood vessels and outflow blood vessels, blood (contrast medium) reflux, cerebral arteriovenous malformation, dural arteriovenous fistula, avascular area, bone mark (internal auditory canal, fundus, supraorbital rim, petrous part, foramen magnum, cervical vertebrae, , clavicle, rib / vertebral number, femoral head, pelvis), tumor blood vessels feeding the tumor / tumor staining, venous occlusion, venous sinus thrombosis, avascular area in the capillary phase, vascular occlusion, shape and distribution of the coil within the aneurysm, position / bulge / shape of the balloon, deviation of the coil into normal blood vessels, insufficient expansion of the stent, degree of adhesion to the blood vessel or twisting, movement of the stent, position of both ends of the stent, positional relationship between the puncture site and the blood vessel (is there a stenosis or is it near a bifurcation?), The present invention may further include a lesion recognition unit that recognizes lesions selected from the group consisting of vascular tortuosity, aortic arch type (how far below the top of the aortic arch the right brachiocephalic artery is), penetration range of liquid embolic material, delay or stagnation of blood (contrast medium) flow, vascular variation (anterior communicating artery, anterior cerebral artery A1, posterior communicating artery, posterior cerebral artery P1, posterior inferior cerebellar artery, anterior inferior cerebellar artery, superior cerebellar artery, superficial temporal artery, each venous sinus, presence or absence and degree of development of each vein), moyamoya vessels (stenosis or occlusion at the tip of the internal carotid artery and development of collateral blood flow beyond that), position of arterial branching and segment (petrosis of the internal carotid artery, cavernous sinus, ophthalmic artery, branching of middle cerebral artery M1), traces of previous surgery (clips, coils, plates, shunt tubes / valves, ventricular tubes, cisternal tubes), position and degree of opening of the WEB device, foreign bodies (dentures, plates), and collateral blood flow. It is also possible to notify the doctor when an abnormality is suspected, or to indicate the area where an abnormality may occur, without indicating the abnormality. In this case, the doctor can make the final decision. Similarly, some aspects of the present invention relate to an image diagnostic device that indicates changes in angiography by comparing the angiography with a previous image or an image taken on a different day. Thus, in some aspects of the present invention, the image processing device may further include an image recognition unit that compares an angiography image in an image with an angiography image previously obtained and stored, and notifies the doctor of changes.For example, changes in the degree of vasospasm, changes in thrombus formation (appearance, disappearance, enlargement, reduction, etc.), release of blocked blood vessels, blockage of blood vessels, displacement of coils, migration of stents, etc. can be indicated.

[0262] <Advantages of the Image Processing Device 1 According to the Embodiment> As described above, the image processing device 1 according to the embodiment can provide a technology that enables a medical professional user to focus on the work in the area of ​​interest during vascular catheter examination or treatment, and supports the user in determining the area of ​​interest.

[0263] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

[0264] <First Modification> Although the above description has been focused on the examination or treatment of cerebrovascular disease, the present invention is not limited to the application of the present invention, and can be applied to the examination and treatment of blood vessels in the circulatory system, such as the heart, peripheral limbs, and abdomen.

[0265] <Second Modification> In the above, an example of a device with two screens as shown in Figs. 7(a)-(b) has been described, but the number of screens is not limited to this, and may be, for example, one screen or three or more screens.

[0266] <Third Modification> In the above, a case has been described in which the X-ray imaging device 3 captures an image of the operative part of the subject P. However, the imaging device that captures the image of the operative part of the subject P is not limited to the X-ray imaging device 3. In addition, the image of the operative part may be captured using a modality such as MRI (Magnetic Resonance Imaging) or an ultrasonic imaging device.

[0267] <Example of embodiment> A part or all of the above embodiments can be described as follows, but the disclosure of the present application is not limited to the following notes.

[0268] (Appendix 1) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; An image processing device comprising: (Appendix 2) the notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, on condition that the region of interest disappears from the image. 2. An image processing device as described in claim 1. (Appendix 3) The notification unit notifies the user on condition that, when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest, a distance between the region of interest and an edge of the image is less than a predetermined threshold distance. 2. An image processing device as described in claim 1. (Appendix 4) The notification unit notifies the user on condition that at least one of a moving distance, a moving speed, and an acceleration of the region of interest in the image exceeds a predetermined threshold. 4. The image processing device according to claim 1 , (Appendix 5) The notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image. 5. An image processing device according to claim 1 . (Appendix 6) The notification unit changes a display manner of the distance on the display device depending on a distance between the region of interest and an edge of the image. 6. The image processing device according to claim 5. (Appendix 7) The notification unit notifies the user on condition that a value obtained by dividing a distance between the region of interest and an edge of the image by a moving speed of the region of interest within the image is less than a predetermined threshold. 7. An image processing device according to any one of claims 1 to 6. (Appendix 8) a marker detector that detects a marker provided on a delivery wire of the embolization coil approaching a region of interest set on a part of a microcatheter that guides the delivery wire; The tracking unit further tracks the detected marker, and the notification unit notifies the user of a timing at which the embolization coil may be disconnected from the delivery wire, when the marker and the region of interest overlap. 8. An image processing device according to claim 1, (Appendix 9) The notification unit notifies the user when the marker passes through the region of interest. 9. The image processing device according to claim 8. (Appendix 10) the notification unit causes a display device to display a distance that the marker should move until the embolization coil is disconnected from the delivery wire. 10. The image processing device according to claim 8 or 9. (Appendix 11) and when a feature amount indicating a shape of the device included in the region of interest satisfies a predetermined condition, the notification unit notifies a user of the image processing device of that fact. 11. An image processing device according to any one of claims 1 to 10. (Appendix 12) the feature value is a curvature, The notification unit notifies the user when a curvature of the device included in the region of interest exceeds a predetermined threshold curvature, or when the tip does not move despite a change in curvature. 12. The image processing device according to claim 11. (Appendix 13) The notification unit notifies the user on condition that a value obtained by subtracting a length of a center line of the blood vessel included in the image or the region of interest from a length of the device included in the image or the region of interest exceeds a predetermined threshold length. 13. The image processing device according to claim 11 or 12. (Appendix 14) The image processing device of any one of appendices 1 to 13, wherein the notification unit notifies the user by displaying the area of ​​interest in a color different from that of the image, changing the font, size or color of the displayed characters, changing the color of the entire screen of the display device or a portion of it, displaying a graphic on the entire screen of the display device or outside the frame or a portion of it, enlarging the area of ​​interest, or changing the color or size of a mark added to the area of ​​interest.

[0269] (Appendix 15) The notification unit notifies the user when a region including a tip portion of a catheter or a tip portion of a guidewire is set as the region of interest and the region of interest has moved or has exceeded a specific range designated on the image. 15. An image processing device according to any one of claims 1 to 14. (Appendix 16) 16. The image processing device according to any one of claims 1 to 15, further comprising a video storage unit that stores images or videos obtained from the image acquisition unit over time. (Appendix 17) 17. The image processing device of claim 16, further comprising a video extraction unit that extracts from the video storage unit video of a certain period before and after the notification unit issues a notification, or video of any time or period specified by the user. (Appendix 18) 18. The image processing device according to claim 17, wherein the extraction period of the image is automatically determined based on at least one of the moving distance, the moving speed, and the acceleration of the area of ​​interest when the notification occurs. (Appendix 19) 19. The image processing device according to claim 17 or 18, wherein the extracted image is displayed on a display device. (Appendix 20) 20. The image processing device of claim 19, wherein the extracted image is automatically displayed repeatedly a predetermined number of times. (Appendix 21) 21. The image processing device of claim 20, wherein the extracted image is displayed based on any operation including play, stop, fast forward, rewind, frame forward, slow playback, and double speed playback. (Appendix 22) 22. The image processing device according to any one of appendices 17 to 21, wherein the time elapsed since the occurrence of the notification, a comparison between the position of the area of ​​interest after the occurrence of the notification and an arbitrary time has elapsed, or a trajectory of the area of ​​interest acquired by the tracking unit is further displayed superimposed on the extracted image. (Appendix 23) 23. The image processing device according to any one of appendices 17 to 22, wherein the extracted image is displayed by cutting out a partial area in the vicinity of the region of interest. (Appendix 24) 24. The image processing device according to any one of appendices 17 to 23, wherein the extracted image is displayed at a position that does not interfere with the display of the region of interest. (Appendix 25) 25. The image processing device according to any one of appendices 17 to 24, wherein the extracted image is enlarged and displayed. (Appendix 26) 26. The image processing device of any one of appendices 17 to 25, wherein the extracted image is displayed simultaneously with the occurrence of a notification or after a predetermined time has elapsed since the occurrence of the notification. (Appendix 27) 27. The image processing device according to any one of appendix 17 to 26, wherein images captured from multiple directions are displayed simultaneously. (Appendix 28) 28. The image processing device according to any one of appendix 2 to 27, further comprising a state estimation unit that estimates a current position and / or velocity of the tip of the catheter or the tip of the guidewire that has disappeared from the image, based on the position, velocity and / or acceleration of the tip of the catheter or the tip of the guidewire immediately before the region of interest disappeared from the image. (Appendix 29) 29. The image processing device of claim 28, wherein a warning is notified to a user when a current position and / or velocity of the region of interest estimated by the state estimation unit exceeds a predetermined threshold. (Appendix 30) 30. The image processing device according to any one of claims 1 to 29, wherein the display device that displays the images displays the two images on two screens of different sizes. (Appendix 31) 31. The image processing device according to claim 30, wherein the display device alerts the user by making a frame portion of one of the two screens glow, change a color, or highlight it. (Appendix 32) 32. The image processing device according to any one of claims 1 to 31, wherein the display device for displaying the image displays a product list of devices for intravascular inspection or treatment. (Appendix 33) 33. The image processing device of claim 32, wherein the display device displays a list of products narrowed down by size or stock. (Appendix 34) 34. The image processing device of claim 32 or 33, wherein the display device displays a list of recommended products based on image analysis results, facility information, or user preference information. (Appendix 35) An image processing device according to any one of claims 1 to 34, which creates, automatically or based on user selection, a surgical record including information on the device used, information on the acquired images, and image analysis results. (Appendix 36) An image processing device as described in any one of Appendix 1 to 35, wherein the notification unit displays, on a display device that displays the image, a numerical value, color, bar, or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition set for each of the regions of interest, or a numerical value, color, bar, or heat map based on a value obtained by applying an arbitrary transformation to a probability distribution. (Appendix 37) The image processing device according to any one of appendixes 1 to 35, wherein the notification unit colors the region of interest with a color or heat map corresponding to the probability that at least one of the regions of interest satisfies a condition set for each region of interest, or with a color or heat map based on a value obtained by arbitrarily converting a probability distribution, and displays the color or heat map on the display device that displays the image, or replaces the probability of satisfying a condition with a numerical value or color and displays the result on the display device that displays the image. (Appendix 38) 38. The image processing device according to any one of appendices 1 to 37, wherein the notification unit notifies the user when, when a region including the tip of a catheter or the tip of a guidewire is set as the region of interest, the region of interest has moved or has exceeded a specific range specified on an image. (Appendix 39) 39. The image processing device of claim 38, wherein the boundary line of the specific range is represented by a straight line, a curve, a circle, a rectangle, or another polygon. (Appendix 40) 40. The image processing device of claim 38, wherein the specific range is superimposed on an X-ray image. (Appendix 41) 41. The image processing device according to any one of claims 38 to 40, wherein the notification unit causes a display device that displays the image to display the distance between the region of interest and an edge of the specific range. (Appendix 42) The image processing device according to claim 41, wherein the notification unit changes a manner in which the distance is displayed on the display device depending on the distance between the region of interest and an edge of the specific range. (Appendix 43) 43. The image processing device according to any one of appendices 38 to 42, wherein the notification unit notifies the user on condition that a value obtained by dividing the distance between the area of ​​interest and the edge of the specific range by a moving speed of the area of ​​interest within the image is less than a predetermined threshold. (Appendix 44) 44. The image processing device of any one of claims 3 to 43, wherein the distance is determined by either a straight-line distance or a distance along a blood vessel. (Appendix 45) 45. An image processing device according to any one of claims 1 to 44, comprising a memory unit that acquires and stores the position and / or shape of an intravascular inspection or treatment device at any time, and the stored position and / or shape of the device is superimposed on subsequent images. (Appendix 46) 43. The image processing device according to any one of appendix 1 to 42, further comprising a lesion recognition unit that recognizes lesions in the image selected from the group consisting of cerebral aneurysm, stenosis, occlusion, thrombus formation, vascular perforation, extravascular leakage of contrast agent, shunt disease, nutrient blood vessels of tumor blood vessels / tumor staining, venous thrombosis, avascular areas in the capillary phase, vascular occlusion, and collateral circulation. (Appendix 47) 44. The image processing device of any one of claims 1 to 43, further comprising an image recognition unit for comparing angiograms in the image with previously acquired and stored angiograms and notifying changes.

[0270] (Appendix 48) A processor of the image processing device acquiring an image including at least an intravascular inspection or treatment device as a subject; acquiring one or more regions including at least a portion of the device included in the image as a region of interest; tracking each of the regions of interest in the image; If at least one of the regions of interest satisfies a condition defined for each of the regions of interest, notifying a user of the image processing device of that fact; An image processing method that performs

[0271] (Appendix 49) On the computer, A function of acquiring an image including at least an intravascular inspection or treatment device as a subject; a function of acquiring one or more regions including at least a portion of the device included in the X-ray image as a region of interest; tracking each of said regions of interest in said x-ray image; a function of notifying a user of the computer when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; A program to achieve this.

[0272] (Appendix 50) An image processing device according to any one of appendix 1 to 44, an imaging device that captures an image of a person having a device for intravascular inspection or treatment inserted therein and transmits the image to the image processing device; An image processing system comprising:

[0273] (Appendix 51) A cerebrovascular catheter surgery support system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with one or more devices inserted into a blood vessel and transmits the image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed area including at least a target area for achieving the purpose of surgery and a device inserted into a blood vessel over time; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with the one or more devices are catheters, guidewires, stents, and / or balloons; When a region including a tip end of a catheter or a tip end of a guidewire, both ends of a stent, and both ends of a balloon is set as the region of interest, the user is notified on the condition that the region of interest disappears from the image or the distance between the region of interest and an edge of the image becomes less than a predetermined threshold distance. system. (Appendix 52) A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the image to the image processing device; Equipped with The image processing device, an image acquisition unit that acquires X-ray images of a fixed region including at least an aneurysm occurring in a patient's blood vessel, a catheter inserted into the blood vessel, and a delivery wire of an embolization coil over time; a region of interest acquisition unit configured to acquire one or more regions including at least a part of the guiding catheter included in the image as a region of interest; a marker detection unit that detects a marker provided on the delivery wire and that approaches one or more regions of interest set in a part of the catheter that guides the delivery wire; and a tracking unit configured to track each of the region of interest and the marker in the image; a notification unit that notifies a user of a timing at which the embolization coil may be disconnected from the delivery wire when the marker and the region of interest overlap; A system comprising: (Appendix 53) the notification unit causes a display device that displays the image to display a distance between the region of interest and an edge of the image or a distance between the marker and the region of interest; Here, the notification unit can change the display manner of the distance on the display device depending on the distance, and the change in display manner includes changing the font, size, or color of characters displayed depending on the distance, changing the color of the entire screen of the display device or a portion of it depending on the distance, displaying a figure on the entire screen of the display device, outside the frame, or a portion of it, enlarging and displaying a region of interest depending on the distance, or changing the color or size of a mark added to the region of interest depending on the distance. (Appendix 54) 54. The system of any one of appendixes 51 to 53, wherein the notification unit can emit a notification sound or transmit vibration depending on the magnitude of the distance. (Appendix 55) 55. The system of any one of claims 51 to 54, wherein the distance is determined either by a straight-line distance or a distance along a blood vessel. (Appendix 56) A cerebral aneurysm coil embolization assist system, comprising: An image processing device; an image capturing device that captures an X-ray image of a patient with a guiding catheter and a delivery wire for an embolization coil inserted into a blood vessel and transmits the image to the image processing device; Equipped with The image processing device, a positional relationship storage unit that stores a positional relationship between the tip of the catheter and the second marker; a position storage unit that stores the distance a between the neckline of the aneurysm and the first marker and the position of the second marker at time t1; a distance estimating unit that calculates a moving distance b from the position of the second marker at time t2 and estimates a distance ab from the aneurysm neckline to the tip of the catheter; a notification unit that notifies a user of the estimated distance; A system comprising: (Appendix 57) 57. The system of claim 56, wherein the estimated distance is represented by a probability distribution. (Appendix 58) 58. The system of claim 56 or 57, wherein the estimated catheter tip position is displayed colored based on a probability distribution.

[0274] (Appendix 59) An endovascular surgery support system comprising: a memory unit that stores a product list of devices for intravascular examination or treatment; a recommendation unit that recommends products to use based on image analysis results, facility information, or user preference information; and a display unit that displays the recommended products.

[0275] (Appendix A1) An image processing device, An image processing device comprising an image acquisition unit capable of acquiring an image including at least an intravascular inspection or treatment device as a subject. (Appendix A2) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with The notification unit notifies the user on a condition that a device that has newly appeared in the image is newly detected as a region of interest. 13. An image processing device according to claim 12 . (Appendix A3) The image processing device according to claim 2, wherein the notification unit notifies the user on condition that a device that has newly appeared on an edge or within a specific region in the image is newly detected as a region of interest. (Appendix A4) The image processing device according to claim 3, wherein the specific area is designated by a user. (Appendix A5) The image processing device according to claim A3, wherein the specific area is automatically designated. (Appendix A6) The image processing device according to claim A5, wherein the specific area is an area that the image processing device automatically determines to be an area that should be confirmed. (Appendix A7) The image processing device according to any one of Additions A2 to A6, further comprising a tracking unit configured to track the region of interest in the image. (Appendix A8) The image processing device according to any one of Appendices A2 to A7, wherein the region of interest is set to include at least a part of an intravascular inspection or treatment device selected from the group consisting of a guiding catheter, a guidewire, an intermediate catheter, a microcatheter, a suction catheter for thrombus retrieval, a marker, a coil, a stent, a filter, an embolic material, an aneurysm embolization device, and a balloon. (Appendix A9) The image processing device according to any one of Addendum A2 to A8, wherein the region including at least a part of the device includes a tip of a device for intravascular inspection or treatment. (Appendix A10) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with The notification unit notifies the user on condition that the region of interest passes through a specific boundary line designated on the image. 13. An image processing device according to claim 12 . (Appendix A11) The image processing device according to claim A10, wherein the specific boundary line is specified by a user. (Appendix A12) The image processing device according to claim A10, wherein the specific boundary line is automatically specified. (Appendix A13) The image processing device according to any one of Additions A9 to A12, wherein the specific boundary line is represented by a straight line, a curve, a circle, a rectangle, or an arbitrary polygon or closed curve. (Appendix A14) The image processing device according to any one of Additions A9 to A13, wherein the specific boundary line is used to detect entry of the device into a blood vessel where entry is undesirable. (Appendix A15) The image processing device according to any one of Additions A9 to A14, wherein the notification unit notifies the user of one or more regions of interest only when the condition is satisfied for the first time. (Appendix A16) The image processing device according to any one of Additions A9 to A15, wherein the notification unit notifies the user only when the condition is satisfied for the first time in an entire region of interest of one or more regions. (Appendix A17) The image processing device according to any one of Additions A9 to A16, wherein the notification unit notifies the user of one or more specific regions designated by the user as a region of interest only when the condition is satisfied. (Appendix A18) The image processing device according to any one of Additions A9 to A16, wherein the notification unit notifies the user of the automatically identified one or more regions as a region of interest only when the condition is satisfied. (Appendix A19) The image processing device according to any one of Addendum A9 to A18, wherein the specific boundary line is displayed superimposed on an X-ray image. (Appendix A20) The image processing device according to any one of Additions A9 to A19, wherein the specific boundary line is redrawn in response to a change in the range or magnification of an X-ray image. (Appendix A21) The image processing device according to any one of Additions A9 to A20, wherein the notification is performed by issuing an alarm sound or by changing a display style of the border. (Appendix A22) The image processing device according to any one of Additions A9 to A21, wherein a different notification method is used depending on a direction in which the region of interest passes through a specific boundary line designated on the image. (Appendix A23) The image processing device according to any one of Additions A9 to A22, wherein different notification methods are used depending on a speed at which the region of interest crosses the particular boundary line. (Appendix A24) The image processing device according to any one of Additions A9 to A23, wherein the notification unit causes a display device that displays the image to display the distance between the region of interest and the specific boundary line. (Appendix A25) The image processing device according to any one of Additions A9 to A24, wherein the notification unit changes a display manner of the distance on the display device depending on a distance between the region of interest and the specific boundary line. (Appendix A26) The image processing device according to any one of Appendices A9 to A25, wherein the notification unit notifies the user on condition that a value obtained by dividing the distance between the area of ​​interest and the specific boundary line by a moving speed of the area of ​​interest within the image is less than a predetermined threshold. (Appendix A27) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with As the region of interest, a region of interest included within a region designated by a user on the image is acquired. 13. An image processing device according to claim 12 . (Appendix A28) The image processing device according to claim A27, wherein the area specified by the user on the image is represented by a rectangle, a circle, or an arbitrary closed curve. (Appendix A29) The image processing device according to claim A27 or A28, wherein candidates for areas that can be specified by the user are automatically displayed. (Appendix A30) The image processing device according to any one of Addition A27 to A29, wherein when a user designates an area, the image temporarily becomes a still image. (Appendix A31) The image processing device according to any one of Additions A27 to A29, wherein when a user designates an area, the image is temporarily played back with a delay. (Appendix A32) The image processing device according to any one of Addendum A27 to A31, wherein when a user specifies an area, a part of the image is enlarged and displayed. (Appendix A33) The image processing device according to any one of Appendices A27 to A32, wherein when a region including a plurality of regions of interest is specified, only the region of interest to be confirmed is automatically selected. (Appendix A34) The image processing device of Appendix A33, wherein the area to be confirmed is determined based on whether the tip of the guide wire is within a certain range, whether the tip of the stent delivery wire is within a certain range, the range over which the 2nd marker of the catheter moves to see when it is OK to cut the coil, whether the filter is within a certain range, whether the tip of the catheter is within a certain range, whether the embolic material is contained within a certain range, the range over which the device remains inside a large blood vessel, an aneurysm, or a critical blood vessel. (Appendix A35) 5. The image processing device of claim 33 or 34, wherein the region of interest to be identified is determined depending on a surgical context. (Appendix A36) The image processing device according to any one of Appendices A27 to A35, wherein the designated area is automatically adjusted when the image is changed. (Appendix A37) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a situation recognition unit that recognizes a situation of a specific surgery based on the image; The image processing device according to claim A1, (Appendix A38) The image processing device according to Appendix A37, wherein the specific surgical situation is at least one situation selected from the group consisting of surgical details, disease name, vascular leakage, vascular perforation, occurrence of thrombus, disappearance of peripheral blood vessels, guidance of guidewire / catheter / balloon, aspiration by catheter, stent placement, balloon expansion, coil insertion, insertion of an aneurysm embolization device, timing of cutting the coil, guidance / placement / retrieval of a filter, injection of liquid embolic material, injection of contrast agent, continuation of a still image for a certain period of time or more, discrimination between mask image and live image, discrimination of imaging area / angle, switching of images, and occurrence of noise. (Appendix A39) a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; The image processing device according to claim A37 or A38, further comprising: (Appendix A40) The image processing device according to claim A38, wherein acquisition of the region of interest is stopped when occurrence of noise is recognized. (Appendix A41) The image processing device according to claim A38, wherein when a still image is recognized to continue for a certain period of time or more, a recommendation to reduce exposure is notified. (Appendix A42) The image processing device according to any one of Appendices A37 to A41, further comprising a recording unit that records the recognized specific surgical situation. (Appendix A43) The image processing device according to any one of Addendum A37 to A42, wherein the specific surgical situation is recognized by a pattern matching, an image recognition, a time series image recognition, a time series difference, or an object detection algorithm. (Appendix A44) The image processing device according to any one of Additions A37 to A43, wherein the recognized device is identified according to the recognized specific surgical situation. (Appendix A45) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a recording unit that records a surgical situation corresponding to a condition when at least one of the regions of interest satisfies the condition defined for the region of interest; The image processing device according to claim A1, (Appendix A46) The image processing device according to appended claim A45, wherein the conditions include at least one procedure selected from the group consisting of puncture, reaching of the guiding catheter to the treatment site, balloon guidance, balloon expansion, stent deployment, catheter guidance, first coil insertion, second or subsequent coil insertion, localization of the coil within the aneurysm, deviation of the coil into the parent vessel, catheter removal, coil removal, and end of procedure. (Appendix A47) The image processing device of any one of appendix A45 to A46, wherein the recorded surgical situation includes still images, video, and / or text information describing the surgical situation. (Appendix A48) The image processing device according to appendix A47, wherein the video is a video of a certain period before and after the point in time when the condition is satisfied. (Appendix A49) The image processing device according to any one of Appendices A45 to A48, wherein the recorded information is modifiable by a user. (Appendix A50) The image processing device according to any one of Appendices A45 to A49, further comprising a report creation unit that creates a report based on the recorded information. (Appendix A51) The image processing device of Appendix A50, wherein the report creation unit creates a summary of the surgery based on recorded still images, videos, and / or text information describing the status of the surgery. (Appendix A52) The image processing device according to any one of Appendices A45 to A51, further comprising a notification unit that notifies a user when a situation of a specific surgery is recorded. (Appendix A53) The image processing device according to any one of Appendices A45 to A52, further comprising a notification unit that notifies the user according to the elapsed time after the situation of a specific surgery is recorded. (Appendix A54) The image processing device of claim A53, wherein the time lapse after a particular surgical situation is recorded is determined based on a balloon inflation time, a guidewire travel time, speed, or acceleration, or a coil travel time. (Appendix A55) The image processing device according to any one of Additions A45 to A54, further comprising a notification unit that notifies the user according to the number of times a particular surgical situation has been recorded. (Appendix A56) The imaging device of claim A55, wherein the particular surgical situation is insertion of a coil. (Appendix A57) The image processing device according to any one of Additions A45 to A56, wherein the image is acquired from a moving image captured in real time or a moving image recorded in the past. (Appendix A58) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with a display unit that displays, on a display device connected to the image processing device, the image including the region of interest and a mark corresponding only to a vertical coordinate of the region of interest beside the image; 13. An image processing device according to claim 12 . (Appendix A59) The image processing device of claim A58, wherein the trajectory of the mark is displayed for a period of time. (Appendix A60) The image processing device according to claim A60 or A61, in which the display format of the mark is different for each area of ​​interest. (Appendix A61) The image processing device according to appendix A60, wherein the display format is a color, a shape, a pattern, a symbol, a character, a caption, or an animation. (Appendix A62) The image processing device according to any one of Addendum A58 to A61, which automatically recognizes and highlights a region of interest to be confirmed. (Appendix A63) The image processing device of claim A62, wherein the region of interest to be confirmed is determined depending on the surgical context. (Appendix A64) The image processing device according to any one of Addendum A58 to A63, wherein only a region of interest designated by a user is displayed. (Appendix A65) The image processing device according to any one of Addendum A58 to A64, which displays only an automatically selected region of interest. (Appendix A66) The image processing device according to any one of Additions A58 to A65, wherein when the mark exceeds a boundary value designated by the user, a notification is given to the user. (Appendix A67) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with When a plurality of regions of interest are tracked, a display unit is further provided which displays the plurality of regions of interest in different display formats. 13. An image processing device according to claim 12 . (Appendix A68) The image processing device according to appendix A67, wherein the display format is a color, a shape, a pattern, a symbol, a character, a caption, or an animation. (Appendix A69) 9. The image processing apparatus of claim 8, further comprising: an image processor configured to automatically recognize and highlight an area of ​​interest for review. (Appendix A70) The image processing device according to any one of Appendices A67 to A69, wherein the region of interest to be confirmed is determined according to a surgical situation. (Appendix A71) The image processing device according to any one of Appendices A67 to A70, which displays only a region of interest designated by a user. (Appendix A72) The image processing device according to any one of Addendum A67 to A71, which displays only an automatically selected region of interest.

[0276] (Appendix B1) An image processing device, An image processing device comprising an image acquisition unit capable of acquiring an image including at least an intravascular inspection or treatment device as a subject. (Appendix B2) An image processing device, an image acquisition unit for acquiring images for intravascular inspection or treatment; a region of interest acquisition unit that acquires, when the image includes an intravascular inspection or treatment device, one or more regions including at least a part of the device as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with The notification unit notifies the user on a condition that a device that has newly appeared in the image or within a specific region, or a part thereof, is newly detected as a region of interest. 13. An image processing device according to claim 12 . (Appendix B3) An image processing device, an image acquisition unit for acquiring images for intravascular inspection or treatment; a region of interest acquisition unit that acquires, when the image includes an intravascular inspection or treatment device, one or more regions including at least a part of the device as a region of interest; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with The output unit outputs to the equipment on the condition that a device or a part thereof that has newly appeared in the image or within a specific region is newly detected as a region of interest. 13. An image processing device according to claim 12 . (Appendix B4) The image processing device according to claim 2 or 3, wherein the notification unit notifies the user on condition that a device or a part thereof that has newly appeared at an edge of the image or within a certain distance from the edge is newly detected as a region of interest. (Appendix B5) The image processing device according to appendix B2 or B3, wherein the output unit outputs to the equipment on the condition that a device or a part thereof that has newly appeared within an edge in the image or within a certain distance from the edge, or within a specific area, is newly detected as a region of interest. (Appendix B6) The image processing device according to claim B2 or B3, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B7) The image processing device according to Appendix B6, wherein the plurality of screens display two or more images selected from a front live image, a side live image, a front mask image, and a side mask image. (Appendix B8) The image processing device of claim B6, further comprising: (Appendix B9) The image processing device according to claim B2 or B3, wherein the specific area is designated by a user. (Appendix B10) The image processing device according to claim B2 or B3, wherein the specific area is automatically designated. (Appendix B11) The image processing device according to claim 2 or 3, wherein the specific area is an area that the image processing device automatically determines to be an area that should be confirmed. (Appendix B12) The image processing device according to claim B2 or B3, further comprising a tracking unit configured to track the region of interest in the image. (Appendix B13) The image processing device according to claim B2 or B3, further comprising a blood vessel recognition unit that recognizes at least a portion of a blood vessel in the image. (Appendix B14) The image processing device of claim B2 or B3, wherein a notification or output is provided when the region of interest falls within a blood vessel having a diameter equal to or less than a threshold, or within a manually or automatically designated blood vessel. (Appendix B15) The image processing device according to appendix B2 or B3, wherein the region of interest is set to include at least a part of a device for inspection or treatment within a blood vessel selected from the group consisting of a guiding catheter, a guidewire, an intermediate catheter, a microcatheter, a suction catheter for thrombus retrieval, a marker, a coil, a stent, a filter, an embolic material, an aneurysm embolization device, and a balloon. (Appendix B16) The image processing device of claim 2 or 3, wherein the region including at least a portion of the device includes a tip of a device for intravascular inspection or treatment.

[0277] (Appendix B17) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with The image processing device according to claim 1, wherein the notification unit notifies the user on a condition that the region of interest passes through or is predicted to pass through a specific boundary line specified on the image. (Appendix B18) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with The image processing device according to claim 1, wherein the output unit notifies the user on a condition that the region of interest passes through or is predicted to pass through a specific boundary line specified on the image. (Appendix B19) The image processing device according to any one of claims 17 to 18, further comprising a tracking unit for tracking the region of interest in the image. (Appendix B20) The image processing device according to claim B17 or B18, further comprising a blood vessel recognition unit that recognizes at least a portion of a blood vessel in the image. (Appendix B21) The image processing device of claim B17 or B18, wherein a notification or output is provided when the region of interest falls within a blood vessel having a diameter equal to or less than a threshold, or within a manually or automatically designated blood vessel. (Appendix B22) The image processing device according to any one of claims 17 to 18, wherein the prediction of passage is based on a distance between the region of interest and a boundary line, or on the distance / velocity of the region of interest. (Appendix B23) The image processing device according to claim B17 or B18, wherein the specific boundary line is specified by a user. (Appendix B24) The image processing device according to claim B17 or B18, wherein the specific boundary line is automatically specified. (Appendix B25) The image processing device according to claim B17 or B18, wherein the specific boundary line is automatically designated when the region of interest has not moved for a certain period of time. (Appendix B26) The image processing device according to appendix B17 or B18, wherein the specific boundary line is automatically specified depending on the surgical situation or is automatically proposed and the user can accept or reject it. (Appendix B27) The image processing device according to appendix B17 or B18, wherein the specific boundary line is automatically released or automatically proposed depending on the surgical situation and the user can accept or reject it. (Appendix B28) The image processing device according to claim B17 or B18, wherein, when the display device that displays the image has a plurality of screens, the specific boundary line specified on one screen is automatically specified at a corresponding position on the other screens. (Appendix B29) The image processing device according to claim B17 or B18, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B30) The image processing device of Appendix B29, wherein the multiple screens display two or more images selected from a front live image, a side live image, a front mask image, and a side mask image. (Appendix B31) The image processing device of claim B29, further comprising: (Appendix B32) The image processing device according to appendix B17 or B18, wherein the specific boundary line is represented by a straight line, a curve, a circle, a rectangle, or an arbitrary polygon or closed curve. (Appendix B33) The image processing device according to claim B17 or B18, wherein the boundary line can be moved, deformed, enlarged, or reduced by a user operation. (Appendix B34) The image processing device according to claim B17 or B18, wherein the specific boundary line is used to detect a blood vessel where entry of the device is undesirable, a diseased area such as an aneurysm or stenosis, or entry into an extravascular space. (Appendix B35) The image processing device according to claim B17 or B18, wherein the notification unit or output unit notifies or outputs only when the condition is satisfied for the first time for each of one or more regions of interest. (Appendix B36) The image processing device according to appendix B17 or B18, wherein the notification unit or output unit notifies or outputs for each of one or more regions of interest only when the condition is satisfied for the first time after a specific point in time. (Appendix B37) The image processing device according to appendix B17 or B18, wherein the notification unit or output unit performs notification or output only when the condition is satisfied for the first time in the entire region of interest of one or more regions. (Appendix B38) The image processing device according to appendix B17 or B18, wherein the notification unit or output unit performs notification or output only when the condition is satisfied for the first time after a specific point in time in the entire region of interest of one or more regions. (Appendix B39) The image processing device according to claim B17 or B18, wherein the notification unit or output unit notifies or outputs a specific area or areas of interest designated by a user only when the condition is satisfied. (Appendix B40) The image processing device according to claim B17 or B18, wherein the notification unit or output unit notifies or outputs the automatically identified one or more regions of interest only when the condition is satisfied. (Appendix B41) The image processing device according to claim B17 or B18, wherein the specific boundary line is superimposed on an X-ray image. (Appendix B42) The image processing device according to claim B17 or B18, wherein the particular boundary line is redrawn in response to a change or movement of the range or magnification of the X-ray image. (Appendix B43) The image processing device of claim B17 or B18, wherein the particular boundary line is redrawn after an interruption and reacquisition of an X-ray image. (Appendix B44) The image processing device according to claim B17, wherein the notification is performed by emitting an alarm sound or by changing a display style of the border. (Appendix B45) The image processing device according to claim B17, wherein the notification conveys some or all of the information regarding which screen, which device, and what event occurred by voice. (Appendix B46) The image processing device of claim B17, wherein different notification methods are used depending on a direction in which the region of interest passes through a particular boundary line specified on the image. (Appendix B47) The image processing device according to appendix B17 or B18, wherein the direction in which the region of interest passes through a particular boundary line specified on the image is automatically recognized or automatically proposed depending on the surgical situation and the user can accept or reject it. (Appendix B48) The image processing device of claim B17, wherein different notification methods are used depending on the speed at which the region of interest crosses the particular boundary line. (Appendix B49) The image processing device according to claim B17, wherein the notification unit causes a display device that displays the image to display a distance between the region of interest and the specific boundary line. (Appendix B50) The image processing device according to claim B17, wherein the notification unit changes a display manner of the distance on the display device depending on a distance between the region of interest and the specific boundary line. (Appendix B51) The image processing device according to claim 17 or 18, wherein the notification unit or output unit notifies or outputs on the condition that a value obtained by dividing the distance between the region of interest and the specific boundary line by a moving speed of the region of interest within the image is less than a predetermined threshold.

[0278] (Appendix B52) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with As the region of interest, a region of interest included within a region designated by a user on the image is acquired. 13. An image processing device according to claim 12 . (Appendix B53) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with As the region of interest, a region of interest included within a region designated by a user on the image is acquired. 13. An image processing device according to claim 12 . (Appendix B54) The image processing device of any one of claims 1 to 5, further comprising a tracking unit for tracking the region of interest in the image. (Appendix B55) The image processing device according to claim B52 or B53, further comprising a blood vessel recognition unit that recognizes at least a portion of a blood vessel in the image. (Appendix B56) The image processing device of any one of claims 1 to 5, wherein a notification or output is provided when the region of interest falls within a blood vessel having a diameter equal to or less than a threshold, or within a manually or automatically designated blood vessel. (Appendix B57) The image processing device according to claim B52 or B53, wherein the area specified by the user on the image is represented by a rectangle, a circle, or an arbitrary closed curve. (Appendix B58) The image processing device according to claim B52 or B53, wherein candidates for areas that can be specified by the user are automatically displayed. (Appendix B59) The image processing device according to claim B52 or B53, wherein candidates for areas that can be specified by the user are automatically displayed according to the surgical situation. (Appendix B60) The image processing device according to claim B52 or B53, wherein the image temporarily becomes a still image when the user designates an area. (Appendix B61) The image processing device according to claim B52 or B53, wherein when a user specifies an area, the image is temporarily played back with a delay. (Appendix B62) The image processing device according to claim B52 or B53, wherein a portion of the image is enlarged when a user specifies an area. (Appendix B63) The image processing device according to claim B52 or B53, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B64) The image processing device of Appendix B63, wherein the multiple screens display two or more images of a front live image, a side live image, a front mask image, and a side mask image. (Appendix B65) The image processing device of claim 63, wherein information from multiple screens is utilized to improve processing accuracy. (Appendix B66) The image processing device of any one of claims 1 to 5, wherein when a region including multiple regions of interest is specified, only the region of interest to be confirmed is automatically selected. (Appendix B67) The image processing device of claim B66, wherein the area of ​​interest to be confirmed is determined based on the importance of multiple devices, the position of the area of ​​interest on the screen, and the surgical scene. (Appendix B68) The image processing device of claim B66, wherein the region of interest to be confirmed is determined depending on the surgical context. (Appendix B69) The image processing device of any one of claims 1 to 5, wherein the designated region is automatically adjusted if the image is changed.

[0279] (Appendix B70) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a situation recognition unit that recognizes a situation of a specific surgery based on the image; The image processing device according to claim B1, (Appendix B71) The image processing device described in Appendix B70, wherein the specific surgical situation is at least one situation selected from the group consisting of surgical details, disease name, vascular leakage, vascular perforation, occurrence of thrombus, disappearance of peripheral blood vessels, guidance of guidewire / catheter / balloon, aspiration by catheter, stent placement, balloon expansion, coil insertion, insertion of aneurysm embolization device, timing of cutting coils, guidance / placement / retrieval of filter, injection of liquid embolic material, injection of contrast agent, continuation of still image for a certain period of time or more, discrimination between mask image and live image, discrimination of imaging area / angle, image switching, and occurrence of noise. (Appendix B72) a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; The image processing device according to claim B70 or B71, further comprising: (Appendix B73) The image processing device according to claim B70, wherein acquisition of the region of interest is stopped when occurrence of noise is recognized. (Appendix B74) The image processing device according to claim B70, wherein when an injection of a contrast agent is recognized, or when a 3D image is acquired, or when a cone beam CT (CBCT) is acquired, acquisition of the region of interest is stopped, or acquisition is performed but no notification is provided. (Appendix B75) The image processing device according to claim B70, wherein when a still image is recognized to continue for a certain period of time or longer, a recommendation to reduce exposure is notified. (Appendix B76) The image processing device according to any one of appendices B70 and B71, further comprising a recorder for recording the recognized specific surgical situation. (Appendix B77) The image processing device of any one of appendices B70 and B71, wherein the particular surgical situation is recognized by a pattern matching, image recognition, time series image recognition, time series differencing, or object detection algorithm. (Appendix B78) The image processing apparatus of any one of claims 70 to 71, wherein the recognized device is identified depending on the recognized specific surgical situation. (Appendix B79) An image processing device as described in appendix B70 or B71, in which a specific boundary line specified on the image is automatically specified or proposed around the region of interest for notifying the user on the condition that the region of interest passes through the specific boundary line specified on the image depending on the recognized specific surgical situation. (Appendix B80) The image processing device according to claim B70 or B71, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B81) The image processing device of Appendix B80, wherein the multiple screens display two or more images of a front live image, a side live image, a front mask image, and a side mask image. (Appendix B82) The image processing device according to claim B80, which utilizes information from a plurality of images to improve processing accuracy.

[0280] (Appendix B83) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a recording unit that records a surgical situation corresponding to a condition when at least one of the regions of interest satisfies the condition defined for the region of interest; The image processing device according to claim B1, (Appendix B84) The image processing device according to Appendix B83, wherein the conditions include at least one procedure selected from the group consisting of puncture, reaching of the guiding catheter to the treatment site, balloon guidance, balloon expansion, stent deployment, catheter guidance, first coil insertion, second or subsequent coil insertion, localization of the coil within the aneurysm, deviation of the coil into the parent vessel, catheter removal, coil removal, and end of procedure. (Appendix B85) The image processing device of any one of appendix B83 and B84, wherein the recorded surgical situation includes still images, video, and / or text information describing the surgical situation. (Appendix B86) The image processing device according to appendix B83 or B84, wherein the moving image is a moving image for a certain period before and after the point in time when the condition is satisfied. (Appendix B87) The image processing device of any one of appendix B83 and B84, wherein the recorded information is modifiable by a user. (Appendix B88) The image processing device according to claim B83 or B84, further comprising a report creation unit that creates a report based on the recorded information. (Appendix B89) The image processing device of claim B83 or B84, wherein the report creation unit creates a summary of the surgery based on the recorded still images, videos, and / or text information describing the status of the surgery. (Appendix B90) The image processing device according to appendix B83 or B84, further comprising a notification unit for notifying a user or an output unit for performing output when a specific surgical situation is recorded. (Appendix B91) The image processing device according to appendix B83 or B84, further comprising a notification unit for notifying a user or an output unit for performing output depending on the time elapsed after the status of a particular surgery is recorded. (Appendix B92) The image processing device of claim B91, wherein the time lapse after a particular surgical situation is recorded is determined based on a balloon inflation time, a guidewire travel time, speed, or acceleration, or a coil travel time. (Appendix B93) The image processing device according to appendix B83 or B84, further comprising a notification unit for notifying a user or an output unit for performing output depending on the number of times a particular surgical situation has been recorded. (Appendix B94) The imaging device of any one of claims B83 to B84, wherein the particular surgical situation is insertion of a coil. (Appendix B95) The image processing device of claim B83 or B84, wherein the image is obtained from a video captured in real time or a video recorded in the past. (Appendix B96) The image processing device according to claim B83 or B84, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B97) The image processing device of Appendix B96, wherein the multiple screens display two or more images of a front live image, a side live image, a front mask image, and a side mask image. (Appendix B98) The image processing device of claim 96, wherein information from multiple screens is utilized to improve processing accuracy.

[0281] (Appendix B99) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with a display unit that displays, on a display device connected to the image processing device, the image including the region of interest and a mark corresponding only to a vertical coordinate of the region of interest beside the image; 13. An image processing device according to claim 12 . (Appendix B100) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with a display unit that displays, on a display device connected to the image processing device, the image including the region of interest and a mark corresponding only to a vertical coordinate of the region of interest beside the image; 13. An image processing device according to claim 12 . (Appendix B101) The image processing device of any one of claims 1 to 5, further comprising a tracking unit for tracking the region of interest in the image. (Appendix B102) The image processing device according to claim B99 or B100, further comprising a blood vessel recognition unit that recognizes at least a portion of a blood vessel in the image. (Appendix B103) The image processing device of claim B99 or B100, wherein a notification or output is provided when the region of interest falls within a blood vessel having a diameter equal to or less than a threshold, or within a manually or automatically designated blood vessel. (Appendix B104) The image processing device of claim B99 or B100, wherein a trajectory of the mark is displayed for a certain period of time. (Appendix B105) The image processing device according to claim B99 or B100, wherein the display format of the mark is different for each region of interest. (Appendix B106) The image processing device according to appendix B99 or B100, wherein the display format is a color, a shape, a pattern, a symbol, a character, a caption, or an animation. (Appendix B107) The image processing device of claim B99 or B100, which automatically recognizes and highlights an area of ​​interest to be confirmed. (Appendix B108) The image processing device of claim B92, wherein the region of interest to be confirmed is determined according to a surgical context. (Appendix B109) The image processing device of claim B99 or B100, which displays only a region of interest specified by a user. (Appendix B110) The image processing device of claim B99 or B100, which displays only an automatically selected region of interest. (Appendix B111) The image processing device of any one of claims 1 to 5, further comprising: a notification to a user when the mark exceeds a boundary value specified by the user. (Appendix B112) The image processing device according to claim B99 or B100, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B113) The image processing device of claim B112, wherein the plurality of screens display two or more images selected from the group consisting of a front live image, a side live image, a front mask image, and a side mask image. (Appendix B114) The image processing device of claim B112, wherein information from multiple screens is utilized to improve processing accuracy.

[0282] (Appendix B115) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with When a plurality of regions of interest are tracked, a display unit is further provided which displays the plurality of regions of interest in different display formats. 13. An image processing device according to claim 12 . (Appendix B116) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a tracking unit for tracking each of the regions of interest in the image; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with When a plurality of regions of interest are tracked, a display unit is further provided which displays the plurality of regions of interest in different display formats. 13. An image processing device according to claim 12 . (Appendix B117) The image processing device according to claim B115 or B116, further comprising a blood vessel recognition unit that recognizes at least a portion of a blood vessel in the image. (Appendix B118) The image processing device of claim B117, wherein a notification or output is provided when the region of interest falls within a blood vessel having a diameter equal to or less than a threshold, or within a manually or automatically designated blood vessel. (Appendix B119) The image processing device according to appendix B115 or B116, wherein the display format is a color, a shape, a pattern, a symbol, a character, a caption, or an animation. (Appendix B120) 7. The image processing device of claim 115 or 116, which automatically recognizes and highlights an area of ​​interest to be confirmed. (Appendix B121) The image processing device of claim B120, wherein the region of interest to be confirmed is determined depending on a surgical context. (Appendix B122) The image processing device of any one of claims 1 to 6, wherein only a region of interest specified by a user is displayed. (Appendix B123) The image processing device of any one of claims 115 to 116, wherein only an automatically selected region of interest is displayed. (Appendix B124) The image processing device according to claim B115 or B116, wherein, in a case where the display device for displaying the image has a plurality of screens, processing is performed for each screen. (Appendix B125) The image processing device of claim B124, wherein the plurality of screens display two or more images selected from the group consisting of a front live image, a side live image, a front mask image, and a side mask image. (Appendix B126) The image processing device of claim B124, wherein information from multiple screens is utilized to improve processing accuracy.

[0283] (Appendix B127) A method for creating a machine learning model, characterized in that training is performed using cerebral vascular images obtained from multiple directions as input. (Appendix B128) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; Equipped with The image processing device according to claim 1, wherein the notification unit issues a notification when the region of interest includes a guidewire and the length of the guidewire or the area of ​​segmentation within a specific region including the tip of the guidewire exceeds a certain threshold. (Appendix B129) An image processing device, an image acquisition unit for acquiring an image including at least an intravascular inspection or treatment device as a subject; a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; an output unit that outputs information about at least one of the regions of interest that satisfies a condition defined for each of the regions of interest to a device connected to the image processing device; Equipped with The image processing device according to claim 1, wherein the output unit outputs when the region of interest includes a guidewire and the length of the guidewire or the area of ​​segmentation within a specific region including the tip of the guidewire exceeds a certain threshold. [Explanation of symbols]

[0284] 1. Image processing device 10...Storage section 11. Control section 110...Image acquisition unit 111... Region of interest acquisition unit 112 Tracking Unit 113... Notification Department 114 Marker detection unit 115... Distance measurement section 116 Output section 117...Video storage unit 118...Video extraction section 119... State Estimation Unit 2...Display device 3. X-ray imaging device 4. External Devices 20 CPU 21. ROM 22...RAM 23. Storage 24 Input / Output Interface 25 Input section 26 Output section 27...Storage medium 30...X-ray irradiator 31. X-ray detector 32...Bed D...Device E···Embolization coil P...Subject S...Image Processing System

Claims

1. An image processing device, an image acquisition unit that acquires an image including at least an intravascular inspection or treatment device as a subject; a situation recognition unit that recognizes a situation of a specific surgery based on the image; An image processing device comprising:

2. 2. The image processing device of claim 1, wherein the specific surgical situation is at least one situation selected from the group consisting of surgical details, disease name, vascular leakage, vascular perforation, thrombus formation, peripheral blood vessel disappearance, guide wire / catheter / balloon guidance, catheter aspiration, stent placement, balloon expansion, coil insertion, aneurysm embolization device insertion, timing of coil removal, filter guidance / placement / retrieval, injection of liquid embolic material, injection of contrast agent, continuation of still images for a certain period of time or more, discrimination between mask image and live image, discrimination of imaging region / angle, image switching, and noise occurrence.

3. a region of interest acquisition unit that acquires one or more regions including at least a part of the device included in the image as a region of interest; a notification unit that notifies a user of the image processing device when at least one of the regions of interest satisfies a condition defined for each of the regions of interest; 3. The image processing device according to claim 1, further comprising:

4. 4. The image processing device according to claim 3, wherein acquisition of the region of interest is stopped when occurrence of noise is recognized.

5. 4. The image processing device according to claim 3, wherein when an injection of a contrast agent is recognized, or when a 3D image is acquired, or when a cone beam CT (CBCT) is acquired, acquisition of the region of interest is stopped, or acquisition is performed but no notification is provided.

6. 4. The image processing device according to claim 3, wherein when it is recognized that a still image continues for a certain period of time or more, a recommendation to reduce exposure to radiation is given.

7. 3. The image processing device according to claim 1, further comprising a recording unit that records the recognized specific surgical situation.

8. The image processing device according to claim 1 or 2, wherein the specific surgical situation is recognized by a pattern matching, image recognition, time series image recognition, time series difference, or object detection algorithm.

9. 3. The image processing device according to claim 1, wherein the device to be recognized is specified according to the recognized specific surgical situation.

10. 3. The image processing device of claim 1, wherein a specific boundary line is automatically specified or suggested around the region of interest to notify the user on the condition that the region of interest passes through the specified boundary line on the image, depending on the recognized specific surgical situation.

11. 3. The image processing device according to claim 1, wherein when the display device for displaying the image has a plurality of screens, processing is performed for each screen.

12. The image processing device according to claim 11 , wherein the plurality of screens display two or more images selected from the group consisting of a live front image, a live side image, a masked front image, and a masked side image.

13. 12. The image processing device according to claim 11, wherein information from a plurality of screens is used to improve processing accuracy.

14. The image processing device according to claim 1, a region of interest acquisition unit that acquires, as a region of interest, one or more regions including at least a part of an intravascular inspection or treatment device included in an image that includes at least the device as a subject; a recording unit that records a surgical situation corresponding to a condition when at least one of the regions of interest satisfies the condition defined for each of the regions of interest; The image processing device further comprises:

15. 15. The image processing device according to claim 14, wherein the condition includes at least one procedure selected from the group consisting of puncture, reaching of the guiding catheter to the treatment site, balloon guidance, balloon dilation, stent deployment, catheter guidance, first coil insertion, second or subsequent coil insertion, localization of the coil within the aneurysm, deviation of the coil into the parent vessel, catheter removal, coil removal, and end of treatment.

16. 16. The image processing device according to claim 14, wherein the recorded surgical situation includes still images, videos, and / or text information describing the surgical situation.

17. 16. The image processing device according to claim 14, wherein the moving image is a moving image for a certain period before and after the time when the condition is satisfied.

18. 16. The image processing device according to claim 14, wherein the recorded information is modifiable by a user.

19. 16. The image processing device according to claim 14, further comprising a report creation unit that creates a report based on the recorded information.

20. 20. The image processing device according to claim 19, wherein the report creation unit creates a summary of the surgery based on the recorded still images, videos, and / or text information explaining the status of the surgery.

21. 16. The image processing device according to claim 14, further comprising a notification unit that notifies a user or an output unit that performs output when a specific surgical situation is recorded.

22. 16. The image processing device according to claim 14, further comprising a notification unit that notifies a user or an output unit that performs output depending on the time that has elapsed since the status of a specific surgery was recorded.

23. 23. The image processing device of claim 22, wherein the time lapse after a particular surgical situation is recorded is determined based on the inflation time of the balloon, the movement time, speed, or acceleration of the guide wire, or the movement time of the coil.

24. 16. The image processing device according to claim 14, further comprising a notification unit that notifies a user or an output unit that performs output depending on the number of times a specific surgical situation is recorded.

25. 16. The image processing device according to claim 14 or 15, wherein the particular surgical situation is the insertion of a coil.

26. 16. The image processing device according to claim 14, wherein the image is obtained from a video captured in real time or a video recorded in the past.

27. 16. The image processing device according to claim 14, wherein when the display device for displaying the image has a plurality of screens, processing is performed for each screen.

28. 28. The image processing device of claim 27, wherein the plurality of screens display two or more images selected from the group consisting of a live front image, a live side image, a masked front image, and a masked side image.

29. 28. The image processing device of claim 27, wherein information from multiple screens is used to improve processing accuracy.

Images